Yes - Websters defines a vermin as:
"1 a: small common harmful or objectionable animals (as lice or fleas) that are difficult to control b: birds and mammals that prey on game c: animals that at a particular time and place compete (as for food) with humans or domestic animals"
Note that any raccoon would gladly compete with you for food.