If the brakes are worn and have worn a groove in the brake drum, you won't be able to remove the drum without first backing off the brakes. At the bottom of the inside of the backing plate you should see a slot. Use a small pry bar or a screwdriver to back off the adjuster.