All moral issues ultimately come down to suffering and happiness. It is by definition immoral to increase the suffering of someone. That's why it shouldn't be considered immoral to do drugs, because you're not increasing the suffering of anyone, except, perhaps yourself.
Racism is a moral issue because it has the potential to directly increase the suffering of others. If, for example, a police officer is a racist, he is very likely to increase the suffering of racial minorities by targeting them for abuse, arrest, or any other form of unfair treatment.