By definition, if you are afraid of someone being gay for irrational reasons, you are a homophobe. However, if you merely believe it to be immoral, you may still be referred to as a homophobe, which may be confusing because the disapproval of homosexuality is not the same as the irrational fear of homosexuality.
So, one way to put it: if you have a fear of homosexuals that cannot be explained, you are a homophobe. If you disapprove of homosexuality for some sort of reason, you may still be referred to as a homophobe, even though you are not necessarily "afraid" of them, much less irrationally afraid.