Not at all
I've seen doors done in a natural finish hung in painted frames that looked great.
and doors done in the same color as the walls while the door frame was in the trim color.
Be brave, the worst thing that can happen is you'll have to paint the door over, how long will that take?