Generally, the phrase used in American English is "On the contrary." However, that was not your question was, "Does to the contrary make sense?" The answer to your question is, yes it does make sense. While, On the contrary is the preferred usage, still to the contrary makes sense. On the other hand, even the French au contraire makes sense!