Not if you have enough credible witnesses to this oral defamation, and can prove that it somehow harmed you (loss of business, etc). You may have to bring the witnesses in person, rather than by...
The best way to prove your were someplace is to have a witness. Another way, is to record it with a camera. Another possibility is to have your name on a sign in sheet, like at a doctor's office.
Ask him what aspect of your relationship makes him feel uncertain or insecure. Listen to what he says; but make the decision that is healthiest for you overall.
If he says that you can only "prove"...
You have to be found guilty by a preponderance of the evidence. It is very difficult to prove a negative, however, a good prosecutor can present the facts in a manner that makes an innocent man...