Answer:
Yes, in most cases a person can counter sue or counter claim. Of course you need to check with you specific legal jurisdiction.
Notwithstanding that caveat, if what you are asking is "can I sue someone for having sued me", generally, the answer is "yes." A defendant may counterclaim or file a claim in the same cause against the Plaintiff for "malicious prosecution in a civil matter," (or something to that effect based on the jurisdiction of the law.)
You can also sue under a different set of facts or understanding. I.e. Joe sue Anne alleging that Anne owes Joe $10,000 for back rent. Anne could also Sue Joe alleging that he owes her $2,000 to finish the bathroom renovation. Anee withheld the $10,000 as payment for the bathroom renovation. These are two separate, yet connected cases.
When the counterclaim arises from the same transaction or occurrence as the main claim, it is generally referred to as a "compulsory counterclaim", or words to that effect. It is significant, because if the claim is not asserted as a counterclaim, many jurisdictions will prevent it from being brought as a separate lawsuit. In contrast, if a claim exists against the same party, but arises from a different transaction or occurrence, it generally may, but does not have to be asserted within the main claim-it may be brought as a separate action. Again, it is important that the law of the jurisdiction be researched to determine governing rules.
Unfortunately, such a cause of action usually cannot be heard until the original matter has been heard. The burden is still on the party who brings a cause of action to prove that cause of action. That means, the Defendant would have to show all the necessary elements to prove malicious prosecution of a civil matter.
As an aside, many attorneys will expressly request sanctions against another party for frivilous or vexatious litigation.