Answer:
Having an affair with a married man, leaves some kind of dirt on your conscience. Everyone knows it's wrong. No matter what the circumstances are. He is just married to someone else.
He loved or loves his wife, then there's you.
In the middle unfortunately.
You don't have to be responsible for letting the wife know of her husband's terrible mistakes, though unless you are quitting this affair with him.
Otherwise telling the wife about it just because you want to 'see' her hurt is unfortunately not spiritually or intellectually productive for your growth in this world.