Answer:
If you're the one married to her, then it's your wife, so yes.
But if you're just another individual on the outside of the marriage, no, it is not.
Though it wouldn't solely be on you if she agreed to have sex with you, you could and would still be partly responsible for hurt feelings, confusion, anger, and possible regret left in the wake of the affair.