You said the magic word 'justly' and the ex wife is hurt and angry. She poured her life into her husband and he left her for you. You and her ex husband knew it wasn't going to be easy when he left her. It's his responsibility! No one twisted his arm to do what he did. You don't need to be involved. He had the affair with you and he walked out on his wife and possibly children, so it's up to him to get a backbone and deal with his ex wife.