to prove such a serious accusation, the children would have to be very neglected, eg: no nutritional meals, unkempt appearance, physical abuse etc..this would need documentation by a Social service...
The factors used to determine that a parent is unfit are generally governed by state laws with child endangerment being the determining factor. The following include some of the reasons a parent may...
A cheating wife is not necessarily an unfit mother, unless she is not looking after the children properly and that fact isn't only the purview of a cheating wife.
I would think you would need to prove it in court. Whatever area you think they are unfit, you will need physical proof, not just hearsay, for example: drug addiction, mental illness, sexual abuse,...