Answer:
I would think you would need to prove it in court. Whatever area you think they are unfit, you will need physical proof, not just hearsay, for example: drug addiction, mental illness, sexual abuse, neglect or abandonment. I would talk to your local Social Services and ask them what steps need to be taken.