Polygamy is illegal and it is called gossip if people remark on it. It is no one's business, but the man and any women involved or the law if he is reported.
It is unreasonable to expect a father to stop talking to his son. You may not realize how really inappropriate and potentially harmful it is to do that. You don't give details, and if there are legal...