Answer:
No. it is not valid to dismiss anyone's opinion whose first language is not English. The two have nothing to do with each other. I can explain things in my field of expertise just as well in several different languages.
On the other hand if the person's English is not at the point where he or she can not understand the question or follow the argument, that is different. In that case, the opinion may not be on topic and have absolutely nothing to do with the subject matter at hand. In that case the opinion may be dismissed not so much as a bad answer but as an irrelevant answer.
Answer
It is harder to honor the person's opinion when you are having difficulty understanding the words. However, you may really miss something if you don't consider the meaning. Plenty of slow talking southerners have "snowed" people who thought they were smarter than the bumpkin talking to them.