Answer:
Not concrete on this, But I would believe its origin lies in "Old Wive's Tales", that generally revolve around what was observed by being near or in hay and gaining a fever afterwards.
Hay Fever is another term for seasonal allergies. Seasonal allergies are often pollen related, and hay tends to be a producer of particularly allergy inducing pollen.