Flemming's work was with Penicillium notatum, which I've heard was originally found on an orange or a cantaloupe, but I think these stories are more fancy than truth. I would guess the truth to be that a small mould culture infected a Staphylocuccus culture Flemming was working with, and produced "... a halo of inhibited growth", which Flemming then cultured and found to be Penicillium notatum.
The problem so often encountered throughout history is that the species of Penicillium was not always either recorded or correctly identified, nor is this an easy thing to do with bread moulds, when using primitive equipment. Some species of Penicillium have an antibiotic effect, some are harmless and some are toxic to humans.
Penicillium is made of various types of fungi. With Alexander Flemming, he left a pile of food for weeks and discovered that it was good types of mould.