Molds are everywhere; we're constantly inhaling mold spores. Different molds create a lot of different kinds of different compounds that people can be allergic to. Certain molds even make antibiotics themselves, and some people are even allergic to these compounds, such as penicillin. In fact, humans got the whole idea for antibiotics from soil microbes in the first place.
There are a lot of different kinds of organisms in compost though: bacteria, archaea, protozoans, nematodes, as well as fungi (mostly saprophytic, symbiotic fungi and not just molds). The specific kinds should be very common; many of the same species found in the dirt in your yard assuming it was healthy. Most compost and earthworm castings are also dominated by bacteria, although even earthworm castings will contain some fungi.
Interestingly enough some fungi are actually used as bio-control agents to protect against other, much much worse fungi such as pythium and fusarium and other pests. Certain strains of the very common Trichoderma sp. have been developed and marketed for this purpose. They work by producing their own own antimicrobials, stimulating host-plant resistance, and competing directly against other organisms.
But, there is actually a species of Trichoderma that causes toxic house mold. Honestly though, I would be more worried about the potential for anaerobic microbes in manures, or breathing in guano dust contaminated with histoplasmosis. Dust and any potential for related lung irritation would probably be the main concern when working with soil\potting mix also (which may or may not contain guano). Ideally you would be wearing at least a dust mask or filter mask\respirator, especially when working with guano.