I guess what you have to ask yourself is "Do I want to rub fungus into my skin?".
As a guy who's allergic to mold, I would say no...
Some people will come on here saying yes. Some will likely even recommend you "wash" your buds in hydrogen peroxide to "kill" the mold and make your buds "safe". In reality, the health problems associated with molds and mildews are largely caused by the toxins they produce. Killing the mold does not make these toxins magically disappear.