Best case scenario, pulling the mushrooms helps but the spores are in the soil, they always keep coming if the conditions are damp.
You do water your plants, correct?
Worst case scenario, ditch the soil and go with sterilized soil (very expensive), or cook the dirt in your oven or nuke it, but be ready to put up with the stench, and you probably have enough problems with nosy neighbors as it is.
Theres always nematodes, but you may not want to introduce more bugs. LoL
Or keep vacuuming out the top inch of soil with a garage vac until they are gone and keep replacing that top one inch layer new soil.
I hope this helps you.
Peace