I'm always finding spiders on my outdoor plant. The yellow spider you are describing sounds like a goldenrod spider....I'll see if I can find a picture. I've found them small like you described and about as big as a quarter.
Personally, I leave them....spiders eat other bugs, not plants.
This is a goldenrod spider...
This guy has yet to be identified...