That sounds really weird. I would expect to water to slightly raise the ph, not drop it below 5. What are you measuring it with? Your ph meter may need calibration. What does it say the ph of the water is?
As for quick fixes, walmart should have some aquarium ph adjusters, but i personally would never trust them on my babies. I only use ph up/down made specifically for hydroponic applications. You can find some on ebay pretty cheap.