We must avoid extremes. The idea behind an indoor grow is to simulate ideal outdoor conditions 24/7. A light breeze is what we're trying to copy, not an out-and-out gust of wind. Your plants will be fine if you ease off the wind, but if you constantly batter them with a direct wind source, they'll suffer by drying out... not unlike a heat stress light burn.
The air exchange is important mostly because it keeps your grow enviornment constantly refreshed with new air so that the plants can come into as much contact as possible with the co2 that exists naturally in the air. If you're blowing fans directly onto your plants to cool your grow area, you would be much better off cooling your light bulb directly with an exhaust fan and ducting.