Exhaust ducting back-flow from outside air/weather while fans off?

Totally new to this, got a 6" duct with duct-fan booster (local grow shop!) 160cfm kicking hot air away. May be overkill, but the box sits at 86-90F all day, and hotter if the box is closed. Cant even close the box.... :< SO. Now there is this great setup to get rid of allll that hot air. While exhaust system is on, hopefully it will pull the temp down by 10-15F, as air will be going out the window. The plan is to have the duct booster off at night, turning off at the same time the lights turn off for their 12/12 cycle (maybe save extra money turning exhaust off @ night). On the inside of the box, there is an extra desk fan circulating air for her to breath during the night.

So the question. What about the hot back-flow air and weather from outside? Will it come back in the box while she sleeps? The fan is set to "off" during the night. That would allow air/weather to come BACK in from outside, no? Will this be a problem? How would you the outlet to stop outdoor weather back-flow issues (its hot outside...)? Should the exhaust fan be left on 24/7? Don't really even know if outdoor air/weather back-flow is going to be an issue, as the ducting system hasn't had the chance to be installed just yet. Thanks for the input everyone!
 
That looks like it could help a bit, anything else from the other pros? My main concern is water from heavy rain, and warm air through cracks back throughout the ducting.
 
Trying to see any other ideas for my issue? Could use the backdraft damper Monkey referred me to, but I don't think the fan provides enough push to open the damper and still allow enough airflow to get hot air out of the grow box.
 
Top