Hey there. I don't think anyone will be able to give you an simple answer to this as there's not really enough information here. I can tell you that a 6" fan to vent 400W would do fine in my climate, as the ambient air temperature is quite low for most of the year. The warmer the air coming into and surrounding the box is, the higher the temperature inside the box will get after it's been warmed by the light. When you say your first box got up to the 110s after an hour, what temperature was the air outside the box at the time?
You mention that your new box is bigger than the first. This is good as there is a greater surface area through which heat can pass via conduction from the inside to the outside of the box through the walls, independent of ventilation. This can make a big difference if the walls of the box are thin, if the box is located in a cool place, and if air can flow freely over the outside surfaces allowing the walls of the box to cool themselves. If the opposite is true, if the box has thick walls and is placed in a warm confined space with no airflow over the outside surfaces, then the new box will reach almost exactly the same temperature inside as the smaller box did, though it will take a bit longer to heat up. Though it seems counter-intuitive, a small well-insulated box behaves very much like a big well-insulated box given identical ventilation, it just takes longer to reach it's maximum temperature as there's more box to heat up. The amount of heat put out by the light and the amount of air drawn out by the 6" fan will stay the same. If you're making the box from thick pieces of wood and the inside surfaces are painted white or coated in mylar, then the few extra cubic feet of space inside won't make a huge difference, it will just take longer to heat up. If there's good scope for airflow on the outside surfaces of the box then making a wall or two out of something like sheet metal will make a bigger difference than increasing the box size, as those walls wouldn't warm up much and so wouldn't heat the air inside the box.
You mention that cool tubing is out of the question, but there may be ways to direct the heat coming off the surface of the light directly into the exhaust fan. The temperature at the top of the box doesn't really matter, just the space in which the plants are growing. If you can make some sort of a hood for the light, so that the heat rising off the surface of the light/ballast gets sucked straight out the exaust without circulating around inside the rest of the box, then it will make a big difference to the temperature lower down in the box. A sheet of thin metal folded into a wide cone shape, attached to the exhaust fan and positioned directly above the light might suck up most of the warm air coming off the surface of the light/ballast before it can warm the rest of the box. Is something like this feasible or is the light too big?
Similarly it will help if surfaces lower down in the box have some air blowing over them from a secondary fan, as this will help transfer heat from surfaces warmed by the light into the air and out the exhaust. In a closed box with just one fan (exhaust) the cool air coming in the passive vents will flow in straight lines to the exhaust. Hotspots will develop in the box where there is no airflow. Eventually the hot spots heat up enough that you get convection currents rising up from them and heat starts to flow out the exhaust at the same rate that the light generates it, but by that stage the whole box will be hot. You'll want every surface low down in the box (floor, walls, pot and the plants themselves) to have some air flowing over them so that heat extraction works well from the moment the light is turned on. You want the cool air coming in to swirl around a bit, picking up heat, before it's sucked out. That said, you'll want to avoid swirling air at the very top of the box as much as you can, otherwise the heat rising off the surface of the light and ballast will warm the whole box, instead of rising up and straight out the exhaust like you want. A small fan or two blowing horizontally at about 18-24 inches above the floor should keep the plants nice and cool without messing with airflow above the light too much.
I hope this helps some, and good luck with the build.
