Depending on how much you take off and how often you do it, you could be fine defoliating those lower side branches.
Don't take too many off in one shot so you don't stress the plant.
On my first main line grow I was really strict with a couple plants, let one go kinda wild, but on one...