My thoughts, and they're just thoughts, as to why it is a non-issue is the timing. The plants growth and flowering stages are largely finished at this point. The buds left on the plant are ripening but not gaining too much size and mass. There isn't much to stunt.
You're also taking the flowers...