I say trim more. It could just be me but I think the more leaves you remove the smoother your smoke ends up. We always make bho out of the sugar leaves so it isn't like you lose anything. But if you like it that way it isn't going to make you any higher if you clean it up more. Congrats on a successful harvest.
Personal preference, there's certainly no reason to except for appearance. I'd leave them as-is myself... there are some quality trichs on the sugar leaves that will get you baked just the same.