Oh, I have no doubt there's aphids on there. But the damage sounds pretty severe for aphids. If you had a massive aphid outbreak you would see tons of aphids clusters on the bottoms of the leaves, sticky shiny residue all over (not THC trichomes), and you'd probably see clusters of the winged ones all over the stems.
When you say the buds got loose and dry, I'm thinking it's bud rot caused by worms. "bud worms"
Worms are these tiny little caterpillar fuckers. I most often find them in the early mornings on the actual bud. They tend to hide out in the daylight. They eat their way to the center of the bud and shit and eat in there. You won't really know it's in there until it's too late and you see a rot spot. You can usually see the dead sugar leaves, or dead/rotten buds, break it off and find worm and/or worm shit.
Monteray BT or Spinosad is some of the best stuff for worm battles. Pick off moth eggs too.