When I spray, it's slightly "milky" white, if you mean this with "white buildup"? I NEVER wash it off - this is the point, it needs to stay on the plants for max. effect.(I never had ANY negative effect on my plants from spraying with the soap and Neem, ever)
In regards to spraying in flower, it is never "recommended" to spray anything (on the buds) in flower, so if you can only spray the leaves but avoid the buds.
BUT...you have always the option to "wash your buds" in case you had to spray in flower this should get rid of anything you might have sprayed.
(Search "bud washing" in google since I am not sure whether it's allowed to link to another forum here. In a nutshell, you harvest and then dunk and swirl your buds in water with baking soda/lemon juice and then two times in clear water to really "wash" them) . It's actually effing easy.