Some ppl I know actually pluck out the leaves during flowering, they say its to expose the buds to more light thus making them fatter. I dont know if thats true but if the plant can take such treatment and still thrive I really dont see any reason for you to be worried over a few yellow leaves.