True about the bugs being brought inside, if you have other plants this could be a problem. The only real negative I see is the lack of UV rays on your plant which will help increase resin production. I grow in the mountains and move plants in and outside all the time to get the benefits of outside and keep them inside when the weather is bad. Really it all comes down to what you want to do and feel most comfortable with, how much time you devote moving them but just a simple fulltime move from outside to inside except for bugs no worries.
Happy Growing