Do your own damn yard work!!! That sucks to hear man. I was working in my vegetable garden outside and accidently left my garage door open. Neighbor came over to converse, and I kept getting whifs of Diesel and Kush. Then I realized my garage was open and was like, damn I hope this dude is cool. I think as long as you are on good terms with the person, they have little reason to go out of their way to see you jailed.
With that being said, better safe than sorry for sure! I know my neighbors fairly well, and they know I'm a good citizen, in college, full-time job etc...I think a pissed off yard man would be a bit of a different situation.
To put your mind at ease, the police need a lot more than a yard man's word to obtain a search warrant. They would need to obviously smell it from the street, well off your property. Or at least lie about it.