CAVU, my experience with FAA common sense using radar has been the same as yours. It is the property owners and other pilots who generally complain.

I was paying for gas in my yellow and white 172 and overheard the office person tell someone on the phone that it was their instructor and she was going to nail him. Just then another yellow and white 172 pulled up to the pump. I asked her what area the complaint was from and explained that I was at 200' on my pipeline right of way just south. She let me pay but otherwise indicated I should get lost.

Most property owners get used to low flying aircraft. Most FBOs let you know of any complaints without hassle. Most other pilots, except at your home airport, fuss when they see you. After safety, that is the best reason to stay low, stay quiet, and give way to all other aircraft.