I've just started caching in the D.C. area and I can understand what you mean. The first few caching trips I took where I had to go through a the back woods of a neighborhood or a restaurant felt kinda weird. I don't know if it's that a sorta 'social contract' that keeps 'normal' people from not going off the beaten path, but I definitely felt a sorta invisible resistance (let me know if I'm crazy or if anyone else felt this the first few times).
In fact the first time I dragged a few friends along they kept saying, 'In a neighborhood? we aren't going in a neighborhood right?' Anyway, don't know if that's just the Northern Virginia suburbs thing or if that's common throughout the states...
that's actually one reason i love this sport, it's really nice to explore the world in a way i would never have done in my normal daily life!