I think this is the key. We have to be above it all. We need to be watching over our communities. We have to take the time to educate with brochures and presentations in our community... So that we always look like the good guys, because we are.
Perhaps as part of this geocaching community we need to stop acting stealthly and start wearing a "uniform"; the orange safety vest and yellow trash bag in hand when caching in urban locations. That way we are announcing to the world, we are geocaching, we are picking up trash, we are on neighborhood watch and reporting suspicious activity. Hey we are the good guys.
Personally I would hate to see geocaching limited to remote wooded locations, because I am not much of a hiker anymore. So suburban and city park caching is what I'm into.