You may find this strange or even objectionable, but there is a national organization, formed because of similar problems. Those similar problems involve banning of their activities on a local basis, even passing laws against them, although the activities of the individuals are inherently legal, moral, and ethical and partaken of by hundreds of thousands of people world wide.
I'm referring to the American Association for Nude Recreation (http://www.aanr.com). Take a look at their web site and notice the national, regional, and local aspects of the organization. A collection of local groups and individuals, both commercial and co-op and just people and their families who come together in a single organization to promote their activities and permit them to practice their activities in freedom.
It seems to me that Geocaching could get some ideas from the AANR experience.
Crimestopper