I understand your concern...however...herein exists a great opportunity -- why not educate the public about geocaching? One can cast the sport in a favorable light so as to increase the public's exposure as well as improve perception.
For example, NEXT on CNN will have a segment on the sport of geocaching this weekend. Using a different medium, I wrote an article that was recently published in "Adventure Sports Magazine" about geocaching. My purpose for writing the article included the reasons made above (i.e., increase exposure and improve perception). Did the article do any good? I haven't a clue yet...but I believe that it was the correct proactive step for me to take.
Lastly, as an Adventure Racer and active member of my county's Search and Rescue team, I *always* ensure that I have matches within my day & 24-hour pack. Yes, we can't control what people will do if they find matches in a cache -- I suppose that a real nut case could intentionally swing an ammo can and knock someone out. Unfortunately, even people that we rely upon within our society have violated our trust -- Volunteer Fireman Arrested for Arson, as just one of many examples.
--steve