I guess I'm a bit suprised that it would be at all problematic to explain to anyone. Everyone that I've talked to about it either knows someone that knows someone that is into it or seems to think it sounds like a neat/fun reason to get out and be active.
Then again I live in Arkansas, the Natural State, and most people here seem to understand that any reason whatsoever is more than enough to go out into nature. Actually, I would think that sort of understanding is pretty universal among we primates. Are there actually people that can't understand the idea of walking about among the trees or other unknown just for its own sake? Geocaching, as a way of encouraging that sort of natural adventurism, doesn't seem that hard to convey.