You nailed it dead on for me, too. I enjoy the fact that caching isn't just looking for junk in the woods. I'm still puzzled how I can spend hours using software to try to find the best possible routes to the most possible caches in a single day. Not just to increase my number of finds, but simply for the satisfaction of doing something I enjoy.
I'm a geek who hadn't spent much time outdoors in quite a few years. It wasn't for the lack of interest, but the motivation wasn't there. The very welcome side-effect of Geocaching on me is I haven't spent a single weekend laying on the couch or watching TV in bed since I found the sport/hobby. Sure, I've spent several hundred dollars more than any sane person should on a GPSr, hiking boots, backpack, first aid kit, etc., but I see it as an investment in something I plan to do for a long time to come.