My friends, co-workers, family, etc., etc., etc. have been calling me a nerd (among other things) for years - geocaching is just something new for them to target this 'accolade' on...
I think it's cool - those who think it's nerd-y definately don't know what they are missing!