The only time I have actually encountered someone who looked suspicious, the person happened to be another cacher. It was lucky that we ran into him, becasue it was our first time searching for a micro, so he showed us what it looked like, and where it was. Then, since it was a multi cache, we continued to find the second half of the cache together (along with my dad and my friend who are my caching partners).
I've gotten some strage looks when it was only me and a friend (without my dad) but no one has actually come up to me. If someone did ask me, I would just tell them that i was Geocaching and then walk away.