There's actually a blind spot in perception that allows this to work--if you look out of place, but act as if you are really supposed to be there, no one will notice.
When I grab a cache, I don't use stealth at all, and I hunt in Boston. No one takes notice. The less suspicious you look, the less people will care.