I know it's not justified but yes, I always feel guilty lurking around. It's to the point where I have personally banned all caches with a "muggle" icon. I feel less suspicious when I have the kids with me. For some reason, their presence makes it look less suspect.
My 2 police encounters, one involving the bomb squad, hasn't helped my apprehension.