I too have been stared at with suspicion while I "loitered" around. Here's what I have discovered; nobody notices a person with a tape measure and a pad of paper. They assume you are a contractor and in between "measurements" when I am poking around and looking for the cache, they all assume I'm just doing some mental engineering.
Works like a charm.