I had a similiar encounter while hunting for a benchmark around a CSX rail line. Looked like the BM might be around the rail line (My first BM), so I went down the tracks a bit before doubling back and finding the BM closer to the road. About the time I found it, I was stopped by a CSX employee who asked, "Can I ask what it is you're doing, what with terrorism being what it is today". I explained the Geocaching game to him, which must have satisfied him as he didn't call for backup.
Not a terrorist, but perhaps just a middle aged geek with a new toy!