Yes this situation could have been uglier. I do carry a firearm, but nevertheless it's best to have good verbal skills in these situations than a quick draw. To someone who is that defensive about their territory and doesn't know anything about geocaching, the explanation is going to sound doubly like bull. However, when confronted with threatening speech or body posture, responding with what could be perceived as a return threat or a spin will only escalate the situation. Keep a non-threatening posture, and speak calmly and firmly. Take all insults and don't respond to accusations defensively. Rather, continue explaining your business. Some things inevitably become physical, but by and large if the person encountered isn't planning to use force in the first place it won't get to that point unless provoked to it. Try to figure out what is causing the other person to be aggressive or defensive and work the conversation about alleviating that.
After 10 years of working in the prison system, I've learned the best protection you can have is interpersonal skills.