As a geocacher a hunter and a gun toter on a daily basis ultimately in my opinion there are several problems
One whether you had the right to be there or not it was hunting season and some respect should be given to that.
Two as a geocacher you threatened the whole community by going back in while the hunter was still there and attempting to show him where the cache is
Three you possibly turned off a new geocacher to the sport
Finally to your question of whether the hunter had any extra rights. Well I say yes and no to that. While the hunter acted inappropriately and he did not have a "gun" he did have a weapon. I would assume based on your post you did not. So in my opinion he had no additional right to be there, but unfortunately for you he did have additional means to enforce his rights. Like it or not you should have recognized that, left the area and stayed out of the area for your own safety that to me is common sense.