As both a hunter and a rookie geocacher, I'm glad this question was raised. My thoughts are that Geocaching is a year round activity, while hunting has very specific and short seasons. We can geocache outside of hunting season so why risk an accident (I hate that word - there is no such thing as a hunting "accident"...) and regardless of safety and any personal views on the ethics of hunting, why ruin a hunter's day by traipsing through his favorite spot on opening day.
My advice would be to stick to caches that you know are in public places like parks, urban/suburban locations, or other areas where hunting is prohibited. Save the other stuff for after hunting season. Why risk it??
And if you do go into the woods during hunting season, for GOD'S sake please wear as much Blaze Orange as you can so that you can be seen from afar. Not just a hat, but a vest or jacket as well. As much as you can wear. It's not a matter of being mistaken for a deer... it's a matter of seeing you standing 200 yards on the other side of that deer before a hunter pulls the trigger. Hunters are trained to be sure of both their target AND their backstop before they fire. Make sure that you are highly visible so that you don't mistakenly become that backstop. It's easy to distinguish between man and deer. It's not so easy to spot a man 300 yards away in the woods wearing a brown jacket. The hunter may be upset that you were there, particularly if you've wandered onto private land. but at least you'll be alive to hear about it! Besides, many states require that you wear orange in the woods anyway during hunting season, whether you are hunting or not. Check your local/state laws.
One last thing... it's your job to be alert as well! You should be looking for orange the whole time you are in the woods, and if you spot a hunter, make sure he sees you and sees what direction you are travelling in, then quietly and quickly leave the area. Keep moving in the direction he last saw you travelling in and don't double back. That way he'll have an idea of where you are and when you will be safely out of range.
Hope this helps!