I don't wear anything special for geocaching normally (unless dressing for the weather). I normally wear cargo pants and bright coloured t-shirts. If I need a jacket is usually dark red (spring jacket) or orange (winter jacket). I also wear a hat, but I wear it everywhere so its not hobby specific.
One thing I do carry just for caching is a bag I gave the original name of "my geocaching bag". Its a green canvas messenger style bag (it was labelled as a "jump bag" in the store - I think its a remake of a bag a paratrooper would jump with, but in essence its just a canvas messenger bag). In the bag I carry the following:
extra batteries for my GPS
USB cable for GPS
pens & mechanical pencils for signing logs
trail food (usually granola bars)
first aid kit
bug spray
sun block
gloves (for when I need to stick my hands in dirty places)
flashlight
GPS
compass
emergency whistle
caribiner
ziplock bags
stoop & scoop bags (optional if one caches without a canine companion
I often carry trade items in it as well, and when I am on multi-day trips, a netbook computer.
The bag & its various contents have served me well.