I'm relatively new to this GC game - yes, it's only a game remember (the kids like "Treasure Hunting") but here's my contribution to this long running thread.
I have a Garmin Etrex Yellow. I also have a PC and laptop on which I run Memory Map with full GB coverage at 1:50 000 and some at 1:25 000 and use EasyGPS and GSAK for admin of routes/waypoints/caches etc. with a serial connection to the GPS. I also have a bluetooth PDA with USB connection, onto which I can load as many maps/portions of maps as my 1GB card allows and, better still, a bluetooth box that connects to the Etrex which can then sit in the top pocket of a rucksack while the PDA is in a jacket/trouser pocket ready to view. The GPS and bluetooth box get left on while the PDA is switched on when needed and takes less than 10 secs to acquire the GPS data and give a fix. A fairly comprehensive setup!
But, on the other hand......
I bagged 5 caches yesterday, 3 of them in quite remote and hilly country and all was done with just an OS map and compass and the Yellow Etrex, with the GPS only coming out of the bumbag when getting close to the caches. The cheap, cheerful and utterly reliable and accurate way of finding them - unless you are using old batteries!
So, my message is that yes, you can obviously spend a fortune getting gadgets that do everything but carry you to the cache and sign the log, but the reality is that a very basic and cheap setup will do the job perfectly and you will probably learn more about navigating in the process and you will have a nice collection of maps to fill shelves and pore over and make plans....
If using the Etrex Yellow, I would recommend something like EasyGPS (free) for admin and transfer of waypoints unless you can download direct to the GPS from Geocaching.com, which I don't think is possible (yet.....?)
Whatever you do, just get out there and happy caching, everyone!