I just went through this process and feel much the same way you do about picking.
I overanalyze everything and want to make sure I get the "right" one.
The real answer is there is no "right one". Through extensive reading and asking for a TON of opinions, I ended up narrowing it down to the Garmin 60CSX, and the Garmin Vista HCX.
There are a bunch of links comparing them, and equal numbers of people arguing that either one will do the job for Geocaching very well.
I ended up settling for the Vista HCX as it was cheaper, had almost identical features, screen was brighter and clearer to me, and was a little smaller. I got it from Amazon a few days ago for $219, and ordered a Sandisk 2GB microSD card to go with it for another $25. I plan to ask my wife to order me the City Nav Maps for my birthday for another $100.
The best thing I did was find a place that had them both, and I held them side by side to figure out which I liked better - button configurations, screens, etc. Good luck with your search.