The first couple of times I tried, I did not have much fun either. The fun part is finding one. Not finding it is... less fun. Especially if the location is an empty parking lot, like one of my first failure to find experiences. At least hiking in a park is a nice place to be while not finding something.
I had two big problems. First, I was using a GPS that is designed for driving directions. The coordinates it shows are not precise enough (DD° MM' SS" instead of DD° MM.MMM') so I could get close to the right spot, but not always find it. Also, it does not show you if the GPS signal is weak or strong.
Second, I did not know where to look. When I went to that empty parking lot, I was supposed to look inside a lamp post base. That is a common hiding spot, but I did not know that, so I looked all over the bushes and wasted a lot of time. Everybody else could find it easily, because they looked inside the lamp post.
If you are totally frustrated, find someone who knows how to find a geocache, and ask them to take you on a trip. There is probably a geocaching group in your area, and if you can find them, one of them will probably be able to help you.
It is so much more fun when you FIND the cache you are looking for.