I am constantly seeing damage by geocachers. Perhaps the underbrush in the area I live (western WA) is more sensitive then other places, but there is a trail and trampled area around almost any cache placed in the woods. I found one cache in some local woods by following a trail through trampled and broken salal to the base of a tree where the moss and underbrush was completely flattened and torn up. I never had to even glance at my GPSr. It looked like a herd of elephants had been through. And that cache had only been up for 12 days. A couple days ago I went for a cache in deep woods that, as it turned out, had coords that were fairly far off. The entire area around the original coordinates was trampled. In the description they put a set of updated coords. The area around those coords was basically flattened. The area around the cache itself turned out to be in good shape. But where people had been searching extensively looked terrible.
This is a touchy subject with me because I've really been noticing a lot of damage recently. A local park has a cache hanging in a bush. The bush has leaves that are turning brown because the branches are broken and freshly broken off branches were obvious.
When you are walking around in the woods it is impossible not to leave some sign of your passage. But if you show a little bit of care it is easy to minimize your damage. For instance, if I'm in the woods I try to return via a different route so I don't pound in a geotrail. If I find smashed down brush I like to fluff it back up so it isn't so obvious to the next person.