I'm sure that geocaching brings a few people into the woods that otherwise would not be there. Personally I have been wandering around and exploring the woods regularly since I was a child. If the only reason you go in the woods is to hunt a geocache, I can understand why you might think that any human sign you find indicates other cachers but there are many other reasons for people to be in the woods. Bent grass and a few broken twigs are not damage to my way of thinking. I have only hunted about a dozen caches so far and with the exception of a couple of micro's, I could spot them from quite a distance because something just looked out of place. Mother nature will never neatly stack bark and limbs on top of her Tupperware, and camouflaged ammunition cans do not occur naturally near the root ball of fallen trees.
I will admit that I have seen a bit more trash on the trails since I started looking for caches so it could be true that the increased traffic contributes more to that problem, especially on the easier trails. Less so once you get onto the game trails where it takes a bit more effort to progress and not bleed all over the thorn bushes and such.
I do not need to be reminded that I have limited experience geocaching, these are my observations based on the experience I do have.