I think some common sense is needed. And in all honesty many reviewers and geocachers can be over the top. For example, if I have a mailbox and post at the end of my driveway, drill a hole in it and install a flap - it's not allowed as I'm defacing property. What if I build the mailbox from scratch with the intention of containing a geocache so it always had a hole and flap. Am I defacing property?
What about all the books destroyed for the creation of library caches? Obviously if I take a book from the library and start cutting it up I'm in the wrong. But if I own the book or ask a friend for a book it's all okay.
Just like if my friend owns a chain link fence and says I can modify it (drill a hole in it) I'm not defacing or destroying.
Having permission is the difference between vandalism (defacing) and having an outside wall painted by an artist (art).