Would have been funny if during the 15 minutes you were 'hiding and waiting for him to leave' he had been out there searching for the cache with his trusty GPS and had to log a DNF because you took the cache off in your car. That would make a great story even better to see his DNF the same day.
If by ivy, you mean poison ivy, wouldn't it be best to post a DNF or at least a Note so that others can be warned and bring gloves or the owner can take care of the cache maybe by moving it a few feet from the poison ivy and updating the coords?