Actually, it seems to happen the other way around for me. I spend a lot of time finding good places to hide letterboxes, then someone piggy-backs a geocache on it and my letterbox ends up with no stamp because someone took it and left a plastic car.
This despite the bold lettering on the lid that says "This is not a geocache."
Of course, your mileage may vary.