MTH - you are not the only cacher who has concerns here. We also have a bird-box cache and got a couple of comments similar to yours back in September, so you are not alone. I will say that we had no critical comments for the first two years and 100-odd cachers who found it. Then a reasonably experienced cacher made a comment like MTH's and then the cachers immediately afterwards agreed. However, since then, no more concerned comments. Maybe that is just evidence that most cachers do not wish to be seen to criticise.
I must admit to having a long think when we saw the comments. I can understand the concerns and if you have to get close to a bird-box to work out whether it is real or fake, then you have already disturbed it if it is real.
On balance, I felt that our bird-box was sufficiently obviously fake (too low, not fixed to the tree, has our caching name on the outside, hole obviously blocked) that I managed to convince myself that it was OK. I do think that any real bird-box will be mounted where it is very difficult for humans to access. If we were looking for a cache and saw a bird-box high up in a tree, we would leave it alone, even if we thought it might be fake (unless the cache page made it obvious that it was the cache).