Why ask? If you dont want to put other cachers in danger, pull it! I would hate to be the owner of a cache that someone warned me about a angry crazy lady nearby and I didnt pull it and someone got shot later.Think of your fellow cachers.
Of course, It wouldnt be a bad idea to make sure neighbors know about near caches and that there OK with them.
Wait! Can you give me the GC code?