+Pprime (P`) Posted December 7, 2013 Share Posted December 7, 2013 I have suddenly noticed many (not all) images cannot be found on their cache pages. This is rather important to me since many caches have important information on them in the form of a picture which is meant to be stored on the cache page. Seems it's not just me too. Can it be investigated please? eg: http://coord.info/GC3BTQD Quote Link to comment
+Walts Hunting Posted December 7, 2013 Share Posted December 7, 2013 It appears that only the important one can't be seen. When I try I get a 404 error which indicates the picture is not at the location specified. Have you tried contacting the CO, advising him of the problem and maybe he will send it to you. This doesn't look like it needs an investigation since it is a singular occurrence. Quote Link to comment
Pup Patrol Posted December 7, 2013 Share Posted December 7, 2013 (edited) Have you tried contacting the CO, advising him of the problem and maybe he will send it to you. The OP is the CO of this cache. Yes, the "note from Captain Hook" image is missing. Have you changed the location of that image? The other images are still there, so it seems odd that only the one is a problem. I didn't check, but I'm wondering if there is a size limitation now, that maybe the missing images are too large? B. Edited December 7, 2013 by Pup Patrol Quote Link to comment
Moun10Bike Posted December 7, 2013 Share Posted December 7, 2013 I have confirmed that there is an issue with a subset of images. You can see occasional broken images in the gallery of most any profile you examine. I've notified our engineering team of this issue. It is likely related to the planned downtime that we had on Thursday afternoon. Quote Link to comment
+Walts Hunting Posted December 7, 2013 Share Posted December 7, 2013 Have you tried contacting the CO, advising him of the problem and maybe he will send it to you. The OP is the CO of this cache. Yes, the "note from Captain Hook" image is missing. Have you changed the location of that image? The other images are still there, so it seems odd that only the one is a problem. I didn't check, but I'm wondering if there is a size limitation now, that maybe the missing images are too large? B. Sorry, I did miss it was his cache. Kinda thought he might have mentioned that in the post. Quote Link to comment
+Pprime (P`) Posted December 7, 2013 Author Share Posted December 7, 2013 Thankyou Moun10Bike And sorry Walt, yes I am for that one, but as noted it is affecting a lot of caches not just mine. I spent an evening sending images of my puzzles to several people who had the time to work on them this weekend ( or had hoped I'd just give them answers I suppose ), but I chose a simpler one for an example. I look forward to the answer in due course. Quote Link to comment
+Walts Hunting Posted December 7, 2013 Share Posted December 7, 2013 That cache sounds like a gem that I would go for if I was around there. Great story on the page. Quote Link to comment
+funkymunkyzone Posted December 8, 2013 Share Posted December 8, 2013 I have suddenly noticed many (not all) images cannot be found on their cache pages. This is rather important to me since many caches have important information on them in the form of a picture which is meant to be stored on the cache page. Seems it's not just me too. Can it be investigated please? eg: http://coord.info/GC3BTQD Some of my cache pages have been affected also - annoyingly, puzzles where the images are important, so I've had to re-upload the images and edit the descriptions. Although, in one case it was the trigger to get me to finally reformat the page to look nice ever since the GS cache page redesign a while back forced descriptions to limit to 670px wide... Anyway, I hope this is sorted soon, so I don't have to go through all our cache pages! Quote Link to comment
Moun10Bike Posted December 9, 2013 Share Posted December 9, 2013 The engineering team identified the issue over the weekend and is working on restoring the missing images. We hope to have everything corrected by the end of today. Quote Link to comment
+-Mark- Posted December 10, 2013 Share Posted December 10, 2013 We hope to have everything corrected by the end of today. Seems like "today" wasn't monday There are still images missing (404). Quote Link to comment
+Der Wald-Pirat Posted December 11, 2013 Share Posted December 11, 2013 My avatar is missing, too! Quote Link to comment
cezanne Posted December 11, 2013 Share Posted December 11, 2013 (edited) We hope to have everything corrected by the end of today. Seems like "today" wasn't monday There are still images missing (404). I just have been notified that a picture of one my caches is missing. I made a hot fix on my own by uploading it again and linking to the new picture. However also other of my caches are affected and I'm not in the mood to check all pictures and search for the missing ones and upload them again etc. That's a whole lot of work just for nothing ..... I've also encountered myself several other effected caches. I'd wish that the issue is fixed really soon. Cezanne Edited December 12, 2013 by cezanne Quote Link to comment
Moun10Bike Posted December 11, 2013 Share Posted December 11, 2013 The initial attempt to restore missing images failed and the IT team is still working on it. Even once the restore is complete, there will be some data loss. This will consist of the most recent images that were lost after the last backup completed. I would recommend re-uploading any images you see missing. Quote Link to comment
+LinkaLovers Posted December 11, 2013 Share Posted December 11, 2013 (edited) ....after the last backup completed.... To make it easier: When was the last good backup made? Edited December 11, 2013 by LinkaLovers Quote Link to comment
Moun10Bike Posted December 11, 2013 Share Posted December 11, 2013 ....after the last backup completed.... To make it easier: When was the last good backup made? End of August. Quote Link to comment
cezanne Posted December 11, 2013 Share Posted December 11, 2013 ....after the last backup completed.... To make it easier: When was the last good backup made? End of August. Are you serious? I can hardly believe that. All the images that are missing for my caches are older and moreover I have backups somewhere and have already restored the most important pictures, but that's still a considerable effort even though I do not own that many caches. I can understand that the gallery pictures are not saved more often, but the cache descriptions and the images that are part of the cache descriptions should be saved much more often and I also do not understand how so many pictures can just disappear. Cezanne Quote Link to comment
+irisisleuk Posted December 11, 2013 Share Posted December 11, 2013 A lot of our photos are missing as well, on our cache pages (sometimes needed for a puzzle) and also with logs for caches and trackables. We have made thousands of photos (each year), so restoring them means going through large databases of photos, so I rather not even think about having to do this. Can I at least be sure that photos that we've placed before August 2013 will eventually be restored so I won't have to look into those? Or is this not sure yet either? Quote Link to comment
Moun10Bike Posted December 11, 2013 Share Posted December 11, 2013 We believe that we will be able to restore all of the missing images from before August. It is the images that were uploaded after then that will be permanently lost and that will need to be re-uploaded. Quote Link to comment
+ecanderson Posted December 12, 2013 Share Posted December 12, 2013 ....after the last backup completed.... To make it easier: When was the last good backup made? End of August. To reiterate a prior post -- SERIOUSLY?Who on earth is your CTO? Kathleen Sebelius? Quote Link to comment
cezanne Posted December 12, 2013 Share Posted December 12, 2013 (edited) We believe that we will be able to restore all of the missing images from before August. It is the images that were uploaded after then that will be permanently lost and that will need to be re-uploaded. In fact, as the process of restoration takes so long cache owners will have to re-upload the important photos that are older as well. I wonder whether Groundspeak has ever thought of dealing with the photos that are part of cache descriptions in a different way and create backups of them much more often which should be feasible. It is of course inefficient to create backups of all photos including the galleries very often. May I ask something related: How often are backups done for the cache descriptions (the text)? This might be of interest to the cache owners as some might decide that they need to improve their own backup policy. I often realize that many cache owners (not all of them) have copies of the first version of their cache description, but perform changes only online and so do not have backups available (which is bad in any case). Maybe Groundspeak could add a few sentences on making backups (on the cachers side) to the help center or another location on the website. Up to now cachers might have assumed that Groundspeak is creating regular backups anyway. For example, at my work we create dayly backups (I did not expect GS to do that for gallery images as this is infeasible anyway). Cezanne Edited December 12, 2013 by cezanne Quote Link to comment
Moun10Bike Posted December 12, 2013 Share Posted December 12, 2013 The image server is backed up ~daily, and the database server (containing cache descriptions and logs) ~hourly. The issue here is that we have experienced a major failure of the image backup server and we are having issues getting the vendor, who have identified the problem, to respond in a timely fashion. Quote Link to comment
+ecanderson Posted December 12, 2013 Share Posted December 12, 2013 Hmmm... I may have been hasty - well, a little bit. When you answered "When was the last good backup made?"with "End of August.", and now note that "The image server is backed up ~daily," my new take-away from that is that your last AVAILABLE good backup dates back to August, and that the server that has held the "~daily" backups since then has failed? Or is it that the backups have been corrupted since August and nobody noticed? The "~daily" makes me feel better, but aren't you guys arranging for off-site storage of those backups on a regular basis? There should still be other copies somewhere if this was done using best practices. Quote Link to comment
+KegNot Posted December 12, 2013 Share Posted December 12, 2013 I'm still missing some pictures in cache descriptions. These descriptions and their pictures haven't been changed for some time now, so they should be in the August backup. When do you expect the restore to be complete? Quote Link to comment
cezanne Posted December 12, 2013 Share Posted December 12, 2013 I'm still missing some pictures in cache descriptions. These descriptions and their pictures haven't been changed for some time now, so they should be in the August backup. When do you expect the restore to be complete? I guess that they have not even started the restore process. Quote Link to comment
Moun10Bike Posted December 12, 2013 Share Posted December 12, 2013 I'm still missing some pictures in cache descriptions. These descriptions and their pictures haven't been changed for some time now, so they should be in the August backup. When do you expect the restore to be complete? I guess that they have not even started the restore process. We have been consistently trying to restore since the weekend. We have run into issues that the server vendor has recognized as their problem, but they have not responded with a solution for us. Until then, we cannot estimate when the issue will be resolved. Quote Link to comment
cezanne Posted December 12, 2013 Share Posted December 12, 2013 We have been consistently trying to restore since the weekend. We have run into issues that the server vendor has recognized as their problem, but they have not responded with a solution for us. Until then, we cannot estimate when the issue will be resolved. Thanks a lot for your regular updates. I did not mean to say that GS has not started working on the issue, but only meant that it appeared to me that no restoration of pictures has happened so far. Quote Link to comment
+KegNot Posted December 13, 2013 Share Posted December 13, 2013 <cut> We have been consistently trying to restore since the weekend. We have run into issues that the server vendor has recognized as their problem, but they have not responded with a solution for us. Until then, we cannot estimate when the issue will be resolved. Tricky. I would not be very happy with such a supplier... I think I'm not going to wait and start replacing the missing images. I want Geocachers to be able to "do" my puzzles and caches... Thanks for the explanation. Quote Link to comment
+KegNot Posted December 13, 2013 Share Posted December 13, 2013 Mmm. I now see the links to the images have changed. Previously: http://img.geocaching.com/cache/25066027-febb-417d-873b-decc4c19d115.jpg Now: http://imgcdn.geocaching.com/cache/large/25066027-febb-417d-873b-decc4c19d115.jpg Both seem to work though. The example in the documentation suggests it should be the img.geocaching.com one. Anyone any comments on this? Quote Link to comment
+KegNot Posted December 13, 2013 Share Posted December 13, 2013 In the end I had to fix a total of two images, so that was not a big chore. I'm done for now. Keep the server healthy Groundspeak! Quote Link to comment
+Viajero Perdido Posted December 14, 2013 Share Posted December 14, 2013 Previously:http://img.geocachin...ecc4c19d115.jpg Now: http://imgcdn.geocac...ecc4c19d115.jpg Both seem to work though. <aside> Both should continue to work, and I wouldn't worry about the difference. Quite likely, one form of the name simply maps to the other via a simple server rule, or will once they finish moving some stuff around. If that's the case, they can easily let it continue to work that way (both forms valid) forever. Just an educated guess. </aside> Quote Link to comment
+small oaks Posted December 16, 2013 Share Posted December 16, 2013 (edited) http://imgcdn.geocaching.com/cache/large/755d614e-5577-4164-8799-57e008f1ad50.jpg I hope this is fixed soon. Edited December 16, 2013 by small oaks Quote Link to comment
+DanPan Posted December 16, 2013 Share Posted December 16, 2013 What is the status? Is the restore process still ongoing? What are the prospects? Quote Link to comment
+Nighthawk700 Posted December 17, 2013 Share Posted December 17, 2013 I lost my avatar today. I uploaded a re-make of the avatar, but it isn't as crisp as the original. Hopefully you can restore it, or I can figure out which program I used that got the best crisp size reduction. Quote Link to comment
+Rikitan Posted December 17, 2013 Share Posted December 17, 2013 I'm still missing some pictures in cache descriptions. These descriptions and their pictures haven't been changed for some time now, so they should be in the August backup. When do you expect the restore to be complete? I guess that they have not even started the restore process. We have been consistently trying to restore since the weekend. We have run into issues that the server vendor has recognized as their problem, but they have not responded with a solution for us. Until then, we cannot estimate when the issue will be resolved. This problem effects 9 of my 50+ listings currently. Please, let us know when you'll complete the restore process. So I'll know how many listings should I try to fix. Quote Link to comment
Moun10Bike Posted December 18, 2013 Share Posted December 18, 2013 We were finally able to resolve issues with our server backup vendor last night. Restore is now underway, but could take the rest of the month. We don't yet know the total number of files that were permanently lost, but we know there are some. Quote Link to comment
+aka Momster Posted December 18, 2013 Share Posted December 18, 2013 Thank you for keeping us updated, Moun10Bike. I have corrected some of my broken images already, but with more than two dozen left to fix, I think I'll wait and see if some get restored. Quote Link to comment
Moun10Bike Posted December 20, 2013 Share Posted December 20, 2013 I have just heard from our IT team that all possible recovery processes for the image issue that started two weeks ago have now been completed. Through a very long restore process, we've managed to recover approximately 95% of the compromised images, which represented 2.52% of the full image store. Unfortunately, 5% of the images were unable to be recovered, which represents 0.126% of our total image store, or about 37,000 images (not including thumbnails, scaled versions, etc that are regenerated on the fly). Any users who at this point are still missing images will be required to upload them again to have them visible on the site. Quote Link to comment
+KegNot Posted December 20, 2013 Share Posted December 20, 2013 Great news! Thanks for the update! Quote Link to comment
+ecanderson Posted December 22, 2013 Share Posted December 22, 2013 Here's hoping that your off-site backup service gave you more than just an apology for putting you through this exercise Quote Link to comment
+Lil Devil Posted December 23, 2013 Share Posted December 23, 2013 We were finally able to resolve issues with our server backup vendor last night. Restore is now underway, but could take the rest of the month. I have just heard from our IT team that all possible recovery processes for the image issue that started two weeks ago have now been completed. Through a very long restore process, we've managed to recover approximately 95% of the compromised images, which represented 2.52% of the full image store. Wow. How did an estimated 2 week restore get completed in 3 days? Either those guys are really really good at making restores go faster, or really really bad at estimating completion time Quote Link to comment
+wmpastor Posted January 23, 2014 Share Posted January 23, 2014 (edited) Hmmm... I may have been hasty - well, a little bit. When you answered "When was the last good backup made?"with "End of August.", and now note that "The image server is backed up ~daily," my new take-away from that is that your last AVAILABLE good backup dates back to August, and that the server that has held the "~daily" backups since then has failed? Or is it that the backups have been corrupted since August and nobody noticed? The "~daily" makes me feel better, but aren't you guys arranging for off-site storage of those backups on a regular basis? There should still be other copies somewhere if this was done using best practices. Right. This business does not exist without the website and its data. Anything less than "best practices" is foolish. Edited January 23, 2014 by wmpastor Quote Link to comment
+wmpastor Posted January 23, 2014 Share Posted January 23, 2014 We were finally able to resolve issues with our server backup vendor last night. Restore is now underway, but could take the rest of the month. I have just heard from our IT team that all possible recovery processes for the image issue that started two weeks ago have now been completed. Through a very long restore process, we've managed to recover approximately 95% of the compromised images, which represented 2.52% of the full image store. Wow. How did an estimated 2 week restore get completed in 3 days? Either those guys are really really good at making restores go faster, or really really bad at estimating completion time Or they follow the sound business practice of estimating conservatively & delivering faster/better/more. Quote Link to comment
+2Abendsegler Posted January 25, 2014 Share Posted January 25, 2014 I have just heard from our IT team that all possible recovery processes for the image issue that started two weeks ago have now been completed. Through a very long restore process, we've managed to recover approximately 95% of the compromised images, which represented 2.52% of the full image store. Unfortunately, 5% of the images were unable to be recovered, which represents 0.126% of our total image store, or about 37,000 images (not including thumbnails, scaled versions, etc that are regenerated on the fly). Any users who at this point are still missing images will be required to upload them again to have them visible on the site. Great job, many thanks! All of my missing images are back. Quote Link to comment
+Astradyne Posted February 10, 2014 Share Posted February 10, 2014 I read a lot about Image data bases and stuff. Funny thing is, that the Images missing from a new listing I'm currently working on aren't uploaded on Geocaching.com but linked via HTML-code <img src>. even stranger - one of them used to be shown and the disappeared as I renewed the page. Why aren't These Images shown while others from the same source URL can be seen? It This surely cannot be the result of a problem with the GC data bases. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.