+TeamGeoduck Posted July 4, 2009 Share Posted July 4, 2009 Thanks for the up to date FB and twitter server restore updates. Now I cannot wait for GC to come back up. Link to comment
Keystone Posted July 4, 2009 Share Posted July 4, 2009 See the summary post from Elias in the Announcements section. Elias, thanks for returning early from your vacation to help restore services after the fire in the data center! Discussions of the site problem belong in the Geocaching.com Web Site forum, so I am moving this thread over to there. Link to comment
+Tequila Posted July 4, 2009 Share Posted July 4, 2009 Things could have been a lot worse. Glad to see no one was injured. Let's hope the servers come back up ok. Echo Keystone's comment: Appreciate Elias efforts. Link to comment
+Mopar Posted July 4, 2009 Share Posted July 4, 2009 Amazing job, all things considered. Thanks! Link to comment
+sbell111 Posted July 4, 2009 Share Posted July 4, 2009 I knew that their was badness when GC.com, WM.com, WiG.com, GS.com, and the forums were all down. I suspected that this was related, but I wasn't sure. Thanks for the hard work to all involved. Link to comment
Motorcycle_Mama Posted July 4, 2009 Share Posted July 4, 2009 Thanks for all the hard work. Glad no one got hurt. Link to comment
+Cryptid Posted July 4, 2009 Share Posted July 4, 2009 I just knew Obama had taken over GC too. Link to comment
+kbarhow Posted July 4, 2009 Share Posted July 4, 2009 See the summary post from Elias in the Announcements section. Elias, thanks for returning early from your vacation to help restore services after the fire in the data center! Discussions of the site problem belong in the Geocaching.com Web Site forum, so I am moving this thread over to there. Everyone can now come back in off the ledge. Just goes to show things seem to fail at the most inopportune time. Tnks for all the effort and everyone have a Happy Fourth! Link to comment
+The Blorenges Posted July 4, 2009 Share Posted July 4, 2009 Well done and big thanks for getting those hamsters up-and-running again... Much appreciated! MrsB Link to comment
+geocyclist94 Posted July 4, 2009 Share Posted July 4, 2009 Thank god for offline databases. With GSAK I was still able to go geocaching Link to comment
+edscott Posted July 4, 2009 Share Posted July 4, 2009 Thanks to the staff for the work to get the site up and running again. I promise to refrain from complaining about any site related issue for the next 30 days... Take an extra week off and head back into the woods. Link to comment
+yorelken Posted July 4, 2009 Share Posted July 4, 2009 Thanks, folks. Hope you get some holiday time. Happy Fourth, all you folks in the US. Link to comment
+Von-Horst Posted July 4, 2009 Share Posted July 4, 2009 Thanks for all your hard work. Sorry your vacation was ruined Elias Link to comment
+Buddaman Posted July 4, 2009 Share Posted July 4, 2009 Thanks for working all night to get GC up , Tell the head frog to put a extra fly in your bowl for the great work on your weekend off.. Link to comment
+Shilo Posted July 4, 2009 Share Posted July 4, 2009 I want to echo the Thank you and great job comments. Especially Elias flying back to Seattle from vacation. And to the rest of the Groundspeak crew who was at Fisher Plaza all night and probably still awake checking on things. Unfortunately, the timing was awful. No fault of Groundspeak. You guys and gals can't catch a break. I'm going to try and run a PQ and go geocaching with new data instead of 2 week old data today. To all the US citizens here, Happy Independence Day. To the rest of ya, have a great Saturday Link to comment
+Coyote's Girl Posted July 4, 2009 Share Posted July 4, 2009 Thank goodness everyone's OK. I knew something serious had happened. Great job on the fas respose. Link to comment
+csaws Posted July 4, 2009 Share Posted July 4, 2009 Oh thank you, I was really bummed the last 24 hours. I sat at the firehouse all day and try as I might could not get on to GC from anywhere, the firehouse, iphone web or the app. I had one near the firehouse saved and we got invited to be in the parade that this place was having... got permission from the owner (long story but basically there is a cemetery here that is being moved to widen a road and it sits on their property, owner said "yeah you can go look for it"... but I think it is gone) Link to comment
+Harry Dolphin Posted July 4, 2009 Share Posted July 4, 2009 Recovering from withdrawal! Thanks guys, you did a great job getting GC back up quickly. Thanks!! Link to comment
+TEAM LOCO Posted July 4, 2009 Share Posted July 4, 2009 Looks like there was an early 4TH of July up there. Glad to see things are back up. A BIG THANK YOU to all involved on getting things going again.... GS Link to comment
+TazD Posted July 4, 2009 Share Posted July 4, 2009 24 hours without geocaching.com and somehow we all survived. Thanks to all those at Groundspeak for all you did to get the site back up and running, especially to Elias for cutting your vacation short to get back to help! Very nicely done everyone. Link to comment
+sseegars Posted July 4, 2009 Share Posted July 4, 2009 If you Twitter, Jeremy had a running commentary during the whole outage. I thought that was very cool. Thanks for everyones hard work and sacrifice on this problem. Link to comment
+The Caching Coulters Posted July 4, 2009 Share Posted July 4, 2009 I tried to find a twitter feed but couldn't. What is it? Link to comment
GOF and Bacall Posted July 4, 2009 Share Posted July 4, 2009 This was linked to in our local forum. http://twitter.com/locuslingua Link to comment
+Don_J Posted July 4, 2009 Share Posted July 4, 2009 Things could have been a lot worse. Glad to see no one was injured. Let's hope the servers come back up ok. Echo Keystone's comment: Appreciate Elias efforts. I was afraid that I had go and find all those caches again. A human being shouldn't be allowed to have so much fun. Lucky for me, I had grabbed all my data for my "kick your butt" hike, hours earlier. I tried to post a forum reply at 11:04 on Thursday night, and it was obvious that something was seriously wrong. I quickly copied my text, and saved it in Word, before it disappeared, forever. Link to comment
+Don_J Posted July 4, 2009 Share Posted July 4, 2009 Looks like there was an early 4TH of July up there. Glad to see things are back up. A BIG THANK YOU to all involved on getting things going again.... GS Barely. A 12 mile hike though the mountains, helped. I want to echo the thanks... Link to comment
+searchjaunt Posted July 4, 2009 Share Posted July 4, 2009 Despite that everything is up again, I don't see - all things considered - any reason to be joyful at all. I've written my thoughts down in http://searchjaunt.idizaai.be/overpowered-or-underpowered/ Link to comment
+Lil Devil Posted July 4, 2009 Share Posted July 4, 2009 Despite that everything is up again, I don't see - all things considered - any reason to be joyful at all. I've written my thoughts down in http://searchjaunt.idizaai.be/overpowered-or-underpowered/ Wow. Who peed in your cornflakes? As was so eloquently said in another thread... While geocaching is a 24x7 sport, it is NOT mission critical. No one is gonna die or somehow be harmed if we don't get real time data for 24 - 36 hours. Link to comment
+critterkeeper Posted July 4, 2009 Share Posted July 4, 2009 Elias says in his Announcements thread that all services are up and running now, but I'm still having no luck getting a PQ to run for my parents' hometown. I'm leaving to visit them any minute now, and I won't be able to update my Colorado while I'm there, so I've been waiting all day for the PQ to run. I can't wait much longer. Is anyone else having the same problem? If it's not working, they apparently don't realize it and need to know. If it's just me, does anyone have any suggestions for how I can get results? I know things might be busy, but it normally only takes a few minutes, occasionally an hour, I just didn't expect it to take over six hours even with heavy traffic! Link to comment
+Frank Broughton Posted July 4, 2009 Share Posted July 4, 2009 Ya, no PQ's here either. Link to comment
+sTeamTraen Posted July 4, 2009 Share Posted July 4, 2009 Despite that everything is up again, I don't see - all things considered - any reason to be joyful at all. I've written my thoughts down in http://searchjaunt.idizaai.be/overpowered-or-underpowered/ One of those thought was: The servers are even slow for paying as for non paying members. I even stopped counting the times that I get a SQL errors when I want to read the forums at 8AM CET As I believe is documented here somewhere, the forums are offline for about 10 minutes every day at exactly that time, for scheduled backups. Try reading them at 0745 or 0815. This is nothing to do with the forum server performance issues (which seem to have been helped quite a bit with Elias's new index). Of course, Groundspeak could have the servers duplicated across two continents with real-time duplication of every transaction, but I doubt if there's a forum hoster on the planet who does that. Link to comment
+todd300 Posted July 4, 2009 Share Posted July 4, 2009 (edited) Thanks for the quick work and for the twitter updates. I had to text a fellow cacher for an update and he sent me the twitter link. I was then able to follow the progress on getting GC dot com up and running again. My only withdrawal was getting notifications of finds, DNF's and new caches that I get in my e-mail daily. I already had over 2000 caches in my GSAK database still unfound so it's not like I could not go out in the field if I wanted to. I'm taking a break though due to a nasty poison ivy rash on my arms from searching for a cache near some poison ivy. Tomorrow will break my longest Geoslump (currently at 4 days without a find, tying my longest slump). But I'm not worried about that either. Soon as the poison ivy rash heals up again, I'll be back out in the field. I may use this time off though to finally get around and planning some of my own hides that I've been neglecting to do for the last couple weeks. Again, thanks to Elias and Jeremy for a job well done. I'm sure PQ's will start running again tomorrow. I usually get mine in the early mornings. I didn't get mine today, but that's because when I usually get my PQ's is about when the server was still down at the point. Notifications started coming in in late morning and they still are. So I'm sure PQ's will come soon enough (I only have one PQ scheduled for tomorrow). If PQ's are not running, be patient. They will soon enough, I'm sure. And to searchjaunt..who pissed in your wheaties? Edited July 4, 2009 by todd300 Link to comment
+Shilo Posted July 5, 2009 Share Posted July 5, 2009 Despite that everything is up again, I don't see - all things considered - any reason to be joyful at all. I've written my thoughts down in http://searchjaunt.idizaai.be/overpowered-or-underpowered/ Although I agree with SOME of what you wrote, the thing is, if you don't like it go start your own caching service and do it the way you want. Base it in Europe and have your date, time format and push your own centric rules for authoring geocaching. No one is stopping you. This is way off topic anyways. I do my own criticizing of Groundspeak but a lot(at least 2 that we heard about) of people returned home from their personal lives and spent a lot of hours trying to correct a bad situation and lost sleep so everyone who uses this service could again. Lest your foreigners remember is was a US holiday as well. That was appreciated by 99% of everyone here no matter how we feel about the everyday issues. Link to comment
+hallycat Posted July 5, 2009 Share Posted July 5, 2009 Thanks for the hard work Elias, Jeremy and the rest of the GS staff. How neat that the worldwide community banded together to pass the word on the online updates via Twitter and Facebook Link to comment
+todd300 Posted July 5, 2009 Share Posted July 5, 2009 BTW, PQ's are working again. I got today's PQ about a couple hours ago. Link to comment
+searchjaunt Posted July 5, 2009 Share Posted July 5, 2009 Despite that everything is up again, I don't see - all things considered - any reason to be joyful at all. I've written my thoughts down in http://searchjaunt.idizaai.be/overpowered-or-underpowered/ Wow. Who peed in your cornflakes? As was so eloquently said in another thread... While geocaching is a 24x7 sport, it is NOT mission critical. No one is gonna die or somehow be harmed if we don't get real time data for 24 - 36 hours. Nobody peed on my flages (I ate an egg ). Having said this, I don't understand why everybody's happy whilst there is no reason to. Within your philosophy, you'd probably undergo everything without question since nothing is mission critical as long as your life doesn't depend on it. Things happen(ed) that one can't cover with the cloak of charity anymore. Link to comment
+sTeamTraen Posted July 5, 2009 Share Posted July 5, 2009 I don't understand why everybody's happy whilst there is no reason to I'm not. Fixed that for you. Of course, it could be that you have a monopoly on the truth and we're all deluded, and should be protesting in the streets that the private company with whom we choose to do business to the tune of $30 a year, does not run its operations the way that we would if our necessarily limited understanding of the personnel, technical, and financial constraints on their organisation turned out to be 100% correct. Within your philosophy, you'd probably undergo everything without question since nothing is mission critical as long as your life doesn't depend on it. Well, that might even be quite a good way to live one's life - I wish I could manage the stressful moments in my life that way more often than I do - but in any case it's a straw man argument. Link to comment
+searchjaunt Posted July 5, 2009 Share Posted July 5, 2009 Of course, it could be that you have a monopoly on the truth and we're all deluded, and should be protesting in the streets that the private company with whom we choose to do business to the tune of $30 a year, does not run its operations the way that we would if our necessarily limited understanding of the personnel, technical, and financial constraints on their organisation turned out to be 100% correct. Let me assure you that I don't have the monopoly on the truth at all, in the contrary. But groudspeak has the monopoly on the gc.com data and if they wish to do business with 30$/member/year it's OK with me, but than one doesn't have to say that one doesn't have any money. As said in the blog, either they raise the membership fee, either they revise their business model and see other (free) services how they seem to cope with it. As a paying 'customer', I think I have the right to question things as things go wrong. That counts for all goods I pay for. Link to comment
+nashuan Posted July 7, 2009 Share Posted July 7, 2009 I sat in on a GSAK class a couple of weeks ago and was going to finally dive into this program I've heard so much about when the server went down - ARGHHHHHH ! I had caching plans for the 4th of July, and was eager to load all the new caches into GSAK and then choose the ones I wanted to transfer to my GPSr. When I saw the site had gone down, I didn't think much about it at first, but after a few hours, I started to get antsy. A few hours after that I was "eager" (ya, I'll stick with that description) to find out what had happened. I checked into my local state-forum (New Hampshire) and found that others were asking questions too. Soon thereafter, I tracked down Jeremy's Twitter feed (never used Twitter before) and though still nervous about not having the info I wanted at my fingertips, at least I was getting updates as to where we stood. I'm not tech-savvy enough to know if we could have had redundant servers in separate locations - if that is even possible, and maybe this could have been solved as easily as throwing a switch, but I will say that with what we had set up that everyone did a great job in getting things back up quickly under the circumstances - thank you for your efforts. Link to comment
+cron Posted July 7, 2009 Share Posted July 7, 2009 I do my own criticizing of Groundspeak but a lot(at least 2 that we heard about) of people returned home from their personal lives and spent a lot of hours trying to correct a bad situation and lost sleep so everyone who uses this service could again. Lest your foreigners remember is was a US holiday as well. That was appreciated by 99% of everyone here no matter how we feel about the everyday issues. Wow. I guess Groundspeak, as a company, is responsible for the service they're offering. I also guess some people in that company must make a living out of it. It's great they came back to fix things up, but don't forget it's their business and that's what their clients were expecting them to do. They can charge overtime if they wish. You might also want to note that a better model would have prevented the need to come back. If everyone's philosophy about the "life and death situation" is identical, I'll be happy to sell all of you a service to mow your lawn. That will be 30$ a year. Pretty cheap. I'll also provide the lanwmower and the gas. I'll probably never show up, but that's ok with all of you as this is a non-critical situation and no life is in danger. I accept checks and paypal. Link to comment
+sTeamTraen Posted July 7, 2009 Share Posted July 7, 2009 (edited) If everyone's philosophy about the "life and death situation" is identical, I'll be happy to sell all of you a service to mow your lawn. That will be 30$ a year. Pretty cheap. I'll also provide the lanwmower and the gas. I'll probably never show up, but that's ok with all of you as this is a non-critical situation and no life is in danger. I accept checks and paypal. A more appropriate analogy would be if you promised to mow my lawn every day, with occasional outages for sickness or your car breaking down on the way. For $30 a year, I'd forgive you the one time in nine years when you were 29 hours late because the bridge was out. I wouldn't criticise you for choosing to live in a suburb which is connected to my part of town by only one highway bridge and not two. This site is remarkably reliable, given the limited amount of resources which are put into it . Sometimes it's slow, sometimes a software update doesn't go to plan, but I reckon overall uptime is well north of 99.8%, 24/7. That kind of number maybe wouldn't keep Google's shareholders happy, but for a site running on what Jeremy himself calls "hobby code" it ain't bad either. AFAIK, Geocaching.com has never been hacked and has never lost anybody's cache data, and all that despite using The Evil Microsoft Platform. If you get a chance, I strongly recommend a visit to Groundspeak's offices in Seattle. It doesn't take very long, and you will have a different perspective on the company afterwards. Edited July 7, 2009 by sTeamTraen Link to comment
+cron Posted July 7, 2009 Share Posted July 7, 2009 A more appropriate analogy would be if you promised to mow my lawn every day, with occasional outages for sickness or your car breaking down on the way. Sometimes it's slow, sometimes a software update doesn't go to plan, but I reckon overall uptime is well north of 99.8%, 24/7. I guess you got the sense of my post. It's true they have a very good uptime. Unfortunately, the 0.001% downtime is what clients see (I know, I'm in that business). I don't know how to say that in English, but "le passé n'est pas garant de l'avenir" (loosely translated by "past doesn't make future"). What never happened can happen anytime, and that's why you have to be prepared. I think they learned some lessons and they will react appropriately (based on the revenues they have). A revised model could give them more money to react in a better way, but it's a choice they will make. Link to comment
Recommended Posts