Jump to content


  • Posts

  • Joined

  • Last visited

Everything posted by Elias

  1. At approximately 4 PM Pacific Time (GMT -7), we released a site update which should have corrected the performance issues we experienced throughout the day. We'll continue to monitor the site closely this evening, but if you do notice anything not working correctly, please let us know. -Elias
  2. There was an issue with the Market and our app didn't get released. Its been corrected and should be available now. Elias
  3. Tonight around 7:00 PM PST (GMT -8), we'll be performing some memory upgrades to the servers which handle these forums. The downtime should be quick and we're only anticipating this work will take about 30 minutes. Elias
  4. We did have an issue where generated Pocket Queries were not available for download. This has been fixed and all previously generated PQs should now be available to download via the download tab. Elias
  5. We're still tracking down some random performance issues, but the site should be stable now. We'll continue to monitor it closely throughout the day. Elias
  6. We've identified the issue causing the site slowdowns and are working to resolve them. We should have this resolved shortly. Elias
  7. As Groundspeak and Geocaching.com have grown, the demands on our production network infrastructure have increased significantly. As a result, on Tuesday December 14, 2010 at 8:00 PM PST (GMT -8), we will be performing a number of major upgrades to our production network. These upgrades include a new network design as well as new hardware to help increase the performance, scalability, and security of the network. We're estimating that the downtime may take up to 6 hours, and during that time, all Groundspeak sites and services including Geocaching.com, the Groundspeak Forums, Pocket Queries, Geocaching Mobile Applications, and other 3rd party applications that access our API will be inaccessible. While we're giving ourselves a 6 hour window to cover anything unexpected that might come up, we'll be doing everything possible to minimize the downtime and are confident that we'll have the site back up much sooner. Thanks, Elias
  8. The short answer: I made a mistake. The long answer: We started getting reports from Road Runner servers that we were sending email to them too fast. As a result, they started heavily throttling us, and email to Road Runner customers was backing up in our mail queue faster than they'd let us send them. The workaround we were given was to reduce the number of simultaneous connections to their servers. To do this, I had to create a number of special exception rules on our mail server to reduce the connection count for just Road Runner domains. There are a number of domains I had to create rules for: rr.com, roadrunner.com, etc..., and during the copy and paste process, I made a type-o on the roadrunner.com rule which caused all email to the roadrunner.com domain (and only the roadrunner.com domain) to bounce. As I was watching the logs after I created the rules, I saw lots of email being delivered to Road Runner servers following the connection limits I had specified, and so I erroneously assumed that all the rules were working correctly. This has been corrected, and I do see email for the roadrunner.com domain being delivered successfully. I sincerely apologize for this mistake. Elias
  9. Tonight around 10:30 PM Pacific Time (GMT -7), we'll be taking the database server offline to apply some important updates. The downtime for this work is anticipated to take under an hour. Elias
  10. The three PQs that ran first thing this morning did generate and were successfully delivered to your Gmail account. Do you have Gmail set to forward those emails to another address, or are you accessing your mail though an IMAP or POP-based email client? Gmail is usually extremely reliable, so I'm wondering if there's something else that may be affecting you. I'll send you our mail server logs in a PM. Elias
  11. I checked the mail server logs and the email we delivered was a little over 400 KB. Something downstream from Groundspeak must have corrupted your file. Apr 26 18:45:15 signal postfix/smtpd[1957]: A8AA44A0046: client=twiki.Groundspeak.com[] Apr 26 18:45:15 signal postfix/cleanup[11953]: A8AA44A0046: message-id=<20100427014515.A8AA44A0046@signal.Groundspeak.com> Apr 26 18:45:15 signal postfix/qmgr[18538]: A8AA44A0046: from=<noreply@geocaching.com>, size=415495, nrcpt=1 (queue active) Apr 26 18:45:16 signal postfix/smtp[29367]: A8AA44A0046: to=<>, relay=aspmx.l.google.com[]:25, delay=1.1, delays=0.1/0/0/1, dsn=2.0.0, status=sent (250 2.0.0 OK 1272332716 10si7288524pzk.119) Apr 26 18:45:16 signal postfix/qmgr[18538]: A8AA44A0046: removed Elias
  12. The new image server is in place and appears to be working correctly. There will be some images from the last couple of days which haven't been copied over yet that will appear to display correctly, but will produce errors when using some of the site's image editing tools such as rotatating an image. We've started copying over those remaining images to the new server, but it will take several days to complete. I'll post an update here again when all of the images have been moved over. Elias
  13. Other than the second login screen you're seeing, this is the expected behavior. When you explicitly click "Log out", we invalidate all your sessions and log you out from everywhere. This is a security measure we put in place a long time ago to protect your account. I did a quick test and didn't see the second login screen, but I'll keep trying to see if I can reproduce it. Elias
  14. Starting in the next few minutes, we'll be swapping in new image server hardware which will hopefully alleviate the performance issues we've been experiencing during peak times. Over the course of the next hour or so, you may notice some strange behavior with images like broken links, but those should correct themselves shortly. I'll post another update here when the swap has been complete. Elias
  15. Elias

    Querry havoc

    I checked the mail server logs and found that your ISP rejected the emails. The message your ISP's server gave was: 571 5.7.1 Message contains spam or virus or sender is blocked : 17080:3159550304|6A6C (in reply to end of DATA command)) You'll need to contact your ISP and ask them to unblock emails from Groundspeak. Let me know if there's anything else I can do to help you track down this issue. Elias
  16. We have a bug on file for this and it should be fixed in the next release. Elias
  17. Thanks for the report, I've opened a bug for this. Elias
  18. Those are unrelated to the specific image server issue we had which affected image uploads. Elias
  19. This should be resolved. Please let us know if you encounter this again. Elias
  20. We may already have this bug on file, but I added a new one anyway. We'll take a look at this soon. Elias
  21. At approximately 12:40 AM Pacific Time (GMT -8), we made a change to the networking configuration of the Geocaching site. While we don't anticipate any issues from this change, if you do notice something not working correctly, please let me know. Thanks! Elias
  22. The site is back up! Thanks for your patience during this downtime! Elias
  23. Its a little after 1:00 AM and the upgrades went well. We're waiting for the restore to complete, but should have the site up in about an hour. Elias
  24. Its now 10:00 PM Pacific Time (GMT -8) and we're still completing the backups of the data. We're estimating that we're about a hour behind schedule so the site probably won't be back up until a little after 1 AM. If anything changes to this estimate, I'll post another update here. Elias
  25. On Tuesday, February 23rd starting at 8 PM Pacific Time (GMT -8), we'll be performing upgrades to the Geocaching.com database server in preparation for the increase in Geocaching over the spring and summer months. The expected downtime for this work is 4 hours and during that time, all Geocaching.com services including the web site, Pocket Queries, the Geocaching iPhone application, Geocache Navigator, and other 3rd party applications that access our API, will be inaccessible. We'll have a webpage set up on the Geocaching site with this downtime notice and we'll provide status updates if the downtime exceeds the 4 hour maintenance window. We'll also post updates to this forum thread as well. Thanks, Elias
  • Create New...