Elias
Members-
Posts
900 -
Joined
-
Last visited
Everything posted by Elias
-
Geocaching.com site update August 18th 2011
Elias replied to OpinioNate's topic in Geocaching HQ communications
At approximately 4 PM Pacific Time (GMT -7), we released a site update which should have corrected the performance issues we experienced throughout the day. We'll continue to monitor the site closely this evening, but if you do notice anything not working correctly, please let us know. -Elias -
There was an issue with the Market and our app didn't get released. Its been corrected and should be available now. Elias
-
Tonight around 7:00 PM PST (GMT -8), we'll be performing some memory upgrades to the servers which handle these forums. The downtime should be quick and we're only anticipating this work will take about 30 minutes. Elias
-
We did have an issue where generated Pocket Queries were not available for download. This has been fixed and all previously generated PQs should now be available to download via the download tab. Elias
-
We're still tracking down some random performance issues, but the site should be stable now. We'll continue to monitor it closely throughout the day. Elias
-
We've identified the issue causing the site slowdowns and are working to resolve them. We should have this resolved shortly. Elias
-
As Groundspeak and Geocaching.com have grown, the demands on our production network infrastructure have increased significantly. As a result, on Tuesday December 14, 2010 at 8:00 PM PST (GMT -8), we will be performing a number of major upgrades to our production network. These upgrades include a new network design as well as new hardware to help increase the performance, scalability, and security of the network. We're estimating that the downtime may take up to 6 hours, and during that time, all Groundspeak sites and services including Geocaching.com, the Groundspeak Forums, Pocket Queries, Geocaching Mobile Applications, and other 3rd party applications that access our API will be inaccessible. While we're giving ourselves a 6 hour window to cover anything unexpected that might come up, we'll be doing everything possible to minimize the downtime and are confident that we'll have the site back up much sooner. Thanks, Elias
-
The short answer: I made a mistake. The long answer: We started getting reports from Road Runner servers that we were sending email to them too fast. As a result, they started heavily throttling us, and email to Road Runner customers was backing up in our mail queue faster than they'd let us send them. The workaround we were given was to reduce the number of simultaneous connections to their servers. To do this, I had to create a number of special exception rules on our mail server to reduce the connection count for just Road Runner domains. There are a number of domains I had to create rules for: rr.com, roadrunner.com, etc..., and during the copy and paste process, I made a type-o on the roadrunner.com rule which caused all email to the roadrunner.com domain (and only the roadrunner.com domain) to bounce. As I was watching the logs after I created the rules, I saw lots of email being delivered to Road Runner servers following the connection limits I had specified, and so I erroneously assumed that all the rules were working correctly. This has been corrected, and I do see email for the roadrunner.com domain being delivered successfully. I sincerely apologize for this mistake. Elias
-
Tonight around 10:30 PM Pacific Time (GMT -7), we'll be taking the database server offline to apply some important updates. The downtime for this work is anticipated to take under an hour. Elias
-
The three PQs that ran first thing this morning did generate and were successfully delivered to your Gmail account. Do you have Gmail set to forward those emails to another address, or are you accessing your mail though an IMAP or POP-based email client? Gmail is usually extremely reliable, so I'm wondering if there's something else that may be affecting you. I'll send you our mail server logs in a PM. Elias
-
I checked the mail server logs and the email we delivered was a little over 400 KB. Something downstream from Groundspeak must have corrupted your file. Apr 26 18:45:15 signal postfix/smtpd[1957]: A8AA44A0046: client=twiki.Groundspeak.com[66.150.167.139] Apr 26 18:45:15 signal postfix/cleanup[11953]: A8AA44A0046: message-id=<20100427014515.A8AA44A0046@signal.Groundspeak.com> Apr 26 18:45:15 signal postfix/qmgr[18538]: A8AA44A0046: from=<noreply@geocaching.com>, size=415495, nrcpt=1 (queue active) Apr 26 18:45:16 signal postfix/smtp[29367]: A8AA44A0046: to=<>, relay=aspmx.l.google.com[209.85.222.10]:25, delay=1.1, delays=0.1/0/0/1, dsn=2.0.0, status=sent (250 2.0.0 OK 1272332716 10si7288524pzk.119) Apr 26 18:45:16 signal postfix/qmgr[18538]: A8AA44A0046: removed Elias
-
The new image server is in place and appears to be working correctly. There will be some images from the last couple of days which haven't been copied over yet that will appear to display correctly, but will produce errors when using some of the site's image editing tools such as rotatating an image. We've started copying over those remaining images to the new server, but it will take several days to complete. I'll post an update here again when all of the images have been moved over. Elias
-
Other than the second login screen you're seeing, this is the expected behavior. When you explicitly click "Log out", we invalidate all your sessions and log you out from everywhere. This is a security measure we put in place a long time ago to protect your account. I did a quick test and didn't see the second login screen, but I'll keep trying to see if I can reproduce it. Elias
-
Starting in the next few minutes, we'll be swapping in new image server hardware which will hopefully alleviate the performance issues we've been experiencing during peak times. Over the course of the next hour or so, you may notice some strange behavior with images like broken links, but those should correct themselves shortly. I'll post another update here when the swap has been complete. Elias
-
I checked the mail server logs and found that your ISP rejected the emails. The message your ISP's server gave was: 571 5.7.1 Message contains spam or virus or sender is blocked : 17080:3159550304|6A6C (in reply to end of DATA command)) You'll need to contact your ISP and ask them to unblock emails from Groundspeak. Let me know if there's anything else I can do to help you track down this issue. Elias
-
We have a bug on file for this and it should be fixed in the next release. Elias
-
Thanks for the report, I've opened a bug for this. Elias
-
Those are unrelated to the specific image server issue we had which affected image uploads. Elias
-
This should be resolved. Please let us know if you encounter this again. Elias
-
We may already have this bug on file, but I added a new one anyway. We'll take a look at this soon. Elias
-
At approximately 12:40 AM Pacific Time (GMT -8), we made a change to the networking configuration of the Geocaching site. While we don't anticipate any issues from this change, if you do notice something not working correctly, please let me know. Thanks! Elias
-
Geocaching.com Scheduled Downtime - Feb 23, 2010
Elias replied to Elias's topic in Geocaching HQ communications
The site is back up! Thanks for your patience during this downtime! Elias -
Geocaching.com Scheduled Downtime - Feb 23, 2010
Elias replied to Elias's topic in Geocaching HQ communications
Its a little after 1:00 AM and the upgrades went well. We're waiting for the restore to complete, but should have the site up in about an hour. Elias -
Geocaching.com Scheduled Downtime - Feb 23, 2010
Elias replied to Elias's topic in Geocaching HQ communications
Its now 10:00 PM Pacific Time (GMT -8) and we're still completing the backups of the data. We're estimating that we're about a hour behind schedule so the site probably won't be back up until a little after 1 AM. If anything changes to this estimate, I'll post another update here. Elias -
On Tuesday, February 23rd starting at 8 PM Pacific Time (GMT -8), we'll be performing upgrades to the Geocaching.com database server in preparation for the increase in Geocaching over the spring and summer months. The expected downtime for this work is 4 hours and during that time, all Geocaching.com services including the web site, Pocket Queries, the Geocaching iPhone application, Geocache Navigator, and other 3rd party applications that access our API, will be inaccessible. We'll have a webpage set up on the Geocaching site with this downtime notice and we'll provide status updates if the downtime exceeds the 4 hour maintenance window. We'll also post updates to this forum thread as well. Thanks, Elias