Jump to content
Sign in to follow this  
Followers 6

Geocaching.com Offline

Recommended Posts

At approximately 3:22 PM PST (GMT-8), the entire building containing Groundspeak's Colocation provider lost power. As a result all Groundspeak servers lost power and were ungracefully shut down. At approximately 3:59 PM, power was restored to the building.


Over the last few hours I've been running diagnostics on each of the Groundspeak servers. Other than a failed hard disk, no major hardware problems were found. Numerous soft disk errors were found and corrected. There were also a number of errors found in the forums database which was also corrected. No errors were found in the Geocaching.com database.


As of 8:06 PM all Groundspeak services have been restored.


At this time, the true cause of this event is unknown. The building our servers are in is part of the City of Seattle's emergency power grid. We're lucky to be on that grid as the main building tenant happens to be a TV station responsible for this area's Emergency Broadcast System. The building itself has backup generators in the event of a power loss, and our colo provider, Internap, has battery backup power supplies that cover their entire facility. But somehow, the impossible happened; the building lost power, the building's generators didn't fire up, and Internap's battery backup system failed.


I'll post more details on the cause of this outage as soon as I receive them.


:D Elias

Share this post

Link to post
Sign in to follow this  
Followers 6