Working Through Server Issues

[Jeremy]

The server issues have been particularly bad lately, so I'll be sitting at the office on Saturday to work throught them. The site may periodically go down as I make changes to the code to offload some of the work. Please be patient during this time while we work through the issues. I'll post status updates as I go.

[Jeremy]

I started today by offloading the travel bug logs to our readonly database. This shouldn't cause any huge latency issues with the log data on the page but it is one often used query that doesn't require a lot of updates.

 

(latency is the time between an update to a page and when you see that update)

 

Additionally I took all of the information that is required for the little stats graphic on your personal page so the database queries go to the other machine.

 

Next week we're testing a server that stores pre-rendered data into memory for quick retrieval. What this means is that before it hits the database it will check this data to see if it has already been rendered. If it has it saves a trip to the database. Since we're in a server farm environment there are a lot of queries that happen that are pretty redundant - by centralizing this data we can ultimately save on resources.

 

More to come...

[Jeremy]

Google Earth queries are now moved to the Readonly database as well. I did notice another bug with description text that I have logged for further review.

 

edit - the issue was the description of some listings. I removed the short description for now so it works. view cache details will get you the added info for now.

Edited April 1, 2006 by Jeremy

[Jeremy]

I've been watching the site over the last few hours and it seems that this latest volley of changes has helped somewhat. The database is very responsive as is the web site. I'll continue to look for ways to speed it up.

[Jeremy]

Server too busy errors at the moment are due to my resetting of the application pool settings. I'm attempting to have the site reset every 4 hours to limit the amount of server too busy errors as the site spins back up on a restart.

[iryshe]

I just adjusted some settings on the site. It'll take a few minutes to spin back up.

[Jeremy]

I think I tracked down one of the big issues with the site spinning back up. Apparently there was one query, the one you see on the front page that shows the weekly statistics, that was holding up all of the other processes. To temporarily fix the problem I went ahead and made that query static so you won't see any changes to the stats until I do it manually. Apparently this helped since the next spin up didn't take any time at all.

 

I'll continue to monitor this tonight but at the moment there is a new log being entered into the database every second. It must be a great day to be outside!

[Jeremy]

I'm updating the servers now. There will be some server too busy errors as it recompiles.

[Jeremy]

Well, with the latest changes to the site we have significantly reduced the impact on the database, which has been the worst bottleneck. Much of it had to do with isolating some of the worst database offenders and fixing their impact on the overall usage of the web site.

 

We're not satisfied with this, however, and will be addressing Session state and better caching (not geocaching, memory storage) to keep up with the constant rise of interest in geocaching (yes, the GPS game).

 

Sorry again for the poor performance of the web site over the last few weeks. This does unfortunately happen when we have an uptick in the number of folks involved with the activity but the software we're building is now written with scalability in mind. The problem is we're still converting even some of the oldest hobby code from the year 2000.