Jump to content

Another Question About The Slow Website


Cheminer Will

Recommended Posts

This thing of extremely -- or, in a number of recent cases, nonexistent -- slow server performance is becoming astounding. At 8 AM Eastern time this Saturday morning, I could not even access the server for cache listings at geocaching.com. I have no idea why the folks at geocaching.com cannot simply contact their webhosting vendor and ask them to incrase bandwidth capacity or to add more servers, whichever would be the most appropriate response in this case. I suspect that such persistent problems will likely drive some folks away from the sport, or at least drive them away from geocaching.com. I doubt the slow response time from the administration is due to lack of funds; between my wife and I alone, we support three Premium accounts.

 

As you known from my past posts, there are a great many things about this site and service that I appreciate very much. Please understand that this note is offered not as a criticism but as a reminder and a bit of an urging that you (administrators) may wish to pick up the phone and speak to your webhosting folks, and quickly arrange to upgrade capacity. As someone with a strong IS/IT background, there seems little reason to me why such problems should be persisting for this long after they first surfaced.

Edited by Vinny & Sue Team
Link to comment

Here's an idea that's bound to be unpopular with many...

 

When the server load gets too high, how about restricting logging of caches and TBs to premium members?

 

Told you it would be unpopular...

 

Paul

 

That would be fine by me :laughing:

 

It wouldn't be going against the theory of not having to pay to play, because those who don't have PM's could still get the coords., but would simply have to wait to log.

 

It would also seemingly tempt more people into getting a PM, which could eventually equal more server power to speed things up.

 

...but I say this in a moment when I am unable to log, so take it for what it's worth.

 

Speaking as someone whose household supports 3 Premium member accounts (and sometimes a fourth for cache maintenance and management purposes), I agree with these folks. The site could be set up on a kind of triage basis so that Premium members have greater priority to access the site, and enjoy greater speed of access. This does not deprive non-paying members of access to the site, but simply means that there may be times when service is slow or not available.

Link to comment

Some numbers - just FYI - shows trend by day - based on my pocket queries

 

I analyzed the "last logged" date shown in gsak for 1546 unique caches logged in the past 4 weeks for my location around Milwaukee Wi - this info shows the trend - not total number of log entries - but you can get a general idea...

 

Last logged date for past 28 days

 

Mon = 128 - 8%

Tue = 131 - 8%

Wed = 195 - 13%

Thu = 169 - 11%

Fri = 95 - 6%

Sat = 263 - 17%

Sun = 565 - 37%

-----------

1546 caches located around Milw WI

 

So as we already know - trying to make log entries on Sat/Sun might be an experiment in futility. :laughing:

 

-Cheers

Edited by Seek+Hide
Link to comment

Some numbers - just FYI - shows trend by day - based on my pocket queries

 

I analyzed the "last logged" date shown in gsak for 1546 unique caches logged in the past 4 weeks for my location around Milwaukee Wi - this info shows the trend - not total number of log entries - but you can get a general idea...

 

Last logged date for past 28 days

 

Mon = 128 - 8%

Tue = 131 - 8%

Wed = 195 - 13%

Thu = 169 - 11%

Fri = 95 - 6%

Sat = 263 - 17%

Sun = 565 - 37%

-----------

1546 caches located around Milw WI

 

So as we already know - trying to make log entries on Sat/Sun might be an experiment in futility. :laughing:

 

-Cheers

 

Although I do not doubt those percentages, I still had trouble this past Monday and Tuesday (and Friday, the lowest day according to those statistics) due to "server too busy" errors. It's not just a weekend problem anymore. Otherwise, my broadband connection works just fine. I know they're "workin' on it" and that the problem worsens in spring months, but it's become a perpetual issue.

Link to comment

Nearly two hours now tryiny to get PQs for tomorrow ........

 

 

 

Server Too Busy

Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.

 

Exception Details: System.Web.HttpException: Server Too Busy

Link to comment

Some numbers - just FYI - shows trend by day - based on my pocket queries

 

Last logged date for past 28 days

 

Mon = 128 - 8%

Tue = 131 - 8%

Wed = 195 - 13%

Thu = 169 - 11%

Fri = 95 - 6%

Sat = 263 - 17%

Sun = 565 - 37%

-----------

1546 caches located around Milw WI

 

So as we already know - trying to make log entries on Sat/Sun might be an experiment in futility. <_<

 

-Cheers

 

We had trouble getting on, (server errors) yesterday. So I think the problem is unrelated to load if Friday is such a slow day. It now seems the site can be unavailable or slow at any day and at any time. I think the problem is now big enough so it will have to be fixed soon. Good for us, bad for all the overworked techies that have to fix it!

Link to comment

okay, i'm still pretty sure that there are a lot of providers that can take this kind of traffic without problems. but another idea ??

 

offline logging.

 

there is probebly somebody (no not me, i just have the idea) that can build a small program where you just enter the data (cacheID, date, time, TB numbers that you dropped and picked up and the textlog) and that will put it on the site when there is a free connection.

 

Now there are people that start screaming that they don't have a DSL or Cable connection... well thats just to bad for you. For the others that do have it it will be possible to enter the data and just send it. the program then will try to upload it and when it's done it will tell you in the program....

 

stupid idea ?? tell me a better one...

 

just trying to help here from the netherlands <_<

 

btw: i love geocaching haahahaha

Link to comment

The site issues have nothing to do with bandwidth. The problem cannot be solved just by adding servers, because the architecture of the application does not currently allow the live transactional database to be served from more than one machine. There is a read-only database used to reduce traffic on the live database.

 

The problems can only be resolved by improving performance at the database level, and eventually working towards a more scalable database design. Right now, there really is not a single low-cost RDBMS which is drop-in scalable to multiple servers. Oracle has some features which SQL Server doesn't have with regard to active-active clustering, but such a change would be extremely large and expensive. SQL Server has some great scalability, but it requires configuration expertise and design changes to applications as well. How much work should be done where can only really be determined by those who know the system well and those who can identify the performance bottlenecks which will have the most impact.

 

If any of you recall ebay and paypal's massive availability problems a few years ago, it can even happen to well-funded and experienced database shops.

Link to comment

 

offline logging.

 

 

This is not a realistic solution (though it's a nice feature). The issue is that the site can take quite a bit of traffic. What it has difficult doing is ramping to accomodate bursts of traffic - for instance when everyone returns to post log entries at the same time. This is a different issue than, say, eBay or PayPal when the payments are more or less evenly distributed over the week.

 

Changing a user's behavior is not the solution. The site needs to keep up during high-traffic times. Offloading some of the more heavy queries to replicated machines like readonly databases helps. Also just making your queries respond better is important. There isn't any big picture solution at this point. We just have to pick the low hanging fruit and keep moving up the tree until the site responds well to these bursts of traffic.

 

For those seeing issues on days like Friday is because we're actively messing with the servers to tweak performance. It is no symptom for another issue. The database is ultimately the issue here.

Link to comment

 

the problem is really simpel: the webhost can't handle this kind of traffic. there are providers and servers that handle much much more then just a few geocachers.

 

hotmail/google/yahoo/gmail/ they have thousends of hit per second and you never see a timed out on google....

 

 

never say never I've had hotmail AND google time out on me...

 

And Yahoo

Link to comment

In the dutch forum we also discussed the GC.com troubles. We discussed some options:

  • Maybe it's possible to give the users who are premium members some higher priority. They are paying for the service, the non-paying members have to wait some longer at bizzy hours, but they can still do their hobby.
  • Maybe it's an idea to think about non-microsoft controlled systems. We all know that for example Linux servers can be made much more stable.

Link to comment

At this point in time there is a log entry a second being entered into the database.

 

 

hum so jeremy how many logs are there in a day?? a month, a year??

it would be cool to see it as it happends ( thats after you all get the site servers working again with out problems) and if it is possable to do it like that...

 

 

 

just for fun i did the math on it

 

so if there was some one logging at 1 log per sec

 

it will be

 

1 day

 

around 86,400 logs

 

and about

 

2,629,743.83

logs per month

 

then

31,556,926 logs a year

 

dang...

----------------------------------------------------------------------------------

keep up the good work guys

 

 

reason for edit the stupid

would not work!!!
Edited by Charles Iverson
Link to comment

hum so jeremy how many logs are there in a day?? a month, a year??

it would be cool to see it as it happends ( thats after you all get the site servers working again with out problems) and if it is possable to do it like that...

 

In the last 7 days, there have been 181186 new logs written by 28385 account holders.

Link to comment
  • Maybe it's an idea to think about non-microsoft controlled systems. We all know that for example Linux servers can be made much more stable.

 

No that is not the problem.

 

Microsoft products are more than able to handle this load. It is not a question of the product not being stable, it is an issue of scalability. It becomes tiresome to always hear people blame Microsoft and how great Linux is.

 

Microsoft SQL and IIS are more than up to the task here. Both products run in environments much larger than here. The issue is one of programming. Better code needs to be written as well as more tweaking to the database setup. Several times in the past TPTB have admitted to poor programming practices. Also it seems there has also been poor planning in predicting and adjusting for growth. There are tools that can stress load the servers and see what breaks where and when.

 

Also for those that have been talking about more bandwidth -- again, that is not the issue. The fact that one gets error messages from the server shows you are getting through. As for throwing more machines to fix the problem, I don't know. I don't know exactly how the system is configured so it is hard to say. But from what I have been able to put together that does not seem to be an issue either. It sounds like there is enough horsepower. I continue to think that better code needs to be put in place to make better use of the hardware there.

Edited by GrizzlyJohn
Link to comment

Grizzly,

 

Application pools are limited to 1.2 gigs of memory. Though the machine has the capacity to handle far more there seems to be a limitation on the ability for IIS to handle more than that per application pool. The Server Too Busy errors occur when the application pool resets and there are multiple connections while this happens which bogs down the server. The machine is too busy to handle the incoming requests and handle the compile for the system.

 

We use extensive caching and, again, garbage collection built into IIS apparently cannot handle both the garbage collection and the incoming requests from users. It essentially craps out and fills up the available memory and *pop* we have Out of Memory exceptions while the machine itself has gigs and gigs of memory free.

 

Although at busy times the database does get pretty bogged down, the inefficiency with the caching on IIS and .net 1.1 is causing a lot of issues when it should be helping the speed of the site. If we didn't have to refresh the pool every 2 hours we could store a majority of the site in memory and reduce overall load on the web site.

 

We'll be moving to 2.0 in the near future (once the blasted licenses arrive), but in the meantime we're looking at a solution that will allow us to share a memory cache across multiple servers and bypass the inefficiencies of IIS. That means the geocaches would optimally sit in memory and be accessible to all the web servers which will mean that the database will only be used for unique queries like custom searches from an origin or complicated Pocket Queries.

 

We're doing the same with session handling which is currently running on the same machine.

 

We've been running profiler on the database over the last week or so and have adjusted some of the indexing on the database to help with response times on popular queries. Additionally we have replaced some redundant queries related to the log table to speed up the ability to post logs. At this point we're hitting the wall with this application pool issue - but we're making progress.

 

Although I do note that the site was originally built as a hobby site so there's a lot of hobby code, it's definitely a different animal compared to even a year ago. We're getting there but it is resource intensive and the premium memberships only support so much labor.

Link to comment

Thank you for all your hard work.. I completely understand. I run the local Geocaching website in my area and it was nothing but a headache when I got it up and running.. I had a SQL problem but quickly fixed it. My site handles NO WHERE NEAR your traffic.

 

I have a solution!

 

Geocaching is moving towards a point where 3 dollars a month can't support the infastructure. I don't know what type of bandwidth you have, but I'm on a shared 0C3 and the donations I get in from my site help but don't cover all.

 

I would be willing, not speaking for everyone else of course, to pay more a month for a "Gold" membership. Say... 5-10 dollars a month. Give gold members a special icon, and a member card, or a coin or something..

 

Don't know...

 

Once again THANKS again for all your hard work.. I know that $36 a month just doesn't go very far when you look at hiring people to maintain a database (Chief engineer of 2 television stations, I know)

 

Baralak

Link to comment

Jeremy, thank you very much for taking the time to explain where the challenges lie. I appreciate this very much. With that in mind, here a a few thoughts at this point, as I look toward the future of this organization as the sport continues to grow. I realize that a small fraction of members paying Premium Membership fees of $30 to $36 per year is likely not enough to really support the kind of IS/IT infrastructure needed, particularly the programming needs and recoding of old code. Here are my suggestions:

 

1) I would like to second Baralak's suggestion for creating a Gold level of membership, with a higher per-month fee.

 

2) I would like to repeat the suggestion that the site may wish to employ some sort of triage system, where Premium and Gold members have preferential access to high-speed service, and where non-paying members continue to enjoy access to the site, but at lower levels of speed/response during peak load times, and with some degree of triage where some modest percentage of users in that pool is denied service entirely during peak load times, on a dynamic basis. This would not only reduce loading on servers at peak load times, but it would provide an additional incentive to non-paying members to start supporting the site, resulting in a greater number of folks paying for Premium memberships (and presto, more income for the organization!)

Link to comment

that´s not funny anymore :laughing: in the past there where only problems during the weekend. now we have the situation, that over the hole week messages like "timeout", "server busy" ... occured. is anybody taking care of this problem ? it is not possible anymore to prepare myself for the next days. once again : THAT´S NOT FUNNY ANYMORE !

Link to comment
that´s not funny anymore :laughing: in the past there where only problems during the weekend. now we have the situation, that over the hole week messages like "timeout", "server busy" ... occured. is anybody taking care of this problem ? it is not possible anymore to prepare myself for the next days. once again : THAT´S NOT FUNNY ANYMORE !

 

You may want to review the 224 posts prior to yours. :D

Link to comment

Although I do note that the site was originally built as a hobby site so there's a lot of hobby code, it's definitely a different animal compared to even a year ago. We're getting there but it is resource intensive and the premium memberships only support so much labor.

 

Maybe a bit more advertising revenues would help. I certainly do not want to encourage such things as pop up ads and some of the other annoying marketing that I find on some sites, but I don't have a problem with a banner ad here and there. I actually look at banners and may follow-up. I would not follow-up on a pop-up if they were selling $20 bills for $10. Could the same high traffic that causes the problem be part of the solution? When I think of all the geocachers who drive Jeep products, there should be some bucks there for sure. FWIW, CharlieP

Link to comment

The real problem is that too many people think geocaching is about finding two dozen 1/1 caches in a single day and posting a generic one-line log to each one when they get home.

 

:)

 

<Did I just say that?>

 

Dipping briefly into my dark side, I have been soooo thinking much the same thing for the past two weeks but have been biting my tongue to keep from saying it for fear that if I did say it, the Wrathful Avenging Stark Fist of God would turn me into a giant squishy green caterpillar! My dark side has also been saying that if people want to place or seek lame 1/1 caches (not that all 1/1s are lame), they should be required to not only be Premium members, but also to pay a monthly surcharge for the privilege of doing so. Of course, this is the same dark side that thinks that people who do not file DNFs should be spanked. Oops. . ! My dark side is very bad. I will sign off now and will go spank my dark side and then make it sit in a dark corner and eat only moldy old porridge infested with mealworms for three days.

Link to comment
Dipping briefly into my dark side, I have been soooo thinking much the same thing for the past two weeks but have been biting my tongue to keep from saying it for fear that if I did say it, the Wrathful Avenging Stark Fist of God would turn me into a giant squishy green caterpillar! My dark side has also been saying that if people want to place or seek lame 1/1 caches (not that all 1/1s are lame), they should be required to not only be Premium members, but also to pay a monthly surcharge for the privilege of doing so. Of course, this is the same dark side that thinks that people who do not file DNFs should be spanked. Oops. . ! My dark side is very bad. I will sign off now and will go spank my dark side and then make it sit in a dark corner and eat only moldy old porridge infested with mealworms for three days.

 

I've just dropped by this thread to say

 

YYYYEEEAAAAAAAAAAAAAAAARRRRRRRRRGH!!!!!!

 

 

Whew!

 

Thenk yew.

 

OXYGEN! I NEED OXYGEN!!

Link to comment

It's nice to come back from dinner and not be reading a lot of hate mail.

 

Dinner? You had the luxury time to eat dinner while the site was um...well, working just fine? Glad to hear it and to experience very few timeouts today while planning a Spring Break cache outing. :)

Link to comment
The Server Too Busy errors occur when the application pool resets and there are multiple connections while this happens which bogs down the server. The machine is too busy to handle the incoming requests and handle the compile for the system.

 

dont know if you thought about or yet tested it ... ngen could help you with precompiling your website ...

 

 

Edit: wow, there are a lot of Ressources for .NET Performance ... a realy interesting job to evaluate all these :)

Checklist: ASP.NET Performance

Improving .NET Application Performance and Scalability

 

Edit2: some more to ngen ... NGen Revs Up Your Performance with Powerful New Features ... ok, it a dotnet 2.0 article ... but should basically apply to dotnet 1.1, too

Edited by Team BMW-Biker
Link to comment
My dark side has also been saying that if people want to place or seek lame 1/1 caches (not that all 1/1s are lame), they should be required to not only be Premium members, but also to pay a monthly surcharge for the privilege of doing so. Of course, this is the same dark side that thinks that people who do not file DNFs should be spanked.
So, newbies who are not yet confident enough to place tougher caches should have to pay more then experienced cachers?

How about DNFs? Since they also add to traffic, should they be reserved to Premium members.

Tell your dark side it's just not thinking straight. :(

Link to comment

... My dark side has also been saying that if people want to place or seek lame 1/1 caches (not that all 1/1s are lame), they should be required to not only be Premium members, but also to pay a monthly surcharge for the privilege of doing so.

I also believe that anyone who likes anything about this game that is different than my preferences should have to pay more than I do. Great idea!

 

:):anibad:

Link to comment

It's nice to come back from dinner and not be reading a lot of hate mail.

 

Thanks for your patience (some of you) :laughing:

 

nobody is hating you or the work you do for "us" we just hate to see the site not working perfect :)

 

but we love you :anibad:

Edited by datisgaaf
Link to comment

I think the site for the most part is working now.

 

We found 38 caches over the weekend and were able to have them all logged by Monday night. We had a few time outs on Sunday night so we waited until Monday to finish and had no problem with any of them. We also logged about 30+ coins we had seen at an event with no problems.

 

Thanks for the work in getting this fixed. You guys are great. :anibad:

Link to comment

It's nice to come back from dinner and not be reading a lot of hate mail.

 

Thanks for your patience (some of you) :)

 

nobody is hating you or the work you do for "us" we just hate to see the site not working perfect :laughing:

 

but we love you :anibad:

 

This is true! I detected in most of the posts -- even most of the most irate ones -- about the problems very little hostility toward the site admins, and rather just frustration and bewilderment.

Link to comment

Another suggestion from left field, maybe it could help for people like me.

 

I use the my account page frequently, often just to look at the page of caches close to my home. Every time I do this, the servers have to pull up every single cache I've found in the past month, my log for that cache, it's current status, and my stats sig. I really don't care. If there were an option somewhere to limit the caches shown to 10 or something, and another one to deactivate my stats sig, the page could be loaded much faster.

 

Being able to turn off the (xxx found) after member's names would probably help, too. Or maybe update it daily, instead of constantly?

Link to comment
Guest
This topic is now closed to further replies.
×
×
  • Create New...