mambero

Agreed, and this is why we'll occasionally tweak the settings. As soon as we publish numbers, we'll be expected to keep them. In order to keep the site available for everyone, we need to be able to prevent people from abusing the system, whether they realize they are or not. This is the beginning of the process.

You will see captcha when you are throttled, and we'll be monitoring captcha use to catch abusers. That's the only question I can answer directly for two reasons. First, answering the others gives too much information to those who look to abuse our system with automated tools. Second, we'll be continually changing these settings, so any info I give now will be short lived.

For the last hour, the update looks solid. We've seen two users use the Captcha form and 4 users repeatedly suspended. And we can see that those 4 users were using automation to retrieve data. Looking forward to reviewing the logs tomorrow. A sentence I never thought I'd say or type.

We're in the process of upgrading the site now with updated throttling code. Here's what we've added: 1) If you are throttled, the page you'll see will have a Captcha form, which will allow you to enter text to prove that you are not a bot. When you pass the Captcha test, we'll remove the throttle and you'll be able to browse the site again. We'll also log information the information we used to determine why you were throttled. This will help us identify bugs in our algorithm and perhaps users who do not realize they're using a tool that over burdens our servers. Passing the Captcha test does not prevent you from being throttled again. In effect, it resets our stats and lets you start anew. If you trigger the throttling code again, you’ll be shown the Captcha form again. 2) We’ve updated the algorithm based on data we've received over the last week. We'll continue to update this to make it as accurate as possible. That’s all I can say about that.

The other two ran at 2:07 PST. If you haven't received them yet, please let check your spam folder and if they're aren't there, let us know.

Questions and observations: 0.01% of users per hour Where is the 80,000 premium users from? Are basic users affected as well? I thought they were, but perhaps I was mistaken. All users must pass through the gauntlet.

80,000 premium members, so .01% would be a grand total of 8 users being throttled? I kinda think from the volume of this thread that may be off just a tad. You would think that was off tad given the tone of the thread. I haven't calculated numbers from before we fixed the time sync issues, but after the sync, yes, we're looking at an avg of 10 users per hour being throttled. This does not include accounts we have identified as bots, which have a much higher rate of being throttled. To be fair to the tone of the thread, most of the complaints were filed before we fixed the time sync bug. At that point, I would be (and was) quite frustrated as well.

OK - scary things are happening... I'm agreeing with sbell111. I think this is one of the signs of the coming of the Apocalypse. Anyway, it isn't really clear - are people still being throttled after the time sync issue is fixed? We're seeing less than .01% of users/hr being throttled. We have yet to determine how many of those are valid. We're still seeing reports from users that they're being throttled, but very few in comparison to before the time sync issue. We plan to release an update to the throttling code tomorrow to gather more information to help determine what issues still exist.

Could you please point use to even one example of a Greasemonkey script that has caused the throttling trigger to trip? I'm having a tough time seeing a script engine blamed for a problem possibly caused by server synchronization issues. I'm no Greasemonkey guru, but it seems to me that everything it does is client-side and occurs only after the html has already been fetched (anybody that knows differently, please educate me). A GM script can fetch things from the server if the programmer decides to do so. But I haven't seen any of that in the scripts that I've been using. In fact, Lil Devil's most recent script for the maps pages should actually lessen the load on the servers by preventing all the extra cache-loading while scrolling. Blaming GM scripts is taking a cheap shot - blaming users opening tabs on their own is just plain low. I honestly don't know enough about the GM scripts being used to know whether they fetch additional data from the site or not. We know that it's possible and thus can't rule it out as a possibility. I also appreciate that there are some talented GM script creators who make our site easier to use. They are not being targeted. We are just doing our due diligence of investigating all possibilities.

So IP address is one of your primary criteria for deciding whether activity comes from a bot or not? There are at least 3 premium members in my office. We all appear to come from the same external IP. If we all access the site simultaneously, you're saying we stand a higher chance of being flagged as TOU-violators? Not if you're logged in.

For the next few weeks, Sunday PQs will be delivered later than usual if you haven't received your PQ by 6am PST. We have several plans in place to improve this over the next few weeks. I am in Ca. last night our reveiwer published 12 new caches so, I thought I would run a PQ and go and get them.. that was Saterday night at 11 pm or so.. and now its 2:40 (1420 pst) and as of yet there are not PQs. That has really messed up our caching day... ugh. Any ideas as to what is going on? Shastablue2 I can see that you had two PQs run today at 3 & 4 pm, and you currently have no PQs selected to run. I don't see any as having run yesterday. Were those the PQs you were expecting? Last night, did you select them to run for Saturday or Sunday. If Sunday, it looks like they were near the end of the list.

We found a culprit. The time sync for one of our servers had failed and it was two minutes faster than the others. This could have caused our algorithm to think some users were accessing the site at a higher rate than they were. We got the time synced and saw the rate of suspensions drop. We're not convinced this was the only culprit and will continue to investigate for other causes.

The challenge is determining normal users from bots. Bots access the system the way normal users do, accept that the bots can access the system much faster. We've set limits that we've only been able to exceed by repeatedly hitting F5 on a throttled page, without taking time to let the page fully refresh, let alone read any of the page. If you're being throttled, please let us know what pages you were browsing before you first saw the throttling code, along with the browser you're using and any plugins and grease monkey scripts you may be using for the site. We are trying to reproduce the problem so we can fix it.

One thing that may be confusing is that not every page is throttled. The main page for example is not throttled. Even if you're throttled, you can access the home page. The only pages we've throttled are those that heavily use the db or allow site scrapers to violate the terms of use. So, you may have triggered throttling code, gotten the error, went back to the home page, thought you were good-to-go, and then went back to a throttled page and gotten the "Whoa" message again. It's easy to think that you were throttled a second time when you were most likely still throttled from the first time.

I wish this were true. It would be an easy way to relieve the pain for a certain group of users. Unfortunately, several of the bots were using sock puppet accounts (thanks to Jon Stanley for getting me hip with the lingo), and some IPs use more than one paid account, and some of the bots were using accounts from more than one IP. So, it's not like we're being targeted by an intelligence agency, but it's more than just someone who turned on a screen scraper from their home server.

Well, I can guarantee that we're not off enjoying our weekend. And we all agree that this was not an ideal time to upgrade the site, but it was also not an ideal time to sit back and watch the site and mobile apis become unresponsive as they have in Sundays past. We're in the proverbial "rock and a hard place" scenario. In response to users who are reporting being temporarily suspended, we've loosened the settings a bit to let more people in. We have not yet been able to reproduce a scenario in our tests where we falsely identify super humans, so we're not able to identify what there is to fix. We'll be upgrading the site early next week to capture more info. On the positive side, we've been able to identify numerous bots that have been heavily hitting our site and causing the significant site slowdowns you've experienced the last few weeks. For the last several hours, we've been seeing large traffic to the site and it has remained responsive. If we had not removed those bots, the site would certainly be at a crawl right now and the mobile apis would likely be unresponsive, given what we know from the last few weeks. To those who have been intermittently blocked from the site due to false positives, know that I'm truly sorry and working feverishly to identify what could be causing this, even on this holiday. My wife is not very happy about it, and thus, it's a far stretch to say that I'm off enjoying my weekend.

Are you still seeing this. We've tried to reproduce the problem, but it's working fine for us on the major browsers.

I've added a note in the bug to post here when it's fixed.

For the next few weeks, Sunday PQs will be delivered later than usual if you haven't received your PQ by 6am PST. We have several plans in place to improve this over the next few weeks.

Are you still seeing this? I just ran the query used by the PQ generator and the date for that log was March 5. Just ran my My Finds PQ before posting this message and it still showed as March 6. I change it in GSAK to March 5 but when I load a new My Finds it changes back to March 6. I have some ideas about why this is happening, but due to work to improve site performance for this weekend, don't have the bandwidth to work on it at the moment. Thus, bug #15877 was born at approximately 12:52 PST.

Are you still seeing this? I just ran the query used by the PQ generator and the date for that log was March 5.

We seem to be through the peak period. We've resumed the normal processing load for pocket queries, so if you haven't seen yours yet, you should in the next couple of hours.

We'll need more info to investigate. Can you give us the cache code along with the original & revised date and time of the log?

Yes, the servers are under a heavy load right now. This is our busiest time of the week. We're monitoring performance and tweaking what we can.

It looks like the ones you've created that haven't run have don't have any days selected.