Apology: Weekend Pocket Query Issues

[Jeremy]

Over the weekend there were a lot of issues with Pocket Queries being generated. For that I sincerely apologize.

 

In an attempt to speed up the Pocket Query Generator, the application(s) that processes tens of thousands of Pocket Queries a day, there was a cascading number of issues that resulted in duplicate Pocket Queries going out, Pocket Queries without logs and Pocket Queries not being sent out for a certain amount of time. Also in the process, Pocket Queries were brought down to a trickle, resulting in Pocket Queries not going out at all.

 

We're taking this issue very seriously at Groundspeak, and we'll be focusing on increasing the capabilities over this next weekend to ensure this never, ever happens again. By increasing this capacity we'll also be prepared to offer Pocket Queries with 1,000 results instead of 500. This will happen by the anniversary of geocaching (May 2, 2010). As those veterans to geocaching know, I'm not usually one to offer dates, but this one is solid.

 

Again, I offer my apology for this weekend's issues.

[+StarBrand]

Thanks Jeremy.

 

Wow a Thousand - and by my own Birthday as well!!

[+CoffeeFamily5]

Let me be the second to say THANK YOU for all the hard work and effort you folks at GS put into this system. It means a lot to me as a new member (well, 6 months is kinda new) to see someone come on here and offer an apology. Too many sites out there have the mentality of, "If they don't like it, Oh well." and it is refreshing to know that Groundspeak stands out in a crowd such as that as a company who cares.

Edited March 8, 2010 by CoffeeFamily5

[Jeremy]

I think the biggest failure for us was our failure to communicate the issues over the weekend. Whenever we touch our production machines in the future, however trivial, we'll make sure to communicate this to you as well. In addition we'll continuously monitor the forums to make sure there are no issues coming up from our changes.

[OpinioNate]

At this time the service is again functioning properly but there is a substantial queue of unsent queries - we estimate it will catch up around 11am Pacific.

 

If you queue'd up a My Finds query and did not receive it please contact our support staff (contact@geocaching.com) and they will reset it for you. Use the subject line "My Finds Reset" for fastest service.

 

I am so sorry for the inconvenience. I hope you were still able to enjoy a fun weekend of geocaching the old fashioned way. We are taking this lapse in service seriously and making every effort to prevent this happening in the future.

 

Edit: changed 11 hours to 11am.

Edited March 8, 2010 by OpinioNate

[+t4e]

thank you for the update

 

i agree that communication is important, but sometimes in the rush of fixing a problem we get busy trying to fix it and forget...so you have my understanding there

 

just don't let it happen again

 

one suggestion i have for such cases is perhaps have something in red at the top of the page linking to the forum announcement, similar to what was during the last downtime maintenance

 

hopefully everyone will see it and there won't be endless threads being started

[jholly]

Thank you for the information. The only real complaint I had was the silence for most of the weekend. Elias set a very high standard with the downtime notice and the updates during the database conversion. Thank you for your efforts to improve the service and the efforts that went into the fixes this weekend.

[+Bear and Ragged]

Sounds good!

Thanks for the info/update.

[+Matthew 7:7 Too]

Many thanks for the update. I appreciate the apology, and agree the most infuriating problem was the complete lack of any acknowledgement of the problems over the weekend by Groundspeak staff. If they could have responded to the many forum threads reporting the problems users were experiencing, at least we'd have had some clue that we were being heard!

 

Chris

[Jeremy]

Thank you for the information. The only real complaint I had was the silence for most of the weekend.

 

Yes. That was a problem. I was clear to the team today that they are more than welcome to post in the forums if they need to. There's no need to have any artificial barriers between the community and the Groundspeak Lackey Developers.

[nikhilsinghal]

Firstly, I would like to apologize sincerely for the frustration that the community has gone through during the weekend. Secondly, as the person responsible for the development and maintenance of geocaching.com, I totally own up the responsibility of this failure.

 

The team is working around the clock on fixing this issue and also ensuring that this does not happen again in the future. Moving forward, I promise to keep the community aware of any such changes in as real time as possible.

 

Once again, my sincere apologies.

Nikhil Singhal

Director of Web Development

[+pklong]

1000 caches in a PQ makes pklong very happy. Thanks Groundspeak!

[+JeremyR]

Thanks a million thousand

[+Ducky]

Thanks for the update

[+rryder]

Ah...so THAT's why my caches had no logs!!

[+Frank Broughton]

Thank you for the explanation. I was wondering why I received 45 copies of one PQ and many many copies of others. Living in the boonies and being on a limited bandwidth Verizon Wireless modem for my ISP I was wondering what crazy friend of mine was sending me a huge email attachment, then the PQ's kept pouring in... haha

 

Jeremy apology accepted. It showed much class on your part to send it. I know so many of us rag on the ways things are done from time to time but this post of yours has earned you much respect from me.

 

Again - THANK YOU!

[+MikieP]

First of all, thanks for the sincere apologies, the acknowledgement that communications are critical, and the information about raising the 500 cache limit per PQ to 1000.

Now my question: of the five queries I have scheduled for today, I received the two odd numbered queries around 8:30 this morning, but am stll waiting on the three even numbered queries. Is that particular server still processing backlog, or has something else popped up?

[+Viajero Perdido]

Well handled.

 

My compliments, also, for the great communications of late, and I happily look forward to the bigger, happier PQs.

 

Cheers,

VP

[+leejas72]

Is it fixed yet? I don't want to try again. I thought it was GSAK so I uninstalled and then reinstalled. Thought it was my GPS reset to factory settings. Glad to know that it's something that will be fixed and I don't have to try to figure it out. LOL not so computer savvy.

[+PDOP's]

Thanks for the update and for all the hard work.

[+Markwell]

First of all, thanks for the sincere apologies, the acknowledgement that communications are critical, and the information about raising the 500 cache limit per PQ to 1000.

Now my question: of the five queries I have scheduled for today, I received the two odd numbered queries around 8:30 this morning, but am stll waiting on the three even numbered queries. Is that particular server still processing backlog, or has something else popped up?

At this time the service is again functioning properly but there is a substantial queue of unsent queries - we estimate it will catch up around 11am Pacific.

[+leejas72]

Maybe there is something wrong with my GPS. I'm still not getting anything and now a black box is opening when I try to send to GPS. C:/Windows\system32

[jholly]

Maybe there is something wrong with my GPS. I'm still not getting anything and now a black box is opening when I try to send to GPS. C:/Windows\system32

 

Is it really C:/?

[+leejas72]

Maybe there is something wrong with my GPS. I'm still not getting anything and now a black box is opening when I try to send to GPS. C:/Windows\system32

 

Is it really C:/?

 

Yes it is. Now I'm really freaking out thinking that I've screwed up my computer or GPS or something.

[+sledgehampster]

To Jeremy, the crew at Groundspeak and Approvers. As a Charter Member (although I lapsed so now I am "only" a Premium Member) I would like to say THANKS!

 

I have watched this go from an underground hobby to the large community that it is today and that is wonderful. I have met many new friends and seen places I would not have seen otherwise as a result of geocaching.

 

Being able to load 500 caches (soon to be 1000!) at a time into my DeLorme PN-40 was only once a dream when cache pages were printed one at a time.

 

Sure there have been growing pains but that is to be expected. Nothing great started out perfect. Time and trials make products better.

 

Again, just a tip of my hat and a THANK YOU for Jeremy,the Team of Lackeys and Approvers. You have unknowingly made my life (as well as tons of others) more interesting and enjoyable thru your work and selfless determination...

Edited March 8, 2010 by sledgehampster

[+beejay&esskay]

By increasing this capacity we'll also be prepared to offer Pocket Queries with 1,000 results instead of 500.

 

Since it hasn't been explicitly stated, let me ask...

 

5 PQs/day with a maximum possible of 1000 each?

All types of PQs will have the new 1000 maximum (and bookmark lists are enhanced to handle 1000 entries)?

[jholly]

Maybe there is something wrong with my GPS. I'm still not getting anything and now a black box is opening when I try to send to GPS. C:/Windows\system32

 

Is it really C:/?

 

Yes it is. Now I'm really freaking out thinking that I've screwed up my computer or GPS or something.

 

It should be C:\, just like rest of the path. And why are you sending things to C:\Windows\system32? That is a place for system files, not gpx or other user files. But this is very off topic for this discussion, open an new thread.

Edited March 8, 2010 by jholly

[OpinioNate]

By increasing this capacity we'll also be prepared to offer Pocket Queries with 1,000 results instead of 500.

 

Since it hasn't been explicitly stated, let me ask...

 

5 PQs/day with a maximum possible of 1000 each?

All types of PQs will have the new 1000 maximum (and bookmark lists are enhanced to handle 1000 entries)?

 

5 per day @ 1,000 each. I can't promise the bookmark lists will increase in size but we can talk about it (in another thread).

[+Pax42]

Can some one please acknowledge if this is also is related to the problem of many people being sent GPX 1.0.1 PQs instead of GPX 1.0 PQs as their preference is set to. It's been causing a lot of grief to Delorme Cache Register users since the most recent site update. Going into the profile and changing the preference back and forth a few times seems to be working as a temporary fix.

[+g-people]

Can some one please acknowledge if this is also is related to the problem of many people being sent GPX 1.0.1 PQs instead of GPX 1.0 PQs as their preference is set to. It's been causing a lot of grief to Delorme Cache Register users since the most recent site update. Going into the profile and changing the preference back and forth a few times seems to be working as a temporary fix.

 

THANK YOU! My Delorme is a mess now....Not only can I not get PQs to load as geocaches vs. waypoints, I cannot even get caches sent individually via the plug in now. Will this be fixed at any point?

[+leejas72]

Can some one please acknowledge if this is also is related to the problem of many people being sent GPX 1.0.1 PQs instead of GPX 1.0 PQs as their preference is set to. It's been causing a lot of grief to Delorme Cache Register users since the most recent site update. Going into the profile and changing the preference back and forth a few times seems to be working as a temporary fix.

 

THANK YOU! My Delorme is a mess now....Not only can I not get PQs to load as geocaches vs. waypoints, I cannot even get caches sent individually via the plug in now. Will this be fixed at any point?

 

I have a Delorme also and it's all screwed up too. I hope this is fixed and things get back to normal. I keep getting an error message in GSAK but I don't know if it's the PQ's or GSAK

[+Bear and Ragged]

I keep getting an error message in GSAK but I don't know if it's the PQ's or GSAK

I'd take it as being the PQ at fault. If you still get problems later in the week with new PQ's it might be GSAK...

[+mambero]

As the developer working on PQs over the weekend, I want to apologize to you for the frustrating experience you've had this weekend with the PQs. While Nikhil has offered to own up to it, it was my mistake that introduced meyham you witnessed. You may hear from others on my team, but they're just being big hearted. I know this ruined many geocaching plans and while I know I can't give you back a lost weekend, I can keep you informed of the current status and our plans for this week with PQs.

 

Current status: At this point it looks like we have several hours left. This morning we were on track for 11am, and then more caches were added and the response times from the database slowed down due to site traffic. It was this slow performance that I was attempting to fix this weekend. Believe it or not, there are now some improvements as we're getting significantly less database timeouts than we were before.

 

Plans for the week: There are 4 main processes that need to happen to deliver your PQs. 1) Find the pending PQs for the day. 2) Determine which caches will fulfill each PQ. 3) Fetch those caches/logs/etc 4) Email them to you and post them to a website.

 

* It turns out #3 is where we spend most of our time, so we're going to fixing that this week. For you tech types, we'll be using a distributed cache.

* Another plan is to start the PQs to processing at 8pm PST. If we have time this week, we plan to prioritize processing by Latitude so those of you across the Pond and further will have your PQs during the day, and to allow our side of the Pond to still log their caches for the day.

We want to make sure the generator is stable this weekend, so processing by Latitude may get pushed, but it is the plan. Any thoughts on this?

* The database is accumulating a large amount of data. We're looking into how we can better tune it for better response times.

 

I will post again today when the main PQ load has been processed. Currently we have 12,000 remaining.

 

I know this got a bit techie, but I want to make sure you have a full understanding of what we're doing to make up for this weekend and ensure it doesn't happen again.

 

Drew Burlingame

[+Biggi_H]

By increasing this capacity we'll also be prepared to offer Pocket Queries with 1,000 results instead of 500. This will happen by the anniversary of geocaching (May 2, 2010).

 

great !! Thank you to Groundspeak

[+mambero]

Can some one please acknowledge if this is also is related to the problem of many people being sent GPX 1.0.1 PQs instead of GPX 1.0 PQs as their preference is set to. It's been causing a lot of grief to Delorme Cache Register users since the most recent site update. Going into the profile and changing the preference back and forth a few times seems to be working as a temporary fix.

 

I'm looking at the code now and we're pulling the version directly from your profile setting, and looking at your profile, you're set for GPX 1.0. Let me know if you're still having problems.

[jholly]

As the developer working on PQs over the weekend, .... <snip>

 

Drew Burlingame

 

Thank you Drew, a most excellent post. You gave me some insight on the process I was unaware of and it helps to better understand the problems you are facing. Thank you for your efforts, I look forward to enjoying the fruits of your labor.

[GOF and Bacall]

Thanks for the update guys. I knew it was a tech type problem as I had no clue what was going on.

 

The larger PQs will be appreciated by many, I am sure.

 

Keep up the great work.

[+Pax42]

Can some one please acknowledge if this is also is related to the problem of many people being sent GPX 1.0.1 PQs instead of GPX 1.0 PQs as their preference is set to. It's been causing a lot of grief to Delorme Cache Register users since the most recent site update. Going into the profile and changing the preference back and forth a few times seems to be working as a temporary fix.

 

I'm looking at the code now and we're pulling the version directly from your profile setting, and looking at your profile, you're set for GPX 1.0. Let me know if you're still having problems.

Thank you for looking into it. I however, have not been having the problem since I did the fix of changing my preference setting back and forth a number of times. Many other people though keep posting with this problem. The resolution was discovered by some folks on the Delorme forums but not everyone reads the forums so they don't know why Cache Register has stopped working for them. Even some Garmin users have noted they were receiving the 1.0.1 attributes even though their preference was set to 1.0. This issue was introduced in the last update and is probably affecting hundreds of people.

 

Edit to add: These are the steps we have been providing people to fix the problem until Groundspeak could look into it.

 

1- go to GC.com and click on YOUR PROFILE

2- then click on Your Account Details

3- scroll to the bottom of that page to Your Preferences

4- if it is set to 1.0.1, skip to step 10

5- click on change

6- find GPX Version Preference and change it to 1.0.1

7- click on Save Changes

8- click on change again

9- if the GPX Version Preference setting now says 1.0, go back to step 6 (yes really)

10- find GPX Version Preference and change it to 1.0

11- click on Save Changes

12- re-run the PQs that have the version problem, the new PQs should be OK

Edited March 8, 2010 by Pax42

[+Aerodoq]

* Another plan is to start the PQs to processing at 8pm PST. If we have time this week, we plan to prioritize processing by Latitude so those of you across the Pond and further will have your PQs during the day, and to allow our side of the Pond to still log their caches for the day.

We want to make sure the generator is stable this weekend, so processing by Latitude may get pushed, but it is the plan. Any thoughts on this?

 

Sounds great ... except I hope you mean you're processing by longitude not latitude

 

A>

[+mr007s]

Many thanks guys, I knew you would get it fixed.

[+TotemLake]

Jeremy et al,

 

I've always said you guys are an amazing bunch particularly in the face of adversity. Looking forward to seeing y'all during GW8.

[Spatial Distortion]

Over the weekend there were a lot of issues with Pocket Queries being generated. For that I sincerely apologize.

 

In an attempt to speed up the Pocket Query Generator, the application(s) that processes tens of thousands of Pocket Queries a day, there was a cascading number of issues that resulted in duplicate Pocket Queries going out, Pocket Queries without logs and Pocket Queries not being sent out for a certain amount of time. Also in the process, Pocket Queries were brought down to a trickle, resulting in Pocket Queries not going out at all.

 

We're taking this issue very seriously at Groundspeak, and we'll be focusing on increasing the capabilities over this next weekend to ensure this never, ever happens again. By increasing this capacity we'll also be prepared to offer Pocket Queries with 1,000 results instead of 500. This will happen by the anniversary of geocaching (May 2, 2010). As those veterans to geocaching know, I'm not usually one to offer dates, but this one is solid.

 

Again, I offer my apology for this weekend's issues.

 

1000, Woo Hoo

Thanks Jeremy

[+weathernowcast]

Over the weekend there were a lot of issues with Pocket Queries being generated. For that I sincerely apologize.

 

In an attempt to speed up the Pocket Query Generator, the application(s) that processes tens of thousands of Pocket Queries a day, there was a cascading number of issues that resulted in duplicate Pocket Queries going out, Pocket Queries without logs and Pocket Queries not being sent out for a certain amount of time. Also in the process, Pocket Queries were brought down to a trickle, resulting in Pocket Queries not going out at all.

 

We're taking this issue very seriously at Groundspeak, and we'll be focusing on increasing the capabilities over this next weekend to ensure this never, ever happens again. By increasing this capacity we'll also be prepared to offer Pocket Queries with 1,000 results instead of 500. This will happen by the anniversary of geocaching (May 2, 2010). As those veterans to geocaching know, I'm not usually one to offer dates, but this one is solid.

 

Again, I offer my apology for this weekend's issues.

Thanks a 1000!

[+beejay&esskay]

* Another plan is to start the PQs to processing at 8pm PST. If we have time this week, we plan to prioritize processing by Latitude so those of you across the Pond and further will have your PQs during the day, and to allow our side of the Pond to still log their caches for the day.

We want to make sure the generator is stable this weekend, so processing by Latitude may get pushed, but it is the plan. Any thoughts on this?

 

Drew Burlingame

(As I has been noted, it's a little worrisome that the Groundspeak guys aren't clear on latitude vs longitude. )

 

How would starting at 8pm affect the 5/day rule? I noticed on Saturday some of my Sunday scheduled PQs running early. (Maybe an early test...or unrelated.)

 

Because of the other problems Sunday it wasn't clear if these early running PQs were counted as Saturday or Sunday runs. If Sunday, there needs to be a clear indication of that.

[+mambero]

* Another plan is to start the PQs to processing at 8pm PST. If we have time this week, we plan to prioritize processing by Latitude so those of you across the Pond and further will have your PQs during the day, and to allow our side of the Pond to still log their caches for the day.

We want to make sure the generator is stable this weekend, so processing by Latitude may get pushed, but it is the plan. Any thoughts on this?

 

Sounds great ... except I hope you mean you're processing by longitude not latitude

 

A>

 

The idea is for those who wake up first to get the first PQs, so I believe by Latitude is correct, and we may also parse into longitude chunks. We plan to get the run times down to about 6 hours, which will be an interesting challenge when we move to 1000 caches per PQ.

[+mambero]

* Another plan is to start the PQs to processing at 8pm PST. If we have time this week, we plan to prioritize processing by Latitude so those of you across the Pond and further will have your PQs during the day, and to allow our side of the Pond to still log their caches for the day.

We want to make sure the generator is stable this weekend, so processing by Latitude may get pushed, but it is the plan. Any thoughts on this?

 

Drew Burlingame

(As I has been noted, it's a little worrisome that the Groundspeak guys aren't clear on latitude vs longitude. )

 

How would starting at 8pm affect the 5/day rule? I noticed on Saturday some of my Sunday scheduled PQs running early. (Maybe an early test...or unrelated.)

 

Because of the other problems Sunday it wasn't clear if these early running PQs were counted as Saturday or Sunday runs. If Sunday, there needs to be a clear indication of that.

 

What would you like the indication to look like? "This is one of your Sunday PQs" in the e-mail?

[+Chrysalides]

The idea is for those who wake up first to get the first PQs, so I believe by Latitude is correct, and we may also parse into longitude chunks.

Latitude is degrees N / S of equator. Doesn't really affect sunrise / sunset (well, except for the axial tilt). Time zone is usually decided by longitude (E / W of prime meridian).

What would you like the indication to look like? "This is one of your Sunday PQs" in the e-mail?

How about "Pocket Query : [PQ name] scheduled for [date]" ?

 

Appreciate the work and the explanation, thank you.

[+Zemmy]

Will the limit of PQs total change from 40?

[+mambero]

It looks like we're about 30 minutes away from finishing the PQs for the day. As for progress from the week, we've been able to improve database responsiveness and we've added the code to start processing a day in advance. It still needs to be tested, but we plan to have it go live tomorrow or the day after.

[jholly]

* Another plan is to start the PQs to processing at 8pm PST. If we have time this week, we plan to prioritize processing by Latitude so those of you across the Pond and further will have your PQs during the day, and to allow our side of the Pond to still log their caches for the day.

We want to make sure the generator is stable this weekend, so processing by Latitude may get pushed, but it is the plan. Any thoughts on this?

 

Sounds great ... except I hope you mean you're processing by longitude not latitude

 

A>

 

The idea is for those who wake up first to get the first PQs, so I believe by Latitude is correct, and we may also parse into longitude chunks. We plan to get the run times down to about 6 hours, which will be an interesting challenge when we move to 1000 caches per PQ.

 

Oh, the sun now rises in the south? No, LONGitude. They are the ones running north to south, or south to north depending on your point of view. Latitudes run around the sphere east to west. The equator is latitude 0 degrees.