Jump to content

Instant Notify not working


mkkm

Recommended Posts

Ones I've noticed that I (mellers) did NOT receive a notification of are:

 

Portsmouth's prose (http://coord.info/GC6PR8X) published 7th August 2016

Something Fishy (http://coord.info/GC6P9WD) published 2nd August 2016

I can't give details of who did, but this may be discernible from the logs of any FTFers on the cache pages.

 

I (mellers) DID get notification of:

Pourri arbre (http://coord.info/GC6N4MK) published 9th July and understand that Broyleboxers did NOT.

Link to comment

I maintain a pocket query that shows me all geocaches (of all types) that have been published in the past month. I almost never actually run the query, I use it to get a list of the new caches in my local area (within a 60-mile radius of my home location). I also have Instant Notifications set up to notify me of all new caches published within a 50-mile radius (the limit for Instant Notifications).

 

I previewed that pocket query just now, and I never received notifications for the following newly-published caches:

 

GC6PX2F Rock On!

Published 08/08/2016

 

GC6PW41 Trail bridge

Published 08/07/2016

 

GC68RNP Alley Oops!

Published 08/06/2016

 

GC6JYQC It keeps the Bugs Away!

Published 08/03/2016

 

GC6PM1P Who's your author?

Published 08/03/2016

 

GC6PV3B A Walk in the (dog) Park

Published 08/07/2016

 

GC6P44Y Bellefontaine Bike Trail Cache

Published 07/25/2016

 

All of the above caches are located less than 50 miles from my home location, and I should have received notifications, and didn't.

 

I count 7 notifications, just for new caches, I didn't receive over the past month. Heaven knows what other (non-publication) notifications I should have received but didn't.

 

The total number of new caches published in this area in the past month within 50 miles of my home location (according to the pocket query) was 81. I was never notified of 7 of them.

 

I rarely, if ever, go for an FTF, but this is still pretty irritating.

 

--Larry

Link to comment

Mellers - I got the FTF yesterday on GC6PR8X Something Fishy but hadn't received a notification for it when it was published despite being set up to do so. It was just sheer luck that I did a search yesterday on unfound caches within 10 miles of me and sorted them by published date so spotted it. The CO is very worried he set the D/T rating too high or that the puzzle is too difficult because it took nearly a week for the FTF to go in an area where the FTF hounds are ferocious.

Edited by prawm
Link to comment

Thanks for all the reports. We don't need any more examples. We will dig into it and hopefully find a fix soon.

 

Cindy / Frau Potter

 

Please keep us posted on the progress and let us know when there is a solution.

 

Thx

 

Hallo Frau Potter,

It looks like there is no viable solution since last week.

That's a pity but still, could you at least provide us with some feedback ? That would much appreciated.

Thx

Edited by Gillala
Link to comment

Sorry for the delay in providing an update. It is a very busy time. The engineers are looking into it and we hope for a solution soon. We don't need any more examples. You have proven that there is definitely a problem. This is a PM feature, so we will work to make sure it is fixed. Thanks for your patience.

 

Cindy / Frau Potter

Link to comment

report:

GC6PE7N

has not shown up in my text message alert nor my email alert of new caches

 

It published this morning. I found out about it from the CO on a FB page that no one had looked at the new cache (premium cache)

 

I have all those boxes checked (it also took about 24 hours for the second puzzle email to arrive in my email, the first puzzle email took much longer)

 

thanks,

Lynn

Link to comment

Sorry for the delay in providing an update. It is a very busy time. The engineers are looking into it and we hope for a solution soon. We don't need any more examples. You have proven that there is definitely a problem. This is a PM feature, so we will work to make sure it is fixed. Thanks for your patience.

 

Cindy / Frau Potter

 

I'm sorry for posting,my reviewer referred here and I had not read more comments since the last time I had looked at this.

:unsure:

Link to comment

Thanks for the reports of problems with Instant Notifications. We made a change to the architecture of our notification system on July 26. It appears that those changes resulted in some UNintended changes. To help us troubleshoot, it would be useful if we had examples with usernames and GC codes. For example, a GC code for a geocache that was published recently and resulted in emails being sent to _______ [specific username] but not to ___________[specific username]. We are trying to determine if notifications are successful in certain circumstances, but not in others.

 

Thanks for your help in identifying the problem.

 

Cindy / Frau Potter

You now have numerous responses showing that this is still happening and setting "tips and tricks" does not solve the problem. You need to give a high priority to getting to the root of this as for many notifications are one of the main feature for which cachers become premium members. So if this incentive is "broken" it could have financial consequences resulting from fewer cachers becoming premium

Link to comment

I used to receive all notifications for all cache types for new hides. It's been hit or miss these past three weeks with some coming through and others not coming through at all, particularly events. The only reason I know is that I run a daily PQ of unfound caches within a certain distance from home. That way I can see all the events and new caches in the area.

Link to comment

I'm going to add this as it seems to be a new development on the perceived problem of this only affecting some cachers on some caches... and not all cachers on some caches.

 

Following a discussion on a local forum, we have concluded that for at least two caches published this morning, no cachers at all were notified.

 

For the following two caches published dated 13/08/16 (Seattle time):

Gaming Evolution (http://coord.info/GC6N86R) and

Hayling Billy (http://coord.info/GC6Q64Y)

not one cacher who contributed to the discussion could report having had their expected notification for either despite correctly having a notification for two multis and two events published within range at the same approximate time. The following cachers ALL reported NOT having received it:

badcats

Titannia1

mellers

molemon

GeoJaxx

Boomshanka

common_smiths

LyndseyW85

Martybartfast

Daffodil27

blueteddy

Fay, R+R

thekennelat79

shirty demon

Tonydev

FfiLli

Broyleboxers

Link to comment

Given that this is a crucial premium member feature, will we be compensated for the loss of service?

 

I know several premium members who are very concerned that their monthly FTF streak may be broken due to this feature not being available. I expect they would not even be happy with a months free membership if their streak is broken.

 

Surely if it is known which release introduced the fault the change could be backed out and the change rectified in a test environment.

 

More information on what if any progress is being made, how long it will be before we can expect a solution and what if any compensation we can expect.

 

Thanks

Neil

Edited by Spire67
Link to comment

Surely if it is known which release introduced the fault the change could be backed out and the change rectified in a test environment.

I still can't figure out why this hasn't been done. It seems to be well-known which change caused the problem ("We made a change to the architecture of our notification system on July 26."), so it should just be a matter of rolling back that change (or release/build/whatever). I can understand how this bug could have been missed in UAT due to its intermittent nature, but the change should have been rolled back as soon as it became apparent that it was causing significant issues in the production environment. Leaving the broken code in place for nearly three weeks is just nonsensical. Missing out on some FTFs isn't a big deal in the grand scheme of things, but there are event organizers that have been let down by one of the best tools for raising awareness of their event and their attendance will likely suffer.

Link to comment

As this is an ongoing issue I wanted to provide an update on our efforts and some background. As you by now likely know we recently switched over to a new system for handling our notifications. The change to the new notifications architecture involves multiple systems operating as one. Each of these systems were updated and deployed together in order to migrate away from the old notifications architecture. "Rolling back" at this point will involve guaranteed downtime for the entire system and all users will be affected.

 

Due to its sporadic nature (my most hated type of bug) it is extremely hard to pinpoint the source of the issue. This new system was tested for months before being fully rolled out and the truth is that at no point did we identify an issue with missing notifications. An increasing amount of logging is being added to shine a light into every corner of the system. We are also working to correlate all of the logs into a single unified picture that will reveal where the hole exists.

 

I understand that this has been extremely frustrating and it might seem like little is being done, but as one of the developers who is actively working on this project I can guarantee that we are looking at everything and working to identify and fix this issue. Additional developers are being pulled in to review associated systems in order to speed up our analysis.

 

No one is happy with the fact that notifications are going missing and I know there will be questions about how and why we changed something that seemed to be working. Please believe me that this change was necessary and that we are doing everything we can to return to business as usual under the new system.

Link to comment

We wondered why the new multi-cache we had put out prior to the Mission GC "find a multi" last weekend got only a few finds, almost no one in the area got a notification it was published. On the plus side, I found a cache this afternoon that was published yesterday and got a FTF today since there was no notification it was published either. Usually it would have been claimed within an hour. I've received 2 notifications out of 5 published caches that I can see looking at the map in the past week.

Link to comment

I find it fascinating that a feature that is ignored by many premium members can be considered "a crucial feature" by others.

I have always been under the impression that notification is one of the most crucial features for many cachers.

 

Personally, the PQ and the API are the most crucial premium features for me, as I don't trust Groundspeak's servers and prefer to keep my GSAK database up-to-date. It's been a while since I'd stopped playing the FTF side game, but I've already failed to add some local events to the GBA Event Calendar which some local cachers (including myself) now rely on, resulting multiple cachers failed to attend events they were to attend otherwise (example). Yes, one can say that I should still be able to keep monitoring new events some other ways, and I'm trying, but that's going to be a lot of work for me.

Link to comment

In the past week or so, I have been getting double email notifications on most caches and then NOT getting any notifications on others. We like to stay active caching while we are still near home base and look forward to these notifications to keep us doing so, mostly because we have found 96% of all the caches locally.

Have read all the comments in this forum and wanted to input my experience before emailing HQ as I know I would rather have someone working on the issue at hand rather than spending time reading and answering emails!

 

Hope this is rectified soon as we believe this is a major technological function for geocachers using geocaching.com. Pretty much goes with the whole idea of what geocaching is...being notified that they are out there...right?

 

Keep us updated HQ...PLEASE!!!!

Link to comment

I was at an event this evening and I noticed one of my trackables on the table. It had been in a supposedly un-found mystery cache of mine. I now realize that I am not getting notifications for found-it logs on my caches. I have no idea how many have been missed and I am not going to dig through every cache to find out. The one that I do know about is for GC6Q1K8 - The Girls from YC -found by user Alamogul today (8/17/2016.) Things are getting worse, not better.

Link to comment

Thanks for the update... BUT...

"Rolling back" at this point will involve guaranteed downtime for the entire system and all users will be affected.

It's pretty clear that many users have confirmed they are already affected and that those which haven't may not realise they have been affected. I think it may be pretty likely that all users have already been affected to a greater or lesser extent and either don't know or haven't realised... so I'm not sure that citing an effect for all users is logical reasoning against rolling back. Mind you, I'm not a software developer so I'm on dangerous ground in discussing something I don't know much about ;).

 

It seems that assuming not all users are already affected is like asking someone who has left their landline phone accidentally unplugged "Did you miss any calls? Were you affected?" "I don't know," would be the answer, "I didn't even realise."

Edited by mellers
Link to comment

It's pretty clear that many users have confirmed they are already affected and that those which haven't may not realise they have been affected. I think it may be pretty likely that all users have already been affected to a greater or lesser extent and either don't know or haven't realised... so I'm not sure that citing an effect for all users is logical reasoning against rolling back.

 

You make a fair point and I agree that was a poor choice of words to explain the situation. At this time we are planning on continuing to move forward with the new notification system and are making progress on identifying the issue and resolving it.

 

In an effort to remain transparent, in the last 24 hours we have deployed 10 updates across 4 separate systems which is making it possible to see everything related to notifications that is happening, at once (opposed to split across individual reports). I appreciate everyone's patience as we continue to analyze the situation and work towards a resolution.

Link to comment

Thanks for the reports of problems with Instant Notifications. We made a change to the architecture of our notification system on July 26. It appears that those changes resulted in some intended changes. To help us troubleshoot, it would be useful if we had examples with usernames and GC codes. For example, a GC code for a geocache that was published recently and resulted in emails being sent to _______ [specific username] but not to ___________[specific username]. We are trying to determine if notifications are successful in certain circumstances, but not in others.

 

Thanks for your help in identifying the problem.

 

Cindy / Frau Potter

 

For GC6QCWC neither uwezi nor pmsl got a notification, but user SmurfKastrull did.

Link to comment

DATA POINT: As of Sun 8/21/2016 @ 7:17 AM, the most recent cache that I've noticed that did NOT have a notification email was GC6Q52W "What Three Words #3" (Unknown Cache) in California, United States created by lmt000. I noticed it live on Thu 8/18/2016 @ 4:09 AM Pacific Time. I have received emails on all since 8/18/2016 @ 4:09 AM.

Link to comment

I realize it felt like a long time with little progress, but we are at the point where we feel confident that the issue has been resolved and all notifications are being sent. For those that are interested I will provide a summary of the issue and the resolution.

 

When a geocache was published (from an existing system) a message was placed in a queue for the new notification system and then the geocache status was updated (also in the existing system). The new notification system would attempt to process a notification, but the geocache was still listed as "unpublished" so no notification would be sent. If the system was under heavy load and the queue was backed up it would process the notifications after a sufficient delay that the geocache had been updated to "published" and so the notification was sent.

 

This worked in the old system because it ran slower and it essentially had a built in delay that was long enough to allow the geocache to be published before processing the notifications. The issue was not identified early on in the new system because we were unintentionally allowing notifications to be sent for unpublished geocaches for the first 24 hours after the new system went live.

 

The fix was to update the existing system that was handling the publishing of geocaches to make sure that the geocache status was updated to "published" *before* adding the message to the queue. With this change the geocache was then always in a consistent state by the time it reached our new notification service. Troubleshooting and load testing of this issue was initially focused on finding an issue in the notification service and did not take into consideration the inconsistent geocache status because that part of the system had not been changed. It was not until we significantly increased the amount of data that was being collected around every notification that we were able to understand that (in some cases) we were handling inconsistent data.

Link to comment

I realize it felt like a long time with little progress, but we are at the point where we feel confident that the issue has been resolved and all notifications are being sent. For those that are interested I will provide a summary of the issue and the resolution.

 

When a geocache was published (from an existing system) a message was placed in a queue for the new notification system and then the geocache status was updated (also in the existing system). The new notification system would attempt to process a notification, but the geocache was still listed as "unpublished" so no notification would be sent. If the system was under heavy load and the queue was backed up it would process the notifications after a sufficient delay that the geocache had been updated to "published" and so the notification was sent.

 

This worked in the old system because it ran slower and it essentially had a built in delay that was long enough to allow the geocache to be published before processing the notifications. The issue was not identified early on in the new system because we were unintentionally allowing notifications to be sent for unpublished geocaches for the first 24 hours after the new system went live.

 

The fix was to update the existing system that was handling the publishing of geocaches to make sure that the geocache status was updated to "published" *before* adding the message to the queue. With this change the geocache was then always in a consistent state by the time it reached our new notification service. Troubleshooting and load testing of this issue was initially focused on finding an issue in the notification service and did not take into consideration the inconsistent geocache status because that part of the system had not been changed. It was not until we significantly increased the amount of data that was being collected around every notification that we were able to understand that (in some cases) we were handling inconsistent data.

 

This is excellent information -- thanks!

 

I do the IT so any of the backend stuff is always interesting to me.

 

You have explained it clearly and non-technically -- not always an easy thing to do.

 

Keep up the good work, and thanks for keeping us in the loop.

 

Meanwhile, on the downside, I was enjoying playing the FTF game as I did before I was PMO, back in the day.

 

A combination of refreshing the map and the newest-in-state list led to a couple of my first firsts.

 

It would be fair to say that they were more satisfying than subsequent finds aided by instant notification.

 

During this latest outage everyone was playing that way and I hope that the situation led to more finds by cachers who didn't even know they'd be first.

Link to comment

I realize it felt like a long time with little progress, but we are at the point where we feel confident that the issue has been resolved and all notifications are being sent. For those that are interested I will provide a summary of the issue and the resolution.

 

When a geocache was published (from an existing system) a message was placed in a queue for the new notification system and then the geocache status was updated (also in the existing system). The new notification system would attempt to process a notification, but the geocache was still listed as "unpublished" so no notification would be sent. If the system was under heavy load and the queue was backed up it would process the notifications after a sufficient delay that the geocache had been updated to "published" and so the notification was sent.

 

This worked in the old system because it ran slower and it essentially had a built in delay that was long enough to allow the geocache to be published before processing the notifications. The issue was not identified early on in the new system because we were unintentionally allowing notifications to be sent for unpublished geocaches for the first 24 hours after the new system went live.

 

The fix was to update the existing system that was handling the publishing of geocaches to make sure that the geocache status was updated to "published" *before* adding the message to the queue. With this change the geocache was then always in a consistent state by the time it reached our new notification service. Troubleshooting and load testing of this issue was initially focused on finding an issue in the notification service and did not take into consideration the inconsistent geocache status because that part of the system had not been changed. It was not until we significantly increased the amount of data that was being collected around every notification that we were able to understand that (in some cases) we were handling inconsistent data.

Do you have any insight as to why so many premium members didn't receive the letters about Mission GC? (I did receive all of them personally.)

Link to comment

Thank you for the well-written explanation. As a computer scientist and an affected cache owner, I saw the "unpublished" aspect of the problem. I think I sent you a note about that.

 

Your explanation of the problem and solution shows us why you consider the bug squashed.

 

A local series published yesterday with 10 caches, and it looks like they all had notifications. Yes. Thank you.

Link to comment

I realize it felt like a long time with little progress, but we are at the point where we feel confident that the issue has been resolved and all notifications are being sent. For those that are interested I will provide a summary of the issue and the resolution.

...

As someone who has been a bit harsh in criticizing the actions of Groundspeak in recent times, I'd like to thank not only the devs for tracking down this tricky issue and resolving it, but also HiddenGnome for providing some excellent communications and insight. It would be great if a similar level of communication could be maintained for other significant issues. It helps give the membership confidence that the Lackeys know what they're doing and are working hard to deal with any issues.

Link to comment

Can you please provide more details about the notifications that you believe have gone missing? What kind of notification were you expecting and what are the relevant geocaches? Did another geocacher receive a notification and you did not? Any additional information that might help to pinpoint the issue would be appreciated.

 

Also, please make sure to check your spam folders to make sure that changes with your ISP or mail provider did not cause your notifications to end up in the wrong location.

Link to comment

Can you please provide more details about the notifications that you believe have gone missing? What kind of notification were you expecting and what are the relevant geocaches? Did another geocacher receive a notification and you did not? Any additional information that might help to pinpoint the issue would be appreciated.

 

I did not receive notifications for GC6QG1R, GC6QN6K, and GC6QN6M.

 

I did receive notifications for GC6R75J and GC6R74J.

 

The main difference between the two sets is that the notifications I did not receive were caches closer to the center coordinates for the notification than the others.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
×
  • Create New...