Slow website geocaching.com

[+De Labratjes]

The last few weeks the website of geocaching.com is very slow or not loading at all in the evening time (e.g. 20:00h Amsterdam time +1 GMT)

 

Can anybody tell what is the problem???

 

Are to little servers available? Or are just the cachers in the US of A waking-up?

 

It's very anoying not being able to navigate the website properly and log or search caches.

 

Anybody any idea's?

[+Manville Possum]

The last few weeks the website of geocaching.com is very slow or not loading at all in the evening time (e.g. 20:00h Amsterdam time +1 GMT)

 

Can anybody tell what is the problem???

 

Are to little servers available? Or are just the cachers in the US of A waking-up?

 

It's very anoying not being able to navigate the website properly and log or search caches.

 

Anybody any idea's?

 

I'm guessing it is server issues. The Waymarking site has been very problematic for the last few weeks too.

[+on4bam]

That's been going on for along time. Biggest problems are during weekends when EU starts to log in the evening and the US is waking up.

 

And BTW, in summer Amsterdam is UTC +2.

[Justin]

The last few weeks the website of geocaching.com is very slow or not loading at all in the evening time (e.g. 20:00h Amsterdam time +1 GMT)

 

Can anybody tell what is the problem???

 

Are to little servers available? Or are just the cachers in the US of A waking-up?

 

It's very anoying not being able to navigate the website properly and log or search caches.

 

Anybody any idea's?

 

The issue is not related to server or infrastructure load based on the metrics we capture. It is a tier-1 ISP routing issue. This can be observed when users with the issue modify their route using VPN services or utilities like ZenMate, and the site becomes snappy and responsive.

 

When the problem first manifested, it was primarily among Deustche Telekom users. We had asked users to submit their ping and traceroute data in hopes of isolating the network or hops responsible, and those client to Groundspeak traces seemed to indicate that NTT's network could be the issue. It wasn't until we were able to setup a test client on a Deutsche Telekom DSL network and run bi-directional traces on our own that we started seeing a pattern that eventually caught the attention of our provider.

 

Our provider, Internap, uses a blended BGP solution for our internet access which consists of 7 tier-1 peers (ATT, GTT, NTT, XO, Zayo, Cogent and Qwest). A technology they employ called MIRO provides dynamic optimization of all outbound connectivity and constantly evaluates the best peer for a route. To test each peer individually, we asked Internap's network engineer to effectively disable MIRO for the IP range of the test client setup on DT, and then specify one peer network at a time. After establishing a baseline of normal performance on all 7 peers, we continued troubleshooting during a recent 18:00-20:00 CET poor performance window. These tests ultimately yielded evidence that connections routed across ATT or GTT were more prone to performance problems. So for Deutsche Telekom, that led to a workaround on our side to never use ATT and GTT for those client networks, and that is hopefully showing improvement for the vast majority of DT users. Specific IP details are available here.

 

Within the last few days, we've seen a pattern of new ISPs and locations reported and you are obviously included in that. The list we have compiled includes Cablecom (Switzerland), UPC (Ireland) and Ziggo (Netherlands). However, after doing more research, it appears that Cablecom and Ziggo have affiliation with UPC, so it appears this new round of reports have a common link. Since we don't have the luxury of a test box on one of these networks, it might take a little more time to reach a solution but this issue has been made a priority. Considering the similarities with the DT issue, ATT and GTT could very well be introducing the problems that you're having and excluding those peers for your network range might resolve the problem. We will have to identify the possible CIDR network ranges in use by your provider to apply the workaround, as well as verify the peer in use when the problem is observed.

[+FeyGriffin]

The issue is not related to server or infrastructure load based on the metrics we capture. It is a tier-1 ISP routing issue. This can be observed when users with the issue modify their route using VPN services or utilities like ZenMate, and the site becomes snappy and responsive.

 

When the problem first manifested, it was primarily among Deustche Telekom users. We had asked users to submit their ping and traceroute data in hopes of isolating the network or hops responsible, and those client to Groundspeak traces seemed to indicate that NTT's network could be the issue. It wasn't until we were able to setup a test client on a Deutsche Telekom DSL network and run bi-directional traces on our own that we started seeing a pattern that eventually caught the attention of our provider.

 

Our provider, Internap, uses a blended BGP solution for our internet access which consists of 7 tier-1 peers (ATT, GTT, NTT, XO, Zayo, Cogent and Qwest). A technology they employ called MIRO provides dynamic optimization of all outbound connectivity and constantly evaluates the best peer for a route. To test each peer individually, we asked Internap's network engineer to effectively disable MIRO for the IP range of the test client setup on DT, and then specify one peer network at a time. After establishing a baseline of normal performance on all 7 peers, we continued troubleshooting during a recent 18:00-20:00 CET poor performance window. These tests ultimately yielded evidence that connections routed across ATT or GTT were more prone to performance problems. So for Deutsche Telekom, that led to a workaround on our side to never use ATT and GTT for those client networks, and that is hopefully showing improvement for the vast majority of DT users. Specific IP details are available here.

 

Within the last few days, we've seen a pattern of new ISPs and locations reported and you are obviously included in that. The list we have compiled includes Cablecom (Switzerland), UPC (Ireland) and Ziggo (Netherlands). However, after doing more research, it appears that Cablecom and Ziggo have affiliation with UPC, so it appears this new round of reports have a common link. Since we don't have the luxury of a test box on one of these networks, it might take a little more time to reach a solution but this issue has been made a priority. Considering the similarities with the DT issue, ATT and GTT could very well be introducing the problems that you're having and excluding those peers for your network range might resolve the problem. We will have to identify the possible CIDR network ranges in use by your provider to apply the workaround, as well as verify the peer in use when the problem is observed.

 

The last few weeks the website of geocaching.com is very slow or not loading at all in the evening time (e.g. 20:00h Amsterdam time +1 GMT)

 

Can anybody tell what is the problem???

 

Are to little servers available? Or are just the cachers in the US of A waking-up?

 

It's very anoying not being able to navigate the website properly and log or search caches.

 

Anybody any idea's?

 

I see these routing issues from KabelBW in Germany too - basically the whole of the Geocaching web service, including API calls from GSAC, is unusable after around 8pm CET. It's meaning I have to make sure I have everything done in the morning that I need for evening caching trips, and is also forcing me to log in batches when I have a morning free :-(

[+Outspoken1]

I am going to share this info on the Waymarking website were we have not been able to use the site since June 10th.

 

Thanks, Outspoken1

[Justin]

The issue is not related to server or infrastructure load based on the metrics we capture. It is a tier-1 ISP routing issue. This can be observed when users with the issue modify their route using VPN services or utilities like ZenMate, and the site becomes snappy and responsive.

 

When the problem first manifested, it was primarily among Deustche Telekom users. We had asked users to submit their ping and traceroute data in hopes of isolating the network or hops responsible, and those client to Groundspeak traces seemed to indicate that NTT's network could be the issue. It wasn't until we were able to setup a test client on a Deutsche Telekom DSL network and run bi-directional traces on our own that we started seeing a pattern that eventually caught the attention of our provider.

 

Our provider, Internap, uses a blended BGP solution for our internet access which consists of 7 tier-1 peers (ATT, GTT, NTT, XO, Zayo, Cogent and Qwest). A technology they employ called MIRO provides dynamic optimization of all outbound connectivity and constantly evaluates the best peer for a route. To test each peer individually, we asked Internap's network engineer to effectively disable MIRO for the IP range of the test client setup on DT, and then specify one peer network at a time. After establishing a baseline of normal performance on all 7 peers, we continued troubleshooting during a recent 18:00-20:00 CET poor performance window. These tests ultimately yielded evidence that connections routed across ATT or GTT were more prone to performance problems. So for Deutsche Telekom, that led to a workaround on our side to never use ATT and GTT for those client networks, and that is hopefully showing improvement for the vast majority of DT users. Specific IP details are available here.

 

Within the last few days, we've seen a pattern of new ISPs and locations reported and you are obviously included in that. The list we have compiled includes Cablecom (Switzerland), UPC (Ireland) and Ziggo (Netherlands). However, after doing more research, it appears that Cablecom and Ziggo have affiliation with UPC, so it appears this new round of reports have a common link. Since we don't have the luxury of a test box on one of these networks, it might take a little more time to reach a solution but this issue has been made a priority. Considering the similarities with the DT issue, ATT and GTT could very well be introducing the problems that you're having and excluding those peers for your network range might resolve the problem. We will have to identify the possible CIDR network ranges in use by your provider to apply the workaround, as well as verify the peer in use when the problem is observed.

 

The last few weeks the website of geocaching.com is very slow or not loading at all in the evening time (e.g. 20:00h Amsterdam time +1 GMT)

 

Can anybody tell what is the problem???

 

Are to little servers available? Or are just the cachers in the US of A waking-up?

 

It's very anoying not being able to navigate the website properly and log or search caches.

 

Anybody any idea's?

 

I see these routing issues from KabelBW in Germany too - basically the whole of the Geocaching web service, including API calls from GSAC, is unusable after around 8pm CET. It's meaning I have to make sure I have everything done in the morning that I need for evening caching trips, and is also forcing me to log in batches when I have a morning free :-(

 

I can request a shunt for the 109.192.0.0/15 network, but can you please provide me with the output of http://tracer01.Groundspeak.com during a slow period? Feel free to remove your specific IP.