Jump to content
Sign in to follow this  
Followers 8
Avernar

Ideas for future PQ1K email or direct download link

Recommended Posts

Many of us are unhappy with the no emailing of 501+ pocket queries. I'd like to discuss the ways we would be able to get the PQ1Ks automatically while reducing the load on Groundspeaks servers. Some of the ideas I've had or heard about:

  1. Add an option in the preferences to pick email or download. Large PQs generation could be delayed to reduce stress on the outbound email server. Regular but not too frequent (monthly?) site visits would be a requirement to keep the PQs going.
  2. Use http authentication on the download link instead of session cookies.
  3. An alternate no authentication required download link that auto deletes the file after successful download.

Any other ideas?

Share this post


Link to post

Don't really understand the strategies behind it. Sending big PQ's via Mail shouldn't affect CPU utilisation but rather the network load on Groundspeaks Mail Gateway. But why are uncompressed PQ's still possible? An uncompressed 500 pq is approx. 3 to4 times bigger than a compressed 1000pq.

Also, if you send them you don't need disk space to store them.

Black Friday for GSAK users.

 

Sorry I cannot deliver another Idea except send them via email and disallow uncompressed attachements

Edited by gzxto

Share this post


Link to post

Don't really understand the strategies behind it. Sending big PQ's via Mail shouldn't affect CPU utilisation but rather the network load on Groundspeaks Mail Gateway.

I'm starting to believe that it's the ability to shut down unused PQs is the real motivation behind it.

 

But why are uncompressed PQ's still possible? An uncompressed 500 pq is approx. 3 to4 times bigger than a compressed 1000pq.

Also, if you send them you don't need disk space to store them.

 

Sorry I cannot deliver another Idea except send them via email and disallow uncompressed attachements

Actually, that's a great idea for reducing the load on the email servers.

Share this post


Link to post
[*]Use http authentication on the download link instead of session cookies.

I like this idea. Sounds easiest for a 3rd party developer to work with.

 

If you're right in that the motivation is to prevent all automation (to detect unused PQs) then there's no hope of change. I fervently hope that is not the case.

Share this post


Link to post
[*]Use http authentication on the download link instead of session cookies.

I like this idea. Sounds easiest for a 3rd party developer to work with.

 

If you're right in that the motivation is to prevent all automation (to detect unused PQs) then there's no hope of change. I fervently hope that is not the case.

 

Scraping a link via http authentication is really no different than writing a script to login and send a session cookie, which Raine has stated is a violation.

Share this post


Link to post

If you're right in that the motivation is to prevent all automation (to detect unused PQs) then there's no hope of change. I fervently hope that is not the case.

I don't think it's to prevent automation itself. The problem with email is that they can't tell if someone is actually picking it up and using it.

Share this post


Link to post

Scraping a link via http authentication is really no different than writing a script to login and send a session cookie, which Raine has stated is a violation.

Scraping is automating user input. A direct link in the notify email that requires http authentication would be explicitly designed for automatic retrieval and therefore is not scraping.

Share this post


Link to post

full ACK to those suggestions. doesn't even need to be HTTP auth, a simple unique link for the download would be perfectly sufficient.

 

either

downloadpq.ashx?g=xxx with xxx being a GUID that's uniquely generated for each time the PQ is generated, or

 

downloadpq.ashx?g=xxx&token=yyy with xxx being the regular GUID of the PQ, and yyy being a unique auth token.

 

of course that link would have to be included in the PQ email.

Edited by dfx

Share this post


Link to post

Being the Devil's advocate, the problem with HTTP authentication is that it sends the username and password in the clear. Not to mention that it would require the disclosure of credentials to third party apps. One of these problems is easily mitigated with HTTPS but the second is a problem.

 

Option 4 would be to include a single-use token in the headers of each 'PQ ready' email that would allow one (successful) download of the PQ without a login or knowledge of it's GUID. [edit] Basically what dfx just said :anibad:

 

There are probably countless other options that would also allow an automated retrieval without enabling PQ sharing.

Edited by JeremyR

Share this post


Link to post

Have you guys even looked at the link that's already in use?

 

It uses the GUID of the PQ and doesn't change from one run to the next. So you don't even need it in the email. Just set them in your retrieval tool once, and forget about it.

 

Of course, your retrieval tool does need to be logged in, which requires scraping, so you're dead in the water before you even leave the dock.

Share this post


Link to post
Of course, your retrieval tool does need to be logged in, which requires scraping, so you're dead in the water before you even leave the dock.

exactly, that's the whole point of this thread :anibad:

 

of course i do have a script that logs me in to the website and gets my login cookie, allowing me to do stuff on the website from a script. but 1) i have to change it every time GS changes the website, and 2) i'd rather not use it if there's other options of doing things. this other option currently is using PQs with max 500 caches, so i'll stick to that for now.

Edited by dfx

Share this post


Link to post

Being the Devil's advocate, the problem with HTTP authentication is that it sends the username and password in the clear. Not to mention that it would require the disclosure of credentials to third party apps. One of these problems is easily mitigated with HTTPS but the second is a problem.

Lil Devil's advocate by any chance? :anibad:

 

You already send you're password in the clear when logging on to the site so that doesn't change. And those people doing automatic email retrieval already have their password in GSAK so I don't see any difference there.

Share this post


Link to post
You already send you're password in the clear when logging on to the site so that doesn't change. And those people doing automatic email retrieval already have their password in GSAK so I don't see any difference there.

Just because we already do it doesn't make it a good idea when there's an alternative B) and yes, I'm sure we all trust GSAK but it's not the only caching application out there. If you look at the bigger picture, sites where you previously would have shared your login details to allow third-party access (think Flickr, Twitter, Facebook...) are all moving to token-based authentication where you don't have to give your credentials to the third party & there's a good reason for it - common sense (see oauth for more).

 

[edit] what DunkleAura said. That's twice I've managed that this afternoon :anibad:

Edited by JeremyR

Share this post


Link to post
Of course, your retrieval tool does need to be logged in, which requires scraping, so you're dead in the water before you even leave the dock.

exactly, that's the whole point of this thread B)

 

of course i do have a script that logs me in to the website and gets my login cookie, allowing me to do stuff on the website from a script. but 1) i have to change it every time GS changes the website, and 2) i'd rather not use it if there's other options of doing things. this other option currently is using PQs with max 500 caches, so i'll stick to that for now.

 

and 3) Raine, t he lead developer for the site, will ban you if he catches you doing this :anibad:

 

IMO, simple scraping of this type should be allowed, as long as it's not abusive. The site already has technical protections in preventing scraping abuse. A simple scripted 'login and download url' is not going to add load to the site, especially since it can be done no more than 5 times per day.

Share this post


Link to post
Scraping a link via http authentication is really no different than writing a script to login and send a session cookie, which Raine has stated is a violation.

There is no mechanism for this right now, so it is something Groundspeak has to explicitly put in place for it to work. If they do that, it is implied that accessing it this way has their blessing.

Share this post


Link to post

Just because we already do it doesn't make it a good idea when there's an alternative :anibad:

Not saying we shouldn't improve things. I was just pointing out it wouldn't be worse.

Share this post


Link to post
and 3) Raine, t he lead developer for the site, will ban you if he catches you doing this :anibad:

yes, which is reason for 2) B)

 

IMO, simple scraping of this type should be allowed, as long as it's not abusive. The site already has technical protections in preventing scraping abuse. A simple scripted 'login and download url' is not going to add load to the site, especially since it can be done no more than 5 times per day.

i agree, but i have a strong feeling that it ain't gonna happen :D

 

Scraping a link via http authentication is really no different than writing a script to login and send a session cookie, which Raine has stated is a violation.

There is no mechanism for this right now, so it is something Groundspeak has to explicitly put in place for it to work. If they do that, it is implied that accessing it this way has their blessing.

yes there is a mechanism for this in place. it's the regular HTML login form together with HTTP cookies. perfectly usable from a script if you know how, the only problem is that you're not supposed to do that.

Share this post


Link to post

IMO, simple scraping of this type should be allowed, as long as it's not abusive. The site already has technical protections in preventing scraping abuse. A simple scripted 'login and download url' is not going to add load to the site, especially since it can be done no more than 5 times per day.

Yeah, but if they allow scraping in one situation it may open up a can of worms for them. By having an official non scraping way of auto downloading the PQ would let them still say that scraping is 100% against the TOU.

Share this post


Link to post
The problem with email is that they can't tell if someone is actually picking it up and using it.

Yeah, so? They got the $30, we should be able to get our PQs. I don't use all of the caches in a PQ. They gonna start restricting me to only caches I actually attempt?

 

Whether we actually use the PQ we've legally and legitimately received should not even come into play.

Edited by CoyoteRed

Share this post


Link to post
Yeah, so? They got the $30, we should be able to get our PQs. I don't use all over the caches in a PQ. They gonna start restricting me to only caches I actually attempt?

 

Whether we actually use the PQ we've legally and legitimately received should not even come into play.

technically yes, but trying to avoid waste is also perfectly understandable. nobody's going to have a problem with that as long as it's non-intrusive and doesn't break anything.

 

but since you've mentioned it: i'd love to have PQs give me only caches i'm actually going to attempt! do those crystal balls come with ethernet interfaces yet?

Edited by dfx

Share this post


Link to post

yes there is a mechanism for this in place. it's the regular HTML login form together with HTTP cookies.

Don't like this method regardless of it's TOU problems. Too much to go wrong. If they change the site layout it breaks. What if they internationalize it and it's something other than English?

Share this post


Link to post
Have you guys even looked at the link that's already in use?

 

It uses the GUID of the PQ and doesn't change from one run to the next. So you don't even need it in the email. Just set them in your retrieval tool once, and forget about it.

 

Of course, your retrieval tool does need to be logged in, which requires scraping, so you're dead in the water before you even leave the dock.

Interesting observation, thanks. I haven't requested the same PQ twice, so don't know that the GUID is static. Thank you for that. I see now that a newer version of the PQ replaces the older one. That makes perfect sense - not much point keeping a stale version of the PQ around (well, there are a couple of legitimate reasons, but no need to go into that).

 

I created a shortcut on my system to that URL. So I'm using Firefox as my retrieval tool. I can now have a Windows command line batch file that will launch the URL and retrieve the file(s) automatically, as long as my session cookies are not stale. I don't script the login.

 

I'm not sure if this violates the TOU - there's definitely no screen scraping involved though. I run a batch file and it automatically fetches my PQs using my browser, the same browser I use for all other purposes, and which is logged into Groundspeak because I spend too much time in the forums and on the main site.

 

Seems like a reasonable workaround, but I'd like Groundspeak's blessings before I go ahead with it.

Share this post


Link to post

Can someone just explain to me what is so awful about having to download the file? I am missing something here.

 

And yes, I was using GSAK's get data via email until today. It was nice, but having to download isn't so awful. And I did see I think Raine mention wanting to work on letting GSAK work with it. I can wait.

Share this post


Link to post
Can someone just explain to me what is so awful about having to download the file? I am missing something here.

I can't speak for others. I can only speak for myself. As it should be.

 

There's nothing "awful" about having to download the file manually, by opening the site, logging in if necessary, going to the PQs page, and clicking on the PQ. I've tried the "automatic fetch from email" GSAK method, and I don't use it myself (reason : too little control). However...

 

Part of the reason I'm looking at automation is that this is my nature, and my profession. Manual, repetitive tasks are exactly why computers are created. Finding ways of accomplishing the same thing with less manual effort is the goal. The fact that it takes a lot of effort in order to accomplish that is besides the point - that is effort you put in once.

 

Some people like to search for better ways of doing things. I happen to believe that the desire for improvement is a good thing for all in the long run.

Edited by Chrysalides

Share this post


Link to post
Can someone just explain to me what is so awful about having to download the file? I am missing something here.

I can't speak for others. I can only speak for myself. As it should be.

 

There's nothing "awful" about having to download the file manually, by opening the site, logging in if necessary, going to the PQs page, and clicking on the PQ. I've tried the "automatic fetch from email" GSAK method, and I don't use it myself (reason : too little control). However...

 

Part of the reason I'm looking at automation is that this is my nature, and my profession. Manual, repetitive tasks are exactly why computers are created. Finding ways of accomplishing the same thing with less manual effort is the goal. The fact that it takes a lot of effort in order to accomplish that is besides the point - that is effort you put in once.

 

Some people like to search for better ways of doing things. I happen to believe that the desire for improvement is a good thing for all in the long run.

 

I have faith that in no time Clyde or one of his macro experts will automate the task. :anibad:

Or one of the GM scripters. With GS's approval of course.

 

Another thing, I am ALWAYS logged in, when I am home or at work I always have a browser open. Why would you close it? Only reason I could think of is a shared computer. If that is the case, get another browser and use it exclusively for PQ's.

 

I saw someone mention that folks who have automated requests yet don't visit GC.com should be turned off.

However, believe it or not there may be folks who cache and don't log them.

But now if they request a larger PQ and never download it, then you know they aren't using them and they could be turned off.

Share this post


Link to post
Another thing, I am ALWAYS logged in, when I am home or at work I always have a browser open. Why would you close it? Only reason I could think of is a shared computer. If that is the case, get another browser and use it exclusively for PQ's.

Downloading 5 PQs into GSAK using the manual method requires it to be done sequentially, with 3 to 4 mouse clicks each, moving between windows. Would being able to do this in one click be such a bad thing?

 

I'm not saying it is unusable now. I'm not even saying it is bad. We're just exploring ways to make automation possible without violating the TOU. Some of us view this as a challenge (against a problem, not against Groundspeak). We code, and we don't like to wait for someone else to solve our problems if we can do something about it.

Share this post


Link to post

Don't really understand the strategies behind it. Sending big PQ's via Mail shouldn't affect CPU utilisation but rather the network load on Groundspeaks Mail Gateway.

I'm starting to believe that it's the ability to shut down unused PQs is the real motivation behind it.

Would that really be so bad? If a PQ is run daily or weekly and the owner never comes back to retrieve it, why should limited server resources be expended in creating it? Free up those resources for people who will actually use them!

Share this post


Link to post

<click> download

<click> download

<click> download

<click> download

 

...how hard is that?

 

Is thirty-five seconds going to make such a HUGE difference in your life?

Share this post


Link to post

<click> download

<click> download

<click> download

<click> download

 

...how hard is that?

 

Is thirty-five seconds going to make such a HUGE difference in your life?

A Firefox extension like DownThemAll would make it even easier. A Grease Monkey script might be workable too.

Share this post


Link to post

If a PQ is run daily or weekly and the owner never comes back to retrieve it, why should limited server resources be expended in creating it? Free up those resources for people who will actually use them!

Because they paid $35 just like all the other users of pocket queries?

Share this post


Link to post

Is thirty-five seconds going to make such a HUGE difference in your life?

35 seconds + 5 seconds to log in * 2 accounts = 80 seconds per day * 365 days = over 8 hours

 

Why should I loose 8 hours of my life for something that can be accomplished in about 6 minutes (1 second * 365 days)?

 

But seriously, why would I do something that takes 80 seconds when I can accomplish the same thing in 1 second?

Share this post


Link to post

And I did see I think Raine mention wanting to work on letting GSAK work with it.

That is correct but I hope it's not some secret API that only GSAK knows. There are a few people I've seen mention that they use some other non GSAK workflow.

Share this post


Link to post

<click> download

<click> download

<click> download

<click> download

 

...how hard is that?

 

Is thirty-five seconds going to make such a HUGE difference in your life?

ok. i need to put the GPX on my PDA for import. currently this works by:

 

turn PDA on, enable WLAN, open TC, connect to FTP, download GPX, open beelineGPS, import GPX. done. note how the PC doesn't even come into play here.

 

with the new system it would have to be something like this (assuming the PC is already running):

 

start browser, go to gc.com, look through list of new PQs, click on each of them, select proper download directory, download them, start script to merge them all into one GPX, turn PDA on, take SD card out, put it in card reader, put card reader into PC, mount card, copy GPX on card, unmount card, take card reader and card out, put card back in PDA, start beelineGPS, import GPX. done.

 

understand now?

Edited by dfx

Share this post


Link to post

Let's try to get back to the original topic.

 

Here's something I've been wondering. For all the non-email solutions we've discussed, how would we retrieve our PQs if the site goes down for the day?

 

I would think this would probably hurt the manual downloaders more than us automatic people...

Share this post


Link to post

ok. i need to put the GPX on my PDA for import. currently this works by:

 

turn PDA on, enable WLAN, open TC, connect to FTP, download GPX, open beelineGPS, import GPX. done. note how the PC doesn't even come into play here.

 

with the new system it would have to be something like this (assuming the PC is already running):

 

start browser, go to gc.com, look through list of new PQs, click on each of them, select proper download directory, download them, start script to merge them all into one GPX, turn PDA on, take SD card out, put it in card reader, put card reader into PC, mount card, copy GPX on card, unmount card, take card reader and card out, put card back in PDA, start beelineGPS, import GPX. done.

Wow. I didn't even think about how it would affect PDA users.

Share this post


Link to post
...how hard is that?

 

Is thirty-five seconds going to make such a HUGE difference in your life?

Does it really bother you so much that others are trying to do something that would, at worst, not affect you, and at best, simplify your workflow, that you'd take 30 seconds out of your life to ridicule them?

Share this post


Link to post

Making the manual download as easy as something I'm doing now won't do a bit of good if the site is not up for you to do a download. That's a major drawback over getting emails.

 

How I'm doing it now:

  • I've set up a special mailbox to which only PQs are to go.
  • I've set up a script to check that mailbox daily and put the contents into a special folder.
  • The script deletes anything that is not a ZIP file. (The mail box gets emptied daily and all junk is deleted.)
  • Then on a less frequent schedule GSAK gathers those PQs and I massage as needed. (First manual interaction with the PQs.)

On the other hand, the manual download scheme would go like:

  • Open a browser and navigate to the Pocket Query page via bookmark. (Start of manual interaction.)
  • Click on Pocket Queries Ready for Download Link.
  • Click on the PQ.
  • Click to confirm file name and destination.
  • Repeat multiple times.
  • Then I get to open GSAK and import. Massage as needed.

The major issue is whether geocaching.com is going to be up when I want to download stuff. This site goes down. Happened just yesterday. (Oops, day before now.) For the manual download scheme of updating my caches the site has to be up on my schedule. Otherwise, I go do something else.

 

You who like the manual download scheme can have it while I stay with the email scheme. The email scheme is less prone to failure. Even when it does fail, those who keep an OLDB still have data.

Share this post


Link to post
I would think this would probably hurt the manual downloaders more than us automatic people...

Exactly what I was thinking. There's plenty of hand wringing as it is when the site goes down. Not getting you're weekly fix of PQs because you've opted for manual downloads instead emails would be a major kick in the gut.

 

I'm sticking with an email scheme.

Share this post


Link to post
There's plenty of hand wringing as it is when the site goes down. Not getting you're weekly fix of PQs because you've opted for manual downloads instead emails would be a major kick in the gut.

I can remember plenty of threads where you "automatic people" didn't get your weekly fix because the mail server went down, or your ISP decided to throttle your mail, or whatever. I don't see how the risk of the website being down is much different.

 

If you think email is a lessor risk, just keep your PQs limited to 500 caches.

 

Since I'm one of those "manual people" who just runs PQs when I need them, I'm happy to take the email out of the equation.

Share this post


Link to post

Since I'm one of those "manual people" who just runs PQs when I need them, I'm happy to take the email out of the equation.

 

and to have the potential to get a lot more data when we do run them manually.

Share this post


Link to post
Many of us are unhappy with the no emailing of 501+ pocket queries. I'd like to discuss the ways we would be able to get the PQ1Ks automatically while reducing the load on Groundspeaks servers. Some of the ideas I've had or heard about:
  1. Add an option in the preferences to pick email or download. Large PQs generation could be delayed to reduce stress on the outbound email server. Regular but not too frequent (monthly?) site visits would be a requirement to keep the PQs going.
  2. Use http authentication on the download link instead of session cookies.
  3. An alternate no authentication required download link that auto deletes the file after successful download.

Any other ideas?

How about thinking outside to box... actually my suggestion is very much inside the (server) box; so far inside, that it is outside... whatever.

 

Geocaching.com could host an internal PostOffice for POP only (maybe IMAP as well; no SMTP for the users, just for the system) and without reception of email from outside the domain. Each user would have an email address consisting of their usernumber @geocaching.com. Email older than 1 week would automatically be deleted. Then we could simply point our email clients to those accounts.

 

Another way, why not give us ftp access to the downloadable files.

Share this post


Link to post

How about thinking outside to box... actually my suggestion is very much inside the (server) box; so far inside, that it is outside... whatever.

 

Geocaching.com could host an internal PostOffice for POP only (maybe IMAP as well; no SMTP for the users, just for the system) and without reception of email from outside the domain. Each user would have an email address consisting of their usernumber @geocaching.com. Email older than 1 week would automatically be deleted. Then we could simply point our email clients to those accounts.

 

Now there is an idea. I bet the present GSAK getmail would work real nice with that.

 

Another way, why not give us ftp access to the downloadable files.

Security might be a problem unless we each had our own ftp account. That could be an admin nightmare.

 

But since I'm not highly automated on the PQ's, grabbing them mostly when I go caching, downloading as needed is not a problem. I would like to shut off the email for 500 or less.

Edited by jholly

Share this post


Link to post
Another way, why not give us ftp access to the downloadable files.

Security might be a problem unless we each had our own ftp account. That could be an admin nightmare.

Need not be. There's already an authentication mechanism in place. It won't be your usual ftp server code, but it could work.

 

I like the POP3 idea. Simplify it further. The only email messages in there are PQ results. It's a beautiful idea. You don't need a full POP3 server, just a POP3 interface to the PQ. You can use existing username and existing password for authentication. The only downside is that base64 encoding increases data bandwidth by approx 35%.

Share this post


Link to post
Many of us are unhappy with the no emailing of 501+ pocket queries. I'd like to discuss the ways we would be able to get the PQ1Ks automatically while reducing the load on Groundspeaks servers. Some of the ideas I've had or heard about:
  1. Add an option in the preferences to pick email or download. Large PQs generation could be delayed to reduce stress on the outbound email server. Regular but not too frequent (monthly?) site visits would be a requirement to keep the PQs going.
  2. Use http authentication on the download link instead of session cookies.
  3. An alternate no authentication required download link that auto deletes the file after successful download.

Any other ideas?

How about thinking outside to box... actually my suggestion is very much inside the (server) box; so far inside, that it is outside... whatever.

 

Geocaching.com could host an internal PostOffice for POP only (maybe IMAP as well; no SMTP for the users, just for the system) and without reception of email from outside the domain. Each user would have an email address consisting of their usernumber @geocaching.com. Email older than 1 week would automatically be deleted. Then we could simply point our email clients to those accounts.

 

Another way, why not give us ftp access to the downloadable files.

 

Brilliant. Adminning such a scheme would potentially be a lot of work though.

Share this post


Link to post

I don't see how the risk of the website being down is much different.

Lots different. If the email server is down I can now go to the site and download them.

Share this post


Link to post

and to have the potential to get a lot more data when we do run them manually.

That's only because Groundspeak chose not to email the big ones. I'd rather have email as the primary delivery mechanism and the download as a backup for all sizes.

Share this post


Link to post

Geocaching.com could host an internal PostOffice for POP only (maybe IMAP as well; no SMTP for the users, just for the system) and without reception of email from outside the domain. Each user would have an email address consisting of their usernumber @geocaching.com. Email older than 1 week would automatically be deleted. Then we could simply point our email clients to those accounts.

I knew I forgot a good one. An advantage of IMAP over POP3 is that it has a push mechanism to avoid polling.

 

I like the POP3 idea. Simplify it further. The only email messages in there are PQ results. It's a beautiful idea. You don't need a full POP3 server, just a POP3 interface to the PQ. You can use existing username and existing password for authentication. The only downside is that base64 encoding increases data bandwidth by approx 35%.

The fact that people can download their PQs more than once makes that point moot.

Share this post


Link to post
Guest
This topic is now closed to further replies.
Sign in to follow this  
Followers 8

×
×
  • Create New...