Jump to content

Ideas for future PQ1K email or direct download link


Avernar

Recommended Posts

What happens if I live next to a boundary? Then I would have to download two files.

I've thought about that. They can just take all the different GPX segments you've requested and put them in one ZIP file.

 

Zipping up the GPX segments would take a lot less resources than querying the database.

Link to comment

Concerning the canned PQs and population density is establish "zones" for the greatest benefit to the system.

 

Groundspeak already knows how many folks are getting PQs and where they are. They could create a new table indexed on origins. Then map the density of these. Starting with the most dense areas establish a "Super PQ" for that area. As an example, one of the most dense areas for PQs could be the Atlanta area. Pulling information out of the air with a wild guess, say, there are 1,000 people pulling 5 PQs each to get all of their stomping grounds of the area. That's 5 thousand individual PQs to be processed. Of course, each are custom to the person with founds and owned probably filtered out along with certain other criteria.

 

Groundspeak could come in and say it looks like a lot of duplication is this particular configuration of PQ. They could establish a "Super PQ" and offer it up to any PM. This could be all caches within a 50 mile radius of the geographical center of Metro Atlanta. It could be a file that has 2,000 caches in it. A cacher could download this file, along with his Found caches, and establish a database with all caches in the area complete with his Founds marked. A person who keeps an OLDB wouldn't need to download his Founds because he already has this data. You've just reduced the number of PQs from 5,000 to something considerably less. (Not to 1, because some with continue to opt for custom PQs.)

 

So, you live on the border of canned Super PQ, download the canned PQ and make a custom one fill out your criteria and Groundspeak would still have a massive gain.

 

The point is, the divisions of a canned PQ does not have to geopolitical. It just needs to benefit the largest number of people possible. That would include folks who won't directly benefit from a canned PQ, but who would still benefit from the greatly reduced sever load.

Edited by CoyoteRed
Link to comment

Does a canned PQ make sense? It would definitely cut down on database queries, but it would seem to be more prone to abuse (sharing) and makes it easier for someone to set up an alternate listing site (also against the TOU, but that won't stop some).

 

The problem with canned PQ's is we will have multiple threads proclaiming that GS has no idea what they are doing on the canned PQ's and multiple threads discussing what the future should be for canned PQ''s.

:D

Link to comment

What happens if I live next to a boundary? Then I would have to download two files.

 

<devils advocate>

... and?

 

Two canned files for 10 people in Bramalea to download is still way less resource intensive than those same 10 people running a PQ for caches within 500km of Bramalea individually.

 

</devils advocate>

 

What if we have different finds and I only want to see caches I have not found as I suspect everyone else will?

Link to comment

If you use Firefox, get the DownThemAll extension and experiment with it. It should be possible to do exactly what you're saying here.

 

I'm familiar with DownThemAll, and the danger with it (as with other download accelerators) is that you'll set off the "bot detection" when your computer requests 40 links at once on the site. You risk getting your IP address banned - particularly if your download accelerator makes multiple calls for the same file (to speed up the transfer).

I'm not sure why you'd download 40 links. At 5 PQ per days times 7 days, the most you'd get is 35 :D .

I tried DownThemAll with 5 PQs the other day and so far Groundspeak is not blocking them. It works very well. DownThemAll filters allowed me to select exactly which PQs I wanted and then downloaded them all in just a few seconds.

 

It would be nice if some lackey could confirm that we can use download accelerators to retrieve our PQs and not risk being identified as screen scrapers or other TOU violations.

Link to comment

Back to the original topic "email or web-download",

I think that FTP is still a very well established way for downloads like this.

 

One could leave all things as they are, and additionally provide an ftp service (and include the ftp download link in the email).

Link to comment

Does a canned PQ make sense? It would definitely cut down on database queries, but it would seem to be more prone to abuse (sharing) and makes it easier for someone to set up an alternate listing site (also against the TOU, but that won't stop some).

That's probably the biggest reason why we don't currently have them. The sharing wouldn't be an issue as the canned PQs would be large enough for almost everyone. The alternate listing site issue would be the problem.

Link to comment

Back to the original topic "email or web-download",

I think that FTP is still a very well established way for downloads like this.

 

One could leave all things as they are, and additionally provide an ftp service (and include the ftp download link in the email).

 

That is a fantastic idea. The login would verify you are who you say you are, and give quick access to the files.

Link to comment

What if we have different finds and I only want to see caches I have not found as I suspect everyone else will?

The canned PQs would probably be larger than most GPSr's can hold. You have to use GSAK or similar software to reduce that to what the GPSr can handle as caches and load the rest as POI. Finds would be filtered during the GPSr load.

 

Most people prefer to just drop their PQs onto their devices which is another reason I don't see canned PQs happening.

Link to comment

Back to the original topic "email or web-download",

I think that FTP is still a very well established way for downloads like this.

 

One could leave all things as they are, and additionally provide an ftp service (and include the ftp download link in the email).

 

That is a fantastic idea. The login would verify you are who you say you are, and give quick access to the files.

 

Not only that, but with a tool like FileZilla (or ncftpget) - I can schedule my computer to run out and get my PQs while I sleep, much like my weekly PQs.

Link to comment

That is a fantastic idea. The login would verify you are who you say you are, and give quick access to the files.

I like the FTP idea too. You can see the dates when the files ware created and only grab the new ones.

 

FTP is a little more server intensive than HTTP since you keep the connection open. But since only the power users would be using it then a decent sized connection limit might suffice.

Link to comment

I still like the POP3 interface suggested by another person. Then there's no need to change GSAK. And similar to FTP you have username and password for authentication.

 

Edit : credit where it is due. Hynr came up with the proposal.

Edited by Chrysalides
Link to comment

I still like the POP3 interface suggested by another person. Then there's no need to change GSAK. And similar to FTP you have username and password for authentication.

POP3 might be the easier of the two for both sides, but it will use 33% more bandwith.

 

Both methods still allow Groundspeak to know if people are not downloading them.

Edited by Avernar
Link to comment

Just as a point of clarification, I like most of the ideas that have been presented because most of them make it possible to do what I want to do: Regularly download a large area in a hands off manner.

 

The idea that I and CoyoteRed suggested, a download-able large canned query would work, as would the POP, FTP, forced compression, direct URL's with authentication built in, etc.

 

I believe that for most of us we would be happy to work around some minor extraction methods if we could just have an automated delivery method.

Link to comment

FTP and POP3 access are out.. Just want to poor cold water on the daydreaming :D

 

-Raine

no surprise there from my point of view, but could we then maybe get some other (more elaborate/helpful) "official" statement from GS, are there any plans to "fix" this issue in the near future?

 

if the answer is no, then at least we could put the whole issue to rest (kinda, anyway) and move on...

Link to comment
FTP and POP3 access are out.. Just want to poor cold water on the daydreaming :D

It won't be an actual POP3 server, just a POP3 interface. But if anyone knows it's not going to happen, it would be you :D

 

I know that having a Windows batch file do something like:

 

firefox [url=http://www.geocaching.com/pocket/downloadpq.ashx?g=xxxxxxxx]http://www.geocaching.com/pocket/downloadpq.ashx?g=xxxxxxxx[/url]
sleep 5 seconds, or otherwise check to make sure file download has completed
firefox [url=http://www.geocaching.com/pocket/downloadpq.ashx?g=yyyyyyyy]http://www.geocaching.com/pocket/downloadpq.ashx?g=yyyyyyyy[/url]

 

would work. Still some details to work out (where do the files go, need to configure Firefox to save file instead of launching GSAK, user needs to be logged in, etc.) But before I put more work into it, will Groundspeak allow me to fetch my PQs this way? Or where should I go to ask?

Edited by Chrysalides
Link to comment

FTP and POP3 access are out.. Just want to poor cold water on the daydreaming :D

 

-Raine

 

Ah, but this says POP3 and FTP are out .... doesn't say NO, just says not with these two methods ....

Daydreaming continues!! :)

are you hoping for IMAP, NFS or SMB now? :D

Link to comment

*sigh*

 

I could always print them out and bring them to you via the company jet...

 

man, fidonet brings me back to the days of running a BBS..hearing my bank of modems clicking as they dialed out to do the mail run

 

-Raine

Link to comment

FTP and POP3 access are out.. Just want to poor cold water on the daydreaming :D

 

-Raine

Could someone pass me a towel? Gee, was I ever unprepared for that. :D

 

...but feeling refreshed now that it is over. :)

Edited by Hynr
Link to comment

FTP and POP3 access are out.. Just want to poor cold water on the daydreaming :)

 

-Raine

Thanks for the cold shower.

Going further, to really stop the daydreaming, some words on the "why" and future plans would be more helpful to me.

Even if it's just about a few clicks, it'd make it less tiresome over the months and years.

Link to comment

Further to my suggestion of a "Download All" button, perhaps what would be most handy would be a "Download New" button, and put that button on http://www.geocaching.com/my (say, top right of the screen) so I can just log in, and mash the "Download New" button - which would give me a ZIP file of all the PQs that have changed since the last time I hit the button.

 

That would let me continue to load up GSAK while maintaining a minimum of cerebral activity - one click download.

Calculating which PQs have changed is a minor load (particularly when we're talking about a list of 40 files and their datestamps), and removes the need for me to check which ones I've got in the list (thus I can grab what I need before I've had that morning coffee).

 

GSAK is happy, because I can just toss the downloaded ZIP file on the application window and it'll be extracted and imported.

Link to comment

Further to my suggestion of a "Download All" button, perhaps what would be most handy would be a "Download New" button, and put that button on http://www.geocaching.com/my (say, top right of the screen) so I can just log in, and mash the "Download New" button - which would give me a ZIP file of all the PQs that have changed since the last time I hit the button.

 

That would let me continue to load up GSAK while maintaining a minimum of cerebral activity - one click download.

Calculating which PQs have changed is a minor load (particularly when we're talking about a list of 40 files and their datestamps), and removes the need for me to check which ones I've got in the list (thus I can grab what I need before I've had that morning coffee).

 

GSAK is happy, because I can just toss the downloaded ZIP file on the application window and it'll be extracted and imported.

 

It looks like the GPX file is already zipped up and store on a drive or in a database somewhere based on how fast the file starts downloading when I click on it. If this is true then a download all would have to repackage the files for a single download which would put a load on the servers repackaging the data or it could be sent down in a multi-part response which would require answering where to put each file as they are downloaded.

Link to comment

Further to my suggestion of a "Download All" button, perhaps what would be most handy would be a "Download New" button, and put that button on http://www.geocaching.com/my (say, top right of the screen) so I can just log in, and mash the "Download New" button - which would give me a ZIP file of all the PQs that have changed since the last time I hit the button.

or hey, how about just emailing those new PQs to your account's email address?

 

 

oh wait... :)

Link to comment
FTP and POP3 access are out.. Just want to poor cold water on the daydreaming :)

-Raine

 

Thank goodness as these two ports are two of the worst ways to move binary files on the internet! Let alone security risks.

 

Well, I sure doubt we're getting SCP/SFTP or SSH access for our downloads.

 

FTP isn't THAT bad. I wouldn't want to use it for downloading my tax return, but seriously - we're accessing data that's pretty darn close to public domain - there's nothing in a PQ that isn't already on the website somewhere.

Link to comment
*sigh*

 

I could always print them out and bring them to you via the company jet...

 

man, fidonet brings me back to the days of running a BBS..hearing my bank of modems clicking as they dialed out to do the mail run

 

-Raine

 

Sorry for going OT!

 

Raine what was the name of your BBS? I ran Starpoint Technology Station on WWIV, had four lines in the house and used to get 200 - 250 calls a day. I loved it! I still have a tried and trusted US Robotics Courier 56K v.92 Modem around here someplace.

Link to comment

Well, I sure doubt we're getting SCP/SFTP or SSH access for our downloads.

 

FTP isn't THAT bad. I wouldn't want to use it for downloading my tax return, but seriously - we're accessing data that's pretty darn close to public domain - there's nothing in a PQ that isn't already on the website somewhere.

 

Password in "the clear" :)

Link to comment

Well, I sure doubt we're getting SCP/SFTP or SSH access for our downloads.

 

FTP isn't THAT bad. I wouldn't want to use it for downloading my tax return, but seriously - we're accessing data that's pretty darn close to public domain - there's nothing in a PQ that isn't already on the website somewhere.

 

Password in "the clear" :)

 

True, but there are ways to mitigate such a threat (ie - anonymous download, separate password db). FTP still represents a nice, clean way to transfer files. Email runs into some issues with encoding stuff with POP3, that I'll grant you.

 

Ever notice the use of https:// when you log into geocaching.com ? How about a gold lock or gold title bar in your browser? Didn't think so - your GC.com username/password is very likely already at risk.

 

It boils down to the whole convenience vs security thing. The most secure computer server, is sealed in 10 feet of concrete with no connections of any sort penetrating the barrier. Not very useful, but is it ever secure!

 

We need something that is a balance between what you are protecting (mostly public information), and the cost of the protection, and the usability of the end product.

 

I suspect right now Groundspeak is trying to concentrate on the last two options - cost and usability. Jeremy still needs to make a profit, and there is a balance to be struck between Geocachers with Pitchforks and Geocachers taking his golden egg away from him.

Link to comment
FTP isn't THAT bad. I wouldn't want to use it for downloading my tax return, but seriously - we're accessing data that's pretty darn close to public domain - there's nothing in a PQ that isn't already on the website somewhere.

i disagree, FTP grossly violates the OSI layer model and that alone makes it a horrible protocol (but it's got other issues as well). i've had my fair share of problems with FTP during my lifetime, plus i don't see how HTTP is any worse than FTP for single file downloads.

 

strangly enough though, i don't consider FTP to be insecure, as SSL/TLS for it is pretty well established by now.

Edited by dfx
Link to comment
FTP isn't THAT bad. I wouldn't want to use it for downloading my tax return, but seriously - we're accessing data that's pretty darn close to public domain - there's nothing in a PQ that isn't already on the website somewhere.

i disagree, FTP grossly violates the OSI layer model and that alone makes it a horrible protocol (but it's got other issues as well). i've had my fair share of problems with FTP during my lifetime, plus i don't see how HTTP is any worse than FTP for single file downloads.

 

strangly enough though, i don't consider FTP to be insecure, as SSL/TLS for it is pretty well established by now.

 

Like it or not, FTP has become a major File Transfer standard. Ugly, but worldwide and implemented.

 

The problem you have with venturing into the newer protocols, is that Groundspeak is expecting this to work on 95% of the computers out there. The last thing I see Raine / Opinionate wanting to do is writeup a howto on getting Filezilla installed and configured properly for S/FTP. Whatever solution is picked - you can bet it's gonna have to work on a stock IE7/IE8, Firefox 3 or Safari install without using ActiveX or Java plugins that are a help-desk nightmare.

 

FTP, HTTP, HTTPS are all right there for Grandma to click on the link and be happy, without calling in Nerds on Site. Not sure IE7 does SFTP right out of the box ... but I know Microsoft moved username/password FTP over to Windows Explorer way back so there's that call to Geek Squad right there.

Link to comment
Like it or not, FTP has become a major File Transfer standard. Ugly, but worldwide and implemented.

true that. granted, i have no problem downloading a file off FTP on my PDA at all. i do have a problem downloading the GPX off a cache listing on IE mobile though, as all that clicking the button gets me is a "cache_details.aspx" file, which i then manually have to rename to .gpx in order to import it (for which i cannot use the included file manager of course, as this doesn't even show me any file extensions). on the 3rd party browser i have on there (skyfire) that button doesn't do anything at all. annoying, but i can live with it as i do this only once in a blue moon.

 

but i still wouldn't wanna administrate a large-scale public FTP server, so i can understand GS not wanting to do that. HTTP is just as good for simple downloads and doesn't have many of the problems that FTP has. it could be so simple: make the existing download link accept HTTP auth in addition to the regular cookie. if the cookie ain't there, look if there's HTTP auth. if there is, use that to authenticate the download. voila. doesn't even require any changes to the existing interface.

Link to comment
Does a canned PQ make sense? It would definitely cut down on database queries, but it would seem to be more prone to abuse (sharing) and makes it easier for someone to set up an alternate listing site (also against the TOU, but that won't stop some).

In this respect, it would no different than it is now.

 

There's nothing to stop someone starting a site and gathering enough folks to share the PQs they are getting now. They then post the PQs to that site which then lists the caches. Then other folks who are not members here could get to those caches.

 

The problem for any site that does this is folks would revolt. Ask Jeremy about the fiasco of trying to take over listings of letterboxes way back. That's not to mention "lawyers."

 

The threat of a parasitic listing site has been non-existent for years.

 

The threat of share downloads would be no different either way.

Link to comment
Most people prefer to just drop their PQs onto their devices which is another reason I don't see canned PQs happening.

"Most people?" Not sure how you figure that, but regardless, if they want custom PQs then they could still do that.

 

Options are good.

 

That's why the present PQ1K doesn't effect me as I'm going with the option of sticking with the old way and getting everything by mail. The reason I'm in the discussion is because this option doesn't allow for me to be part of the load reducing scheme just implemented.

Link to comment

"Most people?" Not sure how you figure that, but regardless, if they want custom PQs then they could still do that.

 

Options are good.

Just my observations in the local caching community at events. No formal polls or anything. ;) Getting GSAK set up properly and using it to consolidate and export to the GPSr is not a trivial thing to do. The less tech savvy geocachers simply don't bother and either drop their GPX onto their devices that support that or load them as waypoint through their management software (Mapsource, etc).

 

Yes, options are good. But the stuff that power users tend to want don't seem to be high priority. Then there's the folks who attack every new request posted in the forums.

 

Personally, I'd love to see larger canned PQs.

 

That's why the present PQ1K doesn't effect me as I'm going with the option of sticking with the old way and getting everything by mail. The reason I'm in the discussion is because this option doesn't allow for me to be part of the load reducing scheme just implemented.

Other than the download only My Finds, messed up PQ names when saving their settings, PQs that shut themselves off, and PQs that don't run at all the PQ1K changes don't effect me either. :anicute:

 

I'd love to reduce the load on the server too. I'm guestimating I can get rid of 1 PQ entirely by reducing the slack between my date ranged PQs.

Edited by Avernar
Link to comment

i think the problem with canned PQs is either 1) high bandwidth usage or 2) lots of manual administrative work.

 

canned PQs have the potential to become really huge. the simplest form would be one PQ for each state/country, which would definitely make them huge, and with many people downloading that every day, bandwidth usage would skyrocket.

 

or somebody at GS would have to split the caches into smaller groups that would make sense. but that would require a lot of manual and also continuous (as the cache landscapes change) work.

 

or, you'd be restricted to one canned PQ download per week or even less, which may be sufficient for some, but for other it would kill the purpose.

Link to comment

Just my observations in the local caching community at events. No formal polls or anything. :anicute: Getting GSAK set up properly and using it to consolidate and export to the GPSr is not a trivial thing to do. The less tech savvy geocachers simply don't bother and either drop their GPX onto their devices that support that or load them as waypoint through their management software (Mapsource, etc).

 

Yes, options are good. But the stuff that power users tend to want don't seem to be high priority. Then there's the folks who attack every new request posted in the forums.

 

I think Groundspeak would have a pretty good idea of what people are doing with the pocket queries. If you are expecting Groundspeak to help user not see the adds on the web site I suspect you have a long wait coming. I think they like the revenue from those.

 

Please try not to accuse people of attacking every new request because they don't agree with the poster. They may just have a different opinion or confused this with an open forum where people are allowed to express an opinion.

Link to comment
I think Groundspeak would have a pretty good idea of what people are doing with the pocket queries. If you are expecting Groundspeak to help user not see the adds on the web site I suspect you have a long wait coming. I think they like the revenue from those.

not sure what ads you're talking about. i just disabled my adblocker and followed the link in the notification email for the PQ downloads. there's no ads there.

Edited by dfx
Link to comment

i think the problem with canned PQs is either 1) high bandwidth usage or 2) lots of manual administrative work.

 

canned PQs have the potential to become really huge. the simplest form would be one PQ for each state/country, which would definitely make them huge, and with many people downloading that every day, bandwidth usage would skyrocket.

 

or somebody at GS would have to split the caches into smaller groups that would make sense. but that would require a lot of manual and also continuous (as the cache landscapes change) work.

I'd pay extra for a state/province wide PQ to cover the bandwidth.

 

For splitting, they could just carve things up based on lat/long squares or something once below the state/province level. Or by counties if they have that data. Plenty of ways to automate.

Link to comment
I think Groundspeak would have a pretty good idea of what people are doing with the pocket queries. If you are expecting Groundspeak to help user not see the adds on the web site I suspect you have a long wait coming. I think they like the revenue from those.

not sure what ads you're talking about. i just disabled my adblocker and followed the link in the notification email for the PQ downloads. there's no ads there.

 

The adds on the download page for the PQ's.

 

I think the issue with the over 500 cache PQ's not being mailed out may be that the mail server can't handle the load yet or they may be worried about the size of the email being rejected by mail servers. If it is an issue with the outbound load, hopefully that will be resolved in time. I guess only Groundspeak can comment on that.

 

I am sure there must be some technical reason for not mailing them. I doubt data mining is the issue as it would only take a few friends to team up to do that.

Edited by Keith Watson
Link to comment

 

The adds on the download page for the PQ's.

 

 

If ad revenue is the main goal, Groundspeak could start inserting advertising logs into the PQs, and then offer the Premium-Premium Membership to remove those ads. Just think! Next time you're out looking for a cache, and you pull up the last 5 logs on your Colorado, you'll have information about Poison Ivy blocking products and new items in the Groundspeak store!

 

 

I think the issue with the over 500 cache PQ's not being mailed out may be that the mail server can't handle the load yet or they may be worried about the size of the email being rejected by mail servers. If it is an issue with the outbound load, hopefully that will be resolved in time. I guess only Groundspeak can comment on that.

 

I am sure there must be some technical reason for not mailing them. I doubt data mining is the issue as it would only take a few friends to team up to do that.

 

I'll bet the helpdesk doesn't want to deal with (more) people who exceed their email box maximum message sizes. That's why I asked for an option to enable that, like Groundspeak did with GPX 1.0.1 files on the "Your Account" page. If I *know* I can accept a 30MB GPX file, then let me opt into it.

Link to comment
Guest
This topic is now closed to further replies.
×
×
  • Create New...