Jump to content
Sign in to follow this  
Followers 8
Avernar

Ideas for future PQ1K email or direct download link

Recommended Posts

I think Groundspeak would have a pretty good idea of what people are doing with the pocket queries.

:anicute: How? I have not filled out any surveys on how I use my PQs, nor have I heard of anyone else filling one out. They pretty much have the same data that we have: forum posts and chit chat at events.

 

If you are expecting Groundspeak to help user not see the adds on the web site I suspect you have a long wait coming. I think they like the revenue from those.

If PQs were free. Since they're not I have no reason to look at ads when I get them (by email). And I have no problem paying more to download the PQ1Ks.

 

I do however see the ads when I log my finds, read other peoples logs, browse cache pages online and zip around the geocaching google map.

 

Please try not to accuse people of attacking every new request because they don't agree with the poster. They may just have a different opinion or confused this with an open forum where people are allowed to express an opinion.

There's a difference between voicing and opinion and attacking. Opinions:

 

* That would be too hard too implement practically/technically because...

* I wouldn't need it because...

* I don't think it's a good idea because...

* Maybe, but here's a better idea...

 

Attack:

 

* Why are you bringing this up again?

* Why are we discussing these petty/trivial things?

* I don't need it, so you don't either.

* That's a stupid idea.

* If you don't like it, go somewhere else.

Share this post


Link to post

they may be worried about the size of the email being rejected by mail servers.

Peoples My Finds PQs would have caused issues long ago if that were the case. And they could always auto disable a PQ if a mail server rejects it on size.

Share this post


Link to post

Please try not to accuse people of attacking every new request because they don't agree with the poster. They may just have a different opinion or confused this with an open forum where people are allowed to express an opinion.

There's a difference between voicing and opinion and attacking. Opinions:

 

* That would be too hard too implement practically/technically because...

* I wouldn't need it because...

* I don't think it's a good idea because...

* Maybe, but here's a better idea...

 

Attack:

 

* Why are you bringing this up again?

* Why are we discussing these petty/trivial things?

* I don't need it, so you don't either.

* That's a stupid idea.

* If you don't like it, go somewhere else.

 

That is your opinion and you are entitled to it. My opinion is this thread is going around in circles now.

 

Is there anyone from Groundspeak that can jump in here and answer why PQ's over 500 do not get mailed out?

Share this post


Link to post
The adds on the download page for the PQ's.

i still don't see any ads on that page, with or without ad blocker. am i blind? :anicute:

Share this post


Link to post
The adds on the download page for the PQ's.

i still don't see any ads on that page, with or without ad blocker. am i blind? :anicute:

 

OK, you had me thinking I was crazy. I just looked again. The page where you select your PQ's and download them. There is an add on the left under the menu box.

Share this post


Link to post

Attack:

 

* Why are you bringing this up again?

* Why are we discussing these petty/trivial things?

* I don't need it, so you don't either.

* That's a stupid idea.

* If you don't like it, go somewhere else.

 

That is your opinion and you are entitled to it.

You don't think those are attacks? Interesting.

 

But I agree, this is off topic.

 

Is there anyone from Groundspeak that can jump in here and answer why PQ's over 500 do not get mailed out?

The only reason officially mentioned so far is worries of bounced emails due to attachment size.

Share this post


Link to post

i still don't see any ads on that page, with or without ad blocker. am i blind? :anicute:

Are you using Lil Devil's Hide Nav Bars script by any chance?

Share this post


Link to post

You don't think those are attacks? Interesting.

 

Answer to your question.

 

If you believe they are personal attacks then i guess they are to you.

 

Recommendation.

 

If you are having personal issues with someone you believe to be attacking you then please take it somewhere else and stop filling this forum with it.

 

 

 

If emails being bounced has been an official statement then I guess that answers why we have to download the larger files.

Share this post


Link to post
Are you using Lil Devil's Hide Nav Bars script by any chance?

no i don't. just to be sure, i just disabled GM completely and also disabled adblock again, and did a full page reload. still nothing. i do see the "advertisement / advertise with us" block (which i always see), but there's nothing there.

 

i do see the ad in here (forums) without the ad blocker though.

 

screenshot050610143711.th.png

Edited by dfx

Share this post


Link to post

The only reason officially mentioned so far is worries of bounced emails due to attachment size.

 

I don't believe the Frog said that, but I did and I don't speak for the frog. So far I don't think the frog has said why over 500 is download only.

Share this post


Link to post
Are you using Lil Devil's Hide Nav Bars script by any chance?

no i don't. just to be sure, i just disabled GM completely and also disabled adblock again, and did a full page reload. still nothing. i do see the "advertisement / advertise with us" block (which i always see), but there's nothing there.

 

i do see the ad in here (forums) without the ad blocker though.

 

screenshot050610143711.th.png

That is the right page. Not sure why you are not seeing the add. So neither of us are losing it. That's good to know.

Share this post


Link to post

The only reason officially mentioned so far is worries of bounced emails due to attachment size.

 

Can you provide a link to that?

Share this post


Link to post

The only reason officially mentioned so far is worries of bounced emails due to attachment size.

 

I don't believe the Frog said that, but I did and I don't speak for the frog. So far I don't think the frog has said why over 500 is download only.

OK, this is as official as it gets.

 

http://forums.Groundspeak.com/GC/index.php...p;#entry4312114

 

To avoid issues with emailing large attachments, PQs from 501-1000 caches will be available for download only, although we will still send you an email notification with a link to the page to download it when it is ready. My Finds PQs will also be available as a download only. I'll have more information for you when we release new code later this week.
Edited by Chrysalides

Share this post


Link to post

If emails being bounced has been an official statement then I guess that answers why we have to download the larger files.

They said they wanted to avoid it. Not that it was happening already.

 

The fact that My Finds, which are much bigger, make it through means that it shouldn't be a problem. Plus a solutions was mentioned: Add a user preference option to allow large emails. If it gets rejected, they can auto turn off that option automatically.

Share this post


Link to post

OK, this is as official as it gets.

You beat me to it. BTW, the little red arrow in the quote line takes you to the quoted post for those who don't know.

Share this post


Link to post

My mistake. There was a second official reason for the download only:

 

We'll also be able to tell when a person doesn't download a PQ that they generated, so we can selectivity turn them off and get PQ's out the door even faster to people who do use them! (really excited exclamation mark)

 

I believe this is the real reason. But there are ways to do this while still keeping the email option.

Share this post


Link to post

My mistake. There was a second official reason for the download only:

 

We'll also be able to tell when a person doesn't download a PQ that they generated, so we can selectivity turn them off and get PQ's out the door even faster to people who do use them! (really excited exclamation mark)

 

I believe this is the real reason. But there are ways to do this while still keeping the email option.

 

I don't really like this reason. PQ's are already a paid service. Since I'm being charged for this feature I think I should be able to run them and ignore them if I want.

Share this post


Link to post
Are you using Lil Devil's Hide Nav Bars script by any chance?

no i don't. just to be sure, i just disabled GM completely and also disabled adblock again, and did a full page reload. still nothing. i do see the "advertisement / advertise with us" block (which i always see), but there's nothing there.

 

i do see the ad in here (forums) without the ad blocker though.

 

screenshot050610143711.th.png

That is the right page. Not sure why you are not seeing the add. So neither of us are losing it. That's good to know.

 

My ad space is blank too.

Share this post


Link to post

Thanks folks. :) They got that important bug fixed in record time:

adtmp.png

Share this post


Link to post

Based on the absence of new ideas and circular topics that appear to have little relevance to the new feature, I conclude that we are all out of ideas on the topic of "Ideas for future PQ1K email or direct download link"

Share this post


Link to post

Based on the absence of new ideas and circular topics that appear to have little relevance to the new feature, I conclude that we are all out of ideas on the topic of "Ideas for future PQ1K email or direct download link"

 

I think avernar is taking a nap.

Share this post


Link to post

Thanks folks. :) They got that important bug fixed in record time:

sorry to spoil it for you, but it's still blank for me :)

Share this post


Link to post
Based on the absence of new ideas and circular topics that appear to have little relevance to the new feature, I conclude that we are all out of ideas on the topic of "Ideas for future PQ1K email or direct download link"

I have my solution. I use a cmd line batch file to invoke Firefox to download them.

Share this post


Link to post
I have my solution. I use a cmd line batch file to invoke Firefox to download them.

i wonder if that violates the TOU. kinda sounds "automatic" to me :)

 

i could also download them, even without firefox. i have a script which can reliably log me into gc.com (at least until their next site update). it doesn't parse any of the html pages, thus no "screen scraping" involved. it's not any more automatic than a batch file, but it might as well be against the TOU. so i'm sticking with PQs <=500 for now.

Share this post


Link to post
I have my solution. I use a cmd line batch file to invoke Firefox to download them.

i wonder if that violates the TOU. kinda sounds "automatic" to me :)

I wondered too. I asked but there's no reply. I don't even do automatic logins - if I'm not logged in the download will fail. To be extra safe I don't script anything in firefox.

 

The main purpose is to easily download a 5 PQ set that I run to update a 25 mile radius roughly centered around me. My main concern is not to try to download all of them at once, but include a reasonable delay in between files

 

I know that GSAK can open an IE window, but I haven't experimented with it. Problem is I don't use IE, and chances are it won't be logged in.

Edited by Chrysalides

Share this post


Link to post
I have my solution. I use a cmd line batch file to invoke Firefox to download them.

i wonder if that violates the TOU. kinda sounds "automatic" to me :)

I wondered too. I asked but there's no reply. I don't even do automatic logins - if I'm not logged in the download will fail. To be extra safe I don't script anything in firefox.

 

The main purpose is to easily download a 5 PQ set that I run to update a 25 mile radius roughly centered around me. My main concern is not to try to download all of them at once, but include a reasonable delay in between files

 

I know that GSAK can open an IE window, but I haven't experimented with it. Problem is I don't use IE, and chances are it won't be logged in.

 

This has me wondering, how long it will be before the DOWNLOAD ALL button shows up via a GreaseMonkey script.

Share this post


Link to post
This has me wondering, how long it will be before the DOWNLOAD ALL button shows up via a GreaseMonkey script.

I have to admit I thought about creating that script, but the DownThemAll! Firefox extension already exists and works great, as pointed out in post 106.

Share this post


Link to post

Update from clyde of GSAK, in the GSAK forums.

 

Nate got back to me today. I can't say too much because there is still quite a few unknowns. However, the good news is that they didn't say "no" to GSAK automating the downloads.

 

However, I haven't as yet got an unqualified "yes" and there is still work to be done on the Groundspeak side before we can even begin to see anything on the GSAK side.

 

So this may take a while, but the direction does look promising

 

http://gsak.net/board/index.php?showtopic=...st&p=104965

Share this post


Link to post

Update from clyde of GSAK, in the GSAK forums.

 

Nate got back to me today. I can't say too much because there is still quite a few unknowns. However, the good news is that they didn't say "no" to GSAK automating the downloads.

 

However, I haven't as yet got an unqualified "yes" and there is still work to be done on the Groundspeak side before we can even begin to see anything on the GSAK side.

 

So this may take a while, but the direction does look promising

 

http://gsak.net/board/index.php?showtopic=...st&p=104965

 

Big download all button in GSAK works for me too :)

Thanks for that update!

Share this post


Link to post

Just my 2 cents from the perspective of bcaching.com.

 

Each bcaching user gets a personalized email address that PQs can be sent to automatically and the email processor automatically extracts the PQ and updates the database. bcaching users only have access to caches that they have uploaded via a PQ in the past 30 days which keeps the data fresh but is also a restriction required in the agreement with Groundspeak. Therefore it is necessary to send new PQs on a regular basis and having an automated mechanism makes it relatively painless.

 

So I am very happy that the email option for 500 or less has not been removed, and hope that if it IS removed in the future that an alternative automated mechanism is put into place first. It would also be nice to take advantage of larger PQ sizes.

 

How about embracing the "automated community" by offering a simple sanctioned method for automated retrieval of GPX files.

 

1. Change the notification email to include a direct downloadable link that *does not* require the login step. Manage the authentication by adding an attribute for the username to the URL and add a flag to the file that marks it "downloaded by direct url" after the first time so the direct link can only be used once. After the first time, additional attempts to use the URL could simply redirect to the regular download page or login page if not logged in. That would allow automated mechanisms to still function and still be notified by (a very small) email.

 

For example:

http://www.geocaching.com/pocket/downloadp...b&u=m_and_w

 

2. Once there is a flag on the file that tells you its been downloaded (automated) you could offer another mechanism for automated downloads that does not require an email notification; might be useful for GSAK. Offer a URL with a private guid that a user can use to request the NEXT available download. Return the next file that has not been downloaded this way and flag it as downloaded.

 

The URL could be the same as the last one except the guid would be permanently generated for the individual user and displayed on their download page for use in 3rd party apps.

Share this post


Link to post

I think avernar is taking a nap.

Nope. Was on the road. Nap came after.

 

There's still a few ideas popping up here and there in the thread. Whatever Groundspeak does to automate it, I really hope it's a public API and not a hidden one for a select few devs.

Share this post


Link to post
The adds on the download page for the PQ's.

i still don't see any ads on that page, with or without ad blocker. am i blind? :)

 

Yes sir you is! :)

Share this post


Link to post
The only reason officially mentioned so far is worries of bounced emails due to attachment size.
I don't believe the Frog said that, but I did and I don't speak for the frog. So far I don't think the frog has said why over 500 is download only.
OK, this is as official as it gets.http://forums.Groundspeak.com/GC/index.php...p;#entry4312114
To avoid issues with emailing large attachments, PQs from 501-1000 caches will be available for download only, although we will still send you an email notification with a link to the page to download it when it is ready. My Finds PQs will also be available as a download only. I'll have more information for you when we release new code later this week.

If this really is the reason--to "help" the customer by not giving them the option of having their email bounce--then it's a pretty poor reason. I have an ISP that allows multi-megabyte attachments. In fact, just to test the theory I archived the 18 PQs waiting to be processed and sent them to myself. A 7.7 mb file came through no problems.

 

A good scheme for this would be go ahead and send the larger files. If larger emails are getting bounced from certain email addresses, then turn off that option. Keeping the same scheme and provide for better error trapping would have been a better way to go.

 

Now that PQ1Ks have been delivered and once the bugs have been ironed out, perhaps this would be a good upgrade.

Share this post


Link to post

FTP and POP3 access are out.. Just want to poor cold water on the daydreaming :P

 

-Raine

Could someone pass me a towel? Gee, was I ever unprepared for that. :)

 

...but feeling refreshed now that it is over. :D

 

Raine even makes small babies cry :)

 

Seriously though, I saw those ideas and cringed. Writing an FTP *CLIENT* is hard enough. With PASV and all the other stuff to deal with. I certainly can't see the effort being worth it to do a server.

 

And POP3 isn't a file transfer protocol. Sheesh ;)

 

Presumably Groundspeak already has a defined API methodology with authentication. At least I assume that's what Geocache navigator and the iphone/etc app use to post field notes.

 

We just need that same methodology expanded to get our giant PQ's.

 

Raine, I presume automating the PQ downloads before you guys provide a method to do so will get me a similiar "RefreshAllGPX" nastygram?

 

Jason

Share this post


Link to post

In fact, just to test the theory I archived the 18 PQs waiting to be processed and sent them to myself. A 7.7 mb file came through no problems.

You know you're a Geocaching addict when you read the above and think he deleted those 18 PQs...

Share this post


Link to post

Presumably Groundspeak already has a defined API methodology with authentication. At least I assume that's what Geocache navigator and the iphone/etc app use to post field notes.

 

We just need that same methodology expanded to get our giant PQ's.

No we don't. That methodology/API is secret and not available to everyone.

Share this post


Link to post
And POP3 isn't a file transfer protocol. Sheesh :)

This discussion is only of academic interest since Raine already *cough* rained on this particular parade.

 

Ah, young grasshopper, but it can be. Surely you've received files in an email attachment?

 

It actually provides all the things to this particular need (which is the automatic retrieval of PQ):

 

1. The ability to authenticate a user (USER/NAME or APOP)

2. The ability to see what is the list of files available for download (STAT, LIST, TOP)

3. The ability to retrieve the file (RETR)

4. The ability to delete it after you're done (DELE)

 

Date of PQ generation, PQ name, all map cleanly into RFC-822 date and subject. The PQ itself is MIME encoded into the body of the message.

 

There is no need to provide an actual POP3 server. It is only a POP3 interface into the existing data store.

Share this post


Link to post

Presumably Groundspeak already has a defined API methodology with authentication. At least I assume that's what Geocache navigator and the iphone/etc app use to post field notes.

 

We just need that same methodology expanded to get our giant PQ's.

No we don't. That methodology/API is secret and not available to everyone.

And they only allow access from certain blocks of IP addresses.

Share this post


Link to post
There is no need to provide an actual POP3 server. It is only a POP3 interface into the existing data store.

but, in order to satisfy all those many email clients out there and make sure they can actually use this interface, the whole plethora of RFC 1939 needs to be implemented. it's "only" 23 pages as POP3 is a relatively simple protocol, but the details of getting everything right and bug-free, scalable and with good performance can get quite hairy and is everything but trivial. as a developer i would immediately bail out of such a plan, simply because there's many much easier ways to accomplish the same thing :)

Edited by dfx

Share this post


Link to post
There is no need to provide an actual POP3 server. It is only a POP3 interface into the existing data store.

but, in order to satisfy all those many email clients out there and make sure they can actually use this interface, the whole plethora of RFC 1939 needs to be implemented. it's "only" 23 pages as POP3 is a relatively simple protocol, but the details of getting everything right and bug-free, scalable and with good performance can get quite hairy and is everything but trivial. as a developer i would immediately bail out of such a plan, simply because there's many much easier ways to accomplish the same thing :)

Getting anything right, bug-free, scalable and with good performance can be hairy and non trivial. Otherwise developers will be out of work <_<

 

Other than the commands I listed, the only other thing that needs to be added is NOOP (I volunteer for that one), QUIT, and RSET. The optional commands are, well, optional.

 

Why would you bail from a plan just because it is non trivial? You bail from it if you don't have enough resources, or if the return on investment doesn't make sense, not because it is non trivial.

 

The purpose of this thread is actually to discuss ways of accomplishing direct download automatically. Can you list some of these many easier ways? And because disclaimers seem to be so popular in the forums now, let me add that I'm saying this not as sarcasm or an attack, but because I respect your views and genuinely wants to have more options listed.

Share this post


Link to post
Why would you bail from a plan just because it is non trivial? You bail from it if you don't have enough resources, or if the return on investment doesn't make sense, not because it is non trivial.

of course, i didn't say i would bail out because it was non-trivial, i said i would bail out because there's easier ways to do the same thing.

 

The purpose of this thread is actually to discuss ways of accomplishing direct download automatically. Can you list some of these many easier ways? And because disclaimers seem to be so popular in the forums now, let me add that I'm saying this not as sarcasm or an attack, but because I respect your views and genuinely wants to have more options listed.

the simplest way of doing the same thing is probably an RSS feed. an RSS client essentially does the same thing as a POP3 client (in this scenario): it connects to a service, optionally authenticates via HTTP, requests a list of available items and then downloads them. the advantage is that RSS is much more flexible and generic, it's actually meant to provide a service like that, the clients allow the users to filter the results in whatever way they wish (i.e. download different PQs into different directories), and it can be set up on existing infrastructure (HTTP servers).

Edited by dfx

Share this post


Link to post
the simplest way of doing the same thing is probably an RSS feed. an RSS client essentially does the same thing as a POP3 client (in this scenario): it connects to a service, optionally authenticates via HTTP, requests a list of available items and then downloads them. the advantage is that RSS is much more flexible and generic, it's actually meant to provide a service like that, the clients allow the users to filter the results in whatever way they wish (i.e. download different PQs into different directories), and it can be set up on existing infrastructure (HTTP servers).

Yes, that is definitely a nice clean interface - an authenticated RSS feed. I hope Groundspeak is taking notes :)

Share this post


Link to post

Indeed - subscription service - brilliant.

 

I concur: much more elegant than ftp or pop. I don't know about details such as security, but I would imagine that it could include automatic detection of new versions and issues, simultaneous loading of all accumulated resources, etc.

 

Gee, that could be exciting.

 

Wether it is the "simplest" I personally don't care. Hopefully a certain programmer is not too averse to this idea (I'll go ahead and put the towel in the dryer so it will be nice and warm for you :) just in case).

Share this post


Link to post

of course all those interfaces (RSS, POP3, FTP) aren't without problems either. first of all, people would have to store their password in the respective application, which i'm sure not everybody wants to do, so those people would have to revert to manually entering their password every time, kinda killing the "automatic" aspect of the whole setup.

 

if GS were to implement such a system and were to care about this issue, they could give users a seperate authentication token for the feed/accounts. this could either be the existing session ID, which would be unusable as it becomes invalid every time you log out, or could be a new unique auth token, which means additional work for GS and therefore isn't likely to happen (at least not in the near future).

 

the second problem is that people would have their clients set up to periodically check for new items to download, which even is the default behavior for RSS clients. GS may have worries about server load there (even though the load impact of an RSS feed would really be miniscule if done properly).

 

if those points are critical, then the only option is to fall back to email delivery. include a unique, pre-authenticated download link in the email and be done with it. disadvantage of this over the other ideas is that it doesn't get rid of all the emails.

 

ps: of course there's ways to deal with the password issue, for example by employing a one-way hash function together with an unknown secret value to produce the authentication token. let HASH() be a such a function, for example a folded and mime64 encoded MD5 (to make it shorter). so instead of using the password to authenticate the feed/downloads/accounts, they could use the result of e.g. HASH(SECRET + USERID), or HASH(SECRET + PASSWORD). this would make it safe to store the "password" in any client, as it can't be used to find the actual password for the account (at least not without knowing the SECRET).

Edited by dfx

Share this post


Link to post
ps: of course there's ways to deal with the password issue, for example by employing a one-way hash function together with an unknown secret value to produce the authentication token. let HASH() be a such a function, for example a folded and mime64 encoded MD5 (to make it shorter). so instead of using the password to authenticate the feed/downloads/accounts, they could use the result of e.g. HASH(SECRET + USERID), or HASH(SECRET + PASSWORD). this would make it safe to store the "password" in any client, as it can't be used to find the actual password for the account (at least not without knowing the SECRET).

I believe they did something similar with GoogleEarth KML files. You used a token to show you're allowed to get the file. Could have been a random token saved in the table, though, too.

 

It's fairly easy to secure a URL to a particular person.

Share this post


Link to post
And POP3 isn't a file transfer protocol. Sheesh :ph34r:

This discussion is only of academic interest since Raine already *cough* rained on this particular parade.

 

Ah, young grasshopper, but it can be. Surely you've received files in an email attachment?

 

It actually provides all the things to this particular need (which is the automatic retrieval of PQ):

 

1. The ability to authenticate a user (USER/NAME or APOP)

2. The ability to see what is the list of files available for download (STAT, LIST, TOP)

3. The ability to retrieve the file (RETR)

4. The ability to delete it after you're done (DELE)

 

Date of PQ generation, PQ name, all map cleanly into RFC-822 date and subject. The PQ itself is MIME encoded into the body of the message.

 

There is no need to provide an actual POP3 server. It is only a POP3 interface into the existing data store.

 

The bottom line is this: uuencoding a binary file to ascii to send it via email is a decent size hit on the CPU and RAM the two limited items of a server. Bouncing emails and bandwidth are a non issue really. CPU time & RAM usage are the main considerations of ANY server as they are limited, So, it seems there is much barking up a wrong tree as to reasoning for no longer using email to move binary files.

Share this post


Link to post
of course all those interfaces (RSS, POP3, FTP) aren't without problems either. first of all, people would have to store their password in the respective application, which i'm sure not everybody wants to do, so those people would have to revert to manually entering their password every time, kinda killing the "automatic" aspect of the whole setup.

...

ps: of course there's ways to deal with the password issue, for example by employing a one-way hash function together with an unknown secret value to produce the authentication token. let HASH() be a such a function, for example a folded and mime64 encoded MD5 (to make it shorter). so instead of using the password to authenticate the feed/downloads/accounts, they could use the result of e.g. HASH(SECRET + USERID), or HASH(SECRET + PASSWORD). this would make it safe to store the "password" in any client, as it can't be used to find the actual password for the account (at least not without knowing the SECRET).

If I understand what you're saying, the secret is still being stored in the application, in encrypted form or otherwise, so the paranoid will remain paranoid, and those who don't care will not care either way.

 

For one example of secure transmission of password, you can look at APOP command of POP3. Challenge / response has one advantage over a fixed secret hash - it is more resistant to a replay attack.

Share this post


Link to post
The bottom line is this: uuencoding a binary file to ascii to send it via email is a decent size hit on the CPU and RAM the two limited items of a server. Bouncing emails and bandwidth are a non issue really. CPU time & RAM usage are the main considerations of ANY server as they are limited, So, it seems there is much barking up a wrong tree as to reasoning for no longer using email to move binary files.

I agree that there are better ways of doing it, I'm merely addressing your analysis of this particular weakness of this method.

 

uuencoding / base64 encoding can be done fast, efficiently and without significant memory or CPU overhead, through the use of lookup tables.

Share this post


Link to post
The bottom line is this: uuencoding a binary file to ascii to send it via email is a decent size hit on the CPU and RAM the two limited items of a server. Bouncing emails and bandwidth are a non issue really. CPU time & RAM usage are the main considerations of ANY server as they are limited, So, it seems there is much barking up a wrong tree as to reasoning for no longer using email to move binary files.

Don't know how the discussion got off onto security. This is not banking.

 

The resistance of sending archival information in differential PQs is the main hold back of reducing throughput required for the system. Folks who update their databases whether maintaining an OLDB or something less encompassing only need the data that has changed in the past week. Groundspeak refuses to provide any indication of which caches are archived so we know to pull those caches from consideration. The only way for us to know is pull a full dataset and figure out which ones are missing.

 

Expanding the differential scheme to send only the data that has changed for any single cache, then the saving become huge. As it is, data that has not changed is useless and a waste of space.

 

Such a change could make PQ1Ks smaller than regular PQs now while actually providing the same coverage to a cacher's caching area.

Share this post


Link to post

Since we have the 'resources' tangent - I'd be curious as to which would cost Groundspeak more overall (memory, CPU, bandwidth)

 

Push type system: Create GPX file, and email out via SMTP - compress and encode GPX file in email message, fire it out the door, gone. Handle rejected emails. I'd say here the entire resource hit occurs when the PQ is generated as it's emailed out right away. Could make scheduling resource use easier but also double trouble.

 

Pull type system: RSS/POP3 - create GPX file and update RSS pointer or leave a 'message' in mailbox, which client polls. Client requests data which is then compressed and transferred or simply transferred. Slight overhead for clients continually polling the server for 'updates' - likely on something like a 10 min, 30 min or 60 min interval. Two resource hits on the CPU/RAM here - once to generate the PQ, and again at some undetermined time when the client requests the data. Will tend to occur within minutes of the original PQ generation but not a guarantee.

 

Hybrid Push/Pull system (current): Create GPX file, email out *minus data* via SMTP. Client logs into website and retrieves binary data via HTTP. Two resource hits on the CPU/RAM here - once to generate the PQ, and again at some undetermined time - however the HTTP download hit is likely very small, more likely a bandwidth hit here.

 

---

 

The question is, what's being optimized by the PQ1K restriction? DB load? Storage space? Bandwidth? Magic Smoke utilization?

Share this post


Link to post
Guest
This topic is now closed to further replies.
Sign in to follow this  
Followers 8

×
×
  • Create New...