Jump to content
Sign in to follow this  
Followers 3
Hynr

Suggestions For Pq Generation System

Recommended Posts

It seems to me that the PQ generation system is no longer able to keep up with the needs of the clients (us). We are not getting what we paid for and that is a fundamental violation of our trust in the company.

 

It is pathetic to push the blame on the clients asking us to be more efficient and run fewer PQs. According to Jeremy: “Create a new PQ” if your’s doesn’t run as promised. I assume that this is being voiced with some frustration (which we don’t hear). I find it hard to believe that it is a financial choice to not add more computing power, because for just 100 clients that have been added, you expect to see additional $3K annual revenue which can pay for another number cruncher. Presumably thousands have been added to the ranks to render the PQ generation capacity inadequate. It would help to know if the choice to not keep up with demand has been a financial decision.

 

These problems were obvious already a few months ago and it was clear already then that getting the clients to be more efficient would be just a temporary band aid because the customer base is growing so rapidly. Can you go through our accounts again and uncheck all the regularly scheduled PQs to speed things up? Maybe, but I’m sure you know that this time the band aid will last much less time. I currently have only those checked that did not run when they were supposed to. Many other users are probably in the same situation.

 

I do recall the discussions a few months ago when we last saw these problems. I recall that many folks made some suggestions and we did see some of these ideas implemented. So in the interest of getting some improvements implemented I am starting this thread to have folks with ideas voice them here.

 

If you have a constructive suggestion (i.e. other than something like “please make it go faster” or “Yeah let’s do that”) then please feel free to speculate here as to what you think might be helpful.

Share this post


Link to post
It seems to me that the PQ generation system is no longer able to keep up with the needs of the clients (us). We are not getting what we paid for and that is a fundamental violation of our trust in the company.

 

Speak for yourself not behalf of me!

 

I personally would like see people answer question mentioned is this thread: link and generally explain what kind of queries they need. Because query system is rather flexible it allows us to create really silly queries which burdens the system too much.

 

So analysing the needs (reasonable) would probably help...

Share this post


Link to post

Until TPTB change the current prioritization scheme this problem will continue to get worse.

 

Here is my idea to change the scheme:

Have the PQ scheme not "remember" so long. Limit it to remembering what was run in only the last 24 - 48 hours.

 

Here are my tweaks of some ideas that I have seen proposed before on these forums. These don't necessarily change the scheme but they supplement it with additional options to get the data you want:

 

Implement the option to download the results of the PQ immediately from the website instead of generating the PQ and then emailing it. It has never made sense that the only way to get the PQ results in 1 file is to have them emailed to you. This way you could lower the strain on the PQ server.

 

Generate "pre-generated" PQ's for all states and allow the ability to download one state per day per account. These PQ's could include all but permanently archived caches. They could also include the last 25 logs for each cache. You could then use a third party application to tweak the caches for your needs.

 

Generate "pre-generated" PQ's for all states that include just the caches that have been published/updated/modified/archived in the past 72 hours, these could have all logs for the past 72 hours in them. Limit the downloads to 3 per day per account. You could then use a third party application to tweak the caches for you needs.

 

Change the limits from 5 daily PQ's with up to 500 caches each to 2500 caches per day using up to 5 PQ's. This could allow the PQ system to be even more flexible. This way might help reduce the number of PQ's needed to be run by each person.

Share this post


Link to post

Change the limits from 5 daily PQ's with up to 500 caches each to 2500 caches per day using up to 5 PQ's. This could allow the PQ system to be even more flexible. This way might help reduce the number of PQ's needed to be run by each person.

 

Who will need 2500 caches per day?

Share this post


Link to post
It seems to me that the PQ generation system is no longer able to keep up with the needs of the clients (us). We are not getting what we paid for and that is a fundamental violation of our trust in the company.
Speak for yourself not behalf of me!

I think the folks in charge understand that I am not an elected spokesperson for the masses. My statement is based on my observation here at the forum where we are once again indundated with PQ problems, many of which appear to be related to lack of capacity.

 

I personally would like see people answer question mentioned is this thread: link and generally explain what kind of queries they need. Because query system is rather flexible it allows us to create really silly queries which burdens the system too much.

I thought about contributing to that thread, but that thread assumes that the problem is a financial issue. We don't know that it is or isn't. If the system cannot deliver a PQ for me when I ask for it now, how will it do it if I promise to pay extra for extra PQs. So while that link shows that there is even more demand for PQs that some clients would like to see satisfied, it does not illustrate how the problem might be solved.

 

So analysing the needs (reasonable) would probably help...

That's my thought. And I would agree with your initial notion: let everyone express themselves based on their own situation, not based on any notion what everyone wants.

Share this post


Link to post
Generate "pre-generated" PQ's for all states and allow the ability to download one state per day per account. These PQ's could include all but permanently archived caches. They could also include the last 25 logs for each cache. You could then use a third party application to tweak the caches for your needs.

 

Combine this with the turning off of the scheduled PQs, sending a separate email to the account email detailing the change, and placing a link on each state page for the whole state download would do wonders.

 

What this would accomplish:

  • It would start taking the load off the PQ severs.
  • The separate email would effectively alert the users. Not like last time with announcement was with the PQ and no one saw it.
  • Would reduce, or eliminate, data aggregators like myself and thus the over all load would go down.

After a month, turn the scheduling back on.

 

Limit the full downloads to 1 a day if you're worried about exposing your dataset to the world. :anitongue:

 

Break up the huge states into regions.

 

Create weekly differentials, and encourage aggregating, to reduce web server bandwidth.

Share this post


Link to post

Change the limits from 5 daily PQ's with up to 500 caches each to 2500 caches per day using up to 5 PQ's. This could allow the PQ system to be even more flexible. This way might help reduce the number of PQ's needed to be run by each person.

Who will need 2500 caches per day?

I don't see how your question is relevant to this thread. 2500 cache records is the data volume limit now; Team yGEOh is asking to have the distribution be different in the interest of improving efficiency and/or satisfying his/her needs. I think we can leave it up to Raine and Jeremy to do the math to figure out whether this suggestion will contribute to solving the problem.

Share this post


Link to post

I keep adding my PQ problem to these threads because I haven't gotten any resolution to mine.

Besides the problems common to everybody else, I cannot get my "All Finds PQ". It has not been successfully run for five weeks. It says it is queued, but never runs. It is caught in any e-mail filters. The indicator that says when it last ran, resets to the last successful run (6/16) every night after I attempt to run it.

After I sent 3 help tickets on it (was getting no response) I got one e-mail saying they were working on it, but nothing else, or no response by e-mail or here in the forums. I know of another area cacher with the same problem.

Share this post


Link to post

Wacka, next time I see you, we'll have a discussion about hijacking threads and how annoying that is. Note this is a thread about ideas on how the problems might be resolved. See you later. - Hynr

Share this post


Link to post

According to Jeremy: “Create a new PQ” if your’s doesn’t run as promised.

 

Historically, Hynr, your posts are written to be pretty antagonistic. This post doesn't seem to be particularly different, but I'll address this statement above to clarify your false assumption that it was somehow mentioned in frustration.

 

First, we're aware of the issue with Pocket Query generation. Although throwing servers at a problem is a common and misunderstood way to address something, it isn't a simple problem you can solve by throwing money at it.

 

We are, however, looking into the issue and continuing to tweak the system. We're also adding new features, like caches along a route, that are designed not to impact the Pocket Query generator and hopefully alleviate the continuous queries that people say they have to do in order to get caches along a route the "old way". By creating these specialized queries we hope to steer people to this feature instead of bulk downloads.

 

The suggestion I wrote above is meant to be a way to alleviate the frustrations of people who want their caches now, not my frustration. It works and is a perfectly reasonable way to get that query.

Share this post


Link to post

Change the limits from 5 daily PQ's with up to 500 caches each to 2500 caches per day using up to 5 PQ's. This could allow the PQ system to be even more flexible. This way might help reduce the number of PQ's needed to be run by each person.

 

Who will need 2500 caches per day?

 

I do.

 

It's not because I might find that many in a day, it's to keep my off line database up to date. Why do I need an off line database? Because of the 5 log limit and PQ server problems. Why do I need 2500 caches? Because recently archived caches aren't included in the files therefore the differential PQs (changed in last 7 days) are incomplete.

 

There's a couple of ways to reduce server load and still allow folks to get what they want when they want. Allow archived caches in last days, all logs in last days, and combine two more queries into one. The aggregators' PQs would drop to about a quarter or less.

Share this post


Link to post

It's not because I might find that many in a day, it's to keep my off line database up to date. Why do I need an off line database? Because of the 5 log limit and PQ server problems.

And you need an off-line database of every log because....?

Share this post


Link to post

 

It's not because I might find that many in a day, it's to keep my off line database up to date.

 

That's weird. You don't seem to have a Premium Membership. How is this possible?

Share this post


Link to post

 

It's not because I might find that many in a day, it's to keep my off line database up to date.

 

That's weird. You don't seem to have a Premium Membership. How is this possible?

:anitongue:

Share this post


Link to post

I'm not a computer programmer, so this is from ignorance. Maybe the solution will have to be clean all the PQs off and have everybody regenerate them.

My earlier post was out of frustration (just got a "we're working on it" about 3 weeks ago). There is (was?) no way to see the status of your request and I would have appreciated a "We don't have any idea what is causing that" or a "It's related to the main PQ problems" reply.

My earlier post also pointed out that there are other problems with the PQ servers than the main one that is discussed all over the place.

 

(This paragraph added on edit)

The PQ I have been talking about finally came between my two posts! I won't be "hijacking" threads again.

Edited by Wacka

Share this post


Link to post

This discussion has prompted me to do something I have been neglecting. That is, to remove the PQ's I used to generate to get caches along specific routes. They're still active & I need to remove them. Now, if others would follow suit - nice reduction of load on the server...

Share this post


Link to post

Pre-generated GPX Cache segments

 

You might already be doing this, but what about pre-generating each <WPT>?

 

This could only work as a special type of PQ. One with none of the previously found caches so each are the same for everyone. We've already got a way to get all of our founds, anyway.

 

Figure how many times you have to compile any one particular cache during a run. I'm sure in heavily populated areas it is a bunch.

 

Store the compiled <WPT> data, including logs, in a record as text. Run the query looking only at coordinates, waypoint, type, etc. Use the waypoint to pull the pre-compiled data and append to the end of the string that will become the GPX file.

 

It would be heavy on the front end, but pretty sporty on the rear.

 

With optimized coding like this, you could implement a policy that these go to the front of the que to encourage its use.

 

Oh, oh, but wait, there's more! Don't pre-compile, but when you do generate a <WPT> then store and as you go through the queries look to see if it's been compiled pull it from that list. The queries would get faster and faster as they run and you don't compile any un-used caches.

Edited by CoyoteRed

Share this post


Link to post

 

And you need an off-line database of every log because....?

I keep asking this of various cachers as well......answer is always because they WANT to (not need).

 

Oh and by the way - The longest I have EVER had to wait for my requested PQ is 3 hours. Typically I have them in under 5 minutes and I can get out ang go caching. The new caches along a route has already saved me dozen or more uncessary PQs for trips I have taken. I'm happy. Just because the system doesn't happen to meet your desires doesn't mean it is entrely broken. Could it be tweaked - sure. Is it perfect - no. Could it do more - probably. Is it pretty great for the tiny $3 per month I pay - yes.

 

Thanks for keeping on top of this and adding more features guys.

Share this post


Link to post

I'm a newbie, so maybe I should just hold my tongue, but it doesn't make sense to me that the PQ can generate a preview, but then you can't download that data right then and there. If the query runs well enough to generate a preview, why not just include a button to click in order to download the data in GPX format based on what the preview generated?

 

I don't mind coming to the site to retrieve info. I'm not asking to eliminate the e-mailed queries, but I don't really need that feature. I'd just like to get the data when I request it, even if it takes a couple minutes to generate. I can always read the forums or tab-browse another site.

 

I'll shut up now....

Share this post


Link to post

I do.

 

It's not because I might find that many in a day, it's to keep my off line database up to date. Why do I need an off line database? Because of the 5 log limit and PQ server problems. Why do I need 2500 caches? Because recently archived caches aren't included in the files therefore the differential PQs (changed in last 7 days) are incomplete.

 

Ok this kind of feedback is constructive and now I understand some of the needs people have.

Of course it's totally different question do I think your need is "reasonable" or does it have sense at all... :anitongue:

 

What come to suggestion some mentioned about ready made queries you can just download sound interesting.

Edited by Geovius

Share this post


Link to post

If the query runs well enough to generate a preview, why not just include a button to click in order to download the data in GPX format based on what the preview generated?

 

The preview is a table of contents. You want the whole book.

Share this post


Link to post

I'm a newbie, so maybe I should just hold my tongue, but it doesn't make sense to me that the PQ can generate a preview, but then you can't download that data right then and there. If the query runs well enough to generate a preview, why not just include a button to click in order to download the data in GPX format based on what the preview generated?

 

When you preview the PQ, it returns back the results against the possible criteria - attributes, location, country, state, terrain, difficulty, has TB, whether or not the user has or hasn't found it, etc.

 

The preview does NOT return the last 5 logs, the names of travel bugs, the description (both long and short) and the other detail that is on the individual cache page. The GPX file DOES have that.

 

I WOULD be in favor of the possibility of returning a single LOC file after a a preview, without all of the extra data. But that's another topic.

Share this post


Link to post
When you preview the PQ, it returns back the results against the possible criteria - attributes, location, country, state, terrain, difficulty, has TB, whether or not the user has or hasn't found it, etc.
I understand Markwell's point - and I know this isn't it - but if there's any computational expense of including TB (and, unlike the others in that list, I suspect there is) I would gladly untick that box and never see TB data in a PQ again.

 

Folks that power hunt TBs are welcome to continue getting them, but a lot of users could untick it.

I WOULD be in favor of the possibility of returning a single LOC file after a a preview, without all of the extra data. But that's another topic.

You can spin the preview and then download the loc instead of the GPX today. But GPX is so much better supported by more tools that it could be GPX and even have key elements of the Groundspeak extensions and be niether computationally more expensive to prepare nor even substantially larger than .LOC. The set I've used in my own tools looks like:

 

<wpt lat="35.895267" lon="-87.0005">

<time>2004-09-30T00:00:00</time>

<name>GCKP78</name>

<desc><![CDATA[stumped in Leiper's Fork]]></desc>

<url>GCKP78.html</url>

<sym>Blue Diamond</sym>

<Groundspeak:cache xmlns:Groundspeak="http://www.Groundspeak.com/cache/1/0">

<Groundspeak:owner><![CDATA[LSUMonica]]></Groundspeak:owner>

<Groundspeak:type>Traditional cache</Groundspeak:type>

<Groundspeak:container>Micro</Groundspeak:container>

<Groundspeak:difficulty>3.5</Groundspeak:difficulty>

<Groundspeak:terrain>2</Groundspeak:terrain>

<Groundspeak:encoded_hints><![CDATA[insert Hint Here.]]></Groundspeak:encoded_hints>

</Groundspeak:cache> </wpt>

 

This subset just so happens to contain exactly the data natively supported by all GPSes supported by GPSBabel. One might deduce that's not a coincidence.

 

There's an education exercise as many users here blur the distinction between "GPX" and "Pocket Query", but that's a solvable problem.

 

But you're right, that's probably another topic, too. This one seem to be reserved for casting stones. :-)

Share this post


Link to post

 

And you need an off-line database of every log because....?

I keep asking this of various cachers as well......answer is always because they WANT to (not need).

 

As to the OP:

 

Reason 1: I travel a lot through work and don't always have access to the internet. In order to make caching as easy and enjoyable as possible I like to have an up-to-date offline database of Ireland (with approx 500 caches this is easy to do with 2 daily PQs). If I have an offline database I can easily cache in areas that I'm visiting on short notice.

 

Reason 2: Most of my cache hunts are in rural areas where there is limited access to the internet. If I want to see a hint/spoiler/cache page I want it on my PDA and the only option is an offline database. This also means I'm less likely to spend time hunting for a disabled/archived/missing cache if I have the most recent logs.

 

Reason 3: I'm permitted a total of 25 PQs a week. Why is there such a problem running 10?

 

Reason 4: I can! I have been able to until recently and so far no one has given a good reason why I can't/shouldn't or explained what is being done to allow me to do so again. This is the most frustrating part, being left in the dark!

 

I'd point you at Reasons 1 and 2 in particular. You didn't respond to my post in the other thread either, maybe youy missed it amongst the other "I Want to"s. I need to maintain an offline db and I'm sure there are many others like me!

 

I'd like to point out again that we're yet to get a good reason as to why the problem has developed*. I applaud Jeremy et al for getting involved in discussions about how to fix the problem but can someone please explain what is causing it. As I said above it's getting very frustrating :anitongue:

 

(*unless I've missed it among the numerous PQ related complaint threads)

Share this post


Link to post

I thought the reason was because of overuse that is straining the system. Like people requesting 25x2500 caches downloaded per week. Or repetitive PQs of entire regions, sucking the caches to an offline database on a daily basis, when most likely there isn't any significant change in the caching landscape between the two.

 

If I'm wrong about what the source of the problem is, I hope Jeremy will correct me.

 

Don't get me wrong, I too have an offline database, but I don't need it DAILY. I get it weekly, and I don't have any problems getting it generated.

 

I'd be in favor of a recurrence of last year's purge. Require the users to come back to the site to reschedule them - at least for a period of a week or two. I'm sure there are people that set up PQs and never GET the data anymore.

Share this post


Link to post

 

Reason 1: I travel a lot through work and don't always have access to the internet. In order to make caching as easy and enjoyable as possible I like to have an up-to-date offline database of Ireland (with approx 500 caches this is easy to do with 2 daily PQs). If I have an offline database I can easily cache in areas that I'm visiting on short notice.

 

Reason 2: Most of my cache hunts are in rural areas where there is limited access to the internet. If I want to see a hint/spoiler/cache page I want it on my PDA and the only option is an offline database. This also means I'm less likely to spend time hunting for a disabled/archived/missing cache if I have the most recent logs.

 

 

Reason 1: I travel a lot through work and don't always have access to the internet. In order to make caching as easy and enjoyable as possible I like to plan ahead by requesting a PQ of caches just before I leave. If I have this PQ within 5 - 10 minutes I can easily cache in areas that I'm visiting on short notice.

 

Reason 2: Most of my cache hunts are in rural Nebraska areas where there is limited access to the internet. If I want to see a hint/spoiler/cache page I can easily view it wih GPX sonar on my PDA via the downloaded PQ. This also means I'm less likely to spend time hunting for a disabled/archived/missing cache if I have the most recent logs. I know they are recent because I requested a new PQ just before setting out.

 

:) You sound just like me - sort of.....

Edited by StarBrand

Share this post


Link to post

This is the way I currently use the PQ system.

 

- Grab of the entire UK database, spread over 25 (and counting) PQs over the week. I like a local database, as the combination of GSAK and Memory Map makes planning walks much quicker and easier than using GC.com alone.

This grab is based on dates, so this refreshes each cache listing once per week. This is needed to pick up on archived caches so they can be dealt with in my database.

If, as someone suggested earlier in the thread, archived caches were included in PQs up to a limit (say 1 or 2 weeks after the archive action) this would remove the need to refresh every single cache every week - rather I (and the many other UK paperless cachers who do things this way) could just ask for the caches that have been updated in the last 7 days, hopefully reducing the database load.

 

- Grab of the recently updated caches within 30 miles of home everyday. This is so I can be confident of having the most recent logs on my PDA without having to interact with the site as much before setting off on a trip.

This used to be achieved by a single query running everyday, but I've now been forced to create 7 duplicates, one running each day.

 

Hope this helps TPTB understand how a few of their customers are trying to use their system.

Share this post


Link to post

In response to StarBrand (I need to type faster!):

 

Point taken but I have a wife, a small child and I run my own business. I don't always have the opportunity to run a PQ before I leave as un(?)fortunately it's rather low down the list of priorities.

 

Automating the two daily PQs with GSAK means that when I sync my PDA for work related items it automatically updates my db at the same time without having to log on to GC.com and wait for a PQ to generate and an email to arrive.

Edited by dino-irl

Share this post


Link to post

In response to StarBrand (I need to type faster!):

 

Point taken but I have a wife, a small child and I run my own business. I don't always have the opportunity to run a PQ before I leave as un(?)fortunately it's rather low down the list of priorities.

 

Automating the two daily PQs with GSAK means that when I sync my PDA for work related items it automatically updates my db at the same time without having to log on to GC.com and wait for a PQ to generate and an email to arrive.

 

I understand your point as well - don't get me wrong. I too have a wife, 2 small children and I own my own business. (told you we were a lot alike!). My point is - given that automated recurring PQs don't run in a consistent fashion, you can still get the data you need for a day (or two) of caching without worrying when/if the query will run. My actual question about offline databases is for those that "need" to store ALL the logs on thousands (or even hundreds) of caches in thier home systems. That has never made much sense to me.

 

Happy caching.

Share this post


Link to post

A number of the posts above address why some geocachers use PQs more heavily than others. I think what we will find is that every user has a slightly different use pattern but that most folks have a "home territory" for which they wish to keep their data fresh. Some users are happy with the last 5 logs and they are satisfied with having the on-line data be their database and simply completely regenerate their data each time. I prefer more old logs, so this approach does not work well for me. My region has about 150 mile radius and is pretty dense with about 10000 caches. I know plenty of other geocachers who have bigger "home regions". Most have much smaller regions and thus much smaller data needs. I would add that I rarely use all the data that I technically could - I find the amount of data that that Groundspeak is sending me to be fine. I doubt that changing those parameters would significantly impact the issue I raised when I started this thread.

 

When I (and presumably others) seek to update information on a large number of caches (thousands), I don't need most of the data that comes down the line. I do want a datum for every cache that I have selected, but I only need the identifying GC code, new logs, my found status, recent changes in status (archived, available, unavailable) and any recent changes in any other data fields. There are some features in the PQ set-up that already helps me significantly. The "Updated last 7 days" is extremely useful, but it has limitations which prevent me from using it exclusively. If along with that setting we could get a complete set of logs from the previous 7 days (without limiting to 5; delivering none if no new logs were generated) and an abbreviated entry for caches that were archived (without coordinates, descriptions, but include the logs from the last 7 days), then this would probably reduce the amount of data being moved by a substantial percentage.

Share this post


Link to post

Who will need 2500 caches per day?

 

I'll answer the question a bit differently than some others.

 

It doesn't matter if I need OR want 2500 caches, or whether I aggregate the info offline or just like to read it around the clock. The point is, that's the package that was sold to me, and that's what I expect to be "in the box". If I buy a box of a dozen donuts, I don't expect to find eight donuts inside when I get home because the donut machine couldn't make as many donuts as customers were requesting.

 

The current problems are a minor annoyance to me personally, and I'm confident that the Grounspeak crew is doing their best to get things corrected, after all, as a business, it's in their interest to satisfy their customer base.

 

In the long run, if there are no good solutions, I'd rather have the PQ benefits reduced (say to three a day instead of five) and have everything work smoothly, than to have something advertised and sold that just isn't realistic, regardless of the cause.

 

*edited to fix typo

Edited by gnbrotz

Share this post


Link to post

It appears to me that the majority of people keeping offline databases (GSAK) are only interested in getting updated information to keep their databases up to date.

 

The requirements for a PQ to do this are (as I see it):

 

1. All updates to caches and ALL additional logs Since the PQ was last run.

 

2. All disabled or archived caches Since the PQ was last run.

 

3. All new caches Since the PQ was last run.

 

Since the PQ list has the last run date stored now it would be an easy task to use this instead of the 7 day limit for updates. For the majority of cachers this PQ would only have to be run once a week at the most.

 

Anyone running it more often would only get what they require for that period only and not the last 7 days and could run single PQ on a wider area to eliminate the need for multiple (and often overlapping) PQs.

 

Inclusion of the archived caches would cut out a step in having to look at the last download date in the database to find ones which are not on the new download.

Share this post


Link to post

It seems to me that the PQ generation system is no longer able to keep up with the needs of the clients (us).

 

I have to agree with Hynr! I've kept track of the success rate of my PQ's for July (at the request of tech support) and I have experienced a 48% failure rate to date. I run 4 queries per day (with a total of 8 queries in my list) and have not had a single query run in 3 days! I believe I'm a fairly typical user and would like to run only 1 query once or twice a week that fetched only updates but that can't work under the current offering. To keep this a constructive criticism I'll list what I think needs to change to reduce the query traffic:

 

1) I need a polygon search (like GSAK offers) -- I am unable to select my area of interest with a circle! I currently have to create multiple overlapping queries to get what I want.

 

2) I need to be able to select my 2500 points/day limit in one query (I really only need about 1700 in my area but you get the idea). As it is now, I have to create multiple queries to capture my area because 500 pts doesn't cut it!

 

3) I need to be able to rely on my "updates only" query running without a problem (currently my biggest issue!) because when it fails I have to manually go back and find what's missing which generally means re-running all 8 queries to get my 1700 points. I have given up on running my update query and now request all 1700 points regularly (obviously adding to the problem!).

 

A few other suggestions:

 

get rid of the "free service" that allows non-paying members to download loc files. It's got to be a big hit on the infrastructure (I used to regularly download about 30 pages of queries before "signing-up"). Let non- members key their coords or pay-up for file downloads!

 

get rid of the "fixed" membership price - Infrastructure costs must rise with the user base and as the demand for better service rises... trying to keep infrastructure cost down by asking your users to "go easy" on the system just doesn't make sense!! Bottom line is that my 4 queries per day are well within what was advertised when I signed up for membership and the service I am receiving is well below that. Ironically, the poor service has forced me to "add" queries and thereby making the situation worse!

Share this post


Link to post

It seems to me that the PQ generation system is no longer able to keep up with the needs of the clients (us).

get rid of the "free service" that allows non-paying members to download loc files. It's got to be a big hit on the infrastructure (I used to regularly download about 30 pages of queries before "signing-up"). Let non- members key their coords or pay-up for file downloads!

 

get rid of the "fixed" membership price - Infrastructure costs must rise with the user base and as the demand for better service rises... trying to keep infrastructure cost down by asking your users to "go easy" on the system just doesn't make sense!!

As a new user, I don't think I would have even done anything with this site unless it had been free to some extent. After using the service free, I elected to upgrade to a premium membership.

 

It's important to keep this free as a lure for new users. If they get hooked, then they'll bite for the premium membership.

 

I've also got to believe that as this sport takes off, the sheer volume of new users will more than offsite any additional infrastructure costs. The store will also take in more money with the added volume.

 

The system ain't broke. It just needs some fine tuning.

Share this post


Link to post

 

The system ain't broke. It just needs some fine tuning.

 

I manage a database administration group for a large organization... I can assure you that a 48% failure rate on queries means it's BROKEN!!

 

This problem is way beyond the tuning stage.

Share this post


Link to post

Who will need 2500 caches per day?

 

I'll answer the question a bit differently than some others.

 

It doesn't matter if I need OR want 2500 caches, or whether I aggregate the info offline or just like to read it around the clock. The point is, that's the package that was sold to me, and that's what I expect to be "in the box". If I buy a box of a dozen donuts, I don't expect to find eight donuts inside when I get home because the donut machine couldn't make as many donuts as customers were requesting.

 

The current problems are a minor annoyance to me personally, and I'm confident that the Grounspeak crew is doing their best to get things corrected, after all, as a business, it's in their interest to satisfy their customer base.

 

In the long run, if there are no good solutions, I'd rather have the PQ benefits reduced (say to three a day instead of five) and have everything work smoothly, that to have something advertised and sold that just isn't realistic, regardless of the cause.

 

This is the best answer I've heard yet to the standard question, "Why do you need all these PQ's?"

 

Well said.

Share this post


Link to post

I decided to try to check on the size difference between getting a set of caches with the most recent 5 logs, versus the same caches with all logs from the last 7 days.

 

I used 4 PQs for the Sacramento CA area, all my “not found” caches; created a new GSAK database with them, then had GSAK export this set of 1686 cache records into a GPX file (including additional waypoints). Resulting gpx file size: 8,118k (1,763k compressed). (The original PQ files had a similar total size)

 

Then I used this information to update my main offline GSAK database (which is reasonably complete for the last few weeks), setting the userflag for the all caches being loaded. Then I copied these flagged records to a new database with all the logs I have and then purged the logs in this new database to eliminate all logs older than 7 days. I created a GPX export from the resulting set. Result: 4,229k (891k compressed).

 

So if the users who are running large PQ datasets are a big draw on the system resources, it seems to me that here is an opportunity to reduce their load by nearly 50%

 

Furthermore, if I do an export for this dataset without any logs, then that gpx file size is 3,892k (833k zipped). So the logs in the original PQ set amount to 4,226k while the logs for the last 7 days amount to 337k.

Share this post


Link to post

Pre-generated GPX Cache segments

...

 

I was puzzling over this and ran into the problem of determining which cache had not been found by the person requesting the query. The point was to reduce or eliminate JOINs and complicated queries.

 

One way perhaps is to create yet another field and stuff the cacher IDs space delimited in it. Then during the query add NOT REGEXP '[[:<:]]$cacherID[[:>:]]' which would return the records that didn't have the cacher's ID in it. With this you'd have a single table with no joins. Not good for everyday work, but this is for a one-time specific use. Tomorrow you start over with all new data.

Share this post


Link to post

CoyoteRed,

 

Just to help me understand what you are suggesting; correct me if I have it wrong: The idea is to preform as much gpx data as feasible and storing this in the Read-only database that the PQ generator uses (assuming that this is not already happening). I.e a buffering or caching scheme.

 

I'm not sure how useful it is to get into programming details because of the many ways that the concept can be implemented and because we know that Raine is quite adept at figuring these details out.

 

Thanks for contributing; keep the ideas coming. - Hynr

Share this post


Link to post

All I know is that I use the pocket queries maybe once or twice a week and only run them the one time when I need them. Heck here in Maine we have less than 1400 caches total for the WHOLE state. I sent out a PQ of All my finds on the 21st and still havent gotten it. So now even though it supposedly was generated and I never received it I have to wait another whole week to generate another one that I might not get. All I want is to update GSAK in my found database. I don't think asking for 700 plus caches is asking too much do you? Help Jeremy!!!! I appreciate all the things that the site provides but I think this pocket query generator isn't working quite the way I was hoping it would. Seems something ought to be done soon. Sorry if I posted this in the wrong thread.

Edited by haffy

Share this post


Link to post

So now even though it supposedly was generated and I never received it

 

That's most likely a problem with your e-mail/ISP and not a problem with the PQ system. Who do you use for your e-mail? AOL users are having trouble as AOL is apparantly throttling stuff again. You ISP could have flagged it, or it could be in a spam folder somehow. Do you have the geocaching e-mail ots in your "safe list"?

Share this post


Link to post

CoyoteRed,

 

Just to help me understand what you are suggesting; correct me if I have it wrong: The idea is to preform as much gpx data as feasible and storing this in the Read-only database that the PQ generator uses (assuming that this is not already happening). I.e a buffering or caching scheme.

 

I'm not sure how useful it is to get into programming details because of the many ways that the concept can be implemented and because we know that Raine is quite adept at figuring these details out.

 

Thanks for contributing; keep the ideas coming. - Hynr

 

One of the things mentioned a while back is the gathering of the logs is the hardest thing for the query to do. It could have been part of the decision to limit it to 5 logs. That and not realizing how useful some of the older logs could be--I mean, who knew?

 

The query has to use all sort of parameters to figure out which caches are in the search and filter based on more criteria. Most of that part is relatively painless. But, it also has to filter based logs. If the database is typical one and the logs are a many-to-one type of record (many logs to one cache) then they would be in a separate table. If you're filtering on "Caches I Haven't Found" then it has to look in a separate table to make sure you haven't found it yet.

 

But then, after all of the records are gathered for the caches you've specified, it has to go back and gather the logs for that cache. ...and it has to do it again for the next person's PQ. ...and again. ...and again. It's the exact same data query performed many times with no change. Even pre-compiling just the logs would help.

 

It would also be the perfect opportunity to change the 5 log limit to a higher one or the "all and only those changed in the last 7 days" model. The later would probably give a significant savings on bandwidth and intermediate database storage while eliminating the need for aggregators to get multiple copies of the same PQ every week.

 

A scheme that would be perfect for aggregators would be when the temporary database is created only the caches that are active, inactive and archived, and that have changed in the last 7 days, would be copied over. A field would be created called "ignore" where the cacheIDs of those who are ignoring and have found the cache are stored. Only those logs written in the last 7 days, but all of the logs written in the last 7 days are rounded up into a single data field.

 

One thing this would do is greatly reduce the data needed to produce the PQ. In my experience only about a quarter to a third of the caches I'm getting actually update the caches I already have in GSAK. Then take into account the number of long archived caches that don't have to be searched. The savings would be significant. In order for this to work for the aggregator, though, it must include recently archived caches. Something has got to tell GSAK the difference between an archived cache and one that simply hasn't changed.

Share this post


Link to post

I'm wondering if the resistance of increasing the 500 cache limit 5 times a day to a single 2500 PQ is memory issues.

 

If that is the case, then I'd break a single 2500 cache query into smaller ones, say 499. Heck, even smaller if the speed would increase. Let's say around 400 caches is the sweet spot for generating the maximum number of caches in the shortest time. You simply divide a larger query into 400 cache chunks and send those out in a single email. Most offline software could handle that without change. You can still have the 2500 a day limit.

 

An advantage to this for the site is the ability to find that sweet spot and operate within that without limiting the end user.

Share this post


Link to post

What about offering a suite of commonly used queries optimised for performance.

Personally I'm only really interested in nearest caches I haven't done, not owned by me, not archived. If a lot of people used these queries would it improve performance?

 

I haven't read the entire thread, so forgive me if this has already been suggested.

Share this post


Link to post

I'd be in favor of a recurrence of last year's purge. Require the users to come back to the site to reschedule them - at least for a period of a week or two. I'm sure there are people that set up PQs and never GET the data anymore.

Rather than just do that every once in a while, would it make sense instead to have all recurring PQs have an expiration date when they're first set up? So people could still get their daily or weekly PQs, but after, say, three months or so, they'd get a message telling them that they need to renew the PQ. (The PQ itself wouldn't be deleted, so they'd just have to go and re-check the desired days-of-the-week box(es).)

 

A lot of unused ones would drop out fairly regularly, I'd think, without it being too much of an annoyance for people to have to re-check a couple of boxes on their PQs four times a year.

Share this post


Link to post

Regions, impersonal queries, and incentives.

 

To eliminate complicated formula to search within a radius, create regions and update them on a set schedule.

 

Taking a few of the above ideas a step further, you could save a little bit of time searching on criteria that is not computed on the fly. Divide areas into regions. Large cities could be their own region while the rest of the state outside that city is another. This has been suggested before.

 

Invariably, this would mean some folks would have the exact same query. Identify these and run that query once and send that one query to them all.

 

So folks would know when regions change and not be caught off guard, make any changes to regions on a set schedule, say the first of every third month or whatever.

 

For some of these ideas to work personalization of the queries would have to be removed. This would mean no Found in the <sym> (or wherever it is as I forget at the moment) and own logs. Allow folks to opt out of that personalization to get more streamlined queries. Most of us can simply get a query of our founds to take care of all of that, anyway.

 

While we're at it, the scheduling could be tweaked to bring those who are using load saving options up to the front to encourage more to use them.

Share this post


Link to post

If running the PQs is the real issue, then why does copying an old query then running it immediately work?

 

And yes, none of mine scheduled for Sunday or today have run. But when I copied one of those and ran it, I got an immediate response. If the system is too clogged to work, then why does the copying work?

Share this post


Link to post

Any scheme that only allows 1 PQ a day is doomed to failure. Just take a look at the number of message where PQs are getting "lost" or blocked by ISP or email clients.

 

Telling people that they need to add geocaching to their email's white list, then cool your heel for 24 hours before trying again isn't going to fly.

Share this post


Link to post

Jeremys post in this thread indicates that there are two PQ queues. One for new never run PQ's and the other for older already run PQ's. It is the old PQ queue that is getting bogged down. In that thread Jeremy says he is going to allow us to "bump" our PQ's into the priority queue instead of creating a copy.

Edited by Team yGEOh

Share this post


Link to post

With my slow Internet connection, I don't want one 2500-cache PQ. It takes long enough to get the 500-cache PQs to finally arrive in my InBox :rolleyes:

Share this post


Link to post
Guest
This topic is now closed to further replies.
Sign in to follow this  
Followers 3

×
×
  • Create New...