PQ Generator Update

[+mambero]

Yesterday I promised to keep everyone up-to-date on our plans for the PQ Generator this week. First things first, all PQs were completed by 10am today. If you should have received a PQ and haven't please check to make sure it is still enable for today.

 

Here's the recap from yesterday:

Plans for the week: There are 4 main processes that need to happen to deliver your PQs. 1) Find the pending PQs for the day. 2) Determine which caches will fulfill each PQ. 3) Fetch those caches/logs/etc 4) Email them to you and post them to a website.

 

* It turns out #3 is where we spend most of our time, so we're going to fixing that this week. For you tech types, we'll be using a distributed cache.

* Another plan is to start the PQs to processing at 8pm PST. If we have time this week, we plan to prioritize processing by Latitude so those of you across the Pond and further will have your PQs during the day, and to allow our side of the Pond to still log their caches for the day.

We want to make sure the generator is stable this weekend, so processing by Longitude may get pushed, but it is the plan. Any thoughts on this?

* The database is accumulating a large amount of data. We're looking into how we can better tune it for better response times.

 

What we did yesterday:

* Raine, the geocaching.com know-all, found two indexes that went missing after our recent upgrade of the db server. When we added them back, we saw PQs throughput increase double. This will get us back to the same processing speed we saw before the database upgrade.

* We've updated the generator to allow us to start processing N hours before midnight (PST). We have a little more testing to do before we set it to run before midnight, but we're on track to start this tonight. See below for more details.

 

These two tasks should allow us to get your PQs to you faster than before. We're putting the distributed cache work on hold because it is complex enough in our environment that we cannot adequately test it before this weekend. Our goal is small, safe, tested changes.

 

We have the following tasks before us:

1) Make the generator reliable. The generator will occasionally stop running, we'll hear about it in the forums and someone will restart it. As of last week, we're now receiving e-mails every half hour with summary info to let us know the generator is still running. The challenge has been that most of us are asleep between midnight and 6am, so the generator could have been down for several hours before we find out. Currently, a couple of us are staying up/waking up frequently to check on these to make sure your PQs are getting out.

 

2) Alert the user when a PQ cannot be processed. Currently, if a PQ can't be processed, we disable it for the day. What can cause a PQ to be disabled:

a ) invalid e-mail accounts. This also included using more than one e-mail address separated by ";". As of yesterday, if you try to enter multiple e-mail addresses, only the first one is used.

b ) if the criteria for a routes base PQ is too complex for us to process in a timely fashion. We'll be looking at how we can improve our algorithms after we address the stability.

c ) if our system has a bug that prevents the PQ from being processed.

 

In cases a & b, we will send you an e-mail alerting you to the issue.

Case c should never happen, and at the same time, it's foolish to think any software is bug free. You are best served if we assume we could have a bug and deal with it. In this case, we will send you an e-mail letting you know the PQ is disable. It makes sense to temporarily disable the PQ so the generator can continue with the other PQs. What should happen though is that we receive an alert and resolve the issue and enable the PQ again for that day.

 

 

Sending PQs before midnight:

all time discussed below is in PST (-8 UTC)

We chose 8pm because that's when our database load starts to subside, and we made it configurable because we know the time may need to change based on feedback from you. To be sure users don't miss out on logs for caches after 8pm, we're only going to process PQs with positive longitude before midnight, and negative longitude after midnight. We've discussing processing all PQs based on timezone of the longitude, but that's a bigger task than we have time for at the moment. Jeremy is working on some of the infrastructure for that at the moment.

 

Q: how will I know if the e-mail I get on Friday is for my Friday PQ or Saturday PQ?

A: we're adding the day to the e-mail message: "Here are the Pocket Query search results in the formats you requested." will become "Here are the Pocket Query search results in the formats you requested for Saturday."

Q: How is my 5 per day limit affected?

A: You will still be allowed only 5 PQs to be processed per day, but we may start processing the 5 for tomorrow today, just not before 8pm. In reality, since we're only processing positive longitudes before midnight, no one should receive more than 5 per day.

 

Back to coding I go,

Drew

Edited March 10, 2010 by drewburlingame

[+Swamper68]

Sounds like a daunting task.

 

Good luck to all of you and I am sure that everyone is looking forward to a more reliable system.

[+Tequila]

Excellent update. Thank you and good luck.

[jholly]

Thanks for the very detailed and excellent update. I can see things are in good hands. Thanks for your efforts and happy coding.

[+beejay&esskay]

If we have time this week, we plan to prioritize processing by Latitude

Drew

Oh, the problems with cut-and-paste.

[+beejay&esskay]

To be sure users don't miss out on logs for caches after 8pm, we're only going to process PQs with positive longitude before midnight, and negative longitude after midnight.

Drew

How do you determine if the PQ is for positive longitudes? The center point? The results?

[Keystone]

Q: how will I know if the e-mail I get on Friday is for my Friday PQ or Saturday PQ?

A: we're adding the day to the e-mail message: "Here are the Pocket Query search results in the formats you requested." will become "Here are the Pocket Query search results in the formats you requested for Saturday."

So long as you're tinkering with the PQ email boilerplate, please change "formats" to "format" or delete "in the formats" entirely.

 

"Formats" is a carryover from long, long ago when you could receive both a GPX file and a MobiPocket file in the same message. Currently, you can only receive the query results in a single "format."

 

Does anyone seriously still request their queries in .LOC format??? That is soooo 2003.

 

Thank you for your report and for the hard work.

[+mambero]

Q: how will I know if the e-mail I get on Friday is for my Friday PQ or Saturday PQ?

A: we're adding the day to the e-mail message: "Here are the Pocket Query search results in the formats you requested." will become "Here are the Pocket Query search results in the formats you requested for Saturday."

So long as you're tinkering with the PQ email boilerplate, please change "formats" to "format" or delete "in the formats" entirely.

 

"Formats" is a carryover from long, long ago when you could receive both a GPX file and a MobiPocket file in the same message. Currently, you can only receive the query results in a single "format."

 

Does anyone seriously still request their queries in .LOC format??? That is soooo 2003.

 

Thank you for your report and for the hard work.

 

Thanks for pointing that out. I've updated the code, but it probably won't see production for tomorrows PQs. I just checked and we do still have users requesting the LOC format.

[+mambero]

To be sure users don't miss out on logs for caches after 8pm, we're only going to process PQs with positive longitude before midnight, and negative longitude after midnight.

Drew

How do you determine if the PQ is for positive longitudes? The center point? The results?

 

Good point. I assume we're talking CAAR and Bookmark PQs. My thoughts are that if we want to make sure a PQ includes all the info from the day before, then if a PQ contains any caches in the W.H., then the PQ will be run after midnight.

 

Also, I was looking at where the 0 longitude falls, and it seems like -27 would be a better cutoff point because it's West of Iceland and Africa.

 

Any thoughts on either of these?

[+mambero]

If we have time this week, we plan to prioritize processing by Latitude

Drew

Oh, the problems with cut-and-paste.

 

Thanks for pointing that out. It has been corrected. Hopefully the mistake can now die gracefully and I'll be able to hold my head high again soon.

[+beejay&esskay]

 

Also, I was looking at where the 0 longitude falls, and it seems like -27 would be a better cutoff point because it's West of Iceland and Africa.

 

Any thoughts on either of these?

Doesn't make any difference to me (being firmly in the middle of the USA) but perhaps someone in Iceland will weigh in.

 

(I was just curious in how you are addressing this major task.)

[+Chrysalides]

In some of the projects I've worked on, for semi mission critical servers, we have periodic tasks performing sanity checks. If you can have a very small PQ request firing off once every half hour or hour, with another process waiting for the email, and alerting an admin if the PQ is not received, you can have a very high system availability. Timing, etc. will need to be tweaked (e.g. to allow for delays during high volume hours, etc.)

 

If My Finds is a different process, then one for that too.

 

Should be more reliable than monitoring the forums for problems Just a small suggestion for your consideration, thanks.

[+mambero]

In some of the projects I've worked on, for semi mission critical servers, we have periodic tasks performing sanity checks. If you can have a very small PQ request firing off once every half hour or hour, with another process waiting for the email, and alerting an admin if the PQ is not received, you can have a very high system availability. Timing, etc. will need to be tweaked (e.g. to allow for delays during high volume hours, etc.)

 

If My Finds is a different process, then one for that too.

 

Should be more reliable than monitoring the forums for problems Just a small suggestion for your consideration, thanks.

 

We implemented a heartbeat mechanism like you've mentioned, but without the alarm. It's an e-mail that comes to a few of us every 10 minutes with some perf data. My goal is to move this to the db and create a couple of services that will alert us when we miss a heartbeat and attempt to restart the service. I think I'll call it the pacemaker.

 

My other goal, as you suggested, is to have the generator and pacemaker alert us when something is wrong so that we can fix it before you notice.

[+hallycat]

 

Does anyone seriously still request their queries in .LOC format??? That is soooo 2003.

 

Thank you for your report and for the hard work.

 

This from the guy whose company is still using IE5?

 

Thanks for the updates and for the team checking things out. If you'd rather you can just post your #'s and we'll only call if they go down

[+Lt.Ranger.Bob]

Maybe all is well but I have two PQ's that are setting in the cue that have not run yet today. They are my last two for today so I guess if they run after midnight I will loss two for tomorrow.

[+mambero]

Maybe all is well but I have two PQ's that are setting in the cue that have not run yet today. They are my last two for today so I guess if they run after midnight I will loss two for tomorrow.

 

The PQ queues are now empty. There was a back up as we migrated the new generator over. I apologize for the delay.

[+mambero]

We just updated the generator. It includes a couple of minor speed enhancements, but mostly focuses on stability. I believe we have fixed the issue that caused the generator to crash every weekend. But given the bug was difficult to reproduce, only time will tell.

 

We were not able to enable the feature to start processing at 8pm PST. It turns out there is a fix we'll need to make to the geocaching.com that will require a redeployment. It was painful for us to find this out as we were so close to having this ready for tonight. The fix will be small, but embedded deep enough that we can't hotfix it. We hope to give this a spin after the next release of the site.

 

Assuming all goes well this weekend, this will be our last update of the generator for a few weeks while we implement more improvements to increase the speed and stability.

[jholly]

Thanks for the info and update. And thank you for your dedication and efforts on fixing these problems. Jeremy will need to give you a boatload of comp time on this one, you've been putting in some very long days.

[+mambero]

Thanks for the info and update. And thank you for your dedication and efforts on fixing these problems. Jeremy will need to give you a boatload of comp time on this one, you've been putting in some very long days.

 

Thanks for noticiiiiiiiiiiiiiiiiiiiiiiiiiiiii ... <drools on keyboard an head hits desk>

[+Markwell]

We gee whiz. Quit spending so much time posting and get back to work! Twenty-four posts in four days from someone who's supposed to be chained to his desk in a dark hole?

 

 

 

Good work! And thanks for keeping us up to date.

[+Matthew 7:7 Too]

As I live just west of the Greenwich meridian, the move to prioritising by longitude sounds good to me! But will those living just to the east (in East Anglia as well as the Netherlands, Germany etc) now get their caches a day later than expected?

 

I'm also delighted to see larger PQs being delivered soon. Presumably generating a fewer number of larger PQs would reduce the server traffic - until users choose to get more frequent updates I suppose??

 

Thanks for your updates on proposed changes.

 

Chris

[+JBnW]

We gee whiz. Quit spending so much time posting and get back to work! Twenty-four posts in four days from someone who's supposed to be chained to his desk in a dark hole?

 

And for that matter, since when are Lackies allowed to claim PM Status?? There goes the neighborhood!!

 

All kidding aside, its good to see the Lackies take the Frog Master's offer to have more of a presence in the forums. I'm not savvy enough to completely understand how it is being done, but do appreciate the updates when the bugs get squished. Thanks!!

[+beejay&esskay]

As I live just west of the Greenwich meridian, the move to prioritising by longitude sounds good to me! But will those living just to the east (in East Anglia as well as the Netherlands, Germany etc) now get their caches a day later than expected?

Chris

As I understand it, the change will cause those to the east to have their weekly PQs run up to 4 hours earlier (8PM Pacific instead of the current midnight Pacific.) And the dividing line is probably just west of Iceland, so you are included with Europe...even if you don't want to be.

 

(Alleged headline in British paper years ago: Storm raging in English Channel. Europe cut off.)

[+ekhoc]

we may start processing the 5 for tomorrow today, just not before 8pm.

Out of curiosity, would the PQ for Saturday (EET), being run on Friday (PST) contain the date of Saturday or Friday?

(Currently the PQs begin to run at 10 AM local time, which I've found rather nice )

 

How about a more complex way, dividing the day into a couple of slices, and then either defaulting the startup time according to longitude, or letting the user to select the slice one would mostly prefer?

[+Markwell]

How about a more complex way, dividing the day into a couple of slices, and then either defaulting the startup time according to longitude...

That's actually what he had said was the ultimate goal in another thread.

 

The problem stands that the PQs should be generated when the servers have the least load. That's why they had been running at midnight Pacific time. If we start running the PQs at midnight local time based on the user's profile, some PQs for those in Berlin, the PQs would start to run at 7 pm in Seattle. It would be interesting to see if the PQ generator can handle the increased load during some peak hours like that...

[+Lil Devil]

Seems to me the best solution would be to split up the 360 degrees of longitude into 24 slices of 15 degrees each, and process one slice every hour, at what would be approximately midnight local time. I wouldn't bother with timezones, since their borders are quite haphazard and would only change things by an hour or two from the 15 degree slice method.

 

This should be based on each PQ's center point, and NOT based on the user's home timezone, because as has been pointed out, the user may be traveling on the other side of the planet. PQs based on a state or country should use an approximate center point of that entity for determining when to run. PQs based on multiple states or countries should just pick one. They're probably all close together anyways.

[+tozainamboku]

Seems to me the best solution would be to split up the 360 degrees of longitude into 24 slices of 15 degrees each, and process one slice every hour, at what would be approximately midnight local time. I wouldn't bother with timezones, since their borders are quite haphazard and would only change things by an hour or two from the 15 degree slice method.

 

This should be based on each PQ's center point, and NOT based on the user's home timezone, because as has been pointed out, the user may be traveling on the other side of the planet. PQs based on a state or country should use an approximate center point of that entity for determining when to run. PQs based on multiple states or countries should just pick one. They're probably all close together anyways.

Or you could just schedule based on the user's timezone and not have to do the processing to figure out which 15 degree slice the center point of the PQ is in or which 15 degree slice to use for CAARs, PQs based on state/country, or PQs based on bookmark list. Most the PQs from the same TZ will be in the same slice. The few times someone is traveling and running a PQ on the other side of the globe may not get the full efficiency of the cache but they probably a small enough percentage to not make a difference. And timezones zigzag so as to not split a single metropolitan area. With the 15 degree slice, if you get a city on the boundary you have some of the PQs run in one slice and the others run in the other slice. Wouldn't it make sense for all the PQs in one metropolitan area to run at the same time and get more efficiency from the cache?

 

The main issue I see with TZs is that many users haven't set theirs. If you were to run all these with the Pacific TZ that might cause a big load for that timezone. Perhaps unspecified timezones could be run randomly and people could be told if they want their PQs to arrive at a predictable time to set their timezone.

[+mambero]

Seems to me the best solution would be to split up the 360 degrees of longitude into 24 slices of 15 degrees each, and process one slice every hour, at what would be approximately midnight local time. I wouldn't bother with timezones, since their borders are quite haphazard and would only change things by an hour or two from the 15 degree slice method.

 

This should be based on each PQ's center point, and NOT based on the user's home timezone, because as has been pointed out, the user may be traveling on the other side of the planet. PQs based on a state or country should use an approximate center point of that entity for determining when to run. PQs based on multiple states or countries should just pick one. They're probably all close together anyways.

Or you could just schedule based on the user's timezone and not have to do the processing to figure out which 15 degree slice the center point of the PQ is in or which 15 degree slice to use for CAARs, PQs based on state/country, or PQs based on bookmark list. Most the PQs from the same TZ will be in the same slice. The few times someone is traveling and running a PQ on the other side of the globe may not get the full efficiency of the cache but they probably a small enough percentage to not make a difference. And timezones zigzag so as to not split a single metropolitan area. With the 15 degree slice, if you get a city on the boundary you have some of the PQs run in one slice and the others run in the other slice. Wouldn't it make sense for all the PQs in one metropolitan area to run at the same time and get more efficiency from the cache?

 

The main issue I see with TZs is that many users haven't set theirs. If you were to run all these with the Pacific TZ that might cause a big load for that timezone. Perhaps unspecified timezones could be run randomly and people could be told if they want their PQs to arrive at a predictable time to set their timezone.

 

Paul and I were talking about this point of processing by TZ/longitude, for that hypothetical point someday when we're spreading the load of processing all PQs across 24 hrs. What if we didn't choose midnight. What if we started processing a TZ at 3am (or a longitude by 3am of its last TZ). That should remove the issues around DST and those areas where PQs cross TZs, ensuring that PQs contain all the previous days info for those caches. My Finds, Bookmarks & in some cases CAAR PQs will not have the previous days info for their TZ, but this is already the case for some TZs. Starting at 3am actually means queuing the PQs for processing at that point. In TZs with large concentration of PQs, the PQs may not finish in the hour, causing the PQs in the next TZ would not start exactly at 3am. The generator would catch up when processing TZs with lower concentrations of PQs. Everyone would receive their PQs sooner than they do now. Any thoughts?

Edited March 12, 2010 by drewburlingame

[+beejay&esskay]

Paul and I were talking about this point of processing by TZ/latitude, for that hypothetical point someday when we're spreading the load of processing all PQs across 24 hrs. What if we didn't choose midnight. What if we started processing a TZ at 3am (or a latitude by 3am of its last TZ)

 

Everyone would receive their PQs sooner than they do now. Any thoughts?

(Old habits die hard, right? Maybe if you made a note on your hand.)

 

Sounds like a completely reasonable plan...as long as it isn't an unmanageable programming task.

 

I hope the page with my PQs listed will make it clear what is going on (time last run in TZ time?)

[+mambero]

Paul and I were talking about this point of processing by TZ/latitude, for that hypothetical point someday when we're spreading the load of processing all PQs across 24 hrs. What if we didn't choose midnight. What if we started processing a TZ at 3am (or a latitude by 3am of its last TZ)

 

Everyone would receive their PQs sooner than they do now. Any thoughts?

(Old habits die hard, right? Maybe if you made a note on your hand.)

 

Sounds like a completely reasonable plan...as long as it isn't an unmanageable programming task.

 

I hope the page with my PQs listed will make it clear what is going on (time last run in TZ time?)

 

Thanks for keeping me on my toes. I've corrected my post.

[+webscouter.]

Paul and I were talking about this point of processing by TZ/latitude, for that hypothetical point someday when we're spreading the load of processing all PQs across 24 hrs. What if we didn't choose midnight. What if we started processing a TZ at 3am (or a latitude by 3am of its last TZ)

 

Everyone would receive their PQs sooner than they do now. Any thoughts?

(Old habits die hard, right? Maybe if you made a note on your hand.)

 

Sounds like a completely reasonable plan...as long as it isn't an unmanageable programming task.

 

I hope the page with my PQs listed will make it clear what is going on (time last run in TZ time?)

 

He is thinking about travelling ALONG a latitude line. If you travelled along a longitude line you wouldn't ever change time zones. Right Drew?

[+Prime Suspect]

Q: how will I know if the e-mail I get on Friday is for my Friday PQ or Saturday PQ?

A: we're adding the day to the e-mail message: "Here are the Pocket Query search results in the formats you requested." will become "Here are the Pocket Query search results in the formats you requested for Saturday."

So long as you're tinkering with the PQ email boilerplate, please change "formats" to "format" or delete "in the formats" entirely.

 

"Formats" is a carryover from long, long ago when you could receive both a GPX file and a MobiPocket file in the same message. Currently, you can only receive the query results in a single "format."

 

Does anyone seriously still request their queries in .LOC format??? That is soooo 2003.

 

Thank you for your report and for the hard work.

You're right that it should be "format", but there are still multiple GPX formats (1.0 or 1.1) that can be selected from the user profile, depending on the GPX software you're using.

[+mambero]

Paul and I were talking about this point of processing by TZ/latitude, for that hypothetical point someday when we're spreading the load of processing all PQs across 24 hrs. What if we didn't choose midnight. What if we started processing a TZ at 3am (or a latitude by 3am of its last TZ)

 

Everyone would receive their PQs sooner than they do now. Any thoughts?

(Old habits die hard, right? Maybe if you made a note on your hand.)

 

Sounds like a completely reasonable plan...as long as it isn't an unmanageable programming task.

 

I hope the page with my PQs listed will make it clear what is going on (time last run in TZ time?)

 

He is thinking about travelling ALONG a latitude line. If you travelled along a longitude line you wouldn't ever change time zones. Right Drew?

 

Right!!!! I was right all along.

[+benh57]

 

He is thinking about travelling ALONG a latitude line. If you travelled along a longitude line you wouldn't ever change time zones. Right Drew?

 

Right!!!! I was right all along.

 

Well no, that's not right at all, since time zones are not straight north-south things. In fact, i can't find a any timezones that don't cross a lot of longitude lines.

 

Map:

http://www.worldtimezone.com/

[+lodgebarn]

I read through these posts and I must confess I can't understand exactly what is being proposed but having worked with time zones before (and don't forget summer time) I know it is a minefield. I reckon it is quite likely we willl get into a never ending list of problems. I live in in England (8 hours different) and often get up before 8:00 and run a couple of pq's which come back quickly on the previous day for me. Are you saying this might change and I won't know which day they will be run on? Are you trying to match everyone's calendars? What happens if I try and create a PQ far away for a holiday? It sound like there is the potential to get very confused. For me it would be very important to have a fixed time as a known reference point.

[+Chrysalides]

What if we started processing a TZ at 3am... Any thoughts?

What would be behavior be if I schedule a PQ after midnight (say, 12:15 a.m.) of my time zone to run on that day? Will it run immediately (if resources are available) or would it wait until 3 a.m.?

 

Actually I find the current system a little awkward for "run on demand" but I guess that is not really what's being discussed right now.

[+mambero]

I read through these posts and I must confess I can't understand exactly what is being proposed but having worked with time zones before (and don't forget summer time) I know it is a minefield. I reckon it is quite likely we willl get into a never ending list of problems. I live in in England (8 hours different) and often get up before 8:00 and run a couple of pq's which come back quickly on the previous day for me. Are you saying this might change and I won't know which day they will be run on? Are you trying to match everyone's calendars? What happens if I try and create a PQ far away for a holiday? It sound like there is the potential to get very confused. For me it would be very important to have a fixed time as a known reference point.

 

You're right, it is a minefield. If we go this route, we will most likely display the time in the cachers TZ along with the TZ. If the cacher doesn't list a TZ, we will either use the PQs TZ or PST. When a PQ runs, we'll be clear about which day it counts for.

[+mambero]

What if we started processing a TZ at 3am... Any thoughts?

What would be behavior be if I schedule a PQ after midnight (say, 12:15 a.m.) of my time zone to run on that day? Will it run immediately (if resources are available) or would it wait until 3 a.m.?

 

It would still be run on that day and would wait until 3 a.m. If you create a new PQ, it will be placed in the high priority queue immediately

Edited March 12, 2010 by drewburlingame

[jholly]

What if we started processing a TZ at 3am... Any thoughts?

What would be behavior be if I schedule a PQ after midnight (say, 12:15 a.m.) of my time zone to run on that day? Will it run immediately (if resources are available) or would it wait until 3 a.m.?

 

It would still be run on that day and would wait until 3 a.m. If you create a new PQ, it will be placed in the high priority queue immediately

 

I need to ask about this. I have a saved PQ, it has been dormant, i.e., not run for some period of time (days-months). Does that mean if I suddenly check to run today and my TZ window has past I have to wait a week? Or will I have to check it for tomorrow and wait for it to run tomorrow? I want that sucker now, I don't want to copy, run and then delete the copy. I think new has to be defined as a PQ that has the uncheck after run attribute set, regardless of how "old" the PQ is. I have some that have been around quite a while but are run at random times days to months apart.

[+Corfman Clan]

What if we started processing a TZ at 3am... Any thoughts?

What would be behavior be if I schedule a PQ after midnight (say, 12:15 a.m.) of my time zone to run on that day? Will it run immediately (if resources are available) or would it wait until 3 a.m.?

 

It would still be run on that day and would wait until 3 a.m. If you create a new PQ, it will be placed in the high priority queue immediately

I think you are asking for a lot of trouble and confusion if you start at some time other than midnight. Let's break this problem down to a few cases:

1. Schedules for Saturday before midnight on Friday. From what you state, this won't run until after 3 a.m. on Saturday
   
2. Schedules for Friday after midnight but before 3 a.m. on Saturday. Will this run right away, "On Demand" or will it run close to a week later?
   
3. Schedules for Saturday after midnight but before 3 a.m. on Saturday. From what you state, this won't run until 3 a.m., just like the first one. So does this means that there is no "On Demand" PQ from midnight to 3 a.m.? It really depends on the answer to item 2 doesn't it?
   
4. Schedules for Saturday after 3 a.m. on Saturday. PQ will run "On Demand"
   

If you start PQ processing by time zone, what would be the benefit of starting the PQ at 3 a.m. instead of midnight since distributing processing throughout the day is already taken care of?

 

I would also like clarification of what you mean by the "high priority queue". Doesn't this queue still follow the same "daily" rules for scheduling? Isn't it more of just a "jump to the front of the line" situation?

 

Thanks

[+Lil Devil]

I need to ask about this. I have a saved PQ, it has been dormant, i.e., not run for some period of time (days-months). Does that mean if I suddenly check to run today and my TZ window has past I have to wait a week? Or will I have to check it for tomorrow and wait for it to run tomorrow? I want that sucker now, I don't want to copy, run and then delete the copy. I think new has to be defined as a PQ that has the uncheck after run attribute set, regardless of how "old" the PQ is. I have some that have been around quite a while but are run at random times days to months apart.

That's not the way I've read the ideas at all. I think that the window would still be 24 hours long, it's just when it starts that is being discussed. Pre-scheduled PQs would be added the the queue to run at this new time, but others would still have 24 hours from that point to click the current day's checkbox to add it the the queue immediately.

 

I could be wrong, though.

[jholly]

I need to ask about this. I have a saved PQ, it has been dormant, i.e., not run for some period of time (days-months). Does that mean if I suddenly check to run today and my TZ window has past I have to wait a week? Or will I have to check it for tomorrow and wait for it to run tomorrow? I want that sucker now, I don't want to copy, run and then delete the copy. I think new has to be defined as a PQ that has the uncheck after run attribute set, regardless of how "old" the PQ is. I have some that have been around quite a while but are run at random times days to months apart.

That's not the way I've read the ideas at all. I think that the window would still be 24 hours long, it's just when it starts that is being discussed. Pre-scheduled PQs would be added the the queue to run at this new time, but others would still have 24 hours from that point to click the current day's checkbox to add it the the queue immediately.

 

I could be wrong, though.

 

I asked the question because the term "new" PQ, as in never run before is being used. I think the term previously unscheduled PQ might be a little less ambiguous since it would cover both cases. But thanks for your comments.

[+mambero]

What if we started processing a TZ at 3am... Any thoughts?

What would be behavior be if I schedule a PQ after midnight (say, 12:15 a.m.) of my time zone to run on that day? Will it run immediately (if resources are available) or would it wait until 3 a.m.?

 

It would still be run on that day and would wait until 3 a.m. If you create a new PQ, it will be placed in the high priority queue immediately

 

I need to ask about this. I have a saved PQ, it has been dormant, i.e., not run for some period of time (days-months). Does that mean if I suddenly check to run today and my TZ window has past I have to wait a week? Or will I have to check it for tomorrow and wait for it to run tomorrow? I want that sucker now, I don't want to copy, run and then delete the copy. I think new has to be defined as a PQ that has the uncheck after run attribute set, regardless of how "old" the PQ is. I have some that have been around quite a while but are run at random times days to months apart.

 

This is a great question. If we start processing at 3am, do we consider 12am or 3am to be the start of the 24 hour processing window for the day. It's certainly easier from a technical standpoint if we consider 3am the start of the window. Thus, a PQ generated at 2am would count towards the previous day. It would obviously be less confusing for the user if midnight was the cutoff. If we use midnight as a cutoff, it's more likely we'll miss logs posted later at night on the previous day, especially for caches across a TZ border that is two or three hours different. We can ease the confusion by showing the current processing day on the PQ page and the time the next day will start.

 

The second point about defining new PQs is another topic we'll be addressing. We plan to simplify that process as well, but I don't have enough info about our plans atm to give details.

[jholly]

<snip>

The second point about defining new PQs is another topic we'll be addressing. We plan to simplify that process as well, but I don't have enough info about our plans atm to give details.

As long as there is a way for me to have PQ's I have defined and saved but run when I want that will be fine. I just don't want to keep redefining the same PQ over and over again for each run.

[+mambero]

If you start PQ processing by time zone, what would be the benefit of starting the PQ at 3 a.m. instead of midnight since distributing processing throughout the day is already taken care of?

 

I would also like clarification of what you mean by the "high priority queue". Doesn't this queue still follow the same "daily" rules for scheduling? Isn't it more of just a "jump to the front of the line" situation?

 

Thanks

 

Starting at 3 a.m. increases the odds of including logs written later at night by cachers. This especially applies for PQs that span TZs, and especially when those TZs could be two or three hours different.

 

A "high priority queue" is one way to implement "jump to the front of the line". The difference being that new PQs can only cut in front of scheduled PQs. They still wait behind other previously queued new PQs.

[+Chrysalides]

This is a great question. If we start processing at 3am, do we consider 12am or 3am to be the start of the 24 hour processing window for the day. It's certainly easier from a technical standpoint if we consider 3am the start of the window.

Just want to point out that there might be DST issues. Both DST of the cacher, as well as your local time zone.

[jholly]

This is a great question. If we start processing at 3am, do we consider 12am or 3am to be the start of the 24 hour processing window for the day. It's certainly easier from a technical standpoint if we consider 3am the start of the window.

Just want to point out that there might be DST issues. Both DST of the cacher, as well as your local time zone.

 

Yeah, some areas don't have DST. Arizona comes to mind along with most of Australia, large parts of Asia, just about all of South America, and just about all of Africa.

[+beejay&esskay]

If we use midnight as a cutoff, it's more likely we'll miss logs posted later at night on the previous day, especially for caches across a TZ border that is two or three hours different.

Does anyone really care about this? For my weekly scheduled PQs it really isn't an issue. For specific caches that I intend to visit for a day trip, I will normally run an "instant" PQ in the morning to make sure nothing of significance has been posted since I last checked.

[+larryc43230]

If we use midnight as a cutoff, it's more likely we'll miss logs posted later at night on the previous day, especially for caches across a TZ border that is two or three hours different.

Does anyone really care about this? For my weekly scheduled PQs it really isn't an issue. For specific caches that I intend to visit for a day trip, I will normally run an "instant" PQ in the morning to make sure nothing of significance has been posted since I last checked.

For caches where I feel the need to get the very latest information, I simply go to that cache's Web page and download the GPX file directly. That way, I don't even have to wait for an e-mail.

 

--Larry