[BUG] PQ GPX file with bad <time> code

[+Mineral2]

Have there been any recent changes to the PQ generation system? Yesterday, while loading PQs into iCaching, I noticed that all of the updated caches were missing their Date Hidden field. At first I thought this might be a problem on their end. But after some fiddling around with the GPX files, I've come to determine that the issue is in fact with the files generated by the Pocket Query.

 

I found an old gpx file I had lying around. The time code for each waypoint looks something like this:

<time>2016-02-29T07:00:00.0000Z</time>

The time code in my most recent pocket queries looks like this:

<time>2016-02-29T00:00:00</time>

The issue here is not any of the numbers, but the Z at the end of the date-time stamp. It's missing in the pocket queries. Sure enough, if I add the Z to the recent PQ and import it into iCaching, the date hidden field is populated with the date it was hidden.

 

That Z is already included in the time stamp for logs, and overall GPX file time stamp. It's just missing from the time stamp directly under the <wpt> opening tag for each geocache.

[+The A-Team]

On ‎3‎/‎17‎/‎2019 at 10:33 AM, Mineral2 said:

The issue here is not any of the numbers, but the Z at the end of the date-time stamp.

Yes, according to the GPX 1.0 schema, the Z appears to be required, indicating that the time is in UTC. I would expect most parsers to choke on these time values if the Z is missing.

[+Mineral2]

Update: This issue also affects single cache GPX files when downloaded from the cache page. It was not an issue in the past as I have a GPX file from 2017 in which the Z is present at the tail of the date-time code.

I believe this will need to be addressed for those who import their PQs via direct GPX download or direct GPX import into database programs to be able to sort and search by date hidden. API download does not seem to be affected at this time.

[+barefootjeff]

I wonder if this is related to this recent "fix" to get event times showing correctly. From the release notes thread:

 

Quote

We have released a fix for GPX files from PQs:

• GPX files downloaded from PQs no longer show the incorrect time (and date) for Event Caches. Instead they are showing the time entered by the CO without a specified time zone.

 

[+Mineral2]

Can anyone here, who uses GSAK, confirm that this is a problem for them? GC Droid manages to import the same PQ files and still show the date hidden. And it's not importing via the API as I did the whole thing on airplane mode.

[+HHL]

3 minutes ago, Mineral2 said:

Can anyone here, who uses GSAK, confirm that this is a problem for them?

Confirmed.

[+The A-Team]

1 hour ago, barefootjeff said:

I wonder if this is related to this recent "fix" to get event times showing correctly. From the release notes thread:

 

Quote

We have released a fix for GPX files from PQs:

• GPX files downloaded from PQs no longer show the incorrect time (and date) for Event Caches. Instead they are showing the time entered by the CO without a specified time zone.

 

 You may be right.

 

Their fix doesn't really appear to fix anything. The GPX schema explicitly says that the time is in UTC. Unless the event is being held in the UTC time zone, this is just replacing one incorrect time with another incorrect time and using incorrect syntax. The local time of the event needs to be converted to UTC, and the syntax fixed.

 

I expect that GSAK and GCDroid are catching this issue and gracefully handling it, but there's still an underlying issue.

[+Mineral2]

I'm not sure that event time really matters for the purpose of a GPX file. In the past, the GPX file used the date that you provide during cache creation and editing as the hidden date, with a generic time set with no real regard to time zone. All databases show is the date. The time is useful for for calendar files, which are created separately from the GPX file, and if I'm not mistaken, when no time zone is denoted, the event will import according to your calendar's default time zone.  Maybe Groundspeak has a system to look up the time zone according to date and coordinates and the times get written to their system as the correct UTC. I haven't looked at any event files to play around with that. All I'm pointing out here is that the change to the date-time stamp in the GPX file affects the way cache management software is reading the data.

Addendum: It appears that event times are encoded with a time zone, but maybe according to the time zone at which the event listing was first created? Events, local or afar, seem to be an hour off when I try to add to my Google Calendar, thus not accounting for the recent shift into daylight savings time. This is a separate issue that requires its own thread.

Edited March 18, 2019 by Mineral2

[+The A-Team]

16 minutes ago, Mineral2 said:

I'm not sure that event time really matters for the purpose of a GPX file.

I'm not sure that I do either, but HQ seems to think so and their fix quoted earlier states that they made the change to fix that. However, as far as I can tell, the "issue" wasn't fixed and the change actually introduced a new issue.

 

As an example, GC84EHK is scheduled to occur at 6pm Pacific Daylight Time on March 21.

Converted to UTC, this would be 1am March 22.

However, a GPX file for this event shows a <time> of "2019-03-21T18:00:00". Ignoring the missing 'Z', this UTC time converts to a local time of 11am March 21, which is wrong. Folks would show up 7 hours early based on the time in the GPX file.

 

This all seems to come back to a misunderstanding of the times in GPX files by HQ. The confusion may come from the use of ISO 8601 time/date standards for formatting the date and time. While that standard states:

Quote

Without any further additions, a date and time as written above is assumed to be in some local time zone. In order to indicate that a time is measured in Universal Time (UTC), you can append a capital letter Z to a time...

...the GPX specification overrides this by explicitly stating that all times are in UTC. Any other use of the time attributes, like dropping the 'Z' and trying to call the time "local", would be outright wrong and will ultimately lead to issues.

 

@robertlipe: As the creator of GPSBabel, I expect that you know a lot about this stuff. Is my interpretation above correct?

[+Mineral2]

Garmin GPS devices don't even deal with the date. It's not displayed anywhere. But, since the system appends the date and time to the description of events, that information is baked into the text of the event and doesn't even need to deal with time zones. I think the GPX file ought to have been left alone to simply reflect the input hide date of the event (even if converted to an incorrect UTC time) so that databases can show the hide/event date, while the time will be present in the description according to the user input time, which is probably written directly into Groundspeak's own database. Again, the time zone is only important since events can be linked with calendar software, which is independent of GPX files.

Edited March 18, 2019 by Mineral2

[robertlipe]

In case need to re-up my credentials, I was also a creator of GPX. We knew that saying "ISO 8601 except..." would be a mixed bag, but I maintain we made the right decision that a persistent object in GPX would always be in UTC just because it made developers think about it.  It makes ME think about it because my time zone is different than where the automated tests run, so any concept of local time leaking in is a failure for us.

There was some squabbling during GPX 1.0 about GPS time vs. UTC but almost nobody really cared about the 18 second (whatever) offset. It's very likely that none of the validators actually test this GPX twist on ISO 8601. (I'd hope that Groundspeak never ships GPX that doesn't validate...)  In formats other than GPX, GPSBabel sees a lot of the formats by smaller companies that accidentally use local time for where the developer is and then it's chaos when someone tries to share tracks if they happen to be made far away (E-W) from the developer. I think that the National Geo state editions did this, as trivia. Each state was considered to be in one time zone, a sure hint that it was written in a small state.

I just checked the GPSBabel reference/ directory which contains 188317 instances of <time> tags, collected over about 18 years of real world apps, most of which are actually legal GPX, but some of which aren't - often to regression our test of handling colloquial (or outright broken) GPX. The four lines that don't end in Z are GPX that came from geocaching.com probably 15 years ago...and I'm pretty sure we were testing Groundspeak: tags, not  core GPX.  So forcing Zulu time isn't just some quirky thing in the GPX spec, real-world programs and GPS do that.

Giving some thought to handling each of the world's 39 time zones, the issue of whether a placer or a placement is at a location that observes DST, which is subject to local laws for start and end dates and can change at very little time, I'll suggest it unlikely that MIneral's guess that they have voodoo that figures this out is unlikely. (If I'm in summer time now and place a event in winter, the UTC offset will change. If the date DST ends changes - as it did in the US in 2006 - what do you do with events on the books?) 

So what they're doing is definitely wrong for about 23/24ths of the world. Not having a Z is local time and it has to be in Zulu/GMT/UTC.  With a lot of thought, it could possibly store event-local time as UTC. If I'm from FL in a layover in TX downloading a PQ for an event I attend in CA, what is "local time"? Do I trust the various calendar programs that'd make our watches go Ding to get that all right? Absolutely not. :-)  

I'd have to think harder about what IS the right thing to do here. Someone would have to think really hard about what happens at an event scheduled for 2a.m. (it happens!) on the weekend of a time change and lots of other stuff.  It probably involves new Groundspeak:tags and teaching software that wants to know about event datetimes about event datetimes as distinct concepts from placement datetimes. Then it could just store and read local time and try to never ever use that for math or a comparison against anything in UTC, which should be everything. Since it's more like a DateTimeSpan than a DateTime, you need another member/field to hold the duration of the event or the end DateTime. Part of the trick will be to plumb this out to correct DateTime pickers in all the editors and viewers that know how to deal with this local time and not changing the concept of an existing tag. For example, overusing PlacementDate as we do now leads a false runup in the event of a mega when the reviwers have been doing the work over time time.

BTW, the individual cache page GPX and the PQ GPX at least used to often creep slightly out of sync with each other. Seems like it should be the same code, but I've seen enough differences over the years to suspect that something was written by copy and paste that probably shouldn't have been. :-)

P.S. This isn't a forum I read so I may be missing context.  Since joining the land of the disabled, I'm mostly an events-only, retired geocacher. I do realize that time handling in general has been problematic for this site for a long time and event time is a welcome concept. Still, I have opinions about everything, qualified or not. :-) 

[+Mineral2]

7 hours ago, robertlipe said:

I'll suggest it unlikely that MIneral's guess that they have voodoo that figures this out is unlikely.

I'm not entirely sure what you're alluding to here.

GPS devices have a time zone map, and will calculate local time based on location and date. I know this because when I cross state lines from Idaho into Montana, my GPS's clock loses an hour. And when I turn it off on the first Saturday in March and turn it back on the next Sunday, the time has gained an hour all on its own. So it's reasonable that Groundspeak either has a time zone map on file, or uses a remote service to reference coordinates with time zones.

What I don't know is whether the UTC offset is baked into the calendar files, and whether that offset is reference based on the time of file creation or the event date. I would hope it's based on the latter. But then why would all of the times be behind an hour? Either DST isn't being taken into account, or something else is going on. Unfortunately, I didn't catch this before DST went into effect to see if this was a year-round issue.

As I said before, I don't care about time in the GPX file, just that the date is in the correct format for for database programs to read it.

[+Corfman Clan]

1 hour ago, Mineral2 said:

I'm not entirely sure what you're alluding to here.

GPS devices have a time zone map, and will calculate local time based on location and date. I know this because when I cross state lines from Idaho into Montana, my GPS's clock loses an hour. And when I turn it off on the first Saturday in March and turn it back on the next Sunday, the time has gained an hour all on its own. So it's reasonable that Groundspeak either has a time zone map on file, or uses a remote service to reference coordinates with time zones.

What I don't know is whether the UTC offset is baked into the calendar files, and whether that offset is reference based on the time of file creation or the event date. I would hope it's based on the latter. But then why would all of the times be behind an hour? Either DST isn't being taken into account, or something else is going on. Unfortunately, I didn't catch this before DST went into effect to see if this was a year-round issue.

As I said before, I don't care about time in the GPX file, just that the date is in the correct format for for database programs to read it.

Time zone and UTC offset are two completely different concepts. You're intertwining them here, so be careful with that. A UTC offset is great for specifying a particular instance in time, not so much for specifying when 4:00PM is at location X in December and again in July, that's when time zones are necessary. I think there is a lot of confusion or short cutting on how this stuff is actually meant to be used that causes problems.

[+Mineral2]

1 minute ago, Corfman Clan said:

Time zone and UTC offset are two completely different concepts. You're intertwining them here, so be careful with that. A UTC offset is great for specifying a particular instance in time, not so much for specifying when 4:00PM is at location X in December and again in July, that's when time zones are necessary. I think there is a lot of confusion or short cutting on how this stuff is actually meant to be used that causes problems.

Doesn't the time zone define the UTC offset for a particular location?

[+HHL]

6 minutes ago, Mineral2 said:

Doesn't the time zone define the UTC offset for a particular location?

It does. ?

[+The A-Team]

3 hours ago, Mineral2 said:

Doesn't the time zone define the UTC offset for a particular location?

Yes, but a time zone will also have extra rules attached like whether Daylight Saving is used and when that changes. A UTC offset is static.

[+The A-Team]

13 hours ago, robertlipe said:

In case need to re-up my credentials, I was also a creator of GPX. We knew that saying "ISO 8601 except..." would be a mixed bag, but I maintain we made the right decision that a persistent object in GPX would always be in UTC just because it made developers think about it.  It makes ME think about it because my time zone is different than where the automated tests run, so any concept of local time leaking in is a failure for us.

<snip>

Thanks for your input. I know this isn't a sub-forum you typically frequent, but I figured you'd be the best person to speak authoritatively on a topic like this. Based on your post, it seems like dropping the 'Z' and redefining the <time> as local simply isn't done by anyone, so it will be interesting to see what happens here.

[+Corfman Clan]

5 hours ago, Mineral2 said:

Doesn't the time zone define the UTC offset for a particular location?

It defines the UTC offset for a particular location at a particular instance in time. The UTC offset for a location may vary throughout the year. The time zone takes all that into account. UTC offset by itself is really only good for a specific instance in time.

Edited March 19, 2019 by Corfman Clan