ClayJar
-
Posts
962 -
Joined
-
Last visited
Posts posted by ClayJar
-
-
I've about finished the updates for Watcher 0.1.31, and it's sitting here nearly ready to be released (pending the fixes to the bug reports from my testing crew). The only other thing I need is to know what the name of the "exported" timestamp node will be.
(.31 has some rather significant improvements and at least one major bug fix, so once it's debugged, I don't want to hang on to it longer than necessary. If I must, I'll go in and disable all the merge logic and release another poorly-merging version, but I'd like to have the smart merge code in there before I start working on exports.)
-
Okay, the GPX (as I received it, at least) had:
quote:
I assume that was a relic of a bit getting shifted between you an me, so I changed it back toquote:
If that had been the problem, you would've gotten an error message, though, so I don't think that happened on its way to you. (You *do* get the GPX files sent to you zipped, right?) Other than that one character getting corrupted by the e-mail, the GPX loaded fine for me in Watcher 0.1.30. I thought I had a problem when I opened it and had 0/99 showing, but I turned off my filters and realized they were all virtuals, which I filter out. Without the filters on, I had 99/99 showing.
-
quote:
If you don't mind, can you e-mail me this GPX? I'll look at it and see why Watcher didn't handle it correctly. I don't know of any outstanding GPX-reading/parsing bugs in 0.1.30, so if there is one in there to find, I'd be grateful if you'd lend me the file so I can find the problem.
Originally posted by Frank and Peggy:FIle 3 had 99 waypoints. EasyGPS reported 38 while Watcher reported 0. Only Spinner found all of them. GPSbabel found 99 on the raw file while it didn't find any from the Spun file.
GPS@C___J__.com (ClayJar, that is.)
-
Well, I've got the code written and waiting. I just need to tell Watcher where to put the exported timestamps and the smart merging will be complete. There's even an optional "Exported" column so you can sort your cache list and see what needs to be checked or re-queried.
-
I just checked one of my PQs from today. Much nicer (even though Watcher doesn't use the field
). -
I replied without looking at my code, which was stupid. (You #Geocache members who kept me up playing Wordox are to blame! Oh, wait, that's me too.)
//gpx/wpt/time *IS* the hidden date, which is why we need the field.
The end.
[This message was edited by ClayJar on February 07, 2003 at 01:11 PM.]
-
quote:
Originally posted by Jeremy (Admin):Sorry I didn't get to this topic sooner. We're currently in the throes of developing the next site for Geocaching, which involves a lot of work. I had no idea how much asp was created for this site until I had to start converting it to dot net!
Hehe... I feel your pain (a teeny, tiny bit). I once converted a *much* smaller application and ended up enjoying the process for a rather-too-long time. I hope you've got cleaner code than I had.
quote:
The reason we need a place to put a timestamp in the caches (as opposed to just at the top of the file) is that Watcher and other programs (GPSBabel, I think), can merge Pocket Query GPX files. When they do that, you can no longer use the single timestamp on the top of the newly created file, since it has waypoints from differing export times. (See this post for the full *why*, if this summary is too short.)
Anyway, I would say that yes, we could support this. I'm just trying to figure out why the timestamp doesn't work for you at the top of the file. As I understand it you just want the additional field for your own purposes so you don't have to create one yourself? We could certainly do that.While Watcher and everyone else *could* use their own namespace to hold the data, it *is* Pocket Query data. It seemed far more logical for it to be in the Groundspeak namespace, even though the per-waypoint exported timestamp does not need to be included in the Pocket Queries -- all the Pocket Query-generated waypoints have one common export timestamp.
-
I really hate to be a bother, and this is getting more and more uncomfortable each time I mention it again, but is there any possibility of hearing *anything* about this?
I don't even need a complete answer; an "ask me next month" would suffice. A yes or no to the idea would be ideal, but if it can't be given, even a "maybe" would be welcome. I just don't want to keep asking about it if you (Jeremy or Elias) are reading this and not answering for some reason.
(I would simply drop the idea entirely, except for the fact that it is a matter of critical importance; stale data *must* not be tolerated when it can be prevented, so one way or another, this node or one like it *must* exist.)
-
I'll be there as early Friday as I can, but certainly in time to set up a tent and camp. Just one question... is this going to be a hydrocache like last time?
-
Well, another little addition while I wait (and a couple little bugs fixed). You can now filter by the eight compass directions (no, there's no ENE, you'll have to settle for E and NE separately). A bug in the name sort and a bug in the directory remembering were also fixed.
Get it at the usual site.
-
Well, since I'm still waiting for a reply (*any* reply) to the GPX thread (and so, I can't add the critically important merge code), I did a quick update in the meantime. (Admin people, are you there?.)
Watcher 0.1.29 features a new "waypoint name" column for all you who mangle your GPX files' waypoint name. Also, you can now select multiple files to merge in the open/merge dialogs. Finally, the distance filter now uses kilometers if you're using kilometers as your distance units.
Get it at the usual waiting area.
-
quote:
Originally posted by Sissy-n-CR: No, I meant so _we_ could put it in and keep a record of our caches. The list could then be sorted on date/time instead of just date. If we had to, we could just make up a time...Hehe. All the timestamps in the GPX (i.e. the so-called "dates") are date and time, so if the application used them, you could indeed set the specific times by hand.
quote:
Wait a minute! I just thought of something. Are the log entry IDs unique? If so, then all you have to do is sort by log entry ID. That'd make it must easier.The log IDs are indeed unique. Assuming people log caches in the order they did them, it would indeed be possible to sort by date and sub-sort by log ID.
quote:
So, then how come finds are all out of order even though I put them in, in the order I found them? Seems like a simple solution.Geocaching.com doesn't sub-sort by log ID. However, since log ID is a value based on the order the log was added to the Geocaching.com database and not necessarily the order the cache was found/not-found/whatevered, doing a sub-sort by log ID would not necessarily help. It's not unfair to consider that they might not be sub-sorting by log ID because it will not necessarily give any better results. (Of course, I always log in order, so for me it would work.) If they ever add the "This Is $USERNAME's Life (Now In GPX-Vision)", I'd probably do a log ID sub-sort, but as for now, there's nothing to do about it...
**Unless** they make the "found" element like:
<Groundspeak:found id="$LOG_ID_NUMBER">$TIMESTAMP</Groundspeak:found>
(That would give enough information to sort the caches by the most logical sort.) -
quote:
I wouldn't mind that, either. However, since that information is not in the Geocaching.com database, and since (as far as I know) Jeremy is neither psychic nor a time-traveller (although either would explain a lot)... in the immortal words of HAL, "I'm afraid I can't do that, Dave."
Originally posted by Sissy-n-CR:I wouldn't mind a found date AND time tag (or whatever you call it). You could make your logs in order of actually finding them.
quote:
Since there is obviously code there already to find if the cache is found, it might be a relatively simple problem to add the found date. If it can be done easily and without a significant performance hit, it would be a valuable addition. It's up to Jeremy, et al, to decide whether it can or should be done, but if they decide to do it, I'll support it.
Also, what would be useful is the inclusion of your log. That way, if your log is not one of the latest 5 logs you can still get it. You'd be able to create a list of the caches you've found, your log entries, and links to the caches AND your log entries on GC.com.As for getting a GPX of everything you've ever done, that would seem very likely to be something that the Pocket Query generator is not currently built to handle. Perhaps at some point they'll add a special "This Is $USERNAME's Life (Now In GPX-Vision!)" feature, which would be immensely cool, but that's a topic for another thread.
-
quote:
I don't think that reasoning is valid. The Pocket Query GPX files are already "tied very closely to a single individual." This is why Watcher asks whether to add caches the GPX says are found to your found list. It would be just as easy to ignore the element *and* any hypothetical element as it is to simply ignore the element, which is the current state of things.
Originally posted by fizzymagic:It's a VERY major change
Because now the GPX file is no longer a generic cache file, but tied very closely to a single individual.
I'm not sure if Geocaching.com wants to add the found date, but I can see no reason not to, other than any additional resources it may use while generating the Pocket Query files. If they decide to add the field, I will support it in Watcher.
-
One of the main uses of Watcher is to collect and then filter the caches in your GPX files so that you can then use EasyGPS, GPSBabel, or any of the HTML, PalmDoc, or other format converters. I've been asked to start working on integrating some of these with Watcher, to make it simple to go to whatever format you want.
It would be nice to be able to tag the cache pages with the timestamp the data was exported, but there is no place in the Groundspeak XSD to preserve that data. I don't want to make another namespace just for Watcher, as that would require *everyone* to support it if we want to use the information cross-application.
Can someone up high take a moment to give us a PASS/FAIL on this thread so we'll know whether we need to all get together to hash out a new XSD or whether one line can be added to the Groundspeak cache XSD to allow us all to be on the same page. (As was said earlier, the Pocket Queries don't need to include the element; it just needs to be listed so we can all follow the standard.)
Well, anyway, I hope someone hears this thread soon. I really don't want to start working on exporting data to other formats until I can note in those exports when each cache entry was generated by geocaching.com, but I'm going to have to start one thing or the other soon... I can't sit still this long.
-
(FYI, MSXML 3.0 came with IE6, but it can be installed with IE5 from the link above, which is for MSXML 3.0 SP2. Watcher currently uses MSXML 2.0, but I may switch to 3.0 if I ever find a reason.
) -
quote:
Cool. I need to pay more attention to the rest of the world; I'm starting to miss really nice features.
Originally posted by robertlipe:It won't mess up GPSBabel, but GPSBabel does grope the Groundspeak extensions for things like Smart Icons and when populating or reading GeocachingDB files to get diff/terr.
(Now all I need to do is get around to the Watcher -> GPSBabel -> **EVERYTHING** layer. I *will* get around to it one of these days... unfortunately, my next free evening is a week from tonight.
...) -
quote:
Actually, we don't know. At some point in the future, Geocaching.com might start adding some "web services" as they're called. If, at that point, they provided an API for checking the status of a cache or list of caches, I could add that to Watcher.
Originally posted by Sissy-n-CR:So, what about that archived waypoint? How do we know it just hasn't been just outside our query?
It might end up being something like "If this cache was exported over X days ago, check it's status on Geocaching.com," and Geocaching.com's servers might reply with a message that says something like "Not Archived, Not Disabled, Last Updated 23 Feb 2004, (Found, Not found, Other, Found, Found)" It doesn't really matter at this point.
For now, though, for us to be able to say that a given cache's information *was* correct at a given point only requires a one-line change to the schema definition. At least we'll be able to say when a cache was last exported, even if there is currently no way of knowing why a cache was not included.
-
quote:
No, it won't mess up Watcher or GPSBabel (GPSBabel doesn't do anything with the Groundspeak extensions other than copy the data, IIRC), but of course, it also won't do anything until the code is updated. I know Watcher's code will support intelligent merging within a day or so of the node being added.
Originally posted by Markwell:GS adding the field is not particularly difficult {dafaultvalue=now()} The problem I would think would be all of the development that has gone into the schema that GS currently provides. Does this goof up GPSBabel or even watcher? I don't know enough about the programming to know.
Also, there's no need to have a default value. The node doesn't need to be in any GPX files created by Pocket Queries, since their cache listings are all of the same vintage. Only when GPX files are merged does the information need to be preserved on a per-cache basis. (So it doesn't need to take any additional resources or file size for Groundspeak to "support" the addition; it's all on the application developer's side, and I'll be glad to do the work.)
-
quote:
Hypothetical situation:I am not getting queries on a weekly basis; I'm collecting hydrocaches in all 50 states and several area countries.I just decided to go hydrocaching in Alabama next weekend, so now I load up my "hydros of 50 states' and more" merged GPX.I can't remember when I got the last Alabama GPX, since I've been collecting caches from all over and I didn't have hard drive space to keep them all.Due to the weather, nobody has logged any of the Alabama hydrocaches since October.Question: Is my information on the Alabama hydrocaches current enough to risk the weekend on the assumption that the caches are still there?
Originally posted by Sissy-n-CR:File "freshness" is also good, but while we are getting queries on a weekly basis, we'll always know we have at least that fresh of a file. Generally.
Really, who cares how old the data is as long as we know it's the latest version? (as far as merging and whatnot.)
*THAT* is why we need to know when the data was exported. You *cannot* answer that question using only last-updated and logs.
-
quote:
Originally posted by Sissy-n-CR:Question: would LASTMODIFIED be better and actually give more information?
Here's my thinking, EXPORTED only gives the date and time of the waypoint exported, but LASTMODIFIED would not only let you determine the newest version of the cache page, but also _how long ago it was modified._ This along with the latest logs could add to how up-to-date the cache is. So a cache that doesn't have a log for a couple of months and the owner has not updated the cache for a year is less likely to be there than one the owner has modified just last week even though the last log was a couple of months ago.
While it may be nice, useful, or even divine to know when a cache page was last modified, that is completely irrelevant to this thread. Why?
The "/gpx/time" node tells you when the GPX was compiled. That is analogous to a "Best if used by:" date on a honey bun. You have no idea when the honey bun was created, but you know that as of "JAN 28 02", it was in nominally digestible condition. If you're a day or two past that, you know that it's probably still fine to consume, but you should check for blue-green spots first. If you're three months over, you know that you should use the honey bun for your next death-by-microwaves experiment, but don't trust it to be edible.
Now, if you know that the honey bun was created three years ago, and it has remained in its hermetically sealed package ever since, and it just expired a week ago, you'll probably say that it's likely still so-called edible. In the same way, if you know that the cache was last updated three months ago, and it's got a decent smattering of logs up until the cache details were exported last week, you'll probably figure it should still be in huntable condition. Having both "last updated" and "exported" could provide useful information.
BUT! If you only have last updated, that's like knowing when your honey bun was created. As you can imagine, that does not answer the question of "is this honey bun consumable?" If it was created yesterday, you're pretty sure it's edible, but if it was created three months ago, did they use enough preservatives to keep it intact that long? Maybe it's "best if used by: JAN 31 04", but maybe it's been expired for two months and thirteen days. How can you tell?
I've got a cache that was last updated well over a year ago. It hasn't had a find in a very long time. If you had "last updated" fields in a merged GPX file with that cache in it, you'd see that there's no new information since 9 or 10 months ago, but what you wouldn't know is how long ago that query was run. You could've run the query 7 months ago, and the cache could've been archived for 6 months, or you could've run the query yesterday with the cache still active.
You need to know when the information was canonical. When did geocaching.com show the cache as it appears in your GPX file? It doesn't matter how long the cache page has looked like that; it only matters that you know that on January 28, 2003 at 8:35am the page *did* look like that.
It would be nice to know last-updated, but as you can see, it doesn't give you the information you need to properly merge multiple GPX files.
-
quote:
Originally posted by Sissy-n-CR:I just looked at something. I took a recently received query and saved it from email. The time stamp was the current time--10:33AM. However, when I extracted it was earlier in the day, 6:50AM.
Wouldn't that be the time it was generated? A kludge, but maybe you could look at file date if it doesn't have an embedded time stamp that you create. Better than nothing.
Of course, Groundspeak embedding the time stamp would be best.
The timestamp of the file in the filesystem is completely arbitrary, and I neither based nor will base anything on the filesystem timestamp. There is a single timestamp in the GPX at "/gpx/time" that contains the time at which that particular GPX file was compiled by the Pocket Query generator. That timestamp can be used for merging GPX files that come from Pocket Queries.
The problem at hand is simply this: If you merge two GPX files, there is not currently a place in the namespace to specify a timestamp for each waypoint in the merged file. Since the merged GPX is potentially made up of components from multiple Pocket Queries, the compilation time is not defined. If we were to merge another Pocket Query with this merged GPX, we could not reliably determine which waypoints are canonical.
Take, for example, three Pocket Queries:
PQ1: January 1, 2003 (All Louisiana traditional caches) PQ2: January 3, 2003 (All caches within 50 miles of me -- some are in Mississippi) PQ3: January 5, 2003 (All Louisiana non-traditional caches)So, I get all three PQs, and I decide that I want to merge PQ1 and PQ3 so that I'll have a merged Pocket Query containing all the caches in Louisiana. I do that, and now I end up with PQ(1+3), a single GPX file with all the Lousiana caches in it.
Well, now I decide that I'm going to take a cache trip up from Baton Rouge all the way to lower Missippi, so I need to merge PQ2 into my GPX so that I'll have it in my PocketPC*. When I go to merge PQ2 with PQ(1+3), I run into a problem. Are the cache listings in PQ2 newer than those in PQ(1+3), or are they older?
If when I merged PQ1 and PQ3 I used the "/gpx/time" for PQ1 as the "/gpx/time" for PQ(1+3), the cache listings in PQ2 will replace the cache listings in PQ(1+3), and so, any caches that were in PQ2 and PQ3 will be replaced with their older listings from PQ2, and any changes or logs on January 3-5 will be lost.
On the other hand, if when I merged PQ1 and PQ3 I used the "/gpx/time" for PQ3 as the "/gpx/time" for PQ(1+3), the cache listings in PQ2 will *not* replace the cache listings in PQ(1+3), and so, any caches that were in PQ1 and PQ2 will *not* be replaced with their newer listings from PQ2, and any changes or logs on January 1-3 will be lost.
On the third hand, if when I merged PQ1 and PQ3, I tagged each cache with a "Groundspeak:exported" node which contained the timestamp from the "/gpx/time" node of the GPX from which that particular cache came, when I merge PQ2, I will be able to tell which caches are newer in PQ2 (because they were compiled at PQ1's timestamp) and which caches are older in PQ2 (because they were compiled at PQ3's timestamp). I will then be able to merge PQ2 with PQ(1+3) to end up with a PQ(1+2+3) that contains the latest cache listing for each cache in the merged GPX.
So, basically, my request is simply to have some place in a merged file to save the individual cache listings' exported timestamps, and that's what I've asked Jeremy, et al, for.
quote:
That's not necessary if the waypoints in the GPX file have their compilation times specified. You wouldn't need to know when the file was last modified, since it doesn't matter when you saved -- it only matters when the data was compiled (since Watcher does not edit the content, only the presentation, of the waypoints in the GPX).
Just thought of something. You might want a "last modified" tag, as well. This would let users know the file is not as it came from Groundspeak. Just a thought.(Incidentally, if this is added, which I can't see why it wouldn't be, Watcher will shortly be updated to display when a cache was last exported. That way I'll be able to show exactly how out-of-date the cache details could be, instead of just saying that they might be old. I can't be specific yet, of course, since there is no way of knowing whether a GPX file was a result of a merge.)
*No, I don't have a PocketPC.
[This message was edited by ClayJar on January 28, 2003 at 08:41 AM.]
-
quote:
The heuristic is simple, really. If a given cache has a "Groundspeak:exported" node, use that as that cache's timestamp, else use the "/gpx/time" node. If neither GPX has ever gone through Watcher, there will be no "Groundspeak:exported" nodes and the "/gpx/time" nodes will be the sole arbitrator of which GPX is canonical. If a given cache *does* have a "Groundspeak:exported" node, that is used instead of the GPX file's "/gpx/time" node when determining canonicity.
Originally posted by Warm Fuzzies - Fuzzy:Of course, creating your own schema and namespace only works if the paragraph I've quoted above is true, and I'm not sure that it is (what if I have a "southern indiana" query and a "northern indiana" query that overlap, and I merge two of those? What if one is significantly newer than the other but neither has ever gone through Watcher?)
(As an aside, if "exported" is not added, I'd probably toss the value into a "Groundspeak:attribute" under "Groundspeak:cache/Groundspeak:attributes". It'll be a rather ugly kludge, but not so ugly as having to extend the namespace myself. It should be a moot point, though, since it's a thoroughly logical addition to the Groundspeak namespace, and it won't cause one iota of extra work for Geocaching.com.
) -
Okay, as you obviously know, I've been working on Watcher for over 6 weeks or so, and it's come a long way. There's just one thing that continues to bother me about it, and I believe I've come to a logical conclusion. Obviously, I am not the judge of that, but the case is substantial and rather clear-cut.
Simply put, there is a pressing need for an element or attribute that can hold the date/time that a given cache waypoint was exported.
Why is this necessary? Watcher (and other programs) can merge several GPX files into one GPX. This is very useful for those people who need multiple queries to cover a given area, or who have several subsets of their local caches returned by different queries. When one GPX file is merged with another, you simply take the date/time of the one GPX and compare it with the next; however, when you take that merged GPX and in turn merge another GPX into it, how can you know which elements are newer?
The short answer is, you can't. You *could* compare log IDs, or you could search for the latest placed cache, or you could try a number of other kludges, but none of them can give you enough knowledge about a given element to tell whether the original one or the merging one is the most current. The only way to know with certainty the date/time a element in a merged GPX file was exported by a pocket query is to have that information in the merged GPX file. The additional line in the .xsd would be something like this (if it were to be an element):
<xs:element name="exported" msdata:Prefix="Groundspeak" type="xs:string" minOccurs="0" />
Note, however, that there is *no* need for Pocket Query-generated GPX files to include this information in the elements. As long as it is in the definition that it *can* be there (but not that it *must* be there), Watcher (or any other GPX-merging application) can simply add the information to the GPX file at the time of merging. (When I merge, I'd just have to mark all the existing caches as exported on the date/time of the original file, and the waypoints being merged would be marked as exported on the date/time of the file being merged.)With this in the schema definiton, I will be able to update my GPX merge code to *always* use the latest available for each cache, which will mean that the possibility of not using the newest data due to a merge will cease to be the problem that it currently is. Since most people have been merging concurrent queries, this has not yet been much of a problem, but with more usage of Watcher and other GPX applications, it's only a matter of time before someone merges an older file into a newer file and ends up with stale waypoints. With a simple line or two, you can make it possible for me to head off the problem before it becomes a problem.
Watcher
in West and Southwest
Posted
That's a bug that Dan (Mr. Easy/ExpertGPS) has told me will be fixed soon (but it isn't fixed yet). In the meantime, you can get around it very easily. Just open the Watcher-exported GPX file in WordPad and replace all "/>" with " />" (to add the space).
There is some work being done to allow Watcher to integrate more closely with GPSBabel. Once that's done, all the work the GPSBabel people have done will be able to benefit Watcher users directly. (It's a good bit of work to connect the two with a simple but full-featured user interface, but thanks to GPSBabel, it's not like I have to write all the communication stuff, which would be rather more difficult.)
I'm glad that Watcher is making itself useful. Watcher 0.1.31 is almost ready. We're just debugging it and waiting for Jeremy to reply about the very small XSD update that it needs for the intelligent merge code.