Jump to content

New BM2GPX Program


YeOleImposter

Recommended Posts

I have been wanting to work on an updated bmgpx program for a long time, someone has beat me to it (thankfully). Foxtrot_xray over at the GSAK forum is working on a great utility that will parse the ngs datasheet files and put them into a gpx file that you can use in whatever program you wish. We are using GSAK but I am sure it could be used by any .gpx program.

 

I really like that the short description is completely user defined -- you can have it output anyway you want.

 

Check it out, see if you can use it, give some suggestions to the author. I sure have, and he may tire of me :(

 

08/20/07 0.5.0.232 Initial Release

- The Benchmark's URL is pointed to NGS's datasheet, not GC.com.

- The short description is read from an HTML template file ('short.html', must be in executing directory with ngs-gpx.exe). You can make the HTML file any way you want, and use tags (like "%%latdec%%") that the program will substitute.

- The full datasheet is included in the Long Description.

- If available, the "Place Date" is the first listed date in the History section. The "Placed By" and the "Owner" are also the first listed in the History section. (If unavailable, the date is 1/1/1800, and the condition/person is "-Unknown-".)

- I've had issues with BMGPX dropping certain datasheets. (Dunno why, haven't looked into it. But BMGPX when it reeads my entire state drops about 3k of the 17k that Georgia has.) I made sure that my program can handle it all and not 'lose' records.

- Support is in place for GPX 1.1 smile.gif It's not used yet, but if it ever is put into use by GSAK, this little bugger's ready.

- Semi-Support for Destroyed marks. If there's a destroyed mark in your TXT file, it'll get loaded into the GPX with the reason it's unpublishable. (See note below.)

 

08/24/07 0.6.0.16

Changes:

1. Modified HTML to produce a cleaner (and more HTML-correct) file for GSAK to use for the 'short description'

2. Disabled the 1/1/1800 date, making it the next valid date found. (In case of NO VALID DATES, it will still be 01/01/1800 however.)

3. The COUNTY from the datasheet is now put into the COUNTRY field in the GPX for importing.

4. Found descriptions as all put into 'logs'. Long description is still full datasheet for now, and no date/whom/condition is transferred along with each log. (See TODO)

5. The coordinate type is stored in the "Container" field. GSAK will accept this, however the user will have to note that if editing the waypoint and the 'Container' field is accidently changed, there's no easy way to set it back to it's default. Editing will also show the 'closest match' in it's pre-defined list to what the type is. (i.e. "Scaled" will show "Small" in the drop-down list.) As long as it's not changed, it won't get updated.

6. The string ALTUTIDE (TYPE) is now put into the USER2 field in GSAK.

 

Things Todo:

-------------

1. The reading and parsing of the History section needs to be redone. Once it is, it will add functionalityto adding the descriptions into the LOG section of the GPX file.

2. Figure out what to put in the long description. (Options to consider are: First description. All descriptions. Full Datasheet.)

3. Add options to running program:

3A. Disable Altitude in User2 field.

3B. Enable passing of own URL to use.

Link to comment

The One I have does a real fine job already.

I just got it so it may be the new one.

 

I have already tried it out in the field with my laptop and it works great with google earth as well.

 

If you increase your disc cache size you can hold an entire county in google earth without having to be online.

Just make sure to zoom in to the benchmarks you plan to visit that day.

 

And of course you can refresh with one of thos WI-FI hot spots.

Thats where the Waymarks for WI_FI hotspots comes in handy.

 

You can read all the descriptions from the NGS conversions.

 

It makes it a little easier.

But my found to not found did not do so good.

I found 1 out of 4 stations.

 

But had the fun of playing with the live tracking in all the programs.

 

GPSTrackmaker has just released a North America map you can now use with the Live tracking feature.

 

Way to many cool gizmos,bells and whistles now a days.

I am becoming a computer geek learning all this new fangled jargin.

Link to comment

I tried this program but it didn't work. It first required me to download NET Frameworks (from Microsoft), which I did (I downloaded the 2.0 version here).

 

The NGS-GPX program would quit on the DESIGNATION line (quoting the designation) on the first PID of the 3 different .DAT files I tried. (I'm running windows XP.) The NGS-GPX program just made a .GPX header file 8 lines long for each .DAT file.

 

Let me know if there's anything else I can do and I will try out this program again. :(

Link to comment

I tried this program but it didn't work. It first required me to download NET Frameworks (from Microsoft), which I did (I downloaded the 2.0 version here).

 

The NGS-GPX program would quit on the DESIGNATION line (quoting the designation) on the first PID of the 3 different .DAT files I tried. (I'm running windows XP.) The NGS-GPX program just made a .GPX header file 8 lines long for each .DAT file.

 

Let me know if there's anything else I can do and I will try out this program again. :D

Sorry about the .NET thing - in my post over in the GSAK forum I stated that I'm /not/ a programmer, I just used what I had access to. :cry: My original post I copied below, for folks here not on GSAK..

 

Like I said, there's no error checking at the time. :o

 

Double-check the two following thins, and if you can e-mail me the GPX files, it'd be appreciated:

1. Verify that the "short.html" file exists in the same directory as the .EXE

2. Verify the directory of the file you're importing can be written to, and if a file with the same name .gpx exists, that it's writable.

 

My e-mail is coyttl @ gmail . com

 

Thanks! :(

Edited by foxtrot_xray
Link to comment

[Edit: Original program post fro GSAK. Explains almost everything, except for the meaning of life.]

Okay, kinda an alternative to BMGPX...

 

The program accepts a text file saved/downloaded from the NGS website - in a datasheet format. It will create a GPX file of the data from the datasheet for loading into GSAK. (Or any other GPX-enabled program..)

 

What it does differently from BMGPX:

 

* The Benchmark's URL is pointed to NGS's datasheet, not GC.com.

* The short description is read from an HTML template file ('short.html', must be in executing directory with ngs-gpx.exe). You can make the HTML file any way you want, and use tags (like "%%latdec%%") that the program will substitute.

* The full datasheet is included in the Long Description.

* If available, the "Place Date" is the first listed date in the History section. The "Placed By" and the "Owner" are also the first listed in the History section. (If unavailable, the date is 1/1/1800, and the condition/person is "-Unknown-".)

* I've had issues with BMGPX dropping certain datasheets. (Dunno why, haven't looked into it. But BMGPX when it reeads my entire state drops about 3k of the 17k that Georgia has.) I made sure that my program can handle it all and not 'lose' records.

* Support is in place for GPX 1.1 smile.gif It's not used yet, but if it ever is put into use by GSAK, this little bugger's ready.

* Semi-Support for Destroyed marks. If there's a destroyed mark in your TXT file, it'll get loaded into the GPX with the reason it's unpublishable. (See note below.)

 

Notes..

 

* The program requires at LEAST .NET 1.1 to be installed. Most systems have a compatible version running.

* Unpublishable codes sometimes have more than one entry. i.e. An unpublishable record may have two lines, the first saying there's no mark at the position, the second stating that the mark was destroyed. I didn't realize this when I programmed it, so if there is more than one line for an unpublishable mark, it'll get thrown into the GPX file twice. I do not know how GSAK would handle this..

* It IS command-line based - basically run the program with the file name as a parameter. Alternatively, drag the file you want to convert onto the icon, and it'll convert. (Same operation as BMGPX)

* I made this for me, and had no intention of actually making it available to anyone else. However, I figured that it can't hurt, so I'm posting the download link here. I'm NOT a programmer, so much of this was a learning experience for me. There are things I'd like to try and do, but due to time and the fact I got what I needed, may not happen. However, some things I'd like to see are:

o Form front-end.

o Better handling of the unpublishable marks.

o Able to allow the user to supply a URL for the mark. (select GC.com or NGS datasheet, or supply a custom one.)

o Better error handling...

* Because of the processing, it's not AS fast as BMGPX, but it can do all of Ga's 17k benchmarks in about 6 minutes (on a 3.0Ghx P4 XP Pro machine, running many other things at the same time.)

 

If it errors out, you will NOT get an error message, nor a summary line. (When it completes, it will tell you how many datasheets it processed.) If you get it to error out, please let me know and send me the exact TXT file you used as input. I'd LIKE to at least get all the processing errors taken care of. smile.gif

 

Build: 0.5.0.232 - Download it here. Just unzip. No installation needed.

 

Comments, suggestions all welcome.

 

Update: 0.6.0.16 (Download It)

Changes:

1. Modified HTML to produce a cleaner (and more HTML-correct) file for GSAK to use for the 'short description'

2. Disabled the 1/1/1800 date, making it the next valid date found. (In case of NO VALID DATES, it will still be 01/01/1800 however.)

3. The COUNTY from the datasheet is now put into the COUNTRY field in the GPX for importing.

4. Found descriptions as all put into 'logs'. Long description is still full datasheet for now, and no date/whom/condition is transferred along with each log. (See TODO)

5. The coordinate type is stored in the "Container" field. GSAK will accept this, however the user will have to note that if editing the waypoint and the 'Container' field is accidently changed, there's no easy way to set it back to it's default. Editing will also show the 'closest match' in it's pre-defined list to what the type is. (i.e. "Scaled" will show "Small" in the drop-down list.) As long as it's not changed, it won't get updated.

6. The string ALTUTIDE (TYPE) is now put into the USER2 field in GSAK.

 

TODO List (In Random Order):

1. The reading and parsing of the History section needs to be redone. Once it is, it will add functionality to adding the descriptions into the LOG section of the GPX file.

2. Add options to running program:

2A. Disable Altitude in User2 field.

2B. Enable passing of own URL to use.

3. Integrate M$ XML interface for writing GPX file. This will allow #4 below.

4. For datasheets with a list of reference marks in the box format, set any mark listed that's also in our GPX file as a child waypoint.

Edited by foxtrot_xray
Link to comment

New release! :)

 

Release: 08/27/07 0.7.0.42

Changes:

1. Changed how the 'log ID' was created. Now to ensure no duplicates are 'accidentally' created, it makes the

ID by: 1<xx><yy><zzzz><#>. Where <xx>, <yy> is the ASCII value of the first two letters of the PIX, and

<zzzz> is the last four digits of the PID. <#> is the log number for that PID. (the '1' is static,

required.)

2. 'short.html' changed to 'long.html', new 'short.html' included. Got one you want to show off? It's

probably better than mine. Send it my way, and I'll include it with the program!

3. Altitude in "User2" padded for sorting. Removed the type.

4. Done re-parsing the history section! Now logs are inserted that show as "Found", "Not Found" or a "Note".

All "Good", "Poor", and "Monumented"/"First Observed" are labeled as FOUND IT. "Mark Not Found" is a

DID NTO FIND. Anything else is a "Note". In the case that there are more descriptions than history lines,

the unmatched ones will be simply "Notes" with the current date/time and "-Unknown-" used.

NOTE: This could create an issue - see PID CK6103. Because one UNKNOWN is actually IN the Description

list, the dates become off by one.

5. Better handling of the History date (in reference to #4 above).

 

Download: Here (0.7.0.42)

 

I know of one issue importing the finds as logs, and I'm still trying to figure out how to correct it.. But please, look at your imported logs, and let me know if you have any issues with it. (Obviously, I couldn't test with the half-million PIDs in the NGS database.) B)

 

Also, if anyone develops a good HTML short and/or long summary, lend it to me, I'll include it with the program. Mine are.. REALLY basic. :)

Link to comment

New release! :rolleyes:

 

Release: 08/27/07 0.7.0.42

Changes:

2. 'short.html' changed to 'long.html', new 'short.html' included. Got one you want to show off? It's

probably better than mine. Send it my way, and I'll include it with the program!

 

Looks like the new "short.html" did not get included. The one in the zip file is dated 8/24 and has the same missing "</style>" style statement at the end of that section. Am real curious what the new one was going to look like <_<

Link to comment

Looks like the new "short.html" did not get included. The one in the zip file is dated 8/24 and has the same missing "</style>" style statement at the end of that section. Am real curious what the new one was going to look like :rolleyes:

Well.. uhm.. all the short has is lines taken out of it.. I'm not really HTML proficient, so.. heh. It's ugly. They're BOTH ugly, in fact! <_<

 

Okay, quick patch. Do NOT try to download the 7.0.42 anymore, it won't work.. :rolleyes:

 

08/28/07 0.7.0.43 Small Fixes

Changes:

1. This time, I REALLY fixed the 'short.html' file. I swear!

2. Added the token %%fullheet%%. If this occurs in anywhere in one of the template HTML files, the *FULL*

datasheet will be inserted. (Best put it in a "preserve whitespace" style!)

 

Download Here (0.7.0.43)

Link to comment

New release!

Noticeable changes:

- Logs are imported now with statuses! (Poor, Monumented, etc.)

- You can now supply your own URL to use!

- Marks with reference points now have those reference points loaded in!

 

For the last one, if you use this and use that feature, test the location (coordinates) of said child waypoints for me. I still haven't been able to lock down the formula yet, so if there's an issue, I'd like to know about it. :D

 

08/30/07	0.7.1.89
===============
Changes:
1. For logs, changed some of the reporting types: "Monumented" and "First Observed" are now put as 'Owner
Maintenance'. "GOOD" is unchanged. "POOR" is 'Needs Maintenance'. "MARK NOT FOUND" is unchanged.
Everything else is still a 'Note'.
2. Helped to improve speed. searching for and replacing the new %%fullsheet%% and the %%findhist%% and
%%finddesc%% tag was seriously slowing the program down.
3. Bugfix: "Unknown" dates would have only been 'fixed' if the FIRST history was unknown.
4. CodeChange: Changed how the 'fullsheet' and 'history' and 'descriptions' were stored in memory.
5. Added Child Points. Using a make-shift function to determine coordinates from bearing/distance. (Two
different methods used (http://www.experts-exchange.com/Programming/Languages/.NET/Visual_Basic.NET/Q_22555684.html)
and (http://www.movable-type.co.uk/scripts/latlong-vincenty-direct.html) gave me wrong results. Math
is different between Javascript/VB6 in .NET, it seems. Have to	track down and fix my bad math.)
6. Added %%referencebox%% token. This will translate to the reference box in the datasheet in plain text.
7. New command-line options:
--url=[url | "ngs" | "gc"] : Allows user to specify what to use as the URL. Technically, all tokens
	used in the HTML files allowed, but only the %%pid%% is really useful. Using "ngs" (default) will
	use the default NGS datasheet. Using "gc" will point it towards the geocaching.com website.
	Otherwise, use your own url, like: "--url=http://tsqmadness.com/lookup.cgi?%%pid%%".
--rua=[yes | no] : Use Approx Reference Points. "no" (default) will skip all reference points whose
	distance starts with "Approx". "yes" will include these as child points. (Obviously, marks  with
	no distance will always be skipped.)

 

Download Here. [0.7.1.89]

Issues? Questions? Concerns? Donations? Let me know. :lol:

Me.

Link to comment

OK, so I downloaded your latest, ran the net frameworks 2.0 when I drop a .dat file on the .exe file I only get this far:

 

<?xml version="1.0" encoding="utf-8"?>

<gpx xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" version="1.0" creator="NGS->GPX" xsi:schemaLocation="http://www.topografix.com/GPX/1/0 http://www.topografix.com/GPX/1/0/gpx.xsd http://www.Groundspeak.com/cache/1/0 http://www.Groundspeak.com/cache/1/0/cache.xsd http://www.gsak.net/xmlv1/1 http://www.gsak.net/xmlv1/1/gsak.xsd" xmlns="http://www.topografix.com/GPX/1/0">

<name>or027.gpx</name>

<desc>Benchmark GPX file created from NGS Benchmark file. (HasChildren)</desc>

<author>Fox Morrey</author>

<email>coyttl@gmail.com</email>

<time>2007-08-30T18:02:00-06:00</time>

</gpx>

 

If I double click the .exe I get a standard windows error report.

 

Any ideas?

 

Thanks

Link to comment

OK, so I downloaded your latest, ran the net frameworks 2.0 when I drop a .dat file on the .exe file I only get this far:

 

<?xml version="1.0" encoding="utf-8"?>

<gpx xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" version="1.0" creator="NGS->GPX" xsi:schemaLocation="http://www.topografix.com/GPX/1/0 http://www.topografix.com/GPX/1/0/gpx.xsd http://www.Groundspeak.com/cache/1/0 http://www.Groundspeak.com/cache/1/0/cache.xsd http://www.gsak.net/xmlv1/1 http://www.gsak.net/xmlv1/1/gsak.xsd" xmlns="http://www.topografix.com/GPX/1/0">

<name>or027.gpx</name>

<desc>Benchmark GPX file created from NGS Benchmark file. (HasChildren)</desc>

<author>Fox Morrey</author>

<email>coyttl@gmail.com</email>

<time>2007-08-30T18:02:00-06:00</time>

</gpx>

 

If I double click the .exe I get a standard windows error report.

 

Any ideas?

 

Thanks

 

Hey,

Verify that you have both "short.html" and "long.html" in the same directory as ngs-gpx.exe (and that they end in "html" and not "htm". If that's okay, send me your DAT file, please - it could have something I didn't plan correctly for. :)

Link to comment

OK, so I downloaded your latest, ran the net frameworks 2.0 when I drop a .dat file on the .exe file I only get this far:

 

<?xml version="1.0" encoding="utf-8"?>

<gpx xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" version="1.0" creator="NGS->GPX" xsi:schemaLocation="http://www.topografix.com/GPX/1/0 http://www.topografix.com/GPX/1/0/gpx.xsd http://www.Groundspeak.com/cache/1/0 http://www.Groundspeak.com/cache/1/0/cache.xsd http://www.gsak.net/xmlv1/1 http://www.gsak.net/xmlv1/1/gsak.xsd" xmlns="http://www.topografix.com/GPX/1/0">

<name>or027.gpx</name>

<desc>Benchmark GPX file created from NGS Benchmark file. (HasChildren)</desc>

<author>Fox Morrey</author>

<email>coyttl@gmail.com</email>

<time>2007-08-30T18:02:00-06:00</time>

</gpx>

 

If I double click the .exe I get a standard windows error report.

 

Any ideas?

 

Thanks

 

Hey,

Verify that you have both "short.html" and "long.html" in the same directory as ngs-gpx.exe (and that they end in "html" and not "htm". If that's okay, send me your DAT file, please - it could have something I didn't plan correctly for. :)

 

Foxtrot,

 

I do have those three in the same directly and they are correctly named as well as the .dat file. I doubt it is anything with the .dat file because I have tried various different file folders and different .dat files. I must have something screwed up.

 

CallawayMT

Link to comment

OK, so I downloaded your latest, ran the net frameworks 2.0 when I drop a .dat file on the .exe file I only get this far:

 

<?xml version="1.0" encoding="utf-8"?>

... deleted text ...

</gpx>

 

If I double click the .exe I get a standard windows error report.

 

Any ideas?

 

Thanks

 

I probably should have said something earlier about this, but same thing happens to me.

I thought it was just my windows setup.

I cannot drag and drop either -- I have to run the program from the dos box --

 

ngs-gpx.exe county.dat

(where county.dat is the file I want to convert)

 

But I have same problem with other programs too -- so just thought it was me.

 

This is a great program! I have just finished updating about 6 counties in my area (nearly 10,0000 benchmarks) and it is doing a great job. Look forward to a better formula for the projecting the reference marks.

 

I also added the following lines in long.html so I would have the box score as part of the description:

 

  <tr>
<td class="tbldataheader">Reference:</td>
<td class="tbldata"><pre>%%referencebox%%</pre></td>
 </tr>

Link to comment

OK, so I downloaded your latest, ran the net frameworks 2.0 when I drop a .dat file on the .exe file I only get this far:

 

<?xml version="1.0" encoding="utf-8"?>

... deleted text ...

</gpx>

 

If I double click the .exe I get a standard windows error report.

 

Any ideas?

 

Thanks

 

I probably should have said something earlier about this, but same thing happens to me.

I thought it was just my windows setup.

I cannot drag and drop either -- I have to run the program from the dos box --

 

ngs-gpx.exe county.dat

(where county.dat is the file I want to convert)

 

But I have same problem with other programs too -- so just thought it was me.

 

This is a great program! I have just finished updating about 6 counties in my area (nearly 10,0000 benchmarks) and it is doing a great job. Look forward to a better formula for the projecting the reference marks.

 

I also added the following lines in long.html so I would have the box score as part of the description:

 

  <tr>
<td class="tbldataheader">Reference:</td>
<td class="tbldata"><pre>%%referencebox%%</pre></td>
 </tr>

 

Thanks,

 

If I run it from dos it does work.

 

CallawayMT

Link to comment

Hrm.. Thanks for letting me know of that guys. I'll take a look at it... Drag&Drop should work.. :/

 

Update:

Got it. it was a simple mistake - one I didn't see until 40+ builds later after going line-by-line comparisons between the two. Interestingly enough, at the end the two programs (mine vs the Javascript) are exact up to the 6th decimal place of each coordinate. Luckily, I feel that my program's more accurate, due to the fun datatypes.

 

Also, important notice - since someone contacted me about it - I am NOT up against Black Dog Tracker's KML program in the least. While I haven't used it (yet - I'm sure, as s strong Google Earth user, I will!) as I understand it his is mostly for GE, whereas mine's aimed at GSAK. :)

 

Mike.

Edited by foxtrot_xray
Link to comment

w00t! New version. Probably the last for a while, unless something major creeps up..

Major notes:

-Two new flags to disable child and log generation. Now does not create altitudes in User2 field unless you specify "--u2alt=yes"

-Now finds as-correct-as-possible coordinates for reference points.

-Can specify own template files.

-Fixed drag & drop.

 

[Edit: The drag&drop still does not work. I thought I had it nabbed, but something slipped through the crack. You CAN drag & drop now, but no templates will be used.]

 

08/31/07	1.0.0.67
Changes:
1. CodeChange: Better parameter organization. Faster handling of parsed templates.
2. Program will not just return an empty short or long description if the files cannot be opened. No more sudden-death quitting due to missing files.
3. Drag & Drop now works! In other words, program wasn't running right if the 'current working directory' wasn't the directory the program was in. This is now fixed.
4. Added two command-line options: "--sdec=" and "--ldec=". --sdec specified the file to use for short description. If invalid or blank, will use default. --ldec is the same, but for the long description.
5. Fixed child-waypoint coordinate finding! w00t!
6. Now defaults to NOT putting the altitude in the User2 field. Use the command-line arg "--u2alt=yes" to enable it. "--u2alt=no" is default and won't put the altitude in the User2 field.
7. Added user flags "--nochild" and "--nolog". The former, if entered on the command-line, will NOT use any reference points. The latter, if specified, will not load any of the station descriptions into the logs for that mark. (Note that the 'referencebox' token is always valid.)
8. BugFix: The %%fullsheet%% token had the descriptions cut out. Fixed.
9. Better history/logging matching. Compares dates now. Is much slower processing, but that's what you get. :) If a description has an unknown date, it will scan the history BACKWARDS to find the firt history entry with an unknown date and link them together.
10. Bugfix: Unknown dates were still not fixed. D'oh! It's all better now. I'm sure of it.
11. Figure this is darn good enough for version 1.0. :)

 

Download Here. (1.0.0.67)

Edited by foxtrot_xray
Link to comment

Hey F-X:

Looking pretty good! Glad someone tackled the BM-GPX issue. Thanks. Downloaded your latest ver 1.0.0.67

1) Tossed out my old .net (it was v1.1, and too old, per error message from BM2GPX)

2) Downloaded & installed .net v2.0

3) All working. Drag & drop worked, .gpx file generated for my home county (Orange, CA)

4) GSAK did not accept the child waypoints. I saved the GSAK error box & dropped it in a .doc file. If you want it, PM me a real email addess. In a nut shell: "Additional child waypoints not added because no matching parents" FYI, it says there were 632 children for the 1701 marks. Seems a reasonable number. The error message also looks like the main and child files have the same file name? That might be the problem?

5) Child waypoints not found in GSAK in the usual manner. I guess not there.

6) Everything else looks pretty good. Still exploring where you put stuff, but much improved functionality from the old BMGPX.

7) One additional thought: How about a short & sweet readme.txt file that has a list of all the current switches & options? Tried the traditional ? and H switches from DOS, but I guess you're not THAT traditional.

Maybe I'll have a few other comments later.

Keep up the good work & thanks.

Link to comment

Hey F-X:

Looking pretty good! Glad someone tackled the BM-GPX issue. Thanks. Downloaded your latest ver 1.0.0.67

4) GSAK did not accept the child waypoints. I saved the GSAK error box & dropped it in a .doc file. If you want it, PM me a real email addess. In a nut shell: "Additional child waypoints not added because no matching parents" FYI, it says there were 632 children for the 1701 marks. Seems a reasonable number. The error message also looks like the main and child files have the same file name? That might be the problem?

5) Child waypoints not found in GSAK in the usual manner. I guess not there.

PM sent re: files.

 

I'll have a readme (and an actual page up on my site) for the program soon, probably over this weekend.) :)

Edited by foxtrot_xray
Link to comment

4) GSAK did not accept the child waypoints. I saved the GSAK error box & dropped it in a .doc file. If you want it, PM me a real email addess. In a nut shell: "Additional child waypoints not added because no matching parents" FYI, it says there were 632 children for the 1701 marks. Seems a reasonable number. The error message also looks like the main and child files have the same file name? That might be the problem?

5) Child waypoints not found in GSAK in the usual manner. I guess not there.

ATTENTION: Disregard the above. My fault. After a subtle nudge from F-X, I updated my old version (6.6.5) of GSAK to the latest (7.1.1.32), and the problem is solved! Lesson learned here. I have a habit of not always upgrading (even when free) - hey - if it ain't broke, don't fix it, right? Well, now all is OK! Thanks F-X!
Link to comment

Program has been working well! 2 feature requests I have thought of while using it:

 

1) Make ignoring destroyed markers the default. I end up editing my ngs.dat file to not include the text at the end so that I don't have to deal with them.

 

2) could the output of the program go to a file? Not sure it is really needed but it whizzes by so quickly I wonder sometimes if I missed something. I guess I could just pipe the output myself, but others may not know know how to do that.

 

BTW, I have edited the FindStatGen macro to produce my benchmark statistics. My numbers are not very impressive compared to the veterans here but it is fun to have the statistics.

 

Here is a link: YeOleImposter BMark Stats

 

I still want to tweak some stuff, like figuring out a way to add DNF data -- but for now it has the important stuff.

Link to comment

#1 I can change over, not a problem. I never thought about leaving it off because, well, I rarely have a file with both published and non-pub sheets in the same file.

 

#2 I can do, but don't think there's a point. My reasoning is this: When it's processing, it only spits out the PID and the Designation. If there's some sort of programming error (I make as assumption and try to read past the end of the sheet, for example) the program will just stop there. It won't print out any error message. (Not being a programmer, I haven't looked into/educamutated myself on serious error-handling yet.) Now, if there's an error in the datasheet (say, there's no longitude, or there's no "History" table, the CORS ID is blank) the program won't do any alert - it just won't populate the stored data with anything. Basically, it ignores what's not there. :huh:

 

If there's another reason you would find it useful (like you think certain datasheets are getting skipped or some such) let me know and I'll see what I can do. B)

 

Program has been working well! 2 feature requests I have thought of while using it:

 

1) Make ignoring destroyed markers the default. I end up editing my ngs.dat file to not include the text at the end so that I don't have to deal with them.

 

2) could the output of the program go to a file? Not sure it is really needed but it whizzes by so quickly I wonder sometimes if I missed something. I guess I could just pipe the output myself, but others may not know know how to do that.

 

BTW, I have edited the FindStatGen macro to produce my benchmark statistics. My numbers are not very impressive compared to the veterans here but it is fun to have the statistics.

 

BTW - nice statistics page! I like it! :D I'd be afraid to see mine, tho. :rolleyes:

 

Here is a link: YeOleImposter BMark Stats

 

I still want to tweak some stuff, like figuring out a way to add DNF data -- but for now it has the important stuff.

Link to comment

No, your reasoning sounds right. I think it was just the number of destroyed markers it listed that I was curious about.

 

BTW, used it again this weekend and it was great to be able to walk to the RMs since your program gives me the coordinates which are automatically loaded in the GPS. Found some that I doubt I would have otherwise.

Link to comment

No, your reasoning sounds right. I think it was just the number of destroyed markers it listed that I was curious about.

Oh, I can explain that - sometimes (quite often, I've found) is that some non-pub marks have multiple entries (for example, not a real sheet:)

>DF0001 NEEDLE IN HAYSTACK		33 22.564  84 55.669   NN
>DF0001 NEEDLE IN HAYSTACK		33 22.564  84 55.669   d

..and in a case like that, my program counts 'DF0001' twice. None of my test sheets had this, so I never came across it until I went through a full state. Since I don't store everything in memory and write as I go, there's no way to 'go back' and modify the WPT entry in the output file. This is why 'destroyed' info is only "semi-working". :)

Link to comment

Does the program handle just one datasheet at a time, or can it break down a file with multiple datasheets in it?

 

Heck yeah! I've done my entire state (Georgia) and let it process 17,5xx at once. :) And, unlike 'BM2GPX', mine won't 'miss' datasheets, like I've had that one do. (Actually, that was the main reason of making a new program..)

Link to comment

Heck yeah! I've done my entire state (Georgia) and let it process 17,5xx at once. :) And, unlike 'BM2GPX', mine won't 'miss' datasheets, like I've had that one do. (Actually, that was the main reason of making a new program..)

 

Just dloaded v 0.5.0.232, and tired a single datasheet and a string of datasheets. Each time the gpx file returned had no waypoints in it. Here is the content of one of the files:

 

<?xml version="1.0" encoding="utf-8"?>

<gpx xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" version="1.0" creator="NGS->GPX" xsi:schemaLocation="http://www.topografix.com/GPX/1/0 http://www.topografix.com/GPX/1/0/gpx.xsd http://www.Groundspeak.com/cache/1/0 http://www.Groundspeak.com/cache/1/0/cache.xsd http://www.gsak.net/xmlv1/1 http://www.gsak.net/xmlv1/1/gsak.xsd" xmlns="http://www.topografix.com/GPX/1/0">

<name>bm.gpx</name>

<desc>Benchmark GPX file created from NGS Benchmark file. (HasChildren)</desc>

<author>Fox Morrey</author>

<email>coyttl@gmail.com</email>

<time>2007-09-15T10:59:02-05:00</time>

</gpx>

Link to comment
Just dloaded v 0.5.0.232, and tired a single datasheet and a string of datasheets. Each time the gpx file returned had no waypoints in it. Here is the content of one of the files:

 

Before trying to figure out the problem, where did you find v0.5.0.232 or was that a typo?

 

I believe the current version is 1.0.1.83 and available at http://ngs.tsqmadness.com.

 

There were some early versions that drag and drop did not work, you had to actually use the DOS command line, but that has been fixed.

Link to comment

Just dloaded v 0.5.0.232, and tired a single datasheet and a string of datasheets. Each time the gpx file returned had no waypoints in it. Here is the content of one of the files:

 

First, download the latest version here. Then try it for me. If, after that, you still have issues, attach or forward or e-mail me the DAT/NGS datasheet you're trying to read, and I'll see what's causing it to barf. Many issues have been addresses which cause errors making the program quit like that.

 

Thanks!

Mike.

Link to comment

Well, it appears that the new version of the datasheets has broken your parser.

 

When I pull up a new set of sheets the first line had said version 7.49 but now it shows:

DATABASE = Sybase ,PROGRAM = datasheet, VERSION = 7.50

 

When I run a diff on the files I notice that the 'space'characters have changed on the * NAD line:

 

version 7.49:

SA1190* NAD 83(1986)-  46 12 37.	 (N)	119 45 52.	 (W)	 SCALED	
SA1190* NAVD 88	 -	   203.929  (meters)	 669.06   (feet)  ADJUSTED  

 

version 7.50

SA1190* NAD 83(1986)	-  46 12 37.	 (N) 119 45 52.	 (W)	SCALED	
SA1190* NAVD 88		 -	   203.929  (meters)  669.06  (feet)  ADJUSTED  

 

Notice the spaces after (1986) and the number of spaces between (N) and the coordinate.

When I search and replace these so that the format matches for this line then the data file processes fine again. Looks like you might need to use some regex matching instead of straight column count? I am not sure how often they change the format.

 

Gary

 

UPDATE: It looks like the spacing was changed so that the columns would line up when the date looks like this:

SD0042* NAD 83(NSRS2007)-  46 54 28.20193(N)	124 06 42.79448(W)	ADJUSTED  

Edited by YeOleImposter
Link to comment

Well, it appears that the new version of the datasheets has broken your parser.

 

Looks like you might need to use some regex matching instead of straight column count? I am not sure how often they change the format.

I'm not surprised. B)

 

I am curious tho, are these being changed on the fly? I mean, I had just saved a 'new' datasheet from NGS and it ran fine(*), which means that the columns in the coordinates hadn't been changed yet. Then you post a line from SA1190 that shows them in different locations - but when I just pulled them up now, it's identical (in spacing) to your SD0042 example.

 

(*) It ran fine save for the descriptions - when I downloaded the sheet, the descriptions started with ", the original version had '. Now they're back to ', for some reason.

 

I'm wary to start making changes until they decide on a format. B)

 

Regarding the regex, I use it in some cases, didn't have to for most of the lines, because I wasn't expecting such a drastic change in the datasheets.. so soon. :P

 

Give me a day or so, and I'll have a version that can handle either datasheet. B) Assuming they don't change it again..

Link to comment

It is interesting that the NGS website has not kept up with the formats. The front page of the data sheets says:

 

Last change to datasheet format was made on 01/01/06

 

yet the last entry on the NGS version histories page says:

 

  The format of the NATIONAL GEODETIC SURVEY Data Sheet was changed
 on 03/29/05 Version = 7.17.

 

But of course the version that we have been using for quite a while is 7.49

 

I just looked for the oldest ngs dat file I still had, last year at this time I had version 7.37

DATABASE = Sybase ,PROGRAM = datasheet, VERSION = 7.37
1		National Geodetic Survey,   Retrieval Date = AUGUST  1, 2006

 

Re the ' or " -- it is interesting that the Tell Me More link for datasheets does not use either, leaving just a space between the PID and the text. So it is amazing how this stuff has changed over time, and just like most of the stuff I do, the changes have not been documented.

 

The 'Tell Me More' does seem to have been updated (even though it says nothing about ' or " for histories). This recent change is documented:

Loading of the National Readjustment data commenced on September 14, 2007.
Before this the format of the position and elevation lines appeared as follows:

AA3495* NAD 83(CORS)-  39 08 02.34046(N)	077 13 15.51884(W)	 ADJUSTED  
AA3495* NAVD 88	 -	   140.76   (meters)	 461.8	(feet)  GPS OBS 

After the readjustment, the position and elevation lines on a datasheet
will appear in a slightly modified format to accomodate the larger datum
tag field (i.e. NSRS2007) as shown in the below examples.
______________________________________________________________
DF9012* NAD 83(NSRS2007)-  42 56 15.39233(N) 071 26 19.03487(W)	ADJUSTED  
AA3495* NAD 83(CORS)	-  39 08 02.34046(N) 077 13 15.51884(W)	ADJUSTED   
RF0849* NAD 83(NSRS2007)-  46 52 08.05186(N) 068 00 53.02328(W)	ADJUSTED
TA0047* NAD 83(1986)	-  48 04 54.20   (N) 090 45 48.42   (W)	HD_HELD1
AC3384* NAD 83(1986)	-  25 57 14.7	(N) 081 43 29.2	(W)	HD_HELD2  
HV0454* NAD 83(1986)	-  38 20 52.	 (N) 076 13 39.	 (W)	SCALED   
DX3756* NAD 83(NSRS2007)-  33 38 08.42412(N) 117 05 10.37961(W)	ADJUSTED 
FQ0856* NAD 83(1986)	-  35 47 36.	 (N) 111 52 56.	 (W)	SCALED	

 

So, looking at this, not only has the 'NAD 83' column been modified, but we now have more than just 'Adjusted' and 'Scaled' at the end of the line.

Edited by YeOleImposter
Link to comment

I was checking NGSREAD yesterday to be sure it wasn't going to get confused with the new confidence table information on datasheets. It wasn't, but in doing that checking I found that I had to fix it to accept a file made from getting a county's present (not archived) data from the NGS datasheet by county site. In getting it fixed to accept that kind of file (different header spacing above the data), I noticed this issue with different positioning of the longitude and latitude on one line of the datasheet and a difference in where the elevation was located on the next line of data in the datasheet.

 

I solved this (I'm programming in C) by searching for the position of the (W) and also the (N) in the lat-lon line and by searching for the last decimal point in the elevation line. After I did that, the two NGS formats' differences were made transparent to NGSREAD. You might consider doing the same thing.

Edited by Black Dog Trackers
Link to comment

I was checking NGSREAD yesterday to be sure it wasn't going to get confused with the new confidence table information on datasheets. It wasn't, but in doing that checking I found that I had to fix it to accept a file made from getting a county's present (not archived) data from the NGS datasheet by county site. In getting it fixed to accept that kind of file (different header spacing above the data), I noticed this issue with different positioning of the longitude and latitude on one line of the datasheet and a difference in where the elevation was located on the next line of data in the datasheet.

Interesting how two people think alike, yet different. In my program, blank space doesn't affect my program at all - so I never had to worry about changing line numbers. (I just keep dropping lines until the first line that has a valid PID in positions 1-6.) Yet I never thought of looking for the lon/lat that way. How are you getting the elevation, then? (Like you said, it changes between the two versions as well. :P

[Edit: Nevermind, Duh. Obviously, just look for (meters).]

 

I like that way of doing it tho. If I can figure out how to do it, I'll change mine. That way, won't be forced into another change if the sheet changes. B)

 

[EDIT: Got it. I think. Everything I tested it on worked fine. New version of my prog will be up in about 5 minutes from now, here. B)]

Edited by foxtrot_xray
Link to comment
How are you getting the elevation, then? (Like you said, it changes between the two versions as well.

[Edit: Nevermind, Duh. Obviously, just look for (meters).]

 

Speaking of meters......

 

Any way to make it an option to use feet instead? This is being plugging this into UserData2 and honestly it always throws me cause I still think in feet.

Link to comment
[Got it. I think. Everything I tested it on worked fine. New version of my prog will be up in about 5 minutes from now, here. B)]

 

I ran the new version (after fixing the backwards slash in your link) and looks like you have a little problem :P Appears you have accidentally dropped the Negative sign in front of the longitude so all my benchmarks are now in China B)

 

-YOI-

Link to comment

I was going to say that I didn't look for meters or feet because the feet moved too in the change from 2006 archives to present day. That's why I was saying that for the elevation I used the last decimal place in the line (C has a function for that).

Oh! Well, aren't you special? :anitongue: I don't have that available to me in VB .NET, so I have to do it like the others, and search for a string, then go from there. <_< You must also use feet then, if you're looking for the last decimal there. <_<

Link to comment
I ran the new version (after fixing the backwards slash in your link) and looks like you have a little problem :anitongue: Appears you have accidentally dropped the Negative sign in front of the longitude so all my benchmarks are now in China <_<

 

-YOI-

D'oh! Will be fixed tomorrow morning. I'll also add a flag to use feet instead of meters. I (re-)broke the link so that it can't be downloaded.

 

Yes, you'll get a 404 error until tomorrow morning. <_<

 

UPDATE: New version up now - 1.0.2.95. Fixed the "Benchmarks In China" issue and also made the program so it could handle datasheets that were made in the East longitude. Also added the flag for using feet instead of meters.

Edited by foxtrot_xray
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
×
  • Create New...