Latest county .dat files?

[Wintertime]

According to the NGS website, the datasheet archives have not been updated since October 2006. I thought there was a place to get month-by-month updates to cover the changes made in a given county since the last archiving process was performed, but I can't find it. So how do I get up-do-date .dat files for an entire county?

 

Patty

[+ltleelim]

According to the NGS website, the datasheet archives have not been updated since October 2006. I thought there was a place to get month-by-month updates to cover the changes made in a given county since the last archiving process was performed, but I can't find it. So how do I get up-do-date .dat files for an entire county?

I've never used the archives because they're always out of date.

 

I'm not sure what other people do, but when I need a whole county then I use the County retrieval method to retrieve the latest datasheets for the county. If you select too many stations, the request will take too long and timeout. You have to select subsets of the list and paste the results together afterwards. I prefer to sort the list by PID to make it easier to figure out where the next request should start. It's not too hard and the datasheets are always up to date.

 

Lloyd

[Wintertime]

I'm not sure what other people do, but when I need a whole county then I use the County retrieval method to retrieve the latest datasheets for the county.

I thought about that, but wasn't sure whether (1) I could download a whole county without something breaking (your suggestion of doing it in batches is a good idea); and (2) the resulting file would work with the DAT2KML awk script. But I just looked at some old .dat and .txt files I have from the NGS database, and they appear to be identical. So maybe I can just change the extension on the file, and DAT2KML will be happy. Anyone tried that?

 

Patty

[+Klemmer]

Patty:

No I haven't tried it, but your question just beat me to it. I was going to raise the same question. I have a bigger problem: the counties down here are so large, to get the whole county, I would have to do it in many segments. It will be a real pain. For example, my home county (Orange) is aver 2800 marks, and I get around to most of the areas in the county fairly often. LA county is over 8000 marks......

 

NGS folks:

So - I sure would like to get some full county .dat files again. There are 6 counties in SoCal that I use from several times per year up to several times per month (some more than others, of course). Can anyone at NGS help me out with this?

Counties I would like: San Diego, Los Angeles, Orange, San Bernardino, Imperial, Riverside.

 

Klemmer

 

Edit: Disregard my comment on Yosemite area if you saw it. My mistake. Above still an issue.

Edited May 9, 2008 by Klemmer & TeddyBearMama

[Wintertime]

Patty:

No I haven't tried it, but your question just beat me to it. I was going to raise the same question.

I went ahead and tried it, and it worked fine. Using Lloyd's tip, to avoid problems, I downloaded the Tuolumne County datasheets in six segments, then used a text editor to combine them. I don't know how many total marks there are, or what the largest segment was, but it worked fine. I copied the file and renamed the copy as .dat instead of .txt; DAT2KML converted it just fine.

 

Still, it would be nice to have updated .dat files on the NGS site.

 

Patty: P.S. I was able to get the Yosemite area counties we will need in one piece.

Cool! I've already done Tuolumne, but will talk to you off-line about getting the others.

 

Patty

[foxtrot_xray]

Since I'm data-scraping, I'd like to second this. I noticed that a few SE stated had their state archived updated earlier this year (as opposed to the others in '06), including my home state. (Which I appreciate!) Would like to see all the other states updated too.

[holograph]

If you have any way to handle shapefiles, you can download the shapefiles by state. The shapefiles have been kept up to date for the most part. It looks like all the states were done around Dec 2007, and a few states have been updated more recently than that.

 

The state datasheet archives, and the monthly datasheet update archives haven't been updated since 2006.

[Wintertime]

Using Lloyd's tip, to avoid problems, I downloaded the Tuolumne County datasheets in six segments, then used a text editor to combine them.

Update: Larry and I have been downloading some other Yosemite-area county files all at once. I.e., select a county, get the listing of stations, select all, and get datasheets for all of them. So far, the only problem either of us has encountered was when he tried to get a county that had more than 1,000 marks. He had to do that in two pieces. But the NGS site has behaved fine for us so far even on counties with several hundred datasheets. So far, no timeouts as Lloyd encountered. Perhaps we're doing it at a time of day when the professional "benchmark hunters" (i.e., surveyors) have all gone home for the day.

 

Patty

[foxtrot_xray]

If you have any way to handle shapefiles, you can download the shapefiles by state. The shapefiles have been kept up to date for the most part. It looks like all the states were done around Dec 2007, and a few states have been updated more recently than that.

Yes - but keep in mind that some of us need more than just the shape files. (For example, if all I needed was GPS coords, then I'd Geocache, and not benchmark hunt! )

 

And some of the SE states HAVE had archives since 2006, as well, like I had previously said.

 

Update: Larry and I have been downloading some other Yosemite-area county files all at once. I.e., select a county, get the listing of stations, select all, and get datasheets for all of them. So far, the only problem either of us has encountered was when he tried to get a county that had more than 1,000 marks. He had to do that in two pieces. But the NGS site has behaved fine for us so far even on counties with several hundred datasheets. So far, no timeouts as Lloyd encountered. Perhaps we're doing it at a time of day when the professional "benchmark hunters" (i.e., surveyors) have all gone home for the day.

 

If it's possible, could either of you (or anyone else that pulls them) send them to me, I'd love to be able to update some of the GE database.

[holograph]

You are welcome to download the archives that I've acquired by going to http://www.holoscenes.com/special/geocac/ds_archive.html

 

These will not be exactly the same as a snapshot of the current NGS datasheets, because the way they were built was :

1. I obtained the NGS archives in 2006 when they became available.
   
2. Each month since, I've downloaded all the datasheets updated in that month.
   
3. I've replaced or added the original downloaded datasheets with the updates.
   

This means, for instance, that datasheets that haven't been updated since 2006 won't have the current NAD83(2007) datum and accuracy estimates. For most purposes that probably won't matter.

 

Legal disclaimer: Use at your own risk! I take no responsibility for the completeness or accuracy of the archives.

 

It was a fairly cumbersome process to generate the 3,219 county archive files and upload them to my web site, so I probably won't update these each month.

[foxtrot_xray]

You are welcome to download the archives that I've acquired by going to http://www.holoscenes.com/special/geocac/ds_archive.html

Thanks! It'll get my GE database up to snuff, then. (And yeah, for myself, anywho, the unupdated marks won't matter.)

 

Is there a function on the NGS site that I missed - that allows you to download only updated sheets?

 

Cheers!

Mike.

[holograph]

You can go to http://www.ngs.noaa.gov/cgi-bin/ds_dates.prl to retrieve datasheets by load date. The retrieval mechanism is the same as the other lookups -- you get a list of PIDs, and you select from them to retrieve the datasheets. By putting in a date range beginning with the last time you retrieved the updates, you get all new updates.

 

edit: P.S. if you are going to download all 3200+ counties, then please pace the downloads so that the site isn't overloaded. The hosting service will probably send me a nasty-gram if the site consumes their server.

Edited May 14, 2008 by holograph

[foxtrot_xray]

You can go to http://www.ngs.noaa.gov/cgi-bin/ds_dates.prl to retrieve datasheets by load date. The retrieval mechanism is the same as the other lookups -- you get a list of PIDs, and you select from them to retrieve the datasheets. By putting in a date range beginning with the last time you retrieved the updates, you get all new updates.

Well, I'll be! I took 'load date' as pretty much the 'set date' of the monument - meaning it would have only given me stations that were MONUMENTED after the date I entered. Al-righty then, thanks.

 

edit: P.S. if you are going to download all 3200+ counties, then please pace the downloads so that the site isn't overloaded. The hosting service will probably send me a nasty-gram if the site consumes their server.

Oh, crap, sorry. I'm stopping now. If you get in trouble or get fined or whatever, please let me know. I'll make it up to you. PM or E-mail me.

 

Thanks!

Mike.

[holograph]

Don't worry about it. The storage and bandwidth are well within the site's limits. It's just that if you have a 'bot or program that is downloading multiple files at a time, or is running non-stop for an hour, it will probably slow down access to the site. And since the site shares a server with other sites, I don't know what other sites are affected. If you want to download all the files, please just do them one at a time, and take a few seconds break between files.

[+Klemmer]

Thanks, Holograph! That was painless, and now I'm updated in all of SoCal for the first time since Oct06. Appreciate your hard work.

Edited May 17, 2008 by Klemmer & TeddyBearMama