Jump to content

UT Topo Mapset almost done - help me pick next state...

IndyJpr

By IndyJpr
in GPS technology and devices

Followers 11

Recommended Posts

Barrikady

Barrikady

Posted
Posted (edited)

For California NED quads, I'm up to Fresno, and done downloading for this morning. I have Garberville, Goldfield, Happy Camp, Hayfork, Healdsburg Hoopa and Isabella Lake to go, for my part. Are the rest being worked on by someone?

Yes. I started from #41

 

So far I have:

 

IVANPAH

LAGUNA HARBOR

LAKE ALMANOR

LAKEPORT

LANCASTER

LAS VEGAS

LAST CHANCE RANGE

LODI

LONG BEACH

LOS ANGELES

 

Help, we can use some more people to download the CA quads...thanks. If you can help, please start downloading from SALTON SEA, #80.

 

.

Edited by Barrikady
Link to comment
Phoenix2001

Phoenix2001

Posted
Posted

For those of you who are sending IndyJpr information other than NED data, it would be helpful if you could check at least some of the POI's, lakes, etc. The data from the USGS (which is often where the state got some of the data) is often out-of-date and sometimes in error. Summit POIs may be a ways off the summit, old reservoirs may be indicated when they no longer exist, etc. That road in the backcountry may not exist anymore. You might be able to fix some of the errors before sending the information to IndyJpr.

 

Thanks to all who are helping!

Link to comment
Marky

+Marky

Posted
Posted
You might be able to fix some of the errors before sending the information to IndyJpr.

How would we know what to fix? I might know my local area, but that 's probably less than 1% of the California data. Do you have some detailed steps on how one might go about fixing the errors?

 

--Marky

Link to comment
Phoenix2001

Phoenix2001

Posted
Posted (edited)
You might be able to fix some of the errors before sending the information to IndyJpr.

How would we know what to fix? I might know my local area, but that 's probably less than 1% of the California data. Do you have some detailed steps on how one might go about fixing the errors?

Note that this doesn't apply to the NED (DEM) data that most of you are downloading.

 

If you only know a small area then there is that much less work to do but anyone going to that area would appreciate the correct data.

 

One person suggested the POI data from the Topografix web site. This is GNIS data converted to GPX format. It can be loaded into Nat'l. Geo. TOPO! and the points compared with the background topo maps. I once found a summit POI that was 1/4 mile away from the actual summit (I reported it to the GNIS manager and it was fixed). Sometimes the feature type is wrong. I.E. "Blue Lake" is actually a lake but the feature type is "stream". I found something like 20-30 errors just scanning around a bit. To find type mismatches, I wrote a program that tried to figure out the feature type from the name and then compared it to the listed type. It output a list of records that needed to be checked.

 

I'm willing to generate a feature type check list for the GNIS data if someone else will check them and get the corrections to IndyJpr in whatever data format he is using.

 

I was particularly thinking of the shape file data for Montana. That's probably similar to the data I can download from the Colorado DOT web site or the USGS SDTS data at:

http://mcmcweb.er.usgs.gov/sdts/index.html

 

In my area I can usually pick out a few reservoirs that aren't anymore. One of the problems I noticed almost immediately with the Colorado topo. was the road going to the top of Pikes Peak going off the north cliff face. A little more scrolling around and I found a road going almost over the top of Ormes Peak instead of around the "base". In mountainous areas, check to see if the curves in the road seem to reasonably follow the contour lines. Those that have a program that can view the data (for instance shapefiles - ArcGIS, ArcExplorer, Global Mapper, etc.) before IndyJpr converts it, might be able to spot some of these problems.

 

Look for data that doesn't make sense - a road through a lake. Maybe it's an old road that was flooded by the reservoir. Maybe the road is drawn in the wrong place. Look for duplicate sets of data.

Edited by Phoenix2001
Link to comment
Marky

+Marky

Posted
Posted
Look for data that doesn't make sense - a road through a lake. Maybe it's an old road that was flooded by the reservoir. Maybe the road is drawn in the wrong place. Look for duplicate sets of data.

That sounds like too much work for a data set as large as California, unless it was split between a group of people. I certainly don't have the time to do it.

Link to comment
IndyJpr

+IndyJpr

Posted
  • Author
Posted
Finished with San Jose. I guess it takes me about 10 minutes or so to do a quad.

Is that 10 minutes for the entire quad (all 4? chunks) or is that for one chunk? If indeed for the entire quad than that is a nice connection you have (and the seamless server must be lightly loaded)...

 

You guys are going to have the NED done really fast. I didn't want to scare you off before :o but the NED downloading is the easier of the two.... :o

 

The NHD (water) data is downloaded by an irregular area (subbasin) so there isn't an easy way to go through the chunks - pretty much have to do some screen prints and mark off the ones you have downloaded. And the process is asynchronous - you pick the chunks and the server says thanks - we'll get back to you..... They do get back to you, sometimes in 5 minutes, sometimes in a couple of hours and sometimes in days... The GOOD NEWS is that there is a lot less data - it'll be 100's of MB for the water data versus multiple GB of elevation data.

 

I'll post the instructions for the water data this weekend.

 

Thanks,

Link to comment
IndyJpr

+IndyJpr

Posted
  • Author
Posted
IndyJpr, how do you decide what water data to keep or do you just keep all the water data. I know in Arizona the water is ridiculous because these streams that "sometimes run" run like once every 100 years.

 

Hi Oz,

 

Well since Utah is my first application of the high res data I'm still doing some testing. Fortunately - for Utah - there seems to be an extra attribute, "IsMajor", which roughly distinguishes between perennial and intermittent streams... I'm not sure what I'll do on the other states...

Link to comment
Marky

+Marky

Posted
Posted
Finished with San Jose. I guess it takes me about 10 minutes or so to do a quad.

Is that 10 minutes for the entire quad (all 4? chunks) or is that for one chunk? If indeed for the entire quad than that is a nice connection you have (and the seamless server must be lightly loaded)...

10 minutes for all 4 chunks. I haven't done any more this evening. I was knitting instead, while I was watching TV. I do a couple now before I go to bed. So far, I've downloaded 6.7 gigs.

 

--Marky

Link to comment
snowfleurys

snowfleurys

Posted
Posted

Finished with San Jose. I guess it takes me about 10 minutes or so to do a quad.

 

The quads I have downloaded from seamless take about 15-20 minutes when at my DSL max of 160k/s

Most of the time its been more like 110-140k/s, or about 20-30 minutes per 'quad' (droped to barely 90 about an hour ago). Seen it as slow as 9, or not even working for days at a time (hardward problems/changes, etc). Then there are the times with 10's of minutes in the extractor stage, building archieve, or waiting for extractor. I've even experienced the number of requests BEFORE me waiting for a extractor INCREASE by 30-40. Many times it has taken 4-5 or even 6 hours to download 2 or 3 files.

Marky, you have had remarkable sucess. My download rate is now down in the 70's.

Link to comment
snowfleurys

snowfleurys

Posted
Posted
Finished with San Jose. I guess it takes me about 10 minutes or so to do a quad.

Is that 10 minutes for the entire quad (all 4? chunks) or is that for one chunk? If indeed for the entire quad than that is a nice connection you have (and the seamless server must be lightly loaded)...

 

You guys are going to have the NED done really fast. I didn't want to scare you off before :wub: but the NED downloading is the easier of the two.... :blink:

 

The NHD (water) data is downloaded by an irregular area (subbasin) so there isn't an easy way to go through the chunks - pretty much have to do some screen prints and mark off the ones you have downloaded. And the process is asynchronous - you pick the chunks and the server says thanks - we'll get back to you..... They do get back to you, sometimes in 5 minutes, sometimes in a couple of hours and sometimes in days... The GOOD NEWS is that there is a lot less data - it'll be 100's of MB for the water data versus multiple GB of elevation data.

 

I'll post the instructions for the water data this weekend.

 

Thanks,

 

Not quite sure how to get this down to the just the appropriate part of the original so I'll leave it all in as this might be helpful for your NHD efforts.

This will be based on my experiences in Oct/Nov, 2007.

NHD is obtainable by 100K quad. I found it by accident while seeing what the data looked like and then took about an hour to find it again. On the NHD-viewer (nhdgeo.usgs.gov) zoom into your area of interest (about half the size of Colorado). On the right side open up base map features. Scroll down to 100K Quad Index.

If this does not appear - you need to zoom in more. Check the box and circle. On left side scroll down. Note that How to Extract now appears - I had to ask them as it never occured to me that such critical info would be initially hidden. Click on polygon extract. With the mouse, make a small box inside the quad you want. Map redraws with selection in blue. Extract window opens - actually before redraw. Top shows info on what is selected. Under 1 - select High Resolution (the default is 1:100,000). Under 2 - select shapefile. Leave 3 unselected. Enter your email address in 4. Select extract. You may or may not get an interim email. The final email will have a hotlink to download the data.

 

Other info (as how is was in 2007)

Yes, more than one quad/basin can be selected - however, the data will all be in one file and the time seemed to go up much more than the individual file sizes would indicate.

Available bandwidth to process was usually more on the weekend - the system is in active use/updating by Federal and State agencies during normal working hours.

The system does go down and seamed to lock-up more if there were large numbers or requests in the que.

Someone at NHD on a work day, needs to realize this and reset the system. Your requests may or may not

be lost. It was quite common for me to find on Monday morning that small requests at numerous times over the weekend would have continous request numbers with only one or two missing.

** IndyJpr please take note ** The data returned will be the FULL feature for everything within or crossing the selected boundary. Waterbodies, areas, flowlines, and lines will not be split at the boundary. Therefore, you will have duplicate features. I would suggest IndyJpr tests how to remove these dupicates, as I remember reading cgpsmapper errors-out if duplicate features are in an mp file.

Also - at the 100K scale many quad names occur more than once (ie Springfield 4 times). The download files produced last fall would include the data for all similiar named quads - and sometimes take days to put together (decreasing the resources available to process other requests). This was especially bad if the quad file name included 'of', as in Craters of the Moon and overedge quads as 'North of ....' There all in one file.

 

Just a few things to be aware of.

Link to comment
snowfleurys

snowfleurys

Posted
Posted
IndyJpr, how do you decide what water data to keep or do you just keep all the water data. I know in Arizona the water is ridiculous because these streams that "sometimes run" run like once every 100 years.

 

Hi Oz,

 

Well since Utah is my first application of the high res data I'm still doing some testing. Fortunately - for Utah - there seems to be an extra attribute, "IsMajor", which roughly distinguishes between perennial and intermittent streams... I'm not sure what I'll do on the other states...

 

IndyJpr,

 

There is a data type field in NHD (actually two, and general word and a more specific number field).

Beaware than some feature types occur as more than just one of point, line, polygon depending on how it was shown on the original 24K. Waterfalls on a single line stream are points, on an area stream they are lines. Rapids can be points, lines, or areas (a series of rapids on a wide stream). Same for dams. Reservoirs, streams, canals, bridges, and levees are also in two catagories. I have an idea of how to handle these, but you and others might differ.

Link to comment
snowfleurys

snowfleurys

Posted
Posted

When I get all four downloads of a quad going at the same time, they average each around 125kbs. With only one going, it is usually in the 300 to 400kbs range.

 

Wow, It never let me do more than 2 at time - would just hang after I pressed download. And never more than half the 160kbs of the DSL. With each file now exposed for twice as long, if seamless burped, both files would have to be restared.

Link to comment
CallawayMT

CallawayMT

Posted
Posted

IndyJpr,

 

Your maps look great, I see that Yogazuu is sending you data for Montana; I would love to get Montana as well. Can you tell me, if I got you a 24k shapefile for the PLSS of Montana is that something that can be added into your maps? There is a 100k shapefile available at NRIS, but 100k does not fit the ground well. I am currently downloading a 143mb shapefile for all of the townships in Montana.

 

Thanks,

CallawayMT

Link to comment
IndyJpr

+IndyJpr

Posted
  • Author
Posted

Hi snowfleurys,

NHD is obtainable by 100K quad.
Thanks for the info! I'll have to check this out.

 

The data returned will be the FULL feature for everything within or crossing the selected boundary. Waterbodies, areas, flowlines, and lines will not be split at the boundary. Therefore, you will have duplicate features.
Yes, I had already noticed this and was wondering if it was going to cause problems... I'll have to do some testing.

 

Thanks again,

Link to comment
IndyJpr

+IndyJpr

Posted
  • Author
Posted (edited)

Hi CallawayMT,

 

Your maps look great, I see that Yogazuu is sending you data for Montana; I would love to get Montana as well. Can you tell me, if I got you a 24k shapefile for the PLSS of Montana is that something that can be added into your maps? There is a 100k shapefile available at NRIS, but 100k does not fit the ground well. I am currently downloading a 143mb shapefile for all of the townships in Montana.

Are you talking about something similar to the Cadastral data found here?

http://nris.mt.gov/nsdi/cadastral/

 

Is that the 100K data you mentioned? I took a look at the statewide set and wow is that a dense set of data but very cool!

 

I think that the data may be too dense if setup with the current zoom levels of the state mapsets. One of these days while I'm waiting for something to compile I'll try making a transparent mapset out of it to see how that works.

Edited by IndyJpr
Link to comment
Barrikady

Barrikady

Posted
Posted

Okay, I'm up to 107 and have 8 left to go. Barrikady, I can start working backwards from 79 if you left me know where you currently are.

 

--Marky

Marky,

 

I have completed IVANPAH, #40, through OAKDALE, #61. I would appreciate your assistance.

 

I am impressed with your Internet connectivity. If I am not mistaken San Jose has Verizon as the local telephone company. Do you have Verizon FiOS connectivity?

 

Barrikady

Link to comment
IndyJpr

+IndyJpr

Posted
  • Author
Posted (edited)

Downloading NHD (Water) Data

 

NHD data can be downloaded from the NHD viewer:

 

http://nhdgeo.usgs.gov/

 

The viewer - when it works - seems to work the best using IE, once the viewer starts up look at the left hand side:

 

nhd_zoom.jpg

 

Click on the "Zoom to State" tool and select your desired state.

 

Now look at the layers dialog on the right hand side:

 

nhd_layers.jpg

 

The square check boxes determine what layers are visible while the round radio buttons determine what layer is active.

 

The following settings are needed:

 

Active: 100K Quad Index

Visible: 100K Quad Index, States (at a minimum)

 

Note that the active layer is indicated at the top of the layers dialog.

 

Please note: Zooming will sometimes cause your visible/active layers to be changed. If you zoom make sure to check your visible/active settings.

 

With those settings the display will look something like this:

 

nhd_view.jpg

 

Now on the left hand side select the "Polygon Extract" tool:

 

nhd_extract.jpg

 

Now in the main display select a 100K quad region. To select a 100K quad region left click and drag a small box inside the region of interest. Please note that if you simply click inside a quad you will get erratic results - you need to drag a small box.

 

After selecting a quad the following dialog will open:

 

nhd_dialog.jpg

 

Select high resolution in the top section, shapefile in next section, leave the third section unchecked and put you email in the fourth section. Hit the Extract button and you should get a confirmation display.

 

At some point in the future (it could be minutes, hours or days and sometimes never) you will get an email with a link for downloading the data.

 

Some Notes

- The 100K quad downloads are required so that I can somewhat easily correct some overlapping issues the data has.

- The system sometime crashes - you'll know when it happens - just check back later and they usually get it reset pretty quickly

- The server can be really slow during prime time, performance is much better late at night or early in the morning.

- Some states, for example Utah, have agencies that have this data available via FTP which is MUCH, MUCH easier. So before you start on a state do some Googling....

 

Let me know if you have any questions....Thanks

Edited by IndyJpr
Link to comment
Marky

+Marky

Posted
Posted (edited)

Okay, I'm up to 107 and have 8 left to go. Barrikady, I can start working backwards from 79 if you left me know where you currently are.

 

--Marky

Marky,

 

I have completed IVANPAH, #40, through OAKDALE, #61. I would appreciate your assistance.

 

I am impressed with your Internet connectivity. If I am not mistaken San Jose has Verizon as the local telephone company. Do you have Verizon FiOS connectivity?

 

Barrikady

I'll take #66 through #79. That leaves you with 4 more to go. Is that ok?

 

Edit: I have AT&T/Pacbell/Yahoo DSL. It was Pacbell back when I started.

Edited by Marky
Link to comment
Barrikady

Barrikady

Posted
Posted

Okay, I'm up to 107 and have 8 left to go. Barrikady, I can start working backwards from 79 if you left me know where you currently are.

 

--Marky

Marky,

 

I have completed IVANPAH, #40, through OAKDALE, #61. I would appreciate your assistance.

 

I am impressed with your Internet connectivity. If I am not mistaken San Jose has Verizon as the local telephone company. Do you have Verizon FiOS connectivity?

 

Barrikady

I'll take #66 through #79. That leaves you with 4 more to go. Is that ok?

 

Edit: I have AT&T/Pacbell/Yahoo DSL. It was Pacbell back when I started.

Thanks for the assistance. I'll grab #62-#65.

 

I also have AT&T DSL. Your throughput certainly seems to be faster than mine.

 

.

Link to comment
Barrikady

Barrikady

Posted
Posted (edited)

I'm done, so it's on to the NHD Data. I'll see if I need any help.

I am also finished with the NED data.

 

I have started to gather the USGS links to the NDH data; it's actually going faster than I expected. I have gone through 15-16 quads and I have received the FTP links to the USGS site that has the data. My request for the data will be available for 14 days from the date of the request. Rather than download the data now, I will download the data within the next few days.

 

Marky, we haven't agreed on a plan for selecting the quads. I will defer to your suggestion. In the meantime I have grabbed links to the NHD data for the following quads:

 

Monterey

Mendota

Fresno

Mt Whitney

Saline Valley

Point Sur

Coalinga

Visalia

Three Rivers

Darwin Hills

Cambria

Paso Robles

Delano

Isabella Lake

Ridgecrest

 

 

.

Edited by Barrikady
Link to comment
Marky

+Marky

Posted
Posted

Yeah, I wasn't sure if I even needed help, that's why I worded it the way I did. I guess I should have said something to avoid duplication. Unfortunately, there wasn't a convenient numbered list. I started at the top and made it through the first 58. I didn't keep a nice list of names like you did. :unsure: Not sure where to go from here. I seem to be having problems accessing the site, do I think I'll just stop for now, and then we can figure out what still needs to be done later.

 

--Marky

Link to comment
Marky

+Marky

Posted
Posted

Okay, so I only seem to have 45 of the 48 NHD I requested and I don't know how to figure out which ones I got and which ones I didn't. :laughing: I have the Reference Polygon ID's of the ones I got, but I have no idea how that refers to a name. Plus, one came back as ID = 0, and that file was way bigger than the others (3 - 4 times bigger). So, I'm pretty much confused about this. Was my problem that I queued up multiple requests, where I should have just submitted one, waited for it to come back, saved it and moved on to the next?

 

--Marky

Link to comment
IndyJpr

+IndyJpr

Posted
  • Author
Posted

How did you get the 100k quad coordinates? Was there a way to do it quickly or?

 

I want to make a list of Arizona quads so I can start downloading all the data.

Hi Oz,

 

I have a 100K index of the US (got it somewhere in Google land), I simply selected all the quads in a state, save the subset to a shapefile and the open the corresponding dbf file within my favorite spreadsheet application. Here's one for AZ:

 

http://www.miscjunk.org/forum/ned/az_quads.xls

 

Hope that helps,

Link to comment
IndyJpr

+IndyJpr

Posted
  • Author
Posted
Okay, so I only seem to have 45 of the 48 NHD I requested and I don't know how to figure out which ones I got and which ones I didn't. <_< I have the Reference Polygon ID's of the ones I got, but I have no idea how that refers to a name. Plus, one came back as ID = 0, and that file was way bigger than the others (3 - 4 times bigger). So, I'm pretty much confused about this. Was my problem that I queued up multiple requests, where I should have just submitted one, waited for it to come back, saved it and moved on to the next?

So far I haven't found any reference whith the NHD zip files that indicate the quad...the way I check is to load them up into GM and see where the gaps are. They may trickle in later - I wouldn't worry about them for now.

Link to comment
IndyJpr

+IndyJpr

Posted
  • Author
Posted
I have all the NED 10m and NHD 24K hydro data for Oregon already on a server. I also have detailed roads and other layers available. Is there a processing outline available that I might take a whack at making this available for "our" Garmin community? I do have copies of Mapsource, gpsmapper, and gpsmapedit applications.

Hi pasayten_pete,

 

Since you have a lot of GIS knowledge/experience you should consider working with shapefiles (versus the MP polish format). Information and examples can be found here:

http://cgpsmapper.com/esri.htm

A tutorial is here:

http://cgpsmapper.com/ESRITipsANDTricks.htm

 

Regardless of which format you choose to work with, the work is the same:

- determine what data sets you want

- find the data

- clean the data (this can take a lot of work)

- process the data (generate contours for example)

- determine your zoom levels (this will probably have to be iterative but you need something for the next step)

- map the data you have to the corresponding Garmin type

- compile you map

 

I realize that's a very high overview. There are a lot of tutorials out on the web plus I'll try to answer any questions you may have.

 

Hope that helps,

Link to comment
Barrikady

Barrikady

Posted
Posted (edited)

Marky,

 

I mailed my California data to IndyJpr today.

 

In addition to the 25 NED quads that you are aware of, I also included about 25 NHD quads.

 

Unless you have grabbed all of the NHD we are probably still short on the NHD data.

 

Go HERE to see the data I sent to IndyJpr

 

I am prepared to grab additional NHD quads, but I just don't know which NHD quads you have already downloaded.

 

What do you suggest?

 

.

Edited by Barrikady
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
Followers 11
Go to topic listing
×
×
  • Create New...