monkeykat Posted March 30, 2007 Share Posted March 30, 2007 Hello Everyone, I started this thread to talk about the Google Maps API, and NGS Database of Benchmarks. First off, stop me if this has been done before, but I haven't found an application that displays benchmarks from the NGS Database in a google map. I read on the message board about importing data into Google Earth, however I didn't look too much into it, how easy it is etc... I frequently use the Geocaching web site's Google Maps Beta feature to look for nearby geocaches. It displays the 20 caches nearest map center, color coded by type, as markers on a google map. However when you try to do something similar for benchmarks, I think the best you can get is the 1 mark you are looking for. I really wanted to have this similar functionality for benchmarks in my area, so I jumped right in and decided to develop my own google map application to display nearby benchmarks. So far it has been an excellent learning experience, and a nightmare all in one. What programming project isn't? Its certainly not done yet, but I had some questions to try and gauge community interest. First off, I have found the Google maps API to be extremely user friendly. It works by assigning a maps key to your specified domain to access the google maps info. However one big issue is the gathering of data, in this case, the benchmarks. The Google Maps API can only display data that you have local to your domain. Meaning if I wanted to display all the NGS benchmarks, I would need to have a full copy of their database on my domain. Not really feasable with a database as big as the NGS. I have found a way to import data from a file on a different domain to my home domain with the google map. I can build hyperlinks to access NGS data and return it to myself in an HTML file. This web page: NGS GET_MARK_LIST page describes how you can get NGS benchmark data in a variety of different ways. In this way I can prompt a user for what region they are trying to find data for, then display it on a google map for them. The part I don't like is the format the dadta comes back in. It gives the following: |Dist|PID...|H V|Vert_Source|Approx.|Approx..|Stab|Designation |----|------|- -|-----------|-------|--------|----|----------- |....|DE6651|. 2|88/ADJUSTED|N300613|W0820940|D...|135.032 |....|DE6647|. 2|88/ADJUSTED|N300658|W0821214|D...|51 CMP As you can see, the coordinates provided for the points is approximate. I can use the PID provided to link to the actual data sheet if the user clicks on the google maps marker, that is no big deal. So would the potential user be interested in getting a localized map of the benchmarks they were planning on seeking, even though the data returned would be approximate? I had hoped to also color code the markers based on scaled or adjusted horizontal coordinates, however the GET_MARK_LIST script provided by NGS returns the Vertical Source data, not Horizontal, so I am not sure I can notify users if the horizontal coords are scaled or not. Therefore, I think all points would have to be considered scaled. I hope to maybe have a test application running after this weekend. And if not then, maybe Easter weekend (the wife will be out of town leaving me to fight between benchmark contest hunting, or google maps applications). So does anyone think this might be useful? I was going to start by allowing users to select a state and county and display the marks. There may be issues here because the maps will slow down when over 100 points are displayed, but I think I can use the maps API to limit the number of points displayed at a time. Kinda like how geocaching limits to 20 the number of nearby caches displayed. Then when you scroll the map around, new marks appear and disappear based on your focus. I was then going to allow other forms of querrying the database, like radius etc. My home page with GoDaddy will not ever allow me to have users upload file, as virus could be lodad to their servers, so I don't think I can allow users to submit a file of PIDs to view, but I may be able to have a text box they can supply them to. In fact godaddy won't even let me read the contents of a file so I am experimenting on another site. So I guess I am asking, is there any interest in an application that can do this? Even though I will probably finish this anyway. Has anything like this been done already? Any comments? I will post a link once I have something remotely functional. Quote Link to comment
2oldfarts (the rockhounders) Posted March 30, 2007 Share Posted March 30, 2007 If you use GSAK and the county downloads from the NGS all you need to do is "Export" the database (as a GPX file) from GSAK to "My Documents" and then drag and drop it on Google Earth. Google Earth will open the file and when you click on the waypoint it will open the benchmark page on GC.com. We have the Geocaching Network KML loaded onto Google Earth, so it looks there for the benchmark page. What are the advantages of your program? John Quote Link to comment
+Black Dog Trackers Posted March 30, 2007 Share Posted March 30, 2007 monkeykat - The extremely valuable tool BMGPX (get it here for free) will convert an NGS .dat file to a .gpx file. Google Earth will open a .gpx file (or a .loc file) and show the benchmark locations in it. (The locations are of datasheet accuracy, not approximate.) Click on a location on the Google map and it will show the Geocaching benchmark datasheet. That is the one leeetle catch with BMGPX. It references Geocaching benchmark datasheet URLs, not NGS datasheet URLs. Not necessarily a big problem, but .... However, I wanted to get NGS URLs. So in this topic, I described how to get both NGS and Geocaching clickable icons on the same map, with virtual witness posts! The trick I used was to take the BMGPX .gpx output, operate on it with a SED script (included in my article) to make a similar .gpx file except that it references NGS URLs when you click on its icons. So in Google Earth, open both of these .gpx files and on the Google map, each benchmark has 2 icons (or just one if the mark isn't in Geocaching!), click one of the benchmark's icons and get the GCBM datasheet and logs, click its other icon to get the NGS datasheet. So unless you want to reinvent (part of) the BMGPX wheel (and/or the GSAK alternative), you might want to incorporate BMGPX in a routine (batch file?) instead of using the NGS radius search routine you referenced. By the way, I made a user-friendly use of the NGS radius search here (go to the "Search for PIDs" section and use the last tool in that section; it has the "NGS Radius Search" button). (Anyone can grab this tool and put it in your own website.) monkeykat, I do think you should pursue your ideas further to make a Google map input routine that's all automatic! I recommend including a Quad search, and a County search, as well as a Radius search. I don't think you should try to process people's requests on your website (present or future). Instead make routines that people can download and use on their own computer. Quote Link to comment
68-eldo Posted March 30, 2007 Share Posted March 30, 2007 I also did the NGS download - BMGPX - Google Earth thing and it makes a big difference when looking for benchmarks. But I did not like the choice of icons Google Earth had so I went to custom icon and changed it to this URL http://www.geocaching.com/mark/images/WptTypes/27.gif. That gives you this for an icon. Quote Link to comment
monkeykat Posted March 30, 2007 Author Share Posted March 30, 2007 Hmm, I did some more poking around on the NGS web site, and I found - Shape Files. Has anyone looked at these shape files before? It seems like they provide the data I am looking for, in a rather compact form. here's an example: 11,20070301,http://www.ngs.noaa.gov/cgi-bin/ds_mark.prl?PidBox=OF1452, -77.4423864528,43.0766217000,OF1452,013 NYGS 1969,NY,MONROE,FAIRPORT (1978),43 04 35.83812(N),077 26 32.59123(W),NAD 83,(1996),ADJUSTED, 198.40 ,NAVD 88 ,VERTCON,,,2,Y,,,,,,,1935 ,19950324,MARK NOT FOUND ,USPSQD,,,D It provides: #FeatureId,DATA_DATE,DATA_SRCE,DEC_LONG,DEC_LAT,PID,NAME,STATE,COUNTY,QUAD,LATITUDE,LONGITUDE,POS_DATUM,DATUM_TAG,POS_SRCE,ELEVATION,ELEV_DATUM,ELEV_SRCE,ELLIP_HT,ELLIP_SRCE,POS_ORDER,POS_CHECK,ELEV_ORDER,ELEV_CLASS,ELEV_CHECK,DIST_RATE,ELLP_ORDER,ELLP_CLASS,FIRST_RECV,LAST_RECV,LAST_COND,LAST_RECBY,SAT_USE,SAT_DATE,STABILITY All delimitted by commas. I downloaded an archived shape file for NY State, and it is only 6.4 megabytes in size. Seems relatively small to me, for the amount of NGS data it holds. I think with this data in a compact form, I could easily parse for the lat and longitude, the PID, the name, and provide a link to the data source when the marker is clicked on. Best thing is it may be small enough to store on the server (which is my friends PC). That would greatly simplify the process of getting the data out, because we can use the GDownloadUrl method to "download" the file into the application for processing. I wonder if it will have a problem parsing a 6 megabyte file? Load times? Or will the marker manager take care of that? Another intriguing possibility. Quote Link to comment
Papa-Bear-NYC Posted March 30, 2007 Share Posted March 30, 2007 I'm using Google Maps (not Google Earth) on a new web page I'm building for my own benchmarks. I wanted it to be web based, so I went with the Google Map API instead of Google Earth. I haven't yet got to the point of doing a big database load of many points, but eventually I'll get to it. Here's the link to the site which is in a prototype stage: New benchmarking page. I'm breaking it down by counties. At first Ill put in just the marks I have found or failed to find, but I may consider eventually getting a whole county up. In any case it will depend on server space, response time, etc. I would never contemplate doing the whole NGS database, just the coiunties I've visited (probably 20? counties). When I get further along and have anything useful to share that others may be interested in, I'll post it. A GPX file is exactly the sort of thing I may use to hold the data. Maybe. Quote Link to comment
+Black Dog Trackers Posted March 31, 2007 Share Posted March 31, 2007 monkeykat - Hey, I never tried the shapefiles before; that's pretty cool. So I downloaded my state (Virginia), imported the 20,312 records into Excel, keeping only some of the first few columns, and swiched the order of those columns a bit. Then I added new columns in between the data columns and put them these things in the new columns: <wpt lat=" " lon=" "><url> </url><urlname>Datasheet</urlname><name> </name> </wpt> Then I concatenated all the columns together into one cell per row, and copied-paste-values the resulting new column. Then I put <gpx> into a new row at the top of the file and put </gpx> in a new row at the bottom of the file and saved as formatted text (space delimited). I then re-named the resulting .prn file to have a .gpx extension. In Google Earth, I opened that .gpx file. The whole state was then covered with icons! I could see the traceries of benchmarks along the highways and byways. Also viewing mountains from the side with benchmark icons on top was pretty cool too. An interesting thing was to find that the Chesapeake Bay Bridge-Tunnel has benchmarks all along its length. There's even a CORS station out in the Atlantic Ocean. Quote Link to comment
2oldfarts (the rockhounders) Posted March 31, 2007 Share Posted March 31, 2007 Simply dragging and dropping the BMGPS files on to Google Earth produces this. The first is the entire state of AZ and the second is the northern half of Coconino county. The icons are hot links to the Geocaching pages. With GSAK the benchmark icons will show the benchmark designations instead of the PID# if you so choose. Your shape file doesn't have the benchmark "official history" (description) which is necessary for locating the scaled benchmarks. That is in the GSAK file and can be easily transferred to a PDA for use in the field. What is the advantage to the system that you are trying to put together? John Quote Link to comment
2oldfarts (the rockhounders) Posted March 31, 2007 Share Posted March 31, 2007 I forgot to add this picture.... It shows a few of the buttons available for GSAK. They are (red arrow) the GC.com benchmark page, (blue arrow) a map other than Google Earth in this case, Mapquest, (green arrow) direct link to the NGS datasheet for the highlighted benchmark. John Quote Link to comment
monkeykat Posted March 31, 2007 Author Share Posted March 31, 2007 To answer John's question, finally, I am not sure that any google maps api application would necesarily have more functionality than what you are doing with Google Earth and GSAK. That does seem like a pretty accurate, and not to difficult way of accessing the data. Thanks for posting the images, I was interested to see what it looked like. Its pretty neat looking. I think what I am looking at is maybe more oriented to the newer user, that hasn't downloaded the county info, and maybe doesn't have google earth. With an online web based application, you might be able to just scroll around near your house and see benchmarks nearby. Or maybe if you are on a vacation to cape cod, you could just go to your location, and instantly have the benchmark data available on the web, without having downloaded the data sheets (even though NGS has made that quite easy). Using the shape file, I was able to create an application that displays all the NYState benchmarks. A new problem, its really SLOW! I should have expected this with trying to load and manage 22,000 markers. I have downloaded the Rhode Island database (much smaller) and easier to manage. But it is still pretty slow. I am currently working on managing the displayable markers. I have made it so you need to zoom in pretty close (level 13) before any markers appear. This helps manage the display better, but there is still the initial load time that kills you. Reading online, I have found that organizing data into an SQL database may drastically improve load time. Apparently SQL queries are faster than text file parsing for data. I haven't worked with SQL database, so hooooray! a new learning opportunity. For now, and I make no guarantees of this working, but here is what I made to access Rhode Island. I wonder if enough people try it out if it will crash the web site or server. Test - Rhode Island Benchmarks Quote Link to comment
+Black Dog Trackers Posted March 31, 2007 Share Posted March 31, 2007 monkeykat - Yes this is fun stuff. Thanks for pointing out the shapefiles thing. I always thought it was some sort of quadrangle download or something. It's nice to be able to download data for a whole state for viewing purposes. Today, I wrote a little awk script to use on a shapefile instead of excel. Much easier! One nice thing about awk is that it is free. Here's the script: BEGIN {FS=","} { if (NR==1) { print "<gpx>" } else { url=$3 lon=$4 lat=$5 pid=$6 printf("<wpt lat=\"%s\" lon=\"%s\">\n",lat,lon) printf("<url>%s</url>\n",url) printf("<urlname>Datasheet</urlname>\n") printf("<name>%s</name>",pid) printf("</wpt>\n") } } END { printf("</gpx>\n") } Next I think I'll put in a couple of switches (they're easy to use in awk), like one to switch to geocaching datasheet URLs instead of NGS URLs, another to look for a particular county, or topo quad, using designation instead of PID, etc. Fun stuff! Quote Link to comment
+Black Dog Trackers Posted April 2, 2007 Share Posted April 2, 2007 I wrote another awk script to use on a .dat file. It will also filter by a quad if desired, and has a switch for geocaching datasheet URLs or NGS datasheet URLs. Fun scripts to experiment with. I was going to do the same for Google Maps (making a .kml file; simple enough) but read that Google maps input has to be from a website. This cooled my interest - putting the simplistic .gpx file I made for map viewing of Virginia stations is 3MB in size and the free website I'm using has a daily 5MB limit! So, my interest remains in Google Earth, not Google Maps. Quote Link to comment
monkeykat Posted April 2, 2007 Author Share Posted April 2, 2007 (edited) Looked into the AWK scripting BDT, seems like a pretty powerful language for parsing files. I've been pretty happy with what I can do with Perl, but its nice to see another option out there in case I choose to pursue it. I too, am pretty amazed at what the Shape files have to offer. I expected some kind of lat/lng box coordinates or something. I'm still putting in some effort on the Google Maps API, althoughGoogle Earth definitely seems more powerful at this point. For now the website based input has been killing me. I decided to look into subdomain management to see if I could set up something like a benchmarking.mydomain.com/ type of site. So each maps project I did could have its own domain area. The subdomain was set up kind of like a linking web site. The files are still stored in my main directory at mydomain, they have a separate folder for them called "benchmarking". After a little tinkering I discovered that even though the files are in mydomain\benchmarking, the Google Maps API says I need to get a new key because the website //benchmarking.mydomain.com is sufficiently different than the Key I had for //mydomain.com. So I got a new key for my "new" subdomain. Then I found I couldn't load any data to my website. After about 2 hours of struggling, including going almost all the way back down to "hello world", I discovered that the function I used to get data (GDownloadUrl), considered the data in mydomain\benchmarking to be in the original domain. So the API said I needed a new key because I was in a new domain, but the file download function said I couldn't access the file data because it felt the file was back in the original domain. Grrrrrr I hope this will all be resolved when I switch over to a SQL Database. As then there will not be a question of where and what domain has the data, it will all be in one database. I'm still enjoying this experimenting more than the programming I am doing at work though... Edited April 2, 2007 by monkeykat Quote Link to comment
+Black Dog Trackers Posted April 2, 2007 Share Posted April 2, 2007 monkeykat - Actually AWK came first and then about 10 years later, PERL was made, I think, as sort of a superset of AWK. After doing some PERL for a while, I decided years ago to switch back to AWK (actually a version of it called GAWK, available here for free) since I prefer its simplicity of operation. It sounds like you have some interesting web challenges to solve. Let us know when you have something for us to try out. Quote Link to comment
atari52oo Posted April 3, 2007 Share Posted April 3, 2007 Monkeykat, Just tried out your Rhode Island test and I love it!! I think your going in the right direction. I look forward to seeing where this project goes. The simplicity of Google Maps is so nice some times... Quote Link to comment
Papa-Bear-NYC Posted April 3, 2007 Share Posted April 3, 2007 (edited) As I mentioned, I've been working on this for my personal web page. The emphasis is not the same as is a general purpose resource, but it may be something useful to some. I have a prototype up of a couple of counties. The link here is for Bergen county, NJ. When it comes up (yes it's a bit slow) you will see a large number of white tear-drop icons plus a few green and a few red ones. The green are those I have found, the red, those I looked for but couldn't find. The map sure does show where I've been looking (along the Palisades). If you hold the mouse over any icon, it will show the station PID and Designation. The green and red are clickable: you will get a page on that particular station. Since this is only a piece of a larger app. there is no real navigation. The process was roughly as follows: 1) Dump the NGS county downloads into GSAK (already did that) 2) use GSAK to keep track of my finds, DNFs, etc. by using various fields. (Starting to do that) 3) Export the GSAK county database to CVS format file. 4) Bring that up in Excel. Use Excel to turn it into a set of javascript variable (use perl if you like). 5) Put them into the Google Maps API. I use Google's MarkerManager but I'm not sure how much it helps with the speed. Here's the link: Test of Google Maps for Bergen County Feedback welcome. If you dare, here's NY's Westchester county: Westchester. It's got almost 900 marks, about twice as many as Bergen. Papa Bear Edited April 3, 2007 by Papa-Bear-NYC Quote Link to comment
mloser Posted April 3, 2007 Share Posted April 3, 2007 By the way, shapefiles are files formatted to work with ESRI mapping applications such as ArcGIS and ArcMap, which are commercial mapping applications. The shapefile format has become a standard in the mapping field and is used to represent geographic information. For some more info check out Wikipedia Shapefiles. Quote Link to comment
monkeykat Posted April 25, 2007 Author Share Posted April 25, 2007 Time for an update. I have been diligently working on my google maps application to display benchmarks. Instead of trying to parse the shape files in their text form, I have written a very easy script that creates a SQL Database for the markers by State. It easily handled the 2450 markers in Rhode Island, and also seems to have added the 22000+ markers in NY State. For now, I just have two pages running. One for Rhode Island, and one for NY State. The app displays only the first 100 benchmarks in the viewing area, but informs the user if there are more. Details are given on the web page on how this works, and how I hope to have it work. I color coded the marks with green for good condition, white for monumented (as they can be in any condition), and red for everything else (usually not founds). There are memory concerns, and even though only 100 marks are displayed for very wide zoom angles, you will still have slow down in your browser. Rhode Island Benchmark Viewer New York Benchmark Viewer I really like the way this turned out, as I can scroll around and look for clusters of marks, and then go to the NGS page and download counties, radius, or just type in the PIDs I want to find. I hope to add a text box for users to enter addresses, or coordinates to jump the map to instead of scrolling. I think, since I am closing in on 100 recoveries, this application is good for now. For people with many recoveries, the Google Earth/GSAK is probably the way to go. Those apps will let you keep track of how many marks you have searched for already, while with this one, you need to just remember where you went. Enjoy, comments welcome, and its still under development. I am planning on adding Pennsylvania and Massachusetts soon. If there is a state you would like to have to goof around with, I think I can add it in minutes with the scripts I have already written. But, I am going on vacation this weekend, and have a full slate of activities for the next few nights. So I can't make any promises as to when I can update things. Quote Link to comment
+NorStar Posted April 26, 2007 Share Posted April 26, 2007 Wow! I just poked around the Massachusetts one you put together (must have just put it up). Really slick! I had been using downloaded county files and put them in an access database, then used MapPoint 2004 to view them. This was fine until recently - for some reason, I can no longer link to any files. This is a nice substitute (the only issue is that I would have to be online to use it - small concession). Maybe some day, I'll 'fix' the other problem or learn how to use the way using GSAK/GPSBabel/GoogleEarth metioned earlier. Quote Link to comment
monkeykat Posted April 26, 2007 Author Share Posted April 26, 2007 I added Massachusetts and Pennsylvania last night. I also made the index.html page a startup page with links to each page dedicated to a state that I have completed. Like a true index. Benchmark Test Viewer State Index Plan on adding the little scale in the corner to show distances. I also noticed that if two marks are at the exact same coordinates, only 1 will be displayed when the user clicks on it. I'll have to figure out how to display more. I think I may be able to do a tab-ed marker display. I think in the future, I will prbably dedicated a domain to these viewers. Something like "benchmarks.scaredycatfilms.<om" or similar. Then update the meta data, and it'll be ready for all to see. As a side note, google maps has very detailed images of areas in North Korea. You can view the capital Pyongyang and see an even more detailed satellite view than about 5 miles down the road from me. Interesting thing is that it is broad daylight, and there are almost NO cars on the road. What an interresting country. Quote Link to comment
+Klemmer Posted April 27, 2007 Share Posted April 27, 2007 monkeykat: Your setup is very, very slick. It operates quickly, and simply. Great work! I enjoyed looking for (and easily found by memory) some of the marks I recovered (or not) in the Buffalo and Niagara Falls area of NY (relatives there, vacations). This is WAY better than what the NGS has on line. Dave and Casey - you guys should see about hiring monkeykat as a consultant (or whatever), and use his setup for ALL states! I'm not sure how closely Dave and Casey (from NGS) monitor these threads - someone should point them here, so they can check it out. I can, but would that be OK, monkeykat? I don't want to "railroad" you...... Meanwhile, monkeykat, what would it take to get you to setup California? HUGE number of marks (almost 63K marks in 2005), although just the southern half would do it for me.... You have an hourly rate? I've got some broadband server space and a URL, currently being used only for our family picture Gallery...... What would it take? PM me if I can do anything. Oh yeah, one suggestion: I wouldn't mind seeing another link for the GC page (if there is one). Klemmer Quote Link to comment
monkeykat Posted April 27, 2007 Author Share Posted April 27, 2007 Meanwhile, monkeykat, what would it take to get you to setup California? HUGE number of marks (almost 63K marks in 2005)... Oh yeah, one suggestion: I wouldn't mind seeing another link for the GC page (if there is one). Klemmer Done. I think California is up and working now from the index page. I was also thinking of adding the geocache link, which was pretty easy to do. When searching the maps I saw recoveries by "GEOCAC" and wanted to see the pictures and logs, but there was no link. Now there is. Enjoy. Quote Link to comment
+Klemmer Posted April 28, 2007 Share Posted April 28, 2007 Yep, California is up and working fine. It's a great tool. I find it an improvement in some ways over what I was doing, which was county gpx files dropped onto Google Earth. In particular, it will save time before traveling outside the local area(s). It also adds an extra dimension of the color coded ballons, so you can get some idea of type, recent finds, etc. Thanks a million! Quote Link to comment
+Klemmer Posted April 28, 2007 Share Posted April 28, 2007 Hmmm.... Some interesting observations: 1) Resolution: I always assumed that Google Earth and Google Maps (in the so called "satellite" mode, which are aerial photos at the lowest "altitudes"), used the same server database. Evidently not. The Google Earth version with Monketkat additions looks WAY better in my area than Google Earth Plus. Your area may differ. 2) On the other hand, the accuracy of an adjusted mark placement on the map is better on Google Earth. I'm sure it's area dependednt, and is not Monkeykat's doing, but Google Earth's "registration" of map vs Lat/Long. Again, your area may differ. Go figure. Always a trade-off, huh? I still like the Monkeykat version. Quote Link to comment
jasir Posted May 1, 2007 Share Posted May 1, 2007 Greetings from AUSTRALIA DO YOU KNOW OF ANY benchmarks IN australia ? Jasir ( Joel ) Quote Link to comment
+Ernmark Posted May 1, 2007 Share Posted May 1, 2007 Greetings from AUSTRALIA DO YOU KNOW OF ANY benchmarks IN australia ? Jasir ( Joel ) ..there aren't any in the Geocaching database, but you can log & visit Australian Trig Points on Waymarking.com Quote Link to comment
monkeykat Posted May 18, 2007 Author Share Posted May 18, 2007 Let's get this thread back on track. I've been working to fix up some of the backend stuff for the google maps API I am working on. Sunny weather and the benchmarking contest have been foiling my coding attempts, but I'm getting it done. I have been working on sliming down the SQL database I store the shapefile data in. Since all that data counts against my disk quota, I am only keeping the essential data from each shape file. I have my shapefile loader php script working great, so adding new states to the database is a breeze. (I do have tons of disk quota left, it just makes sense to slim it down in case I need that space for the future) I've also been trying to get a generic "state page" template done, since all 50 states will need a similar looking page, its best to finish that up before I create 30 states, and have to modify them all by hand. 1 template for easy modification. Then small customizations later. I moved the latest maps to a subdomain on scaredycatfilms. The newest maps can be accessed from: http://benchmarks.scaredycatfilms.com/. Soon I will take the old pages in the /maps directory down, and redirect users to the new ones, if people haven't gone there yet already. I added Hawaii last night as a test to see how quickly I can add a state. Only took about 10 minutes for the first try. I think the time will drop down to only a few minutes once I refine my process. I am going to try and work on clustering soon to increase speed and preformance. Right now the pages still seem to bog down when viewing states at a high level. That's because I pass back all the potentially viewable points, but only display the first 100. So if you chose to view the entire state of California, your browser memory would be filled with 66,000 points, but only displaying 100 of them. I'll see if my clustering works, and if it makes a difference. Quote Link to comment
Papa-Bear-NYC Posted May 18, 2007 Share Posted May 18, 2007 (edited) Good job Monkeykat. I checked out the latest version. Massachusetts is my home state, so I checked around where I used to live. As you know I've taken the county approach using county downloads, but the state wide shapefiles are also appealing. I guess you could say I'm building up from the county level and you're building down from the state level. I've had some success at clustereing using the GoogleMaps MarkerManager. Here's a recent version of my application for Suffolk County, NY which has about 2000 stations: Suffolk County. The big white marketrs spread across the county each represent 32 stations. If you click on one of them it will zoom you in to see those markers. The GMap MarkerManager manages the whole process (after you feed it the data). Here's one of the US-Canada border along the "Highlands" - a ridge line 175 miles long in Maine and New Hampshire. The Highlands. There are 3500 stations but they are concentrated along the line, so it gets really dense. That's why it takes a few extra seconds to start up - it's loading it all into the MarkerManager. For The Highlands, I separated the stations into 3 categories: one group (67 Triangulation stations - gray markers) has no clustering, a second group (194 boundary monuments - green markers) has a 18 in a cluster, zooms in by factor of 8, and the last (3219 Line marker disks - red markers) has 140 to a cluster and zooms in by factor of 64. So if you click on a green cluster marker and click on the link you will see the boundary up close, but not the red markers yet. Click on a red cluster marker and you zoom way in and see everything. If you click on any single marker you'll get a page for the station with a close up map. When you're up close in this area, the Google Map type is useless (no roads or towns there - just trees and mountains). Use the pull down menu in the upper right of the map to switch to USGS Topo. Question to the Google Earth users: How do you manage large numbers of stations packed at high density on your maps? It would be good to see what other schemes folks have come up with. Edited May 18, 2007 by Papa-Bear-NYC Quote Link to comment
+Klemmer Posted May 18, 2007 Share Posted May 18, 2007 (edited) PapaBear: Until recently, I have just dumped a whole county .gpx into Google earth. Being in a dense area (SoCal), I'm talking many thousands of markers in a county. I'm at work without the numbers, but they are large. (edit - LA county is 8189 marks, last .dat file). They all try to display on the screen in google earth. They just keep loading until the whole county is red (or whatever color). With recent hardware (6 months or so, mid Dell grade), it's not too annoying. As you zoom down, of course, they stay put, but get thinned out. Works reasonably well. BUT: You BETTER close all open marks before you quit GE. If not, it crashes trying to load them all, then you have to go into the GE (or other) directory and delete a file...."My Places.kmz" .... it's the "user preferences" (init file, in DOS terms). THAT is annoying if I forget to close open counties before quiting GE. I'm watching what you code experts are doing with interest. I guess I'm an internediate type user. At the momemt, I'm enjoying the GAWK scripted files BDT has done. The additional pop-up info is great. But with all of a state (or region) available on-line, without doing all the .dat -> .gpx -> GE conversions, your approaches are very promising, especially for someone traveling a lot, or maybe less than an intermediate level user (or whatever). I can envision NGS adopting your approach(es). They are already better than what NGS has on line (which I rarely use). What a great tool! Imagine it "real-time" from NGS, or maybe day old data. (edited as above to add LA county at 8189 marks and GE file name) Edited May 19, 2007 by Klemmer & TeddyBearMama Quote Link to comment
Papa-Bear-NYC Posted May 18, 2007 Share Posted May 18, 2007 ... They are already better than what NGS has on line (which I rarely use). What a great tool! Imagine it "real-time" from NGS, or maybe day old data. Ha! You know Holocene can't even get monthly updates anymore. It's a shame but I don't think it's technology they need, it's more man- and woman-power to run their important department. But I do digress, sorry! I'm waiting to see what Monkeykat gets out of server side database access. I may move in that direction if things get bigger. (My son promises to help me set it up) I know just putting up the counties I have or might visit may keep me happy, but I'm also thinking of the general benchmarking public who have no great interest on what mark I visited last week, but rather would like a quick look at someplace they may visit next vacation. So keep in touch, and if you look at something monkeykat or I put up, I aassure you we really appreciate feedback. Quote Link to comment
+Black Dog Trackers Posted May 19, 2007 Share Posted May 19, 2007 (edited) Question to the Google Earth users: How do you manage large numbers of stations packed at high density on your maps? It would be good to see what other schemes folks have come up with. Well, Google Earth just handles it somehow. I have put the shapefile for the whole state of Virginia and it works fine. California puts a strain on my computer and I find that its best to move over to Arizona, zoom in on the map and then sneak back into California. But all 67,000 marks show in California. If you give the GAWK stuff I did a try, you'll see LOTS of benchmarks on your screen! I loaded the whole state of NY and also NJ at the same time and they came up in Google Earth just fine. Zooming in and out on them is no problem. A .dat file for a whole county is more manageable of course, as is loading up 3 or 4 counties to Google Earth at the same time. Getting them from County archive files instead of state shapefiles gives lots more data in the popup balloon too. ========= Klemmer & TeddyBearMama - I have not noticed the problem of sticky files that you did. Every time I close down Google Earth it asks me if I want to save the files I opened with it and I say No. Maybe I have a newer version or something. Edited May 19, 2007 by Black Dog Trackers Quote Link to comment
wvnewbie Posted May 24, 2007 Share Posted May 24, 2007 Instead of starting a new thread (maybe I should be, but.....) can anyone tell me in the most simple of terms if you can get benchmarks on google earth (like you can geocaches) and how. I have to be honest - I am new to this whole thing, and especially benchmarks - and my head is spinning as it regards benchmarks (ie: you found this - but you didn't find IT, etc etc etc). I want to look for benchmarks as a way of complimenting my geocaching hobby, and that is why I would like to see benchmarks on google earth. Any help would be awesome - thanks! Quote Link to comment
Papa-Bear-NYC Posted May 24, 2007 Share Posted May 24, 2007 Instead of starting a new thread (maybe I should be, but.....) can anyone tell me in the most simple of terms if you can get benchmarks on google earth (like you can geocaches) and how. I have to be honest - I am new to this whole thing, and especially benchmarks - and my head is spinning as it regards benchmarks (ie: you found this - but you didn't find IT, etc etc etc). I want to look for benchmarks as a way of complimenting my geocaching hobby, and that is why I would like to see benchmarks on google earth. Any help would be awesome - thanks! There's a recent thread on using Google Earth with benchmarks: Thread There are various options from simple dumping a county download file or GSAK file onto GE, to some excellent work done by BDT to massage the data in to a more useful form. Read through that thread and you will probably come up with some questions you could ask on that thread. Good luck Pb Quote Link to comment
ArtMan Posted May 25, 2007 Share Posted May 25, 2007 I tried to download some shapefiles to play with from the NGS site, but they are returning empty files (or empty ZIP containers). Is this a known site problem or is it a known ArtMan problem? TIA, -ArtMan- Quote Link to comment
monkeykat Posted May 25, 2007 Author Share Posted May 25, 2007 I tried to download some shapefiles to play with from the NGS site, but they are returning empty files (or empty ZIP containers). Is this a known site problem or is it a known ArtMan problem? TIA, -ArtMan- Doesn't look like just an 'ArtMan' problem. I was unable to download shapefiles when using the "Shapefiles" button. I tried NY State and selected Monroe County. I got an empty ZIP file. I used the Archived Shapefiles button in the past to grab information by complete State. That one still appears to work, as I could download Delaware this morning. I've only used the Archived link in the past, so I am unsure if the empty ZIP files are a result of the upgrade, or a problem from before the outtage. Quote Link to comment
monkeykat Posted May 29, 2007 Author Share Posted May 29, 2007 Had a long day of geocaching/benchmark hunting on Saturday, so I took it easy on Sunday, meaning I was able to play around with the Google Maps Application for a while. Recent Updates: 1. Added a Clustering Algorithm to try and average the number of marks available in the viewing area and display a maximum of 100 at a time. Works pretty well I think. Has a problem where it displays a maximum of 199 marks at a time, but once you reach 200 it has enough marks to average. Also, near water you can get some odd marks, like Cape Cod where it averages some mainland and some cape marks and gives you a marker in the ocean. 2. Tried to update the FAQ a little. I will link to the benchmarking FAQ on the geocaching page at some point. 3. Update the state page "template" to have a larger map and link to the other states. 4. Added Arizona, Maine, Vermont, New hampshire, Connecticut, and New jersey to the database of states available to view. I will add more soon, so if you want your state now, send me an email. The index page will remain: Benchmark Viewer Home Page Quote Link to comment
monkeykat Posted June 14, 2007 Author Share Posted June 14, 2007 Another status update. I have all 50 States added to the online database. I think I will add Washington DC soon, as it has a separate shape file. Enjoy Everyone. Maybe I'll add some of the US territories if I have the time... any interest in Guam? I hoped to add a polyline around each state, but I haven't gotten them to work yet period. Let alone around a state. It bears more investigating. Expect some minor tweaks here or there. But mainly cosmetic, I think. The shape files are all loaded in the database. Statistics Are as Follows: States: 50 Apprx Marks: 815,500 Apprx Size: 100MB Keep in mind, I only saved the PID, html link to the data sheet, LAT, LNG, PID, Name, Last recovery, condition, and by whom, to keep the data size down. As always, the link should remain the same: http://benchmarks.scaredycatfilms.com/index.html Quote Link to comment
+YeOleImposter Posted June 14, 2007 Share Posted June 14, 2007 Very cool application! I have bookmarked it for regular use. Nice to be able to get a quick visual of the Adjusted/Scaled Reported/Unreported status of benchmarks in the area. One addition that would make this even more valuable is if you could incorporate Topo maps into it like http://mapper.acme.com has. I use this map all time out of GSAK. Thanks again!! Quote Link to comment
monkeykat Posted June 20, 2007 Author Share Posted June 20, 2007 Another day, and another roll out of the Benchmark Viewer web site. This time I have made a few visible changes, and some behind the scenes. First, the behind the scenes... I have been able to create a State page template so that the only editable regions are the state name, and the major cities. The rest of the page is a template and making one change, changes all 50 states at the same time. This will make updates extremely easy. If I decided to make the map wider, then just change the template, and the change is rolled out to all the other files. Also, I combined my javascript to one file, so again, a change there and it affects all pages. Now the visible changes. I added Topographic and Orthographic imaging. The ortho is very useful in viewing some areas that may not have detailed google satellite views. The topo is extremely useful, and pretty self explanatory. I also added Washington DC, as it had its own shape file. Important: I had to change two files because their two letter abbreviations were reserved words in SQL. For that reason, Indiana, which was IN.html and Oregon which was OR.html can now be found in IND.html and ORE.html. I removed the old OR and IN html files, but tonight I will replace them with a redirect page to the appropriate html page. Enjoy. Quote Link to comment
+Klemmer Posted June 20, 2007 Share Posted June 20, 2007 WOW! Nice work. Greatly improves the usefulness. So when does NGS link start linking to your website? Quote Link to comment
+Black Dog Trackers Posted June 20, 2007 Share Posted June 20, 2007 Nice site ! I especially like the ability to select a topo map view. Quote Link to comment
Papa-Bear-NYC Posted June 25, 2007 Share Posted June 25, 2007 (edited) Hello all I've got some nice new features and lots of enhancements for navigation and presentation for my benchmarking/maps web site. I've been slow in putting up the content, but I've got things to Look the way I want (almost) and since all the software is template, I can add content now without changing any of the design or look and feel. This started out as a way to present my own benchmark hunting and my finds on my web site, but it has suffered what is known as "Mission Creep". Once I got going I figured this was great stuf and why not add XX, then add YY, and so on till I included everything short of curing cancer. But time constraints have had their influence and I guess what I've got is what I'll get. The maps form a three level heirarchy. Region County and Station. Each level map is saved so you can switch back and forth without having to reload. All map context is saved when switching. This becomes a great win especially with the internet response being less that superlative, since the slowest part is the initial loading of the map. The County Map has the most data (= slowest to load ) and should be of the most interest to this group. It has two neat zoom features: one I call the "Custer zoom" which will zoom in from a cluster marker to the underlying stations represented by the cluster. The other which I call the "Box zoom" is something I found on a Google developers group and adapted. You basically draw a rectangle on the map and it zooms in making that rectangle the whole screen. I also use the location window on the top of the browser to save where you are in the application so you can use the back browser button (not working in IE yet) and save it as a link or bookmark. So if you've found a station or area you like, you can get back to it with one fell swoop. Lots of other little niceties, but that would be too much for now. So far I've laid out two regions (which are arbitrary, but could be states of other areas): NY Metro Region and International Boundary Region (has a very high density of stations) I've set the link to point to the first - you can see the parameters in the URL and get the idea of how they work. The NY Metro Region Map will come up (22 counties in NY, NJ and CT) and Monmoth County is pre-selected as the "Current County". Of course you can pick on any other (click on the map or the list) if you like. Once you pick a county, you can "mouse-over" a cluster marker and zoom in, or use the box zoom Here's the link: Benchmarking Maps web page In the next note I'll take anyone who is interested for a walk through for a bit of playing. Edited June 25, 2007 by Papa-Bear-NYC Quote Link to comment
Papa-Bear-NYC Posted June 25, 2007 Share Posted June 25, 2007 (edited) So ... Let's say Harry Dolphin is going out next Saturday. He thinks he'll head down to the Jersery Shore, maybe Ocean or Monmoth County. So he brings up the region map in the above note. He sees Monmoth County is the "Current County", so he decides that might be interesting and clicks on the "Monmoth" label on the map. (Besides, he hates the Garden State and wants to stay a bit closer, so no Ocean County this week.) Now he notices the cluster of markers on that spit of land at the north end of the county. That's Sandy Hook. So he clicks on "Box Zoom" from the zoom controls at the top left, moves the mouse to just above and to the left of the end of Sandy Hook and drags the mouse over the area, A yellow rectangle will outline the selection. He releases the mouse and whamo!, the end of Sandy Hook is zoomed in to fill the whole window. "Wow! Look at that, there must be a zillion markers in that tiny area" (says Harry). But what are all those labels cluttering up the map? Hey those are the PIDs. He thinks that's handy but there are just too many of them. So he clicks on the box at the top next to the zoom controls labeled "Toggle Markers, Labels On/Off" and a little drop down list appears. He clicks the bottom one "Turn Labels Off", and whoosh, they're gone. He decides he doesn't like the boring map, so he goes to the drop down selector in the upper right and selects "Hybrid" as a map type. The map redraws under the markers and he gets to see the sand and sea. "Hey maybe I'll go to the beach this Saturday" (he says). Now to get serious. "I recall there's a famous light house there". so he does another "Box Zoom" over the most concentrated area inside the hook on the west side just south of the tip. (He makes sure not to click "Box Back" or he'd be back where he started!) Now there's a nice closeup look. The white markers are bench marks and the gray ones are triangulation or intersection stations, so he checks out the gray ones. He holds the mouse momentarily over a few markers and a little box pops up giving the name of the station (this is called a "tool tip"). He finds an interesting one down by the Coast Guard Station which says " KV4987 - SANDY HOOK LH LIGHT 1835". Pay dirt! So he clicks on the marker itself and up pops an info window (some folks call these "bubbles") with some specs on this station and a little map. He clicks on the link in the info window and the station page comes up for this station. When the station map comes up, he checks that out and then he clicks on "NGS" in the top panel and gets the datasheet in place of the map (yes, the map is saved). After checking this out, he wants to check out some of the other stations, so he clicks back on "Current County" and he's back looking at the county map. . . . . . Harry does go to Sandy Hook, bags 34 marks and has a great time. (And all because of the maps ). Edited June 25, 2007 by Papa-Bear-NYC Quote Link to comment
monkeykat Posted June 28, 2007 Author Share Posted June 28, 2007 I have to say I haven't given papa_bear enough credit in this thread. He has a lot of neat features on his benchmarking pages, and has pushed me in new directions in my development of the scaredycatfilms benchmark viewers. I am glad he posted a link to his site here, as it is well done and very interesting to look at. Things will cool down a little while development wise on the viewer, as I think they have reached a pretty stable point, and vacation season is approaching for me. I hope to add some sort of mark labeling like A_Tapeworm suggested, so that you can take the google map with the benchmark markers on it and have a label beneath them. I hope to make it somewhat configurable, so you can turn it on or off, and have it only appear when you are zoomed in relatively closely. That way you won't have hundreds of labels all over and not be able to figure out which is which. I also want to add some warnings about not removing disks or damaging them, as the recent article in American Surveyor was a real eye opener for me, and I hope that this viewer doesn't encourage people to seek out and destroy the marks. Quote Link to comment
andylphoto Posted June 28, 2007 Share Posted June 28, 2007 Things will cool down a little while development wise on the viewer, as I think they have reached a pretty stable point, and vacation season is approaching for me. I also want to add some warnings about not removing disks or damaging them, as the recent article in American Surveyor was a real eye opener for me, and I hope that this viewer doesn't encourage people to seek out and destroy the marks. Monkeycat, I have not commented previously, but I want to thank you immensely for your work on this viewer. It has been very helpful for me in visualizing the general location of marks. Organizing datasheets for an outing in order has gotten much easier. The viewer is very helpful for this and other reasons. I also like having the link for both the NGS and GC pages. When you get back into development, something to consider, and others can offer input as well. I don't even know if this is possible. Any way to make it so that the balloon appears for a particular mark when the mouse is hovered over it, and disappears automatically when the mouse is moved? This is something I'd like to see. Having the box "sticky" when clicked (the current behavior) is nice. But there are times when I'd just like to see the info, and not necessarily have it stay. Whether or not this comes to pass, I'll continue to enjoy your work. Thank you again! Andy Quote Link to comment
Papa-Bear-NYC Posted June 28, 2007 Share Posted June 28, 2007 (edited) When you get back into development, something to consider, and others can offer input as well. I don't even know if this is possible. Any way to make it so that the balloon appears for a particular mark when the mouse is hovered over it, and disappears automatically when the mouse is moved? This is something I'd like to see. Having the box "sticky" when clicked (the current behavior) is nice. But there are times when I'd just like to see the info, and not necessarily have it stay. Whether or not this comes to pass, I'll continue to enjoy your work. Thank you again! Andy I'll jump in here on this question. The answer is yes. An InfoWindow can be opened on a variety of events. What you want is a "mouseover" event. If you look at my Monmoth County map form the above link, the large white icons have this function enabled. I'll warn you however, that using mouseover to launch these InfoWindows can get annoying. Just moving the mouse across the screen to get to a marker to click on can bring up unwanted InfoWindows, which might actually obscure the mark you homing in on. (Actually I turn them on but not off. But you can also do that.) For crowded maps, I prefer the "tooltip" idea, where a very small window with information in it appears. The nice thing about them is you have to place the mouse over them for a second or two for the thing to appear. Microsoft uses them all the time and if you put the mouse on most icons in this Forum, they use them too (these too can get annoying. Like the ones that come up when you're trying to look at a listing of the content's of a folder). I use tooltips for my low level markers. You can see these if you zoom in on the county map and thereby expose the low levels markers (just click on the link in one of those annoying balloons that pop up over the big white markers ). Here's the link again for Monmoth county: Map Edited June 28, 2007 by Papa-Bear-NYC Quote Link to comment
monkeykat Posted July 4, 2007 Author Share Posted July 4, 2007 Well, since rain has thwarted my July 4th Geocaching (I was originally going to go out rain or shine, but then decided I really didn't want to look for a bunch of tricky micros in the heavy rain), I have decided to add a few things to the benchmark viewers. As a couple users suggested, I have been able to add a toggle button to turn on and off PID Labels from each of the markers. It will only add the label on non cluster markers. Also, it slows things down a little so I suggest leaving the labels off while you search, and then turn them on when you are zoomed into your search area. I have also added the "tool tip" labels that show up over marks, much like when you put your mouse over an image and see the ALT text. These are on all the time. I also turned back on the scrolling that shows the complete Marker Info window. I had this turned off because the map would always try and refresh when the scroll was completed. Now I have the refresh turned off whenever an info window is open. Hope this works well for everyone. Added some warnings about not removing markers. Enjoy. Quote Link to comment
+PilotMike Posted August 8, 2007 Share Posted August 8, 2007 This is cool. I found a link in another thread to the scaredycat page before I made it to this thread. (I had a bunch of threads open in separate tabs). I began reading this thread at the beginning and thought that I really should put a link to that site so everyone can benefit. It would be great for what they're talking about. It turns out, this was the brainstorming behind that great site I just found! Congrats and keep up the good work! Now you just need to get it linked or built into the benchmarks' sites on gc.com so we can view the map from there... Quote Link to comment
monkeykat Posted October 1, 2007 Author Share Posted October 1, 2007 I got back to doing a little work on the benchmark viewer this weekend, since I didn't have enough time to plan any benchmark or geocaching excursions. One thing I had been working on is generating LOC files for the GPS. I wanted to have a one stop shop for datasheets and LOC files. Before, my process was to use the viewer to seek out marks and get the data sheets. Then I needed to go to geocaching.com and find the marks I was interested in and get the LOC files. The good news is that I think you can all do it from one site now. I added a Generate LOC button to the PID Viewer on the right side of the screen. When you click the button, all PIDs you have in the list will be added to a text area below the google map. You can hide or show the text area using two additional buttons. The LOC data is erased and generated again each time the button is pressed. The user will need to copy the data from the text area and place it in an application such as notepad. Save the text as a .LOC file and you are ready to go! I would have liked to create the file for you, but my service provider prevents that type of dynamic operations. I think it could potentially be used by people creating malicious programs on their servers dynamically. I also had trouble trying to create the file in a popup window because Internet Explorer kept trying to interpret the XML instead of treat it like text.' Enjoy, and happy benchmarking. Quote Link to comment
monkeykat Posted October 1, 2007 Author Share Posted October 1, 2007 I hope all users are enjoying the benchmarking viewer site. I thought I would share some of the statistics gathered by my service provider. These statisitcs are for my entire domain, which includes the Main Web Site ScaredyCatFilms.com as well as the benchmarking site: http://benchmarks.scaredycatfilms.com/index.html Most of the stats should be hits from the benchmark viewer, although I do get a surprising amount of hits on the images page for my wallpapers featuring indy car driver Danica Patrick. If only Benchmarking had Sex Appeal. Monthly Statistics: The Monthly Report identifies activity for each month in the report time frame. Remember that each page hit can result in several server requests as the images for each page are loaded. Month | Num of requests | Num of page requests 1. March 2007 | 728 | 224 2. April 2007 | 9,897 | 1,047 3. May 2007 | 12,413 | 1,217 4. June 2007 | 23,424 | 1,867 5. July 2007 | 51,031 | 3,699 6. August 2007 | 43,119 | 1,962 7. September 2007 | 31,861 | 1,585 The more significant of the stats is the last column, the number of page requests, as the middle column includes images. I think there are a lot more images since July onwards, as that is when I went from text to a more graphical layout. Not sure about the huge jump in July. It may have coincided with heavy activity on the geocaching forum, and/or search engine spider hits. The number of page requests seems pretty steady in the 1,500 range, if you toss out the March and July anomalies. It doesn't say I have 1500 users, just 1500 requests. I am sure 30 of those are mine. Seriously though, its not about the page hits, I did this project as a way to visualize the benchmarks in my area and learn about SQL and the Google Maps API. Since I had the room, I decided to add the other states, and post for all to see. I hope others find it useful. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.