Need Help Designing a "Test" Between Receivers......

[Alphawolf]

O.K...After months and months of reading everyone's opinion about which GPSRs are the "best" at getting and holding a sat. lock, and which are the most accurate, Josh and I (hi Josh) decided to develop a "test". It occured to us, we have access to just about every current model from both big names. We want to put them to a controlled, objective test to analyze both accuracy and sensitivity to sat. lock under tough conditions. We want to do this with good scientific methodology, so as to control as many variables as possible. So....Help us out! Give your ideas as to how to "test" these things. We are on a cheap budget (I could spare some pocket change) so don't design any NASA stuff here! If nothing else this project will be fun. First though, give us your suggestions as to a name for this project.......

 

"Never take a sunset for granted. Stop what you are doing and enjoy it. You never know when it may be your last"

[+phantom4099]

Well your going to want to compare the accuracy to a set bench mark (get them from the main page) that was surveyed recently or checked (I not sure what category is best, but avoid one that are scaled off maps, may want to go to GPS surveyed). Then try to pick some out in the open, and some that in more wooded areas.

You then probably want to download SA watch or visual GPS (this will give you more precise reading than the GPSr alone will, and will let the position average as long as you want). On SA watch you can set a reference point (survey marker) to see how far off you are (you may have to adjust the datum that the survey marker used to wgs84 which SA watch uses).

 

You probably want to let each receiver sit for the same amount of time (give it some time before you start so you get complete info). Also try to do this for 30 mins, and if possible try to do each test at the same time of the day so the constellation is basically the same, and try to do it when the weather is the same so moister does not effect the readings.

 

You probably want to do one unit at a time so not to have cross interference and allow them to have basically the same view of the sky (or at least what is reasonable out side a scientific test).

 

The only way I know of to test the moving accuracy of a given GPSr is to check how well the track matches up to a aerial photo graph. USAphotomap works great for this (www.jdmcox.com).

So this should not be too expensive given you have access to a laptop, GPSr, and cables. SA watch has a free 30-day trail, and visual GPS has a free version. This can also be done with a pocketPC with visualGPS.CE

 

Wyatt W.

I hope this makes sence.

 

The probability of someone watching you is directly proportional to the stupidity of your actions.

 

[This message was edited by phantom4099 on December 14, 2002 at 11:30 PM.]

[+EtrexRose]

Im sure you have been there but: www.gpsinformation.net does all kind of crazy tests, I think all of your troubles could be solved by emailing your question to them. You wont be dissapointed.

[+geospotter]

Find a tennis court (or football field) with an open view of the sky. Turn on both (all) units to be tested and slowly (very slowly) walk the lines of the court. Download your tracks and compare.

 

If testing more than two units attach them to a piece of plywood so that no unit interferes with another and walk all of them at the same time.

 

With any test be sure to do them all at the same time.

[+yooper_gps]

..., very carefully, with a Garmin GPSMAP 76S and a Magellan Sportrak Pro. I then did a really careful comparison between 2 GPSMAP 76S units - one bought mail order, and one bought at a local chain store. I found that small differences in how one unit is held will deliver variances in how they receive. Make sure you try to do things EXACTLY the same so that you really are comparing apples and apples. That includes (I think) making sure that the batteries are all pretty new. Signal reflections seem to cause some odd things as well.

 

FWIW, I went with the 76S from the mail order place. It was demonstrably more sensitive to establishing and holding satellite fixes than the other 76S. Go figure. The reasons that the Sportrak Pro lost out were: crummy graphics, and my overall distruct of the averaging thing, and recent discussions regarding how that unit my be "saying" that it has a fix, but in fact, it doesn't, and the accuracy goes off into the weeds. I liked the Garmin software better too. But no question - the Sportrak Pro is one fine unit, and I went back and forth over it for about a week. If the 76S went away and I had to own a Sportrak Pro, I would be very happy. It did seem to be able to establish satellite fixes amazingly fast. I also wanted to eval a Meridian Plat but no one around here had them. That is a unit that I may want to check out in the future. But for now, my 76S makes me very happy.

 

Doug

[+Cachetrotters]

>...very carefully, with a Garmin GPSMAP 76S and a Magellan Sportrak Pro. I then did a really careful comparison between 2 GPSMAP 76S units - one bought mail order, and one bought at a local chain store. I found that small differences in how one unit is held will deliver variances in how they receive.>

 

What data did you record; did you base this test on relative accuracy or absolute accuracy; what standard of measurement did you use; did you eyeball everything, use a stopwatch, or software with an acceptable track record such as SA Watch?

 

 

Signal reflections? What reflections were those? You measured them how? You knew they were there because...?

 

<...It was demonstrably more sensitive to establishing and holding satellite fixes than the other 76S. Go figure.>

 

Why should we go figure? We want to know how you demonstrated it and then figured it. What criteria did you use for this sensitivity test?

 

 

OK. Fair enough, in part. Software and graphics can be explained as your personal opinion, and would be fair grounds for choosing one model over another by anyone. No problem there.

 

What are your citations for these "recent discussions" and do you know how authoritative this "off into the weeds" information is? Did you observe such?

 

Also, am I to understand that Garmin does not use averaging any more? What does it mean that you don’t trust averaging overall?

 

 

Must not have been too crappy after all.

 

 

Except, of course, it might have been lying to you because it was "off into the weeds" somewhere.

 

Sorry to be so hard, but this is how opinion becomes misinformation, then fact. ‘Barracks lawyers and barracks doctors’.

 

Don

[rfriel1]

i just bought a magellan meridian from frys for $99. you are comparing the garmin map76s which the lowest price i have ever seen is near $400. that is almost a 4 to 1 price difference between the two units that you were comparing.

 

if you are going to do a real test, you should pick units that have similar prices, NOT a 4 to 1 price difference. who would drop almost $400 on their very first gpsr if they didn't even know if it would be useful to them. you can't beat dropping $99 for a new magellan meridian. if i don't like my magellan meridian, i'll chuck it out, no regrets, i'd have to choke down a big one to chuck out a $400 garmin gpsr.

[+Cachetrotters]

The first thing I would do, and I do this regularly at work, is contact the manufacturer for information about how they do it and how they arrive at the specifications they advertise. Since Engineering and Sales do not often see eye-to-eye, talk to each. Remember, it is Sales function to sell these things, and if that means stretching the Engineering numbers, then so be it.

 

The first goal would be to see if each UUT (unit under test) meets the manufacturer specifications. Without talking to them, you may not know what the specs REALLY mean.

 

For any test of accuracy or precision, one must have the methodology laid out. It must itself be tested for strengths and weaknesses.

 

You must have some accepted standard if you are testing for absolute accuracy. The survey markers mentioned several posts up would be ideal. If you own or have access to property, the boundary corners might provide convenient references.

 

Test as many units at once as you have computers and software. I would not do it with less than 3 systems. Four is better. This will subject each UUT fairly equally with respect to environmental and satellite variables. Then the differences between UUT's are given substance.

 

Ex. If one UUT starts to walk and the others do not, and the walker has two fewer sats at the same time, then you might be able to say it exhibited a tendency to "loose lock" and wander. That is more substantial than telling me that you tested them two hours apart and one "seemed" to be less credible, even though the sun had set and it was raining and it was 25 degrees colder and somebody once told me these things are junk, etc.

 

Lastly, this should be no two-day affair. You need to have several sites with varying coverage, etc. and take multiple measurements at each.

Be prepared to evaluate them statistically with interval and confidence level, like the specs are.

 

Sorry for the lecture, but this is my estimate of just some of what to do and look for in your upcoming test.

 

Regards,

Don

[+slinger91]

Where do you work? You seem to know alot about GPSr. When i go to buy one after the 1st. of the year, I might have to buy from your store. btw, which one do you recomend?

 

 

"If I knew how thirsty I was going to be this morning, I would have drank more last night."

[+Searching_ut]

Alright, a person in the same general geographical area interested in measuring the differences in units. It will be interesting to see how your results compare to what I’ve been getting over the years. The biggest problem I’ve been encountering in measuring data is doing all the comparisons at the same spot at the same time. Reception conditions change so fast and much that comparisons aren’t really any good otherwise.

 

As for what I’d like to see, and the sort of tests I tend to personally run:

 

Real accuracy while stationary, at a known location, under as many different reception conditions as possible.

 

Ability to return to a location under different reception conditions such as terrain, tree cover, etc.

 

Accuracy and repeatability of tracklogs, especially out and back type journeys. (Look on my webpage under “accuracy” “Angels Landing” to see just how out of whack they often get)

 

Mask angle of the different receivers. I’ve found a low mask can be a problem on occasion, or can make the difference in getting or not getting a fix on the side of a mountain where terrain blocks a lot of the sky.

 

NMEA data for reception. (The whole data stream gives lots of good data) Recorded for each unit in the same spot, at as close to the same time as possible. I’ve found this data shows some interesting reception patterns from different antenna types, which will probably surprise you. Real world doesn’t always match the theory I learned about antennas. (PDA’s make great data loggers)

 

Ability to obtain a lock, and how many sats were used in the solution for various locations when the units are allowed to sit and search for a period of time, say 5 minutes. Of course the problem is that the data has to be obtained at the same time and location. If possible, the actual accuracy would be very helpful, but I’ve found it’s very hard to obtain.

 

As a final note, no matter what results you get, you’ll probably find you get an awful lot of people mad at you if you didn’t like “their” receiver.

 

http://home.sprynet.com/~searching_ut/

[+yooper_gps]

...I'll post some more opinions about competing GPS units. Jeez, Cachetrotters - preaching to me about testing methodology like you've got some exclusive corner on the use of the scientific method.

 

I thought it was pretty clear that I was presenting casual, empirical, opinionated info, not something based on some kinda rigorous testing methodology. I don't have time or interest to design and implement a treatise on comprehensive GPS testing. So I went forth and performed some casual comparisons, and this yielded interesting and useful information that I could reasonably use to make a purchase decision

 

Cachetrotters, nice touch: "I didn't see your GPS simulator listed." Uh-huh. Very useful.

 

And rfriel1: As for the differences in price between the Sportrak Pro and the 76S, the Pro was $270 at Wal-Mart, and the 76S was $369 at GPSNow. Not that big of a difference. How the pricing of a Meridian got in there and the 4:1 pricing difference is beyond me.

 

So much for the peaceable world of geocaching.

 

Doug

[Alphawolf]

quote:

---

Where do you work? You seem to know alot about GPSr

---

 

The more you learn, the more you realize you don't know! I work at a large multi-state sporting goods chain (Sportsmans Warehouse) I have sold these things now for 5 years, and owned one for 7. It's like golf though. You only learn enough to humble yourself regularly.

 

"Never take a sunset for granted. Stop what you are doing and enjoy it. You never know when it may be your last"

[+Searching_ut]

Hmmmm I wonder if Alphawolf could be the guy who talked me into buying a brunton windwatch to go with the yellow etrex I picked up at the Riverdale Sportsmans warehouse a few years back. At the time I was looking to upgrade to an actual 12 channel unit, and had had a lot of temperature related trouble with my old Magellan 2000. The wind watch didn't help with the temperature problems of course, but helped me decide I must be crazy to head out in some of the conditions I seek adventure in.

 

Good luck in your testing

 

Jeff

[+Searching_ut]

Opinions, perceptions, and general observations are something I personally find useful, but must admit I like them even better when they can be combined with measureable data. I worked in an Air Force test squadron for many years, and often found much of the data we collected to be very interesting. Sometimes when looking at everything after testing you would find that peoples perceptions during a test didn't mirror the actual results at all. Other times you would find that the tests themselves were flawed, and needed corrected, which was very evident based on peoples observations. The fun part is figuring out where the truth lies.

 

For what it's worth

 

Jeff

[+yooper_gps]

...of a casual test I performed. I'd stand in one spot, without moving, out in an open park area, and let both 76S units do their satellite fixing thing for about 10+ minutes until they seemed "done". I'd have one in one hand, the other in the other, both held up in the air above my head about a foot from each other. Not a dignified picture. I'd watch them do their "aquiring fix" thing on the bar chart, and one seemed to find more satellites, and find them more quickly, and aquire stronger signals from them than the other unit. Then I'd switch hands and try it again, and again, the same one was more sensitive. Both had new batteries. Draw your own conclusions. I'd also set them side by side in the car on the seat, and one was a bit better at holding a fix. Every time.

 

When hiking through the woods with the Sportrak Pro and the 76S, the Pro would aquire more targets more quickly, and generally hold them longer before losing a fix. But it wasn't consistent - sometimes the 76S would hold on a tad longer. My first thought was WOW - this Pro is WAY more sensitive and accurate. But then I read a bunch of opinions of people who talked about the averaging thing with the Pro, and how it might not actually have the fix even though it was saying that it did. Again, not having a ton of time, money, and interest to come up with the canonical testing plan for these things, I had to go with my opinion, based on what I had at my disposal.

 

Who's gonna argue about the graphics quality between the two? Now, it's cool if someone doesn't care about the graphics - I wondered myself if I'd feel that way down the road. Even at this point, I find that I don't really use the graphics as much as I use the numeric read-outs of things like position, distance, speed, etc.

 

Like I said, I wish I had been able to compare the Meridian Plat to the 76S and the Sportrak Pro. It's not clear to me if the Pro uses the same receiver and antenna as the Plat - and the Plat does have a bit bigger screen than the Pro. Once I get more experienced with GPSing, I may end up going to a Plat or a Pro. I could give a rat's *** about brand loyalty or brand zealousness with these things. I just want the "best" one.

 

Doug

[+yooper_gps]

....

[+geospotter]

I don't think the serious tests described above will show you anything. Tuesday's results will not match Wednesday's, Florida's results will not match Finland's, and my results will not match your results. There are just too many variables.

 

I look more toward "real world" use. I trust an engineer over a salesman, but I trust a user over an engineer.

 

Run simple tests to find what works in your part of the world, during the hours that you typically cache. It really doesn't matter if the SuperGPS is better on paper if the MundoGPS performs better in the field. Don't worry about things that you can't test (like reflections), just be aware that they exist.

 

I'll bet not one of us did exhaustive testing (as described above) before buying our units, but I'll bet almost everyone asked someone else about their opinions. I wonder why that is...

 

And I don't think you could find a better source of opinions about GPSs than right here in these forums.

 

geospotter

[Kerry.]

Just as well these were "casual" tests wouldn't like to see anybody take them all that seriously.

 

Cheers, Kerry.

 

I never get lost everybody keeps telling me where to go

[+yooper_gps]

...for being so intense today. Bad today here. I learned a lot from the posts I read.

 

But I am curious - I noticed that my phrasing of "rat's ***" was censored, because it came out as "rat's ***".

 

What's up with that? Is this board run by Blockbuster Video?

 

Doug

[+yooper_gps]

The word "***" gets deleted. Jeez.

[Alphawolf]

quote:

---

Are you the one?

---

Well, that is where I work, but I really hated that thing, so I can't imaging talking you into buying one! In fact, they (Brunton) make several things I am not crazy about. But I do like their compasses!(not the electronic variety)

 

"Never take a sunset for granted. Stop what you are doing and enjoy it. You never know when it may be your last"