Favorite Ratios

[I!]

For those not following the thread, the cutoff is needed because too few samples would cause high spikes in the percentages (i.e. 1 PM find and 1 favourite = 100%).

A maximum a posteriori estimate could neatly sidestep this. Setting aside some lovely mathematics involving beta conjugate priors, it's a simple matter of pre-supposing the [mythical] existence of some P number of premium cachers, F of which awarded a favourite. Then the "score" for a cache is not favourites/premiums, but instead is (favourites+F)/(premiums+P).

 

F and P should be chosen per class of cache (in some sense to be investigated but e.g. by cache type). Simply plot the distribution of ratios of favs/prems for that class and try to fit it to a Beta(F,P) distribution. I'll try to find the time tomorrow to test this idea, accepting that it is not super-likely to work ...

[+Avernar]

Looks like I was right about the raw count being biased to the older caches.

[+StarBrand]

Looks like I was right about the raw count being biased to the older caches.

Based on observations of local caches numbers - I would tend to agree with that statement. What it means to me is up in the air so far. But it does demonstrate the usefulness of the ratio's in finding the truely top favorited caches.

[+addisonbr]

Looks like I was right about the raw count being biased to the older caches.

I'm not entirely convinced about the mechanism... I have a suspicion that if you control for number of finds, older caches are actually penalized.

[+Avernar]

I'm not entirely convinced about the mechanism... I have a suspicion that if you control for number of finds, older caches are actually penalized.

My list doesn't seem to show that. The older caches are spread out all over. There's even some really old ones in the top 20. Hilton Falls Loop which was published in 2005 is number 11. Bushwacker which was published in 2001 is number 13 and it's not one that's easy to get to.

[+Coldgears]

Looks like I was right about the raw count being biased to the older caches.

I'm not entirely convinced about the mechanism... I have a suspicion that if you control for number of finds, older caches are actually penalized.

I'm also not conviced.

[+Avernar]

I'm not entirely convinced about the mechanism... I have a suspicion that if you control for number of finds, older caches are actually penalized.

I'm also not conviced.

Why aren't you convinced? If penalizing older caches is bad, isn't penalizing newer caches bad as well?

[+tozainamboku]

For those not following the thread, the cutoff is needed because too few samples would cause high spikes in the percentages (i.e. 1 PM find and 1 favourite = 100%).

A maximum a posteriori estimate could neatly sidestep this. Setting aside some lovely mathematics involving beta conjugate priors, it's a simple matter of pre-supposing the [mythical] existence of some P number of premium cachers, F of which awarded a favourite. Then the "score" for a cache is not favourites/premiums, but instead is (favourites+F)/(premiums+P).

 

F and P should be chosen per class of cache (in some sense to be investigated but e.g. by cache type). Simply plot the distribution of ratios of favs/prems for that class and try to fit it to a Beta(F,P) distribution. I'll try to find the time tomorrow to test this idea, accepting that it is not super-likely to work ...

very funny.

 

It doesn't take very advanced mathematics to show that if only one or two premium members have found a cache there are big swings in ratio if just one of them were to change their vote (or when the next premium member finds the cache whether they vote of it or not). The pro-ratio group has even stated such - of course with the argument that a small sample does not give as high a confidence level as a large sample.

 

I should point out that I'm not opposed to ratios per se. In fact most non-parametric statistics use some sort of ratio to deal with sample size. For example a political poll will report the percent for candidate A, the percent for candidate B, and the percent undecided. Most people would find hard to compare two polls that had different sample sizes if the raw votes were reported. Note that most polls will also report a margin of error - basically a confidence interval of the percentages.

 

The issue I have is that we are comparing polls on two or more different caches that not only have different sample sizes but have samples that are biased depending on the cache we are looking at. The argument going on whether an old cache is biased to have a higher percentage in the sample than expected or a lower percentage, just goes to show we don't understand yet all the mechanisms which make the sample biased. Do some people not favorite old caches because they aren't going back to their old finds or they just don't remember the old caches? Or are people's old memories faulty so they are more likely to favorite a old find where they remember something but might not favorite a similar recent find where they remember more details. Or perhaps in the old days they found just that one cache so they remember it well while someone finding the same cache today found 30 other caches today, so they don't remember the cache when assigning favorites?

 

With out understanding the way the samples are biased, the margin of error or confidence cannot be computed. It could very well be that even for caches with many premium finders the margin of error is still very large so the rank ordering of caches by ratio is mostly meaningless.

 

Finally, I will reiterate that as far as I am concerned even if the ratio gives an estimate of the probablity that a random cacher will favorite a cache (with some unknown but large margin of error), using this to estimate if I will simply enjoy the cache or not is overkill. I can estimate whether or not I will enjoy a cache using the raw count. Why compute a number that at best gives me a very fuzzy estimate of whether a random finder will favorite the cache. I don't care if a random finder favorites the cache; what I care is whether someone who has similar tastes to mine has favorited the cache. That tells me it is recommended.

 

For caches in my regular geocaching area, the best use of the favorites is to look at the caches I favorited and see who else has favorited many of the same caches. Then I will look at their list of favorites to get some recommendations. When I am not in my regular area, I haven't favorited any caches there yet, so I look at the raw count and I look at cache type, terrain, difficulty and other attributes to find caches I might like. While not as effective as looking at what known cachers recommend, the more individuals who have favorited a cache the better the chances that at least one of them has similar taste to mine.

Edited February 5, 2011 by tozainamboku

[I!]

It doesn't take very advanced mathematics to show that if only one or two premium members have found a cache there are big swings in ratio if just one of them were to change their vote

The Beta(F,P) method has a flattening effect that reduces this phenomenon. Consider a cache whose first five premium visitors went: fave, not, not, fave, fave. Ratio goes 100%, 50%, 33%, 50%, 60%. Ratio with prior F=3,P=10 goes 36%, 33%, 31%, 36%, 40%. There are plenty of reasons not to like this particular estimator of the favouriting-rate; I expect I'll happily shoot down Beta(F,P) after I've tried it on real data.

 

Finally, I will reiterate ... (why any estimate of the favouriting-rate is worse than a raw count)

Well then, as we're reiterating, so will I. Here, again, is why I take the opposite view.

Kermit the Cacher, who we've never met before, hops into the room. "Find me a favourite cache!", he demands cryptically, "You must decide using either the raw count or the ratio!".

 

Which do we pick?

 

Answer:

• Raw count if we wish to maximise the chances that Kermit has *already* favourited the cache.
  
  
• Ratio if we wish to maximise the chances that, visited already or not, this would rank among Kermit's favourites.
  
  

Corollary:

• If you're more interested in following the crowd than finding your own way, you need the raw count.

If you can't understand an argument illustrated by frogs in this forum then frankly there's no hope for you!

[+addisonbr]

I can estimate whether or not I will enjoy a cache using the raw count.

I believe that's true; I simply believe you can make a better estimate if you add ratio data.

[aniyn]

What I am claiming is that caches with very high raw counts have a pretty good chance of being a cache I will enjoy. Caches with a lower raw count have a bigger probability I might not enjoy the cache. Those with low raw count and even cache with zero raw counts are might be the caches I enjoy most. Who knows, it's possible that those with higher ratios I will enjoy more. But this isn't what I'm trying to find out. I'm just trying to avoid caches that suck - as fizzymagic would say. I don't need a ratio to do this. I believe that the ratio would in fact be less useful, because I might not ever look at the cache in touristy location with only 1% favorites. If 50 people liked the cache, I think I will probably enjoy it. Perhaps not as much as a hike or a puzzle, but I will likely enjoy it. (I still look at the cache page so there is still the chance I won't look for it, if there are other caches with lower raw count that seem more interesting at the moment.)

That's great for you then. However, the basic flaw in you entire argument against "the pro ratio crowd" is you automatically assume they're all idiots who can't use more than 1 source of information. "The pro-information crowd" would quite frankly be a better label, if you insist on using one, since more information on which to base their cache searches on is all they're after.

In the meantime, no one is taking away the raw counts. They're not going anywhere. And the ratio is in a tab and you cannot see it unless you want to.

 

The problem with raw counts is fundamentally the same as the problem with ratios - they're both dependent on sample sizes. A ratio is useless with a very small sample size, but a raw count becomes useless with a large sample.

 

And if a cache only gets a 1% favorite rate? That means 99% of the premium members who found it didn't think it worthy of being in their top 10%. While it may still be an enjoyable cache, it would certainly not be an amazing one either.

[+addisonbr]

I'm not entirely convinced about the mechanism... I have a suspicion that if you control for number of finds, older caches are actually penalized.

My list doesn't seem to show that. The older caches are spread out all over. There's even some really old ones in the top 20. Hilton Falls Loop which was published in 2005 is number 11. Bushwacker which was published in 2001 is number 13 and it's not one that's easy to get to.

My suspicion is a bit similar to the 'survivor bias' in finance. When we're only looking at the caches that made the top 20 or top 160, we're not seeing all of the caches that get no (or very few) votes, and it's hard to tell if older caches are suffering from this disproportionately. (It's similar to studies that try to measure the average return of IPOs by taking all of the listed stocks on the exchange and calculating backwards to IPO day - it misses all of the stocks that IPO'd and subsequently went out of business and are no longer listed.)

 

I don't know for sure that it's an effect, but I suspect it could be in play. I am going to try to take a look at this in my area and see what turns up.

 

Another factor that could penalize older caches is people who simply start playing the voting game today moving forward. There is a prolific finder in my area who has only favorited one cache, a decent but nothing particularly notable cache he found a couple of weeks ago. It's easier to 'Favorite' a cache right when you find it, so I think for example decent caches listed in 2011 will have disproportionate Favorite voting as experienced cachers with dozens if not hundreds of Favorite points saved up from older caches spend them at an accelerated rate on the new listings they find.

 

But that's just speculation at this point; it'll be something interesting to look at in a few months.

[+dfx]

The issue I have is that we are comparing polls on two or more different caches that not only have different sample sizes but have samples that are biased depending on the cache we are looking at. The argument going on whether an old cache is biased to have a higher percentage in the sample than expected or a lower percentage, just goes to show we don't understand yet all the mechanisms which make the sample biased. Do some people not favorite old caches because they aren't going back to their old finds or they just don't remember the old caches? Or are people's old memories faulty so they are more likely to favorite a old find where they remember something but might not favorite a similar recent find where they remember more details. Or perhaps in the old days they found just that one cache so they remember it well while someone finding the same cache today found 30 other caches today, so they don't remember the cache when assigning favorites?

 

It doesn't matter, because over time it's gonna even itself out. Consider a random cacher looking at the list of best rated caches and pick a very old cache from that list to hunt for. Or maybe he just looked at the list of oldest caches and went from there, doesn't matter. He goes for it, finds it, and returns back home to log his find. One of two things can happen: 1) he puts a favorite vote in, or 2) he doesn't put a favorite vote in. The reasons why he would do either of that are his personal matter and irrelevant to the statistics. If he put a vote in, the ratio will go up slightly, and if he didn't then it will go down slightly. Now this single occurance doesn't tell us anything, but if the same thing keeps happening over and over again as more cachers hunt for the cache, we can draw a conclusion: if more people put in favorite votes than the ratio previously indicated, then the ratio was too low, or if it's the other way around, then it was too high.

 

So at this point we would know if the ratio was too low or too high, but what do we do with this knowledge? Nothing! Because the ratio has already adjusted itself to reflect this new information. More data = better statistics!

[+StarBrand]

I give up - can a Mod please close this thread down as many/most are no longer observing or commenting on the data now visible but instead making silly arguments as to how ratios should or should not be used and by whom for what.

[I!]

I give up - can a Mod please close this thread down as many/most are no longer observing or commenting on the data now visible but instead making silly arguments as to how ratios should or should not be used and by whom for what.

You don't have to understand the intricacies of every argument. Move along. We're having fun.

[+tozainamboku]

I give up - can a Mod please close this thread down as many/most are no longer observing or commenting on the data now visible but instead making silly arguments as to how ratios should or should not be used and by whom for what.

You don't have to understand the intricacies of every argument. Move along. We're having fun.

That was my feeling too! But the OP is within his rights to have the thread closed, it really has strayed from his original intent.

 

While it is opened still, however, I do like your idea of using a prior knowledge to flatten out the fluctuation of ratio that we see with small samples. We actually know that the average cache should have a ratio of favorites to premium finders of (total number of favorites used /( 10 * total number of available favorite votes). This isn't exact because you get favorite points for attending events and such but can't use the favorites on these caches. Some people also may have received favorite points for logging their own cache or caches they adopted later, and now cannot award these points to those caches. But I'll grant that these may not effect the a priori ratio. So you could use a Beta distribution to compute an a posteri ratio conditional on the small sample of finders on most caches. Of course, I still contend that the probablity that a random cachers favorites a cache is not a good estimate in and of itself as to whether I would enjoy the cache, and once I start looking at other factors, the raw count works better for me.

 

@addisonbr - I see your knowledge of statistics comes from their use by analyst who predict the performance of stocks and other financial instruments. While I am sure that the analysts that use statistics this way understand what the limitations are, the events of the last couple of years should certainly make everyone aware of the dangers of relying on statistics without understanding what they are truly telling us. Banks and brokerage relied on statistics that they didn't understand and look what happened.

 

@dfx - I certainly know the Law of Large numbers. A long time ago when I took Theory of Statistics in college, we learned the formal proof of both the Strong and Weak forms of this law. So we really had to understand just what is claimed. In the week law, as you take more random observations the ratio will approach in the limit the the ratio for the entire population. It doesn't say that it equals the ratio (of course for a finite population once you have sampled everyone you would of course found the actual proportion). In fact, the ratio may move significantly away from the true proportion, but the probability of this happening as you get bigger and bigger samples approaches 0.

The strong law is much more what you seem to want to use. It says as the sample gets larger almost surely the ratio will be the expected value. This means that if we have a big enough sample the ratio will be arbitrarily close to the true proportion. The strong law require more assumptions about the underlying distribution of the population.

In both cases the Law of Large Numbers does not tell us how quickly the convergence is. In certain cases where we know the distribution type of the population we can compute how good the estimate is for a given sample size. We can't make those assumptions about the distribution for favoriting caches. In addition, we certainly can't assume that the premium members who find a particular cache represent a random sample of premium members.

 

At best the ratio is an estimate of the probability that a premium member who would select this cache based on the available information (including the current raw count or ratio of favorites) at the time they went to hunt the cache would end up giving it a favorite. Some people may choose to use this number to decide whether or not they will hunt a cache. My contention is that this number is not the likelihood that a cache will enjoy the cache. There may be a relationship, but there is also a relationship between the raw count and whether a cacher will enjoy the cache. I have given the reasons why I think the raw count is a better predictor of enjoyment than a highly variable and biased estimate of probability of giving out a favorite vote. If you really believe that you are the hypothetical average cacher, then perhaps knowing the average chance that this cache would be a favorite would tell you something - if only the sample for each cache reflected the average cacher.

Edited February 5, 2011 by tozainamboku

[+addisonbr]

@addisonbr - I see your knowledge of statistics comes from their use by analyst who predict the performance of stocks and other financial instruments. While I am sure that the analysts that use statistics this way understand what the limitations are, the events of the last couple of years should certainly make everyone aware of the dangers of relying on statistics without understanding what they are truly telling us. Banks and brokerage relied on statistics that they didn't understand and look what happened.

Yes. I've always felt it's better to understand the limitations of a statistic than to use it blindly, or ignore its existence altogether.

[I!]

btw, I asked GS if I could hoover down 100 or so cache pages + premium-or-not info about the (recent) finders and was turned down ... but turned down in a way that said, more or less, "API's coming soon; ask us again when that's in place". I can hardly wait!

[Brad_W]

The original poster has asked for the topic to be closed; it appears the question has been answered, and the original poster does not seem to be attempting to have the last word.

 

Closing.