Peer Review

[+lumbricus]

Hi,

let's continue the discussion from the 'Marathon events worldwide' thread here.

 

Few minutes ago I read this post from the 'Blue Quasar'.

 

Posted 25 August 2006 - 04:28 AM

 

"The part that says "Please keep in mind, we aren't looking for your personal opinion of the category here." I pretty much totally ignore.

 

Only because I don't think it is worded well. Obviously some aspect of our opinion is required to evaluate potential new categories.

 

So I use these thoughts to guide me, oh and a Magic 8-ball!

 

1... Is this category well thought out and described with detailed requirements?

2... Is this something that would be of interest to a large group of people?

3... Is this a unique category, or at least stands apart for some reason?

4... Is there a regional restriction that has been justified properly?

 

So basically, even if I personally have no interest in the category but I can see how some people would get enjoyment from it, and everything makes sense and is well written... then I vote YEA.

 

Lack of detail, items that are socially perceived as being in bad taste (not to be confused with 'adult in nature' or 'sensitive'), and of course locationally centric for no reason.... then survey says.... NNNNaaaaaa!

 

The Blue Quasar"

 

I think the bold sentence is important.

 

[+Max and 99]

Hi,

let's continue the discussion from the 'Marathon events worldwide' thread here.

 

Few minutes ago I read this post from the 'Blue Quasar'.

 

Posted 25 August 2006 - 04:28 AM

 

"The part that says "Please keep in mind, we aren't looking for your personal opinion of the category here." I pretty much totally ignore.

 

Only because I don't think it is worded well. Obviously some aspect of our opinion is required to evaluate potential new categories.

 

So I use these thoughts to guide me, oh and a Magic 8-ball!

 

1... Is this category well thought out and described with detailed requirements?

2... Is this something that would be of interest to a large group of people?

3... Is this a unique category, or at least stands apart for some reason?

4... Is there a regional restriction that has been justified properly?

 

So basically, even if I personally have no interest in the category but I can see how some people would get enjoyment from it, and everything makes sense and is well written... then I vote YEA.

 

Lack of detail, items that are socially perceived as being in bad taste (not to be confused with 'adult in nature' or 'sensitive'), and of course locationally centric for no reason.... then survey says.... NNNNaaaaaa!

 

The Blue Quasar"

 

I think the bold sentence is important.

 

 

Another thing to add:

 

Even though it's not listed as a criteria, many of us will vote Nay on a category if the officers have little or no Waymarking experience. The logic is that if you haven't waymarked, you don't have the background to be reviewing other people's submissions AND if you haven't demonstrated stick-to-it-ness, will you abandon the category when you lose interest?

[+elyob]

Aside from the criteria which appear for our guidance when we use Peer Review, are there other written instructions for what makes or does not make a good Waymarking category? Question 2: is there any point in adding/revising instructions to the Peer Review process or should the Peer Review process itself be reconsidered?

[+Max and 99]

This is the information under the link "I'd like to create my own category":

 

The Waymarking directory contains categories that focus on a specific type of location. Each category is managed by a Group consisting of at least 3 Premium Members that work together to ensure all the locations submitted to their category adhere to the category goals. If you are a Premium Member and have an idea for a new category that does not already exist in the directory, you can form a Group and submit your category for review by following the steps below.

 

1. Form a Group

 

From any page, click the My Groups link in the right column and then click "Create New Group". Choose a name for your Group, state your Group’s goals and click Submit.

 

Invite at least two other Premium Members to join your Group. If you are new to Waymarking and don't know any other Premium Members you can find some tips for recruiting here.

Promote your new Group members to the rank of Officer. Check out this link for more information on promoting Group members.

 

2. Create a Category

 

Once you have the required number of Officers in your Group, a checklist will automatically appear on your Group page with a link to the category creation form. Take your time writing the category details and research your idea fully before you complete the edits. The Discussion Forum is a great place for tips on category creation.

After you have finished editing your category details you can call for an Officer vote. All officers are required to be in 100% agreement of the category name, description, and variables so be sure everything is in order before calling the vote. When the vote is completed and everyone has voted to accept the category, you may then click the "Send Category to Peer Review" link to initiate the final step.

 

3. Peer Review

 

Peer Review consists of a queue available to all Premium Members allowing them to review, vote, and comment on category hopefuls for a period of 3 days. During this time you will not be able to make changes to your category description, however you may cancel the vote at any time if you left out something important. To cancel a vote, click the red X next to step 4 in the checklist on your Group page.

Small mistakes such as grammar and typos can be fixed after Peer Review if necessary. Once the voting period has passed, and assuming the community liked your category, you will receive a congratulatory email and may begin accepting new waymarks immediately!

[+lumbricus]

Nate wrote 2006:

 

Posted 31 March 2006 - 02:09 AM

 

"The purpose of this thread is to post general tips and tricks for gathering the support one needs to see his or her category listed in the directory.

 

For questions on procedural issues related to creating a Group or Category please visit the FAQ:

http://www.waymarkin...p/faq.aspx?f=-1

 

Or this series of posts in the Getting Started forum:

http://forums.Groundspeak.com/GC/index.php...dpost&p=2097536

 

Do Your Homework!

 

You're more likely to garner support in the form of Group officers if you take the time to post a detailed description of what you hope to accomplish with your category. Don't forget to search the directory first to make sure you aren't duplicating the efforts of other waymarkers.

 

There are a lot of resources around the 'net for you to use in writing your category description (like Wikipedia), and people will be more likely to join up if you've taken the time to do your homework. Little or no preparation on your part will likely result in a lukewarm reception of your idea.

 

Discuss Possible Variables

 

Variables give you a way to gather additional information about a location, or separate waymarks in your category into logical groupings. For instance, for Historic Markers you can have a variable that requires a waymark submitter to choose between a "highway" marker, or a "city" marker. Variables can make or break a category so now is the time to start thinking about how they relate to your category idea. If you need input from the community don't be shy in asking for help.

 

Remember, 99% of the time when someone feels they need sub-categories what they really need are more variables.

 

Be Enthusiastic

 

If you're not excited about the prospect of managing your chosen category, there's little chance anyone else will be either. Pin on your pieces of flare and sell it!

 

Have Your Group Ready

 

Take a moment to create a Group to manage your category before posting your request for support. Start by choosing a Group name and writing out a quick description of your goals for the category so folks know what they're getting into. When you are ready to begin accepting new membership, it is helpful to turn on "Open Enrollment" feature to allow new members to join at their whim.

 

Don't forget to include a link to your Group page when proposing your new category!

 

I will add to this post occasionally and would welcome any input on effective recruiting and category proposal tactics. Happy Waymarking!" http://forums.Groundspeak.com/GC/index.php?showtopic=128144

[+lumbricus]

The post above is not 100% about the peer review, but good to know.

[+DieCacherGirls]

German speaking waymarkers may read this in German

 

+++++++++

 

Dear waymarkers,

 

after struggling with the ‚running events worldwide‘ category we would like to suggest some improvements to the current peer review process. Please consider this as unprejudiced consulting.

 

As an obligatory first step we suggest that you decide whether the current peer review shall continue as an open all member vote or shall be restricted to a real expert decision.

 

If a real expert decision about new categories is wanted, the group of voters could be limited to all existing category leaders and officers, because these are the people with experience about categories.

 

If the open all member vote should be continued, there shall be measures to prevent that voting is influenced by buddies of the category proposers (e.g. geocachers). There shall also be measures to encourage voting based on expertise. This could be done by writing down detailed guidelines for category submittal with all rules like this was done for earthcaches (http://www.geosociety.org/earthcache/guidelines.htm). The guidelines may include category types no longer accepted, e.g. limited availability types or fast food chains. Maybe a checklist can be provided to the proposer. You may even limit the total number of categories if necessary or mark some older categories as grandfathered in the grid.

 

If there are submittal guidelines you shall think about how to observe during peer review if all voters stick to the rules. This is not trivial. Currently there is no one caring for votes apparently wrong (e.g. ‘not global’ votes for global categories). Like in sports someone needs to act as referee, addressing the voter or showing the yellow card (and as a last measure even delete a vote if there is a reason).

 

Currently voting in peer review is without any commitment, some stick to the four criteria, some stick to hidden rules, some stick to their own rules. Peer review is an unorganised process, which may be fine within a small restricted group. If the Waymarking community wants to be open for more members, processes need to be more comprehensible.

 

An optional measure to improve peer review is simple majority instead of 2/3 majority, because it is more common. Additionally there shall be a dedicated timeframe for peer reviews to enable broader participation rate e.g. the first three days of each month, so all members can plan for, also the workforce.

 

With that many proposals for improvement, we back out of the discussion now and hope you make the best of of it.

 

Best Regards, Britta and Renate for the CacherGirls team

[+lumbricus]

...If a real expert decision about new categories is wanted, the group of voters could be limited to all existing category leaders and officers, because these are the people with experience about categories...

 

Silverquill wrote often about this problem. We lost our old forum so I can't cite him. I think that would be a good thing, all leaders and officers are allowed to vote.

[+Benchmark Blasterz]

...If a real expert decision about new categories is wanted, the group of voters could be limited to all existing category leaders and officers, because these are the people with experience about categories...

 

Silverquill wrote often about this problem. We lost our old forum so I can't cite him. I think that would be a good thing, all leaders and officers are allowed to vote.

 

We respectfully disagree. We are enthusiastic WMers, on our way to 1000 ( and not stopping there we hope!). We do not have time now to take on the responsibilities of an officer, but we are involved -- even here in the forum. We are invested in the hobby and are working to grow it and help it succeed by posting quality WMs for others to find and enjoy.

 

We vote in peer review religiously - only missed 1 vote while traveling since we have become active. Under this proposal, we would be voiceless. Why would you not want to hear from all WMs who participate in growing and expanding the hobby?

 

We also don't like a vote referee -- someone with power to cancel (or change) another WM's vote. If you want that system, just submit to GS like for caches and who cares what the WM community's desires are? Have a WM Potentate with complete power to approve or disapprove categories. not democratic not good not healthy for WM we think.

 

We like the 2/3 threshold -- it ensures only quality categories with broad support get activated. Supermajorities are required for several governmental changes-- this is why.

 

We may have more later

[+fi67]

You are disappointed and think there is something wrong here. I do understand that and there is indeed something that should be improved; but it is not the process, it is the documentation that can raise false expectations and lead to frustration.

 

"Peer Review" as is has proven to be surprisingly successful over a long time. The point is: it is not a real peer review, but a vote. There are small but fundamental differences. A vote has to be won, there is no right to succeed just by fulfilling some criteria, and there are no selected experts who decide, it's the public, you and me.

 

This is a good thing and it works. Sometimes you lose a vote and sometimes it is for wrong or doubtful reasons. This has happened to all of us and it will happen again. Still expert review is absolutely not the way to go in my opinion. This would raise more problem than it would solve. Officers should know better than the crowd. Do they? I see many officers that are absent or only care about their own old category, and I see many waymarkers who make valuable comments in the forums and in peer review but do not feel to take the burden of managing a category. Someone to judge given votes and afterwards decide if they are valid? Really? A horrible idea!

 

These votes are not to compare with Facebook Likes at all. Most voters know exactly what they do and why. There are the four criteria, they are guidelines and there are different possible interpretations of them. Most participants have them in mind when they submit their vote. But it's fine to include additional thoughts to find your decision.

 

And then there is the two-thirds quota. It keeps the whole thing going smooth and stable, because it means broader support. Two-thirds are very common around the world in fundamental votes.

 

The category leaders are the ones who decide when the vote begins. I do not see a single reason to change this.

 

I don't see the point in adding a whole bunch of complexity and bureaucracy to something simple that works well (enough). It is not the unorganized process you see. It was created with the flexibility to change over time for the better by itself, and this has mostly happened.

 

There are some problems (like the buddy recruitments): It is not perfect, I know, but not bad enough for the downside risk of complex changes.

 

I can follow your thoughts, but I do not support any single point of your proposed improvements.

 

The whole discussion is academic, anyway. Nothing is going to happen. First: Groundspeak is not going to make any changes at the site in the near future and they have clearly stated that voting permissions for all premium members is a core principle. Second: this is just a forum, there is no "official community" in a sense that they have the power to enforce any change. We are all just users of a website who discuss things here, we are all just the same as you. (Well, except BruceS and his magic powers )

Edited April 4, 2013 by fi67

[+elyob]

I accept that Groundspeak will not be making changes to the process. However, is it still possible to educate future category creators concerning category-types which are no longer acceptable to the community? Thanks to this forum, I now know that limited-availability categories (and many commercial categories) would no longer be accepted. But for those future category creators who actually work outside of this forum, how will they know not to waste their efforts on a category-type that is no longer true to Waymarking?

[+lumbricus]

...But for those future category creators who actually work outside of this forum, how will they know not to waste their efforts on a category-type that is no longer true to Waymarking?...

 

Normally you start thinking about a new category not in the beginning. It's a process, if you a really interested in Waymarking you will read in the forum. I think many read our forum, but only a small part is writing here.

 

But we can do one thing, we can change the text from the category guide

 

"The Waymarking directory contains categories that focus on a specific type of location. Each category is managed by a Group consisting of at least 3 Premium Members that work together to ensure all the locations submitted to their category adhere to the category goals. If you are a Premium Member and have an idea for a new category that does not already exist in the directory, you can form a Group and submit your category for review by following the steps below."

 

It could be clearer that it's not that easy to create a new category. If we change a few sentences, I'm sure that bootron will add these, that's really a good thing and very, very short/small.

[+lumbricus]

...(Well, except BruceS and his magic powers )

 

Hex, hex

 

We, the community, run Waymarking.com to 95% Thanks to all of you!

[+kittyposse]

I share some of DieCacherGirls' discontent with the current peer review concept, but my course of action would be right in the opposite direction and instead of putting up more restrictions and complicating the process, I'd suggest to further facilitate it and ask only one single question in peer review:

 

Would you, personally, like to see this new category added to the list? (yes/no)

 

And that's it, plain and simple. Criteria such as globality, prevalence, redundancy, "wow" factor etc. may still be mentioned as suggestions to think about when voting, but not as guidelines or as a checklist to go through. They'd be merely suggestions that one may just as well ignore.

 

I'd further suggest to do away with the need to add comments to "nay" votes. As it is, many of these comments aren't very substantial and are of little or no value anyway, and then I believe nobody should be forced to justify their vote, regardless whether it be "nay" or "yea". Of course those who'd like to add a comment should still be able to do so, but this should be voluntary for both "nay" and "yea".

 

In the end, it is not important why voters vote the way they do. Important is only the result. If 2/3 of the voters vote "yea" - for whatever reasons - then the new category can be expected to be a success (as ist is welcomed by the vast majority of the community) and it deserves to pass. If, on the other hand, more than 1/3 of the voters say "nay" - again, for whatever reasons - then there is reason to believe the new category would be of limited interest to a substantial part of the community, and it deserves to fail. Of course, this way peer review would still be the box of chocolates ("you never know what youre gonna get") it ever was, maybe even more, but at least this would be clear from the very start and there wouldn't be any false expectations like "If I only did my homework, my category will be likely to pass".

 

But if fi67 is right and nothing's gonna change anyway, then why bother? I guess I'd better save my time and go out and find some new waymarks instead.

Edited April 4, 2013 by kittyposse

[+Max and 99]

 

If a real expert decision about new categories is wanted, the group of voters could be limited to all existing category leaders and officers, because these are the people with experience about categories.

 

 

There are active waymarkers with WAY more experience than some category officers. You can find category officers with zero waymarks visited and only a handful of posted waymarks, so they are in no way an "expert" about categories.

Benchmark Blasterz has a lot more experience than some officers!

Edited April 4, 2013 by Max and 99

[+Max and 99]

 

I'd further suggest to do away with the need to add comments to "nay" votes. As it is, many of these comments aren't very substantial and are of little or no value anyway, and then I believe nobody should be forced to justify their vote, regardless whether it be "nay" or "yea".

 

Comments are not required. Peer Review instructions state you must give a reason for a Nay vote, but you do not have to write a single word.

[+kittyposse]

 

I'd further suggest to do away with the need to add comments to "nay" votes. As it is, many of these comments aren't very substantial and are of little or no value anyway, and then I believe nobody should be forced to justify their vote, regardless whether it be "nay" or "yea".

 

Comments are not required. Peer Review instructions state you must give a reason for a Nay vote, but you do not have to write a single word.

 

I'm baffled. How could I give a reason without writing a single word?

[+Max and 99]

 

I'd further suggest to do away with the need to add comments to "nay" votes. As it is, many of these comments aren't very substantial and are of little or no value anyway, and then I believe nobody should be forced to justify their vote, regardless whether it be "nay" or "yea".

 

Comments are not required. Peer Review instructions state you must give a reason for a Nay vote, but you do not have to write a single word.

 

I'm baffled. How could I give a reason without writing a single word?

 

.

[+elyob]

Edited April 4, 2013 by elyob

[+kittyposse]

 

I'd further suggest to do away with the need to add comments to "nay" votes. As it is, many of these comments aren't very substantial and are of little or no value anyway, and then I believe nobody should be forced to justify their vote, regardless whether it be "nay" or "yea".

 

Comments are not required. Peer Review instructions state you must give a reason for a Nay vote, but you do not have to write a single word.

 

I'm baffled. How could I give a reason without writing a single word?

 

.

 

Now this is getting ridiculous. I find it difficult to accept a fullstop as a reason. And anyway, if this is only kidding, then why insist on "giving a reason" in the first place?

[+fi67]

And anyway, if this is only kidding, then why insist on "giving a reason" in the first place?

A full stop has a lot to say if you carefully listen; similar to 'not global'. The system will not let you submit a Nay with an empty text box.

 

This requirement still makes sense. Not when the idea is not supported at all, but this is just a special case.

Often a cstegory is denied for formal reasons. Weak description, weird visit instructions or none at all, things like that.

 

These reasons help the officers to find out what they can do to improve the category and then resubmit a better version with a real chance to be accepted.

[+Max and 99]

I think this is in agreement with kittyposse (sorry if I'm misunderstanding).

 

I do think we should CONSIDER eliminating the text box in peer review. Any discussion/comments/feedback should be given in the forum discussion where responses can be provided.

 

Just a thought.

[+kittyposse]

 

These reasons help the officers to find out what they can do to improve the category and then resubmit a better version with a real chance to be accepted.

 

Absolutely agreed. That's why I suggested comments should be voluntary, not abolished. There would still be valuable feedback, but a smaller number of nonsense entries only made because the box can't be left blank. This way the remaining ones would be all the more valuable.

[+Max and 99]

Comments are already voluntary.

[+Benchmark Blasterz]

We always leave comments whether we vote yea or nay. We think they acknowledge the hard work of the proposed category owners and provide feedback in a place where many will see it.

 

I find the comments helpful when voting or when gauging if I need to be on the lookout for a probably successful category --

 

Leaving comments only in the forums -- do we know how many see those?

[+Benchmark Blasterz]

 

If a real expert decision about new categories is wanted, the group of voters could be limited to all existing category leaders and officers, because these are the people with experience about categories.

 

 

There are active waymarkers with WAY more experience than some category officers.

 

Benchmark Blasterz has a lot more experience than some officers!

 

[+BruceS]

I think this is in agreement with kittyposse (sorry if I'm misunderstanding).

 

I do think we should CONSIDER eliminating the text box in peer review. Any discussion/comments/feedback should be given in the forum discussion where responses can be provided.

 

Just a thought.

 

I disagree here. The text box and having the vote visible has done more to clean up the voting than anything else. The blind voting which was present before is what led to many categories getting approved that probably should not have. By having the vote visible let people know it is ok to vote no on categories. Yes it influences the voting of some but that was its intent. There have been categories that have gone to peer review that should not have because they were duplicates of other categories but having people vote and identify that problem early on in the vote fixed that problem, remember most waymarkers don't know all 1000 plus categories.

 

Yes categories should be discussed in the forums but there really is no way to enforce that and even if they are discussed in forums what ends up in peer review has sometimes been totally different. The text box gives the opportunity of for feedback. And having that feedback visible to others has cleaned that feedback up considerably. If you think the feedback is harsh now, when it was on publicly visible it was vicious at times.

[+BruceS]

...If a real expert decision about new categories is wanted, the group of voters could be limited to all existing category leaders and officers, because these are the people with experience about categories...

 

Silverquill wrote often about this problem. We lost our old forum so I can't cite him. I think that would be a good thing, all leaders and officers are allowed to vote.

 

I don't think this will gain what you want. The group leader can if he is a group leader of another group (even one not managing a category) promote all his friends to be officers in his other group and you are back to where you are now.

[+lumbricus]

...If a real expert decision about new categories is wanted, the group of voters could be limited to all existing category leaders and officers, because these are the people with experience about categories...

 

Silverquill wrote often about this problem. We lost our old forum so I can't cite him. I think that would be a good thing, all leaders and officers are allowed to vote.

 

I don't think this will gain what you want. The group leader can if he is a group leader of another group (even one not managing a category) promote all his friends to be officers in his other group and you are back to where you are now.

 

Perhaps only leaders and officers who manage a approved category? But I like Benchmark Blasterzs comments so this idea is not so good

[+Team Sieni]

One little think I'd like ... an email saying "Cream Cheese Sculptures has been submitted to peer review" (Other categories are available)

 

So I'd know when a new one had come up.

[+Benchmark Blasterz]

How about a running vote tally? Sometimes Mama Blaster runs out of fingers and toes -- ha ha!!

 

besides, Politicals are always interested in the vote totals --

[+Max and 99]

How about a running vote tally? Sometimes Mama Blaster runs out of fingers and toes -- ha ha!!

 

besides, Politicals are always interested in the vote totals --

 

Would the running vote tally on peer review include the anonymous ones?

[+silverquill]

Trying to catch up on the forums.

 

Yes, I have written about this in the past, so have we lost access to our old forums? I don't see them anymore. If Groundspeak has deleted them, then I think that is a mistake. Are they really that hard up for storage space? Don't see the reason for it.

 

I do think there are flaws in the whole process for creating new categories. We have categories that are created by one person with two sock-puppet accounts as officers, or other non-participating accts as officers. The single and only requirement for participation is paying Groundspeak for a premium membership, so it is all for sale.

 

This carries over into the peer review voting. I've seen some votes where nearly half of those voting had zero waymarks. Presumably these were geocaching friends recruited by the category creator to stuff the ballot box and get his category approved. I don't know of a way of preventing such abuses. (I suppose the same thing could happen if someone were intent on voting down a category.)

 

My main theme is that active participants should be the ones regulating the hobby, not just someone who has bought the privilege. I would like to see a minimum experience requirement set for someone to be able to create a group and category, be an officer or vote in peer review. I don't know that there is a way to do that, even if Groundspeak were inclined to do so, which they are not. Their philosophy is, the less interference the better. I agree with that to a point, but it does make things messy.

 

It is true that I think the category criteria should be rewritten, but not eliminated. We need to have some sort of guidelines. The criteria have served us well, but no longer fit the current needs and directions of Waymarking, in my opinion. The "global" criterion is one that I think needs some serious work. We have at least a third of our categories that are limited to a single country or region, for instance. I think we also need to make room for smaller, niche categories. Is a "wow factor" really necessary for a good category?

 

Obviously peer review will remain a subjective thing. Thanks for reposting BQ's excellent essay on this issue. The question is not, "Do I personally like this category?" Even if it seems boring to me, if a group is interested enough to create and manage a category, then who am I to rain on their parade? I look at other factors such as how well the category is defined, what the posting requirements are, clarity of the writing, objective criteria for waymark approval, and just overall sensibility.

 

While what we have is not perfect, peer review is better than no review, and there have been improvements. I think it is good that vote comments can be made visible to other voters, for instance, and that the officers can see all comments at the end of the vote. This is a good feedback mechanism. Still, this does not mean that we shouldn't discuss these issues and try to make them better. We'd have to come to Groundspeak with some concrete proposals, but since we are just individuals sharing a forum, there is no real mechanism for doing so.

[+fi67]

There is always room for improvement. Some minimum activity requirement for voting might be a good idea, and the criteria are not worded perfectly.

 

But in my opinion, some "fuzziness" is a strength of them, this leaves room for interpretation and allows to community to shift the common understanding without the need to rewrite. I think the "global" criterion is the clearest one, and I cannot remember any not-global category passing peer review with the (many) exception of additional countries for those categories that were geographically divided from start. There is much more disagreement on the meaning of prevalent or redundant.

 

While we all know that the whole process is a bit messy, I cannot remember any real incident. There was no category proposal ever approved just because of buddy votes, that should not have been, or anything like that.