Announcement: Be excellent to each other.


Caravel Forum : Caravel Boards : General : How do you rate holds? (What's a perfect 10?)
1
Page 2 of 3
3
New Topic New Poll Post Reply
Poster Message
Schik
Level: Legendary Smitemaster
Avatar
Rank Points: 5381
Registered: 02-04-2003
IP: Logged
icon Re: How do you rate holds? (0)  
michthro wrote:
Well, I guess I'm way out there, but I'd prefer constructive criticism. It just seems obvious to me that free work goes hand in hand with constructive criticism, as opposed to a mere rating, where the architect doesn't necessarily know why someone doesn't like his hold. In other words, I don't like "give it 2/10, and *if you want to* tell the architect why". I prefer "If you really must give it 2/10, tell the architect why, and be nice about it. It's not your divine right that people should spend months building holds for you to criticise."
Okay, let me rephrase:

Hold ratings are there to give users an indication of how hard and how good a hold is before they play it.

Holds are each given their own thread on a board created specifically for Holds, so that people can give the architects constructive criticism, or pats on the back, or whatever else.

Let's see.. there are 3657 posts on the Holds board, and there have been 2233 hold ratings at this point. So people are talking about the hold more often than "just" giving it a rating. I agree that regardless of whether or not you give a hold a rating, you should make a comment in the Hold's thread to say what you thought about it. You don't have to say "I give this hold X brains and Y overall", but I'm sure the architects would love to hear the good and bad about their hold. Specific comments about a hold are worth a lot more to architects than a number is.

____________________________
The greatness of a nation and its moral progress can be judged by the way it treats its animals.
--Mahatma Gandhi
09-28-2006 at 05:47 PM
View Profile Send Private Message to User Send Email to User Show all user's posts High Scores Quote Reply
agaricus5
Level: Smitemaster
Rank Points: 1838
Registered: 02-04-2003
IP: Logged
icon Re: How do you rate holds? (0)  
I've followed the discussion here for a while, and would like to add my thoughts to the pool too. I apologise if I'm simply repeating what someone has already said, although I hope I'm not.

Schik wrote:
-I don't like rating down a hold that someone obviousy put a lot of time and effort into, even if I think it's poor quality despite that. I feel that rating holds is not the same as rating, say, authors (of books), to continue with an idea from the other thread. Architects are not making money or anything out of providing us with their work, which makes a big difference. It's similar to the old free-open-source attitude of if you don't like it, don't use it, and keep your wise-cracks to yourself; constructive criticism welcome, but saying it's worth 2/10 isn't constructive criticism. I even refrain from rating down poor quality holds that were thrown together in half an hour, but that's changing fast.
You seem to think the ratings are aimed mostly towards the architects, while I think they're for prospective players. Sure, architects want to get high ratings, but the rating system wasn't invented to give architects a pat on the back - it was to give players a way to make a somewhat informed decision about what hold they should try next. Rating something (2/10 or anything else) isn't meant to be constructive criticism. If you want to do that, then make a post in the hold's thread, and give as detailed criticism as you want to. But please rate the hold 2/10, if that's what it's worth, so that others know the hold's not all that great. If the architect learns from your criticism and makes the hold better, you can always change your rating.
Unfortunately, Schik, I disagree with you about what you think the rating system is useful for. I think a hold's score is important for Architects too, not because of vanity (although it can be encouraging), but for the exact same reason that it is important to players - the rating tells you what people who have played it think about it in general, and whether or not it is considered a "good" or "bad" hold. This is because it highlights whether or not people think the Architect built a good hold in terms of whether it fulfils its purpose as a hold, i.e. to be fun, playable, challenging, creative, unusual, or any other rating criterion that the hold was created to score highly in, depending on exactly which hold is being considered. Architects can then use this to consider if their style of hold is popular or not, for whatever reason, whether stylistic, structural or even because of bugs, and then use this information along with any comments to identify what could be improved next time.

The changing of opinions over a hold's lifetime, however, I agree, should be free to change if people think about holds differently; in the world of fashion, for example, what is popular today might not be so next month. Of course, holds aren't likely to undergo massive popularity changes in months, but like what michthro said, holds that were popular and fun to play a year ago may not be considered so now.

However, I believe the point being made, (implicitly), is that the rating criteria for holds is now far too complex to be representatively shown in just two, overall general ratings. A hold's true rating is as complicated as the total opinion of all the people who can, have, and will play it, and given that there are probably more than 500 DROD players around at present who have opinions about at least one hold, I doubt just two criteria are going to be enough to express the average opinion about a hold even fairly accurately.

For example, someone may not like Bavato's Dungeon because it is full of large hordes, lots of orb puzzles, and a lack of lenience towards certain types of mistakes, and would rate it 3 for tedium and 2 for frustration (i.e. lots of effort is required for little reward).

However, such a person may also like the variety of different puzzles available, the number of different concepts one must use, the aesthetics of the hold, and the fact that under the tedium and frustration, there were many puzzles that were highly enjoyable. So, the person might consider it a 8 for creativity and variety, 9 for the puzzles (i.e. how good the puzzles were at being, well, "good puzzles"), and 8 for overall coherence (i.e. as a hold, how well it fitted together and how much effort appeared to be put in to make it look that way).

So the person faces a bit of a dilemma - does he vote 9 to reflect his overall positive experiences, or maybe 2 to reflect his dislike of the more tedious aspects of the hold? Probably the most obvious thing to do would be to take the average of the scores and give it a fun rating of 6.

However, that's not quite satisfactory in the sense that it doesn't quite represent how "Fun" this person thought the hold actually was. In fact, this person didn't really enjoy it very much, since although he (I shall assume a gender for pronoun purposes) really enjoyed the variety of puzzles present and how many different things there were to do, he really found that tedious hack and slash techniques and having to play 50 moves sequences over several times in maybe half the rooms made the whole effort to play it not incredibly worthwhile, and reduced the fun factor of the hold. So, averaging what he would consider fun, he would actually consider it a 4. The 6 rating would also not reflect exactly what he thought about the quality of the hold (an overall rating criterion Eytan mentioned earlier, I think). Basically, although he didn't quite enjoy it, he was nonetheless impressed by the well-thought out puzzle ideas and the overall layout and aesthetics of the hold, and knows that while he doesn't like manipulating large hordes, someone else might think very differently. So, he would consider it to be a work of art that is of high quality, and rate it a 9 for that, even though he didn't really enjoy playing it.

I suppose an analogy would be that of, say, a building like the King's College Chapel, Cambridge. I personally don't find this kind of architecture interesting to look at or visit (I'm more into things like imaginary dungeons). However, I am impressed with the amount of quality work that's appears to have gone into building it (it is large and intricate, for example and doesn't look like it is made of over-sized Lego bricks :)), and know that its quality is therefore to be respected. So I can't really say I don't like it because I don't like that kind of architecture, and similarily, I can't say I like it because I think it was well-built; they're two completely different things.

So, my suggestion is to have at least one more rating criterion for holds. Right now, we have a "Fun" rating, which reflects how enjoyable it is for a person to play the hold, but this doesn't say much about the hold's quality, which is, I think, often a separate thing for people, and so would require a separate rating. Obviously, I guess the terms "fun", "quality", and any other rating term that someone else might use would need to be defined loosely to avoid ambiguity, and again, it won't entirely solve the problem of showing representative hold opinions, but at least it would help to make it more accurate (if a little more bulky).

Edit: I think the general consensus is probably similar; more rating criteria are probably needed, as Eytan also suggests.

____________________________
Resident Medic/Mycologist

[Last edited by agaricus5 at 09-28-2006 06:20 PM]
09-28-2006 at 05:53 PM
View Profile Send Private Message to User Send Email to User Show all user's posts This architect's holds Quote Reply
Banjooie
Level: Smitemaster
Avatar
Rank Points: 1645
Registered: 12-12-2004
IP: Logged
icon Re: How do you rate holds? (0)  
Michthro, in that other thread, said:
And everyone still keeps missing the point that it might be a bad idea to discourage new architects.

Yes, because when everybody downvoted Tokyokid into non-existence, he stopped putting out holds.

When everybody openly mocked Jeff_Ray's initial efforts (An event #1, anyone?) , he stopped putting out holds. (Which have gotten much better, incidentally.)

I'm not sure where this hypothetical sensitive architect is, but it certainly hasn't applied to anyone who ever has had serious downvoting here, at least since I've come by.



[Last edited by Banjooie at 09-28-2006 06:37 PM]
09-28-2006 at 06:37 PM
View Profile Send Private Message to User Show all user's posts This architect's holds Quote Reply
michthro
Level: Smitemaestro
Rank Points: 679
Registered: 05-01-2005
IP: Logged
icon Re: How do you rate holds? (+1)  
As I said in the other thread, you don't prove a universal statement by giving a few examples. Christ. Why am I arguing with someone who doesn't even understand that?
09-28-2006 at 06:43 PM
View Profile Send Private Message to User Show all user's posts Quote Reply
Ezlo
Level: Smitemaster
Avatar
Rank Points: 1214
Registered: 01-08-2006
IP: Logged
icon Re: How do you rate holds? (0)  
Because we shouldn't lie for the person who might just get offended.
09-28-2006 at 06:45 PM
View Profile Send Private Message to User Send Email to User Show all user's posts This architect's holds Quote Reply
Briareos
Level: Smitemaster
Avatar
Rank Points: 3516
Registered: 08-07-2005
IP: Logged
icon Re: How do you rate holds? (0)  
Ezlo wrote:
Because we shouldn't lie for the person who might just get offended.
Well, it's one thing participating in a discussion, and another being the creator of what's being discussed.

The latter probably needs a bit more asbesto in your clothing... *cough*

np: Radiohead - Let Down (OK Computer)

____________________________
"I'm not anti-anything, I'm anti-everything, it fits better." - Sole
R.I.P. Robert Feldhoff (1962-2009) :(
09-28-2006 at 06:52 PM
View Profile Send Private Message to User Send Email to User Visit Homepage Show all user's posts Quote Reply
Ezlo
Level: Smitemaster
Avatar
Rank Points: 1214
Registered: 01-08-2006
IP: Logged
icon Re: How do you rate holds? (0)  
I would much rather perfer totally honest opinions about my holds, so that my next ones would be better. If people can't take constructive critism, then they shouldn't make any more holds anyway.
09-28-2006 at 06:55 PM
View Profile Send Private Message to User Send Email to User Show all user's posts This architect's holds Quote Reply
michthro
Level: Smitemaestro
Rank Points: 679
Registered: 05-01-2005
IP: Logged
icon Re: How do you rate holds? (+1)  
Ezlo wrote:
Because we shouldn't lie for the person who might just get offended.
There's a difference between being honest and something like:
Anyway: I can understand playing three rooms of this hold, and going (lol spoiler)
Click here to view the secret text
There's also nothing constructive about that example. *Constructive* criticism, yes. That's what I've been saying all along. Trying to be funny at the expense of others, no.

09-28-2006 at 07:02 PM
View Profile Send Private Message to User Show all user's posts Quote Reply
Schik
Level: Legendary Smitemaster
Avatar
Rank Points: 5381
Registered: 02-04-2003
IP: Logged
icon Re: How do you rate holds? (+1)  
agaricus5 wrote:
Unfortunately, Schik, I disagree with you about what you think the rating system is useful for.
I don't think it's useless for architects, but nor do I think it's the most important thing for them. Here, I'll give you two (fake) reviews for Bavato's:
#1:
Click here to view the secret text

#2:
Click here to view the secret text
Now for the money questions:
Which one of these is worth more to the architect?
Which one of these is more likely to be shown in-game when people are looking at holds they could download?
I think a hold's score is important for Architects too, not because of vanity (although it can be encouraging), but for the exact same reason that it is important to players - the rating tells you what people who have played it think about it in general, and whether or not it is considered a "good" or "bad" hold.
It is important to architects, but that's not its main purpose. It doesn't exist so that architects know they made a hold that is a 7.4. It exists (and I know... I made it) so that players have a good idea how good and how hard a hold is going in.
So the person faces a bit of a dilemma - does he vote 9 to reflect his overall positive experiences, or maybe 2 to reflect his dislike of the more tedious aspects of the hold? Probably the most obvious thing to do would be to take the average of the scores and give it a fun rating of 6.
I look at "overall" as "How much do I want to suggest that others play this hold?" It's a pretty personal decision. If the things that bothered you really took away from your experience, you should probably lower your vote a notch or two. If they didn't, then don't.
...more rating criteria are probably needed
I disagree. I imagine we could easily come up with 10 or more things that we could rate the hold on, but is that really for the players or for the architect? If it's for the architect, then simply tell them.

By the way, this constructive criticism of the rating system is worth much more than if you all just said "Current Hold Rating System? 5."

____________________________
The greatness of a nation and its moral progress can be judged by the way it treats its animals.
--Mahatma Gandhi
09-28-2006 at 07:06 PM
View Profile Send Private Message to User Send Email to User Show all user's posts High Scores Quote Reply
agaricus5
Level: Smitemaster
Rank Points: 1838
Registered: 02-04-2003
IP: Logged
icon Re: How do you rate holds? (0)  
Schik wrote:
I disagree. I imagine we could easily come up with 10 or more things that we could rate the hold on, but is that really for the players or for the architect? If it's for the architect, then simply tell them.
But...

By the way, this constructive criticism of the rating system is worth much more than if you all just said "Current Hold Rating System? 5."
...as you say, more information is better than less. If the difficulty and fun ratings were combined into one single "Overall" rating, then imagine how much less information would exist in that rating. You could argue that difficulty and fun are not correlated, and that's why they are separate, but I could also argue that to some people fun and difficulty are the same thing, and the quality of a hold isn't to do with either. I think the key is to find the right level of information that gives a more reasonable level of accuracy to a rating, without it becoming too bulky for most people to use, or ambiguous because two criteria overlap greatly in most people's opinions.

I don't think it's useless for architects, but nor do I think it's the most important thing for them. Here, I'll give you two (fake) reviews for Bavato's:
#1 is way more useful for me than #2. But, #2 is the one shown in the DROD download page. So, what if I am looking for a hold to play, saw #2, and not #1, and I happen to hate large horde rooms and orb puzzles? Do I download the hold and find out for myself whether the contents are fun, and if I did do so, how far would I have to go to make a judgement on whether I want to play the hold or not? I could risk disliking and not seeing the rest of the hold if I don't see comment #1 and don't get past level 2, for example.

In such a case, the rating system fails because of its simplicity; what exactly does the "average fun rating" tell you as a player?

____________________________
Resident Medic/Mycologist

[Last edited by agaricus5 at 09-28-2006 07:21 PM]
09-28-2006 at 07:20 PM
View Profile Send Private Message to User Send Email to User Show all user's posts This architect's holds Quote Reply
Schik
Level: Legendary Smitemaster
Avatar
Rank Points: 5381
Registered: 02-04-2003
IP: Logged
icon Re: How do you rate holds? (0)  
agaricus5 wrote:
By the way, this constructive criticism of the rating system is worth much more than if you all just said "Current Hold Rating System? 5."
...as you say, more information is better than less. If the difficulty and fun ratings were combined into one single "Overall" rating, then imagine how much less information would exist in that rating.
Sure, but your suggestion is to change it to

"Current Hold Rating System? 5, 7."

That still tells me (the Hold Rating System architect) not a whole lot. I think for giving feedback to the architect, nothing is going to work as well as writing down your constructive criticism.

If we added another thing you can rate, what would happen to the 2000+ ratings we currently have? Would "overall" get copied into the new field? Would it just be blank? And you know someone will come along in a year and say "You know, we really need to be able to rate the scripting in a hold too. Let's add that to the things we can rate."

The system isn't set in stone, but I don't see anything that makes me want to change it yet.

____________________________
The greatness of a nation and its moral progress can be judged by the way it treats its animals.
--Mahatma Gandhi
09-28-2006 at 08:01 PM
View Profile Send Private Message to User Send Email to User Show all user's posts High Scores Quote Reply
Ezlo
Level: Smitemaster
Avatar
Rank Points: 1214
Registered: 01-08-2006
IP: Logged
icon Re: How do you rate holds? (0)  
I've an idea.

Let's make it so you have to post to vote.

When you're making a post in hold topic, there will be the scoring and difficulty bar as usual. There will be no other way to vote. If you don't want to vote when you post, just leave it blank.

Your vote will appear over your post, and the overall will be at the top as usual. This will encourage people to give comments as well as vote.
09-28-2006 at 08:07 PM
View Profile Send Private Message to User Send Email to User Show all user's posts This architect's holds Quote Reply
eytanz
Level: Smitemaster
Avatar
Rank Points: 2708
Registered: 02-05-2003
IP: Logged
icon Re: How do you rate holds? (0)  
Schik wrote:
agaricus5 wrote:
By the way, this constructive criticism of the rating system is worth much more than if you all just said "Current Hold Rating System? 5."
...as you say, more information is better than less. If the difficulty and fun ratings were combined into one single "Overall" rating, then imagine how much less information would exist in that rating.
Sure, but your suggestion is to change it to

"Current Hold Rating System? 5, 7."

That still tells me (the Hold Rating System architect) not a whole lot. I think for giving feedback to the architect, nothing is going to work as well as writing down your constructive criticism.

If we added another thing you can rate, what would happen to the 2000+ ratings we currently have? Would "overall" get copied into the new field? Would it just be blank? And you know someone will come along in a year and say "You know, we really need to be able to rate the scripting in a hold too. Let's add that to the things we can rate."

The system isn't set in stone, but I don't see anything that makes me want to change it yet.

Well, as I said in another thread ( I forget which), what I really would appreciate is adding a display of the standard deviation in addition to the mean score, which would give a fuller picture. I agree that just adding more scores is an iffy premise.

____________________________
I got my avatar back! Yay!
09-28-2006 at 08:15 PM
View Profile Send Private Message to User Show all user's posts This architect's holds Quote Reply
eytanz
Level: Smitemaster
Avatar
Rank Points: 2708
Registered: 02-05-2003
IP: Logged
icon Re: How do you rate holds? (+1)  
Ezlo wrote:
I've an idea.

Let's make it so you have to post to vote.

When you're making a post in hold topic, there will be the scoring and difficulty bar as usual. There will be no other way to vote. If you don't want to vote when you post, just leave it blank.

Your vote will appear over your post, and the overall will be at the top as usual. This will encourage people to give comments as well as vote.

No, that'll make things much worse. It should be as easy as possible to vote, so the most people will vote, and then the scores will most accurately reflect general opinion. By restricting the voters to a few, you make these few a lot more influential, and the most vocal people are not normally the people best suited for speaking for everyone (I know I'm not).

____________________________
I got my avatar back! Yay!
09-28-2006 at 08:17 PM
View Profile Send Private Message to User Show all user's posts This architect's holds Quote Reply
Ezlo
Level: Smitemaster
Avatar
Rank Points: 1214
Registered: 01-08-2006
IP: Logged
icon Re: How do you rate holds? (0)  
What do you mean? How is this limiting people who can vote? Everyone can post.
09-28-2006 at 08:18 PM
View Profile Send Private Message to User Send Email to User Show all user's posts This architect's holds Quote Reply
eytanz
Level: Smitemaster
Avatar
Rank Points: 2708
Registered: 02-05-2003
IP: Logged
icon Re: How do you rate holds? (0)  
Ezlo wrote:
What do you mean? How is this limiting people who can vote? Everyone can post.

Yes, but only few do. Basically, the more work you make it to vote, the less people will vote.

"limiting" was a bad wrong word choice - I should have said "reducing".

(edit - I didn't actually say "limiting", I said "restricting". But, even though I am a semanticist, I don't want to quibble on the meaning of individual words right now, and I think my point above is clear).

____________________________
I got my avatar back! Yay!

[Last edited by eytanz at 09-28-2006 11:10 PM]
09-28-2006 at 08:19 PM
View Profile Send Private Message to User Show all user's posts This architect's holds Quote Reply
Schik
Level: Legendary Smitemaster
Avatar
Rank Points: 5381
Registered: 02-04-2003
IP: Logged
icon Re: How do you rate holds? (0)  
eytanz wrote:
Well, as I said in another thread ( I forget which), what I really would appreciate is adding a display of the standard deviation in addition to the mean score, which would give a fuller picture. I agree that just adding more scores is an iffy premise.
Yeah, I saw that. Where do you think this would fit in? One place in particular? Everywhere that the mean score is displayed?

____________________________
The greatness of a nation and its moral progress can be judged by the way it treats its animals.
--Mahatma Gandhi
09-28-2006 at 08:19 PM
View Profile Send Private Message to User Send Email to User Show all user's posts High Scores Quote Reply
eytanz
Level: Smitemaster
Avatar
Rank Points: 2708
Registered: 02-05-2003
IP: Logged
icon Re: How do you rate holds? (0)  
The latter, unless there's a particularly good reason not to. At least, it should be displayed on the hold's header and in the game.

____________________________
I got my avatar back! Yay!
09-28-2006 at 08:21 PM
View Profile Send Private Message to User Show all user's posts This architect's holds Quote Reply
Schik
Level: Legendary Smitemaster
Avatar
Rank Points: 5381
Registered: 02-04-2003
IP: Logged
icon Re: How do you rate holds? (0)  
eytanz wrote:
The latter, unless there's a particularly good reason not to. At least, it should be displayed on the hold's header and in the game.
One concern I have is on the holds page, there's not a whole lot of room to put it anywhere, it's already packed so tight. On the Holds board it wouldn't be an issue of course. I also wonder if enough people know enough about standard deviation to make this useful to add.

____________________________
The greatness of a nation and its moral progress can be judged by the way it treats its animals.
--Mahatma Gandhi
09-28-2006 at 08:43 PM
View Profile Send Private Message to User Send Email to User Show all user's posts High Scores Quote Reply
Schik
Level: Legendary Smitemaster
Avatar
Rank Points: 5381
Registered: 02-04-2003
IP: Logged
icon Re: How do you rate holds? (0)  
Okay, I added it to the threads on the Holds board so we can look at it and see if it's something worth adding elsewhere. If we keep it, I'll clean up the presentation. :)

____________________________
The greatness of a nation and its moral progress can be judged by the way it treats its animals.
--Mahatma Gandhi
09-28-2006 at 09:29 PM
View Profile Send Private Message to User Send Email to User Show all user's posts High Scores Quote Reply
Tim
Level: Smitemaster
Avatar
Rank Points: 1979
Registered: 08-07-2004
IP: Logged
icon Re: How do you rate holds? (+2)  
I must say that the suggestions mentioned in the current active threads are quite interesting. According to them, I should give a hold a score based on:

10 for minimum
+1 if only one person can finish it
+1 if it's the architect's first hold
+1 if the hold has no mentionable story
+1 if there are no breaks between rooms/levels
+1 if it does not motivate me to finish the hold
+1 if the architect's family says it's worth a 10
+1 if the rooms looks terrible
+1 if someone else like it a lot and wants everyone vote like him/her
+1 if it is not completable
+1 if everyone likes it but me
+1 if it changes the holds often and invalidates someone else's score
+1 if it's an old hold
+1 if it's a SmS hold

Schik, can you please change your scoring system to 1-25?

(I will write a section how I really rate holds later (as I can't think really clear at the moment),
but I do want to thank all these nice people here who want to tell me what to vote instead.)
[/sarcasm]

____________________________
The best way to lose customers is to let little kids running loose on a forum with too many mod points.

[Last edited by Tim at 09-28-2006 09:40 PM : sarcasm tag]
09-28-2006 at 09:39 PM
View Profile Send Private Message to User Show all user's posts This architect's holds Quote Reply
michthro
Level: Smitemaestro
Rank Points: 679
Registered: 05-01-2005
IP: Logged
icon Re: How do you rate holds? (0)  
Tim wrote:
+1 if someone else like it a lot and wants everyone vote like him/her
Now you're really irritating me, Tim. I'm the one who's repeatedly been accused of wanting everyone to vote like me, despite reapeatedly having spelled out that I do not expect anyone else to vote the same way as me. I'm sorry if by some coincidence that wasn't aimed at me, but I think it's unlikely. I have tried again and again to make it clear that I'm concerned about new architects being put off, something that has been said before on another topic and no-one argued then. That, and I want *constructive* criticism. I really don't see what's wrong with wanting that, and I really don't see why caring about new architects being put off is such a bad idea now that I'm saying it. And no, I'm not saying something ridiculous such as that holds by new architects should be rated at least 4 or something.

I'm looking forward to your rating criteria, and finding ways to twist them into reasons to rate your holds down, just like you're twisting people's words into ridiculous statements.

I wouldn't do that, but you see what I'm getting at?

09-28-2006 at 10:33 PM
View Profile Send Private Message to User Show all user's posts Quote Reply
Tim
Level: Smitemaster
Avatar
Rank Points: 1979
Registered: 08-07-2004
IP: Logged
icon Re: How do you rate holds? (0)  
michthro wrote:
Tim wrote:
+1 if someone else like it a lot and wants everyone vote like him/her
Now you're really irritating me, Tim. I'm the one who's repeatedly been accused of wanting everyone to vote like me...
Being intoxicated is not a good way to write clearly, so please excuse me. It's definitely not aimed at you. I've done this more than once myself (which, if you don't mind, I don't want to dig up those holds again).
09-28-2006 at 10:54 PM
View Profile Send Private Message to User Show all user's posts This architect's holds Quote Reply
michthro
Level: Smitemaestro
Rank Points: 679
Registered: 05-01-2005
IP: Logged
icon Re: How do you rate holds? (+2)  
Great. And now I'm arguing with a drunk. Mind you, I actually prefer it, and at least you'll be sober tomorrow. (Well, I hope.) Have fun.
09-28-2006 at 11:04 PM
View Profile Send Private Message to User Show all user's posts Quote Reply
agaricus5
Level: Smitemaster
Rank Points: 1838
Registered: 02-04-2003
IP: Logged
icon Re: How do you rate holds? (0)  
Schik wrote:
Okay, I added it to the threads on the Holds board so we can look at it and see if it's something worth adding elsewhere. If we keep it, I'll clean up the presentation. :)
Just as a bit of information, the statistics of the scoring system are probably quite interesting with lots of votes. The distribution of scores is highly likely to be non-normal, probably bi-modal (with the smaller peak near the low end), and also probably skewed well over towards the top.

Basically, the graph showing the number of votes for each score is likely not to be bell-shaped like the Normal Distribution, has two peaks, not one, with a small peak near the lower score end, and a bigger one around the mean, and the right side of the second peak (i.e. the number of people who voted 9 and 10) is likely to be reasonably high compared to the peak.

Although the distributions are not normal, there is a theorem called the Central Limit Theorem that states roughly that for any distribution with a mean, (μ), and finite standard deviation (σ), if you pick at random a certain number of people or things from it (in this case, you could pick a number of DRODders at random and ask them what they would vote), and average the numbers, you'll find that if you did it over and over many times, the average mean you'd get will be μ, and the means would be distributed approximately normally (i.e. the graph of means would be roughly bell-shaped), with a standard deviation of σ/√n, where n is the number of things you picked.

So, for example, Bavato's Dungeon, with 45 votes, has a mean score of 8.8, and a standard deviation of 2.13.
I expect that because the mean is near 9, that the peak is at around 9 (i.e. the largest number of votes were 9s), and that there are also quite a few 10s as well. The standard deviation is quite large, meaning the scores are probably very widely spaced. So, I expect there to be many 8s, fewer 7s, and probably almost zero 6s, with a small peak between 3-5, and maybe one or two 1s and 2s.

If I were then to do a random experiment by picking 45 hapless DRODers at random from the population and ask them to vote on Bavato's Dungeon, and do this many many times (let's say, 100 times), if the 45 votes the hold has now are representative of what people think, then...

I would find the average scores for the groups of people will probably have a mean of something near 8.8. I would also find the standard deviation of these averages to to be about 0.318, and the average scores put onto a graph would probably look roughly like a bell-shaped Normal curve. As nearly 95% of all the values will probably lie between 2 standard deviations of the mean, I can be probably 95% sure that the true mean (i.e. the mean I'd get if I made everyone vote, including those who will play DROD in the future) lies between 8.16 and 9.34.

For another example, HIJK has 22 votes, a μ of 8.8 and a σ of 0.89. The σ is much smaller, and so consequently, probably most people voted between 7 and 10, with very few low votes. Assuming the current votes are representative of everyone, we can be 95% sure that the true mean rating for the hold is between 8.42 and 9.18. Basically, it is more likely that the average rating for HIJK is around 8.8, while the average rating for Bavato's Dungeon is more likely to move either up or down, rather than remain at 8.8 (it's probably more likely to fall, given that it has already fallen over the last 9 months).

And I guess that's about it.

____________________________
Resident Medic/Mycologist

[Last edited by agaricus5 at 09-28-2006 11:37 PM]
09-28-2006 at 11:13 PM
View Profile Send Private Message to User Send Email to User Show all user's posts This architect's holds Quote Reply
Tahnan
Level: Smitemaster
Avatar
Rank Points: 2459
Registered: 11-14-2005
IP: Logged
icon Re: How do you rate holds? (+1)  
OK, that last post only made me dizzy, so I'll throw out a couple of somewhat older comments:

First: would be as helpful (more helpful? helpful instead?) if we could see what the votes are, rather than the standard deviation? That is, if you could (e.g.) click the text "Average User Rating" and see the range, displayed in the same way that poll results are displayed? That way you could quite literally tell "ah, ok, there's a 2 on there, but everything else is in the 8-10 range", which should--I believe--go a long way to mitigating michthro's concerns about discouraging architects. (It's a poor architect who's discouraged by a single 2 in a field of 9s.)

Second: I agree with eytanz's advocation of keeping voting as simple as possible; for that reason, I don't think it's a good idea to separate "quality" and "fun". And, to be honest, I don't know that I'm generally qualified to rate "quality"--for instance, I happen to love trapdoor puzzles but hate tar-on-trapdoor puzzles; I like Perfection-like "move at exactly the right time" puzzles, and don't really like horde management puzzles. Given that: I don't think I can really adequately rate the quality of a hold that has tar-on-trapdoor puzzles; since I don't like any of them, I don't have any good sense of whether a particular hold's instances are well-done or not. (Similarly, I can tell you exactly how good I think the rooms in Perfection are; but when it comes to the "war" series, well, I know what I think of it, but not really whether they're especially good horde-management rooms.)
09-29-2006 at 12:28 AM
View Profile Send Private Message to User Show all user's posts High Scores This architect's holds Quote Reply
agaricus5
Level: Smitemaster
Rank Points: 1838
Registered: 02-04-2003
IP: Logged
icon Re: How do you rate holds? (0)  
Tahnan wrote:
First: would be as helpful (more helpful? helpful instead?) if we could see what the votes are, rather than the standard deviation? That is, if you could (e.g.) click the text "Average User Rating" and see the range, displayed in the same way that poll results are displayed? That way you could quite literally tell "ah, ok, there's a 2 on there, but everything else is in the 8-10 range", which should--I believe--go a long way to mitigating michthro's concerns about discouraging architects. (It's a poor architect who's discouraged by a single 2 in a field of 9s.)
That statistical analysis was only useful if you cannot see the graph (like now), and I agree, a graph would be far more useful than simply the s.d., at the expense of the system being less simple overall.

Second: I agree with eytanz's advocation of keeping voting as simple as possible; for that reason, I don't think it's a good idea to separate "quality" and "fun". And, to be honest, I don't know that I'm generally qualified to rate "quality"--for instance, I happen to love trapdoor puzzles but hate tar-on-trapdoor puzzles; I like Perfection-like "move at exactly the right time" puzzles, and don't really like horde management puzzles. Given that: I don't think I can really adequately rate the quality of a hold that has tar-on-trapdoor puzzles; since I don't like any of them, I don't have any good sense of whether a particular hold's instances are well-done or not. (Similarly, I can tell you exactly how good I think the rooms in Perfection are; but when it comes to the "war" series, well, I know what I think of it, but not really whether they're especially good horde-management rooms.)
With the horde management vs. precision move sequence preference, I'm exactly the opposite of you. I vastly prefer the former over the latter. Nonetheless, I don't feel inclined to give Perfection a 3 simply because I really don't like having to work out for an entire hold exactly which 100-move sequence in maybe 10^7 that I need to use for each room. I know that the rooms themselves are well-designed, and the puzzle ideas very sound; it's just that I don't like their actual incarnations. However, that makes it difficult for me to rate it, based on what I've seen. I don't want to give it a 9 for its ingenuity, and yet not a 3 for what I find frustrating. 6 doesn't really do justice to the hold, in my opinion, and so I'd have to settle for 7 or 8, even though that's not quite what I think of it.

Generally, though, on this system, I find I rate on both fun and quality, take the highest number, and subtract 1 if the two are more than 3 points from each other. It usually shows my opinion quite well, but not always, as is the case with Perfection and A Quiet Place, for example.

____________________________
Resident Medic/Mycologist
09-29-2006 at 12:53 AM
View Profile Send Private Message to User Send Email to User Show all user's posts This architect's holds Quote Reply
Banjooie
Level: Smitemaster
Avatar
Rank Points: 1645
Registered: 12-12-2004
IP: Logged
icon Re: How do you rate holds? (0)  
michthro wrote:
I'm the one who's repeatedly been accused of wanting everyone to vote like me, despite reapeatedly having spelled out that I do not expect anyone else to vote the same way as me.

You don't seem to expect everyone else to vote the same way, you just don't want anyone else to vote the other way.

And why was this thread won by a drunken man?
09-29-2006 at 03:50 AM
View Profile Send Private Message to User Show all user's posts This architect's holds Quote Reply
eytanz
Level: Smitemaster
Avatar
Rank Points: 2708
Registered: 02-05-2003
IP: Logged
icon Re: How do you rate holds? (0)  
Banjooie wrote:
michthro wrote:
I'm the one who's repeatedly been accused of wanting everyone to vote like me, despite reapeatedly having spelled out that I do not expect anyone else to vote the same way as me.

You don't seem to expect everyone else to vote the same way, you just don't want anyone else to vote the other way.

I must say, Michthro, that Banjooie just summarized exactly what I thought you were saying as well. So if that's not what you meant, than either you weren't communicating very clearly, or me and Banjooie have reading comprehension issues.

____________________________
I got my avatar back! Yay!
09-29-2006 at 04:23 AM
View Profile Send Private Message to User Show all user's posts This architect's holds Quote Reply
Banjooie
Level: Smitemaster
Avatar
Rank Points: 1645
Registered: 12-12-2004
IP: Logged
icon Re: How do you rate holds? (0)  
It's not your divine right that people should spend months building holds for you to criticise."

It's not your divine right that people should spend months building holds for you to rate it a 10.

It goes both ways, and you've yet to tell me why a high rating is inherently more justified than a low rating.
09-29-2006 at 08:29 AM
View Profile Send Private Message to User Show all user's posts This architect's holds Quote Reply
1
Page 2 of 3
3
New Topic New Poll Post Reply
Caravel Forum : Caravel Boards : General : How do you rate holds? (What's a perfect 10?)
Surf To:


Forum Rules:
Can I post a new topic? No
Can I reply? No
Can I read? Yes
HTML Enabled? No
UBBC Enabled? Yes
Words Filter Enable? No

Contact Us | CaravelGames.com

Powered by: tForum tForumHacks Edition b0.98.8
Originally created by Toan Huynh (Copyright © 2000)
Enhanced by the tForumHacks team and the Caravel team.