Brewing a Better Rating System

Hi, I'm Mike from Steepster. We thought we'd share our new ratings system we just deployed with HN as we think it's relevant for products with customer reviews, ratings, etc. It's our attempt to combat the 4.3 dilemma (discussed here recently: <a href="http://news.ycombinator.com/item?id=883890" rel="nofollow">http://news.ycombinator.com/item?id=883890</a>).<p>Background: Steepster is a community site for tea drinkers to share their tasting notes, get recommendations, and discover new teas.<p>Feedback appreciated!

I love that the sliding meter shows tick marks for your previous rankings of other teas - the UI reflects that your judgement of a particular tea is relative to your other experiences.

The problem with this system is in the sorting. The list of "Highest Rated" teas is dominated by results where 1 person rated the tea 100.<p>Steepster should use a Bayesian average (<a href="http://en.wikipedia.org/wiki/Bayesian_average" rel="nofollow">http://en.wikipedia.org/wiki/Bayesian_average</a>) so that the uncertainty of a small number of ratings is reflected in the sorting.

The genius is adding some previous scores. I always struggle to rate stuff fairly without anything obvious to compare it with.

This is THE best implementation of a ratings system I've seen. Very good job.

I feel like this really combines the best of the granular 4 star systems with the specificity of a percentage rating. Really good stuff, I'd be interested to hear a follow up with user feedback on this approach and how it holds up long term.

This is cool, but I think virtually all rating systems suffer from the same basic problem: there's no way to turn it up to 11.<p>Take movies, for example. They are usually rated on a four-star scale. And yet, a three-star movie is a clear success. Few movies can realistically aspire to more than three stars. Even many four-star movies are really just trying desperately to avoid two-star land. Francis Ford Coppola was sure he was going to be fired any day from <i>The Godfather</i>. The production crew and actors on <i>Star Wars</i> thought it was practically a joke. Please, God, let <i>Star Wars</i> not be a <i>B</i> movie, they must have been thinking.<p>When you say ★★★ out of ★★★★, you make it look like it wasn't good enough: 75%. Movies really should be rated on a three-star scale: ★★★ out of ★★★; ★★★ = <i>A</i> = 100%. Anything else is gravy.<p>So, rate tea on a three-star scale. Three stars means "excellent tea, no clear way to make it better". ★★★½ means "Whoa, there <i>is</i> something better than ★★★!" ★★★★ means "This is <i>The Godfather</i> of tea! This tea makes me an offer I can't refuse."

In this context, it looks like the user has incentive to rate (to get better suggestions), but on rating in general:<p>Why even ask people how they feel?<p>Depending on the content, you can analysis how they use it to get a much more accurate rating. For video: Did they watch the entire thing? Did they leave after a few seconds? Did they share it somehow?<p>That (slightly off-topic) being said, this looks great.

Something else you could think about in a rating system like this would to instead of using generic faces, you could associate each with a common tea that most tea lovers have tried.<p>The notches kind of do this but theres always the risk of someone rating there first tea 80, then deciding subsequent teas after are better so they need to be rated higher, when the first one should have been more around 60.

I think main reason sliders aren't used is that users find it too troublesome, hence up/down and 5 stars are mainly used. I remember from my pys class that a 7 point rating system is best. But the 5 stars' simplicity and ubiquity probably trumps the benefits gained by a 7 point system. I think the best compromised is a 5 star UI implemented as 6 points by allowing 0 point assignments.

This is very cool thank you. I posted earlier today about the need for a better rating system than the five star system. I'm really glad to see you working on a great solution. Thanks!

Meh...