Star Ratings
How sure can you be that a product is good from its star ratings? Enter the ratings of two products, choose what quality means and a prior, and read off the probability that each clears the bar, and that one is better than the other.
Quality
Prior
Ratings
Product A
posterior above the bar mean and 95% interval observed bar c
- Pr(quality > c | ratings)
- Posterior mean
- 95% credible interval
- Posterior
a product is good if its quality exceeds c
Product B
posterior above the bar mean and 95% interval observed bar c
- Pr(quality > c | ratings)
- Posterior mean
- 95% credible interval
- Posterior
The model
A product has an unknown rating distribution p = (p1, …, p5): the share of everyone who would give it 1 to 5 stars. Each rating is a draw from p, so with counts n1, …, n5 the likelihood is ∏ pknk. The conjugate prior is Dirichlet(α1, …, α5), with density ∝ ∏ pkαk − 1, and the posterior is Dirichlet(α1 + n1, …, α5 + n5): each αk acts as k-star ratings counted before the data, and the Beta–Bernoulli update is the two-category case.
Mean rating. Quality is the average rating the product would get from everyone, μ = Σ k pk, a number between 1 and 5. Its posterior has no simple closed form, so the page draws 40,000 rating distributions from the posterior Dirichlet (normalized Gamma draws), computes μ for each, and takes the curve, the mean, the interval and the probabilities from these samples; probabilities are accurate to about ±0.005.
Share of positive ratings. Grouping 4 and 5 stars as positive and the rest as negative reduces the model to a Bernoulli one: quality is θ = p4 + p5, the prior is Beta(α, β), and with h positive and t negative ratings the posterior is Beta(α + h, β + t).
Priors. Flat: every rating distribution is equally plausible, Dirichlet(1, …, 1) or Beta(1, 1). Jeffreys: Dirichlet(½, …, ½) or Beta(½, ½), the prior that does not depend on how the parameter is written down; it puts extra weight near the edges. Skeptic: ratings counted before the data that lean low, ten pseudo-ratings averaging 2.4 stars, or Beta(2, 8), two positive and eight negative. Custom: any pseudo-counts.
Comparing two products. Given the ratings, the two qualities are independent, so Pr(A is better than B) = Pr(μA > μB) is computed over all pairs of one sample from each posterior.