Statistics algorithm for sorting ratings looks fishy
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
software-center (Ubuntu) |
Triaged
|
Medium
|
Unassigned |
Bug Description
Here's the current code snippet for sorting the Software Center Ratings:
def wilson_score(pos, n, power=0.2):
if n == 0:
return 0
z = pnormaldist(
phat = 1.0 * pos / n
return (phat + z*z/(2*n) - z * math.sqrt(
def calc_dr(ratings, power=0.1):
'''Calculate the dampened rating for an app given its collective ratings'''
if not len(ratings) == 5:
raise AttributeError(
tot_ratings = 0
for i in range (0,5):
tot_ratings = ratings[i] + tot_ratings
sum_scores = 0.0
for i in range (0,5):
ws = wilson_
sum_scores = sum_scores + float((i+1)-3) * ws
return sum_scores + 3
This looks very fishy to me, as we are calculating 5 different wilson scores per rating and summing them. This is slow, and probably wrong. I'm not 100% sure about what the right method to use is, however I did find the question asked on Math Overflow:
The current answer there suggests using a standard normal distribution for large samples, and a T-distribution for low ones (we don't do either)
This website suggests a slightly different Wilson algorithm:
http://
I will go further, and assert that we are making a conceptual error in trying to estimate a mean rating in the first place: ratings are fundamentally ordinal data, and thus a mean doesn't make much sense for the same reason that "excellent" + "terrible" does not balance out to "mediocre". However, taking medians and percentile data is very much valid measurement.
I will research this question a bit more, and probably post a question on the beta stats stackexchange site for advice. Intuitively, though, I think we may want to have a ratings algorithm that sorts primarily based on median, and then for the large number of cases where two apps have the same median (since we only have 5 ratings), we then compute a wilson score for the lower bound of the probability that a rater of App A would rate >= median vs < median.
tags: | added: client-server |
Here is my question on Cross Validated, the statistics stack exchange: http:// stats.stackexch ange.com/ questions/ 19115/how- do-i-sort- an-ordinal- list-of- user-generated- ratings- data