Hi folks, if you haven’t encountered me before, I’m Justin French, Product Owner of the Marketplaces. As I’ve previously mentioned in other threads, the rating system is the next major pain point we’d like to focus on in improving the Marketplaces this year. There’s certainly no shortage of feedback and ideas!
Like many recent projects, we’re taking this potentially epic piece of work and splitting it into smaller pieces we can confidently ship really soon, and some more we will look at later.
Show a breakdown of the average
This is an incredibly popular and long-running request! We’re redesigning the sidebar of an item page to accomodate a little more detail, rather than just the overall average rating — buyers will see the number of people who have rated the item at each step (5 stars, 4 stars, etc), the total number of ratings received and the precise numerical value of the average (e.g. “4.6” or “3.9”). This work is mostly complete and will ship soon!
Rounding to half stars
Also requested, we’ll start rounding to the nearest half-star rather than whole star. Currently an average rating of “4.6” is rounded to “5”, “4.3” is rounded to “4”. By rounding to the nearest half star, an average “4.6” or “4.3” will be rounded to “4.5” instead. Some items will go “up” a half star, and others “down”, but overall, the graphical representation will be closer to the real numerical average. This work is mostly complete and will ship soon.
Encouraging more ratings
We want to encourage and remind buyers to provide a rating for any of their unrated purchases. We’ll start with an email follow-up a week after purchase and measure what effect this has on the number of ratings per item. This is also mostly complete and will ship soon.
What are we rating, exactly?
We don’t explicitly state anywhere what a buyer should be thinking about when they’re rating an item. Is it the technical quality? Aesthetics? The author’s support? Flexibility & configuration options? The author’s willingness to help with questions or customisation? The purchasing experience? The installation process? Value for money?
To help us better understand what’s already taking place, we’re going to run a survey, interview a range of authors and buyers, and start collecting the primary reason (selected from a short list of predefined options) from buyers as they rate each item. There’s no plans to display this data on the site (not even to the author), but it will help us better understand how the rating system is currently being used, and to plan our approach to some of the remaining feature requests.
Speaking of feature remaining requests…
Who rated this, and why?
An often-requested feature is buyers providing a comment when they rate an item, displayed alongside their username — some propose this should be required for all ratings, others just for one and five star ratings. There’s a lot of hidden features we might need here too (moderation & dispute resolution, marking reviews as unhelpful, etc).
The underlying goals of these requests are:
- improve transparency for everyone
- provide authors with insights to help improve their items
- provide relevant ratings to buyers to help their purchasing decisions
These goals are important to us, but we’re not starting on the build right now — the data we’re collecting and the research we’re working on will help us design the details of a system that best meets the community’s needs, as well as ours.
We’ll update you with what we’ve learned (and our next steps) in a month or two.
We’re also working with a statistician to explore some alternatives to the existing method of averaging we use (including bayesian and median, which have been brought up a few times by the community). These are only experiments to review what happens to the data at this point. If it looks like we’re going to make a change in the future, we’ll keep you in the loop.
Ratings are forever?
Right now, a rating is forever — we calculate the average using every available rating since the item was first uploaded, which means it’s a mixture of historical and current ratings. We’re working with a statistician to explore the impact of ignoring older ratings. If an item has greatly improved over time, this might lead to a rating which is more reflective of what you’re buying today. Again, if it looks like we’re going to make a change in the future, we’ll discuss what we’ve learned.
I wanted to share this plan before we deploy the first batch of improvements so that you aren’t left wondering “is that it?”. There’s plenty more to be done, and this is the first step!
Edit: Made a small clarification to “What are we rating, exactly?” 4pm AEST (about 5hrs after the post).
- Forum Superstar
- Exclusive Author
- Sold between 10 000 and 50 000 dollars
- Russian Federation
- Has been a member for 2-3 years
- Most Wanted Bounty Winner
- Referred between 10 and 49 users
- Bought between 10 and 49 items
- Grew a moustache for the Envato Movember competition
Sounds great! Thank you for the information.
Best Regards, Andrey
Sounds really great! My only wish now is to have it implemented as soon as possible
Good stuff! Thanks for the thorough insights.
This will be nice! I have a couple 4 star items that may actually be 4.5 stars
- Has been a member for 5-6 years
- Author has had an Item Featured
- Author was Featured
- Bought between 100 and 499 items
- Referred between 200 and 499 users
- Exclusive Author
- Microlancer Beta Tester
- Author had a Free File of the Month
This is very exciting news and makes me very happy that finally something is happening here! Good job Justin and whole team!
Great News. Can’t wait to see it active
Great to know about that. I’ve been waiting for this for years . Cheers to the team!
Great News ! I’m looking forward to see the result