The crowd was never the problem

In 1906, at a livestock fair in Plymouth, Francis Galton watched 787 people pay sixpence each to guess the dressed weight of an ox. He expected the crowd to be foolish. When he collected the tickets, the median guess was within one percent of the true weight, and the mean was one pound off [1]. The result, published in Nature the following year, is the founding story of the wisdom of crowds, and a century later it became the founding premise of the social web.

It is worth noticing what the crowd at Plymouth had going for it. Each guess was written on a sealed ticket, so nobody could see the others. A guess cost money, so nobody was playing. Most of the guessers were farmers and butchers who had handled cattle, so most were better than chance. And there was a true weight, waiting at the end, against which every guess could be checked. James Surowiecki, who made the story famous, named the conditions a crowd needs to be wise: diversity, independence, decentralisation and a way to aggregate [2]. The ox had all four.

The web kept the arithmetic and dropped the conditions.

What the web built

The second generation of the internet, the one we now call Web2, was built on a single democratic idea: every voice counts once. A star rating is an average of everyone who clicked. An upvote is a vote. A product with ten thousand reviews outranks one with ten. Wikipedia would be written by anyone, Reddit would be ranked by everyone, and the restaurant down the street would be judged by whoever walked in. If enough opinions were collected, the truth would rise.

For a while, in low-stakes places, it did. But the design made three assumptions that Galton’s fairground never made. It assumed every voice was equally informed. It assumed every voice was honest, or at least that dishonest ones would be drowned out. And it assumed the voices were independent, when in fact every one of them could see the running total before it spoke. Where the stakes rose, where money, reputation, grades or elections were on the line, each of those assumptions failed in its own way, and the failures have names.

The average that drifted to the ceiling

The first failure is quiet. On platforms where people rate each other, the ratings drift upward until they say nothing. Apostolos Filippas, John Horton and Joseph Golden studied an online labour market over several years and found that average feedback rose steadily toward the maximum, not because the workers improved but because raters felt pressure to leave an “above average” score and did not want to harm the person on the other side. Their conclusion was blunt: since the potential to harm is what makes a rating informative, “reputation systems sow the seeds of their own irrelevance” [3].

The pattern is everywhere once you look. On eBay, Chris Nosko and Steven Tadelis found the published “percent positive” score so compressed that the median seller had near-perfect feedback and buyers could not tell good from bad. The useful signal turned out to be silence: the buyers who left no feedback at all. Counting them as a quiet negative produced a measure that actually predicted which sellers would disappoint [4]. On Airbnb, Georgios Zervas, Davide Proserpio and John Byers found that nearly 95 percent of properties carried an average of 4.5 or 5 stars, while the same cities’ hotels averaged 3.8 on sites where guests had no host to face [5].

A grade that everyone receives is not a grade. The ceiling is where an unweighted average goes when the raters have a reason to be kind, and the person who pays is the excellent host, driver or seller who can no longer be distinguished from the adequate one.

The voice that could be bought

The second failure is louder. If every voice counts once and a voice costs nothing to create, voices will be manufactured. Michael Luca and Georgios Zervas examined the reviews Yelp’s own filter flagged as suspicious and found that roughly 16 percent of restaurant reviews were caught, that the flagged reviews were more extreme than the rest, and that a restaurant was most likely to post fake praise when its reputation was weak and most likely to receive fake attacks when a competitor opened nearby [6]. Fraud, in other words, was not random noise. It was a rational response to a system that counted every star the same.

The scale is now industrial. Amazon reported that in 2023 it blocked more than 250 million suspected fake reviews and took legal action against more than 150 brokers who sell them [7]. In August 2024 the United States Federal Trade Commission issued a rule banning fake reviews outright, with penalties of up to roughly $52,000 per violation, including reviews written by people who do not exist and reviews generated by AI [8]. A regulator does not write a rule against a nuisance. It writes one against a market.

The same mechanism works on culture. When a Star Wars film divided its audience in 2017, a researcher at the University of Southern California examined the tweets sent to its director and found that about half of the negative ones came from bots, sock puppets or politically motivated accounts, including a cluster that matched the signature of Russian troll operations [9]. Two years later Rotten Tomatoes changed its audience score so that only people who could prove they had bought a ticket would count [10]. Notice what the fix was. It did not add more voices. It weighted them, by evidence that the voice had seen the thing it was judging.

The herd

The third failure undoes Galton’s sealed tickets. On the web, every voice can see the tally before it speaks, and that changes what it says. Lev Muchnik, Sinan Aral and Sean Taylor ran a randomised experiment on a social news site, giving some comments a single artificial upvote the moment they were posted. That one vote made the next reader 32 percent more likely to vote up as well, and the herding that followed raised the comment’s final score by 25 percent on average [11]. Nothing about the comment had changed. Only the number next to it.

Matthew Salganik, Peter Dodds and Duncan Watts had shown the deeper consequence years earlier. They built an artificial music market with more than fourteen thousand participants and let some of them see what others had downloaded. In the worlds with social influence, the same songs became hits in one world and flops in another. Quality still mattered at the extremes: the best songs rarely failed and the worst rarely succeeded. Everything in between was decided by whoever happened to click first [12]. A crowd that can see itself does not aggregate independent judgments. It amplifies early ones.

The feed

The fourth failure is the one with bodies. In 2018 Soroush Vosoughi, Deb Roy and Sinan Aral examined every verified true and false news story on Twitter from 2006 to 2017, about 126,000 stories shared by three million people, and found that falsehood travelled “farther, faster, deeper, and more broadly than the truth” in every category. A false story was 70 percent more likely to be retweeted than a true one. The bots, which everyone blamed, spread both at the same rate. It was humans who preferred the lie [13].

A feed that ranks by engagement is not unweighted. It is weighted, by the one quantity that correlates with novelty and outrage rather than with truth. The results are now a catalogue. In April 2013 a Reddit community set out to find the Boston Marathon bombers and settled on a missing college student, Sunil Tripathi; his family had to take down the page they were using to search for him, and the site later apologised for “online witch hunts and dangerous speculation” [14]. In December 2016 a man walked into a Washington pizzeria with a rifle because a story he had read online said children were being held in its basement; the building has no basement [15]. In March 2018 the chair of the United Nations fact-finding mission on Myanmar said social media had played a “determining role” in the violence against the Rohingya, and that in Myanmar “social media is Facebook” [16].

None of these were failures of information. The information existed, and in each case it was wrong. They were failures of weighting: nobody in the system was asking which of these voices had earned the right to be believed.

The classroom

The fifth failure is the one Fairgrade started with, and it is the same failure in a smaller room. Peer assessment, letting students grade one another, was the classroom’s version of the Web2 promise, and the evidence on it is instructive. A meta-analysis of forty-eight studies found that peer marks correlate with instructors’ marks at about 0.69: real information, but far from reliable, and worse when students had to judge several dimensions at once [17]. When Coursera opened its courses to hundreds of thousands of students and had no choice but to use peer grading, Chris Piech and colleagues at Stanford analysed 63,199 peer grades and found that the single most important improvement they could make was to model each grader’s bias, because graders were systematically generous or harsh in ways that a plain average could not see [18].

That is the Uber problem with a syllabus attached. A student grading a classmate has every reason to be kind and none to be exact, and the average drifts upward until it stops discriminating. Instructors know this, which is why peer assessment has been confined to participation points and low-stakes exercises for decades. It was never that students cannot judge. It is that counting every judgment once throws away the information about which students judge well.

The experts fail the same way

It would be comforting to think the problem is amateurs, and that a panel of credentialed judges escapes it. It does not. The panel is still an unweighted crowd, weighted by title rather than by record.

Three credit rating agencies, staffed by professionals and relied upon by law, certified the securities at the centre of the 2008 crisis as investment grade. The Financial Crisis Inquiry Commission concluded that “the failures of credit rating agencies were essential cogs in the wheel of financial destruction” [19]. In 2015 the Open Science Collaboration attempted to reproduce one hundred findings published in three of psychology’s leading journals, every one of which had passed peer review. Thirty-six percent replicated [20]. In the same study, researchers who were allowed to bet real money on which findings would hold predicted the outcomes correctly 71 percent of the time, better than the same people answering a survey [21]. The peer-review process had no weight on its judges. The betting market did: conviction backed by money.

What it costs

Put the five failures side by side and the social cost comes into focus. When a rating can be inflated, bought, herded or flooded, the rational reader learns to discount all of them. The honest actor pays most, because the honest signal is the only one that cannot be manufactured on demand. Institutions lose the ability to tell good from bad, and with it the ability to reward good. And the public, sensing that every number has a thumb on it, stops believing numbers.

That is where we are. The 2025 Edelman Trust Barometer found that 70 percent of people across twenty-eight countries believe that government officials, business leaders and journalists deliberately mislead them, an all-time high, and that six in ten carry what the report calls grievance against the institutions around them [22]. One can argue about causes. One cannot argue that a society which has spent two decades aggregating unweighted opinion has come out of it trusting more.

The first essay on this blog ended with a question: once anyone can produce the work, who do we believe actually understands it? Web2’s answer was to ask everyone and count. That answer has been tested at scale for twenty years, and in every place where it mattered, it failed. Not because crowds are foolish. Because the crowd was never the problem. The weight was.

The voice that counts for more

Consider how the world already behaves when it is paying attention. When Berkshire Hathaway discloses that it has bought a stock, the price moves, and it stays moved. Gerald Martin and John Puthenpurackal showed that an investor who simply copied Warren Buffett’s disclosed holdings a month after they were made public would have beaten the S&P 500 by about 10.75 percentage points a year from 1976 to 2006 [23]. On 17 October 2008, with the financial system in freefall, he wrote in the New York Times under the headline “Buy American. I Am.” The market fell for five more months, then rose for a decade [24]. In 2007 he bet a million dollars that an index fund would beat a basket of hedge funds over ten years; it returned 8.5 percent a year against their 0.3 to 6.5 percent [25].

Why does one man’s judgment carry that weight? Not his title; he has none that matters. Not his certainty; he writes a letter every year listing his mistakes. He carries it because for sixty years his judgments have been checked against outcomes in public, and the record is better than the crowd’s. The market weights him. It simply does so informally, after the fact, and for almost nobody else.

The same pattern runs through every field, and so does its opposite. In 1847 Ignaz Semmelweis had the doctors in his Vienna clinic wash their hands in chlorinated lime, and maternal deaths fell from 18 percent to under 2 percent. The consensus of his profession rejected him, some of its members insulted by the suggestion, and he died in an asylum [26]. In 1984 Barry Marshall drank a culture of Helicobacter pylori to prove that bacteria, not stress, caused stomach ulcers, against the settled view of gastroenterology. He received the Nobel Prize twenty-one years later [27]. In January 1986 an engineer named Roger Boisjoly argued on the eve of the Challenger launch that the cold would compromise the booster seals. His managers overruled him, and the Rogers Commission later confirmed that he had been right [28].

In each case a crowd of credentialed judges was wrong and one judge with evidence was right, and in each case the system weighted by position rather than by record. Philip Tetlock turned this into an experiment. His Good Judgment Project recruited thousands of ordinary volunteers to forecast geopolitical events, kept score, and found that the top two percent, identified purely by their track record, beat professional intelligence analysts who had access to classified information by about 30 percent. The aggregation that performed best gave more weight to the forecasters whose records were better [29].

The lesson is not to find the guru. Buffett has been wrong, loudly, and said so. Semmelweis was also abrasive and his contemporaries had reasons, bad ones, for not listening. The lesson is that judgment leaves a record, that the record predicts, and that a system which ignores it in favor of counting heads, or counting titles, will be wrong in exactly the ways the last twenty years have been wrong.

The mathematics was there all along

None of this should have surprised anyone, because the mathematics of when crowds are wise was worked out before the web existed. In 1785 the Marquis de Condorcet proved that if each member of a jury is more likely than not to be right, and they judge independently, a majority is very likely to be right and becomes more so as the jury grows. He also proved the converse: if each member is more likely than not to be wrong, the majority is confidently, overwhelmingly wrong, and more so as it grows [30]. Galton’s farmers were above one half. A feed full of strangers reacting to a headline is not.

In 1982 Shmuel Nitzan and Jacob Paroush went further and asked what the best possible rule is when judges differ in competence. The answer is a weighted vote, and the weights are not a matter of taste. The optimal weight on each judge is the logarithm of the odds that they are right: a judge who is right 90 percent of the time should count for roughly twice a judge who is right 75 percent of the time, and a judge at 50 percent should count for nothing at all [31]. That is the formula for the ox, for the classroom and for the committee. Web2 set every weight to one.

The few places where the web did weight are the places where it worked. In 1998 two graduate students at Stanford proposed ranking web pages not by how many times a keyword appeared but by how many other pages linked to them, with each link weighted by the credibility of the page that gave it. They called it PageRank, and it beat every search engine that counted [32]. Twenty-four years later, the team behind Community Notes on Twitter published their algorithm: a note on a post is shown only if it is rated helpful by people who usually disagree with each other, so that agreement across a divide counts and agreement within a faction does not. Posts with a note shown were reshared a quarter to a third less [33]. The eBay fix was weighting. The Rotten Tomatoes fix was weighting. The replication market was weighting by conviction. Every repair that held was the same repair.

What a crowd needs

A crowd is wise under conditions, and the web removed them. Independence, which herding destroys. Honesty, which cheap identity destroys. A majority better than chance, which an uninformed audience does not supply. And above all a weight on each voice that reflects how often that voice has been right where the answer could be checked.

The last condition is the one that repairs the others. A system that weights by earned credibility makes fraud expensive, because a bought voice has no record. It makes herding detectable, because a voice that only ever agrees with the tally has no information of its own. It makes bias visible, because a judge who is consistently generous to friends, or consistently harsh, leaves a pattern. And it makes the Semmelweis case come out differently, because the doctor whose patients stopped dying has a record that a title cannot outweigh.

The crowd was never the problem. Credibility is the weight, and it has to be earned against outcomes, estimated rather than asserted, and re-earned as the record changes. The next question is how to estimate it, for every judge at once, at the same time as the thing they are judging. That is the subject of the next essay.

References

1. Galton, F., “Vox Populi,” Nature, 7 March 1907; see Wallis, K. F., “Revisiting Francis Galton’s Forecasting Competition,” Statistical Science, 2014. https://arxiv.org/pdf/1410.3989
2. Surowiecki, J., The Wisdom of Crowds, Doubleday, 2004.
3. Filippas, A., Horton, J. J., Golden, J., “Reputation Inflation,” Marketing Science, 2022; NBER Working Paper 25857. https://ideas.repec.org/p/nbr/nberwo/25857.html
4. Nosko, C., Tadelis, S., “The Limits of Reputation in Platform Markets: An Empirical Analysis and Field Experiment,” NBER Working Paper 20830, 2015. https://www.nber.org/papers/w20830.pdf
5. Zervas, G., Proserpio, D., Byers, J. W., “A First Look at Online Reputation on Airbnb, Where Every Stay is Above Average,” Marketing Letters, 2021. https://www.doi.org/10.1007/s11002-020-09546-4
6. Luca, M., Zervas, G., “Fake It Till You Make It: Reputation, Competition, and Yelp Review Fraud,” Management Science, 62(12), 2016. https://dash.harvard.edu/entities/publication/73120379-0abd-6bd4-e053-0100007fdf3b
7. PYMNTS, “Amazon Uses Technology and Legal Action to Combat Fake Reviews,” 2024. https://www.pymnts.com/cybersecurity/2024/amazon-uses-technology-and-legal-action-to-combat-fake-reviews/
8. Wilson Sonsini, “FTC Issues Final Rule Banning Fake and Misleading Consumer Reviews and Testimonials,” August 2024. https://wsgr.com/en/insights/ftc-issues-final-rule-banning-fake-and-misleading-consumer-reviews-and-testimonials.html
9. Bay, M., “Weaponizing the Haters: The Last Jedi and the strategic politicization of pop culture through social media manipulation,” 2018; reported in Quartz, October 2018. https://qz.com/1410761/the-last-jedi-haters-were-manipulated-by-bots-and-trolls-study
10. TechCrunch, “Rotten Tomatoes’ new verified audience score,” May 2019. https://techcrunch.com/2019/05/23/rotten-tomatoes-verified-audience-score
11. Muchnik, L., Aral, S., Taylor, S. J., “Social Influence Bias: A Randomized Experiment,” Science, 341(6146), 2013. https://seanjtaylor.com/2013/08/08/science-paper-on-social-influence-bias.html
12. Salganik, M. J., Dodds, P. S., Watts, D. J., “Experimental Study of Inequality and Unpredictability in an Artificial Cultural Market,” Science, 311(5762), 2006. https://www.cs.toronto.edu/~bor/303s19/docs/salganik_dodds_watts.pdf
13. Vosoughi, S., Roy, D., Aral, S., “The spread of true and false news online,” Science, 359(6380), 2018. https://www.media.mit.edu/publications/the-spread-of-true-and-false-news-online/
14. Salon, “Reddit apologizes for falsely accusing student of bombings,” April 2013. https://salon.com/2013/04/23/reddit_apologizes_for_falsely_accusing_student_of_bombings_partner
15. CBS News, “Charges filed against suspected ‘Pizzagate’ Comet Ping Pong gunman,” December 2016. https://cbsnews.com/news/edgar-maddison-welch-charges-filed-against-suspected-pizzagate-comet-ping-pong-gunman
16. Al Jazeera, “UN: Facebook had a ‘role’ in Rohingya genocide,” March 2018. https://www.aljazeera.com/news/2018/3/13/un-facebook-had-a-role-in-rohingya-genocide
17. Falchikov, N., Goldfinch, J., “Student Peer Assessment in Higher Education: A Meta-Analysis Comparing Peer and Teacher Marks,” Review of Educational Research, 70(3), 2000. https://doi.org/10.3102/00346543070003287
18. Piech, C., Huang, J., Chen, Z., Do, C., Ng, A., Koller, D., “Tuned Models of Peer Assessment in MOOCs,” Educational Data Mining, 2013. https://arxiv.org/pdf/1307.2579
19. Financial Crisis Inquiry Commission, Final Report, January 2011, Conclusions. https://www.govinfo.gov/content/pkg/GPO-FCIC/pdf/GPO-FCIC.pdf
20. Open Science Collaboration, “Estimating the reproducibility of psychological science,” Science, 349(6251), 2015. https://doi.org/10.1126/science.aac4716
21. Dreber, A., et al., “Using prediction markets to estimate the reproducibility of scientific research,” PNAS, 112(50), 2015. https://doi.org/10.1073/pnas.1516179112
22. Axios, “Trust in CEOs erodes, new report shows,” on the 2025 Edelman Trust Barometer, January 2025. https://www.axios.com/2025/01/19/edelman-2025-trust-barometer
23. Martin, G. S., Puthenpurackal, J., “Imitation is the Sincerest Form of Flattery: Warren Buffett and Berkshire Hathaway,” 2008. https://www.valuewalk.com/imitation-sincerest-form-flattery-warren-buffett-berkshire-hathaway/
24. Buffett, W. E., “Buy American. I Am.,” The New York Times, 17 October 2008. https://www.nytimes.com/2008/10/17/opinion/17buffett.html
25. InvestorDaily, “Buffett wins 10-year bet with hedge funds,” 2018. https://www.investordaily.com.au/buffett-wins-10-year-bet-with-hedge-funds
26. “Ignaz Semmelweis,” Wikipedia, with sources on the 1847 clinic mortality figures. https://en.wikipedia.org/wiki/Ignaz_Semmelweis
27. The Nobel Prize in Physiology or Medicine 2005, Barry J. Marshall and J. Robin Warren. https://www.nobelprize.org/prizes/medicine/2005/
28. Chapman University, “Whistle blower in ‘Challenger’ disaster donates personal papers,” on Roger Boisjoly. https://news.chapman.edu/?p=2381
29. Tetlock, P. E., Gardner, D., Superforecasting: The Art and Science of Prediction, Crown, 2015; and Popular Science, “How a US intelligence program created a team of ‘Superforecasters’.” https://www.popsci.com/technology/superforecasters-future-predictions/
30. “Condorcet jury theorem,” Encyclopedia of Mathematics. https://encyclopediaofmath.org/wiki/Condorcet_jury_theorem
31. Nitzan, S., Paroush, J., “Optimal Decision Rules in Uncertain Dichotomous Choice Situations,” International Economic Review, 23(2), 1982; the result is restated in Berend, D., Kontorovich, A., “Consistency of weighted majority votes,” 2013. https://arxiv.org/pdf/1312.0451
32. Brin, S., Page, L., “The Anatomy of a Large-Scale Hypertextual Web Search Engine,” 1998. http://infolab.stanford.edu/~backrub/google.html
33. Wojcik, S., et al., “Birdwatch: Crowd Wisdom and Bridging Algorithms can Inform Understanding and Reduce the Spread of Misinformation,” 2022. https://arxiv.org/abs/2210.15723