Collective intelligence you can trust

Fairgrade exists to make collective intelligence trustworthy. That sentence carries three claims, and this essay is about all three. The first is that the cost of producing an answer has fallen to nothing, and the cost of knowing which answers to trust is about to define the economy. The second is that the obvious way to find out whom to trust, ask everyone and count, has been tried at the scale of the whole internet and has failed wherever it mattered. The third is that there is a better way, older than the web and grounded in probability rather than in data alone: weigh every judgment by the demonstrated credibility of the person who made it, estimate that credibility from the evidence rather than from a title, and let the two converge. We proved it first in a place where the answer can be checked, which is the university classroom. The essays that follow this one develop each claim in turn. This one is the map.

What changed

In March 2023 a New York lawyer filed a brief citing six court decisions that did not exist. When the court asked for copies, he asked the tool that had written the brief whether the cases were real, and it assured him that they were. The judge fined him and ordered him to write to every judge whose name had been attached to an opinion never written [1]. The lesson was not that he had used a machine. It was that he had asked the machine to check itself, and a machine that generates will always say yes, because it has no way to know.

Three years later the people building these machines said the same thing in public. In September 2026 Dario Amodei wrote that the industry “must slow the pace at which we improve the capabilities of AI models” and committed Anthropic to giving independent evaluators permanent, employee-level access to its systems; within hours Sam Altman agreed that “we need to pace the frontier” [2]. The makers of the machines conceded that generation had outrun verification and that the remedy was human judgment. What they did not say is how to know which humans to trust.

That question has already arrived everywhere the output is easy to produce. Among British undergraduates, 95 percent now use generative AI in some form and 94 percent use it for assessed work [3]. Three in four knowledge workers use it on the job [4], and the people most exposed to its output trust it least: developers adopted the tools almost universally in 2025 while their trust in the accuracy of what the tools produce fell to 29 percent [5]. The essay, the brief, the report, the code and the application all still arrive looking finished. None of them any longer proves that the person who submitted it understood it. Every institution that inferred understanding from the artifact, which is to say every transcript, credential, peer review and hiring process we have, is now measuring something that has gone quiet.

Why the obvious answer failed

The internet’s first answer to “whom do you trust” was to ask everyone and count. It had a respectable pedigree. In 1906 Francis Galton collected 787 guesses of an ox’s weight at a county fair and found the average within a pound of the truth [6]. In 1785 the Marquis de Condorcet had proved why: when each member of a jury is more likely than not to be right, and they judge independently, the majority is very likely to be right. He also proved the converse, which the web forgot. When each member is more likely to be wrong, the majority is confidently, overwhelmingly wrong, and more so as it grows [7].

Galton’s crowd had sealed tickets, a price for guessing, farmers who knew cattle, and a true weight at the end. The web kept the arithmetic and removed the conditions. Ratings drifted to the ceiling, where they say nothing. Reviews were bought at industrial scale; Amazon alone blocked more than 250 million suspected fakes in 2023 [8]. Visible tallies turned independent judgments into herds. And a feed that ranks by engagement rewards what travels, which is not what is true: across eleven years of Twitter, a false story was 70 percent more likely to be retweeted than a true one, and it was humans, not bots, who preferred the lie [9]. The credentialed version failed the same way. Three rating agencies certified the securities at the centre of the 2008 crisis, and a hundred peer-reviewed psychology findings, re-tested in 2015, replicated 36 percent of the time [10].

None of this was a failure of crowds. It was a failure of weight. Every voice was counted once, or counted by its volume, or counted by its title, and in 1982 two economists had already shown what the best rule actually is: a weighted vote, with each judge’s weight set by how often that judge is right [11]. The web set every weight to one. The second essay in this series is about what that cost.

What the method asks

Ask a group of friends for a restaurant and you do not treat their answers equally. You ask, without naming the questions, whether each person has been right before, whether they know the subject, and whether they have a reason to say what they said. Those are the three questions. The Marcianos call them judgment, mastery and bias, and their method asks them of every judge, on evidence, with nobody trusted in advance.

Judgment is the track record: people whose past assessments proved good are trusted more, with an adjustment for those who run consistently harsh or consistently lenient. Mastery is demonstrated expertise: it carries weight early, when there is little else to go on, and is discounted when the record contradicts it. Bias is evidence of a tilt: favouring friends, favouring work that resembles one’s own, scoring everything high to be done quickly, each of which leaves a pattern the method looks for.

The part that distinguishes this from every reputation system the web built is that credibility is not assumed and then applied. It is estimated at the same time as the quality of the work being judged. The judgments imply which judges are credible, the credibility re-weighs the judgments, and the two converge. Disagreement among judges is not noise to be averaged away; it is information about the judges. What comes out is a verdict with a stated confidence, which tells the reader how much to believe it. The method is grounded in probability theory rather than in data alone, which means it works from a single class rather than from a million examples, and it can explain every weight it assigns, which means it can be checked, corrected and improved.

There is a fourth element, quieter than the three questions and as important: who judges what. Work has to be distributed sparsely, so no judge is overloaded; with enough overlap that judges can be compared; anonymously, so friends do not grade friends; and in a way nobody can steer or predict, so that a record cannot be farmed. The distribution is part of the measurement. The third essay in this series is about all four.

Why higher education first

Fairgrade is built for any decision that rests on many human judgments of uneven reliability. We chose to prove it in the university, with the grade itself, for five reasons.

It is where the method was born. Sonia and Anthony Marciano are professors at NYU Stern, and the method was built in their own courses to solve their own problem, then refined there for over a decade. By our own count it has been used in more than sixty courses with more than five thousand students, with the instructor’s own grade as the check each time; the Science page sets out what that record showed. In 2020 Fairgrade won the $100,000 Rennert Prize at NYU’s Entrepreneurs Challenge [12]. “My husband and I have been full-time professors in top-rated business schools since 1996,” Sonia Marciano has written. “The problem we solve goes to the heart of the mission of education: how the learning of students is assessed.”

It has a ground truth. Most problems in collective intelligence have no answer key; you cannot easily check whether a hiring panel was right. Grading is different. The instructor’s judgment exists, and a method that claims to find the signal in a crowd should be made to prove it where the proof is unavoidable.

It is where the failure is felt, and where it is most expensive. American colleges and universities spend about $702 billion a year [14], and the single most powerful thing they could give a student with that money is the one thing that does not scale. In 1984 Benjamin Bloom showed that a student tutored one to one performed two standard deviations better than a student in a classroom, and he named the search for group methods that could match it the “2 sigma problem” [15]. Feedback is among the most powerful influences on learning that educational research has measured [16], and in a class of two hundred it is rarely given, because giving it well takes time that grows with every student. The grade that arrives weeks later, as a number without a reason, is what the scaling problem looks like from a student’s chair.

The question has changed. Once anyone can produce the work, the artifact stops proving understanding, and detecting the machine does not help: the detectors are unreliable and fall hardest on honest students writing in a second language [17]. The honest answer is to let students use the best tools available, so that access to the tool no longer decides who can produce the work, and to add one step: every student also evaluates the work of peers, under the instructor’s rubric, and the quality of that evaluation is measured. The artifact can be generated. Judgment cannot be outsourced without leaving a trace, because credibility is earned against the consensus over time. This is not only assessment. Students who review classmates’ work go on to write better work themselves, more so than students who only receive feedback, a finding replicated in writing classrooms for fifteen years [18]. To judge well is to learn the material a second time, in the harder direction.

And it pays for the rest. Education matters on its own terms; fixing how it assesses is worth doing for its own sake. It is also how the method earns the right to be trusted elsewhere. Every course that runs on it is another place where its verdicts were checked against a known answer, and that record is the credibility we will carry into every field that follows.

Where it goes

The same mathematics applies wherever observers of uneven reliability judge the same hard question, and we state what follows as research directions, not products. Referees, judges and review committees are crowds of uneven reliability, and the evidence that they favour famous names and familiar institutions is thorough [19]. Forecasting and risk are places where the track record should outweigh the title, and where a method that weights by past accuracy is a method for surfacing the analyst who saw it coming. Talent review is bias’s best-known home, and a method built to detect and discount bias is its natural repair. Public discourse is, at bottom, an unweighted crowd; making record, expertise and interest visible lets the reader decide whom to believe, which is the direction the one platform repair that has worked, Community Notes, already took when it required a note to be rated helpful by people who usually disagree [20].

Even the laboratories that asked in September for independent evaluators were describing this. They want humans with a record and real access, and they have no way to find them except by appointment, which is reputation with a budget. Measurement is the other way, and it works only where the judgment can be checked. Each of those fields will be earned the way the first one was.

What this blog is

These essays carry the argument in order. The first, after this one, is about what changed: generation has outrun verification, in the classroom, the office, the journal and the hiring funnel. The second is about why counting everyone failed, and why the crowd was never the problem. The third is about the three questions and the fourth element, and what it takes to answer them for a thousand judges at once. The ones that follow are about grading the judgment, about the economy that forms when verification carries the premium that generation has lost, about where the mathematics applies next, and about why infrastructure for trust should be held as a public good rather than owned by one company.

Two things about how they are written. They use names without titles, because the whole argument is that credibility is earned and not conferred. And they are written with the help of the same tools they describe, which is as it should be: the tools are good at structuring an argument, and useless at deciding whether it is true. That part is ours, and it is checked the same way everything here is checked, against a reference you can open.

The place to start was the place where the answer could be checked. That was the classroom. The rest follows from there.

References

1. Mata v. Avianca, Inc., S.D.N.Y., June 2023. https://en.wikipedia.org/wiki/Mata_v._Avianca,_Inc.
2. MediaNama, “Anthropic wants independent evaluators inside frontier AI labs; OpenAI agrees,” September 2026. https://www.medianama.com/2026/09/223-anthropic-evaluators-frontier-ai-labs/
3. Higher Education Policy Institute, “Student Generative AI Survey 2026,” Report 199, March 2026; summarised by Advance HE. https://advance-he.org/knowledge-hub/higher-education-policy-institute-hepi-student-generative-ai/
4. Microsoft and LinkedIn, “2024 Work Trend Index,” May 2024. https://blogs.microsoft.com/blog/2024/05/08/microsoft-and-linkedin-release-the-2024-work-trend-index-on-the-state-of-ai-at-work/
5. Stack Overflow, 2025 Developer Survey, reported by InfoWorld, July 2025. https://infoworld.com/article/4031673/ai-use-among-software-developers-grows-but-trust-remains-an-issue-stack-overflow-survey.html
6. Galton, F., “Vox Populi,” Nature, 1907; see Wallis, K. F., “Revisiting Francis Galton’s Forecasting Competition,” 2014. https://arxiv.org/pdf/1410.3989
7. “Condorcet’s jury theorem,” Wikipedia. https://en.wikipedia.org/wiki/Condorcet%27s_jury_theorem
8. PYMNTS, “Amazon Uses Technology and Legal Action to Combat Fake Reviews,” 2024. https://www.pymnts.com/cybersecurity/2024/amazon-uses-technology-and-legal-action-to-combat-fake-reviews/
9. Vosoughi, S., Roy, D., Aral, S., “The spread of true and false news online,” Science, 2018. https://www.media.mit.edu/publications/the-spread-of-true-and-false-news-online/
10. Open Science Collaboration, “Estimating the reproducibility of psychological science,” Science, 2015. https://doi.org/10.1126/science.aac4716
11. Nitzan, S., Paroush, J., “Optimal Decision Rules in Uncertain Dichotomous Choice Situations,” International Economic Review, 1982; restated in Berend and Kontorovich, 2013. https://arxiv.org/pdf/1312.0451
12. NYU Stern, “Winners Announced in NYU Stern $300K Entrepreneurs Challenge,” 2020. https://www.stern.nyu.edu/experience-stern/news/winners-announced-in-nyu-stern-300k-entrepreneurs-challenge-2020
13. National Center for Education Statistics, “Fast Facts: Expenditures,” total expenditures of degree-granting postsecondary institutions, 2020–21. https://nces.ed.gov/fastfacts/display.asp?id=75
14. Bloom, B. S., “The 2 Sigma Problem: The Search for Methods of Group Instruction as Effective as One-to-One Tutoring,” Educational Researcher, 13(6), 1984. https://doi.org/10.3102/0013189X013006004
15. Hattie, J., Timperley, H., “The Power of Feedback,” Review of Educational Research, 77(1), 2007. https://doi.org/10.3102/003465430298487
16. Liang, W., et al., “GPT detectors are biased against non-native English writers,” Patterns, 2023; reported in The Markup. https://themarkup.org/machine-learning/2023/08/14/ai-detection-tools-falsely-accuse-international-students-of-cheating
17. Lundstrom, K., Baker, W., “To give is better than to receive: The benefits of peer review to the reviewer’s own writing,” Journal of Second Language Writing, 18(1), 2009, https://doi.org/10.1016/j.jslw.2008.06.002; and Cho, K., MacArthur, C., “Learning by reviewing,” Journal of Educational Psychology, 103(1), 2011. https://doi.org/10.1037/a0021950
18. Tomkins, A., Zhang, M., Heavlin, W. D., “Reviewer bias in single- versus double-blind peer review,” PNAS, 2017. https://pmc.ncbi.nlm.nih.gov/articles/PMC5715744/
19. Wojcik, S., et al., “Birdwatch: Crowd Wisdom and Bridging Algorithms can Inform Understanding and Reduce the Spread of Misinformation,” 2022. https://arxiv.org/abs/2210.15723