Whom do you ask?

Ask a group of friends for a restaurant. Six answers come back, and you do not treat them equally. You remember that one friend’s last three recommendations were good. You know another eats out constantly and has opinions worth hearing, and a third is enthusiastic about everything, so his praise carries no information. You notice that a fourth is recommending his cousin’s place. By the time you choose, you have done three things without naming them. You asked whether each person had been right before. You asked whether they knew the subject. And you asked whether they had a reason to say what they said.

That is the whole of it. Judgment, mastery and bias are the three questions people already ask when they decide whom to believe, and they ask them so naturally that the questions disappear into the answer. The second essay on this blog ended by saying that the crowd was never the problem, the weight was, and that the weight has to be earned. This essay is about what earning it means: how those three questions can be asked of a thousand judges at once, on evidence, with nobody trusted in advance, and why the way the work is handed out matters as much as the way it is scored.

Three questions everyone already asks

Consider the first question, the record, in three settings where it can be checked.

On the quiz programme Who Wants to Be a Millionaire, a contestant who was stuck could telephone the cleverest person they knew or poll the studio audience. The friend, an expert chosen in advance, was right about 65 percent of the time. The audience, a room of strangers who had each paid for a ticket, was right 91 percent of the time [1]. The audience was not wiser than the friend on any one question. It was better as a record, because across hundreds of questions a crowd in which most people are slightly better than chance, answering independently, outperforms one person who is often better than chance. That is the arithmetic of the ox, and of Condorcet’s jury, and it only works because each member of that audience was at least a little informed and nobody could see the others’ answers [2].

Warren Buffett’s record is the opposite case: one person, checked in public for sixty years. An investor who merely copied Berkshire Hathaway’s disclosed holdings a month after they became public would have beaten the market by about 10.75 percentage points a year for three decades [3]. The market weights his judgment because his judgment has a verifiable history, not because of his title, which is “chairman,” and not because of his certainty, which he discounts in every annual letter by listing what he got wrong.

Michael Burry’s record is the hardest case, and the most instructive. In 2005 he concluded that the American mortgage market would collapse and bought insurance against it. His investors, who had trusted his stock-picking for years, did not believe him; some demanded their money back, and he had to stop them withdrawing it, which brought lawsuits. By June 2008 his fund had returned 489 percent since inception, net of fees, and the bet had made his investors $725 million [4]. For two years the record said one thing and the crowd said another, and nobody could know which was right until the outcome arrived. A record is only legible after the fact. That is its nature, and it is the first reason a record cannot stand alone.

Is a track record safe?

The question deserves a direct answer, because the whole method rests on it. A track record is a store of having been right. Four things can make that store misleading.

It can be luck. In 1997 Mark Carhart showed that mutual funds which beat the market in one year tended, modestly, to do so again the next [5]. When James Choi and Kevin Zhao repeated the analysis for 1994 to 2018, the effect was gone: in the modern period “a fund’s performance is completely unpredictive of its returns in the future” [6]. A short record of being right is consistent with skill and with chance, and the two look identical from the outside.

It can be farmed. Goodhart’s law, in Marilyn Strathern’s formulation, is that “when a measure becomes a target, it ceases to be a good measure” [7]. A judge who learns that agreeing with the consensus builds credibility will agree with the consensus, and the record will reward conformity rather than judgment.

It can go stale. Being right about one subject in one decade says less about another subject in the next, and a system that does not let credibility decay will trust the past at the expense of the present.

And it entrenches. Robert Merton named the Matthew effect in 1968: the accruing of greater recognition to those who already have it, and the withholding of it from those who have not yet made their mark [8]. Once a reputation exists, people defer to it, and the deference looks like confirmation. The scientific literature on peer review is a catalogue of this. When a computer science conference randomly split its reviewers between those who could see authors’ names and those who could not, the ones who could see names were 1.63 times as likely to recommend accepting a paper by a famous author and about twice as likely for one from a top company [9]. Twenty years earlier, two Swedish researchers had shown that a woman applying for a postdoctoral fellowship needed roughly two and a half times the published output of a man to receive the same competence score from the reviewers [10]. Ignaz Semmelweis, whose patients had stopped dying, had no reputation to be deferred to, and the profession deferred to the men who did.

So a track record is safe under five conditions, and the method exists to supply them. The record must be measured against outcomes that could be checked, not against applause. It must be held with uncertainty, so that a short record counts for little and a long consistent one counts for much. It must be specific to a domain and re-earned as the record changes. It must be paired with the other two questions, mastery for the newcomer who has no record yet and bias for the judge who is building one strategically. And the work the record is built from must be handed out in a way nobody can steer, which is the subject of the section before last.

Mastery, and what it is for

The second question, whether the judge knows the subject, matters most at the beginning. A new judge has no record. Demonstrated expertise is the only evidence available, so it carries weight early, and it is discounted when the record begins to contradict it. That is how people treat a new doctor: the qualification earns a first hearing, and what follows depends on what happens to the patients.

Where institutions go wrong is in treating position as mastery. Universities appoint teaching assistants to grade because instructors cannot, and then treat their marks as expert marks. The evidence says otherwise. At King’s College London, more than 95 percent of graduate teaching assistants surveyed worked beyond their contracted hours and more than a third worked ten or more unpaid hours to mark student work [11]. An engineering education study found that a majority of teaching assistants met the criteria for burnout, with grading and feedback among the tasks most associated with it [12]. And when researchers had 28 graders, with the profile of typical teaching assistants, mark the same forty programming assignments in groups of four, the groups could not agree: an inter-rater reliability of about 0.2 for correctness and under 0.1 for code quality, where 0.8 is the threshold for acceptable agreement. When the same assignment was shown to a grader twice, only one of 22 graders gave it the same marks both times [13].

Fatigue does the rest. An analysis of tens of millions of grading records from a large public university’s learning management system found that assignments graded later in a session received lower scores, more negative and less polite comments, and more post-grade complaints, and that because the system sorted submissions alphabetically, the disadvantage fell systematically on students whose surnames came late in the alphabet [14]. None of these graders were careless people. They were tired people with a title, judged as if the title were evidence. Mastery is evidence of what a person has shown they know. It is not a job description.

Bias leaves a pattern

The third question is whether the judge has a reason to say what they said, and it is the one people are worst at answering about themselves. The friend recommending his cousin’s restaurant may believe every word. The bias is visible only from outside, as a pattern.

The patterns are well documented. Students grading classmates tend to score friends higher and people they dislike lower; in one study the individual effect was clear even though it washed out in the aggregate [15]. Reviewers favour famous names and prestigious institutions without intending to, which is why a three-year trial at an ecology journal found that hiding authors’ identities reduced the advantage that authors from wealthy, English-speaking countries had enjoyed [16]. Graders under load score generously to be finished, or harshly after a run of good work, or in whatever direction the previous submission pushed them. And every evaluator is kinder to work that resembles their own.

Each of these leaves a signature. A judge who is consistently higher than the weighted verdict for one group of people and not another, or consistently high across the board, or consistently near the running tally and never away from it, is showing a pattern that a single score cannot show but a record across many items can. That is why bias is estimated rather than assumed, and why it is estimated at the same time as the other two questions. Some of it can be removed at the source: anonymity, which the ecology journal’s trial demonstrated, takes the name off the work before the judgment is made. The rest has to be found in the record.

Estimated together, not assumed

Here is the part that distinguishes the method from every reputation system the web built, and it can be said without any mathematics.

In 1982 two economists proved what the best possible voting rule is when judges differ in competence: a weighted vote, with each judge’s weight set by how often that judge is right [17]. The theorem assumes you already know how often each judge is right. In life you never do. The students who grade each other, the referees who review a paper, the analysts who rate a bond, the strangers who rate a restaurant: nobody has measured any of them in advance, and the only evidence of their credibility is the very judgments you are trying to weigh.

The method’s answer is that the two unknowns are estimated together. The judgments imply which judges are credible; the credibility re-weighs the judgments; the two converge on a verdict and on a set of weights that are consistent with each other and with whatever ground truth exists. Nobody is trusted in advance. Credibility on all three questions is inferred from the record, and it comes with an honest statement of how much evidence it rests on. What comes out is not a score. It is a score with a stated confidence, which is a different and more useful thing, because it tells the reader how much to believe it. The mechanics of the convergence are set out in the Marcianos’ paper and belong there; what matters here is what it produces and what it refuses to assume.

Web2 never did this. It set every weight to one and called the result a rating. The feed set every weight to engagement and called the result relevance. The expert panel set every weight by title and called the result peer review. Each was an assumption about credibility standing in for a measurement of it.

Dissemination, or who judges what

There is a fourth pillar, and it is the one nobody talks about because it happens before any judgment is made. Who evaluates what, and in what order, is not administration. It is part of the measurement.

Consider what can go wrong if it is left to chance or to choice. If judges pick their own items, friends grade friends and rivals grade rivals, and the bias section above becomes the whole story. If every judge sees every item, the burden is intolerable and the fatigue effects from the Michigan records take over. If each judge sees only a few items and no two judges share any, the records cannot be compared and credibility cannot be estimated at all. If a judge can see the running tally, the herding that the second essay described sets in. And if the assignment can be predicted, it can be farmed.

So the distribution has rules, and the rules are constraints that have to hold at once. It must be sparse: each judge sees a small share of the work, and each piece of work is seen by a small number of judges, because anything else does not scale and degrades the judges it exhausts. It must overlap: the judges’ assignments have to intersect enough that any two can be compared through the items they share, which is a problem agricultural statisticians solved in the 1930s, when they needed to compare many varieties with few plots and worked out how to arrange incomplete blocks so that every comparison could still be made [18]. It must be anonymous and unsteerable: nobody chooses their own items, nobody knows whose work they hold, nobody sees what others gave. It must balance: workload spread evenly, conflicts of interest excluded, and perspectives mixed so that agreement means something, which is the principle Community Notes rediscovered when it required a note to be rated helpful by people who usually disagree before it would show one [19]. And it must randomise order, which is the one remedy the Michigan study recommended and the simplest.

A distribution that meets those constraints is what makes the three questions answerable. It is what makes a record hard to farm, a bias visible, a burden fair, and a credibility estimate honest. Fairgrade calls it intelligent assignment, and it is the quiet half of the method.

What this answers

The first essay asked who to believe once anyone can produce the work. The second asked why counting everyone failed. The three questions and the fourth pillar are the answer to both, and it is worth walking back through the places where the problem was found.

In the classroom, the problem was that the artifact no longer proves understanding and the people appointed to judge it are exhausted and inconsistent. The method does not appoint judges. It lets everyone judge, distributes the judging so that no one is overloaded and no one can steer it, and measures the judging. The teaching assistant problem dissolves not because the assistants are replaced but because a title stops standing in for mastery, and the instructor’s own judgment becomes the ground truth the method is checked against rather than the bottleneck it was.

At work, the problem was workslop: output that reads as finished, a manager who cannot tell from the output who understood the task, and colleagues who have stopped trusting each other. A measured record of judgment is exactly the signal that went quiet. The person who catches the invented citation, the almost-right code, the hollow analysis, leaves a record of catching it, and that record is what the manager was trying to infer from fluency and cannot.

In the wider economy, the problem was a premium on verification with no way to price it. A record of judgment, held with a stated confidence, is a priceable thing. It is the credential the market has been reaching for since the artifact stopped meaning anything.

On the web, the problem was fraud, inflation, herding and the flood. Weight by earned credibility and a bought voice has no record; inflation cannot drift a weighted verdict that discounts the judge who is generous to everyone; herding is a pattern the record shows; and a flood of generated voices carries no credibility to be weighed. The platforms that repaired themselves, by verified tickets, by bridging algorithms, by markets with money at stake, each reinvented one of the three questions. The method asks all three, together.

And for the laboratories that asked in September for independent evaluators with a record and real access: this is how such evaluators would be found at scale. Not by appointment, which is the Matthew effect with a budget, but by measurement, in places where their judgment can be checked.

The anchor

A verdict from a crowd, weighted by the demonstrated credibility of the people in it, with a stated confidence, produced by a distribution of work that nobody could steer. That is what the three questions and the fourth pillar yield, and it is the one thing a generator cannot supply for itself, because a generator has no record, no mastery it can demonstrate against an outcome, and no way to show it is free of the tilt of whatever it was trained on. The anchor has to come from people, and it has to be earned.

The restaurant question is easy because you know your friends. The method is what it takes to answer the same question when you do not: when the judges are strangers, the work is plentiful, and the only evidence of anyone’s credibility is the judgments themselves. Grading was the first place to prove it, because grading has an answer key. The next essay is about what it means, for a student and for a university, to grade the judgment.

References

1. The accuracy figures for the two lifelines, 91 percent for the audience and 65 percent for the phone-a-friend, are reported in Surowiecki, J., The Wisdom of Crowds, Doubleday, 2004, and restated in “The Crowd Can Make You a Millionaire,” PHCP Pros. https://phcppros.com/articles/3995-the-crowd-can-make-you-a-millionaire
2. “Condorcet’s jury theorem,” Wikipedia, with the 1785 source. https://en.wikipedia.org/wiki/Condorcet%27s_jury_theorem
3. Martin, G. S., Puthenpurackal, J., “Imitation is the Sincerest Form of Flattery: Warren Buffett and Berkshire Hathaway,” 2008. https://www.valuewalk.com/imitation-sincerest-form-flattery-warren-buffett-berkshire-hathaway/
4. “Scion Asset Management,” Wikipedia, on Scion Capital’s 2000 to 2008 returns, the credit default swap position and the investor dispute. https://en.wikipedia.org/wiki/Scion_Asset_Management
5. Carhart, M. M., “On Persistence in Mutual Fund Performance,” The Journal of Finance, 52(1), 1997. https://doi.org/10.1111/j.1540-6261.1997.tb03808.x
6. Choi, J. J., Zhao, K., “Did Mutual Fund Return Persistence Persist?,” NBER Working Paper 26707, 2020; summarised by Yale Insights. https://insights.som.yale.edu/insights/does-mutual-fund-s-past-performance-predict-its-future
7. “Goodhart’s law,” Wikipedia, with Strathern’s 1997 formulation. https://en.wikipedia.org/wiki/Goodhart%27s_law
8. Merton, R. K., “The Matthew Effect in Science,” Science, 159(3810), 1968. https://pubmed.ncbi.nlm.nih.gov/5634379
9. Tomkins, A., Zhang, M., Heavlin, W. D., “Reviewer bias in single- versus double-blind peer review,” PNAS, 114(48), 2017. https://pmc.ncbi.nlm.nih.gov/articles/PMC5715744/
10. Wennerås, C., Wold, A., “Nepotism and sexism in peer-review,” Nature, 387, 1997. https://www.cs.utexas.edu/~mckinley/notes/ww-nature-1997.pdf
11. Times Higher Education, “King’s College London graduate teachers speak out,” May 2015. https://www.timeshighereducation.com/news/kings-college-london-graduate-teachers-speak-out/2020248.article
12. Berta, M., Pembridge, J. J., “Exploring Burnout among Graduate Teaching Assistants,” ASEE Annual Conference, 2019. https://portfolio.erau.edu/en/publications/exploring-burnout-among-graduate-teaching-assistants/
13. Messer, M., Brown, N. C. C., Kölling, M., Shi, M., “How Consistent Are Humans When Grading Programming Assignments?,” 2024. https://arxiv.org/abs/2409.12967
14. Pei, J., Wang, Z., Li, J., “30 Million Canvas Grading Records Reveal Widespread Sequential Bias and System-Induced Surname Initial Disparity,” EAAMO, 2023; a later version analyses 70 million records. https://conference2023.eaamo.org/papers/EAAMO23_paper_118.pdf
15. Sousa, F., Alves, T., Gama, S., Jorge, J., Gonçalves, D., “Impact of Social Relationships on Peer Assessment in E-Learning,” 2024. https://arxiv.org/abs/2407.01077
16. Fox, C. W., “Double-blind peer review affects reviewer ratings and editor decisions at an ecology journal,” Functional Ecology, 2023. https://doi.org/10.1111/1365-2435.14259
17. Nitzan, S., Paroush, J., “Optimal Decision Rules in Uncertain Dichotomous Choice Situations,” International Economic Review, 23(2), 1982; restated in Berend and Kontorovich, 2013. https://arxiv.org/pdf/1312.0451
18. Fisher, R. A., The Design of Experiments, 1935, and the incomplete block designs that followed from it. https://en.wikipedia.org/wiki/The_Design_of_Experiments
19. Wojcik, S., et al., “Birdwatch: Crowd Wisdom and Bridging Algorithms can Inform Understanding and Reduce the Spread of Misinformation,” 2022. https://arxiv.org/abs/2210.15723