When several sources publish an expectation for the same player and they do not agree, there is something to learn from the disagreement itself, independent of who is closer to right.

We assumed for a long time that we knew what it meant. We were wrong in a way that took a redesign to fix, and the redesign is the only reason we now count sources at all.

We Treated Spread as Uncertainty and It Was Often Coverage

The intuitive reading is that when sources disagree, the underlying situation is uncertain. Wide spread means nobody is sure; narrow spread means the outcome is well understood.

That is sometimes true. It is also frequently an artefact of who happens to be publishing. A player covered by many sources tends to show a narrow spread because the sources are watching each other. A player covered by two shows whatever those two happen to think, and the spread between them says more about the pair than about the player.

So we had built a feature that we believed measured uncertainty and that was substantially measuring how much attention a player received. Those two things are correlated, which is what let the mistake survive — attention and predictability genuinely do travel together — but they are not the same, and the cases where they came apart were exactly the cases we cared about.

The clearest symptom: lightly-covered players in smaller competitions were being flagged as highly uncertain, systematically, regardless of how consistent their record was. Our corpus said one thing and our spread feature said another, and the spread feature was winning arguments it should have lost.

Counting the Sources Before Measuring Their Disagreement

The fix was to stop treating the spread as a single number and start carrying the count alongside it. A spread computed from two sources and a spread computed from nine are different measurements that happened to be stored in the same column.

Once the count was available, most of the strange behaviour explained itself. We now require a minimum number of contributing sources before spread is allowed to influence anything, and below that minimum we fall back to the corpus and say so. The fallback is used more often than I expected — a substantial share of the players we look at are covered thinly.

The other change was to stop intersecting sources when merging them. That had been quietly discarding any category not present in all of them, which both narrowed the pool and made the count meaningless. Combining and counting is strictly more informative than intersecting, and it is the same amount of code.

“Two sources agreeing isn't consensus,” Dana said. “It's two. You were writing it down as though it were a finding.”

What We Had Been Systematically Wrong About

For most of a season we were biased against exactly the players we should have been most interested in. Thinly-covered, consistent performers in smaller competitions are, in principle, where an evaluation shop earns its keep — plenty of history, not much attention. Our uncertainty feature was penalising them for the attention part.

That is the most expensive error in this notebook, and not because of any individual bad rating. It is expensive because it was a systematic tilt away from our own best use, running quietly for months, produced by a feature we were confident we understood.

We only found it because somebody asked why a particular competition never appeared in the output. There was no alert for this and I cannot think of one that would have caught it. A feature that measures the wrong thing consistently does not look like a malfunction; it looks like a result.

Carrying the Count Everywhere, Forever

Every aggregate in the system now carries the number of observations it was computed from, and nothing may consume an aggregate without having access to that count. It is a slightly tedious constraint and it has caught three further versions of the same error in other places.

The general form of the lesson, which I would put above the specific fix: a summary statistic without its sample size attached is not a measurement, it is a rumour. We had several of those and did not know it.

We also now check, quarterly, whether any competition or category is systematically absent from our output. Not because we expect balance — we do not — but because absence was how this surfaced, and it surfaced by luck.

The competition that started the whole investigation still appears rarely. The difference is that we can now say why, and the reason is about its record rather than about who was watching it.

Note: PlayerGem is a fictional analytics shop and these accounts are invented. Nothing here is a pick, a recommendation, or betting or investment advice, and the players, teams and competitions described do not exist.