The Corpus Entry We Built Twice

Sometime in the second half of a basketball season we noticed that one player's scoring history was pulling harder on our shrinkage calculation than the sample size warranted. The estimate kept resisting the base rate. We checked the gates, checked the stat version, checked the ingestion log. Everything looked clean. It took Remi about forty minutes of cross-referencing import timestamps to find it: the entry existed twice, under two slightly different internal identifiers, and both copies had been graded, both had been weighted, and both had been treated as independent evidence for eleven weeks.

We had, in effect, given one player two votes in a process designed to give every player one. The corpus had been lying to us with our own data, in our own handwriting, and we had been reading the output and nodding along.

This is a piece about how that happened, what it cost us in calibration terms, and the specific thing we changed — and the specific thing we didn't, because we were not sure enough to.

How the same player entered the corpus on two separate occasions without us noticing

The shop pulls performance data from a central feed and assigns each player a unique internal key on first ingestion. That key is generated from a combination of name string and sport type. The problem, which in retrospect is almost comically simple, is that name strings are not stable. A feed that renders a name one way in October may render it differently in January — a hyphen dropped, a suffix added, an accented character flattened to its ASCII equivalent. When the January version came through, our ingestion script did not recognise it as a duplicate. It created a second key. Both keys were valid. Both passed every gate.

The two entries did not have identical histories. The first had eleven weeks of graded performance behind it. The second, created mid-season, had only the current week's data — but because it was new, our system treated it as a short-history entry and applied lighter shrinkage. The first entry, with its longer history, was being pulled toward the base rate appropriately. The second was not being pulled at all. Together they produced an estimate that was too confident and too high, and we had no automatic check that would have caught the collision.

What made it survivable, barely, is that the duplication affected only one player in that window. What made it embarrassing is that we had written, in the documentation for this exact system, that unexplained drift in a corpus entry was a signal worth investigating. We had documented the symptom. We had not built a detector for it.

The deduplication pass we ran after finding the double entry

Once Remi confirmed the duplicate, we ran a full retroactive deduplication pass across the active corpus. The logic was straightforward: for any two entries whose graded performance histories overlapped by more than seventy percent of their dates, flag them for manual review. Seventy percent was not a principled number — it was the threshold at which we stopped getting false positives on players who genuinely shared a common surname. We tried sixty percent first and spent a day reviewing coincidences.

The pass surfaced four other near-duplicates. Two were genuine: a hockey player whose name had been transliterated inconsistently across two feed updates, and a soccer player who had been entered once under a club-registered name and once under a national team registration. Both had been running in parallel for shorter windows than the basketball case — three weeks and five weeks respectively — and neither had accumulated enough graded history to meaningfully distort the output. We merged them, kept the longer history, and closed the keys.

The other two flagged entries turned out to be different players who happened to share a surname and a position type in the same sport. The overlap in their histories was coincidental. We reviewed them manually, confirmed they were distinct, and added a secondary identifier — a positional subtype combined with an approximate career-start window — to reduce the chance of future false flags. It was an inelegant solution and we knew it.

"The problem with fixing this kind of thing is that you're always building the fix for the case you just found. The case you haven't found yet has a different shape." — Remi

We also added a post-ingestion check that fires whenever a new key is created: it runs the same seventy-percent overlap test against the existing corpus before the entry is confirmed. If it trips, the entry goes into a review queue rather than live population. In the three months since, the queue has caught two genuine duplicates and eleven false positives. We consider that ratio acceptable without being confident it stays that way.

What the duplicate actually did to our calibration that week

We went back and reconstructed what our estimates would have looked like with only the correct, single entry in place. The difference in the point estimate was not large — roughly four percent above where a clean corpus would have placed it. That sounds modest. The problem was not the estimate itself; it was the confidence interval we attached to it.

Because the duplicate entry was being treated as independent evidence, our system believed it had more data than it did. A larger effective sample produces a narrower interval. The interval we published internally that week was about eighteen percent narrower than a correctly-sized sample would have justified. We were more certain than we had any right to be, and the certainty was entirely an artifact of counting the same player twice.

This is the kind of error that is particularly hard to catch through calibration review alone, because the calibration check compares stated confidence to observed outcomes — and if the outcome happens to land inside the artificially narrow interval, the error looks like a success. It was only because the estimate sat at the high end of even the inflated interval that Remi got suspicious and started pulling the thread. Had the player performed closer to his historical mean that week, we might have logged it as a clean result and never looked.

We have written before about entries that pass every gate and still mislead. This was a version of that — except the misleading was not in the data itself but in the architecture that held it. The data was fine. The container was wrong.

What we changed in the corpus structure and what we deliberately left alone

We kept the seventy-percent overlap check as a permanent part of ingestion. We kept the manual review queue. We kept the secondary identifier — the positional subtype and career-start window — as a tiebreaker for the name-collision problem. Those three things cost us about two hours of setup time per quarter to maintain and have paid for themselves in caught errors.

What we did not change is the name-string-based primary key. Remi argued, reasonably, that we should move to a hash of multiple fields — name, sport, position, and a career-start year derived from the earliest data point in the corpus. That would make collisions much harder. It would also make the system harder to audit by hand, because the keys would no longer be human-readable. When something goes wrong in the corpus — and things go wrong in the corpus — we want a person to be able to look at a key and know immediately what it refers to. We traded some collision resistance for legibility and we are not certain that was the right call.

The deeper issue, which the deduplication pass did not address, is that our ingestion script had no concept of a player's identity across time. It knew about name strings. It did not know that name strings change. A similar version of this problem — where a change in how a stat is recorded creates a false discontinuity — had come up before, and we had treated it as a stat-versioning issue rather than an identity issue. In retrospect they are the same problem wearing different clothes: the corpus assumes the world is more stable than it is, and the world keeps finding new ways to move.

We added a change-log field to every corpus entry that records the last time its key was touched and why. It does not prevent the problem. It makes the problem visible faster, which is the most we felt we could honestly promise.

There is a version of this story where the lesson is "validate your identifiers." That version is true and also not quite the point. The part that stayed with me is that we had eleven weeks of graded output that looked normal, passed calibration review, and was being generated by a process that was silently wrong. Eleven weeks is not a short window. I am not sure what the right amount of structural paranoia is — whether you check everything, whether you build detectors for failure modes you haven't seen yet, or whether you accept that some errors only become visible after they've run long enough to leave a mark.

Note: PlayerGem is a fictional analytics shop and these accounts are invented. Nothing here is a pick, a recommendation, or betting or investment advice, and the players, teams and competitions described do not exist.