Trusted Longest, Turned Out Shortest

For a long time, one entry in our basketball corpus carried more weight than almost anything else we had. It was old. It had survived several rounds of gate revisions, a full calibration cycle, and at least two separate attempts by Renata to find a reason to discount it. It had a clean grading record — not spectacular, but consistent — and when we pulled it into a new model pass it behaved. We trusted it the way you trust a colleague who has never given you a reason not to.

The problem, which we did not find for almost eighteen months, was that the apparent depth was a counting illusion. The entry spanned four seasons on paper. In practice, the player had missed large portions of three of them. What we had was not four seasons of evidence. We had roughly sixty-two individual performance events, scattered unevenly across a long calendar window, with gaps we had logged as absences but never subtracted from our confidence in the total. The entry looked old. It was not deep. Those are different things, and we had been treating them as the same thing.

When we finally ran the actual event count, Renata put it plainly: "We were trusting the span, not the substance." That sentence has lived on a sticky note above the corpus terminal ever since, which is the kind of thing you do when you have been embarrassed and want to make sure you remember why.

Four Seasons That Were Really Sixty-Two Games

The shop's oldest working principle is that sample size beats recency. A proven performer with a long, dense history outranks a player who has been excellent for six weeks. We have tested this belief more than almost any other, and it has held up more than almost any other. The problem is that "sample size" requires you to count the right thing.

We had been counting seasons. Four seasons is a long time. Four seasons of a player who appeared in roughly fifteen games per season — due to injury, rotation changes, and one extended absence we had categorised vaguely as "personal" — is sixty-two games. Sixty-two games is not a short corpus for a young player. For an entry we were treating as a foundational reference, one we leaned on when newer data conflicted with itself, it was thin.

The calendar span had done something specific to our judgment: it made the entry feel stable. We associate stability with age, and age with reliability. A corpus entry that has been sitting in the system for four years without causing problems reads as trustworthy in a way that a six-month entry never does, even if the six-month entry contains more actual observations. This is exactly the kind of informal rule we had made up and never formally tested — that longevity in the system was a proxy for depth. It was not.

What We Tried When We First Noticed the Gap Counts Were Off

The discovery came sideways, as these things usually do. We were not auditing this entry specifically. We were trying to understand why a different entry — a newer one, with eighteen months of dense activity — kept losing out to older entries in our weighting scheme. The weighting scheme applied a multiplier for corpus age. We were investigating whether that multiplier was correctly calibrated.

Marcus pulled the event counts for the top twenty entries by age-weight and sorted them by events-per-season. The entry in question appeared near the bottom of that list. Not the absolute bottom — there were two entries with even sparser records, both of which we quietly retired that same afternoon — but low enough that it was obvious something was wrong with how we had been thinking about it.

"The issue isn't that sixty-two is a small number in isolation. The issue is that we were acting like it was two hundred and forty. The age multiplier had inflated our stated confidence to a level the underlying count couldn't support."

— Marcus

We tried, first, to simply reweight the entry using the corrected event count rather than the season count. That was the conservative move: keep the data, adjust the confidence. The entry dropped significantly in our weighting hierarchy, which felt right. What did not feel right was that it continued to appear in model passes at all, given that we now knew our confidence in it had been miscalibrated for over a year. There is a version of this story where we quietly updated the weight and moved on. We did not take that version.

Eighteen Months of Miscalibrated Confidence, Honestly Counted

The grading record for this entry, re-examined after the event count correction, was not catastrophically wrong. That was almost the more uncomfortable finding. If the entry had been obviously bad, we would have caught it earlier through normal calibration review. Instead, it had been quietly overstated — we had treated it as a high-confidence reference when it was, at best, a moderate-confidence one, and the ratings it had influenced had reflected that misplaced certainty in ways that were too small to flag individually but consistent enough to matter in aggregate.

We went back through eighteen months of grading logs. In periods where this entry had been a primary reference point, our stated confidence averaged about twelve points higher than our observed accuracy warranted. Twelve points is not a rounding error. It is the difference between a well-calibrated output and one that systematically overpromises. This is the kind of failure that survives every gate because the gates are checking for different things — availability, activity level, recency of events — and none of them are checking whether the age multiplier is inflating confidence beyond what the event count supports.

The thing we got wrong, stated plainly: we had built a gate structure that was very good at catching entries that were obviously insufficient and not at all good at catching entries that looked sufficient but weren't. The entry had cleared every gate on every pass. The gates were checking the right conditions and measuring the wrong denominator.

What the Corpus Audit Procedure Looks Like Now

We retired the entry. That was the first decision and the easy one. The harder decision was what to change in the process so that an entry like this could not accumulate that kind of influence again without the event count being checked against the stated confidence.

The age multiplier still exists. We did not abandon the principle that older entries carry more evidential weight — that principle is sound, and we have tested it enough to be reasonably sure. What we changed was the denominator. The multiplier now applies to a rolling event count within the corpus window, not to the number of seasons spanned. A four-season entry with sixty-two events gets treated as a sixty-two-event entry. A two-season entry with two hundred and ten events gets treated as a two-hundred-and-ten-event entry. The calendar span is now recorded but not scored.

We also added a periodic audit step — quarterly, run by whoever did not build the entries being reviewed — that checks every top-weighted corpus entry for the ratio between its age-based confidence and its actual event density. The first audit caught three more entries with the same structural problem. None of them were as influential as the basketball entry, but all three had been carrying more weight than their event counts justified. We retired one and reweighted two.

Renata pointed out, correctly, that there is a different failure mode where you stop trusting an entry without being able to articulate why — instinct catching something the process missed. This case was the opposite: the process had given us every reason to keep trusting, and the instinct had been quiet. I am not sure which failure mode I prefer. The one where instinct catches the problem is at least faster.

The shop's working principle is still that sample size beats recency. We have not revised that. What we have revised is our confidence that we were counting the sample correctly — and I am not entirely sure we are counting it correctly now, only that we are counting a different thing than we were before, and that the different thing has held up for two audit cycles, which is not the same as being right.

Note: PlayerGem is a fictional analytics shop and these accounts are invented. Nothing here is a pick, a recommendation, or betting or investment advice, and the players, teams and competitions described do not exist.