The Grading Window We Trusted Too Long

For most of the shop's first two seasons, the grading window was thirty days. That was the number we had settled on before we published anything — not because we had tested it rigorously, but because it felt proportionate. Long enough to catch delayed box scores and corrected stat lines, short enough that a rating wasn't being judged against a player's performance from a different month of a different team's schedule. Thirty days. We wrote it down, we told each other it was the rule, and then we more or less stopped thinking about it.

The window held. The calibration reports from those early seasons showed nothing alarming. We were hitting roughly what we said we would hit, and the gaps between stated confidence and observed accuracy were small enough that we could attribute them to noise. That was probably true. It was also, as it turned out, a coincidence of timing — the structure of the sport calendars we were working with had made thirty days a reasonable proxy for something we had never actually defined. When that structure changed, the window didn't change with it. We kept trusting it because it had always been enough.

The moment we noticed the problem, we had already been running on a flawed grading surface for most of a third season. That is the part that still bothers me — not that we were wrong, but that the mechanism hiding the wrongness was the same one we had built to find it.

Why thirty days started lying without telling us

The grading window's job is simple: it defines the period inside which a rating is scored against what actually happened. A rating issued on a given day gets evaluated against outcomes that occur within the window. If the outcome falls outside the window — a game postponed past day thirty, a stat correction filed on day thirty-one — the rating is marked incomplete and excluded from calibration. That is the design. It is not a subtle design.

The problem we hit was not with the window's edges. It was with what the window was sampling. During our first two seasons, the basketball corpus we were drawing from had a relatively even distribution of game density across the calendar — teams played at roughly consistent intervals, and a thirty-day window captured a representative slice of any player's activity. Then the scheduling pattern shifted. One of the invented leagues we were tracking moved to a condensed midseason format that clustered games into tight two-week blocks separated by longer rest periods. Our thirty-day window, depending on where it landed in the calendar, was now capturing either a full burst of activity or almost none.

A window that captures a burst looked great. The ratings scored against a high-activity period had more outcomes to grade against, more variance to absorb, and the calibration numbers came out clean. A window that caught mostly rest days produced thin samples, and thin samples were being quietly excluded as incomplete rather than flagged as a structural problem. We were, without realizing it, grading the window's luck rather than the rating's accuracy. The calibration surface we were so comfortable with was partly a product of which windows happened to land on busy stretches.

What we tried when we finally looked at the sampling distribution

Dara was the one who caught it. She had been running a side audit on incomplete-rating rates by calendar position — not looking for this problem specifically, just maintaining a habit of checking what was being excluded before it disappeared into the denominator. She brought a chart to a Wednesday session that showed incomplete rates spiking in a pattern that tracked almost exactly with the rest-block schedule. "This is not random exclusion," she said. "This is the window eating itself in the off-weeks."

"The window was always a proxy for 'enough games.' We just never wrote down how many games that was. We wrote down days instead, and days and games stopped being the same thing."

We spent the next two weeks rebuilding the grading logic around a games-played threshold rather than a calendar threshold. Instead of closing a rating's evaluation at thirty days, we closed it when the player had appeared in a minimum number of contests after the rating was issued — a number we settled on after reviewing the corpus for what sample size had historically produced stable grade distributions. The calendar window became a backstop: if the games threshold wasn't reached within sixty days, the rating was marked incomplete. But the primary trigger was appearances, not time.

We then re-ran the prior season's calibration using the new logic. That was the uncomfortable part. The recalculated numbers were meaningfully different from what we had published — not catastrophically, but enough to matter. Our stated confidence had been running about four percentage points above our actual accuracy on the basketball corpus. Under the original window, that gap had been masked by the sampling asymmetry. Under the new one, it showed up plainly. This connected to something we had already written about fixing the window before results are visible — the logic holds, but only if the window is measuring what you think it is.

The four-point gap we had been publishing past for eight months

Four percentage points sounds small. In calibration terms it is not small — it is the difference between a shop that is honest about its uncertainty and a shop that is systematically overconfident. The whole premise of this work is that stated confidence should match observed accuracy. A shop that claims seventy percent confidence and hits seventy percent has done something real. A shop that claims seventy percent and hits sixty-six has a story to tell about why, and the story should not take eight months to surface.

The cost was not to any external audience — we do not publish our calibration tables in a form anyone could act on, and the shop's interest here is explicitly intellectual rather than consequential in any downstream sense. The cost was internal and methodological. We had been using a grading surface that flattered us slightly, and we had been doing it long enough that the flattery had started to feel like the baseline. When Dara's audit forced us to recalculate, the first instinct in the room was to check whether the new method was wrong rather than whether the old one had been. That instinct is worth naming: it is exactly the bias the calibration process is supposed to catch, and it showed up anyway.

We had also, in at least two internal reviews, looked directly at the incomplete-rating rate and decided it was within acceptable bounds. It was within acceptable bounds. It was also patterned in a way that should have prompted a question about what was driving it. We looked at the number and not the shape. That is a gate failure of a different kind — not the kind where a gate breaks, but the kind where a gate passes everything and you forget to ask why.

What the window looks like now, and what we are still not sure about

The games-threshold logic has been running for about five months. The calibration numbers on the basketball corpus have stabilized at a gap we consider honest — stated confidence and observed accuracy within one and a half percentage points across the last full grading cycle. That is better than the masked four-point gap, and it is probably closer to the truth of what the ratings are actually doing.

We kept the sixty-day backstop. A rating that doesn't accumulate enough appearances in two months is almost certainly evaluating a player whose situation has changed — injury, reduced role, something the corpus didn't anticipate — and marking it incomplete is the right call. What we stopped doing was treating the backstop as the primary mechanism. Time is a proxy. Appearances are closer to what we actually care about.

We also added a quarterly audit of the incomplete-rate distribution by calendar position, which is the thing Dara was already doing informally. Making it formal means it has to produce a written output, which means someone has to look at the shape and not just the number. That is a small process change with a disproportionate history behind it. We have written before about the specific failure mode of adjusting a window to protect a rating you already like — this is the opposite problem, but it shares the same root: a window that isn't being actively questioned will quietly start serving the ratings instead of grading them.

The soccer and hockey corpora are still on calendar-based windows. We have not found the same sampling asymmetry there, but we have also not looked as carefully as we looked at basketball after Dara's audit. That is an honest statement about where the work currently stands.

The thing about a grading mechanism that works for two seasons is that two seasons is long enough to stop examining it. I am not sure there is a principled answer to how often you should audit the auditor — at some point the regress has to stop somewhere, and wherever it stops is a place you are trusting because it has always been enough.

Note: PlayerGem is a fictional analytics shop and these accounts are invented. Nothing here is a pick, a recommendation, or betting or investment advice, and the players, teams and competitions described do not exist.