Grading the Window Instead of the Rating

The grading review that week was supposed to be routine. We had a batch of basketball ratings from a three-week run in mid-season, a clear outcome record, and a spreadsheet that had been sitting closed for long enough that nobody remembered building it. We opened it expecting the usual mixture of hits, misses, and a few that landed so close to the line they were useless as evidence either way. What we found instead was a hit rate that looked better than it had any right to be — and a window definition that had been quietly adjusted sometime in the previous month without any of us writing it down.

The window is the fixed interval between when a rating is published and when it is graded. Ours had always been twenty-four hours for single-game ratings. At some point, apparently, someone had extended it to thirty-six for a subset of basketball entries, on the reasonable-sounding logic that certain stat categories settle differently depending on whether the game falls early or late in a back-to-back. The logic was not wrong. The timing of the change was the problem: it happened after the ratings had already been published, which meant the window was being chosen with partial knowledge of how the outcomes had gone.

We had not cheated deliberately. That is the part I find genuinely unsettling in retrospect. We had optimized the measurement instrument without noticing we were doing it, and the instrument had obligingly returned numbers that felt like progress. The ratings had not improved. The grading had softened around them.

How a window adjustment can make a bad batch look adequate

A grading window exists to prevent exactly this. When the window is fixed in advance — before any outcome is known — it acts as a constraint on the shop's ability to rationalize. When it shifts after the fact, even by twelve hours, even for a principled reason, it becomes a variable. And a variable that correlates with outcomes is no longer a measurement tool; it is a selection mechanism.

In this particular batch, the extended window absorbed several late-resolving games that had initially looked like misses under the original twenty-four-hour definition. Under thirty-six hours, two of those flipped to hits. The batch moved from a hit rate that would have prompted a real conversation to one that felt like noise. We would have filed it and moved on.

Remi caught it, and only because she was cross-referencing timestamps for an unrelated calibration check. She came to me with a printed log and a very flat expression.

"The window changed on the fourteenth. The ratings were published on the twelfth and thirteenth. That's not a methodology update — that's grading on a curve after you've seen the test."

She was right. The shop's grading window has to be fixed in advance — that is not a preference, it is the structural guarantee that the grading means anything at all. We had violated it without a meeting, without a memo, and without any apparent awareness that we were doing so.

What we did when we found the drift in the basketball batch

The first thing we did was reconstruct the batch under the original twenty-four-hour definition. This meant pulling the raw timestamps back out of the log, re-running the outcome matches, and accepting the two flipped ratings as misses. The corrected hit rate was lower. Not catastrophically, but enough that it sat in a range we take seriously rather than one we file quietly.

Then we went back further. If the window had shifted on the fourteenth without documentation, we had no guarantee it hadn't shifted before — or that this was the only category where it had happened. We spent two days auditing the prior six weeks of grading logs against the window definitions that had been recorded at the time of publication. We found one other instance, smaller, in a hockey batch, where a window had been extended by six hours. That one had not changed any outcomes, but it had not been written down either.

We also, and this is the part that felt worse, went back and looked at whether we had done something similar in an earlier period that we had already filed as clean. We found no smoking log entry, but we also found gaps in the documentation that meant we could not rule it out. That is a different kind of problem — not a confirmed error but an unverifiable one, which in some ways is harder to sit with.

We re-graded the basketball batch formally, published the corrected figures in the internal record, and added a step to the grading process: the window definition must be logged at the time of publication, not at the time of grading, and any proposed change to a window definition takes effect only on ratings published after the change is recorded.

What the corrected batch actually told us about the ratings themselves

Here is where it gets more complicated. Once we had the honest numbers, we had to interpret them — and the honest numbers were not obviously worse than what we would have expected from the method at that stage of the season. The hit rate under the corrected window was lower, but it was not dramatically out of range for a mid-season basketball batch with the corpus we had at the time. The miss cluster was real but small.

What that meant was that the window drift had not been concealing a crisis. It had been concealing a mediocre-but-not-broken result. We had accidentally made a fine batch look good, rather than accidentally making a bad batch look fine. The distinction matters less than I initially hoped it would.

The actual cost was not to the ratings — it was to the calibration record. Calibration, as we use it, is the comparison between stated confidence and observed accuracy across a long run. When a window shifts mid-batch, the denominator of that comparison is no longer clean. We could not use those six weeks of basketball data in the calibration audit without a footnote that undermined the audit. We flagged them as compromised and excluded them from the quarterly calibration review, which meant we had six fewer weeks of basketball signal than we thought we did going into the second half of the season.

That exclusion was the right call, and it was also the call that stung. We had done the work. We had published the ratings on time, graded them, and kept the record. The record turned out to be partially untrustworthy through a procedural failure that had nothing to do with whether the underlying ratings were good. This is a specific flavor of frustrating that I do not think has a clean resolution.

The rule we kept, and the one we almost talked ourselves out of

We kept the twenty-four-hour window for single-game basketball ratings. The argument for extending it — that late back-to-back games resolve differently — is not wrong, and we spent a full afternoon discussing whether a tiered window structure would be more accurate. It probably would be, in a narrow sense. A window calibrated to the actual settlement pattern of each stat category would produce cleaner grade distributions.

We decided against it anyway, for the same reason that a window agreed on in principle but not enforced in practice is worse than a blunt window that everyone actually follows. The more parameters a window definition has, the more surface area it has for drift. A single number, applied uniformly, is easier to audit and harder to rationalize around. We chose auditable over precise.

The other thing we kept was Remi's framing of the problem. She had called it "grading on a curve after you've seen the test," and that phrase became the internal shorthand for a specific failure mode — any adjustment to a measurement instrument that happens after partial outcome information is available. It applies beyond window definitions. It applies to gate thresholds, to corpus inclusion criteria, to the shrinkage parameters we set before a grading cycle. Any of those can drift in the same direction this one did, for reasons that feel principled in the moment.

We did not add more rules. We added one question to the pre-grading checklist: was this parameter set before any outcome was known? If the answer is uncertain, the batch gets flagged before grading starts, not after. The doctrine of never deleting a rating only holds if the grading attached to it is trustworthy — and trustworthy grading requires that the instrument measuring it was locked before the results came in.

I still do not know whether the window change on the fourteenth was made by me or by someone else in the shop. Nobody remembers doing it, and that might be the most honest thing the whole episode produced: a reminder that procedural drift rarely has a clear author, which makes it considerably harder to stop than the kind of error that someone has to decide to make.

Note: PlayerGem is a fictional analytics shop and these accounts are invented. Nothing here is a pick, a recommendation, or betting or investment advice, and the players, teams and competitions described do not exist.