The Grading Window We Opened Early

The grading window is supposed to be fixed in advance. We have written that rule down, argued about it, and agreed to it more than once. The whole point of a window is that it does not move because you are nervous, or hopeful, or just curious about how a rating is tracking. The moment the window becomes negotiable, the grade starts measuring your mood rather than the player's performance, and the calibration work that follows is worthless because it is grading a process you did not actually run.

We opened one early anyway. Not because we were nervous. Not because the rating was in trouble. We opened it because a colleague wanted to see how a particular basketball corpus was holding up mid-season, and the language used was just a look, which is the most dangerous phrase in this building. A look is not a grade. A look does not count. A look is how you give yourself permission to let partial results inform a rating that is not finished being tested yet.

What followed was a few weeks of low-grade contamination that took us longer to identify than it should have. This is the account of how it happened, what it cost, and what we changed — which is less than you might expect, because the lesson turned out to be narrower than the embarrassment suggested.

Why opening the window early felt harmless at the time

The rating in question covered a basketball player we had been tracking for two full seasons. The corpus was deep — well over the minimum sample we require before a rating earns a gem designation — and the player had been consistent enough that the rating felt, in the informal language we use internally, settled. Settled ratings are the ones you stop worrying about. They have earned their confidence interval through repetition, not through hope.

Mara, who runs our grading queue, flagged the window as open when it was not. She did not do this maliciously; she did it because the corpus genuinely looked complete at a glance, and because the player had already accumulated enough post-rating events to make a preliminary read feel reasonable. The distinction between enough events to be tempting and enough events to be valid is exactly the kind of distinction that disappears when you are inside a busy week.

The problem was not that the early data was wrong. It was actually favorable — the rating was tracking well, the player was performing close to what the corpus predicted, and the preliminary numbers were the kind that make you feel good about the process. That is precisely when the contamination is hardest to see. If the early look had shown the rating failing, we would have noticed the irregularity immediately, because bad news makes you check your work. Good news makes you nod and move on.

"The early numbers were fine, so nobody questioned whether we should have been looking at them at all. That's the tell. We only audit the process when it's hurting us."
— Mara

What we did once we realized the window had been opened

When I noticed the grading queue entry had a timestamp inconsistent with our fixed window schedule, my first instinct was to close it and pretend the preliminary read had not happened. This is the instinct you have to resist, because it is the same instinct that produces a window that keeps moving to protect a rating you have already become attached to. Closing the record does not undo the fact that the early data had been seen and discussed.

Instead, we logged the irregularity, noted the date the window was opened and the date it should have been opened, and flagged the rating for a secondary calibration check once the proper window closed. The idea was to run two grades — one from the early open, one from the correct date — and compare them. If they diverged meaningfully, we would know the early look had done real damage. If they matched closely, we would have evidence that the contamination was minor.

We also went back through the grading queue for the prior three months looking for other cases where the window timestamp did not match the scheduled open date. We found four more. Two were closures, not openings — windows shut before the game was fully resolved, which is a separate failure mode we have written about. One was a genuine data entry error. One was another early open, on a hockey rating, that nobody had flagged.

What the early look actually changed, and where we were wrong about the damage

Here is where I have to be precise about what we got wrong, because we got two things wrong and they pointed in opposite directions.

The first wrong thing: we initially assumed the early open had caused us to over-trust the rating. Our theory was that seeing favorable preliminary data had made us less likely to apply the shrinkage we normally apply to confident estimates — that we had, in effect, let the early look reduce the pull back toward the base rate. When we ran the comparison grade, that theory was partially confirmed. The final rating was about four percentage points more confident than our calibration history for this stat type would have justified. Not catastrophic, but real.

The second wrong thing: we assumed the hockey rating we found — the other undocumented early open — had caused similar over-confidence. It had not. The hockey rating had actually been under-graded, because the early data happened to catch a rough stretch that resolved before the proper window would have opened. The analyst who looked early had, without logging it, slightly discounted the rating as a result. The preliminary read had made them more cautious, not less. So the effect of an early open is not uniform. It depends entirely on what you see when you look.

That asymmetry is what made the whole episode harder to write up cleanly. We wanted a rule violation with a consistent consequence, and we got a rule violation with a consequence that depended on luck. The process failure was identical in both cases. The outcome damage was not. This is the kind of thing that makes you distrust your own post-mortems, because you can always find a way to make the story neater than it was. We tried not to.

It is worth noting that this failure is a cousin of the one where a window gets agreed on and then quietly set aside — the difference being that in that case the agreement was explicit and the violation was collective. Here the agreement was implicit and the violation was individual. Both produce the same calibration noise.

What we changed in the queue, and what we decided not to change

The concrete change was straightforward: the grading queue now requires a second timestamp field, populated automatically, that records when the window was actually accessed rather than when it was scheduled to open. The two fields are compared during the weekly audit, and any gap of more than six hours triggers a flag. This does not prevent an early open — nothing prevents a determined analyst from looking — but it makes the irregularity visible in the record rather than invisible in someone's memory.

We also added a step to the secondary calibration check: any rating that was viewed before its scheduled window now gets its confidence estimate pulled an additional half-step toward the base rate, regardless of what the early data showed. This is a blunt instrument. It penalizes the hockey rating that was already under-graded, and it probably over-corrects in cases where the early look was genuinely harmless. We kept it anyway, because the alternative — trying to assess whether each individual early look was contaminating — is exactly the kind of discretionary judgment that created the problem in the first place.

What we did not change: the window schedule itself. Mara argued, reasonably, that a shorter window would reduce the temptation to look early, because there would be less time between the rating and the grade. I was not persuaded. The window length is calibrated to give the player enough events to produce a meaningful sample, and shortening it to solve a discipline problem would be trading one error for a larger one. Sample size is the thing we are most careful about. We have made enough mistakes by treating the window as flexible that we were not going to solve a flexibility problem by introducing more flexibility.

The basketball rating, for what it is worth, came out of the secondary calibration check slightly deflated from where it had been. The player continued to perform consistently. We do not know whether the deflation was correct or whether it was the blunt instrument doing unnecessary work. That ambiguity is not comfortable, but it is honest.

The thing I keep returning to is that the early look felt like curiosity and not like interference — and I am not sure that distinction is as meaningful as we wanted it to be. Curiosity is how the shop justifies most of what it does. It is also how analysts talk themselves into exactly the kind of mid-process peeking that degrades the process. I do not know where the line is between healthy attention and premature judgment, and I am not sure the queue timestamp solves that problem so much as it makes the problem legible.

Note: PlayerGem is a fictional analytics shop and these accounts are invented. Nothing here is a pick, a recommendation, or betting or investment advice, and the players, teams and competitions described do not exist.