The Window We Trusted Because We Set It First

The grading window is supposed to be the boring part. You pick a duration — how long after an event you will wait before scoring a rating against what actually happened — and then you leave it alone. The whole point is that it is fixed in advance, immune to whatever you learned in the meantime. We wrote that rule down. We believed it. And then we spent several months discovering that writing a rule down and believing it are not the same thing as following it.

The specific failure was not that we moved the window after the fact. We have done that, and it is embarrassing in a different way — moving the window to protect a rating is at least a recognizable sin, the kind you can name and correct. What happened here was stranger: we trusted the window because we had been the ones to set it. We had argued about it, documented it, and settled on it in a meeting that felt rigorous at the time. The authorship gave us a confidence that the reasoning did not actually support.

It took a calibration review to surface it. We were right at roughly the rate we expected, which initially read as a clean result. Then Priya pulled the breakdown by sport and the numbers stopped looking clean almost immediately.

Why the window felt safe before we tested it

We use a fixed grading window because the alternative is worse. The window has to be set before you see results — that is not an aesthetic preference, it is the only way calibration means anything. A window you adjust after the fact is not a window; it is a narrative device that makes your ratings look better than they were.

When we originally set the window, we ran it through three sports and two full seasons of historical data. The number we landed on — I will call it the anchor duration, because the specific figure is less important than the mechanism — survived that test without obvious damage. We recorded the decision, noted the sample sizes, and moved on. This is more process than most shops apply, and I think that is exactly where the overconfidence came from.

The problem is that passing a historical test tells you the window was appropriate for the data you used to test it. It does not tell you the window is appropriate for the data you will generate going forward. Those are different claims, and we had been treating them as identical for long enough that nobody was checking.

"We tested it on basketball and baseball and called it general. That's not a generalization, that's a sample size problem with extra steps." — Priya

She was right. Hockey and soccer have different rhythms of performance — the intervals between events, the clustering of fixtures, the way a player's output can compress into a short stretch and then go quiet — and our anchor duration had never been validated against either. We had assumed transferability. We had not earned it.

What we ran when we finally decided to check

The review took longer than it should have. Part of that was capacity; part of it was that nobody was eager to open a file that might require us to regrade a year's worth of ratings. Marcus flagged it first, then let it sit on the agenda for two cycles before Priya's calibration breakdown forced the issue.

We pulled every rating we had issued across all five sports over the preceding fourteen months and re-ran the grading under three alternative window lengths — one shorter than our anchor, one longer, and one sport-specific duration that Priya had sketched out informally. We were not trying to find a better window. We were trying to find out whether the one we had was doing what we thought it was doing.

The results were uneven in a specific way. For basketball and baseball, the anchor duration held up. The calibration gap — the difference between stated confidence and observed accuracy — was within the range we had seen in the original validation. For hockey and soccer, it was not. The window was too short for hockey, where performance clustering meant we were often grading before a player's actual response to a situation had fully resolved. For soccer, the problem ran the other direction: the window was long enough that late-season fixture congestion was contaminating grades with noise that had nothing to do with the original rating.

Neither of these failures was dramatic. The calibration gaps were not large. But they were consistent, and consistency is the thing that distinguishes a structural problem from random variance. We had a structural problem.

What the prior authorship actually cost us

The direct cost was fourteen months of ratings in two sports that were graded against a window we cannot now fully defend. We have not deleted any of them — quiet deletion is not how we operate — but they carry an asterisk in our internal records that will follow them for as long as we keep records.

The subtler cost was to the calibration signal itself. Calibration is only useful if the grading is honest, and grading is only honest if the window is right. Fourteen months of hockey and soccer ratings were telling us something about our confidence levels, and some of what they were telling us was wrong — not because the ratings were bad, but because the instrument we used to grade them was miscalibrated. We were learning from noise and calling it signal.

What I keep coming back to is the mechanism of the error. We did not skip the validation step. We did not set the window arbitrarily. We did the work, documented it, and then extended the conclusion beyond what the work supported. The confidence was not irrational given what we had done — it was irrational given what we had not done, which is a harder thing to see from the inside.

Marcus put it more plainly than I would have: "We trusted it because we remembered arguing about it. Arguing about something is not the same as being right about it." He is correct, and I found it annoying that he said it so easily.

What the window looks like now, and what we are still unsure about

We now run sport-specific grading windows. Basketball and baseball kept the anchor duration because it earned that position. Hockey moved to a longer window — roughly forty percent longer — which reduced the clustering problem without introducing the fixture-congestion noise we had seen in soccer. Soccer got its own duration, shorter than the anchor, with a secondary check at the original anchor length that we use as a cross-reference rather than a grade.

Tennis we left alone, which is either principled or lazy depending on how the next review goes. The sport's structure is different enough — discrete matches, no team context, relatively clean performance intervals — that the anchor duration seemed defensible on first principles. We noted the assumption explicitly this time rather than letting it ride on the general validation.

The process change that mattered most was not the window lengths themselves. It was adding a step that forces us to record, for every window we set, which sports and sample sizes it was validated against. The documentation existed before; what did not exist was a rule that prevented us from applying a window outside its validated scope without a new validation first. That gap is what let us run hockey and soccer on a basketball-and-baseball window for over a year without anyone formally noticing.

We also went back through the basketball and baseball ratings from the same period and re-examined whether the anchor duration's performance there was genuine or whether we had simply been lucky that the window fit those sports' rhythms. The honest answer is: probably some of both. The validation held, but the sample was not as large as I would like, and I am less certain than I was that the anchor duration is correct for basketball and baseball rather than merely not-obviously-wrong.

There is a version of this piece that frames the sport-specific windows as the solution. I want to resist that framing. What we have now is a better-defended set of assumptions. Whether those assumptions are right is a question the next fourteen months will answer, and we will publish whatever that answer is, including the parts that are uncomfortable. That commitment is the only thing that makes any of the rest of this worth doing — the grading is only honest if we do not get to choose which grades count.

The thing I am still not sure about is whether any validation process protects you from the specific confidence that comes from prior authorship — the sense that a decision you argued through carefully is a decision you can stop questioning. It might be that the only defense is a rotation of who challenges which assumptions, so that the person most attached to a rule is never the one responsible for testing it. We have started doing that informally. Whether it will be enough is not something I know yet.

Note: PlayerGem is a fictional analytics shop and these accounts are invented. Nothing here is a pick, a recommendation, or betting or investment advice, and the players, teams and competitions described do not exist.