Grading a Window We Closed Too Early

The grading window is supposed to be the honest part. Everything before it — the corpus, the gates, the reading of lines, the shrinkage — is judgment dressed up in procedure. The window is where we stop arguing and just look. A player either did the thing or he didn't. The rating either held or it didn't. We record the result and move on. That is the theory.

What happened last winter was simpler and more embarrassing than any methodological failure. We closed a grading window on a basketball rating before the game had actually ended. Not before the final whistle in some technical sense — before the fourth quarter was finished. The player in question had already hit his threshold with several minutes remaining, the number was sitting there looking clean, and someone on our end logged the grade as a pass. The game then proceeded to do what games do: it got complicated. By the time the clock hit zero, the stat line looked different enough that the grade would have been a fail.

We caught it the next morning. That part, at least, went correctly. But the catch only happened because Renata was cross-referencing final box scores against our logged grades as part of a routine calibration check, and she noticed the timestamp on the grade didn't match the game-end time in our source data. Without that check, the error would have sat in our record quietly inflating our accuracy on a stat type we were already, it turns out, too confident about.

Why a game that wasn't over produced a grade we trusted

The immediate cause was a logging habit that had grown up around a different kind of event. In tennis and baseball, there are natural stopping points inside a match where a rating can be resolved cleanly — a set is over, an inning is complete, the relevant counting stat cannot change for the next twenty minutes. We had built our grading workflow around those rhythms, and at some point the same workflow got applied, without adjustment, to basketball.

Basketball doesn't stop. A player's point total, assist count, rebound number — all of it is live until the final buzzer, and in close games the final minutes are disproportionately active. The player we had graded early had accumulated his threshold total in the third quarter and into the fourth, and whoever logged the result made a reasonable-feeling judgment that the number was safe. It wasn't a reckless call. It was the kind of shortcut that accumulates quietly when a process works well enough for long enough that people stop reading it carefully.

The deeper problem was that we had no rule against it. The grading window has to be fixed in advance — that principle was already in our doctrine, but it addressed the question of when to open the window, not when to close it. We had written down the start condition and left the end condition implicit. "When the game ends" should not require documentation, and yet here we were.

What we did when Renata found the timestamp mismatch

The first thing we did was reverse the grade. That was not a difficult decision — the final box score was unambiguous, and the rating had failed against the actual final number. We logged the corrected result, noted the original error and its cause, and flagged the entry for our calibration review at the end of the quarter.

The second thing we did was pull every basketball grade from the preceding six weeks and check each timestamp against the corresponding game-end time. This took most of a day. We found two other instances where a grade had been logged before the final buzzer, both of them passes that remained passes after the game ended — so the grade itself didn't change, but the procedure had been wrong in the same way. We corrected the timestamps and noted them.

Then we wrote the rule down. The grading window for any basketball rating closes at confirmed game end, sourced from the same feed we use for final box scores, and the grade cannot be logged until that confirmation is in. Renata added a validation step to the logging tool that compares the grade timestamp to the game-end timestamp and throws an error if the grade is earlier. It took her about an hour to build. We had not built it before because we had not imagined needing it.

"The problem wasn't carelessness," Renata said, when we were writing up the incident. "The problem was that the process had a gap we'd never stress-tested because the gap only shows up in close games in the fourth quarter. Most games, it doesn't matter. So we never saw it."

That observation mattered more than the fix itself. The validation step closed the specific hole. But the reason the hole existed was that our testing had been done on games where the outcome was settled well before the end, and close games had never broken anything before. We had been protecting our ratings in ways we hadn't fully named.

What the inflated accuracy number had already cost us

The corrected grade changed our basketball accuracy figure for the quarter by less than one percentage point. If that sounds small enough to dismiss, the calibration implication is the part that isn't small. We had been comparing our stated confidence on basketball ratings to our observed accuracy, and the observed accuracy had been sitting slightly above where our stated confidence predicted. We had interpreted that gap as evidence that our basketball corpus was in good shape — that our estimates were, if anything, appropriately conservative.

After the correction, the gap narrowed. Not enough to reverse the interpretation entirely, but enough to make us uncertain about it. The corpus might still be in good shape. Or the previous quarter's numbers might have contained other early-close errors we hadn't found. We didn't know, and the honest answer was that we had been drawing a calibration conclusion from a record that had at least one confirmed contamination in it.

We went back further. We checked the prior two quarters of basketball grades the same way — timestamps against game-end times. We found nothing, which was reassuring but not conclusive, because the logging tool we used before Renata's update didn't always record timestamps with enough precision to be certain. Some entries showed only a date, not a time. Those we could not clear.

The thing we got wrong, stated plainly: we had been reporting our basketball calibration figures with more confidence than the underlying record warranted. Not because the numbers were fabricated — they weren't — but because we had not validated the data quality of the grading record itself before treating it as a reliable signal. We had applied the discipline of writing thresholds down in advance to the rating side of the process and then skipped the equivalent discipline on the grading side. That asymmetry was ours.

What the process looks like now, and what we still can't rule out

The validation step stayed. Every basketball grade now requires a confirmed game-end timestamp before it can be logged, and the system won't accept an entry that fails that check. We extended the same logic to hockey, which has overtime and shootout periods that create a similar problem — a player's stat line can change in the extra period in ways that would affect a rating graded at the end of regulation. We had not thought carefully about hockey overtime before this. We thought carefully about it after.

For tennis and baseball, we reviewed the existing logic and left it in place. The natural stopping points in those sports are genuine — a completed set, a completed inning — and the ratings we build in those sports are scoped to those intervals deliberately. The issue was never that we had the wrong logic for tennis; it was that we had applied tennis logic to basketball without noticing the difference.

We also added a data-quality flag to our calibration reports. When we publish our accuracy and calibration figures internally, they now carry a note indicating what percentage of the underlying grades have been timestamp-verified against a confirmed game-end source. For the current quarter, that number is high. For anything before Renata's fix, it is lower, and the figures from those periods are labeled accordingly.

What we cannot rule out is that there are other process gaps of the same shape — places where a procedure that works correctly in one sport got carried over to another without the stress-testing that would reveal the difference. The gates process, the corpus maintenance, the line-reading heuristics: all of them were built in contexts that may not transfer cleanly. We have looked at each of them since this incident, and we have not found anything. That is a different statement from "there is nothing to find."

A grading record is only as honest as the timestamps inside it, and a timestamp is only as honest as the process that wrote it down. We believe our current record is clean. We believed that before, too.

Note: PlayerGem is a fictional analytics shop and these accounts are invented. Nothing here is a pick, a recommendation, or betting or investment advice, and the players, teams and competitions described do not exist.