The Gate We Closed Too Late

The gate existed. That was the thing that made it hard to write up. It was not a missing check or a feature we had never built — it was a functioning piece of the pipeline that had simply stopped doing what we thought it was doing, and we did not notice for eight days because the ratings it was passing looked plausible. Plausible is the most dangerous output a broken gate can produce.

The gate in question was our availability check for a basketball corpus we had been maintaining for roughly two seasons. Its job was narrow: confirm that a player flagged as active was not just rostered but genuinely expected to feature in the night's event. We had written about the distinction before — available is not the same as expected to feature — and we believed we had built that distinction into the check. We had not built it as thoroughly as we believed.

What followed was not a catastrophic failure. No single rating was grotesquely wrong. The damage was quieter than that: a mild but consistent directional skew across eleven days of output, discovered only when we ran the grading window at the end of the month and noticed the pattern. By then the ratings had been scored, filed, and in three cases cited internally as evidence that a particular player profile was stabilizing. They were not evidence of anything except a leaky gate.

What the Gate Was Supposed to Catch

The basketball corpus carries a field we call feature confidence — a rough internal estimate of whether a player is likely to log meaningful minutes on a given night. It is not a prediction of playing time; it is a disqualification trigger. If feature confidence falls below a set threshold, the candidate does not get scored. It sits at gate two, just after the availability check and just before we do anything computationally expensive.

The threshold had been calibrated against two seasons of data and it worked acceptably — meaning it was wrong at a rate we had documented and were not embarrassed by. The problem was not the threshold. The problem was the data feeding it. Somewhere around day three of the affected window, the source we were pulling status updates from began returning stale records for a subset of players. The records were not obviously stale. They carried recent timestamps. They simply reflected a prior status rather than the current one.

Players who had been listed as probable two days earlier were still being passed as probable. Players who had been downgraded to questionable or out were not. The gate saw clean inputs and passed them. Everything downstream — the scoring, the shrinkage, the final rating — ran on a false premise about who was actually going to play.

"The gate didn't fail. It did exactly what it was told. We just told it the wrong thing and didn't check whether what we were telling it was still true." — Renata

Renata had flagged something adjacent to this problem the previous season, when we were working through a different availability leak in the hockey corpus. Her note at the time was that status fields age faster than any other data we handle and that our refresh logic was optimistic. We had updated the hockey pipeline. We had not revisited basketball with the same scrutiny. That is the version of this story I find hardest to tell.

How We Tried to Detect It Mid-Window

On day six, a colleague named Dov ran a spot check on three candidates who had been rated the previous night. He was not looking for a gate failure — he was checking something unrelated about how we were handling back-to-back scheduling in the corpus. In the process he noticed that one of the three players had been listed as out in a secondary source we do not use for the pipeline but do occasionally consult manually. He flagged it as a possible one-off discrepancy and moved on.

That was the moment we could have caught this earlier. We did not, because a single discrepancy between a primary and secondary source is not unusual, and because the rating for that player had already been graded against actual output — the player had not featured, the rating had missed, and we had logged the miss as an ordinary error. It looked like noise. It was signal.

On day nine, Dov ran the same spot check for unrelated reasons and found two more discrepancies. At that point we pulled the full window and compared our pipeline's status records against the secondary source for every candidate in the affected period. The pattern was immediately visible: a consistent subset of players, all sharing the same roster status update cadence, all being passed through the gate on stale data.

We closed the gate — meaning we suspended scoring for the affected corpus — on day nine. The contaminated window was eleven days. We had the gate failure for eight of them before we had any reason to suspect it, and three more while we confirmed what we were seeing.

What Eleven Days of Stale Data Actually Damaged

The direct damage was thirty-one ratings across the basketball corpus. Of those, nineteen involved players whose status had genuinely changed between our last clean refresh and the night they were scored. Twelve were probably fine — the stale record happened to match the actual status — but we could not distinguish those cleanly, so we voided all thirty-one. They sit in the archive marked as contaminated, not deleted, because quiet deletion is not something we do.

The subtler damage was to the calibration record. The grading window has to be fixed in advance — that is not a preference, it is the only way calibration means anything — and we had already fixed ours. The contaminated ratings fell inside it. We could not retroactively exclude them without corrupting the calibration logic, so they count. The month's calibration report for basketball shows a confidence-accuracy gap that is partly a real gap and partly an artifact of a data freshness problem. We noted this in the report. It still looks like a worse month than it was, and there is no clean way to separate the two effects.

The internal citations were the part that cost us the most time. Three separate notes in our working documents had used ratings from the contaminated window as evidence that a particular player type — high-usage guards with compressed rotation — was behaving consistently with our prior model. We had to retract those notes and rerun the analysis with the voided ratings excluded. The conclusion did not change dramatically, but the sample size dropped enough that the conclusion was no longer supportable with the confidence we had attached to it. We downgraded it from a finding to a hypothesis. That is the correct category for it. It should have been there from the start.

There is also the version of this where we ask whether the ratings would have been different if the gate had been working. Honestly: probably not by much. Most of the players with stale statuses were probable rather than out, and probable players mostly did feature. The directional skew was real but not large. The problem was not that we produced dramatically wrong ratings. It was that we produced ratings we had no right to produce, and then used them as if they were clean. That distinction matters to us more than the magnitude of the error, which I recognize sounds like a way of minimizing it. It is not.

What Changed in the Gate After This

The immediate fix was a freshness check on every status field before it is passed to gate two. If the timestamp on a status record is older than a threshold we set at four hours for game-day inputs, the candidate is held rather than passed or failed — held meaning it waits for a fresh pull rather than proceeding on stale data. This sounds obvious in retrospect. The reason it was not already there is that building it required us to store timestamp provenance alongside the status field, and we had been lazy about that infrastructure because it felt like overhead.

The less obvious change was to how we handle single-source discrepancies. Dov's day-six spot check had surfaced a real signal and we had logged it as noise because one discrepancy was within normal variance. We now flag any discrepancy between our primary source and the secondary source — not as a gate failure, but as a review trigger. A human looks at it before the next scoring run. This slows the pipeline by a small amount on the nights it fires. That is an acceptable cost.

We did not add a third source. That was a discussion we had, and the argument against it was that more sources create more reconciliation problems, and that the basketball corpus punishes you for data complexity in ways that are not always obvious until you are already inside a bad window. Renata's position was that the problem was refresh cadence, not source count, and that adding a third source with the same cadence would give us more confidence in stale data rather than less. She was right, and we did not add the third source.

The thirty-one voided ratings remain in the archive. The calibration report for that month carries a footnote. The hypothesis about high-usage guards is still a hypothesis. None of this is resolved — it is just correctly labeled now, which is the most we can usually say about a miss after the fact.

The thing I keep returning to is the three days between Dov's first spot check and the day we pulled the full window. A single discrepancy looked like noise because single discrepancies usually are noise, and because the rating had already been graded and filed and we had moved on. I do not know what a better version of that moment looks like in practice — a policy that treats every one-off as a full investigation would paralyze the pipeline. But I am not entirely comfortable with the answer we landed on either, which is to require two discrepancies before we act. Two is a threshold we chose because it felt right, and "felt right" is not a calibration method.

Note: PlayerGem is a fictional analytics shop and these accounts are invented. Nothing here is a pick, a recommendation, or betting or investment advice, and the players, teams and competitions described do not exist.