The Gate We Inherited and Never Questioned

For a long time we had a gate that read, roughly: if the player has not appeared in the last fourteen days, remove them from consideration. Nobody wrote it down as a principle. It showed up in the first version of the evaluation code, copied from a basketball corpus we built before the shop had any real methodology, and it migrated forward untouched through every subsequent revision. When we moved into baseball, then hockey, then soccer, we carried it with us the way you carry a box from one apartment to the next without opening it.

I want to be precise about what happened, because the failure is instructive in a way that a cleaner story would not be. The gate was not wrong for basketball. Fourteen days without an appearance in a sport with an eighty-plus game regular season is a meaningful signal — it usually means something is wrong with availability, and availability is exactly what that gate exists to test. The gate was doing its job. The problem was that we never asked whether the job was the same job across sports, or whether the fourteen-day number was derived from anything at all. It was not. Someone had typed it once and no one had typed anything different since.

We discovered this in the middle of a hockey evaluation cycle, when a colleague named Priya pointed out that we were systematically discarding players who had simply been on the road stretch of a schedule. In hockey, fourteen days between home appearances is ordinary. We had been treating ordinary as disqualifying for over a year.

What the Fourteen-Day Gate Was Actually Measuring

Once Priya flagged it, we went back through the grading logs and looked at every candidate the gate had removed in hockey over the previous fourteen months. The number was not enormous — maybe twelve percent of the total pool — but it was not random. The gate was disproportionately removing players from teams with aggressive road schedules and players who had been rested ahead of a playoff push. In other words, it was removing exactly the kind of seasoned, high-sample candidates the corpus is supposed to protect.

The gate had been designed to catch a specific kind of noise: the player who has been absent because of injury, personal leave, or a quiet demotion that hasn't been formally announced. In basketball, fourteen days is almost always one of those things. In hockey, it is sometimes just Tuesday. The gate could not tell the difference, and we had never asked it to.

What made this harder to catch was that the gate's output looked clean. It was removing candidates, which is what gates are supposed to do. Nothing in the downstream results waved a flag, because the candidates it removed were simply gone — they never reached grading, never reached calibration, never appeared in any error report. A gate that lets the wrong things through is visible in the grades. A gate that removes the wrong things is invisible until someone counts what is missing.

How We Tried to Fix the Number Without Fixing the Logic

Our first response was to change fourteen to twenty-one. I am not proud of this. It was the kind of fix that feels like analysis but is actually just arithmetic — we picked a bigger number because the problem seemed to be that the number was too small, and we did not stop to ask whether any number was the right instrument.

We ran the revised gate through a back-test on the hockey corpus. The false-removal rate dropped by about half, which felt like progress. Priya pointed out, correctly, that we had not actually established what the right threshold was for hockey — we had only established that twenty-one was less wrong than fourteen. Those are not the same thing.

"We optimized toward a number we invented to replace a number we inherited. At no point did we ask what the gate was supposed to be measuring. We just made the box bigger."
— Priya

She was right, and the back-test confirmed it in a way we did not want to see. When we looked at the candidates the twenty-one-day gate was still removing in hockey, many of them were structurally identical to the ones the fourteen-day gate had been removing — road-schedule absences, managed rest, nothing that implied unavailability in any meaningful sense. We had shifted the cutoff without ever defining the underlying concept.

The correct question, which we eventually got to, was: what does an absence actually mean in this sport, at this point in the season, for a player with this kind of usage history? That question cannot be answered with a single number applied uniformly. It requires the corpus to do some work — specifically, it requires knowing how often a player of a given type goes fourteen days, or twenty-one days, between appearances under normal conditions. If the base rate for that player type is high, the absence means nothing. If it is low, it means something. We had built a gate for one sport and applied it to four, which is a failure of design, not a failure of the number.

What We Got Wrong Before We Understood What We Were Doing

The honest accounting is this: for roughly fourteen months, our hockey evaluations were running on a corpus that had been pre-filtered by a basketball heuristic. Every gem that survived that period had survived a gate that was not designed for it. Some of those gems were probably correct for the wrong reasons. Some candidates that should have been gems were never scored at all.

We do not know the full extent of the damage, because we cannot grade candidates who were removed before grading. That is the specific cruelty of a gate error versus a scoring error. We have a rule about never quietly deleting a rating — but a candidate who never became a rating is not covered by that rule, and the gate was operating upstream of it. The rule protected the output. It did not protect the input.

There was also a subtler cost. When we finally rebuilt the gate properly — sport-specific thresholds derived from actual base rates in the corpus, with a secondary check on schedule density — the new gate removed fewer candidates overall. That sounds like an improvement, and in accuracy terms it was. But it also meant we had been understating our own uncertainty for over a year. We thought we were evaluating a filtered, clean pool. We were evaluating a pool that had been filtered by the wrong instrument. The calibration numbers from that period are not trustworthy, and we marked them accordingly in the logs.

A colleague named Dom, who handles most of the calibration reviews, said he had noticed the hockey calibration was slightly flatter than expected — less variance in the confidence intervals than the basketball numbers showed — but had attributed it to the sport rather than to the gate. That is a reasonable thing to attribute it to. It was also wrong, and it is the kind of wrong that feels obvious only after you know the answer.

What the Gate Looks Like Now, and What We Are Still Not Sure About

The rebuilt gate uses sport-specific absence thresholds derived from the corpus itself. For each sport, we calculated the median inter-appearance gap for active players at various points in a season, then set the disqualification threshold at two standard deviations above that median. A player is removed not because they have been absent for an arbitrary number of days, but because their absence is unusual relative to players like them in the same sport at the same point in the year.

This is better. It is also more complex, which means more places to be wrong in ways we have not found yet. The old gate was wrong in a way that was at least simple — one number, one sport's logic applied everywhere. The new gate is wrong in ways that will take longer to surface, because the thresholds are derived from the corpus, and if the corpus has a systematic gap in it, the thresholds will inherit that gap. We are aware of this. We have not solved it.

We also added a manual review flag for any candidate removed by the absence gate, so that a human looks at the schedule context before the removal is finalized. This is the kind of thing that sounds obvious in retrospect. It was not obvious when the gate was working quietly and nothing was visibly broken. The lesson — if there is one, and I am wary of calling it that — is that the gates worth auditing are not the ones that seem to be failing. They are the ones that seem to be fine. The gate that removed candidates silently and cleanly for fourteen months never generated a complaint. It never produced a result that looked wrong. It just removed things, and we let it.

Priya now runs a quarterly audit of every gate against the sport it is being applied to — not to check whether the gate is removing candidates, but to check whether the candidates it removes make sense given what we know about the sport's structure. It is not a perfect check. It is a slower version of the question we should have been asking from the start, which is: what, exactly, is this gate measuring, and did we ever decide that was the right thing to measure?

We inherited the gate from ourselves — from an earlier version of the shop that was moving fast and did not stop to write down why. I am not sure that version of the shop was wrong to move fast. I am not sure this version is right to move slower. What I keep coming back to is that the gate looked healthy the entire time it was wrong, and I do not have a reliable way to know which of our currently healthy gates are doing the same thing right now.

Note: PlayerGem is a fictional analytics shop and these accounts are invented. Nothing here is a pick, a recommendation, or betting or investment advice, and the players, teams and competitions described do not exist.