The Gate We Trusted Because We Built It

The gate in question was the availability check — the one that decides, before anything else runs, whether a player is actually expected to feature in the event being evaluated. We built it ourselves over about three weeks in a quiet stretch between seasons, and it worked well enough in testing that we stopped thinking about it. That was the mistake. Not the logic inside the gate, which was defensible, but the confidence we attached to a thing we had made with our own hands.

There is a particular kind of trust that forms when you have written every line of a process yourself. You remember the reasoning behind each decision. You remember the edge case you anticipated and handled. What you do not remember, because you cannot, are the edge cases you did not anticipate — and those are precisely the ones that will eventually walk through the door looking perfectly valid.

This piece is about the eleven weeks during which our availability gate passed a category of candidate it should have stopped, and about the fact that we only found the leak because a colleague got suspicious about something unrelated. Left to our own assumptions, we might have trusted that gate for another full season.

What the Gate Was Supposed to Do, and What It Actually Did

The availability gate has one job: confirm that a player is expected to participate before the rest of the process touches them. A player who is injured, suspended, rested, or otherwise absent from competition is not a candidate. Evaluating their expected performance against a line that already prices in their absence is not analysis — it is noise dressed up as signal.

Our implementation checked three things in sequence. First, whether the player appeared on an active roster for the event. Second, whether any official status indicator flagged a limitation. Third, whether the player had logged meaningful participation in a recent prior event — our proxy for "actually playing" rather than "technically available." All three had to clear. If any failed, the candidate was dropped.

The flaw was in the third check. We had defined "meaningful participation" using a minute threshold calibrated against basketball and hockey, where playing time is recorded in consistent increments and a low number reliably signals a limited role. When we extended the same gate to other sports, we adjusted the threshold numerically but kept the underlying assumption: that participation volume was a stable proxy for expected involvement. In soccer, it is not. A player who appears for twelve minutes as a late substitute in one match may start the next. Our threshold passed them because the number cleared. The gate read that as availability. It was not.

The result was that a subset of candidates — players with recent low-minute appearances in soccer — moved through the availability check and into scoring. They were not unavailable in any formal sense. They were simply players whose expected role was genuinely uncertain, and the gate had been designed to resolve that uncertainty rather than flag it.

The Eleven Weeks We Spent Trusting Our Own Work

We ran the gate as designed from the start of the evaluation period through eleven weeks of soccer activity. During that time, somewhere between thirty and forty candidates cleared availability on the strength of a recent substitute appearance and were scored as if their participation in the next event was reasonably certain. Some of them did start. Some of them did not feature at all. The gate could not tell the difference, and because we had not built any downstream check for that specific failure mode, neither could we.

Remi flagged it. He had been reviewing calibration outputs for a different project and noticed that our soccer candidates were showing a wider confidence gap than the other sports — not dramatically wider, but consistently so, across several weeks. He brought it to the group not as a gate problem but as a corpus question, wondering whether thin sample sizes were inflating our stated confidence on certain profiles.

"The gap isn't random. It's the same direction every time — we're more confident than we should be, and it's concentrated in one sport. That's not a corpus problem. That's something upstream."

He was right, and we should have seen it before he did. When we traced the calibration gap back through the pipeline, the availability gate was the first place where soccer candidates diverged from the other sports in any structural way. The minute threshold was doing what we had told it to do. We had just told it the wrong thing.

What Eleven Weeks of Misplaced Confidence Actually Cost

The direct cost was to calibration. A gate that passes uncertain candidates as if they were certain ones does not merely add noise — it adds systematically biased noise. Every candidate who cleared on a substitute appearance and then did not feature was a case where our confidence was stated too high. Across eleven weeks and somewhere between thirty and forty candidates, that is a meaningful dent in the calibration record we keep on ourselves.

We publish our error rate. That is the whole premise. So the cost was not abstract — it is in the numbers we report, and it will remain there. We do not go back and revise the record because a flaw was later found. The eleven weeks stand as written.

The subtler cost was the one Remi's comment pointed at without quite naming: we had been less suspicious of this gate than we would have been of a gate someone else had built. Closing a gate for the right reasons requires being willing to question the reasoning behind it, and we were not doing that here. The gate was familiar. We remembered building it. That memory functioned as a substitute for ongoing scrutiny, which is exactly the kind of thing that sounds obvious in retrospect and is genuinely hard to catch in the moment.

There was also a smaller, more specific error embedded in the original build that we had to acknowledge: when we first extended the gate across sports, we documented the threshold adjustment but not the assumption underneath it. If we had written down "this threshold works because participation volume is a stable proxy for expected involvement," we might have noticed sooner that the assumption needed sport-specific validation. We wrote down the number. We did not write down the reasoning. That is a process failure, not a logic failure, and it is the kind that compounds quietly.

What We Changed and What We Chose to Leave Alone

The fix to the gate itself was not complicated. For soccer candidates, we replaced the single minute threshold with a two-part check: the threshold still runs, but any candidate whose most recent appearance was under a defined ceiling — set conservatively — is now held in a separate review queue rather than passed or dropped outright. The queue does not score those candidates automatically. It flags them for a manual read of available context before the gate resolves. More candidates die at the gate now. That is the intended outcome.

We did not extend the new logic to the other sports immediately. The temptation was to rebuild the whole gate from scratch with sport-specific rules for all five, but that felt like overcorrecting in a way that would introduce new assumptions we had not tested. The soccer adjustment addressed a demonstrated failure. Rebuilding basketball's availability logic to fix a soccer problem would have been the same mistake in a different direction — trusting our own reasoning about a thing we had not yet seen break. We have written before about rebuilding a gate after a specific failure, and the lesson we keep relearning is that targeted repairs age better than comprehensive redesigns.

The documentation practice changed more substantially. Every gate now carries a written statement of its underlying assumption — not just the threshold or the logic, but the claim about the world that the logic depends on. The minute threshold depended on a claim about participation volume as a proxy. That claim is now written down next to the threshold. When the claim stops being true, someone will notice, because the claim is visible.

Whether that is sufficient is genuinely unclear. The gate that leaked for eleven weeks was not undocumented — it was documented in a way that felt complete at the time. The assumption was just one level deeper than what we had written. There is always another level.

The thing that stays with me about this one is not the calibration gap or even the eleven weeks — it is the specific texture of the trust we had in that gate. It was not blind confidence. We could have explained the logic in detail. We just never turned that explanation toward the question of whether the logic was right for every sport it was touching, because explaining a thing and interrogating it are different activities, and familiarity makes it easy to mistake one for the other. I am not sure how you build a process that reliably catches that distinction. I am not sure it can be built.

Note: PlayerGem is a fictional analytics shop and these accounts are invented. Nothing here is a pick, a recommendation, or betting or investment advice, and the players, teams and competitions described do not exist.