The Gate We Closed Right for the Wrong Reason

The player cleared every prior gate. Corpus was deep enough — a little over two seasons of consistent data in a team sport, which is more than we usually have at this point in a process. The availability signal came back clean. The line had moved in a way that, historically, correlates with a player being on a full training load. We were ninety seconds from scoring the candidate when Priya looked up from her monitor and said the player's name with a slight frown, the way she does when something has not resolved yet.

We closed the gate. The player did not feature that night — a late scratch confirmed about forty minutes before the event began. So in the ledger, that entry reads as a correct closure. Gate closed, candidate disqualified, outcome consistent with the decision. For a long time, that is all we wrote down.

What we did not write down, and what took embarrassingly long to surface, is that we closed it for a reason that had nothing to do with what actually happened. We were right. The reasoning was not. And in a shop whose entire premise is that the process is what we grade — not just the outcomes — that distinction is the whole point.

Why "correct outcome" and "correct reasoning" are not the same ledger entry

The gate in question is what we call the participation gate. Its job is narrow: confirm that the player is expected to be on the field, court, or ice for enough of the event that the stat we are rating is actually achievable. It is not a health gate. It is not a form gate. It is a single, binary question — will this person play enough to be evaluated? — and the answer either clears or kills the candidate.

What Priya had noticed was a discrepancy in two participation signals that were supposed to agree. One source showed the player as active. One showed a status flag we had not seen before, a code that our ingestion pipeline was silently mapping to "active" because it did not match any known inactive code. The flag, it turned out, was a precautionary designation that the team used for players whose availability was still being assessed. Not inactive. Not active. Genuinely undetermined.

We closed the gate because the signals disagreed. That was the reasoning. Two sources, one ambiguous flag — close the candidate, move on. This is actually the kind of leak we have written about before: a gate that appears to be working because outcomes look right, while the underlying logic is doing something else entirely. The player was indeed scratched. But we did not know that when we closed the gate. We closed it because we were uncertain about status, which is a different and weaker reason than confirming unavailability.

How we tried to reconstruct what the gate actually knew

The reconstruction took about three weeks, spread across other work, which is why it sat so long. We went back through the ingestion logs for that flag code — the precautionary designation — and found it had appeared on forty-one candidates over the prior eight months. Of those forty-one, thirty-seven had eventually played. Four had not, including the one we caught.

So the flag was not a reliable signal of unavailability. It was a reliable signal of uncertainty. Those are not the same thing, and we had been treating them as equivalent. The gate was closing candidates who were uncertain rather than candidates who were unavailable, and most of the time those candidates went on to play. We were throwing away valid candidates at a rate we had not measured because the outcomes — the ones we did catch — looked fine.

"You can't grade a gate by the candidates it correctly closes. You have to grade it by the ones it closes that shouldn't have been." — Priya

She said that during the reconstruction and I wrote it on the whiteboard because it is the most compact version of the problem I have heard. We had been grading our gates the wrong way. We were counting correct closures as evidence of a working gate, when the actual test is whether the gate is closing the right things for the right reasons. A gate that closes everything would have a perfect record of correct closures. That is not a useful gate.

This connects to a broader problem we have documented elsewhere — the tendency to build rules that feel rigorous without testing whether they are actually measuring what we think. The participation gate felt airtight. It had a clear input and a clear output. What it did not have was a validated mapping between the flag codes it was reading and the real-world states those codes were supposed to represent.

What the wrong reasoning actually cost over eight months

Thirty-seven candidates closed incorrectly. That is the number we arrived at, and I want to be precise about what "incorrectly" means here — not that they would have produced good ratings, but that they were disqualified before we ever had the chance to find out. We do not know how many of those thirty-seven would have cleared the remaining gates. We do not know how many would have graded well. We lost the information entirely.

There is a compounding problem. Because the gate was producing correct outcomes often enough — the four genuine scratches out of forty-one flags — we had no reason to look at it. The calibration check we run on gates is outcome-based: did the gate close candidates who then failed to participate? Four out of four is a perfect record if you only look at the closed candidates who were confirmed scratches. We were not looking at the closed candidates who played. That was the gap.

It is a version of the same error we made in a different context with a corpus that looked internally consistent but was measuring the wrong thing from the start. The structure was sound. The inputs were wrong. And because the structure was sound, we trusted the outputs longer than we should have.

What it cost in concrete terms: eight months of a gate operating on a faulty assumption, thirty-seven candidates discarded without proper evaluation, and — most durably — a false sense of confidence in a part of the process that warranted more suspicion than it was getting. That last one is the cost that does not appear in any log.

What we changed in the gate and what we left alone

We did not rebuild the participation gate. The structure was correct — binary question, single decision, no scoring until it clears. What we changed was the flag mapping. Every status code from every ingestion source now has an explicit classification: active, inactive, or undetermined. Undetermined does not close the gate. It triggers a secondary check, which is a manual confirmation step that takes about four minutes and has a logged outcome.

The four-minute step was the argument. Dani pushed back on it during the review, reasonably, on the grounds that adding a manual step to a gate introduces its own failure modes — human error, inconsistency across the people doing the check, the temptation to default to "active" when the check is inconclusive. She was right about all of that. We added it anyway, because the alternative was continuing to close candidates on the basis of uncertainty rather than confirmed unavailability, and we had eight months of evidence that the alternative was worse.

What we left alone is the gate's position in the sequence. It still runs before scoring, before the line is read, before caps and shrinkage are applied. The lesson — if I am allowed to call it that loosely — is not that the gate was in the wrong place. It is that a gate can be correctly positioned and correctly structured and still be operating on a broken assumption about what its inputs mean. The discipline of writing down what a threshold is supposed to measure before you look at the data would have caught this earlier. We had written down the threshold. We had not written down the mapping.

The grading entry for that original candidate still reads as a correct closure. We did not change it. The outcome was correct. But we added a note to the record — "closed for the wrong reason" — because the shop's error rate is supposed to reflect the quality of the reasoning, not just the alignment of outcomes. A correct result produced by faulty logic is a miss with good luck attached, and we try to count those honestly.

I am still not certain we have the right secondary check. Four minutes is fast enough to feel manageable and slow enough to feel like a real step, which might mean it is exactly the right length or might mean we designed it to feel rigorous rather than to be rigorous. The question I keep returning to is whether there is any version of a gate that grades its own inputs — something that does not just ask "did the candidate play" but "did the flag that closed this candidate actually mean what we thought it meant." I do not know how to build that without creating a loop that defeats the purpose of a gate. But I think about it.

Note: PlayerGem is a fictional analytics shop and these accounts are invented. Nothing here is a pick, a recommendation, or betting or investment advice, and the players, teams and competitions described do not exist.