The Gate That Changes With the Checker

There is a gate in our process that checks whether a player is actually expected to feature in the event being evaluated. Not injured, not resting, not listed as probable but quietly understood to be out. It sounds straightforward. For about two years, we believed it was. Then we pulled the logs on a disagreement between two colleagues and found that the same candidate, checked on the same morning, had been passed by one person and held by another — not because the underlying information was different, but because the two of them were applying the gate differently in their heads.

We called it an availability check. What it actually was, it turned out, was a judgment call wearing the clothes of a rule. The written definition said something like: the player must be reasonably expected to participate at their normal role level. That word "reasonably" was doing an enormous amount of work, and we had never noticed because the outputs usually agreed. When they stopped agreeing, we had no mechanism to resolve it. We just had two people who had each been running the gate for months and had each developed a slightly different version of it.

This is the piece about that problem. Not about fixing it — we are not sure we have — but about what it looks like when a gate that appears structural turns out to be personal.

How Two People Ended Up Running Different Gates With the Same Name

The availability gate is, in theory, one of the cheapest checks to run. Public information, quick to find, binary in principle. Either the player is in or they are not. The problem is that "in" has gradations that the binary framing ignores. There is available, and then there is available-but-on-a-minutes-restriction, and then there is available-in-the-sense-that-nobody-has-said-otherwise-but-something-seems-off. Each of those is a different answer, and our written gate definition did not distinguish between them.

Nadia ran the gate conservatively. If there was any public signal of reduced participation — a coach's comment, a lighter practice load, a pattern of short appearances in the prior week — she held the candidate. She had been burned early by a player who was technically available and contributed almost nothing, and she had internalized that experience into the gate without writing it down.

Marcus ran the gate more literally. If the player was listed as active and no injury designation existed, the candidate passed. He had a different formative experience: he had watched us hold candidates who then played full minutes and produced normally, and he weighted the cost of over-exclusion more heavily than the cost of letting a marginal case through.

Neither of them was wrong, exactly. They had each built a reasonable gate from the same starting definition. The problem was that we had one gate on paper and two gates in practice, and we had no way of knowing which version any given candidate had passed through.

What We Tried When We Finally Acknowledged the Split

The first thing we tried was documentation. We sat down and wrote out every sub-case we could think of: full participation, limited participation, practice non-participant but game-day active, load management, undisclosed. We assigned each one a disposition — pass, hold, flag-for-review — and published it internally as the canonical gate definition. This took about a week and felt thorough.

It was not thorough. Within a month, Nadia had flagged two candidates that the new definition technically passed, and Marcus had let through one that the new definition should have held. The written rules had not transferred the underlying judgment; they had just created a new surface for disagreement. We had documented the outputs of the old gate without capturing the reasoning that produced them.

"The issue isn't the definition," Nadia said, after the third dispute in a week. "The issue is that we're both pattern-matching to different sets of cases we remember. The document doesn't have our memories in it."

The second thing we tried was calibration against historical cases. We pulled every candidate from the prior eighteen months that had been marked as a close availability call and asked both of them to reclassify each one under the new definition. The agreement rate was better — around eighty percent, up from what we estimated was closer to sixty — but the twenty percent that still diverged turned out to be exactly the cases that mattered most: the ones where the player's participation was genuinely ambiguous and the gate was doing real work.

We also tried assigning the gate to a single person on any given day, so at least the internal variation would be one person's variation rather than two people's. This helped with consistency within a session. It did nothing for consistency across sessions, and it introduced a different problem: whoever ran it last had effectively set the standard for that day, and there was no guarantee that standard matched the previous day's.

What the Inconsistency Actually Cost the Ratings

The direct cost was noise in the corpus. When we went back and tagged every availability-gate decision by who had made it, we found a statistically detectable difference in the pass rate between the two checkers — Nadia held about fourteen percent more candidates than Marcus did on ambiguous cases. That meant the pool of rated candidates was slightly different depending on who had been on gate duty, and our calibration grades were being computed against two subtly different populations without our knowing it.

This is the kind of thing that is easy to wave away. Fourteen percent on ambiguous cases sounds small. But ambiguous cases are not randomly distributed — they cluster around certain player types, certain sports, certain points in a season. In hockey, availability ambiguity spikes around back-to-back scheduling. In basketball, it spikes around load management patterns. The noise was not uniform; it was concentrated exactly where the gate was supposed to be earning its keep.

The cost I am more embarrassed about is that we did not find this through our own auditing. We found it because a colleague from outside the gate process asked why a particular candidate's rating had been withdrawn and then reinstated within the same week. When we traced it back, we found it had been held by Nadia on Tuesday, passed by Marcus on Thursday, and that neither of them had known the other had touched it. The corpus had a duplicate entry for the same candidate, scored against the same event, with different gate dispositions. This is exactly the kind of structural failure that trusting a gate because you built it will produce — the assumption of consistency in a process that was never formally verified to be consistent.

We had also, it turned out, made a version of this mistake before in a different context. A gate we had added for one sport and never formally reviewed had drifted in application over time for almost the same reason: the person who originally built it had left implicit knowledge in their head rather than in the documentation, and the people who inherited it had each reconstructed it slightly differently. We had seen this failure mode and still reproduced it.

What the Gate Looks Like Now, and What We Are Still Not Sure About

We kept the gate. That is not a trivial statement — there was a period where Marcus argued, not entirely without logic, that an availability gate this prone to subjective drift was introducing more noise than it removed, and that we would be better served by a stricter binary rule even if it meant passing some candidates we should have held. We did not go that direction, mostly because the gate's purpose is real: a candidate evaluated without checking availability is a candidate that can fail for reasons entirely outside the scope of what we are measuring. The gate is annoying and imprecise, but removing it would be worse.

What we changed is the resolution mechanism. Any candidate that triggers a sub-case in the written definition — anything that is not a clean pass or a clean hold — now requires both checkers to log their disposition independently before a decision is made. If they agree, it moves. If they disagree, it goes to a third review, and the third reviewer's notes become part of the permanent record for that candidate type. We are, in effect, using the disagreements to build a case library that the written definition never had.

The case library is still small. We have been running this process for about seven months and have collected sixty-odd adjudicated cases. That is not enough to be confident the library is representative, and we are aware that the cases in it are the ones we knew to flag — the ones we did not know to flag are still being resolved by individual judgment with no record. The most painful gate decisions are rarely the ones that get caught in time to document properly.

Marcus's pass rate and Nadia's pass rate on ambiguous cases have converged to within about four percent over the last quarter. We do not know whether that reflects genuine alignment or whether they have each learned to anticipate what the other would flag and adjusted accordingly. Those are different things, and only one of them is the outcome we wanted.

The gate is still running. The case library is still thin. What I keep coming back to is that the inconsistency existed for roughly two years before we found it, and we found it by accident rather than by audit. That suggests there are probably other places in the process where two people are running the same named procedure through different mental models — and that the ones we have not found yet are the ones that will embarrass us next.

Note: PlayerGem is a fictional analytics shop and these accounts are invented. Nothing here is a pick, a recommendation, or betting or investment advice, and the players, teams and competitions described do not exist.