The Gate We Pass on Instinct, Call a Rule Later

For most of last spring, we had a gate that did not exist on paper. It existed in the way that Remi would pause when pulling a candidate's recent activity log and say something like, "this one feels thin in the wrong direction" — and we would both understand what she meant and move on. No written threshold. No label. A shared hesitation that we had apparently arrived at independently, agreed on without discussing, and applied inconsistently enough that it only looked like a rule in retrospect.

The gate concerned role clarity. Specifically: whether a player's recent appearances suggested they were operating in a defined, stable function within their team, or whether the log showed the kind of scattered usage that looks like volume but means nothing coherent. A high assist count one night, a high rebound count the next, minimal minutes the night after that — not because of injury, not because of schedule compression, but because the team was apparently still deciding what to do with them. We had been quietly discarding these candidates for months. We had not written it down.

Writing it down is where it got complicated. Because the moment we tried to formalize what we had been doing, we discovered that we had not, in fact, been doing the same thing. Remi's version of the gate was stricter than mine. Mine had exceptions hers did not. And somewhere in the gap between those two unwritten versions, three candidates had passed through that probably should not have.

The gate that lived only in shared intuition

The problem with an unwritten gate is not that it fails to work. Sometimes it works fine — instinct can be well-calibrated, and we have documented at least one case where a colleague's instinct outscored the corpus on a run of candidates that the formal process rated confidently and incorrectly. The problem is that an unwritten gate cannot be audited. You cannot grade something that was never stated. And a shop whose entire premise is that it publishes its own error rate cannot afford gates that leave no record.

What we reconstructed, after the fact, was that Remi and I had each developed the gate from the same underlying observation — that scattered usage patterns in the corpus were weakly correlated with inflated single-game numbers that didn't repeat — but had drawn the threshold in different places. Her version excluded a candidate if the three most recent appearances showed two or more different statistical profiles with no obvious situational explanation. Mine was looser: I was looking for something more like a general impression of incoherence, which is to say I was not really looking for anything I could define.

This is the version of the problem that doesn't get written up often: not the gate that was wrong, but the gate that was approximately right, applied approximately, producing results that were approximately fine until they weren't. The gate we closed too late on a different candidate earlier that year had at least been documented. We could point to the rule, point to the moment it should have triggered, and say clearly where it failed. This one had no such record. It had only Remi's memory and mine, which disagreed on three specific cases once we sat down to compare them.

Trying to reverse-engineer a rule from what we had been doing

We spent most of a Tuesday trying to write the gate down. The first attempt was too broad — it would have excluded candidates that both of us agreed, separately, were legitimate. The second attempt was so narrow it would only have caught the three cases we already knew about, which meant we were writing a rule to justify past decisions rather than to guide future ones. That is a known trap and we walked straight into it.

The version we eventually settled on had three components. First, a minimum coherence threshold: at least two of the last five appearances had to show the same primary statistical function — not the same numbers, but the same type of contribution. Second, an exception for documented situational causes: injury returns, lineup disruptions with a clear date, schedule anomalies. Third, a flag rather than a hard exclusion for candidates sitting near the threshold — meaning the gate could produce a "review" outcome rather than just pass or fail.

"The flag option felt like a compromise at the time. It probably was. But a gate that forces a second look is better than a gate that forces a wrong answer." — Remi

That third component was new. Neither of us had been applying anything like it in our informal version. We had been treating the gate as binary — pass or discard — which meant candidates near the threshold were being decided by whichever of us happened to be running the log that day. Adding the flag was an admission that the gate was not sharp enough to be binary, and that admission took longer to make than it should have.

We also went back and applied the written version to the three cases we had disagreed on. Two of them would have flagged. One would have passed cleanly under the new rule, which meant my informal version had been wrong to hesitate on it — I had been reading incoherence into a pattern that was actually a documented lineup rotation we had simply not cross-referenced properly. That one was mine to own.

What the unwritten version cost us in grading

The three contested candidates all went through the grading window. One performed roughly in line with its corpus profile. One produced the kind of scattered single-game output that the gate was designed to flag — not catastrophically wrong, but incoherent in exactly the way the pattern had suggested it might be. The third was the one I had hesitated on incorrectly; it graded well, which did not vindicate my hesitation so much as confirm that I had been looking at the wrong thing.

Net effect on calibration: small but real. When we ran the calibration check for that quarter, the role-clarity candidates as a group showed a confidence gap — we had stated higher confidence than the observed accuracy warranted. Not by a large margin. But the gap was there, and when we traced it, the unwritten gate was part of the explanation. We had been passing candidates we were not actually confident about, because the gate that should have flagged them existed only as a feeling.

There is a version of this that is more embarrassing, and it is worth naming: we had been aware, at some level, that the gate was unwritten. We had not written it down partly because writing it down would have required agreeing on it, and agreeing on it would have revealed that we had been applying it differently. Leaving it unwritten was the easier path. This is not a flattering thing to put in print, but it is accurate, and a shop that grades its own misses does not get to skip the ones that are inconvenient.

It also connects to something we have run into before with gates that passed because the corpus looked full — the surface appearance of rigor covering a gap in the actual process. In both cases, the problem was not a lack of data. It was a failure to look carefully at what the data was actually measuring.

What the written version changed, and what it didn't

The written gate has now been in place for two full grading windows. It has flagged eleven candidates in that time, of which we passed six, held four, and discarded one outright. The flagged candidates as a group have graded slightly worse than the non-flagged candidates, which suggests the gate is identifying something real. Whether it is identifying the right thing — whether "role incoherence" is the actual mechanism or just a proxy for something else — we do not know yet. The sample is too small to say.

What changed immediately was the audit trail. Every flagged candidate now has a record: who reviewed it, what the specific concern was, what we decided and why. When the grading comes back, we can trace the decision. That is worth something independent of whether the gate itself is well-calibrated, because it means we can improve it. An unwritten gate cannot be improved. It can only be replaced, after the fact, with something that explains the failures you already know about.

What did not change is harder to articulate. Remi still pauses on certain candidates in a way that precedes any formal check — a hesitation that arrives before she has run the coherence threshold, before she has looked at the exception criteria. Sometimes the formal gate catches what she caught. Sometimes it does not. We have not resolved what to do with that gap, and we are not sure we should try to close it entirely. The gates we have tried to over-formalize have their own failure modes, and some of them are worse than a well-calibrated hesitation that resists definition.

The written rule is better than the unwritten one. We are fairly confident of that. We are less confident that writing everything down is always the right direction to move in.

There is a version of good process where every gate is documented, every threshold is explicit, and nothing passes on a feeling. We keep moving toward that version. What we have not settled is whether arriving there would mean we had built something rigorous, or something that had quietly discarded the parts of judgment that do not survive being written down.

Note: PlayerGem is a fictional analytics shop and these accounts are invented. Nothing here is a pick, a recommendation, or betting or investment advice, and the players, teams and competitions described do not exist.