The Gate We Closed on an Undefined Stat

At some point in the second year of running gates, we added a check for what we called "defensive involvement." The phrase appeared in an internal note, then in a process document, then in the gate itself. Nobody wrote down what it meant. We assumed we all meant the same thing, the way you assume everyone at a table means the same thing by "early" — and then the first time it matters, you find out they don't.

The gate was supposed to disqualify soccer candidates whose role had shifted away from the kind of defensive activity that made their historical corpus meaningful. A player who spent two seasons as a high-volume defensive midfielder and was now operating further forward would have a corpus full of numbers that no longer described what they actually did. The gate was a good idea. The problem was that "defensive involvement" was doing three different jobs in three different parts of the process, and none of those jobs had been written down with enough precision to catch the discrepancy.

We ran that gate for eleven weeks before Priya flagged it. By then we had passed candidates through it on the basis of a count we were not all counting the same way, and failed candidates on a threshold we had never formally set. The embarrassing part was not that we disagreed. The embarrassing part was that we had not noticed we were disagreeing.

What "defensive involvement" was actually doing in the gate

When I went back through the note where the phrase first appeared, I found that I had used it to mean something like: the player's corpus was built on defensive actions, and those actions need to still be happening at a meaningful rate. A rate threshold, in other words — something like "at least X defensive actions per ninety minutes across the corpus sample."

Marcus, who built the implementation, had read it differently. He treated it as a positional flag: if the player's listed role in the current lineup was a defensive one, the gate passed. If the role had shifted to an attacking position, the gate failed. Cleaner to implement. Faster to run. Also measuring something completely different from what I thought I had asked for.

The third interpretation came from the grading step downstream, where we had been using "defensive involvement" as a label for a composite that included both defensive actions and defensive positioning data — a number that had its own definition buried in a calibration document nobody had looked at in four months.

Three definitions, one phrase, zero explicit reconciliation. The gate was functioning. It was just not functioning as a single coherent thing. It was functioning as three slightly different things that happened to produce similar outputs often enough that we had not noticed the divergence.

"The issue isn't that the definitions were wrong," Priya said. "The issue is that we had a gate doing three jobs and we thought it was doing one. When it passed something it shouldn't have, we looked at the wrong definition to explain why."

She was right. And the reason we had never caught it was that the three definitions agreed on the easy cases — the clear defensive midfielder, the clear attacking forward. They only diverged on the candidates in the middle, which were precisely the candidates the gate was supposed to help us think about carefully.

How we tried to write a definition that would hold

We closed the gate. Not temporarily — we stopped passing candidates through it while we wrote down, in one place, exactly what the gate was supposed to test. This felt like it should take an afternoon. It took most of a week.

The first draft was mine, and it was too abstract. I wrote something about "the degree to which a player's current activity profile matches the activity profile underlying their corpus." Marcus pointed out, correctly, that this was not a gate — it was a research question. A gate needs a binary output. You either pass or you don't.

The second draft tried to operationalize it as a ratio: defensive actions in the most recent window divided by defensive actions in the corpus baseline, with a floor. If that ratio fell below a threshold, the gate failed the candidate. This was closer. It was also, as Priya noted, sensitive to sample size in the recent window in a way we had not thought through. A player who had featured in only two matches in the last month would produce a ratio based on very little, and the gate would behave strangely — which is a failure mode we had seen before in a different gate entirely.

The third draft added a minimum activity threshold for the recent window before the ratio was even computed. If the player had not featured enough recently to produce a stable estimate, the gate did not pass or fail them on this criterion — it flagged them for manual review. This felt like cheating at first. It was actually correct. Some candidates genuinely sit in a state where the right answer is "we do not have enough information to run this check."

We also separated the positional flag from the activity ratio, because Marcus's original implementation was not wrong — it was just answering a different question. We kept it as a separate, earlier gate, which is where it belonged. The question "has this player's listed role changed" is worth asking. It is just not the same question as "has this player's actual activity profile changed."

What eleven weeks of an undefined gate actually cost us

We went back through the eleven weeks and re-ran every candidate through the corrected gate. Of the candidates the old gate had passed, the new gate failed a small but nonzero number — players whose listed role had stayed the same but whose activity profile had drifted enough to make their corpus misleading. Of the candidates the old gate had failed, the new gate passed a similar number — players whose role label had changed but whose actual activity was still consistent with their historical data.

Neither group was large. The gate had been right on the obvious cases, as I said, and the obvious cases are most cases. But the candidates it had mishandled were exactly the interesting ones — the players in transition, the ones whose corpus was ambiguous, the ones where the gate was doing the most work. We had been most wrong precisely where we most needed to be right.

There is a version of this story where the lesson is "write your definitions down before you implement." That is true and also useless, because we thought we had. The note where the phrase first appeared felt like a definition to me when I wrote it. It was not a definition. It was a gesture toward a definition, and the gesture was clear enough to everyone reading it that nobody asked for more. We have inherited gates that had the same problem — phrases that felt self-evident to whoever coined them and turned out to contain three different things.

The actual cost was not the eleven weeks. The actual cost was the false confidence. We had been running a gate, checking a box, and the box had been doing something we could not have described if asked. That is a different kind of wrong than a gate that is simply miscalibrated. A miscalibrated gate at least knows what it is measuring. This one did not.

What the gate looks like now, and what we still argue about

The current gate has a written definition that fits in two sentences. It checks whether a player's defensive action rate in the recent window — minimum twelve qualifying appearances — falls below sixty percent of their corpus baseline rate. If it does, the candidate is failed on this criterion. If the recent window is too thin to compute, the candidate is flagged, not failed.

The positional-role check runs earlier, as a separate gate. It fails candidates whose listed role has changed by more than one positional category since the corpus was last validated. That gate has its own definition, its own threshold, its own failure mode. It does not share a name with anything else in the process.

What we still argue about is the sixty percent figure. Marcus thinks it is too lenient — that a player operating at sixty-five percent of their historical defensive rate has already drifted enough to make the corpus unreliable. Priya thinks the threshold should vary by position, because the relationship between role and activity rate is not uniform across the field. I think they are both probably right and that we set the number where we did because we needed to set it somewhere, which is a thing we have done before and will likely do again.

The gate we have now is better than the gate we had. It is also a gate whose threshold was chosen with less rigor than its definition. That tension — between having a clean definition and having a defensible number — is not something the definition exercise resolved. It just made the tension visible, which is at least a more honest place to be. When we discuss why certain sports stress the gates differently, this is usually the kind of friction we end up describing: not that the gate is wrong, but that its precision is uneven.

I am still not sure whether the problem was the phrase or the process. A better phrase might have forced the conversation earlier. A better process would have required a written definition before any implementation began. But I have also seen gates with careful definitions that drifted in practice, because the definition was written by one person and implemented by another and graded by a third, and none of them compared notes until something broke. Maybe the definition is not the thing. Maybe the thing is the comparison of notes, and the definition is just the excuse to have it.

Note: PlayerGem is a fictional analytics shop and these accounts are invented. Nothing here is a pick, a recommendation, or betting or investment advice, and the players, teams and competitions described do not exist.