The Gate We Pass on Days the Corpus Feels Thin

There is a version of the gates problem that everyone at the shop understands immediately and nobody talks about much: what to do when the corpus is not missing but is not quite there either. Not empty — empty is easy, empty is a hard stop — but thin. Fourteen observations where we would prefer forty. A player who was injured for most of two seasons and has been healthy for six weeks. A hockey forward who changed roles at the trade deadline and whose prior numbers describe a different job. The corpus exists. It just does not feel like enough.

The temptation on those days is to pass the candidate through anyway and let the grading system sort it out later. The corpus clears the gate on a technicality — it meets the minimum count, the timestamps are recent enough, the stat version is consistent — and the rating gets issued with a quiet asterisk that nobody sees. We did this for longer than I would like to admit. The asterisk was a form of honesty we were performing for ourselves and not actually publishing.

The gate we eventually built for this situation is not elegant. It is slow, it requires a judgment call at the end, and it fails in both directions with roughly equal frequency. I am writing about it because the failure modes are more instructive than the design.

Why "thin" is harder to catch than "empty" on a corpus check

Empty corpora are caught at the first gate we run, because the count comes back zero or the player has no qualifying appearances in the window. That check takes seconds. Thin corpora are not caught there because they are not empty — they pass the count threshold and proceed downstream.

The problem surfaces at scoring, which is the worst possible place for it. By the time we are computing a rating, we have already spent time on availability checks, role confirmation, and stat-version validation. Discovering at that stage that we are working from fifteen appearances, seven of which came in a different system under a different coach, is not a data problem. It is a gate problem that escaped upstream.

What makes thin so treacherous is that it is a continuous variable masquerading as a binary one. The gate wants to say yes or no. The corpus wants to say sort of. We kept trying to find a sample-size threshold that would convert the sort-of into a clean no, and we kept finding that the threshold was context-dependent in ways that made a single number useless. Forty observations is plenty for a baseball hitter's contact rate. It is almost nothing for a basketball player's defensive positioning in a new scheme. The same count means different things in different sports and different roles, and a gate that ignores that distinction is not a gate — it is a turnstile.

The composite thinness score we built and mostly trusted

We spent about three weeks building what Reyna called a "thinness score" — her phrase, and a better one than anything I had. The idea was to combine four signals into a single index: raw observation count relative to a sport-and-role baseline, the proportion of those observations that fell within the current role context, the time elapsed since the last qualifying appearance, and a volatility flag that triggered when the standard deviation of the candidate's recent outputs was more than two times the cohort average.

"The score doesn't tell you the corpus is bad. It tells you the corpus is nervous. Those are different problems and they need different responses."
— Reyna

A candidate whose thinness score exceeded our threshold — we settled on a value derived from calibration against two prior seasons of graded ratings, not from theory — was held at the gate and flagged for manual review rather than automatically rejected. This was deliberate. An automatic rejection felt like precision we had not earned. Manual review meant someone looked at the flagged candidate and made a call about whether the thinness was disqualifying or merely cautionary.

In practice, manual review meant me, usually at the end of a long session, which introduced its own problems. But the structure was sound: the score surfaced the candidates that needed attention, and the human step acknowledged that the score was not the final word. We were trying to build a gate that admitted its own uncertainty, which is harder than it sounds when you are also trying to run the gate quickly.

The connection between this gate and the shrinkage step downstream matters here. A thin corpus is not just a gate problem — it is also a reason to pull the resulting estimate harder toward the base rate. We documented that link explicitly, because we had passed candidates before on the strength of a corpus that looked full but was concentrated in a single context, and the shrinkage step had not compensated because nobody had flagged the concentration. The thinness score was partly an attempt to catch that failure earlier.

What the manual review step actually cost us in the first month

The honest accounting: in the first month, manual review added an average of eleven minutes per flagged candidate. We flagged more candidates than expected — the threshold we had chosen was slightly too sensitive, and about a third of the flagged candidates turned out to be fine on inspection. Eleven minutes times too-many candidates is not a catastrophic cost, but it is a cost that compounds, and it landed entirely on the review step rather than distributing across the process.

More damaging was the error we made in the other direction. Three candidates that should have been held were passed through manual review because the reviewer — again, me — looked at a thin corpus and pattern-matched it to a similar candidate from the previous season who had graded out well. That is not a method. That is instinct dressed up as oversight, and when we graded those three ratings, two of them were among our worst-calibrated of the quarter. The thinness score had flagged them correctly. The manual step had overruled the flag on weak grounds.

This is the version of the story I find most useful to tell, because it is the one where the gate worked and the person running it did not. We had built something that correctly identified nervous corpora. We had then installed a human in the loop who was capable of being wrong in ways the score was not. The technically-correct-but-useless corpus entry is a known failure mode; the technically-correct-but-overruled gate flag is a different one, and we had not stress-tested for it.

Reyna pointed out, after we reviewed the quarter, that the two badly-calibrated ratings shared a feature: both candidates had high observation counts but low role-context proportions — the corpus was numerically thick but contextually thin, and the overall thinness score had been pulled down by the raw count. The score was weighting count too heavily relative to context. We had built the wrong composite.

What the gate looks like now, and what we still do not know

We reweighted the composite so that role-context proportion carries roughly twice the influence of raw count. This was a calibration decision made against observed grading outcomes, not a theoretical one, which means it is probably wrong in contexts we have not seen yet. We also tightened the manual review criteria: a flag can only be overruled if the reviewer documents a specific structural reason — a known data gap in the feed, a role change that has since been reversed — rather than a pattern-match to prior candidates. Vague confidence is no longer a valid override.

The gate now rejects more candidates than it used to. Some of those rejections are wrong — there are thin corpora that would have produced accurate ratings — but we have no way to know which ones, and that uncertainty is the point. A gate that passes uncertain candidates and hopes the grading catches the errors is not doing gate work. It is doing grading work twice, badly.

We kept the manual review step. We considered removing it after the first-month results and decided the problem was the reviewer's criteria, not the step itself. A gate that is entirely automated has no mechanism for catching the cases where the score is measuring the wrong thing, and we have been down that road before — the gate that leaked for eight days was automated and nobody noticed because there was no human checkpoint to notice at. The manual step is slow and error-prone and we are keeping it anyway.

Sample size beats recency — that is the oldest thing we believe — but sample size measured against what, exactly, is a question the thinness gate has forced us to keep asking. Count beats recency. Context-adjusted count beats raw count. What beats context-adjusted count is something we are still working out.

I do not know whether the right answer is a better composite score or a different gate architecture entirely — something that holds thin-corpus candidates in a separate queue and grades them against a separate baseline rather than forcing them through the same pipe as the well-attested ones. Reyna thinks the queue idea is sound. I think it is sound too, and I also thought the original thinness score was sound, so I am holding that opinion at arm's length for now.

Note: PlayerGem is a fictional analytics shop and these accounts are invented. Nothing here is a pick, a recommendation, or betting or investment advice, and the players, teams and competitions described do not exist.