When the Corpus Is Thin, the Gate Moves

The gate is supposed to be a fixed rule. A candidate either has enough prior performance behind them or they do not, and the number that separates those two states is written down somewhere and does not change based on how we feel about it that week. That was the design. The reality is that we have been moving the gate for years — quietly, inconsistently, and usually in the direction that lets us keep a candidate we were already interested in. I am not proud of that, but it is the accurate description of what happened.

The specific failure is this: a thin corpus is not the same thing across sports, or across stat types within a sport. Forty games of basketball rebounding data is a different kind of thin from forty games of baseball strikeout data, because the variance structures are different and the rate at which those numbers stabilize is different. We knew that abstractly. What we did not do, for longer than I would like to admit, was encode it — which meant every analyst in the shop was running a slightly different version of the gate in their head, and we were calling all of them the same rule.

What follows is an account of how we found that out, what we tried to do about it, and where we are still not confident we got it right. The short version is that the gate we pass differently when the corpus is thin is not a gate we have fully resolved. It is a gate we have made more honest about its own flexibility, which is not the same thing.

The thin-corpus gate was a number pretending to be a rule

For most of the shop's early history, the corpus gate worked like this: a candidate needed a minimum number of logged appearances before we would score them. We had a number. The number felt principled because it came from a calibration exercise we ran in the first year — we looked at how many observations it took before our ratings stopped moving dramatically week to week, and we drew a line there. That is a reasonable way to set a threshold.

The problem is that we ran that calibration exercise on basketball, and then applied the result to everything else. Hockey, soccer, tennis — they all inherited a gate that was derived from a sport with a particular game frequency, a particular stat volatility, and a particular season length. We did not notice this for a while because the ratings in other sports were not obviously wrong. They were just wrong in ways that took longer to surface, which is the most dangerous kind of wrong.

The tell was in our calibration logs. We track whether stated confidence matches observed accuracy — that is the whole point of grading — and when we broke the logs down by sport, hockey candidates who cleared the gate were underperforming our stated confidence by a margin that did not exist in basketball. The gate was letting candidates through in hockey that would not have cleared if the threshold had been set for hockey's variance structure. We had passed the gate because the corpus looked full when, by the right measure, it was not.

How we tried to make the thin-corpus gate sport-aware

The first thing we did was go back to the calibration exercise and run it separately for each sport. This took about three months of actual work — pulling the historical corpus entries, stratifying by sport and stat category, and measuring at what observation count the rolling estimates stabilized to within an acceptable band. We ended up with five different thresholds, one per sport, and a note in the methodology document that said they should be revisited annually.

That part worked reasonably well. The hockey threshold came out nearly double the basketball one, which matched what the calibration logs had been telling us. Tennis came out lower than either, which surprised us until Remi pointed out that the stat types we track in tennis have much tighter natural variance — a player's first-serve percentage does not swing the way a hockey player's shot volume does across a season.

"The number we had was never wrong for basketball. It was just wrong for everything else, and we called it universal because it was easier than admitting we had only really built one sport." — Remi

The second thing we did was add a modifier for stat category within each sport. A basketball player's three-point attempt rate stabilizes faster than their assist numbers, which are more situation-dependent. So the gate is not just sport-aware now — it is sport-and-stat-aware, which means it is a table rather than a number. This is less elegant than a single threshold. It is also more accurate, which is the trade we made.

We also formalized something that had been happening informally: a thin-corpus flag that sits below the gate but above a hard disqualification. A candidate who clears a sport-specific threshold but falls short of a higher confidence threshold gets scored with a shrinkage penalty applied — their estimate gets pulled harder toward the base rate than a full-corpus candidate would be. This connects directly to how our earliest gate is structured: cheap checks first, expensive adjustments later. The thin-corpus flag is an expensive adjustment, and it now happens in the right place in the sequence.

What the sport-aware gate got wrong almost immediately

Here is what we missed: the calibration exercise we used to set the new thresholds was itself built from a corpus that was not uniformly clean. Some of the historical hockey entries had data quality issues we had not fully resolved — specifically, some games logged during a period when our ingestion process was inconsistent. That meant the variance we measured in hockey was slightly inflated by noise, which meant the threshold we set was slightly too high, which meant we were disqualifying hockey candidates who actually had enough clean data behind them.

We did not find this for about two months. During that period, our hockey coverage was thinner than it should have been, and the ratings we did produce were, in retrospect, drawn from a less representative sample of candidates than we intended. The gate was more conservative than it needed to be, and conservative gates have their own failure mode: they do not produce obviously wrong outputs, they just produce fewer outputs, and the ones they exclude are not always the ones you would have wanted to exclude.

This is the version of the problem we had not fully thought through. We wrote at length about corpus entries that survived every gate and still produced bad ratings, but the inverse — a gate that was correctly motivated but set from bad inputs — is subtler and took longer to name. The gate was doing its job. The job had been specified from flawed material.

We corrected the ingestion issue, re-ran the calibration on clean data, and the hockey threshold came down by about fifteen percent. The methodology note now says the thresholds should be re-derived any time we make a significant change to ingestion, not just annually. That is a lesson delivered through embarrassment, which is the shop's primary teaching method.

What the thin-corpus gate looks like now, and where it still moves

The current gate is a table. Each sport has a base threshold derived from clean calibration data, and each stat category within that sport has a multiplier that adjusts the threshold up or down depending on how quickly that stat type stabilizes. A candidate who clears the adjusted threshold is scored normally. A candidate who clears the base threshold but not the adjusted threshold is scored with a heavier shrinkage penalty. A candidate who does not clear the base threshold does not get scored at all — that is still a hard gate, and it has not moved.

What has not changed is the underlying principle: sample size beats recency. A player with three years of data and an ordinary recent stretch outranks a player with six weeks of data and an extraordinary one. The thin-corpus gate is the enforcement mechanism for that principle, and making it more precise was not a departure from the principle — it was an attempt to apply it more honestly across contexts where the principle has different practical implications.

What still moves, and I want to be clear that it moves, is the judgment call about whether a corpus that technically clears the threshold is actually representative. A candidate might have forty logged appearances in hockey, clear the gate, and still have a corpus that is concentrated in a single stretch of games against weak opposition — something we have written about in the context of gates that get passed on instinct rather than explicit criteria. The table does not catch that. A human still catches that, and the human is still making a judgment call that the table is supposed to eliminate.

We have not resolved this. The table is better than the single number was. It is not a complete solution to the problem of knowing when a corpus is actually thin versus merely numerically small, and those two things are not the same.

The gate we pass differently when the corpus is thin is, at bottom, a gate about how much we trust our own count. We count appearances, we count appearances in the relevant context, we count clean appearances after ingestion issues are excluded — and at each stage, the count we have is a little different from the count we thought we had. Whether there is a version of this gate that does not require a human to periodically override the table is something I genuinely do not know.

Note: PlayerGem is a fictional analytics shop and these accounts are invented. Nothing here is a pick, a recommendation, or betting or investment advice, and the players, teams and competitions described do not exist.