Shrinkage is supposed to protect you from yourself. The logic is simple enough: when a confident estimate sits far from the base rate, you pull it back, because confidence is usually the problem. We built the rule into the method early and it saved us from some embarrassing overclaims in the first year. So we trusted it. We trusted it the way you trust a smoke detector — not by checking it, but by assuming it would make noise if something was wrong.

The problem was a soccer player. Invented for this piece, as they all are, but the mechanism was real enough that three of us remember exactly which week it surfaced. The player had a documented statistical pattern — a particular activity rate in a particular game state — that held across a large corpus, something north of two full seasons of granular event data. The estimate our model produced was consistently high relative to the base rate for that category. And so, every time, shrinkage pulled it down. The rating that survived the process looked reasonable. It was wrong, repeatedly, in the same direction.

We did not notice for eleven weeks. That is the part I find hardest to write plainly, but this desk exists precisely so I have to.

How a Protective Mechanism Became a Systematic Suppressor

The shrinkage rule, as we wrote it, works on a single axis: distance from the population mean. An estimate that sits two standard deviations above the mean gets pulled back toward it by a fixed coefficient, regardless of how that estimate was produced. The coefficient was calibrated against the full corpus, which is to say it was calibrated against the average quality of our estimates across all players and all sports.

That is where the design failed. The average estimate in our corpus is built on somewhere between forty and ninety observed events per player. This particular player had over two hundred and thirty. The corpus entry was deep, consistently maintained, and had been tested against grading cycles going back far enough to include a coaching change and a formation shift. The estimate it produced was not a hot streak mistaken for a trend — it was a stable finding with a large sample behind it. But the shrinkage rule did not know that. It saw a number far from the mean and it pulled.

The distinction between a confident estimate that deserves its confidence and a confident estimate that is simply wrong is the central problem in calibration work. We had written about it. We had not encoded it. Accuracy and calibration are genuinely different things, and our shrinkage rule was optimizing for the wrong one — it was reducing variance in our outputs without asking whether the variance was earned.

What We Built to Let Sample Size Modify the Pull

Once we identified the mechanism, the fix seemed obvious: weight the shrinkage coefficient by corpus depth. A player with two hundred-plus observations should be pulled back less than a player with forty-five, because the estimate for the former is resting on more evidence. We had actually described this principle in an earlier piece on how we approach shrinkage doctrinally — the idea that you shrink to resist overconfidence, not to resist data. Somehow the implementation had not inherited that nuance.

Remi built the revised function over about a week. The new coefficient was a sliding scale: below sixty observations, full shrinkage as before; between sixty and one-fifty, a graduated reduction; above one-fifty, a much smaller pull, with a floor we kept in place so that no estimate, however well-supported, escaped the base rate entirely. The floor matters. Even a deep corpus can be measuring the wrong thing.

"The original rule assumed all confidence was the same kind of wrong. That's a reasonable prior when you're starting out. It stops being reasonable when you have two years of data on somebody." — Remi

We ran the revised function backward across the eleven weeks in question. The corrected ratings for that player would have been consistently higher than what we published. The grading confirmed they would also have been consistently more accurate. That part was satisfying for about a day, until we started asking how many other players the original rule had quietly suppressed.

Eleven Weeks of Ratings We Published in the Wrong Direction

The honest accounting: eleven consecutive grading cycles in which the rating for this player was directionally incorrect, not because the underlying estimate was wrong but because the protective layer on top of it was miscalibrated. The error was not random. It was systematic and it ran in one direction, which is the worst kind — random errors wash out, systematic ones compound.

We also found three other players across different sports where the same dynamic was at work, though less severely. In two of those cases the corpus depth was borderline — around one hundred and sixty observations — and the suppression was mild enough that the ratings still graded acceptably. In the third case, a basketball player with a similarly deep file, the pattern was clear once we looked. We had not looked because the ratings graded well on average. Average accuracy had been masking a directional bias, which is exactly the failure mode that publishing a full error rate is supposed to prevent. It did not prevent it here, because we were publishing accuracy numbers rather than directional error distributions. That gap in our reporting is still only partially closed.

There is also a version of this mistake that runs the other way, and we have not fully stress-tested for it. A player with a shallow corpus and an estimate that happens to sit near the base rate will pass through shrinkage without being touched, even if the estimate is essentially noise. The rule protects against overconfident outliers but does nothing about underconfident mediocrities. I am not sure how much that costs us, because by definition those cases grade unremarkably.

What Stayed, What Changed, and What We Are Still Unsure About

The floor stayed. No estimate, regardless of corpus depth, exits shrinkage entirely untouched. That was the one piece of the original design we were unwilling to remove, partly on principle and partly because we could construct scenarios where even a two-hundred-observation corpus was measuring the wrong variable — a player whose role had quietly shifted, for instance, in a way our corpus had not yet absorbed. We have written about that kind of drift before in the context of entries that kept changing shape without an obvious cause. The floor is a hedge against our own blind spots.

The sliding coefficient is now standard. We also added a flag to the grading output that marks any player whose shrinkage adjustment exceeded a certain magnitude in the same direction for three or more consecutive cycles. It is not an alarm — it does not change anything automatically — but it surfaces for review. In the eleven weeks we missed, that flag would have appeared on week three. Whether we would have investigated it promptly is a separate question that I prefer not to answer with certainty.

What we have not resolved is the calibration reporting gap. We publish hit rates broken down by sport and by stat category. We do not yet publish directional error distributions in a form that would make a bias like this visible from the outside. Remi has argued that we should. The counterargument, which I have made and which I find less convincing each time I make it, is that directional distributions are harder to read without context and might be misused. That is a real concern. It is also a convenient one for a shop that would rather not display its systematic errors in a well-lit room.

The rule that saved us in year one was the rule that failed us in year three, and the mechanism in both cases was identical: it assumed all confidence was the same shape. I am not sure we have fully corrected for that assumption, or whether a method small enough to run in this shop ever fully can.

Note: PlayerGem is a fictional analytics shop and these accounts are invented. Nothing here is a pick, a recommendation, or betting or investment advice, and the players, teams and competitions described do not exist.