Shrinkage We Removed for a Colleague's Confidence

There is a version of this story where we made a principled methodological decision, watched it fail cleanly, and learned something. That would be a better story. What actually happened is that Renata made a compelling argument at the wrong moment, we were already tired of the rule in question, and we used her confidence as permission to do what we had been looking for a reason to do anyway. The shrinkage came out. Six weeks later we put it back, slightly stronger than before, and wrote this up.

The rule was a per-sport shrinkage floor — a lower bound on how far any individual estimate could be pulled toward the base rate, set by sport rather than by player. We had introduced it after noticing that our basketball ratings were arriving with a confidence we had not earned, and we had written at the time about why we set that floor differently for every sport rather than applying a single figure across the board. The floors were not elegant. They were blunt instruments that existed to stop us believing ourselves too much, which is exactly the kind of rule that feels unnecessary when you are feeling good about your process.

Renata's argument was not frivolous. She had been tracking a cluster of basketball candidates over a full calendar month and the shrinkage floor was, in her reading, suppressing estimates that the corpus was actively supporting. The players in question had long sample histories. The recent data was consistent with the historical data. The floor was pulling confident estimates inward not because the evidence was thin but because the rule did not distinguish between thin evidence and evidence we had simply decided to distrust on principle. She was right about that. She was wrong about what to do with it.

Why the Shrinkage Floor Existed and Why We Resented It

The floor had been installed after a specific embarrassment. We had produced a set of basketball ratings with stated confidence in the high eighties — meaning we believed our estimates were accurate at that level — and observed accuracy came in closer to sixty-five. That gap is not noise. That is a systematic overstatement of our own reliability, and the calibration check made it impossible to ignore.

The floor was our fix. Pull every estimate at least some minimum distance toward the base rate, regardless of what the corpus said, and the stated confidence would stop outrunning the observed accuracy. It worked. The calibration gap narrowed. We wrote it up without much ceremony and moved on.

The problem, and Renata was correct about this, was that the floor was not smart. It applied the same skepticism to a player with four seasons of dense, consistent data as it applied to a player with eighteen months of patchy returns. Applying shrinkage evenly when the underlying evidence is uneven is its own kind of error — you are not being humble, you are being indiscriminate. The floor treated all confidence as suspect rather than treating unsupported confidence as suspect. That distinction matters, and we had not built it in.

So when Renata brought her month of tracking data and pointed to five or six candidates where the floor was clearly doing nothing useful, we did not have a clean answer. We had a rule that had solved a real problem by being blunter than the problem required. That is not a comfortable place to defend from.

Removing the Floor After Renata Said She Was Sure

We did not remove the floor immediately. We spent about a week arguing about it, which felt responsible. Renata's position was that the floor should be replaced with a corpus-depth check — something that asked how many observations were behind an estimate before deciding how far to shrink it, rather than applying a fixed minimum to everything. That was the right idea. We did not build that. Instead, we removed the floor and told ourselves we would build the replacement shortly.

"The floor isn't humility. It's a habit we dressed up as a principle. If the corpus supports the number, the floor is just noise we added to feel careful." — Renata

I agreed with most of that sentence. The part I should have interrogated was "if the corpus supports the number," because that clause was doing a lot of work. Renata's sample — five or six candidates over one month — was not a corpus. It was a streak of cases where the corpus happened to be dense and the floor happened to be visibly unnecessary. We generalized from those cases to all cases, and we did it because she was confident and we were persuadable.

The floor came out in the third week of a basketball-heavy stretch. For the first two weeks, nothing obviously went wrong. Estimates moved outward from the base rate, stated confidence rose, and the ratings that survived the gates looked sharper than they had with the floor in place. Renata noted this. I noted it too, and I was more pleased about it than I should have been.

What Six Weeks Without the Floor Actually Showed

The calibration check runs on a fixed window. We grade every rating against what happened, stated confidence against observed accuracy, and we do not remove the embarrassing ones. By week four without the floor, the gap had reopened. Stated confidence was running about twelve points ahead of observed accuracy across the basketball ratings — not as bad as the original embarrassment, but bad enough that we could not pretend the floor had been cosmetic.

The specific failure was not the candidates Renata had flagged. Those held up reasonably well. The failure was in the candidates none of us had been watching closely — the ones where the corpus looked full but wasn't, where recent consistency had masked a thinner historical base than the sample count suggested. The floor had been quietly protecting us from those cases. Without it, our confident estimates on borderline corpus entries drifted outward and then got graded, and the grades were not good.

This is a version of something we had seen before. Passing a gate because the corpus looked full is one of the more reliable ways to produce a confident wrong answer, and removing the shrinkage floor had made us more exposed to exactly that failure mode. The floor had not been smart, but it had been load-bearing in ways we had not mapped before taking it out.

The honest accounting: Renata's argument was correct about the floor's design flaw. It was not correct that removing a flawed rule was better than fixing it. We chose removal because fixing it required work we had not scheduled. That is the actual miss here — not that we listened to a colleague, but that we used her argument to justify a shortcut.

What the Floor Looks Like Now, and What We Still Haven't Resolved

We restored the floor after the six-week calibration check made the case we should have made ourselves before removing it. Then we spent another two weeks building the corpus-depth adjustment Renata had originally proposed — something that asks how many independent observations support an estimate and sets the shrinkage minimum accordingly. A player with three full seasons of dense data gets a lower floor than a player with one season and a gap. The floors are still per-sport, because the variance structure is genuinely different across sports and a single figure still does not work.

The new version is better. It is also more complicated, which means it has more places to fail quietly, which means we are watching it more carefully than we watched the old one. That attention is probably the most useful thing the six-week experiment produced.

What we did not fix is the underlying dynamic. Renata was right that the floor was blunt. She was right that corpus depth should inform shrinkage. She was confident, we were persuadable, and we moved faster than the evidence supported. Sample size outranks recency is the oldest rule we have, and we violated the spirit of it by treating one month of supporting cases as sufficient reason to remove a rule that had been calibrated over a much longer window. We knew better. We did it anyway.

The new floor has been in place long enough now that the calibration gap has closed back to roughly where it was before Renata's argument. We have not run the comparison she originally proposed — the one that would tell us whether the new version is actually better than the old one or just differently imprecise. That test is on the list.

I still think Renata was right about the design flaw. I think about that sometimes — whether a rule can be correct and load-bearing at the same time, and whether we would have found a better replacement if we had fixed it under pressure rather than removing it and rebuilding from the gap it left. The new version might only look better because we built it after seeing what the absence cost us, and that is not the same as it being right.

Note: PlayerGem is a fictional analytics shop and these accounts are invented. Nothing here is a pick, a recommendation, or betting or investment advice, and the players, teams and competitions described do not exist.