When Updates Erased the Baseline

At some point in the third season of maintaining this particular entry, we stopped being able to answer a simple question: compared to what? The entry existed. It had rows, it had a history column, it had a confidence weight that had been adjusted eleven times. What it did not have, anymore, was the number we had started with. We had updated it into amnesia.

The player was a basketball forward — invented here, as always, to illustrate a mechanism rather than describe a market. He had a long prior record, which was why we built the entry in the first place. The corpus is supposed to rest on exactly that kind of depth. What happened instead was that each update we applied to keep the entry current quietly displaced a little more of the original signal, the way a river slowly replaces its own banks. We did not notice until we tried to grade a rating against the baseline and found the baseline was us, looking at ourselves.

The problem was not the updates individually. Each one was defensible. Taken together, they had produced something that we kept updating without fully understanding why — and by the time anyone asked the question out loud, the entry had lost the fixed point everything else was supposed to be measured against.

How the baseline disappeared one reasonable edit at a time

The entry had been built on four seasons of per-game data — roughly 260 appearances, which by our own threshold is the floor for treating a corpus entry as load-bearing. That sample was the reason we trusted it. Sample size beats recency; that is the oldest principle we have, and we repeat it often enough that it should be difficult to violate accidentally.

What happened was a sequence of individually sensible decisions. In the first update cycle, we removed a partial season the player had missed to injury, reasoning that it was distorting his per-game averages. Reasonable. In the second cycle, we applied a positional adjustment that we had developed for a different sport and decided was portable. Reasonable. In the third cycle, we re-weighted the two most recent seasons at 1.4x because the player had changed his role — more minutes, different usage — and the older data felt stale. Reasonable. Then we did it again.

By the time we reached the eleventh adjustment, the effective sample the entry was drawing on was not 260 appearances. It was closer to 80, because the re-weighting had mathematically crowded out everything older than eighteen months. We had not deleted the old rows. They were still in the table. They just no longer meant anything, because we had assigned them weights that rounded to zero in practice. The entry looked like a deep corpus. It was a hot streak in a lab coat.

What we tried when we realized the entry had no memory before itself

The first thing we did was pull the edit log. Every corpus entry is supposed to carry one, and this one did — which is how we could reconstruct the eleven adjustments at all. Remi, who maintains most of the basketball entries, looked at the log for about ten minutes and then said something I wrote down because it was precise:

"We didn't corrupt the entry. We improved it to death. Every change made it more accurate to right now and less accurate to anything."

We tried to reverse the re-weighting first, restoring the original sample weights to see what the entry would have said with its full history intact. The result was a number noticeably different from what we had been using — not dramatically, but enough to matter for calibration. The entry had been drifting in a direction the original data did not support, and we had provided the current for that drift ourselves.

We also ran a parallel version of the entry for six weeks: one instance with the accumulated adjustments, one with the original weights restored. We graded both. The older-weighted version performed better in aggregate, which was uncomfortable to write into the log. An entry we had trusted precisely for its age had been quietly made younger by people who thought they were helping it.

What the eleven adjustments actually cost us in calibration terms

The calibration gap was the real damage. We grade on whether stated confidence matches observed accuracy — that is the output that matters to us, not the hit rate in isolation. When we ran the six-week comparison, the adjusted entry showed a confidence weight of roughly 0.78. Its observed accuracy over the same window came in at 0.61. That is a meaningful gap, and it is the kind of gap that compounds: a rating built on an overconfident entry gets shrunk less than it should, which means the cap-and-shrink step that is supposed to protect us from believing ourselves had less to work with.

There was a second cost that was harder to quantify. The entry had passed every gate cleanly — availability, recency of activity, sufficient prior record on paper. An entry that clears every gate and still misleads is a specific kind of failure, because the gates are where we are supposed to catch exactly this. The gates did not catch it because the entry genuinely had 260 rows in the database. The gates cannot read weights.

We had also, at some point in the second season of updates, stopped asking why we were adjusting. The positional correction that we had ported from another sport had never been formally validated for basketball. We applied it because it was available and it felt right. That is not a method. That is instinct with extra steps — and unlike actual instinct, it had no track record attached to it.

What the entry looks like now, and what we locked to prevent this again

We rebuilt the entry from the original rows with no re-weighting beyond the standard recency decay we apply uniformly across the corpus. The positional adjustment was removed pending a proper validation study that, as of this writing, we have not finished. The injury-season removal was partially reversed: we kept the season out of the per-game average but restored it to the appearance count, so the sample size reflects what actually happened.

The more consequential change was structural. Every corpus entry now carries a locked baseline field — the original weighted sample size at the time of first publication — and any update that would reduce the effective sample below 70% of that baseline triggers a review flag rather than applying automatically. The flag does not block the update. It just requires someone to write a sentence explaining why the reduction is justified. That sentence has to exist in the log. In practice, it has slowed us down, which was the intention.

We also added a check that runs quarterly and compares the effective sample to the row count. If the ratio falls below 0.4 — meaning the entry has more historical rows than it is actually using — the entry is flagged for audit. The line sometimes preserves information the corpus has already discarded, and one of the things the quarterly audit is supposed to catch is exactly that divergence: a market that has kept the old signal while our own entry quietly moved on.

None of this fixes the six weeks of ratings we produced from the compromised entry. Those are in the grade log, marked with a notation that the underlying corpus was under review at the time. We do not delete embarrassing entries. That is the whole premise.

The part I keep returning to is that we did not update the entry carelessly. We updated it with attention, each time, for reasons we could have defended in a meeting. The baseline did not disappear because we stopped paying attention. It disappeared because we paid too much of the wrong kind — always to the question of whether the entry was current, never to the question of whether it still remembered where it started. I am not sure what the right frequency of that second question is, or whether a flag and a sentence in a log is actually enough to force it.

Note: PlayerGem is a fictional analytics shop and these accounts are invented. Nothing here is a pick, a recommendation, or betting or investment advice, and the players, teams and competitions described do not exist.