There is a difference between a bad week and a broken week, and the tell is whether the failures are related.

Ours were all the same failure. We produced a normal volume of ratings, at normal confidence, and a large majority of them missed in the same direction, for one reason, which we did not identify until the following Tuesday.

One Cause Wearing Twenty Different Faces

A scattering of misses across a week is uninformative — it is what the error rate is for. What we had instead was a distribution with a shape, and shapes have causes.

Nearly every failing rating was on a player whose recent workload had increased sharply. Our estimates for those players were all too low, consistently, by a similar margin. Twenty-odd separate ratings that were, in effect, one mistake repeated.

The cause was that our activity window had been lengthened three weeks earlier — for a good reason, to reduce noise in a sport with irregular scheduling — and lengthening it had the side effect of diluting exactly the signal that detects a role change. A longer window is more stable and slower. We had wanted the stability and had not costed the slowness.

What makes this a good example rather than an ordinary bug is that the change was correct in isolation, tested, and improved the thing it was meant to improve. It broke something else that nobody thought to check because the two were not obviously connected.

Grading by Cause Instead of by Outcome

We had no mechanism for noticing a shape. Our grading recorded whether each rating was right and rolled that up. A correlated cluster of misses looked identical to an unlucky spell.

What we added is a grouping step: after each grading run, cluster the failures by their characteristics and report any cluster larger than chance would produce. It is crude — a handful of attributes, a simple threshold — and it has caught four correlated failures since, two of which we would otherwise have absorbed into the error rate as noise.

The important part is that it changed what a bad week means. A bad week with no structure is expected and requires nothing. A bad week with structure is a bug report, and it now arrives labelled as one.

“You had twenty misses and one mistake,” Dana said, going through it. “Those are not the same size of problem and you were reporting them the same way.”

Three Weeks Between the Cause and the Symptom

The window change went in three weeks before the bad week. That gap is the expensive part.

In those three weeks we produced ratings that were subtly wrong on a specific population, and they graded out badly at a rate low enough to look like ordinary variance. It only became legible when a run of role changes happened to cluster, which is a matter of luck. Without that cluster we would have carried the defect for months, losing a small amount of accuracy on one population, invisibly.

The other cost was to our confidence in the earlier change. It genuinely had improved stability, and we ended up reverting most of it — not because it was wrong, but because we could not find a version that kept the stability without the blindness. So we gave up a real improvement to close a real hole, and I still do not know if that was the right trade.

Every Change Gets a List of What It Might Break

The practice that came out of this: any change to a window, threshold or weight now ships with a written list of what else depends on that quantity. Not a full analysis — a list, written by the person making the change, of the things they can think of.

It is a weak instrument and it works better than I expected, because the act of writing the list is where the thinking happens. Twice somebody has written the list and then withdrawn the change unprompted.

We also stopped treating tested-and-improved as sufficient. A change can be correct on its own terms and still make the system worse, and the only defence we have found is asking, out loud, what this quantity is also being used for.

The activity window is back roughly where it started, with a comment above it that is longer than the code. That seems to be the pattern with anything we have been burned by.

Note: PlayerGem is a fictional analytics shop and these accounts are invented. Nothing here is a pick, a recommendation, or betting or investment advice, and the players, teams and competitions described do not exist.