The most exciting result we have ever produced was a bug, and it took six weeks to establish that.

I want to write it up carefully, because everything about how we responded to it was reasonable and the outcome was still six weeks of chasing nothing.

A Pattern Too Clean to Be Real

The apparent finding was a consistent relationship between a player's performance and where in the season it occurred. Not subtle — a clear, repeatable pattern, visible in several sports, stable across the whole corpus.

We were appropriately suspicious for about a day. Then we checked it three ways, found it in each, and moved from suspicion to excitement. It was the first thing we had found that felt like an actual discovery rather than a correction to our own machinery.

The pattern was produced by a date comparison that was off by one, in a boundary condition, in the code that assigned each appearance to a segment of the season. A small fraction of appearances were being filed into the wrong segment, and because the misfiling was systematic rather than random, it manufactured a relationship where none existed.

The three ways we checked it all consumed the same segment assignment. So they were not three independent confirmations. They were one confirmation, computed three times, which is a distinction I should have insisted on at the beginning and did not.

The Question That Would Have Ended It on Day One

The question is: what would this look like if it were an artefact? We did not ask it. We asked whether the pattern was there, repeatedly, and got yes each time.

Asking the artefact question forces you to name a mechanism that could fake the result, and then to go and check that mechanism specifically. When Tobias finally did that, he had the cause inside an afternoon — he went looking for ways the segment assignment could be wrong rather than for further evidence that the pattern was right.

The general form: confirmation and falsification are not symmetric amounts of work. Three confirmations cost us three days and told us nothing we did not already believe. One serious attempt at falsification cost an afternoon and resolved it. We spent our effort in exactly the wrong proportion.

“You verified it three times using the same broken column,” Tobias said, entirely without malice. “That's one test.”

Six Weeks, and Something We Nearly Built

Six weeks of attention, most of it mine. We designed a feature to exploit the pattern. We got as far as a working implementation and a plan to deploy it, and the plan included changing how three other things behaved so they could take advantage of the discovery.

Had we shipped that, the harm would have outlasted the bug considerably. The bug was one line. The feature built on top of it touched several components and would have been defended by everybody who had worked on it, including me, and it would have been justified by a pattern that no longer existed once the line was fixed.

The narrow escape is what makes this the entry I think about most. We were not saved by process. We were saved by one person choosing a different kind of question on a Thursday, and I cannot construct a version of our procedures at the time that would have caught it.

Independent Confirmation Means Independent Inputs

Any result we are pleased about now requires at least one check that does not share code or columns with the original computation. That is inconvenient and slow and it is the only thing on this list that would have caught the segment bug.

We also added the artefact question to the standing review: before investigating why a result is true, spend ten minutes on what could make it appear true. It costs almost nothing and it inverts the order we naturally work in.

The pattern, incidentally, does not exist. We looked, afterwards, properly, with the boundary fixed. There is nothing there — which is a duller ending than the story deserved and the correct one.

The habit that has actually stuck is smaller than the rule. When somebody reports a result, the first question in the room is now which columns it was computed from — not whether it is real, not how big it is. Just which columns. It sounds pedantic and it takes one sentence to answer, and it would have caught the segment bug on day one, because the answer would have been the same column three times and somebody would have noticed the shape of that.

The feature is still in a branch somewhere. I have not deleted it, which probably says something I would rather not examine.

Note: PlayerGem is a fictional analytics shop and these accounts are invented. Nothing here is a pick, a recommendation, or betting or investment advice, and the players, teams and competitions described do not exist.