A gate that lets everything through is embarrassing. A gate that lets nothing through is worse, because it produces no bad output at all — and a shop with no bad output looks like a shop having a slow week.

Ours ran that way for most of a month.

The Availability Check That Compared Two Different Things

The gate in question asked whether a player was expected to feature. It did this by comparing an identifier from one source against a list assembled from another.

At some point the format of one of those identifiers changed — same information, different representation. The comparison never matched again. Every candidate was therefore treated as not expected to feature, and every candidate was rejected, correctly according to the logic and wrongly according to reality.

What made this invisible for weeks is that rejection is the normal case. We reject the great majority of candidates every single day and consider that healthy. A gate rejecting everything does not stand out against a background of a gate rejecting almost everything. The output shrank, and it shrank during a stretch when there were genuinely fewer events, so the shrinkage had an innocent explanation available.

Making Each Gate Report Its Own Rate

The immediate fix was a normalisation step on both sides of the comparison, which is dull and obvious in hindsight.

The structural fix was to make every gate report the proportion it rejects, and to record what that proportion normally is. A gate that usually rejects most of what it sees and is now rejecting all of it is a different situation from a gate behaving normally, and the only way to know is to have written down what normal was.

We deliberately did not set thresholds that alert. We tried that and it was unusable — legitimate seasonal swings are large, and we spent a fortnight acknowledging alerts that meant nothing. What we have instead is a small table anybody can look at, and a habit of looking at it when the output seems thin. Less rigorous, considerably more used.

“It was rejecting a hundred percent and the dashboard said ninety-four,” Priya said. “Nobody can see the difference between those two numbers by feel.”

There is a design point buried in this that took us a while to articulate. A gate has two failure modes and they are not symmetric. If it stops rejecting, bad output appears and somebody notices, because bad output is visible and annoying. If it starts rejecting everything, the consequence is an absence — and absences are invisible by construction. Which means the monitoring effort should not be split evenly between the two. Nearly all of it belongs on the quiet failure, and we had put nearly all of ours on the loud one.

A Month of Looking Careful

The cost here is strange, and it is the reason this one bothers me more than the leak did.

We produced almost nothing that month and our error rate for the period looks excellent. The handful of ratings that came through were from paths that bypassed the broken gate, and they happened to grade well. So the honest reading is that our best-looking month in a year was a month when the system was substantially broken, and if we had not found the cause we would have been left with a flattering number and no explanation for it.

That is a genuinely unpleasant thing to notice about your own record. It means a good result is not evidence that the machinery is working, and we had been treating it as though it were. The absence of visible errors was doing a lot of reassuring that it had not earned.

Distrusting the Quiet Weeks

What we kept is a habit rather than a mechanism: when output is unusually thin or unusually clean, that is now a thing to investigate rather than to enjoy. It sounds obvious written down. It was not the instinct we had.

We also stopped comparing identifiers across sources without normalising both sides, everywhere, as a rule. The specific bug is unlikely to recur; the general shape of it — two representations of the same thing that stop matching after an upstream change — has happened three more times in other places since, and the normalisation habit caught all three.

The rate table is still there. It is ugly and nobody has ever asked to improve it, which I take to mean it is being used.

I still do not know exactly which day the format changed. The logs did not keep enough to say, and that gap is its own small lesson about what we were storing.

Note: PlayerGem is a fictional analytics shop and these accounts are invented. Nothing here is a pick, a recommendation, or betting or investment advice, and the players, teams and competitions described do not exist.