For eight days our pipeline ran normally, produced output, logged no errors, and was completely broken.

What made it invisible is that the failure did not remove anything. It narrowed something, upstream, in a way that every downstream component handled correctly.

Eight Days Where Every Candidate Was One of Two Kinds

I found it while looking at something unrelated. Grouping a week of intake by category, every single row belonged to one of two categories — and both of those categories were, for separate and long-standing reasons, permanently ineligible.

So the gates were working. They rejected everything, correctly, all week. The output was thin but not empty, because a few candidates came from a second source that was still healthy. Nothing looked wrong from the outside. No error, no exception, no alert, because nothing had failed — the input had simply changed shape.

The cause turned out to be a merge step that combined several upstream sources. It had started dropping any category that did not appear in all of them, and one source had quietly reduced what it published. The intersection collapsed to two categories, and the merge did exactly what it was written to do.

The reason I want this written down rather than quietly fixed is that it is the most instructive kind of failure we have had. Nothing was wrong with any component. Every step handled its input correctly, logged nothing unusual, and completed. The system was in a state no individual piece of it could recognise as bad, because badness was a property of the relationship between the intake and everything downstream, and nothing was watching that relationship. A pipeline made entirely of correct steps can sit in a broken state indefinitely.

Fixing the Merge, and Then Fixing the Silence

The merge fix was straightforward and took an afternoon: combine across sources rather than intersect, keep a count of how many sources contributed to each candidate, and treat that count as information rather than a filter.

The silence was the real repair, and it took longer. The pipeline had no concept of an expected shape. It could tell you that a step had failed, and it could tell you how many rows it had processed, and neither of those would have caught this — the row count was low but not zero, and lower row counts happen for perfectly ordinary reasons.

What we added is a check on the composition of the intake rather than its size. If the mix of categories or sources moves sharply from one day to the next, that is reported, whether or not anything failed. It is a noisy check and we have tuned it twice.

“You were monitoring whether it ran,” Tobias said. “Nobody was monitoring whether it was still doing the same thing.”

Eight Days of Ratings We Had to Grade Anyway

The ratings we produced that week were not wrong, exactly. They were drawn from a pool so narrow that they were unrepresentative of anything, and there were few enough of them that a couple of ordinary outcomes swung the week's numbers noticeably.

We graded them and kept them, because the rule against deleting anything does not have an exception for weeks we would rather forget. Our record has a visible dent there. When somebody asks about it, the answer is that our intake was broken and we did not notice for eight days, which is a worse admission than a bad week of judgment would have been.

The part I still find uncomfortable is how long it would have gone on. I found it by accident, working on something else. There was no path by which the system would have told us, and the merge had been behaving that way for over a week before I happened to group a table the right way.

Watching the Shape, Not Just the Success

The doctrine that came out of this is short: a job that succeeds is not a job that worked. Every step now reports what it produced, not merely that it finished, and a sharp change in composition is treated as a problem to explain rather than a curiosity.

We also stopped intersecting anything, anywhere, as a matter of habit. An intersection is a silent filter with a failure mode that scales with the number of inputs, and almost every place we had used one, a union with a count attached was both safer and more informative.

None of this would have prevented the original upstream change. It would have told us about it inside a day, which is the most I think we can reasonably ask for.

The composition check has fired four times since. Three were harmless seasonal changes. One was another source quietly reducing what it published, which I take as the check earning its keep.

Note: PlayerGem is a fictional analytics shop and these accounts are invented. Nothing here is a pick, a recommendation, or betting or investment advice, and the players, teams and competitions described do not exist.