Fail Closed Data Validation: Why a Trading Bot Should Refuse to Boot on Bad Data

One morning a container of mine would not stay up. It booted, died, and the orchestrator restarted it; it booted, died, and restarted again. A tidy little crash-loop, once every few seconds, red all down the log. My first instinct was the normal one: something is broken, make it stop. Then I read the log line it was dying on, and my stomach settled instead of dropping. The node was refusing to start because one price series on disk had fewer rows than the last time I had blessed it. It was not broken. It was working. The crash-loop was the sound of a system declining to trade on data it did not trust, and it had just saved me from sizing a real position off a history that had quietly lost a decade overnight.

This piece is about that posture, which almost nobody writes about because it is unglamorous and slightly embarrassing to admit you need: a trading node should refuse to boot on bad data. Fail closed, not fail open. The counter-intuitive part is that a visible outage is the good outcome here, and I will tell the story from both sides, including the time my own gate was too eager and locked me out of a market I should have been in.

Why the quiet failure is the one that ends you

A live node does a frightening amount in its first few seconds. It boots, reads its history off disk, warms up its indicators, and starts placing orders, all before a human is watching. Whatever the data said at that instant becomes the ground truth for every decision the strategy makes that session. So the failure that scares me is not the loud one. A feed that throws an exception is a gift; the process dies, you get paged, you fix it. The failure that ends accounts is the quiet one.

Picture the ways a routine data refresh hands you worse data without raising a single error. A vendor caps free-tier history and your file that started in 2003 now starts in 2011, same format, valid parquet, a decade of regime gone. A download is interrupted and the last week never lands, so the file reads fine and is simply missing the most recent bars your indicators lean on hardest. One job clobbers another's file at the same path, plausible contents, different provenance, possibly a different adjustment convention you cannot see by eye. A cleanup script renames a series a live strategy depends on and it vanishes.

None of these throws. The filesystem is happy, pandas is happy, the strategy is happy. It just trades on a view of the world that is strictly worse than the one you validated, and it does it silently. That is the whole danger in one sentence: the data failures that quietly corrupt are exactly the ones that survive your attention, for the same reason backtest errors lean optimistic. Nobody deletes a strategy that suddenly looks fine. Nobody notices a feed that quietly shrank.

Fail open is how you size a position off a NaN

Here is the default that most hobby systems ship with, usually without anyone deciding to. Bad data arrives, and the code does the polite thing: it fills the gap. A missing price becomes a forward-filled last value, or a zero, or a dropna() that silently deletes the row, or a NaN that sails through a calculation and comes out the other end as a number that looks like data. This is fail open. The system meets corruption and keeps going.

I want to be precise about why this is the dangerous choice, because on the surface it feels robust: the system did not crash, it coped. But look at what it actually did. A NaN close, defaulted to zero, feeds a volatility estimate that collapses to near nothing, which feeds a position sizer that reads near-zero risk and asks for the maximum size the account allows. A forward-filled stale price makes a fast-moving market look frozen, and a mean-reversion signal happily bets against a move that already happened. The silent default did not neutralise the bad data. It laundered it into a confident, wrong number, and confident wrong numbers are what move real money.

That is the trade the two designs make, and it is stark. Fail open buys you uptime and pays with a position you cannot explain. Fail closed spends a visible outage and buys you the guarantee that you never trade on garbage. When the downside is asymmetric, and here it is brutally asymmetric, an outage you can see beats a bad position you can't. I would rather stare at a dead container for an hour than discover at settlement that I sized a trade off a default that was wearing a price's clothes.

The gate, in one sentence

So I put a gate in front of boot. Before the broker is queried, before the trading node is even built, the runner checks the data on disk against a record of what known-good data looked like the last time I blessed it. If the data has regressed, gotten smaller, shorter, or swapped for a different source, the node does not negotiate. It logs a critical, fires an alert to my on-call channel, and exits without touching the account. If the data is clean, it boots and trades.

The one design choice that makes this liveable, rather than a tripwire that fires on every success, is the asymmetry: growth is fine, contraction is the signal. More rows, an earlier start date, a later end, a brand new series, all wave straight through. The gate fires only when a series I already trusted gets smaller, shorter, or quietly replaced. That is deliberate, because you almost never accidentally gain good data, but you can lose it in a dozen ways before breakfast. A gate that blocked on every change would fire on your wins, and a gate that fires on wins is a gate operators learn to disable. The exact fingerprint I record for each file, and precisely what counts as a regression, is the machinery in the book; the principle is the part you can act on today.

The flip side: my gate crash-looped me out of a market

Now the honest half, because fail closed is a stance you can hold too tightly. That first crash-loop was the system working. A later one was the system being an idiot, and it was my fault.

I had wired the gate to be strict and, crucially, to be silent about a category of problem it should have merely warned on. A data source I depended on had a brief outage, and the file it wrote was thinner than usual, not because history was lost but because that morning's pull was genuinely incomplete and would have been fine an hour later. The gate did what I told it: it saw a contraction, refused to boot, and the container went into a crash-loop. Fine so far. The problem was that the alert did not fire the way I thought it did, so the node sat there dying every few seconds and nothing reached me. I found out because a strategy that should have been in the market simply was not, and I went looking. Fail closed had degraded into fail silent, which is its own trap: a halt nobody can see from the outside is indistinguishable from a hang.

The fix was not to weaken the gate. Weakening it would have re-armed the exact failure it exists to catch. The fix was two-part. First, separate the two jobs I had conflated: a blocking quality gate that halts on genuine contraction of known-good history, and a softer, advisory freshness check that warns when data is merely a bit stale or a cron looks late, and does not halt. Conflating them gives you a gate too trigger-happy to live with. Second, and this is the real lesson, fail closed but never fail silent. The halt has to be as loud on the way out as it is firm. Same one-line message, at three volumes: a log line, an alert to the channel I actually read, and a hard non-zero exit the watchdog treats as a real failure. If the gate stops the node, I know within seconds which file contracted, by how much, and what the remedy is, without opening a single log.

The portable rule

If you keep one thing from this piece, keep the shape of the decision, not my implementation of it.

On bad data, refuse to trade and alert loudly. Never proceed with a default that looks like data.

That is it. When a value is missing, corrupt, stale, or all NaN, you have exactly two honest options, and quietly substituting a plausible number is not one of them. Either the data is good enough to act on, or it is not and you halt and shout. The forbidden move, the one that ships in almost every naive system, is the third path: fill the hole with a zero, a forward-fill, a fillna, a dropna, anything that converts "I don't know" into a confident figure the sizer will treat as truth. A NaN that reaches your position sizer is not a small bug. It is the mechanism by which a system sizes a real trade off nothing at all.

And the operational corollary, learned the hard way: the loud part is not optional. A gate that halts silently throws away most of its value, because a halt you can't see becomes a position you can't explain the moment someone reaches for the override "just to get trading again this morning." Make the outage observable from outside the thing it guards, or you have built fail-silent and called it fail-closed.

Where the rest lives

I have deliberately not given you the gate's schema, the exact fields I fingerprint, the diff that decides what a regression is, or the boot-time fast path that makes this cheap enough to run on every start. That machinery, the manifest that stores known-good, the content hash that catches a same-size silent overwrite, the two-handed override discipline for re-baselining on purpose, and the runbook for what you actually do at 3am when the node crash-loops on a data problem, is Part III of Building a Production Quant Trading System. It is a living digital copy on Leanpub and a print paperback on Amazon. A sanitised, Apache-2.0 companion framework, where you can see the gate's shape in real code rather than prose, sits on GitHub.

Carry one sentence out of here: a crash-loop on bad data is not your system failing, it is your system refusing to lie to you. Build it so it refuses loudly.

This is an engineering essay, not investment advice, and it contains no tradable strategy. All figures are illustrative and sanitised, and the war-stories are about bugs, not profits.