Validation that doesn't flatter: costs, out-of-sample and walk-forward

A backtest is an argument, and left to itself it argues in favour of the idea being tested. Nobody sets out to fool themselves, but the usual way of building a backtest makes it easy. This note describes the ways backtests flatter, and the checks Pragma runs so that a hypothesis has to earn its verdict.

Why backtests overstate results

Most backtests are built by someone who wants the idea to work. That alone is not a problem. The problems come from a few habits that tend to follow.

The data used to shape an idea is often the same data used to judge it. Every adjustment made while looking at the history makes the strategy fit that history a little better, and the final test then measures how well it fits, not whether it works.

Trading costs are often left out, or added as an afterthought once the idea already looks promising. By then, the decision to pursue the idea has been made.

Choices made after seeing results leak back into the strategy. A parameter is moved, a filter is added, a period is excluded, each for a reason that seems sensible at the time. Together they turn a test into a search.

None of this requires bad intent. It is the default outcome unless the process is designed against it.

Costs from the first run

In Pragma, spread, commission and slippage are part of the backtest from the first run. They are not a final adjustment.

This matters most for short-horizon ideas. An effect measured in small price moves can look convincing before costs and disappear completely after them. If costs only arrive at the end, a lot of work has already gone into an idea that never had a chance. Counting them from the start means weak ideas fail early, while they are still cheap to drop.

Data the strategy has never seen

Part of the history is held back. The stages that design and implement a strategy never see it.

Only when the strategy is fixed does it meet the held-out data. If the result holds there, that is evidence the strategy captures something real. If it falls apart, the earlier result was most likely the product of fitting. Either way, the test is honest, because nothing about the strategy could have been tuned to pass it.

Rolling windows

One held-out period can still be lucky. Markets change, and a strategy can look good in a single stretch of history for reasons that do not repeat.

Walk-forward testing addresses this. The strategy is fitted on one window, tested on the window that follows, then the process moves forward and repeats. The result is not a single verdict on a single period but a series of them across changing conditions. A strategy that only works in some windows is, at best, dependent on conditions you cannot count on.

Criteria set before the test

Every hypothesis card states its acceptance criteria before validation starts: what the strategy must beat, and under which conditions. The validation stage checks the result against those criteria, not against criteria chosen after the numbers are in.

This removes the most tempting move in research, which is to look at a result and then decide what would have counted as success.

A rejected hypothesis is a useful result

The example on our homepage shows a hypothesis marked rejected. That is deliberate.

Most trading ideas should fail a careful test. A pipeline that rarely rejects anything is not rigorous; it is flattering. Rejections in Pragma are recorded with their reason, whether the effect vanished after costs, failed on held-out data or broke down across windows. That record is useful in itself: it stops the same idea from being retested until it passes, and it shows which assumptions tend to break.

What validation cannot do

None of these checks makes a strategy safe to trade. Simulated results do not predict future performance, and a hypothesis that passes every check can still fail in live conditions.

What validation can do is reduce the chance that you fool yourself. That is the job we built these checks for, and it is why we would rather show a rejection on our homepage than a result that only looks good.