Backtesting Software: The Data Decides
Backtesting software replays a set of trading rules over historical price data and reports what they would have returned. The tool itself matters less than three settings inside it: which data it uses, how it assumes orders were filled, and what costs it charges.
How it works
Backtesting software replays a written rule over old price data and reports the outcome. You supply the entry, the exit, the size and the costs; it walks the bars and prints the account curve. The method it automates is backtesting.
Built-in testers, platforms and writing it yourself are the three categories. A strategy tester inside a charting platform, a scripted environment, and a spreadsheet or a bar-replay tool. Each trades convenience against control.
The three settings that decide the answer
And the data matters more than the software does. Two testers running identical rules on different price histories will disagree, and neither is wrong about its own data. Adjusted or unadjusted prices, missing sessions and bad ticks all move it.
How it assumes you were filled is the hidden setting. When a stop or a limit is touched inside a bar, the tester must guess whether you were filled, and at what price. Identical rules then give different answers.
It makes a beautiful curve far too easy to produce. An optimiser tries thousands of parameter sets in a minute and hands back the best. That is overfitting with a progress bar attached.
Dead companies missing from the data flatter every test. A universe built from today’s listed names excludes everything that was delisted, which is precisely the set that would have hurt.
And using a close you could not have known is the common bug. Deciding at a bar’s open on the basis of that same bar’s close is look-ahead bias. It is easy to write by accident and it produces flawless results.
In practice
A test without costs is not a test. Commission and the bid-ask spread come out of every round trip, and many testers default to neither.
It assumes liquidity that was not there. No tester models the case where volume was too thin to absorb your size.
More history is worth more than more settings. A slow method needs years to produce a sample worth reading, and cleaner data beats more tuning.
Most testers fill a gapped stop at its level. Price cannot trade through an opening gap, so the fill lands on the far side, worse than the level you set.
Which removes the largest risk from the record. Attaching a fixed pessimistic slippage to every stop loss fill is crude, and far closer to reality than assuming none.
Every round trip costs 2% of a bar. On this site’s shared history it is 0.0098 price units, or 2% of a median bar’s range and 45% of the smallest bar. The more often a method trades, the more of its result is that figure.
Five questions to ask of any tester
What data, exactly? Which vendor, which sessions, whether prices are adjusted for splits and distributions, and whether delisted instruments are present. A tester that cannot answer has capped what its output is worth.
What is the fill rule, and what happens on a gap? Find the setting, read what it does, change it and run again. A tester that hides its fill logic is a simulator with an opinion you cannot inspect.
What costs are applied, and can it hold data back? Commission, spread and slippage should be inputs you set, not defaults you inherit. Reserving history the optimiser never sees is the bridge to forward testing and paper trading.
What backtesting software is not
It is not evidence about the future. It describes one history under assumptions you chose.
It is not a trading bot. Testing a rule and running it are separate jobs.
It is not a source of data quality. It reports faithfully on what it was fed.
And it is not the end of the process. The unseen period is the only honest test.
When it fails
In a flat period it reports exactly what happened, and that is the problem. A trend rule tested across a trading range looks broken, and the same rule across a trending stretch looks superb. Neither is a property of the rule.
A daily bar hides the path taken inside the day. If a stop and a target both sit within one bar’s range, no tester can say which was reached first. It picks one by convention, and on those trades the convention is the strategy.
It fails silently on damaged data. A missing session or a bad tick produces trades that never existed, and nothing marks them out.
It fails when the sample is small and nobody says so. A curve built on twenty trades is a story about twenty events, and probability will not rescue a sample that size.
And it fails when it becomes the whole method. In systematic trading a test earns a rule the right to be watched live, and nothing beyond that.
The original data
The counts are lopsided. In research/broker-coverage.json, a scan of the 31,760 videos
in research/search-study-corpus.jsonl, 812 videos carry “backtest” in the title,
from 400 channels, at a median of 2,115 views and a maximum of 1,151,907. “Backtesting” adds 443
more from 232 channels, median 887.
The two things that decide whether any of it means anything are almost unwritten. “Overfitting” appears in 2 videos from 2 channels, median 299 views. “Walk forward” appears in one, at 3,796 views, and “forward test” in three, median 683. “Win rate” gets 211 videos from 130 channels.
The tool is taught exhaustively and the two questions that govern it are not taught at all. 812 videos on running a backtest, against 2 on overfitting and one on walk-forward testing. Whichever software a viewer picks, nothing they watched says how to read its output.
The bar ranges say why that matters. In research/series-measurements.json, from
site/measure_series.py, the median bar range is 0.493, the tenth percentile 0.17 and the ninetieth
1.101 — a ratio of 6.5. A fill assumption small against a median bar is large against the smallest,
at 0.022, and on quiet bars it decides the result.
Run the same rules twice under two different fill assumptions. If the answer changes materially, you have measured the assumption rather than the strategy.
Related
Backtesting is the method this software automates, and the errors it makes easy. Overfitting is the one an optimiser commits faster than any human could. And forward testing is the check no tester can run on itself.
The result I trusted most turned out to rest on a single line I never looked at. My exit was a stop, and the tester was filling it at exactly the level I had drawn, every time, in every condition. Once I made it fill a little worse than that, the method stopped working and I stopped trading it. Now I check the fill rule before I read the curve.
— Michael Whitman
This page is educational, not financial advice. Test every idea on your own charts before risking money.