Backtesting Software: The Data Decides
Backtesting software replays a set of trading rules over historical price data and reports what they would have returned. The tool itself matters less than three settings inside it: which data it uses, how it assumes orders were filled, and what costs it charges.
Covered on this page: TradingView.
How it works
Backtesting software replays a written rule over old price data and reports the outcome. You supply the entry, the exit, the size and the costs; it walks the bars and prints the account curve. The method it automates is backtesting.
Built-in testers, platforms and writing it yourself are the three categories. A strategy tester inside a charting platform, a scripted environment, and a spreadsheet or a bar-replay tool. Each trades convenience against control.
The three settings that decide the answer
And the data matters more than the software does. Two testers running identical rules on different price histories will disagree, and neither is wrong about its own data. Adjusted or unadjusted prices, missing sessions and bad ticks all move it.
How it assumes you were filled is the hidden setting. When a stop or a limit is touched inside a bar, the tester must guess whether you were filled, and at what price. Identical rules then give different answers.
It makes a beautiful curve far too easy to produce. An optimizer tries thousands of parameter sets in a minute and hands back the best. That is overfitting with a progress bar attached.
Dead companies missing from the data flatter every test. A universe built from today’s listed names excludes everything that was delisted, which is precisely the set that would have hurt.
And using a close you could not have known is the common bug. Deciding at a bar’s open on the basis of that same bar’s close is look-ahead bias. It is easy to write by accident and it produces flawless results.
In practice
A test without costs is not a test. Commission and the bid-ask spread come out of every round trip, and many testers default to neither.
It assumes liquidity that was not there. No tester models the case where volume was too thin to absorb your size.
More history is worth more than more settings. A slow method needs years to produce a sample worth reading, and cleaner data beats more tuning.
Most testers fill a gapped stop at its level. Price cannot trade through an opening gap, so the fill lands on the far side, worse than the level you set.
Which removes the largest risk from the record. Attaching a fixed pessimistic slippage to every stop loss fill is crude, and far closer to reality than assuming none.
Every round trip pays twice. Say a share has a 2-cent spread and a $1 commission each way: a 100-share round trip costs $4 before price moves at all. The more often a method trades, the more of its result is that figure.
Five questions to ask of any tester
What data, exactly? Which vendor, which sessions, whether prices are adjusted for splits and distributions, and whether delisted instruments are present. A tester that cannot answer has capped what its output is worth.
What is the fill rule, and what happens on a gap? Find the setting, read what it does, change it and run again. A tester that hides its fill logic is a simulator with an opinion you cannot inspect.
What costs are applied, and can it hold data back? Commission, spread and slippage should be inputs you set, not defaults you inherit. Reserving history the optimizer never sees is the bridge to forward testing and paper trading.
What backtesting software is not
It is not evidence about the future. It describes one history under assumptions you chose.
It is not a trading bot. Testing a rule and running it are separate jobs.
It is not a source of data quality. It reports faithfully on what it was fed.
And it is not the end of the process. The unseen period is the only honest test.
How this connects
The software is not the hard part. Specifying the rule is, and no package removes that.
What a package actually supplies is bookkeeping. It applies your rule bar by bar without hindsight, which is the thing a human reading a chart cannot honestly do.
Cost settings are where results are won or lost. Commission and slippage usually default to zero, and a zero-cost test flatters any strategy that trades often.
Speed is the danger. Re-running with a different parameter takes seconds, so dozens get tried and the survivor is the best-fitted rather than the best.
Forward testing is the check that survives all of this - the same rule, on data the fitting never saw, accepted as the real number even when it is worse.
When it fails
In a flat period it reports exactly what happened, and that is the problem. A trend rule tested across a trading range looks broken, and the same rule across a trending stretch looks superb. Neither is a property of the rule.
A daily bar hides the path taken inside the day. If a stop and a target both sit within one bar’s range, no tester can say which was reached first. It picks one by convention, and on those trades the convention is the strategy.
It fails silently on damaged data. A missing session or a bad tick produces trades that never existed, and nothing marks them out.
It fails when the sample is small and nobody says so. A curve built on twenty trades is a story about twenty events, and probability will not rescue a sample that size.
And it fails when it becomes the whole method. In systematic trading a test earns a rule the right to be watched live, and nothing beyond that.
The original data
The counts are lopsided. In a scan of the 24,971 videos in this site’s search study, 462 videos carry “backtest” or “backtesting” in the title, from 354 channels, at a median of 2,234 views and a maximum of 1,151,907. 273 of them say “backtesting”, at a median of 1,129.
The two things that decide whether any of it means anything are almost unwritten. “Overfitting” appears in 2 videos from 2 channels, median 299 views. “Walk forward” appears in one, at 3,796 views, and “forward test” in three, median 683. “Win rate” gets 169 videos from 136 channels.
The tool is taught exhaustively and the two questions that govern it are not taught at all. 462 videos on running a backtest, against 2 on overfitting and one on walk-forward testing. Whichever software a viewer picks, nothing they watched says how to read its output.
Bar width says why that matters. Suppose a fill rule gives away 5 cents a trade. Against a day that spans $2.00 that is 2.5% of the range; against a quiet $0.20 day it is 25%, and on quiet bars it decides the result.
Run the same rules twice under two different fill assumptions. If the answer changes materially, you have measured the assumption rather than the strategy.
Related
Backtesting is the method this software automates, and the errors it makes easy. Overfitting is the one an optimizer commits faster than any human could. And forward testing is the check no tester can run on itself.
A backtest result can rest on a single line nobody looks at: the fill rule. If the tester fills every stop at exactly the level drawn, in every condition, the curve is flattering you. Make it fill a little worse than that and see whether the method still works, and check the fill rule before you read the curve.
— Michael Whitman
This page is educational, not financial advice. Test every idea on your own charts before risking money. Some links on this page earn a commission if you buy through them. It costs you nothing and it does not decide what appears here or in what order — how these pages are made is set out in our methodology.