Strategy Tester: The Optimizer Lies
A strategy tester is the testing environment built into a trading platform, which replays a strategy over historical data and reports the result. It also runs an optimizer, which ranks parameter combinations. The top row is not an edge; only a later untouched period tells you anything.
Covered on this page: TradingView.
How it works
A strategy tester is the testing environment built into a trading platform. It replays a strategy over stored history and produces a report and an equity curve — backtesting with a graphical interface. The term belongs to the MetaTrader platforms, whose tester also runs an optimizer.
Modeling quality is the first setting and it decides everything after it. The options run from the opening price of each bar, through interpolation within each bar, up to modeling every tick.
Only every tick resolves what happened inside a bar. A strategy with a stop loss and a target inside the same bar cannot be judged without it — the tester would simply assume which was touched first.
The optimizer and the forward test
The optimizer is the most dangerous button in it. It will search hundreds of parameter combinations and hand you the best one, and the best of hundreds of random results always looks like an edge.
A grid search will always find a winning setting, even in data with no edge in it. That is what searching means: it is a probability problem before it is a trading one.
So split the history and test forward on the half you hid. Optimize on the earlier period, then evaluate once on a later one the optimizer never saw — forward testing, run inside the tester.
Parameter stability is the other check, and it costs nothing. Read the rows either side of the winner: a lone peak is an accident, a broad plateau is a candidate.
And the headline number is the least useful line on the report. The drawdown, the trade count, the longest losing run and the spread of results carry the information.
In practice
Set the spread, commission and swap before any run. On many platforms the default bid-ask spread is not the one your account trades, and in forex it widens at the hours a fast strategy likes.
It cannot model a market too thin to fill you. The tester fills your order at the price it chose, whatever the volume on that bar was.
A longer sample beats a better optimization every time. More history means more regimes, and surviving several of them beats being tuned perfectly to one year.
Weekend gaps are where its assumptions break. An opening gap means the market never traded at the levels in between, so a resting order was never fillable.
And it will fill a stop the market jumped straight past. The record then shows a controlled loss rather than the larger one you would have taken.
Costs are not a rounding error at short holding periods. Say a strategy targets 20 cents a trade on a stock where spread and commission come to 3 cents a round trip: costs take 15% of every winning trade before a single loss is counted.
An honest run, in order
An honest run has a fixed order, and the order is the method. Set the costs to what your account actually charges, then set modeling quality to every tick and leave it there.
Split the history before you optimize, not after. Hold back the most recent third and do not look at it. Optimize on the older portion alone, and treat its table as a list of suspects.
Check stability next, because it is free. If a small change to a length or a threshold turns profit into loss, the setting is fitted to noise.
Then evaluate once on the untouched period, and once only. Re-optimizing because the forward result disappointed turns that period into training data. Systematic trading lives on that discipline.
What a strategy tester is not
It is not a prediction. It describes one history under assumptions you chose.
It is not proof of an edge. The top row of an optimization table is a selection, not a finding.
It is not a live account. Slippage, requotes, outages and partial fills all sit outside it.
And it is not automated trading. Running an expert advisor in the tester is rehearsal, not trading.
How this connects
A strategy tester turns a rule into a number, which is the only honest way to argue about one. An idea that cannot be written down cannot be tested.
It needs a rule with no ambiguity. Entry, exit, position size and invalidation all have to be specified, which is itself useful - most discretionary ideas fall apart at exactly this step.
Its output depends entirely on costs you set. Commission and slippage default to zero, and a backtest at zero cost is a test of a market nobody trades in. The bid-ask spread is paid on every round trip whether the test counts it or not.
It lives inside TradingView and runs on Pine Script, so the data, timeframes and instrument coverage are the platform’s.
And a result is not evidence until it survives data it was not built on. Otherwise the test has described the past rather than examined it.
Overfitting, which is what usually goes wrong
Speed is the danger, not the arithmetic. A parameter can be changed and the test rerun in seconds, so dozens of variants get tried in an evening.
The one that survives is the one that best describes that sample. Not the best rule - the best-fitted one, and those look identical on a results screen.
The tell is sensitivity. A rule that works at a 14-period lookback and collapses at 13 and 15 has found an accident. A real effect degrades gently as you move around it.
Splitting the data is the cheapest defense available. Build on one period, test on another you have not looked at, and accept the second number as the real one.
And record the tests you discarded. Twenty variants tried and one kept is a very different claim from one variant tried and kept, but the equity curve looks the same either way - only your own notes preserve the difference.
When it fails
In a trading range the optimizer simply finds a range setting. Every parameter it picks is tuned to whichever regime dominated the sample, so the verdict describes the period, not the strategy.
The second failure is a short history. A few hundred bars holds too few independent events to mean much, and the optimizer will fit all of them.
A third is the broker’s own price data. The tester uses whatever history the platform supplies — short, patched across feeds, or quoted on a spread nobody traded.
A fourth is a strategy that trades rarely. Twenty trades cannot separate method from luck, and optimizing over twenty trades is fitting noise with a progress bar.
A fifth is re-running it after seeing the forward result. That single act converts test data into training data, and nothing short of new history undoes it.
And the sixth is generous fill assumptions. Limits that fill on a touch, stops that fill at their level and no slippage turn a losing method into a profitable one on paper.
The original data
Five videos carry “strategy tester” in a title, at a median of 52,868 views across four channels and a maximum of 228,801. “Expert advisor” returns seven at a median of 11,637, “backtest” or “backtesting” returns 462 at a median of 2,234 across 354 channels, “overfitting” two at 299, and “walk forward” one, at 3,796.
Those counts come from the search study behind this site, a scan of the 24,971 trading and investing videos collected for this site’s search study. Set the tester’s cost fields before anything else, because a result computed without them describes a market nobody can trade.
The question: The best of nine hundred runs. An edge?
That median is the highest of any testing-related term here, roughly twenty-four times the 462 backtesting videos. Very few people teach the tester and a great many want it taught. One video in the whole corpus covers walk-forward testing, the technique that makes an optimizer result mean anything.
The cost point is smaller on the page than it is in practice. The spread is one field in a dialogue box, and here a round trip is 45% of the smallest bar, so leaving it on a default decides the run. Set the real costs before the first run, and judge a strategy by the untouched period, never by the best row in the table.
Related
Backtesting software is the wider category, and a dedicated package gives more control over data quality and costs. Overfitting is the failure the optimizer produces by construction, which is why the top row means so little. An expert advisor is what the MetaTrader tester runs, so its code limits what a report can show.
I look at the neighbors of a result before I look at the result. A row that stands out after an overnight optimization is usually the luckiest combination in the search rather than a rule, and it falls apart on the months you held back. If the settings either side of it lose money, you have found noise.
— Michael Whitman
This page is educational, not financial advice. Test every idea on your own charts before risking money. Some links on this page earn a commission if you buy through them. It costs you nothing and it does not decide what appears here or in what order — how these pages are made is set out in our methodology.