Out-of-Sample Testing: One Rule, Three Flipped Verdicts
Out-of-sample testing checks a trading rule on data that played no part in building it: the history is split, the rule is designed on the first part and judged once on the rest. Split at the end of 2020, one RSI(2) rule flipped its pass-or-fail verdict on all three stocks tested.
Out-of-sample testing is the discipline of keeping some history back. The rule is built and tuned on one stretch of data, then run once, unchanged, on a stretch it has never seen, and only that second result is treated as evidence.
How it works
The data is split into two parts before any testing starts. The first part, the in-sample or build period, is where the rule is designed: its settings chosen, its filters added, its bad ideas thrown out. The second part, the out-of-sample or check period, stays untouched until the rule is final.
Then the finished rule runs on the check period exactly once. If it still clears the bar it was built to clear, that is modest evidence the rule found something real. If it falls apart, the build result was probably describing the past rather than a repeatable pattern.
The reason it works is simple. Every decision made while building a rule is made while looking at the build data, so the build result is partly a record of those decisions. The check data had no vote, so it cannot flatter them.
This differs from forward testing, which waits for new data to arrive in real time. A holdout check uses history that already exists, so it takes seconds rather than months, but it relies entirely on the tester not peeking.
Choosing where to cut
A time split is the usual choice for trading rules. Build on the earlier years, check on the later ones, so the test runs in the same direction as real trading. Mixing random days into each half lets the rule learn from the future of its own check period.
The cut has to leave both halves big enough to judge. A rule that trades a few times a year needs many years on each side, and a split that looks tidy on a calendar can leave the check half too small to say anything.
It should also be chosen blind. Picking 2020 because the chart shows a regime change there is already a decision made with knowledge of both halves.
A worked example
Start with a hypothetical, to show why the build result cannot be trusted alone. Say 20 slightly different settings are tried, and none has a real edge, but each has a 5% chance of passing on luck.
The chance that at least one passes is 1 − 0.95 raised to the 20th power, which is about 64%. So nearly two times in three, the search hands back a “winner” with nothing behind it.
Now run that winner on the check half. Under the same assumption, a luck-only rule passes there 5% of the time, so the check catches the fluke about 95% of the time. That is the whole value of the method in one line of arithmetic, and it assumes the check half is used once.
Then a real case: Microsoft in the build half. From January 2015 to December 2020 the RSI(2) rule took 36 trades on MSFT, and 26 closed higher, which is 72.22%.
The profit factor was 5.141, meaning winners made about $5.14 for every $1.00 losers lost. Against the test’s three bars (60% of trades higher, profit factor 1.3, at least 30 trades), it passed all three easily.
In the check half, January 2021 to August 2026, it took 27 trades and 16 closed higher: 59.26%, with a profit factor of 2.161. That misses two bars, the 60% share and the 30-trade minimum. Nothing in the code changed between the two runs.
What a check half can and cannot say
It can reject. A rule that fails badly on untouched data has shown that its build result did not carry forward, and no amount of explanation changes that.
It cannot certify. A pass on one check period is one draw. It says the rule survived one stretch of later market conditions, not that it will survive the next one.
And it cannot separate the rule from the market it ran in. A dip-buying rule tested in two rising periods will look stable because both periods rose.
The original data
The split was run in August 2026 on one rule and three stocks: RSI(2) under 10, above the 200-day average, on AAPL, MSFT and NVDA, daily bars, same code and costs in both halves. It was a stability check rather than a full walk-forward, and it is the only out-of-sample run in the whole set of swing backtests.
All three verdicts flipped. Apple failed the build half with 58.33% of 24 trades higher, then passed the check half with 63.16% of 38. Microsoft passed, then failed. Nvidia failed the build half on sample size alone, 75.00% of 24 trades higher, then passed with 73.33% of 30.
So the choice made in 2020 would have been exactly wrong. A trader using the build half to pick stocks would have chosen Microsoft alone. Microsoft was the one stock that failed afterwards, and the two stocks left behind were the two that passed.
The flips are smaller than they sound, and that matters. All six runs made money, from +8.18% on Apple’s build half to +86.42% on Nvidia’s check half. The profit factor stayed above 1.3 in every run. What changed was mostly the share of winners and the count of trades, sitting just either side of the thresholds.
Sample size did most of the damage. A half holds 24 to 38 trades, against 55 to 70 for the same rule over the whole window, and every one of the three failures fell under the 30-trade floor. Two of them also fell under 60%.
The halves also lose a few trades at the join. Together they hold 62 trades on Apple against 70 in the full run, 63 against 66 on Microsoft and 54 against 55 on Nvidia.
A trade open across the cut and the 200-day average restarting in each half are the likely reasons; the results file does not record which. Both tables are published: the out-of-sample split and the full-window results. These are simulated past results, not a forecast.
When it fails
It fails the moment the check half is used twice. Run the rule, see a weak check result, adjust a setting and run it again, and the check period has become build data. After a handful of rounds it offers no protection at all, and the final pass means about as much as the original in-sample one.
It fails when the split halves an already small sample. Nvidia’s build half shows how a rule can fail a trade-count floor purely because the window got shorter: 75.00% of trades higher, but only 24 of them.
A threshold chosen for eleven years of data is arguably too strict for five and a half, and loosening it after seeing the result is its own form of peeking.
It fails when both halves share one market. If the build and check periods are both long rises, a long-only rule can pass both without having been tested against a falling or flat market.
It fails when the list of stocks was already chosen with hindsight. A clean time split on a list of the decade’s winners still carries the survivorship problem in full. The split protects against tuning, not against a biased universe.
And it fails as a single number. One check period is one sample of the future. Rolling the split forward through several periods, known as walk-forward testing, gives several, which is why a single holdout is a minimum rather than a finish line.
Related
Overfitting is the error this method is designed to expose, and it explains why a rule tuned on the past can describe that past perfectly. Backtesting covers the full set of mistakes a historical test can make, including the list-selection problem a time split does not fix.
Forward testing is the slower, stricter version of the same idea, recording signals before the outcome exists. And the RSI(2) strategy page describes the rule used in this split, with its results on twelve stocks. The split is a teaching example.
Pick the split date and write it down before running anything, then treat the later stretch as sealed. The first time a disappointing check result leads to one more tweak, the check half has quietly become part of the build, and a fresh stretch of data is needed.
— Michael Whitman
This page is educational, not financial advice. Test every idea on your own charts before risking money.