Overfitting: Describing the Past
Overfitting is a result that fits the particular history it was tested on, including that history's random noise, rather than any repeatable relationship. Each adjustable parameter, and each variant tried, adds another way to fit the past. A fitted system did not stop working; it never worked.
How it works
Overfitting is a result that describes the past and nothing else. A system is tuned until it fits one stretch of history, noise included, rather than anything repeatable. It never worked; it did not stop working.
Every parameter is a dial, and dials get turned. A length, a threshold, a stop loss distance: each is a degree of freedom, another way to bend a system around its data.
Turn enough of them and the old curve looks perfect. With enough degrees of freedom and enough attempts, some combination fits almost any dataset, including pure noise.
Because you fitted the noise, not the signal. Price history holds a little structure and a great deal of accident. Optimization cannot tell them apart.
Which is why it dies the moment it meets new data. The accidents do not recur, so the fitted trading rules have nothing to work on. The drawdown arrives at once.
Counting the ways you can fit
Three parameters are far harder to fit than twelve. The number of dials sets how closely a history can be traced. Fewer means a worse tested result and a more believable one.
And a rule added to fix a bad year is the classic case. It always improves the backtesting result, because it was derived from the outcome it removes. It rarely improves the future.
Costs left out of a test hide the problem completely. Omitted commission, look-ahead bias (using information the moment did not have) and survivorship bias (testing only the survivors) all flatter the result.
More inputs mean more ways to fit the same history. Adding volume filters, correlation screens or session windows feels like rigor. Each is another dial.
A slow system has too few trades to judge at all. A rule trading twice a year yields a few outcomes per decade, too few to separate skill from luck in any backtesting software.
In practice
And a backtest fills a gapped stop that never filled. An opening gap jumps past the level, yet the strategy tester books the exit at the price you asked for.
Move the stop until the test improves and you have done it. Stop distance is a dial like any other. Sliding it through twenty values and keeping the best is twenty tests.
Every round trip pays a spread. On a hypothetical $20 stock quoted a cent wide that is 0.05% of the position on every trade the tuned system takes, and a backtest that leaves it out flatters the fastest settings most.
The tests that help are the ones you cannot tune. Hold a block of data back and never touch it. Run forward testing on unseen periods, and prefer fewer parameters.
Stability is the most useful single check. A system that works at one setting and fails at the settings either side of it has found an accident. A genuine relationship degrades gradually.
Trades per parameter
The arithmetic is simple and unforgiving. Every parameter you fit consumes information from the sample it was fitted on, so the real question is how much sample there is per dial. Sixty trades and twelve parameters leaves five outcomes behind each decision.
Five outcomes cannot separate an effect from a run of luck. The same twelve parameters against several thousand trades is a different proposition, because no single accident can move that result far.
Treat the ratio as a gate before you read the result. Count the trades, count the dials, and count the variants you tried. Where that figure is small, the number in front of you is work for probability, not intuition.
It is the quiet cause behind much of why traders lose money. The sample was always too thin for the confidence placed on it.
What overfitting is not
- Not a system that decayed. A decayed system worked first; a fitted one never did.
- Not simply a losing system. Plenty of honest systems lose without any fitting involved.
- Not the same as optimization. Choosing sensible values is normal; searching until the past is perfect is not.
- Not fixed by more history. More data helps only if you stop tuning against it.
How this connects
It is the central hazard in quantitative analysis. The more variations a researcher can test cheaply, the more certain it becomes that one of them looks excellent by chance.
Indicator settings are the most common dials. Choosing a moving average length or an RSI level because it scored best on one history is fitting, however reasonable the number looks on its own.
Automation hides it well. An algo trading system or a trading bot executes the fitted rules faithfully, so nothing in its behavior signals that those rules describe only the past.
A flattering Sharpe ratio is often the symptom rather than the proof. Smooth historical returns with short losing stretches are exactly what fitting the noise tends to produce.
Forward evidence is the honest check. Paper trading, followed by a trading journal of small live results, measures trading expectancy on data the system never saw.
And the damage arrives through sizing. A trader who trusts a fitted edge sizes positions for it, which raises the risk of ruin in exactly the situation where the edge does not exist.
When it fails
The diagnosis can be wrong in both directions. These are the cases where the label misleads:
- A quiet trading range punishes a sound system. A flat period flattens an honest edge too, and calling that overfitting throws away something that works.
- A short live sample proves nothing either. Twenty live trades cannot confirm a failure any more than sixty backtested trades confirmed a success.
- Fitting hides inside simple systems. Two parameters chosen after four hundred attempts is a heavily searched result wearing a modest disguise.
- Held-back data stops being held back once you use it. Look at your reserved block a second time and it has quietly joined the training sample.
- Some markets genuinely change. Rule changes, new venues and shifting participants can end a real edge, and that is decay rather than fitting.
- A result can be honest and still useless. A stable, few-parameter system with a thin edge is not fitted; it simply does not survive costs.
The original data
The question: Sixty trades, twelve parameters. Trust it?
The site’s video scan shows the imbalance. Across the 24,971 trading and investing videos in the study corpus, 462 titles mention “backtest” or “backtesting” and 2 mention “overfitting”. “Walk forward” appears once and “forward test” three times, at a median of 683 views. “Expectancy”, “risk of ruin” and “monte carlo” appear zero times each.
The audience numbers widen the gap. Those two overfitting videos come from 2 channels, median 299 views, maximum 515, against a median of 15,735 views for the 169 videos about win rate.
So the failure that invalidates a backtest is covered in roughly one video in four hundred. The published curriculum teaches how to produce a backtest, and almost nothing about how one lies to you.
The S&P 500 base rate shows what a tuned result has to beat. From 1950 to 2026, 59% of the index’s ten-day windows finished higher, so a long-only rule tuned until 59% of its ten-day holds win may have fitted the drift rather than found anything.
A few dozen events is already thin, and a system tuned on a sample that thin is describing a few dozen accidents. Before your next optimization run, write the test down first, record how many variants you tried, and reserve a block of data you will look at exactly once.
Related
Backtesting is the process this failure contaminates, which is why a strong tested result deserves suspicion. Forward testing runs on data your tuning could not reach, which is what makes it worth more than any in-sample number. Systematic trading is where the problem lives, because no discretionary trader runs two hundred variants overnight.
I stop tuning when the ugly stretches of an equity curve start smoothing out, because that is the point where a system turns into a description of the sample. A model that tests beautifully across years of history and behaves like a different thing live is not unlucky, it is fitted. The tuning is the problem, not the fix.
— Michael Whitman
This page is educational, not financial advice. Test every idea on your own charts before risking money.