Overfitting

What is Overfitting?

A price series is part signal and part noise. Overfitting is what happens when a set of rules ends up describing the noise. In backtesting the mechanism is rarely a model with too many parameters; it is selection. You try a variant, you look at the result, you try another. Whichever variant you keep is, by construction, the best of everything you tried, and the best of many noisy results is biased upward even when none of them has a real edge. The search itself manufactures the number.

This is why overfitting is so easy to commit without noticing. Nudging a valuation threshold until the equity curve smooths out, adding a filter that removes the worst holdings, shortening the window because the early years "were a different market", switching the rebalance frequency because quarterly looked better than monthly: each step feels like judgment rather than optimization. Collectively they are a search over a space of strategies, and the number of attempts is what determines how inflated the winner's backtest is. Most people never count the attempts.

The signature of an overfit strategy is fragility. Move a price-to-earnings cutoff from 15 to 16 and the ten-year return halves. Shift the start date by one quarter and the ranking of your top holdings scrambles. A genuine effect tends to sit on a broad plateau, where nearby parameter choices give similar answers. An overfit one sits on a narrow spike, and the spike is exactly where the search left you standing.

Why Overfitting Matters in Backtesting

Overfitting corrupts every number downstream of it. The Sharpe ratio, the maximum drawdown, the win rate and the calendar-year table all inherit the selection bias of the strategy that produced them, so you cannot repair an overfit backtest by looking at more of its metrics. The decision at stake is whether the result in front of you is an estimate of the future or a summary of the past, and no further in-sample analysis can answer it.

The published evidence is uncomfortable. Bailey, Borwein, Lopez de Prado and Zhu, writing in the Notices of the American Mathematical Society in 2014, showed that a researcher who tries enough variants can produce an impressive in-sample Sharpe ratio from data with no signal in it at all, and that the number of trials required is small enough to reach in an ordinary afternoon of tinkering. Their conclusion: a backtest reported without the size of the search behind it is close to uninterpretable. Sullivan, Timmermann and White (1999) made a related point about technical trading rules that had looked convincing on the Dow, whose significance largely disappeared once the full set of rules searched was accounted for.

The failure mode is expensive and quiet. You allocate to a strategy whose true expected excess return is near zero, but whose trading costs, taxes and tracking error against a simple index fund are entirely real. The strategy does not blow up; it slowly fails to be worth the trouble, and by the time that is obvious you have paid years of friction for a pattern that was never there.

How SledgeKey Handles Overfitting

SledgeKey has no parameter optimizer, no automatic threshold search, and no button that hunts for the best-performing screen. That is a deliberate choice rather than a missing feature. An optimizer would put a thousand-trial search one click away and hand back the winner with no record of what it beat, which is the most efficient possible way to produce a beautiful and worthless backtest. Because the search happens at the speed of your own re-runs, the trial count stays visible to you.

The platform's defaults make overfit strategies pay for themselves. Transaction costs are applied on both sides of every trade with a default of 0.10% per side, so a rule that only works because it churns will show the churn in its return. Fundamentals are recorded against the date each filing became public, so an apparent edge cannot come from information the strategy would not have had. On the results page, calendar year returns and the rolling 12-month charts break performance apart period by period, which is where a strategy that owes its whole result to one lucky year gives itself away.

A practical protocol inside SledgeKey: keep the number of screening criteria small, because each one is another degree of freedom. Prefer round thresholds to precise ones, since a price-to-earnings cutoff of 14.7 is a confession that you searched. Check the neighbours of any threshold and make sure the result does not depend on the exact value. Then put the finished rules in front of data they have never seen, which is the subject of the companion article on out-of-sample testing.

Common Pitfalls

The first is mistaking complexity for rigor. A screen with nine criteria feels more careful than a screen with three, but every added filter is another dial you turned while watching the output, and a strategy with more dials will always fit the past better and predict the future worse. If a criterion cannot be defended before you look at what it does to the backtest, it does not belong there.

The second is believing you have not optimized because you never wrote an optimizer. Twenty manual iterations of a screen is a twenty-trial search, and it inflates the winner's result exactly as much as an automated one would. The bias comes from the number of things you looked at, not from whether a computer looked at them for you.

The third is the post-hoc economic story. "It makes sense that companies with low debt and high margins outperform" is easy to say after the backtest, and a plausible story can be constructed for almost any pattern a search turns up, including patterns that are pure noise. The story carries weight only when it existed before the test. If the rationale arrived after the result, treat it as decoration rather than evidence.

Related Concepts