Amulya HoldingsResearch
Why most backtests look better than reality
A backtest answers one question: if I had followed these exact rules in the past, what would have happened? It is a useful tool, and every systematic strategy starts with one. But a backtest is built by someone who already knows how the past turned out. That hindsight leaks in, and the result almost always looks better than what you will get with real money.
Here are the four most common reasons, and what to ask before you trust any backtest, including ours.
1. Overfitting: finding patterns in noise
Suppose you test a moving-average rule with 20 possible lengths, 10 stop-loss levels and 5 profit targets. That is 1,000 combinations. Even if none of them has a real edge, a few will look excellent on past data purely by chance. Pick the best one and you have not found an edge; you have found the luckiest fit to history.
Warning signs of overfitting:
- Very specific settings (a 37-day average, a 1.83% stop) that only work in a narrow range.
- Results that collapse when a setting moves slightly.
- Many rules and filters added one by one to remove bad trades.
- Great results in one period and poor results in the next.
2. Costs and slippage left out
In Indian markets every trade pays brokerage, Securities Transaction Tax (STT), exchange charges, SEBI fees, GST and stamp duty. On top of that is slippage: the gap between the price your model assumed and the price you actually got.
For a slow strategy that trades a few times a month, costs are a small drag. For an intraday options strategy that trades every day they can decide everything. Illustration with round numbers: if a rule earns 6 points per trade before costs and costs plus slippage take 4 points, two-thirds of the edge is gone. If costs are underestimated by just 2 points, a "profitable" strategy is actually breaking even.
A credible backtest uses real bid and ask prices where possible and charges the full statutory costs on every trade.
3. Look-ahead and survivorship bias
Look-ahead bias means the test uses information that was not available at the moment of the decision. A classic example is deciding to buy "at the close" using a signal that needs the closing price, which you only know once the market has closed.
Survivorship bias means testing only on companies that exist today. Stocks that were delisted, merged or collapsed drop out of the data, so the past looks kinder than it was. Testing a "Nifty stocks" strategy using today's Nifty list is a common version of this mistake.
4. Too few trades
Thirty trades can look convincing and still be luck. The fewer the trades, the wider the range of outcomes that a strategy with no edge could produce. A larger sample spread across different market conditions, rising, falling and sideways, is far more informative than a short, spectacular run.
The tests that catch these problems
- Out-of-sample testing. Build the rule on one period and test it, unchanged, on a later period it has never seen.
- Full costs from day one. Every trade pays realistic charges and slippage.
- Robustness checks. Nearby settings should give similar results; randomly dropping or shuffling trades should not flip a winner into a loser.
- An independent rebuild. Someone else codes the rule from its written description. Hidden errors and wishful assumptions show up quickly.
- Live validation. Paper trading, then small real trading, until live behaviour matches the research.
These are the five gates we use at Amulya Holdings. Most ideas fail one of them, and that is the point.
Questions to ask about any backtest
- Are all costs and slippage included? What figures were assumed?
- How was the period split between building and testing?
- How many trades, over how many years and market conditions?
- What was the worst drawdown, and how long did it last?
- Is there a live or paper record that matches the backtest?
- Is the provider registered with SEBI, and is any performance shown verified as SEBI requires?
Quick answers
Is a good backtest a guarantee of future returns?
No. A backtest shows how rules would have behaved in the past under the assumptions used. Markets change, and live trading adds costs and frictions a backtest can miss. Treat it as evidence to be tested further, never as a promise.
What is out-of-sample testing?
Out-of-sample testing means designing a trading rule on one period of data and then testing it, without any changes, on a different period it has never seen. A rule that only works on the data it was built on is probably overfitted.
How many trades does a backtest need?
There is no single number, but more is better. A few dozen trades can easily be luck. Hundreds of trades across rising, falling and sideways markets give much stronger evidence.
Education only. This article explains general concepts. It is not research, a recommendation or investment advice, and it does not refer to any security. Amulya Holdings is not yet registered with SEBI. Investments in the securities market are subject to market risks.
Get one email when we launch
Rule-based research for Indian markets, once our SEBI registration is in place.
