What is Backtesting?

Backtesting is a way to test if something would have worked in the past. People use it a lot with money and investments.

There are two main things people backtest:

1. Value-at-risk (VAR) models
2. Investment strategies

Backtesting Value-at-Risk Models

Value-at-risk models try to predict how much money you could lose. Banks and financial companies use them a lot. They want to know the worst-case scenario.

To backtest a VAR model, you take the predictions it would have made in the past. Then you compare them to what actually happened. This tells you if the model is any good at predicting risk.

Here’s how it works:

1. You run the VAR model on past data, as if you didn’t know the future. The model predicts the most money you could lose.
2. Then you look at what really happened in the past. How much money did you actually make or lose?
3. If the losses were bigger than what the model predicted, that’s bad. It means the model thinks there’s less risk than there really is.
4. If the losses were smaller than the model’s worst-case guess, that’s good. The model is predicting the risk pretty well.
5. You do this day after day, trade after trade. Over time, you can see if the VAR model usually gets it right or wrong.

This is important because you don’t want surprises. If you think your risk is small but it’s actually big, you could lose a lot of money. Backtesting helps you trust your risk model before using it for real.

Backtesting Investment Strategies

Investors and traders also use backtesting. They want to test their strategies.

An investment strategy is a set of rules for when to buy and sell. For example:

• Buy a stock when its price goes up 5%
• Sell a stock when its price drops 10%
• Hold 60% stocks and 40% bonds
• Rebalance your portfolio every 3 months

To backtest, you pretend to follow those rules in the past. You look at the price history and see when your rules would tell you to buy or sell.

This creates a “what if” scenario. What if you had followed this exact strategy 10 years ago? How much money would you have made or lost?

The steps are:

1. Get historical price data as far back as you can
2. Write code to simulate your trading rules
3. Run the simulation on the old data
4. Measure the profits or losses your strategy would have made
5. Analyze the results to see if the strategy is good

This is useful because you can test many different strategies very quickly. You can optimize and improve your strategy before risking real money.

Why Backtesting Matters

Backtesting matters because it’s a way to learn from the past. We can’t predict the future, but we can study what worked and didn’t work before.

Better than “Forward Testing”

The opposite of backtesting is just going forward with an idea. You think it will work, so you try it from now on.

This can be very risky. What if your idea only works in your head? When you try it for real, you might lose money.

Backtesting gives you more information first. You can test your ideas on a lot of past data and scenarios. If it worked well in many past situations, it’s more likely to keep working.

Forward testing takes a long time. You have to wait for results. But with backtesting, you can test a 10-year strategy in minutes.

Finding the Flaws

Backtesting helps you find the flaws in your strategies and models. Nothing is perfect. But it’s better to find the flaws before you risk money.

Maybe your investment strategy works great in a bull market. But backtesting shows it fails terribly in a bear market. That’s good to know. You can’t predict bear markets, but now you know you need a plan for them.

Or maybe your risk model predicts small losses. But the backtest shows some huge losses happened that the model missed. Now you know your model is too optimistic. You need to adjust it.

Backtesting is a form of feedback. It’s like when you’re practicing a sport and your coach points out what you’re doing wrong. Painful in the moment, but it helps you get better.

Building Confidence

Once you’ve found the flaws, fixed them, and backtested again, you build confidence. Maybe this strategy that sounded good in your head actually does work.

Backtesting never guarantees your strategy will work in the future. The past doesn’t predict the future. Things change. But it’s a lot better than going in blind.

With a well-backtested strategy, you can put money behind it more confidently. You’re not just guessing and hoping.

Limitations of Backtesting

Backtesting isn’t perfect. It has some important limitations to keep in mind.

Overfitting

A common problem is called overfitting. This means you make your strategy too specific to the past.

For example, say you’re backtesting a strategy for trading oil futures. You find some exact rules that would have made a lot of money for the last 20 years.

But those rules might only work because of what happened in those specific 20 years. Maybe there was a war, or a natural disaster, or a new technology. Your rules fit those exact events perfectly.

But the future won’t have those same events. Your overfit strategy might fail when the future is different than the past.

The solution is to test on a lot of different time periods. Don’t optimize for just one historical scenario. Find rules that work across many different scenarios, even if they’re not perfect for any one scenario.

Past Performance vs Future Results

Backtesting can lull you into a false sense of security. If a strategy worked for the last 50 years, you might assume it will keep working.

But the world changes. Markets change. The strategies that worked in the 1970s might not work in the 2020s.

Backtesting is backward-looking by definition. It doesn’t account for what might change in the future.

Some changes that can break backtested strategies:

• New regulations
• New technologies
• Changes in market structure (like the rise of high-frequency trading)
• Shifts in the global economy
• Demographic changes

The further back you backtest, the more the world has changed since then. An strategy that worked in 1920 might be irrelevant today.

Data Mining Bias

Another problem is data mining bias. With enough backtesting, you can find a strategy that works on any historical data. But it will be a fluke.

Imagine you have a coin flipping strategy. You backtest thousands of different rule sets. Finally, you find the perfect set of rules. It would have turned $1000 into $1 million, if you had used it for the last 20 years of coin flips.

But that’s just random chance. Your rules fit the noise in that specific set of coin flips. They won’t predict future coin flips.

The same thing can happen with backtesting. If you test enough strategies, some will work by pure luck. But that luck won’t continue in the future.

To avoid this, you need to:

1. Have a good reason for your strategy, not just mining for any strategy that works
2. Use out-of-sample testing (test on data you didn’t use to create the strategy)
3. See if the strategy works in different time periods and market conditions