"What can I do about inconsistent out-of-sample performance?"
By Dave Mabe
Here's a question from long-time list member, Steve K. (name used with permission, lightly edited for brevity):
Steve K:
Have you ever run into live trading results that vary widely from optimization?
I'm wondering if I might be curve-fitting. Here's what I've done. I optimized around 1,000 trades over 15 months of data. Equity curve looks very nice, but applying it out-of-sample was basically flat.
Another test using 4,000 trades across 3 years. Out-of-sample? Flat again.
My only conclusion is that I have to backtest a very large amount (10 years? 15 years) of trade data before optimizing. What do you think?
Dave:
One important fact about Steve's strategy that he doesn't mention: his strategy is trading a single futures contract.
Why is that so important?
To create a trade set large enough by backtesting a single instrument, by definition, you're going to have to water down your trade signal.
I have no idea what the signal is, but I'm skeptical that you can find a strong enough edge for your starting point with 1,000 trades across 15 months - from a single instrument.
Apply the same signal across the entire equities market, though, and you'll have more trades than you can handle!
Your starting strategy (pre-optimization) is important.
What Steve's running into is the fundamental reason I prefer applying strategies across equities - it's just easier to create a starting point with a strong signal.
So my advice for Steve:
Take a step back and consider your trade signal
How can you modify it to have more strength?
Adding strength will reduce the number of trades, so you'll need more history
This should get you a stronger starting point, which will make optimizing easier.
Thanks for sharing with the list, Steve, and let us know how it goes with these changes.
-Dave
P.S. Are you sick of second-guessing yourself every time you click buy or sell? Trade with conviction using MabeKit.