"Which robustness tests are best?"
By Dave Mabe
Here's a question from Luca L. (name used with permission, lightly edited for clarity):
Luca L.
Over your 30-year experience, what robustness test(s) have you found to be most useful for evaluating trading strategies?
Dave:
There's always another robustness test you can run. t-tests, Monte Carlo, walk forward, walk forward with more folds, etc.
And there's always a stats nerd that will explain in excruciating detail how your strategy is curve-fitted and can't be trusted.
A few years ago, I shared one of my best-performing strategies, which has been trading live for 15 years in a collaboration with one of the best stats guys in the world.
(I didn't reveal that this strategy was already producing steady profits in live trading.)
He looked at it, did some analysis, and then gave me umpteen reasons why it wouldn't work in live trading.
Hmmm.
I learned in that moment that there's always a robustness test that will show you your strategy is garbage.
And, importantly, you always sound smart and prudent when you say a strategy is curve-fitted.
But that's a great way to stay stuck in backtest world... forever.
Here's the thing:
There's no checkbox you have to check when you wire your profits out of your trading account to certify that your strategy wasn't curve-fitted.
Do you want to be statistically perfect, or do you want to make money?
OK, now back to the original question.
The best "robustness test" is... looking at the equity curve.
And you're not just looking at the right edge and where the curve ends.
You're looking at the path it took to get there.
That describes, in a single snapshot, the emotional pain you'll have to endure.
What it will actually feel like to trade the strategy live.
-Dave
P.S. Do you wish you had a column library that would tell YOU how to make your strategy profitable? My column library is now included with MabeKit Get Instant Access to MabeKit