One of the most difficult problems with creating predictive models in financial markets is to find a system that will have a high accuracy in different market conditions. The identification of the transition between different market regimes will create several contradictory problems, when using machine learning.
1) Over fitting guaranteed
The machine learning classifier will almost for sure start to memorize the connections between features and targets instead of finding the general relations between the two if the data is not split into two set, one for training and one for testing. This is a standard procedure and called out of sample testing.
2) Not testing on all regimes
If the test time period with out of sample testing is not sufficient long enough, the window in time may not cover all different market regimes.
3) Missing a long period of latest market info
If the test time period with out of sample testing is long enough to cover all market regimes, and that untested period is not included in final classifier, the predictive model will miss a lot of information about the connections between features and targets would make up the rules for that predictive model.
4) Not trying all combinations of market regime
A standard walk forward approach will cover must market regimes with the test periods, but every period of training data may not be tested against every combination of market regime.
5) Using future info on past data
By using the data from the period closest in time for testing, we make sure that “market behavior” from the future is not influencing the test period from the past. This condition will not be met by using a standard cross validation procedure.
6) Mixing the time series
Time series data need to be separated based on point in time, or else different market regime will be randomly mixed together, making it impossible to know how well the predictive model are performing in each separate market condition. Also the “market behavior” from the future may be influencing the test period from the past.
Hire a freelance quant, contact me at: mikael[at]furesjo.se