The biggest challenge that any trader or investor faces is building models that not only work well in hindsight, but into the future as well. This is a challenge to any kind of model, including investing on fundamentals, because after all, even fundamental investing is about making decisions about historical patterns we have observed. Even Warren Buffett has the problem, because stocks chosen using old values may not act in today's environment as they would have in the past.
In NeuroShell Trader, one solution was often to reduce the number of "free variables" in the model, so that the model does not actually learn the "noise" instead of the underlying patterns. Neural nets typically try to use in some way all of the information they are given. The ChaosHunter does not suffer anywhere nearly as much from the problem of free variables, because it naturally tends to keep only a few variables, even if you have presented it with too many inputs. So in ChaosHunter that is less of a worry, but in general, we would give ChaosHunter only what we feel are the most likely leading indicators so it doesn't waste time sifting through the chaff to get to the wheat. ChaosHunter also does not suffer from the problem of overoptimization, in our opinion, given that you have enough data as described below.
The major factor in making models that last with NeuroShell Trader, ChaosHunter or anything else right down to discretionary trading is to develop the model or strategy based upon a large, diverse set of relevant patterns from the past. It is also important to allow plenty of future ("out-of-sample") time to evaluate how well the model held up. After all, Warren Buffett would not assume that the patterns he is using to pick stocks is a failure after say only a few weeks. So let's examine what "a large, diverse set of relevant patterns" means.
1. A "large set" is one with say at least 1200 bars, and more is better. If you build a model with less data there are many ways to make profitable models from it, not all of which are robust enough to continue working. As an extreme example consider how foolish it would be if you made all future trading decisions based only upon what the market did today.
2. A "diverse set" is one that has many various types of patterns, so that the formula ChaosHunter produces will have considered many situations that could occur in the future. We would then call the formula "robust". Now a large set will enhance the probability of getting diverse patterns captured, but will not guarantee it. So you should manually examine the price curves in your set to make sure it shows lots of rising, falling, and volatile patterns.
3. A "relevant set" is one that contains the patterns that are most likely to occur in the future, as opposed to ancient patterns no longer likely to recur. In the mid nineties you could have purchased almost any Internet stock and made a fortune, but very few would advise that pattern of buying today. Of course, deciding what types of patterns in prices are relevant is very difficult. It might be easier to describe what patterns are NOT likely to be relevant. That would include steep bull market price gains, and post bubble burst bear patterns. As this tip is written on September 4, 2008, we appear to be in a somewhat volatile market not likely to take off in either direction without some powerful news event. So if we were building models with daily bars, we would do one of two things:
a. Start our data in early 2003 after the strong bull and bear markets, or
b. Start our data around 1997 so we pick up not only the bear market, but the bull market as well.
If you are building intra-day models, make a similar analysis without going back so far in time.
At the end of the day we cannot give you an exact cookbook of how to choose your data periods, because like all of trading, it is more an art than a science. Experiment and come to your own conclusions about what is best for the stocks or other issues you are dealing with. Keep in mind that not every issue is always predictable (has repeating patterns). Sometimes stocks and markets change based on news, and totally new patterns will appear.