Equity Research Products: • Internet-Integrated • PC-Based • Web-Based Equity Analysis Data: • Current • Historical Data Delivery Options: • Dial-Up • CD-ROM • FTP/Internet Backtesting • Backtesting Guide • Glossary • Stump The Quant! Subscription Info

Backtesting Guide Backtesting is a historical trading simulation. You can optimize a stock selection strategy using historical data and then determine whether it can performed as tested currently. You can see how the stocks performed over time when subjected to your current selection strategies and compare how your strategy performed when markets were up and how it performed when markets were down. *** It may be an inescapable aspect of human nature that the first inclination of Zacks Quantitative Analysis software user is to begin with the Trading Strategy Evaluator module. In fact, this option should be the last aspect of the research effort. There are at least three steps should precede the use of the TSE - an understanding of the possible limitations and biases of the data-selection process, a working knowledge of other Zacks modules utilized in the preliminary stages of the backtesting, and a reasoned theoretical model of the source or sources of superior equity performance. Data Limitations and Biases There are numerous articles in professional journals describing the concept of backtesting and its limitations.1 While a full discussion of these issues is well beyond the scope of this guide, two issues are of particular significance to the database users. Look-Ahead Bias: This data bias refers to the use of data in a backtest that was not, in reality, available to the analyst at the tine the backtest assumes. For example, suppose that your model is based, in part, on 12-month P/E ratios below a certain value. In the historical database, the 10/31/95 value for a company with a December fiscal year end may be subject to look-ahead bias. The 10/31/95 value is the 10/31/95 price divided by the sum of the 9/30/95, 6/30/95, 3/31/95, and 12/30/94 EPS values. The ratio should be based on earnings information available as of 10/31/95. Since many December-fiscal-end companies may not have announced third quarter earnings on or before October 31, 1995, this implies that the 10/31/95 ratio should be calculated using the 6/30/95, 3/31/95, 12/31/94, and 9/30/94 EPS values. Zacks software can alleviate this problem thorough its ability to lead and lag variables. In addition, leading and lagging, when appropriately used, can test the sensitivity of a strategy to the vital issues of information availability. Survivor Bias: This bias refers ti strategies that recommend stocks on the basis if a criterion associated, is some statistically significant sense, with a greater-than-average tendency to fail through bankruptcy or liquidation. Zacks users can, by utilizing Zacks backtesting databases, avoid this problem to a large extent (Zacks backtesting databases contain both the currently active and research companies). Other Statistical Traps Data Mining: Testing every possible combination of data points, finding a spurious but persistent relationship, and attempting to attach meaning to the results after the test has been conducted. Example: Predicting the direction of the NASDAQ over the course of a year based on the winner of the Super Bowl. How to avoid this problem: Begin your research with a rationale for your investigation and an ordered set of hypothesis. Over-fitting: Using questionable statistical techniques to over-fit your model to the data in question. Example: Building models with large number of factors, several of which are highly correlated or do not add information to the model. How to avoid this problem: Run a multiple regression of your model and check your results to confirm that your factors are significant. Do not rely on R2 as a measure of your model's descriptive ability: examine your residuals carefully. Units of Data: Check the factors in your model to insure that the units for the items are consistent, and that you are cognizant of the number of significant digits in your results. Example 1: Creating a daily dollar volume measure using the monthly volume figure without dividing by the number of trading days in a month. Example 2: Comparing the third decimal point of the IC, when there is only one significant digit to the right of the decimal point. How to avoid this problem: Check Zacks Database Appendix for exact item definitions and units. Selecting a Benchmark The following should be considered, when selecting a benchmark: What is the purpose of this backtest? What is the definition of a success in this backtest? (Exceeding the return of an index? Exceeding the return of the universe of stocks in the backtest? etc.) Should the benchmark be market-weighted or equal weighted? Potential Problems: The size effect can dominate any results in an equal weighted portfolio A readily defined index such as the S&P 500) may not be consistent with the style being examined   1See, for example, Robert L. Hagin, "Engineered Investment strategies: problems and solutions" (in Equity Models and Valuation Models, ICFA, 1988, pp.42-50) and Ronald N. Kahn, "What Practitioners Need to Know - About Backtesting" (Financial Analyst Journal, July-August 1990, pp.17-20).

Support | FAQ | Demo | Order Now! About Zacks | Contact Zacks | Knowledge Base |Feedback Copyright © 2001 Zacks Investment Research, Inc.