This installment of Forecasting 101 discusses the importance of tracking forecast accuracy, explains the basic approach and illustrates how to use the results to improve your forecasting process.
Four reasons to track forecast accuracy
1. Improving your forecasting process requires the ability to track accuracy.
Forecasting should be viewed as a continuous improvement process. Your forecasting team should be constantly striving to improve the forecasting process and forecast accuracy. Doing so requires knowing what is working and what is not.
For example, many organizations generate baseline forecasts using statistical approaches and then make judgmental adjustments to them to capture their knowledge of future events. Organizations that track the accuracy of both the statistical and adjusted forecasts learn where the adjustments improve the forecasts and where they make them worse. This knowledge allows them to focus their time and attention on the items where the adjustments are adding value.
2. Tracking accuracy provides insight into expected performance.
A forecast is more than a number. To use a forecast effectively you need an understanding of the expected accuracy.
Within-sample statistics and confidence limits provide some insight into expected accuracy; however, they almost always underestimate the actual (out-of-sample) forecasting error. This is due to the fact that the parameters of a statistical model are selected to minimize the fitted error over the historic data. The parameters are thus adapted to the historic data, and reflect any of its peculiarities. Put another way, the model is optimized for the past—not for the future.
Generally speaking, out-of-sample statistics (i.e., historic forecast errors) yield a better measure of expected forecast accuracy than within-sample statistics.
3. Tracking accuracy allows you to benchmark your forecasts.
If you are lucky enough to be in an industry with published statistics on forecast accuracy, comparing your accuracy to these benchmarks provides insight into your forecasting effectiveness. If industry benchmarks are not available (usually the case), periodically benchmarking your current forecast accuracy against your earlier forecast accuracy allows you to measure your improvement.
4. Monitoring forecast accuracy allows you to spot problems early.
An abrupt unexpected change in forecast accuracy is often the result of some underlying event. For example, if unbeknownst to you, a key customer decides to carry a competing product, your first indication might be an unusually large forecast error. Routinely monitoring forecast errors allows you to spot, investigate and respond to these changes early on—before they turn into bigger problems.
Building a Forecast Archive
Tracking forecast accuracy requires that you maintain a record of previously generated forecasts. This record of the previously generated forecasts is referred to as the forecast archive.
The table above shows a very simple forecast archive for a single product. The first row contains the forecast that was generated in December 2009. The second row contains the forecast that was generated in January 2010, etc.
If your forecasting process generates multiple forecasts (e.g., statistical forecast, adjusted forecast, sales person’s forecast, etc.), then all of the forecasts should be saved to the forecast archive.
Viewing Tracking Reports
Once the archive has been established, it can be used to generate reports comparing the archived forecasts to the actual sales. Due to the volume and complexity of the data, this is best accomplished using either a dedicated software solution such as Forecast Pro TRAC or an internally developed solution that utilizes a relational database—it is not a job for Excel.
The screenshot above shows a sample accuracy tracking report. Due to its cascading-like appearance, this style report is often referred to as a waterfall report. The top half of the numeric section (the “Forecast Report” section) displays the actual demand history and the archived forecasts for the periods being analyzed. The lower half displays summary statistics for different lead times.
The screenshot above displays an exception report flagging all items where our most recent forecast for last month deviated from actual demand by more than 50%.
As we mentioned previously, routinely monitoring forecast errors allows you to spot, investigate and respond to problems—before they turn into bigger problems. Exception reports are used to streamline the monitoring process, reducing the need for manual review and allowing you to focus on the items where human attention is most needed.
Conclusions
Tracking forecast accuracy is an essential part of the forecasting process. If you cannot assess the accuracy of your current process, it is very difficult to improve it. In addition, tracking forecast accuracy provides insight into expected performance, enables you to benchmark your forecasts and allows you to spot, investigate and respond to problems earlier.
About the author:
Eric Stellwagen is the co-founder of Business Forecast Systems, Inc and the co-author of the Forecast Pro software product line. He has consulted widely in the area of practical business forecasting and spends 20-30 days a year presenting workshops on the subject. He has worked with many leading firms including Coca-Cola, Procter & Gamble, Merck, Blue Cross Blue Shield, Nabisco, Owens-Corning and Verizon. He has presented seminars and workshops under the aegis’s of many groups including the Institute for Professional Education, the American Production and Inventory Control Society, the University of Wisconsin, the Institute for Business Forecasting, the World Research Group, the International Institute of Research, the Electric Power Research Institute, the International Communications Forecasting Association and the International Institute of Forecasters. He is currently serving on the board of directors of the International Institute of Forecasters and on the practitioner advisory board of Foresight: The International Journal of Applied Forecasting.
