Press
Release
FOR IMMEDIATE RELEASE
June 28, 1999
FORECAST PRO BEATS THE COMPETITION IN THE LARGEST FORECASTING
COMPETITION EVER HELD
June 28, 1999, 19th International Symposium on Forecasting, Washington,
DC. Final results from the M-3 forecasting competition were presented
today by Professor Michele Hibon of the French business school INSEAD.
Designed to evaluate the accuracy of different forecasting methods,
the competition was sponsored by the International Journal of
Forecasting and is the largest and most comprehensive empirical
forecasting study ever performed. The most striking result was the
performance of the fully-automated Forecast Pro for Windows package
(Version 3), which significantly outperformed all of the other software
approaches as well as 18 out of 19 academic teams.
The study compared the accuracy of 26 different approaches used
to prepare 3,003 forecasts based on historic demand data. The data
were selected to cover as wide of a range of data types as possible
(e.g., microeconomic, macroeconomic, industrial, financial and demographic)
and included weekly, monthly, quarterly and annual series. Each
entrant was free to use any method he or she wished to prepare the
forecasts. Nineteen of the approaches were implemented by forecasting
experts from academic institutions such as the Wharton School, Case
Western Reserve, INSEAD and the Imperial College (London). Approaches
included techniques like exponential smoothing models, Box-Jenkins
models, neural networks and rule-based approaches. Human judgment
and statistical expertise played a significant role in many of the
approaches. The remaining seven approaches were implemented using
commercially available, fully automated forecasting packages—Forecast
Pro, SmartForecasts, ForecastX, Autocast and Autobox (three variations).
The researchers evaluated the forecasts against the actual future
values of the series.
The results surprised many who assumed that a computer program
would not be able to outperform human experts. The results did not
surprise Dr. Robert Goodrich, the author of Forecast Pro and president
of Business Forecast Systems. According to Goodrich, “Over
the last ten years, our clients have used Forecast Pro to forecast
hundreds of millions of time series. Whenever unusual forecasts
are reported, we carefully analyze the situation and make improvements
to our algorithms. While it is difficult to fine-tune a subjective
human decision-making process, it is fairly straightforward to fine-tune
quantitative algorithms in a computer program. The result is that
Forecast Pro recognizes and responds appropriately to many more
special circumstances than a human practitioner would have encountered
and would be able to keep track of.” Goodrich also noted that
the M-3 competition is the second time that Forecast Pro has outperformed
human practitioners and other forecasting packages in a formal study.
The first time was in a much smaller study conducted by Keith Ord
and Sam Lowe of Penn State and published in the February, 1996 issue
of The American Statistician.
Final results from the M-3 competition are posted at the Wharton
School’s Web site (www-marketing.Wharton.upenn.edu/forecast/m3-competition.html.),
and will be published in The International Journal of Forecasting
in the near future, and likely be the basis of a book.
M-3:
AVERAGE SYMMETRIC MAPE (MEAN ABSOLUTE PERCENT ERROR): ALL DATA
|
|
|
Forecasting
Horizons
|
Average
of Forecasting Horizons
|
SOFTWARE
|
1
|
2
|
3
|
4
|
5
|
6
|
8
|
12
|
15
|
18
|
1-4
|
1-6
|
1-8
|
1-12
|
1-15
|
1-18
|
Forecast Pro V3
|
8.6
|
9.6
|
11.4
|
12.9
|
13.3
|
14.3
|
12.6
|
13.2
|
16.4
|
18.3
|
10.64
|
11.69
|
11.86
|
12.14
|
12.60
|
13.19
|
|
Autobox-1
|
9.8
|
11.1
|
13.1
|
15.1
|
16.0
|
16.8
|
14.2
|
15.4
|
19.1
|
20.4
|
12.30
|
13.67
|
13.78
|
14.00
|
14.56
|
15.23
|
|
Autobox-2
|
9.5
|
10.4
|
12.2
|
13.8
|
13.8
|
14.9
|
13.2
|
15.2
|
18.2
|
19.9
|
11.48
|
12.44
|
12.63
|
13.10
|
13.70
|
14.41
|
|
Autobox-3
|
9.7
|
11.2
|
12.9
|
14.6
|
15.8
|
16.5
|
14.4
|
16.1
|
19.2
|
21.2
|
12.08
|
13.43
|
13.64
|
14.01
|
14.57
|
15.33
|
|
Autocast
|
9.1
|
10.0
|
12.1
|
13.5
|
13.8
|
14.7
|
13.1
|
14.3
|
17.7
|
19.6
|
11.20
|
12.21
|
12.40
|
12.80
|
13.34
|
14.01
|
|
* ForecastX
|
8.7
|
9.8
|
11.6
|
13.1
|
13.2
|
13.9
|
12.6
|
13.9
|
17.8
|
18.7
|
10.82
|
11.73
|
11.89
|
12.22
|
12.81
|
13.49
|
|
SmartForecasts
|
9.2
|
10.3
|
12.0
|
13.5
|
14.0
|
15.1
|
13.0
|
14.9
|
18.0
|
19.4
|
11.23
|
12.34
|
12.49
|
12.94
|
13.48
|
14.13
|
ACADEMIC TEAMS
|
|
ADAPTA
|
15.5
|
16.2
|
17.1
|
18.2
|
18.1
|
18.9
|
15.7
|
18.6
|
20.9
|
22.3
|
16.74
|
17.33
|
16.98
|
17.21
|
17.62
|
18.12
|
|
AMM1
|
9.8
|
10.6
|
11.2
|
12.6
|
13.0
|
13.5
|
14.1
|
14.9
|
18.0
|
20.4
|
11.04
|
11.76
|
12.43
|
13.04
|
13.77
|
14.63
|
|
AMM2
|
10.0
|
10.7
|
11.3
|
12.9
|
13.2
|
13.7
|
14.3
|
15.1
|
18.4
|
20.7
|
11.21
|
11.95
|
12.62
|
13.21
|
13.97
|
14.85
|
|
ARARMA
|
9.7
|
10.9
|
12.6
|
14.2
|
14.6
|
15.6
|
13.9
|
15.2
|
18.5
|
20.3
|
11.83
|
12.92
|
13.12
|
13.54
|
14.09
|
14.74
|
|
AutomatANN
|
9
|
10.4
|
11.8
|
13.8
|
13.8
|
15.5
|
13.4
|
14.6
|
17.3
|
19.6
|
11.23
|
12.38
|
12.58
|
12.96
|
13.48
|
14.11
|
|
B-J automatic
|
9.2
|
10.4
|
12.2
|
13.9
|
14.0
|
14.8
|
13.0
|
14.1
|
17.8
|
19.3
|
11.42
|
12.41
|
12.54
|
12.80
|
13.35
|
14.01
|
|
COMB S-H-D
|
8.9
|
10.0
|
12.0
|
13.5
|
13.7
|
14.2
|
12.4
|
13.6
|
17.3
|
18.3
|
11.10
|
12.04
|
12.13
|
12.40
|
12.91
|
13.52
|
|
DAMPEN
|
8.8
|
10.0
|
12.0
|
13.5
|
13.7
|
14.3
|
12.5
|
13.9
|
17.5
|
18.9
|
11.05
|
12.04
|
12.14
|
12.44
|
12.96
|
13.63
|
|
FLORES-PEARCE-1
|
9.2
|
10.5
|
12.6
|
14.5
|
14.8
|
15.3
|
13.8
|
14.4
|
19.1
|
20.8
|
11.68
|
12.79
|
13.03
|
13.31
|
13.92
|
14.70
|
|
FLORES-PEARCE-2
|
10.0
|
11.0
|
12.8
|
14.1
|
14.1
|
14.7
|
12.9
|
14.4
|
18.2
|
19.9
|
11.96
|
12.77
|
12.81
|
13.04
|
13.61
|
14.29
|
|
HOLT
|
9.0
|
10.4
|
12.8
|
14.5
|
15.1
|
15.8
|
13.9
|
14.8
|
18.8
|
20.2
|
11.67
|
12.93
|
13.11
|
13.42
|
13.95
|
14.60
|
|
NAIVE1
|
11.6
|
12.5
|
14.6
|
16.1
|
16.5
|
16.9
|
15.4
|
16.0
|
20.5
|
22.1
|
13.69
|
14.70
|
14.92
|
15.24
|
15.84
|
| 
