Demand forecasting: when the spreadsheet is no longer enough

In almost every distribution, manufacturing or retail business there is a forecasting spreadsheet. Someone competent built it over the years, and it works reasonably well. Until the day demand changes pattern and nobody understands why in time. The question is whether clearly better methods exist today, and the research has an answer.

The M5 competition

The M competitions, run since 1982 by Spyros Makridakis, are the world benchmark for forecasting methods. The fifth edition, in 2020, used real Walmart data: 42,840 sales series arranged in a hierarchy, from the overall total down to each product in each store, with forecasts required for 30,490 product and store combinations. A total of 5,507 teams from 101 countries took part (Makridakis, Spiliotis and Assimakopoulos, 2022).

The result marked a turning point. It was the first M competition in which all the top methods were pure machine learning and clearly better than every statistical benchmark.

The winning method was 22.4% more accurate than the best statistical benchmark. The top five all beat the 20% mark.

What the winners did differently

Three ingredients recur among the top methods:

1. Learning from many series at once. Instead of one model per product, a single model trained on all products and stores. The authors call this cross-learning, and practically all of the 50 best methods used it.
2. External information. Prices, promotions and calendar events. Even in classic statistical methods, adding these variables improved accuracy by 6% to 13%.
3. Gradient-boosted decision trees. Most of the top methods used LightGBM, a fast and robust machine learning technique that is now widely available.

Where the limits lie

The competition also shows what machine learning does not solve. The gain was large at aggregate levels, around 40% for total sales, but small at the level of each product in each store, where it was about 3%. Demand for an individual item, on a given day, in a given store, remains largely unpredictable. Only 7.5% of teams beat the benchmark method. And the authors point out that the results come from one retailer and one sector.

What this means for your operation

• Forecast at the level where you make decisions: if the decision is the weekly order to a supplier by product family, that is where the gain is greatest.
• Use the information you already have. Prices, promotions and the calendar almost always sit in a system somewhere, and the M5 shows they are worth more than a sophisticated algorithm without them.
• Plan stock around the uncertainty. A good model returns an estimate with its uncertainty: items with a stable forecast need less safety stock, and uncertain items need more.
• Before you switch, run the model alongside your current method for a few weeks and measure the errors of both, so the decision is made with numbers from your own operation.