Demand forecasting and performance evaluation in today’s disrupted consumer demand-driven supply chain environment has become an extremely challenging discipline for business planners to master. For instance, current forecasting performance metrics for intermittent demand have shortcomings that are not easily overcome. In particular, the widely used Mean Absolute Percentage Error (MAPE) metric is unusable in this context when no demand is encountered. To embrace more agile and reliable approaches, I will consider a new metric for multi-step ahead forecasts with fixed horizons (e.g. Leadtime demand forecasts) by using relative entropy measures from information theory.

Before dealing with intermittent demand forecasting applications, I will examine a lead-time demand forecasting example without intermittency first. The approach will be equally applicable. I have added the notation in the spreadsheet, so you can follow this with your own data.

In the dataset, the holdout data (in italics) are shown in the row for year 2016. A fiscal, budget or planning year does not necessarily have to start in January, so lead-time or forecasting cycle totals are shown also, as we will be using them in our calculations. For a twelve-month holdout sample, we have created three forecasts by judgment, method and model:

For a judgment forecast, we will assume last year actuals as the forecast for the next year. This forecast profile is Year-1 FP. For a method, we use the constant MAVG_12 FP, which is simply the average of previous 12-month history. The MAVG12 method has a level profile.

The model forecast is based on the ETS (A,A,M) which has a local level, multiplicative seasonal forecast profile and is a State Space forecasting model described in Chapter 8 of my book. The model was selected in automatic mode since the profile is a deterministic trend/seasonal profile.

We start by defining a multi-step ahead forecast with a fixed horizon (e.g. lead-time forecast) as a Forecast Profile (FP). A forecast profile can be created by a model, a method, or informed judgment. The forecast profile of a m-step-ahead forecasts makes up a sequence FP = { FP(1), FP(2), . . . , FP(m) }. For our example, we will assume that the forecasts are monthly starting at time t = T and ending at time t = T + m, where m = 12. Typically, lead-times could be 2 months to 6 months or more, the time for an order to reach an inventory warehouse. For operational and budget planning, the time horizon might be 12 months. This 12-month pattern is called a Forecast Profile.

At the end of the forecast horizon or planning cycle, there is a corresponding Actual Profile AP ={ AP(1), AP(2), . . . , AP(m) } to compare with FP for an accuracy assessment. Familiar methods include the Mean Absolute Percentage Error (MAPE). The problem, in case of intermittent demand, is that some AP values can be zero, which leads to undefined terms in the MAPE.

We are going to evaluate forecasting performance by considering the information in the Forecast Profile using information-theoretic measures. Then we can assess how the forecast profile diverges from the pattern or profile in the actuals. 

Coding a Profile Into an Alphabet Profile

For a given Profile, the Alphabet Profile is a set of weights whose sum is one. That is, a Forecast Alphabet Profile is FAP = {f1, f2, . . . fm ], where fi = FP (i)/ . In other words, we simply divide each forecast by the sum of the forecasts over the horizon. This will give us fractions whose sum equals one. The Actual Alphabet Profile is AAP = {a1, a2, . . . am], where ai is the ratio of the actual to the Total Leadtime actuals. When we compare a profile with the corresponding alphabet profile, we note that the pattern does not change.

An Information-theoretic Accuracy Measure of Forecast Profile Performance

The performance of the process that created the Forecast Profile is of interest because we will come up with a ‘distance’ metric between the Forecast Profile and Actual Profile that has its basis in Information Theory. We notice in the graphs that the alphabet profilehas the same pattern as the corresponding source profile from which it was created. The alphabet profile is necessary in the construction of the ‘distance’ metric.

The relative entropy or divergence measure is used as a performance measure in various applications in meteorology, neuroscience and machine learning. I use it here as a measure of how one forecast profile diverges from another profile. A performance measure for the forecast alphabet profile (FAP) is given by a Kullback-Leibler divergence measure, which can be interpreted as a measure of dissimilarity or ‘distance’ between the actual alphabet AAP and forecast alphabet FAP profiles. The measure, known as the Kullback-Leibler divergence, is non-negative and is equal to zero if and only if ai = fi (I = 1,2, . . m). This happens when the alphabet profiles overlap, or what we might consider as 100% accuracy.

A Decomposition of the Divergence Measure D(AAP, FAP)

The MAVG_12 FP is level, as is the Naïve_1 (NF1) FP and simple exponential smoothing (SES) FP. In these cases,

where m = 12. The first term is known as the entropy H of AAP and is a measure of the information in AAP. Hence, D(AAP, FAP) = ln (12) – H (AAP), in this example.

For a single forecast, Actual (A) minus Forecast (F) is the accuracy of the forecast, so D(AAP, FAP) can be viewed as a measure of accuracy for a single Actual (AP) and Forecast (FP) profile. That is how I propose D(AAP, FAP) can be used for assessing the performance of multi-step-ahead and lead-time demand forecasts.

I am unaware of where this accuracy measure has been previously used in the forecasting literature or used in demand forecasting software or applications. Someone might wish to bring this to my attention.

In a previous blog on my website and a LinkedIn article, I gave a spreadsheet example of the Forecast Profile Performance (FPP) index to measure intermittent demand forecast accuracy. I invite you to join the LinkedIn groups I manage and share your thoughts and practical experiences with intermittent data and smarter demand forecasting in the supply chain. Feel free to send me the details of your findings, including the underlying data used without identifying descriptions, so I can continue to test and validate the process for intermittent data.