Imputing Missing Values
Time series generated from measured data often have missing points, because in practice it is not uncommon for the data loggers to fail to record measured values over shorter periods of time. The measurement data are then in principle equidistant (equally spaced), but show gaps. These gaps are either just missing data values (in equally spaced time series) or completely missing points in time (in unequally spaced time series).
In order to synchronize these time series with other measurement data and to process them further, it is necessary to fill the gaps with meaningful values. This is done using the method Impute missing values/data. Based on a time grid, it automatically identifies missing data in the form of missing data values (NaNs) and, in unequally spaced time series, also in the form of missing time points. The impute procedure is based on the assumption that the measurement data should be equally spaced.
The method automatically identifies the seasonal structure of the time series and takes it into account when supplementing the missing measurements. The appropriate forecasting method is selected automatically.
For very short missing periods (the size of which is determined by the user), the measured values are interpolated linearly, while for larger gaps data based on exponential smoothing (according to Dannecker, 2015) are added.
The following figure shows an example of electricity demand with a seasonal structure where missing data have been supplemented (red).
Applying the Method
To supplement the missing parts of a time series, use the ribbon items Methods → Analyze → Impute missing values (see following figure)
or the context menu in the Project explorer Methods → Change values → Impute missing data (see following figure).
In the window that opens (see following figure), select the one to be imputed under Time series. Only one time series can be selected.
Finding Gaps
Click on the Find gaps button (see figure below).
The gaps are identified.
The dialog window of the method Impute missing values displays the first Gap with the first and last point in time and the length in number of time steps.
The drop-down list contains the other gaps that can be selected. Use the Previous gap and Next gap buttons to navigate from one gap to the next. A click on the Focus button focuses the plot on the gap in the plot window.
Interpolating
In the Interpolate short intervals section, select the interval length in the desired unit (steps, seconds, minutes, or hours) up to which the missing values are to be supplemented by interpolation. Start the interpolation with the Interpolate button.
The interpolated missing values are deleted from the drop-down list of Gaps. To display them again, click the Find gaps button again.
Saisonal Completion
For the other intervals that are to be completed according to a pattern, select the Season (period) length in hours, days, or weeks (see following figure).
Procedere
Results for Equally and Unequally Spaced Time Series
The following figure shows the time series generated by the method.
While the figure above shows the completion of an equally spaced time series, the following two figures illustrate the completion of an unequally spaced time series. The first of the two figures shows the period with gaps: not only empty data points, as in the equally spaced time series above, but also missing time points, which were discovered by the imputation method, inserted into the time series, and filled with meaningful data in the second figure.
The unequally spaced time series is supplemented with the missing time points until it is equally spaced. The resulting time series therefore contains more data points than the original time series.
Calculation Method
Expert knowledge is not necessary to apply the method. In the following section, particularly interested users will learn details about the method, which runs automatically in the background of the simple user interface.
In its simplest form, exponential smoothing calculates missing values by recursion \( \begin{equation} \begin{aligned} s_t= αx_t+(1-α) s_{t-1} \end{aligned}\end{equation}\) using measured values \( \begin{equation} \begin{aligned} \{x_t\}\end{aligned}\end{equation}\), where \( \begin{equation} \begin{aligned} 0<α<1 \end{aligned}\end{equation}\) is the smoothing factor.
The applied method from time series analysis for the prediction of seasonal behavior, the triple exponential smoothing (Holt 2004), also called Holt-Winters Smoothing, depends on three parameters \( \begin{equation} \begin{aligned} 0<α,β,γ<1\end{aligned}\end{equation}\) and the seasonal length \( \begin{equation} \begin{aligned} S\end{aligned}\end{equation}\). The time series under investigation is broken down into three components:
“Approximately” means that the components are actually composed of somewhat more complex expressions.
The parameters \( \begin{equation} \begin{aligned} \alpha\end{aligned}\end{equation}\), \( \begin{equation} \begin{aligned} \beta \end{aligned}\end{equation}\), and \( \begin{equation} \begin{aligned} \gamma\end{aligned}\end{equation}\) control a weighting of the composition of the components from current values and values in the past. If \( \begin{equation} \begin{aligned} S\end{aligned}\end{equation}\) is the length of the season, the three components are as follows:
\( \begin{equation} \begin{aligned} l_t=\alpha⋅(y_t-s_{t-S})+(1-\alpha)⋅(l_{t-1}+b_{t-1}) \end{aligned}\end{equation}\),
\( \begin{equation} \begin{aligned} b_t=β⋅(l_t-l_{t-1} )+(1-β)⋅b_{t-1}\end{aligned}\end{equation}\),
\( \begin{equation} \begin{aligned} s_t=γ⋅(y_t-l_t )+(1-γ)⋅s_{t-S} \end{aligned}\end{equation}\).
With these components, starting from a point in time \( \begin{equation} \begin{aligned} t\end{aligned}\end{equation}\), usually the last observed value, a prediction can be made for any number of steps into the future:
\( \begin{equation} \begin{aligned} y_{t+r}=l_t+r⋅b_t+s_{t-(S-1)+r ̃} \end{aligned}\end{equation}\) mit \( \begin{equation} \begin{aligned} r ̃=(r-1)\ mod\ S\end{aligned}\end{equation}\).
For fully automatic calculation, the smoothing factors \( \begin{equation} \begin{aligned} \alpha\end{aligned}\end{equation}\), \( \begin{equation} \begin{aligned} \beta \end{aligned}\end{equation}\) and \( \begin{equation} \begin{aligned} \gamma\end{aligned}\end{equation}\) must be determined from the existing measurement data. These parameters are calculated using the Nelder–Mead method (also downhill simplex method) developed by John Nelder and Roger Mead in 1965. Different test statistics are used to evaluate the quality of a parameter set. Supplemented data are compared with existing observation data, and a sum of squared errors is calculated.
The parameter for the season length \( \begin{equation} \begin{aligned} S\end{aligned}\end{equation}\) can either be set by the user or automatically determined by the maximum autocorrelation.
