Q: Errors when usinig gluonts of LSTNet: GluonTSDataError

Question

I've been studying time series forecasting, and I'm trying to learn how to use gluon-ts&python. Here is the source code of gluon-ts:

https://github.com/awslabs/gluon-ts/

I tried to use LSTNetEstimator module, however, it turns out an error as follow. And I found that all the discussions of gluon-ts is about DeepAR. Is there anyone that may come to help?

GluonTSDataError: Input for field "target" does not have the requireddimension (field: target, ndim observed: 1, expected ndim: 2)

I think it has something to do with my custom dataset, which includes a dataframe with the shape of [320000,3] and columns of ['time','Power_Cdp','Power_Dp']. here is the trainset:

from gluonts.dataset.common import ListDataset
from gluonts.model.lstnet import LSTNetEstimator
from gluonts.mx.trainer import Trainer


training_data = ListDataset(
               [{"start": LSTNet_df.index[0], "target": LSTNet_df['Power_Cdp'][:-10000}],
               freq = "15min")

estimator = LSTNetEstimator(freq="15min", prediction_length=24*4, context_length=24*4,
                            num_series=48*4, skip_size=72*4, ar_window=24*4, channels=32,
                            trainer=Trainer(epochs=10))
predictor = estimator.train(training_data=training_data)   # Error 

The full report is bellow.

GluonTSDataError                          Traceback (most recent call last)
<ipython-input-311-ad48fdc20df5> in <module>
      7                             num_series=48*4, skip_size=72*4, ar_window=24*4, channels=32,
      8                             trainer=Trainer(epochs=10))
----> 9 predictor = estimator.train(training_data=training_data)

~/.local/lib/python3.8/site-packages/gluonts/mx/model/estimator.py in train(self, training_data, validation_data, num_workers, num_prefetch, shuffle_buffer_length, cache_data, **kwargs)
    192         **kwargs,
    193     ) -> Predictor:
--> 194         return self.train_model(
    195             training_data=training_data,
    196             validation_data=validation_data,

~/.local/lib/python3.8/site-packages/gluonts/mx/model/estimator.py in train_model(self, training_data, validation_data, num_workers, num_prefetch, shuffle_buffer_length, cache_data)
    145         transformed_training_data = transformation.apply(training_data)
    146 
--> 147         training_data_loader = self.create_training_data_loader(
    148             transformed_training_data
    149             if not cache_data

~/.local/lib/python3.8/site-packages/gluonts/model/lstnet/_estimator.py in create_training_data_loader(self, data, **kwargs)
    216     ) -> DataLoader:
    217         input_names = get_hybrid_forward_input_names(LSTNetTrain)
--> 218         with env._let(max_idle_transforms=maybe_len(data) or 0):
    219             instance_splitter = self._create_instance_splitter("training")
    220         return TrainDataLoader(

~/.local/lib/python3.8/site-packages/gluonts/itertools.py in maybe_len(obj)
     21 def maybe_len(obj) -> Optional[int]:
     22     try:
---> 23         return len(obj)
     24     except (NotImplementedError, AttributeError):
     25         return None

~/.local/lib/python3.8/site-packages/gluonts/transform/_base.py in __len__(self)
     99         # NOTE this is unsafe when transformations are run with is_train = True
    100         # since some transformations may not be deterministic (instance splitter)
--> 101         return sum(1 for _ in self)
    102 
    103     def __iter__(self) -> Iterator[DataEntry]:

~/.local/lib/python3.8/site-packages/gluonts/transform/_base.py in <genexpr>(.0)
     99         # NOTE this is unsafe when transformations are run with is_train = True
    100         # since some transformations may not be deterministic (instance splitter)
--> 101         return sum(1 for _ in self)
    102 
    103     def __iter__(self) -> Iterator[DataEntry]:

~/.local/lib/python3.8/site-packages/gluonts/transform/_base.py in __iter__(self)
    102 
    103     def __iter__(self) -> Iterator[DataEntry]:
--> 104         yield from self.transformation(
    105             self.base_dataset, is_train=self.is_train
    106         )

~/.local/lib/python3.8/site-packages/gluonts/transform/_base.py in __call__(self, data_it, is_train)
    122         self, data_it: Iterable[DataEntry], is_train: bool
    123     ) -> Iterator:
--> 124         for data_entry in data_it:
    125             try:
    126                 yield self.map_transform(data_entry.copy(), is_train)

~/.local/lib/python3.8/site-packages/gluonts/transform/_base.py in __call__(self, data_it, is_train)
    126                 yield self.map_transform(data_entry.copy(), is_train)
    127             except Exception as e:
--> 128                 raise e
    129 
    130     @abc.abstractmethod

~/.local/lib/python3.8/site-packages/gluonts/transform/_base.py in __call__(self, data_it, is_train)
    124         for data_entry in data_it:
    125             try:
--> 126                 yield self.map_transform(data_entry.copy(), is_train)
    127             except Exception as e:
    128                 raise e

~/.local/lib/python3.8/site-packages/gluonts/transform/_base.py in map_transform(self, data, is_train)
    139 
    140     def map_transform(self, data: DataEntry, is_train: bool) -> DataEntry:
--> 141         return self.transform(data)
    142 
    143     @abc.abstractmethod

~/.local/lib/python3.8/site-packages/gluonts/transform/convert.py in transform(self, data)
    127         value = np.asarray(data[self.field], dtype=self.dtype)
    128 
--> 129         assert_data_error(
    130             value.ndim == self.expected_ndim,
    131             'Input for field "{self.field}" does not have the required'

~/.local/lib/python3.8/site-packages/gluonts/exceptions.py in assert_data_error(condition, message, *args, **kwargs)
    114         exception message.
    115     """
--> 116     assert_gluonts(GluonTSDataError, condition, message, *args, **kwargs)

~/.local/lib/python3.8/site-packages/gluonts/exceptions.py in assert_gluonts(exception_class, condition, message, *args, **kwargs)
     93     """
     94     if not condition:
---> 95         raise exception_class(message.format(*args, **kwargs))
     96 
     97 

GluonTSDataError: Input for field "target" does not have the requireddimension (field: target, ndim observed: 1, expected ndim: 2)

Answer 1

@darth baba I tried again lately, and this time I used DeepVAR. Here is what got. I reset "target": ... , stepped into my code and found

gluonts.exceptions.GluonTSDataError: Array 'target' has bad shape - expected 1 dimensions, got 2.

This is because I set "target": train_df[['Power_Cdp', 'Power_Active_Fan']], which might be reported as an error at ./gluonts/dataset/common.py +385. I found self.req_ndim != value.ndim , which suggested that I could have input a wrong shape of target, thus I reset input like "target": train_df.index[:] and this problem solved.

But it reports an another error,

gluonts.exceptions.GluonTSDataError: Input for field "target" does not have the requireddimension (field: target, ndim observed: 1, expected ndim: 2)

To be sincerely, this one confused me a lot, for the gluonts's code seems so complicated. I checked again and found it reports error at ./gluonts/transform/convert.py +129. It seems like expected_ndim is not equals to value.ndim in this case.

However, after rewriting expected_ndim manually, the training progress worked eventually.

I have no idea whether this modification is right. Although the avg_epoch_loss are decreasing, the final forecast is not as good as expected.