Feature/metadata (#2656)

* Added metadata attribute, carried along through operations

* changelog

* refactored metadata into a pandas Series

* fix lags_past_covariates dict type breaks lags_future_covariates when output_chunk_shift>0 (#2655)

* if clause modified to correct bug

* contribution added to changelog

* implemented modifications from reviewer

* unit tests added

---------

Co-authored-by: “authierj” <“[email protected]”>

* fix failing tfm load with pytorch>=2.6.0 (#2658)

* Allow to strip training data for storage and memory reason on save for GlobalForecastingModel (#2649)

* Add option to drop training_series during save() for GlobalForecastingModel

* Update error on predict with no training series, update tests

* Update CHANGELOG.md

* Add _clean fonction to get a cleaned instance of the models

* Use shallow copy to get clean model

* Update darts/models/forecasting/regression_model.py

Co-authored-by: Dennis Bader <[email protected]>

* Update darts/models/forecasting/torch_forecasting_model.py

Co-authored-by: Dennis Bader <[email protected]>

* Define _clean() in ForecastingModel

* Fix typo in docstring

Co-authored-by: Dennis Bader <[email protected]>

* Reformulate comment

Co-authored-by: Dennis Bader <[email protected]>

* Update CHANGELOG.md

Co-authored-by: Dennis Bader <[email protected]>

* Add clean argument in model.save for conformal and ensemble models

* Update docstring

* Remove additional training params on clean, update tests

* Fix _clean and save logic, update save/load test

* Update darts/models/forecasting/forecasting_model.py

Co-authored-by: Dennis Bader <[email protected]>

* Remove map_location on model load, set cpu as default accelerator

* Update CHANGELOG.md

* Add covariate to save/load test

* Update CHANGELOG.md

* Add _clean() to clean sub-model, update unit tests

* Add _clean() to clean sub-models, update unit tests

* minor refactor

* further upadtes

* pickle tfm model wrapper instead of torch save / load

* update changelog

* last fix

---------

Co-authored-by: Dennis Bader <[email protected]>

* adapted tests for metadata

* changelog

* add missing metadat propagation and fix tests

* metadata changed to dict

* tests adapted, one removed for metadata

* remove metadata from timeseries generation

* update timeseries

* extend unit tests

* update changelog

* udpate changelog

---------

Co-authored-by: Jules Authier <[email protected]>
Co-authored-by: “authierj” <“[email protected]”>
Co-authored-by: Dennis Bader <[email protected]>
Co-authored-by: Jonas Blanc <[email protected]>
Co-authored-by: authierj <[email protected]>

Author: 5 people

CHANGELOG.md

@@ -11,6 +11,7 @@ but cannot always guarantee backwards compatibility. Changes that may **break co
 
 **Improved**
 
+- Added support for embedding metadata in `TimeSeries`. You can now attach any type of metadata as a dictionary to your time series, either at series creation using parameter `metadata` (or `metadata_cols` with `from_group_dataframe()`), or add it to an existing series via the `TimeSeries.with_metadata()` method. Unlike static covariates, metadata is not utilized by Darts' models during the forecasting process. However, the metadata is propagated through all downstream tasks, preserving the integrity and continuity of the information throughout the data pipeline. [#2656](https://github.com/unit8co/darts/pull/2656) by [Yoav Matzkevich](https://github.com/ymatzkevich), [Jules Authier](https://github.com/authierj) and [Dennis Bader](https://github.com/dennisbader).
 - Improved `CatBoostModel` documentation by describing how to use native multi-output regression. [#2659](https://github.com/unit8co/darts/pull/2659) by [Jonas Blanc](https://github.com/jonasblanc)
 - `TimeSeries.from_dataframe()` and `from_series()` now support creating `TimeSeries` from additional backends (Polars, PyArrow, ...). We leverage `narwhals` as the compatibility layer between dataframe libraries. See the `narwhals` [documentation](https://narwhals-dev.github.io/narwhals/) for all supported backends. [#2661](https://github.com/unit8co/darts/pull/2661) by [Jules Authier](https://github.com/authierj)
 - Added new export methods: `TimeSeries.to_dataframe()` and `TimeSeries.to_series()`. These methods support exporting `TimeSeries` to DataFrames and Series for all backends supported by `narwhals`. See the `narwhals` documentation [documentation](https://narwhals-dev.github.io/narwhals/extending/) for all supported backends. [#2701](https://github.com/unit8co/darts/pull/2701) by [Jules Authier](https://github.com/authierj)

darts/dataprocessing/transformers/midas.py

@@ -488,4 +488,5 @@ def _create_midas_df(
             values=arr,
             columns=cols,
             static_covariates=static_covariates,
+            metadata=series.metadata,
         )

darts/models/forecasting/baselines.py

@@ -409,6 +409,7 @@ def _target_average(self, prediction: TimeSeries, series: TimeSeries) -> TimeSer
             columns=series.components,
             static_covariates=series.static_covariates,
             hierarchy=series.hierarchy,
+            metadata=prediction.metadata,
         )
 
     def _params_average(self, prediction: TimeSeries, series: TimeSeries) -> TimeSeries:
@@ -444,4 +445,5 @@ def _params_average(self, prediction: TimeSeries, series: TimeSeries) -> TimeSer
             columns=prediction.components[: likelihood_n_params * n_components],
             static_covariates=None,
             hierarchy=None,
+            metadata=prediction.metadata,
         )

darts/models/forecasting/forecasting_model.py

@@ -593,12 +593,12 @@ def _build_forecast_series(
             input_series if input_series is not None else self.training_series
         )
         return _build_forecast_series(
-            points_preds,
-            input_series,
-            custom_components,
-            with_static_covs,
-            with_hierarchy,
-            pred_start,
+            points_preds=points_preds,
+            input_series=input_series,
+            custom_columns=custom_components,
+            with_static_covs=with_static_covs,
+            with_hierarchy=with_hierarchy,
+            pred_start=pred_start,
         )
 
     def _historical_forecasts_sanity_checks(self, *args: Any, **kwargs: Any) -> None:
@@ -1266,6 +1266,7 @@ def retrain_func(
                             if not predict_likelihood_parameters
                             else None
                         ),
+                        metadata=series_.metadata,
                     )
                 )
             else:

darts/models/forecasting/kalman_forecaster.py

@@ -152,6 +152,7 @@ def _predict(
             columns=self.training_series.columns,
             static_covariates=self.training_series.static_covariates,
             hierarchy=self.training_series.hierarchy,
+            metadata=self.training_series.metadata,
         )
 
         series = series.append(series_future)

darts/models/forecasting/varima.py

@@ -115,6 +115,7 @@ def _differentiate_series(self, series: TimeSeries) -> TimeSeries:
                 df=series.to_dataframe(copy=False).diff().dropna(),
                 static_covariates=series.static_covariates,
                 hierarchy=series.hierarchy,
+                metadata=series.metadata,
             )
         return series