Kalman filter #751
Kalman filter #751
Conversation
…othing due to timeseries dataset vs forecastdataset in postprocessor
Signed-off-by: Guilly Kolkman <guilly.kolkman@sia-partners.com>
packages/openstef-models/src/openstef_models/presets/forecasting_workflow.py
Show resolved
Hide resolved
packages/openstef-models/src/openstef_models/presets/forecasting_workflow.py
Outdated
Show resolved
Hide resolved
| *feature_adders, | ||
| HolidayFeatureAdder(country_code=config.location.country_code), | ||
| DatetimeFeaturesAdder(onehot_encode=False), | ||
| KalmanPreprocessor( |
There was a problem hiding this comment.
Is there a reason to only add it to xgboost? Did it not perform well in gblinear?
Also whether we want to add it to the preset, depends on the results also. I would probably leave it out (for now).
There was a problem hiding this comment.
We added this for convenience, we will take it out. Good you noticed, thank you.
packages/openstef-models/src/openstef_models/transforms/general/kalman_filter.py
Outdated
Show resolved
Hide resolved
|
|
||
| # Preprocessor: operates on TimeSeriesDataset | ||
| class KalmanPreprocessor(BaseKalman, TimeSeriesTransform): | ||
| """Apply Kalman Smoothing to time series data to reduce noise and improve temporal consistency. |
There was a problem hiding this comment.
Is there a specific source you used for the implementation? If so, I think it is nice to add it to the docs.
There was a problem hiding this comment.
The implementation was done to mimic other transforms. No additional sources were used
|
|
||
| import pandas as pd | ||
| from pydantic import Field | ||
| from sktime.transformations.series.kalman_filter import ( |
There was a problem hiding this comment.
Can you add this as a dependency? I don't think we have it already.
There was a problem hiding this comment.
added as dependency of openstef models
packages/openstef-models/src/openstef_models/transforms/general/kalman_filter.py
Outdated
Show resolved
Hide resolved
packages/openstef-models/src/openstef_models/transforms/general/kalman_filter.py
Outdated
Show resolved
Hide resolved
| transform = KalmanPreprocessor() | ||
| result = transform.fit_transform(dataset) | ||
|
|
||
| expected_values = pd.DataFrame( |
There was a problem hiding this comment.
It would also be nice to add a tests for the multiple column case where all columns are transformed.
Implemented Kalman filter as both a pre and post processing option with tests.
The implementation follows the sktime implementation of KalmanFilterTransformerFP build on filterpy. The kalman filter reduces noise and improves results. However, the method is computationally more expensive.
For the preprocessor you can add it in the preprocessing list in workflows of models with, for example:
preprocessing = [ *checks, *feature_adders, HolidayFeatureAdder(country_code=config.location.country_code), DatetimeFeaturesAdder(onehot_encode=False), KalmanPreprocessor( selection=config.kalman_features, ), *feature_standardizers, ],The kalman_features are currently based only on weather features.
For postprocessing a flag is added which can be set to True or False depending on the user's need.