Skip to content

Feat/metric crps#3089

Merged
dennisbader merged 5 commits into
masterfrom
feat/metric_crps
Apr 26, 2026
Merged

Feat/metric crps#3089
dennisbader merged 5 commits into
masterfrom
feat/metric_crps

Conversation

@dennisbader

@dennisbader dennisbader commented Apr 26, 2026

Copy link
Copy Markdown
Collaborator

Checklist before merging this PR:

  • Mentioned all issues that this PR fixes or addresses.
  • Summarized the updates of this PR under Summary.
  • Added an entry under Unreleased in the Changelog.

Note: this is a continuation of @Boubker10 's work on #3088.

Fixes #1112.

Summary

Added two new metrics for probabilistic time series forecasting:

  • crps (Continuous Ranked Probability Score): a per-time-step proper scoring rule that measures the compatibility of a predictive sample distribution with a scalar observation. Reduces to the Absolute Error (AE) when all samples are identical.
  • mcrps (Mean CRPS): mean of crps over the time axis, analogous to MAE for probabilistic forecasts.

Both metrics follow the existing metric API (support for component_reduction, series_reduction, time_reduction, multi-ts, multivariate).

Other Information

Formula used (energy score formulation):

CRPS = (1/N) * Σ|x_i - y| - (1/(2N²)) * Σ_i Σ_j |x_i - x_j|

Tests added in darts/tests/metrics/test_metrics.py covering:

  • raises ValueError for deterministic input
  • perfect prediction → CRPS = 0
  • numerical correctness against manual calculation
  • multivariate and multi-ts consistency

@dennisbader dennisbader mentioned this pull request Apr 26, 2026
3 tasks
@codecov

codecov Bot commented Apr 26, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 96.24%. Comparing base (aa16909) to head (ddd3206).

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #3089      +/-   ##
==========================================
- Coverage   96.30%   96.24%   -0.07%     
==========================================
  Files         160      160              
  Lines       17263    17280      +17     
==========================================
+ Hits        16625    16631       +6     
- Misses        638      649      +11     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@dennisbader dennisbader merged commit 16fda02 into master Apr 26, 2026
9 checks passed
@dennisbader dennisbader deleted the feat/metric_crps branch April 26, 2026 13:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Consider adding 'strictly proper' scoring rule metrics for probablistic forecasts (eg CRPS score)

2 participants