Skip to content

Conversation

@AddyM
Copy link

@AddyM AddyM commented Oct 10, 2025

Fixes #2846

Summary

This PR improves error handling when backtest() is called with insufficient future_covariates, replacing cryptic IndexError and TypeError messages with clear, actionable ValueError exceptions. It also fixes a bug in the residuals() method where covariate parameters were not being properly forwarded.

Changes Made

1. Enhanced Error Messages in backtest()

**Added validation to detect when future_covariates are provided but do not cover the required time range. New error messages now include: **

  • Specific problem description with timestamps
  • Exact number of missing time steps
  • Clear resolution steps with code examples
  • Alternative approaches (encoders, relative index)

Before:
model.backtest(series, future_covariates=short_cov, forecast_horizon=6)
IndexError: index 1 is out of bounds for axis 0 with size 1

After:
model.backtest(series, future_covariates=short_cov, forecast_horizon=6)
ValueError: TFTModel requires future_covariates to extend beyond the series end.

 **Problem: Series ends at: 2020-04-09, Future covariates end at: 2020-03-15
                     Required: 2020-04-15 (series end + 6 steps) 
                     Missing: 31 time steps**

  **Solution: Extend future_covariates to cover the full series + 6 steps.**

2. Added Validation Helper Method
Created _validate_future_covariates_for_forecast() to validate:

  • Covariates start at or before the series start time
  • Covariates extend at least forecast_horizon steps beyond the series end
  • Validation only occurs when covariates are provided (model handles None case)
  • Works for both single series and sequences

3. Fixed residuals() Bug
The residuals() method accepted past_covariates and future_covariates parameters but didn’t forward them to the internal backtest() call, causing potential validation failures or incorrect results. Now fixed.

4. Design Decisions
Validation added only to backtest() (not historical_forecasts()) to preserve flexibility for internal calls (e.g., ensembles). Only validates when covariates are provided; lets model handle None case. No API or behavioral breaking changes.

Testing:

  • Added a comprehensive test suite: test_backtest_error_messages.py
    • Covers 5 main scenarios: Missing covariates, IndexError → ValueError conversion, TypeError → ValueError conversion, Regression tests without future_covariates, and edge cases for exact boundary conditions.
  • All tests pass locally:
    • 5/5 new tests pass
    • All existing backtest tests pass
    • All residual tests pass
    • No regressions detected

Aditya Mehra added 2 commits October 10, 2025 09:02
…() and fix residuals() bug

1. Add validation for future_covariates coverage in backtest()
   - Clear error messages when covariates start too late or don't extend far enough
   - Replaces cryptic IndexError/TypeError with actionable ValueError
   - Only validates when covariates are provided; model handles None case
   - Validation only in backtest() to preserve flexibility in historical_forecasts()

2. Fix bug in residuals() method
   - residuals() now properly passes past_covariates and future_covariates to backtest()
   - Previously these parameters were accepted but not forwarded

3. Add comprehensive test suite
   - 5 tests covering insufficient covariate scenarios in backtest()
   - Tests coverage validation (too short, starts too late)
   - Validates no regression for models without covariates
   - Tests exact boundary cases

Fixes unit8co#2846
@AddyM AddyM requested a review from dennisbader as a code owner October 10, 2025 13:53
@AddyM AddyM changed the title Improve error messages for insufficient future_covariates in backtest() and fix residuals() bug Improve error messages for insufficient future_covariates in backtest() and fix residuals() bug 2846 Oct 10, 2025
@AddyM AddyM changed the title Improve error messages for insufficient future_covariates in backtest() and fix residuals() bug 2846 Improve error messages for insufficient future_covariates in backtest() and fix residuals() Oct 10, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Improve error message when future covariates are missing during backtest() call

1 participant