Fixed the cache diffusion ci/cd

[jingyu-ml]

What does this PR do?

Type of change: Bug fix

Overview: Fixed the cache diffusion CI/CD issue related to Torch 2.9.

Usage

pytest tests/examples/diffusers/test_cache_diffusion.py::test_sdxl_benchmarks -v -s

Testing

Before your PR is "Ready for review"

• Make sure you read and follow Contributor guidelines and your commits are signed.
• Is this change backward compatible?: Yes/No
• Did you write any new necessary tests?: Yes/No
• Did you add or update any necessary documentation?: Yes/No
• Did you update Changelog?: Yes/No

Additional Information

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 74.64%. Comparing base (263b2b7) to head (787ca10).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #620      +/-   ##
==========================================
- Coverage   74.65%   74.64%   -0.02%     
==========================================
  Files         183      183              
  Lines       18542    18542              
==========================================
- Hits        13843    13840       -3     
- Misses       4699     4702       +3     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.