How to deal with CI that machines are changing

[marcosschroh]

Hi,

Thanks for this library. This is not an issue, is more than a question about how to deal when either when a python version changes or a OS machine changes.

We are running during the CI the command pytest tests/test_benchmarks.py --benchmark-compare --benchmark-compare-fail=mean:10% which it compares the result against the previous benchmark, the problem is that sometimes the machines are not the same per PRs, then the command fails due ERROR: --benchmark-compare-fail requires valid --benchmark-compare. Any ideas how to bypass the error as we can not control the machines types during the CI? How the file to compare is selected?

[pkit]

Double check that your --benchmark-compare actually points to a path that is resolvable in Github Actions.
We had that problem when it was not.
For Github Actions the path that benchmark runs in is the root of the repo.

[SchoolGuy]

I am using dedicated GitHub Runners that are managed by Hostinger's Fireactions. These bare-metal runners are in my DC and are exclusively reserved for running the performance testsuite. This seems to work rather well.

Otherwhise I have realized that with a bigger number of iterations these fluctuations seem to be smoothed out for a lot of test-types.