possible build improvement: not so many wheels every single time?

[lgritz]

There are times we build the wheels FOR REAL -- releases, pushing a tag before a release (we push the tag then wait for CI to pass before actually doing the "draft a release"), and the nightly build where we want the full set of wheels built.

But there are also cases where we just want to test that we haven't broken the wheel-building process, but don't care about the wheels themselves as artifacts, such as ordinary pushes of the cherry-picks we are making to the dev-3.0 branch that eventually will be the next patch release, and also ordinary PRs where we touch the Python bindings in some way and want to double check we haven't accidentally broken wheel-building.

Now for the latter class of cases, do we really need each of these pushes or PRs to launch >30 wheel-building jobs that exercise every permutation, or is building just one wheel for each platform enough to confirm we haven't broken anything? And if so, what's the expedient way to fix the job to achieve this?

[lgritz]

To be even more precise, I think we have four levels of wheel need from CI:

1. Publish to PyPI -- only for tagged supported-branch releases from the official repo.
2. Generate all wheels for verification and as artifacts -- when pushing release tags (precursor to drafting a release), and for nightly builds, and changes to the wheel.yml workflow.
3. Make sure we haven't broken the wheel building -- on PRs or pushes that change the python bindings, and pushes to the release dev branch (i.e. as we cherry pick things into dev-3.0). Maybe only a subset of wheels, perhaps one per major platform, is enough for this?
4. No wheel building needed at all -- for most ordinary PRs or pushes that don't change anything about the python bindings.

Note that tier 3 does not currently exist. Those cases fall under 2, leading to ~36 jobs to build every possible flavor of python wheel.

[EmilDohne]

I think this is a good compromise, we wont run the risk of having broken wheels as the nightly jobs will pick up any issues and tracking down which commit broke it is fairly straightforward since there isnt too many commits per day.

I think this would be a good step to reduce the number of builds!

[lgritz]

"Assigning" to @zachlewis because I know he's messing with the wheel building at the moment and wanted to remind him, in case implementing any of this is especially convenient while he's making other changes. These are all "good to have" but not emergencies by any stretch.