Advice on transitioning large source code base from v2.8 to v3.x

[Artem-B]

I need to migrate a large code base from CCCL 2.8 to 3.1. The problem is that it can't be done at once. The sources are maintained by different teams, some of the sources are external projects, and even if we do get them all to have patches ready, by the time all of them are done, the patches will likely be out of date because of the ongoing work.
The practical reality is that the source code must remain buildable with both v2.8 and v3.1 until transition is complete.

https://nvidia.github.io/cccl/cccl/3.0_migration_guide.html lists a lot of things that have changed and are likely to break things. On my toy build attempts a lot of things, as expected, do break. Now I need to figure out how to transition multiple users that currently use v2.8 to v3.1, one at a time. The best (but terrible) idea I've came up so far is the good old preprocessing based on CUB/THRUST/CCCL version macros. While it's OK for small projects, that's going to be borderline unmanageable in my case.

I wonder if it would be feasible to provide some sort of compatibility shim that would allow current v2.8 users to continue working in the builds that use v3.1, and then let individual teams migrate to the native v3.1 APIs on their own schedule. Sprinkling uniform #include <cccl_v28_compat.h> and decoupling the users' own code upgrade from the build-wide CCCL upgrade would make things work much better.

Perhaps there are other approaches I should consider.

Thoughts? Ideas?

@jrhemstad @miscco

[jrhemstad]

Hey @Artem-B. Thanks for starting this discussion.

I totally empathize with needing to support unfortunate situations like that 😅. Just one of those unglamorous things that come up in the realities of software engineering.

To help us even begin to start wrapping our head around what this would take, is there any way you could give us an idea of what % of the surface area of CCCL we're talking about?

Maybe to start could you get us a list of all the unique thrust/cub/cuda symbols used across the various projects you're supporting?

[miscco]

We had some shims in place for rapids to work with the unfortunate situation that some new features were not yet in place but got deprecated immediately.

#if CCCL_MAJOR_VERSION >= 3
using cuda::maximum;
using cuda::minimum;
#else
using thrust::maximum;
using thrust::minimum;
#endif

They were recently removed here

From my experience, the most common issues were indeed the functors that we replaced with cuda::std ones. The only bigger problem here was cuda::{maximum, minimum} as that only came in in 3.0

Also one of the biggest issue we had were uses of the experimental memory resource, that changed heavily due to being experimental and us deciding on new names. So if a project uses rmm downstream there is some work ahead

[miscco]

That said, if there is anything you need from our side I am happy to collaborate / brainstorm ideas

[Artem-B]

if there is anything you need from our side I am happy to collaborate / brainstorm ideas

Thanks. I'll pick a representative project next week and see what breaks. I should have more specific questions to ask then.

So far I've tried my luck with RMM and that didn't go well. It appears that the very promising rapidsai/rmm#2017 got further tweaked by rapidsai/rmm#2113 and now requires CCCL v3.1. :-(

I vaguely recall that in the past CUB had a way to place it in its own namespace. If there's a way for different versions of CCCL to co-exist in the same executable, that may provide a gradual transition path from the old version to the new one, or even let some old un- or poorly-maintained projects to remain on v2.8.

[bdice]

We generally only support a single CCCL version in each RAPIDS release, which includes RMM.

RMM has used experimental CCCL memory resource features for several years, and CCCL ABI stability guarantees do not extend to these experimental features. Parts of the memory resource code in CCCL are becoming stable in CCCL 3.1 and 3.2 (currently in development), so RMM's decision to require CCCL >=3.1 (and soon >=3.2) is driven by the need for RMM to have CCCL's major-version ABI stability guarantees going forward. I am hopeful that RMM will be compatible with a wider range of CCCL versions at build time at some point after CCCL 3.2, but I expect we will still only build and test a single version of CCCL. There are also a number of API breaks in RMM 25.12 (and I expect similarly for 26.02) as we migrate towards the structure of the new CCCL memory resources.

Here is the CCCL support matrix for recent/upcoming RAPIDS releases, which may help you in this migration:

• RAPIDS 25.06 used CCCL 2.8.3
• RAPIDS 25.08 used CCCL 3.0.2
• RAPIDS 25.10 officially used CCCL 3.0.2 and also unofficially supported building with CCCL 3.1.
• RAPIDS 25.12 will use CCCL 3.1.x, but won't be able to support 3.2 yet (not yet released).
• RAPIDS 26.02 will hopefully migrate to CCCL 3.2 - not set in stone yet, but I am hoping to work on this. I believe 3.2 will stabilize all the CCCL components we depend on in RMM, and will hopefully allow RMM to float more freely to versions after 3.2.

[Artem-B]

Thank you for the timeline. It's useful. We're currently just few commits shy of 25.08 appear to be buildable with CCCL v2.8.

Given my hard requirement that stuff must be buildable with both v2.8 and v3.1, existence of works-with-one-CCCL-version-only packages means that I will need to have multiple versions of those packages, and select the right one, depending on CCCL version used by the build. In the worst case this kind of replication may need to be done for other downstream dependencies. E.g. if RAPIDS libraries introduce breaking changes of their own, their users would need version-dependent changes or versioned copies, too.

[miscco]

There is no way to use different versions of CCCL at the same time. We had a never ending flood of bugs arising from accidentally mixing different versions.

I would assume that reverting rapidsai/rmm#2113 should be reasonably possible (at least internally) as it only drops conditional compilation.

@bdice what is your stance on allowing older CCCL or was there some other issue there?

[Artem-B]

There is no way to use different versions of CCCL at the same time. We had a never ending flood of bugs arising from accidentally mixing different versions.

Yup. Been there, done that. Both mixing, and fixing. :-/

To think of it, I could probably get by (or, at least minimize the impact) with a guaranteed lack of naming conflicts/ODR violations between two versions on the ABI level, as long as the build guarantees that only one version of the library is ever used for compiling a given source file, and the code does not exchange CCCL types across API calls. Sort of the level of co-existence similar to what we have between libc++ and libstdc++ on the ABI level. It will not solve all migration issues, but it may give me some wiggle room to reduce the scope of the problem a bit.

[bdice]

Responded above about CCCL/RMM compatibility. #6540 (reply in thread)

If there is anything I can help with here, please let me know!