Add Rust SDK

[dreveman]

We've implemented a Rust SDK for Perfetto at Rivos that could be upstreamed to the Perfetto repo if there's interest in having that be part of the main repo. This SDK doesn't have to be part of the upstream repo but we think that would be preferred for the following reasons:

• Rust is an increasingly popular language and having first-class support for it should be a priority.
• These bindings are implemented on top of the C ABI and use a protozero plugin similar to the C plugin to generate native Rust code for efficient encoding of protos. Having these proto bindings be up to date with latest perfetto release would be ideal.
• The "-sys" crate support both the usage of an installed shared perfetto library and "vendored" sources. The latter rely on amalgamated source for the C shared library, which is easy to support as part of the perfetto release process.

I'll crate a pull request with the bindings for more detailed review but here are some of the features:

• Use the C ABI and protos to generate safe bindings.
• Low level "-sys" crate for raw C ABI bindings.
• Higher level "perfetto" crate provides a safe interface, uses common Rust patterns, and is recommended for normal usage.
• "perfetto-derive" crate provides proc-macros to make tracing of function calls even easier. Just tag a function and entry/exit and arguments passed will be traced if a specific track event category is enabled.
• Public API fully documented, including example usage that is checked at compile time. All unsafe Rust usage is also documented.
• Includes examples and a comprehensive set of unit tests.

The pull request will have files with Rivos copyright header and doesn't include CI integration but that could be adjusted if we decide to merge this.

@primiano @LalitMaganti

[LalitMaganti]

Hey! Thanks for the context on the bug and sending us the PR! Really exciting stuff but also a whole new area for us to think about when it comes to long term maintenance of this.

Primiano and will discuss this together and also with the rest of the team and then come back to you with a comprehensive answer on how we want to deal with.

[dreveman]

Hey! Thanks for the context on the bug and sending us the PR! Really exciting stuff but also a whole new area for us to think about when it comes to long term maintenance of this.

Primiano and will discuss this together and also with the rest of the team and then come back to you with a comprehensive answer on how we want to deal with.

Sounds good. Thanks for considering this.

[LalitMaganti]

Update: had an initial discussion with Primiano. We've set up a team meeting on Friday to discuss how we want to handle both this PR and future "weighty" external contributions like this in the future.

I will take some notes of the meeting and share them here after that happens. I'll also try and write up a doc with our policy as well (similar to our committer doc at https://perfetto.dev/docs/contributing/become-a-committer)

[dreveman]

Thanks for the update. Sounds like a good plan.

[primiano]

Hey we discussed this extensively today in our weekly, here's a summary.

First of all, wow, this is an incredible contribution! Thanks so much. I don't speak rust but I skimmed through it and seems very well written.

TL;DR we'd love to proceed with this in a /contrib folder. We'd exclude for now /contrib from Android Tree builds and google3. There are some open questions below, but no big deal breaker. Let us know what you think.

Now, more in details:

Team responsibility / expectations

The biggest fear this triggered is the fact that very few people in our team speak Rust, and the ones that do don't have a ton of professional experience with it. So one of the fear is that if this gets used within Google or Android, and one day there is some large scale change / some push due to some changes in those environemts etc, we would be committing ourselves to have to be on call on something we are not very familiar with.
But all in all we think this can be mitigated with:

• Moving this in a contrib/ folder. See RFC-0007: Community Maintained Code #3404 which highlights what that entails, but overall boils down to lowering expectations on how much our teams is responsible for it.
• Not exposing contrib/ to Android or Google3 so we won't get business-critical dependencies on something we don't own/control. This is pragmatically the most important aspect for us.
• There's an open question: many projects in Android are written in Rust, and it's possible that once this lands various teams will ask to use this SDK in Android's Rust. we will cross that bridge when we get there. I think we will ask that either somebody from Google becomes a co-maintainer and take the responsibility of dealing with android-bearing needs.

So I'd say: can you move this under contrib/rust-sdk/?

CI coverage

• I'd not block the CL on this, but I think the right approach here would be:
• We can set up and pay the cost of a CI bot (we use Github Actions) that covers building/testing with rust
• We would make the bot FYI only and not block PRs on it.
• We'll try to not break rust, but I'd like to avoid scenarios where somebody in our team is trying to cut a release and now we are blocked on rust not working.

Who will release the crates and when?

There is an open question on who will own/release the rust crates. There are various aspects to consider here:

On one hand it would be nice if the rust releases were in sync with our monthly(ish) release process (https://perfetto.dev/docs/contributing/sdk-releasing)

OTOH that might add extra work for the team.
If Google were to release the crates, then we would have to be involved in each PR to the rust-sdk as we'd be accountable for it. But that goes against the model of having an external maintainer for it and doesn't scale as a solution. Also you wouldn't want to wait for our codereview queues to get stuff done.
But if we are hands off on contrib/, then we shouldn't release crates for code we didn't review.

We'd prefer if you owned the rust crates releases. It would be nice, but you don't have to keep in sync with our releases.
Note that there is an empty https://crates.io/crates/perfetto that somebody created. Maybe reach out to the owner to see if they are willing to give back the crate? otherwise we'll have to pick another name.

Autogenerated protos
I know one of the reason why you want this in contrib/ is to keep the pz.rs in sync.
FYI we decided to move to an eventually-consistent model where:
proto-generated sources are still checked into the repo
we will no longer require that proto-generated code is in sync on every CL (it is making CLs very noisy and annoying for people)

We will have some bot that on a daily/weekly basis will regenerated the various proto-dependent files
People can still run tools/gen_xxx if they make a change that has runtime dependencies for their CL, as it is today

libperfetto_c

I see that you created a libperfetto_c amalgamation for the vendored code. We were just discussing it recently (independently of rust) if we should have an amalgamation for the C SDK.
Feels like this add one more argument for doing so.
@ddiproietto was planning to take care of this (although he is leaving the team in a few days)

FYI we libperfetto_c.{h,cc} will exist only on releases branches like it's the case for the amalgamated SDK

trace_packet.pb.rs

There is only one thing I don't fully like about the PR, it is this the presence of this:
gpu_counter_event: GpuCounterEvent, msg, 52,
gpu_render_stage_event: GpuRenderStageEvent, msg, 53,
gpu_log: GpuLog, msg, 63,

I understand why you did it (That's what you need), but that feels somewhat arbitrary. Is that TracePacket going to grow every time somebody needs a new proto?

To be completely honest this is not a new problem and we have the same issue already with the C SDK, so you are not making the problem any worse.
Our team should invest some time to figure out a way to split:
The core proto-generated classes that are required for perfetto to work (e.g. ClockSnapshot, track event, sequence flags)
The other protos that other might want to use (Without generating a monster binding file which is inconvenient for everybody)

Question for you:
would it be possible to split that TracePacket into TracePacket_Core and TracePacket_GPU ? I don't understand enough of rust and didn't look to deep in the way your codegen works to tell.

[dreveman]

Thanks for discussing this and creating RFC-0007. Makes sense and looks good to me.

I've updated the PR according to the RFC in addition to some other changes:

• Added a top-level Rust SDK README.md with build instructions and and overview of the SDK
• Made the vendored build feature default as my plan is to publish crates with that as it makes it much easier to use.
• README.md and cargo file changes needed to prepare for crate publishing.

I also dropped the Gpu protos for now. I agree that including that is very arbitrary. For my use case it's easy to handle the need for those downstream. It might require a small patch to the trace_packet.pz.rs for now but I'll think more about how that can be avoided in the future. Ideally we allow more proto bindings to be provided by other crates. e.g. we can have a perfetto-protos-gpu crate for gpu data source related protos.

A bot that will regenerated the various proto-dependent files sounds good. That will remove the need to do that manually for the rust SDK.

I'm happy to own Rust crate publishing. I'll try to keep this up to date with perfetto releases. Shouldn't be hard as with the latest changes to the PR I really just need to run cargo publish for each crate.

WRT CI coverage; What you suggested sgtm and I'll work on getting that running as a follow up.