Python: drop betterproto dep/deprecate Python bindings entirely?

[woodruffw]

betterproto is now unmaintained, per danielgtaylor/python-betterproto#673. There's an upcoming fork of it, but it (1) looks pretty young, and (2) considers itself a "major redesign," which we probably don't want to hitch ourselves to immediately.

This puts the Python bindings in a pretty tough spot: there aren't that many good Protobuf parsers/libraries out there for Python that satisfy our requirements (idiomatic modern Python 3, minimal native deps, type hints, etc.).

Given that, I think we have two options:

1. Soft-deprecate the Python bindings (i.e. sigstore-protobuf-specs on PyPI) and move forwards with a "pure JSON" modeling approach, i.e. with handwritten models. This will be a little tedious but shouldn't be that bad for Python in particular, since we only use a subset of the protobufs anyways.
2. Conduct a search for a new long-term Protobuf library for the Python bindings (one that's pure Python, idiomatic, has type hints, etc.). Google's proto-plus looks really nice, but it's a wrapper around the official protobuf and therefore brings in a native C build.

CC @haydentherapper @jku for thoughts 🙂

[woodruffw]

xref sigstore/sigstore-python#1049 as well for the sigstore-python side.

[haydentherapper]

Do you have any plans to use gRPC? If no, (1) seems like a fine approach, though I don't have a sense of how much work that is.

Is it OK to continue to rely on the deprecated library? Are there features missing from the library or have we seen a lot of bugs such that we'd want an actively maintained library? Of course longer term we should move off it, just thinking about the priority of this.

[woodruffw]

Do you have any plans to use gRPC? If no, (1) seems like a fine approach, though I don't have a sense of how much work that is.

I think the plan is to stay on REST/HTTP as long as we can retain feature parity that way 🙂 -- I'm not opposed to gRPC per se, but IIUC it'd fully commit us to a dependency on protobuf (i.e. that exact dep, not just any library we want), which would put it the API + native deps situation I'm trying to avoid as much as possible.

(I think the amount of work involved isn't too bad -- I've been experimenting with it for a day or so, and I think it'll only take me another ~day to port over the parts that sigstore-python needs.)

Is it OK to continue to rely on the deprecated library?

I think it's probably not the worst thing, although betterproto has a handful of well-known bugs that users of sigstore-python will probably hit eventually. I think the bigger medium-term risk is that we use betterproto's Pydantic model generation, and the API will eventually begin to diverge there as Pydantic makes breaking changes.

[jku]

My hunch is option 1 is not awful for reasons William listed.

Is it OK to continue to rely on the deprecated library? Are there features missing from the library or have we seen a lot of bugs such that we'd want an actively maintained library? Of course longer term we should move off it, just thinking about the priority of this.

I don't think this needs to block releases but it is something I'd like to see in near future rather than just "long term"... betterproto has been a bit of a problem for a while, cutting losses seems like the right call.