PEP 780: Update to use sys.abi_info

peps/pep-0780.rst

@@ -8,7 +8,7 @@ Status: Draft
 Type: Standards Track
 Topic: Packaging
 Created: 21-Mar-2025
-Python-Version: 3.14
+Python-Version: 3.15
 Post-History:
   `05-Aug-2024 <https://discuss.python.org/t/60007>`__,
   `26-Mar-2025 <https://discuss.python.org/t/86013>`__,
@@ -22,15 +22,16 @@ Post-History:
 Abstract
 ========
 
-This PEP defines using ABI features as environment markers for project
-dependencies, through a new ``sys_abi_features`` environment marker. :pep:`508`
-(later moved to :ref:`packaging:dependency-specifiers`) introduced environment
-markers to specify dependencies based on rules that describe when the
-dependency should be used. This PEP extends the environment markers to allow
-specifying dependencies based on specific ABI features of the Python
-interpreter. For this, it defines a set of `ABI Features`_ and specifies how
-they are made available for `environment markers <pep-780-sys_abi_features_>`_
-as a new marker variable, ``sys_abi_features``.
+This PEP defines the use of information about the interpreter ABI as
+environment markers for project dependencies through a new ``sys_abi_info``
+environment marker variable. :pep:`508` (later moved to
+:ref:`packaging:dependency-specifiers`) introduced environment markers to
+specify dependencies based on rules that describe when the dependency should
+be used. This PEP extends environment markers to allow specifying dependencies
+based on specific information about the ABI of the Python interpreter. For
+this, it defines a set of `ABI Features`_ and specifies how they are made
+available for `environment markers <pep-780-sys_abi_info_>`_ via a new marker
+variable, ``sys_abi_info``.
 
 Motivation
 ==========
@@ -57,23 +58,19 @@ marker for free-threading`_ Discourse thread:
   wheels. Picking up the wrong Cython version is causing a lot of obscure build
   failures and runtime crashes. It would be beneficial if the metadata could
   express that (c.f. `Require Cython Pre-release for Free-Threaded Python`_).
-- CFFI has no support for free-threading yet, and `Armin Rigo, one of the
-  maintainers, has stated
-  <https://github.com/python-cffi/cffi/pull/143#issuecomment-2580781899>`__
-  that it may be a good idea to fork ``cffi``, implement support for
-  free-threading, and only come back to the CFFI project with a single large PR
-  that adds support after the functionality "is reasonably well-tested (either
-  as tests or, better in this case, tested by being in use in various other
-  projects)". There are a lot of projects that depend on ``cffi``. They are
-  likely fine to start depending on a fork for free-threading only, however
-  depending on a fork for >=3.13 or for all Python versions seems like a much
-  larger ask, and more disruptive for distribution packagers.
-
-While these concrete examples may be addressed later this year by Cython and
-CFFI making compatible releases, the same issue is going to repeat further up
-the stack. The free-threading rollout is expected to take several years, and an
-environment marker for free-threading will make that rollout significantly
-easier.
+- CFFI has only recently gained support for free-threading, and for a
+  significant period of time, it looked likely that support would first be
+  added in a fork. There are a lot of projects that depend on ``cffi``, which
+  are likely fine to start depending on a fork for free-threading only, however
+  depending on a fork for >=3.13 or for all Python versions seemed like a much
+  larger ask, and more disruptive for distribution packagers. While the
+  situation has been resolved with CFFI 2.0.0, other projects may face similar
+  issues.
+
+While concrete examples may be addressed with compatible releases, the same
+issue is going to repeat further up the stack. The free-threading rollout is
+expected to take several years, and an environment marker for free-threading
+will make that rollout significantly easier.
 
 Another important ABI feature that is not yet covered by environment markers is
 the bitness of the interpreter. In most cases, the ``sys_platform`` or
@@ -135,7 +132,7 @@ Relation to Other PEPs
 This PEP extends environment markers with set semantics for ABI features.
 :pep:`751#additions-to-marker-expression-syntax` includes a similar extension
 for lock file specific environment markers; although the two have been
-developed indepedently, they are compatible where they overlap in terms of the
+developed independently, they are compatible where they overlap in terms of the
 new set semantics.
 
 Specification
@@ -149,60 +146,71 @@ and "**OPTIONAL**"" in this document are to be interpreted as described in
 ABI Features
 ------------
 
-ABI features are intrinsic properties of the Python interpreter, expressed as
-simple, understandable strings. However, not all features are equally
-applicable to all Python interpreters or Python versions. For example, the
-distinction between free-threaded and GIL-enabled interpreters is only relevant
-for CPython 3.13 onwards, but the bitness of the interpreter is relevant for
-all interpreters.
+ABI features are intrinsic properties of the Python interpreter.
+However, not all features are equally applicable to all Python interpreters or
+Python versions. For example, the distinction between free-threaded and
+GIL-enabled interpreters is only relevant for CPython 3.13 onwards, but the
+bitness of the interpreter, i.e. the number of bits in a pointer, is relevant
+for all interpreters.
 
 All interpreters MUST handle the following ABI features as stated. ABI features
 that are restricted to particular interpreters MUST NOT be provided by other
-interpreters. The features are subdivided into groups and for each group there
-MUST be exactly one feature present, except when the group is marked as
-optional, in which case there MUST be at most one feature present.
+interpreters.
 
-``free-threading`` or ``gil-enabled`` (only CPython)
-    If the Python interpreter is free-threaded, the ``free-threading`` feature
-    MUST be present and the ``gil-enabled`` feature MUST NOT be present.
-    Otherwise, the ``gil-enabled`` feature MUST be present and the
-    ``free-threading`` feature MUST NOT be present.
+In Python 3.15 and later, the ABI features are determined by the new
+:external+py3.15:py:data:`sys.abi_info` attribute. In earlier Python versions,
+the ABI features are the same as for Python 3.15 and given as follows:
 
-``debug`` (only CPython, optional)
-    This ABI feature is reserved for the ``--with-pydebug`` build of CPython.
-    If the interpreter is a CPython interpreter with ``Py_DEBUG`` capabilities,
-    the ``debug`` feature MUST be present. On POSIX systems, this corresponds
-    to the Python expression ``"d" in sys.abiflags``.
+``pointer_bits``
+    The bitness of the interpreter, that is, the number of bits in a pointer,
+    typically ``32`` or ``64`` [#bitness]_.
 
-``32-bit`` or ``64-bit`` (optional)
-    The bitness of the interpreter, that is, whether it is a 32-bit or 64-bit
-    build [#bitness]_. If the bitness is unknown or neither 32-bit nor 64-bit,
-    this feature MUST NOT be present.
+``free_threaded`` (only CPython)
+    If the Python interpreter is free-threaded, ``free_threaded`` MUST be
+    ``True``, otherwise it MUST be ``False``.
 
-.. _pep-780-sys_abi_features:
+``debug`` (only CPython)
+    This ABI feature corresponds to the
+    :external+py3.15:py:attr:`sys.abi_info.debug` field.
 
-The ``sys_abi_features`` Environment Marker
--------------------------------------------
+``byteorder``
+    Indicates the byte order of the interpreter. The value MUST be either
+    ``"little"`` or ``"big"``.
 
-To make ABI features available in dependency specifications, a new environment
-marker variable, ``sys_abi_features``, is added to the format of dependency
-specifiers.
+.. _pep-780-sys_abi_info:
 
-To do this, we need to extend the grammar laid out in :pep:`508` and maintained
-in the :ref:`packaging:dependency-specifiers` and document the possible values.
+The ``sys_abi_info`` Environment Marker
+---------------------------------------
 
-The grammar is extended to include the ``sys_abi_features`` marker variable by
-augmenting the definition of ``env_var`` as follows::
+To make ABI features available in dependency specifications, a new environment
+marker variable, ``sys_abi_info``, is added to the format of dependency
+specifiers. It is built to reflect the value of ``sys.abi_info``, which is a
+new attribute of the ``sys`` module introduced in Python 3.15 as a simple
+namespace. To represent its fields in the environment marker variable, they are
+expressed as a frozen set of strings of the form ``f"{name}::{value}"``, where
+``name`` is the name of the field and ``value`` is the string representation of
+its value.
+
+The ``sys_abi_info`` Environment Variable
+'''''''''''''''''''''''''''''''''''''''''
+To do this, we need to extend the grammar laid out in :pep:`508` and maintained
+in the :ref:`packaging:dependency-specifiers` and document the possible values,
+to include the ``sys_abi_info`` marker variable by augmenting the definition of
+``env_var`` as follows::
 
     env_var       = ('python_version' | 'python_full_version' |
                      'os_name' | 'sys_platform' | 'platform_release' |
                      'platform_system' | 'platform_version' |
                      'platform_machine' | 'platform_python_implementation' |
                      'implementation_name' | 'implementation_version' |
-                     'sys_abi_features' |
-                     'extra' # ONLY when defined by a containing layer
+                     'sys_abi_info' |
+                     'extra' | 'extras' | 'dependency_groups' # ONLY when defined by a containing layer
                      )
 
+This makes ``sys_abi_info`` available as a variable in marker expressions.
+
+The Values of ``sys_abi_info``
+''''''''''''''''''''''''''''''
 Like the grammar, also the overview table of environment markers in
 :ref:`packaging:dependency-specifiers` is augmented to add the following row:
 
@@ -212,39 +220,86 @@ Like the grammar, also the overview table of environment markers in
     * - Marker
       - Python equivalent
       - Sample values
-    * - ``sys_abi_features``
-      - no direct equivalent available
-      - ``{'free-threading', '64-bit'}``,
-        ``{'gil-enabled', 'debug', '32-bit'}``
+    * - ``sys_abi_info``
+      - ``sys.abi_info``
+      - ``{'free_threaded::True', 'pointer_bits::64'}``,
+        ``{'free_threaded::False', 'debug::True', 'pointer_bits::32'}``,
+        ``{'byteorder::little', 'pointer_bits::64'}``
 
 With these additions, ABI features can be used in dependency specifications via
 the ``in`` operator to test for the presence of a feature, or the ``not in``
 operator to test for the absence of a feature.
 
+This allows for marker expressions such as::
+
+    "free_threaded::True" in sys_abi_info
+    "pointer_bits::32" not in sys_abi_info and "debug::True" in sys_abi_info
+
+Note that this is forward compatible in the sense that future additions to the
+set of ABI features will not be in conflict with the grammar, such that tools
+encountering unknown ABI features in a dependency specification will naturally
+evaluate the test as false. This empowers package authors to decide on the
+default resolution in this case. For example, a package author may decide to
+require a dependency only if an ABI feature is present, by using a presence
+test like ``"new_feature::True" in sys_abi_info``, which will evaluate to
+false on current interpreters that do not support the new feature, or an
+absence test like ``"new_feature::False" not in sys_abi_info``, which will
+evaluate to true on current interpreters.
+
+While covering all ABI features specified at the moment, this mechanism is only
+meaningfully applicable where the ABI feature is unambiguously representable as
+a string. Currently, all ABI features are either booleans, integers, or
+strings, which all satisfy this requirement.
+
+Should the future reveal a need for ABI features that are not unambiguously
+represented as strings, like floating point numbers or more complex data
+structures, this PEP would need to be revisited to address those.
+
 Examples
 ========
 
 Require Cython Pre-release for Free-Threaded Python
-----------------------------------------------------
+---------------------------------------------------
 To require a pre-release of Cython only for a free-threaded Python interpreter,
 the following dependency specification can be used::
 
-    cython >3.1.0a1; "free-threading" in sys_abi_features
-    cython ==3.0.*; "free-threading" not in sys_abi_features
+    cython >3.1.0a1; "free_threaded::True" in sys_abi_info
+    cython ==3.0.*; "free_threaded::False" in sys_abi_info
 
 Require SciPy Unless on 32-bit ``win32``
 ----------------------------------------
 To require SciPy unless on a 32-bit interpreter on Windows, the following
 dependency specification can be used::
 
-    scipy; platform_system != "Windows" or "32-bit" not in sys_abi_features
+    scipy; platform_system != "Windows" or "pointer_bits::32" not in sys_abi_info
 
 Require NumPy for a Free-Threaded Interpreter With Debugging Capabilities
---------------------------------------------------------------------------
+-------------------------------------------------------------------------
 To require NumPy only for a free-threaded interpreter with debugging
 capabilities, the following dependency can be used::
 
-    numpy; "free-threading" in sys_abi_features and "debug" in sys_abi_features
+    numpy; "free-threaded::True" in sys_abi_info and "debug::True" in sys_abi_info
+
+Require Free-Threaded Fork of PyYAML
+------------------------------------
+PyYAML initially decided not to support free-threading in its mainline
+distribution before wider testing of free-threaded Python in the real world.
+To facilitate this testing, a free-threaded fork, PyYAML-ft, was created.
+To require this fork, for a free-threaded Python interpreter, without forcing
+it also on the GIL-enabled interpreter, the following dependency specification
+can be used::
+
+    pyyaml-ft; "free_threaded::True" in sys_abi_info
+    pyyaml; "free_threaded::True" not in sys_abi_info
+
+Note that in the absence of the features described in this PEP, early adopting
+projects have had to resort to using the fork for *all* builds of a potentially
+free threading Python interpreter; typical dependencies than look like::
+
+    pyyaml; python_version<"3.13"
+    pyyaml-ft; python_version>="3.13"
+
+which is an undesirable side effect for GIL-enabled environments.
 
 Backwards Compatibility
 =======================
@@ -257,6 +312,15 @@ However, the introduction of the feature has implications for a number of
 ecosystem tools, especially those which attempt to support examination of data
 in ``pyproject.toml`` and ``requirements.txt``.
 
+In particular, current tools generally work with a fixed list of environment
+variables, either in their own implementation (e.g. `uv`_) or via an
+established package like `pypa/packaging`_ (e.g. pip).
+Hence, it is principally impossible to add any new environment markers without
+making new packages using these markers incompatible with older versions of
+these tools.
+However, once these tools support the proposed marker variable, changes in its
+contents are handled gracefully.
+
 Audit and Update Tools
 ----------------------
 
@@ -360,6 +424,8 @@ Footnotes
    particularly on platforms that allow the execution of either kind of binary.
 
 
+.. _uv: https://github.com/astral-sh/uv/blob/86e7b2e97a193ce3f506804be4c41c611191fa72/crates/uv-pep508/src/marker/tree.rs#L170
+.. _pypa/packaging: https://github.com/pypa/packaging/blob/8f13a4309aa620bf643cd508304f783b456e8c49/src/packaging/_tokenizer.py#L63
 .. _Python experimental support for free threading: https://docs.python.org/3/howto/free-threading-python.html
 .. _Python free-threading guide: https://py-free-threading.github.io/
 .. _Environment marker for free-threading: https://discuss.python.org/t/environment-marker-for-free-threading/60007