Skip to content

Commit 7f0cc23

Browse files
rgommersrgommers
committed
DOC: add documentation about using shared libraries
1 parent 30e9656 commit 7f0cc23

File tree

2 files changed

+255
-0
lines changed

2 files changed

+255
-0
lines changed

‎docs/how-to-guides/shared-libraries.rst‎

Lines changed: 254 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,254 @@
1+
.. SPDX-FileCopyrightText: 2024 The meson-python developers
2+
..
3+
.. SPDX-License-Identifier: MIT
4+
5+
.. _shared-libraries:
6+
7+
**********************
8+
Using shared libraries
9+
**********************
10+
11+
Python projects may build shared libraries as part of their project, or link
12+
with shared libraries from a dependency. This tends to be a common source of
13+
issues, hence this page aims to explain how to include shared libraries in
14+
wheels, any limitations and gotchas, and how support is implemented in
15+
``meson-python`` under the hood.
16+
17+
We distinguish between *internal* shared libraries that are built as part of
18+
the project, and *external* shared libraries that are provided by project
19+
dependencies and that are linked with the project build artifacts.
20+
For internal shared libraries, we also distinguish whether the shared library
21+
is being installed to its default system location (typically
22+
``/usr/local/lib`` on Unix-like systems, and ``C:\\lib`` on Windows)
23+
or to a location in ``site-packages`` within the Python package install tree.
24+
All these scenarios are (or will be) supported, with some caveats:
25+
26+
+-----------------------+------------------+---------+-------+-------+
27+
| shared library source | install location | Windows | macOS | Linux |
28+
+=======================+==================+=========+=======+=======+
29+
| internal | libdir | no (1) | ✓ | ✓ |
30+
+-----------------------+------------------+---------+-------+-------+
31+
| internal | site-packages | ✓ | ✓ | ✓ |
32+
+-----------------------+------------------+---------+-------+-------+
33+
| external | n/a | ✓ (2) | ✓ | ✓ |
34+
+-----------------------+------------------+---------+-------+-------+
35+
36+
.. TODO: add subproject as a source
37+
38+
1: Internal shared libraries on Windows cannot be automatically handled
39+
correctly, and currently ``meson-python`` therefore raises an error for them.
40+
`PR meson-python#551 <https://github.com/mesonbuild/meson-python/pull/551>`__
41+
may improve that situation in the near future.
42+
43+
2: External shared libraries require ``delvewheel`` usage on Windows (or
44+
some equivalent way, like amending the DLL search path to include the directory
45+
in which the external shared library is located). Due to the lack of RPATH
46+
support on Windows, there is no good way around this.
47+
48+
.. _internal-shared-libraries:
49+
50+
Internal shared libraries
51+
=========================
52+
53+
A shared library produced by ``library()`` or ``shared_library()`` built like this
54+
55+
.. code-block:: meson
56+
57+
example_lib = shared_library(
58+
'example',
59+
'examplelib.c',
60+
install: true,
61+
)
62+
63+
is installed to ``libdir`` by default. If the only reason the shared library exists
64+
is to be used inside the Python package being built, then it is best to modify
65+
the install location to be within the Python package itself:
66+
67+
.. code-block:: python
68+
69+
install_path: py.get_install_dir() / 'mypkg/subdir'
70+
71+
Then an extension module in the same install directory can link against the
72+
shared library in a portable manner by using ``install_rpath``:
73+
74+
.. code-block:: meson
75+
76+
py3.extension_module('_extmodule',
77+
'_extmodule.c',
78+
link_with: example_lib,
79+
install: true,
80+
subdir: 'mypkg/subdir',
81+
install_rpath: '$ORIGIN'
82+
)
83+
84+
The above method will work as advertised on macOS and Linux; ``meson-python`` does
85+
nothing special for this case. Windows needs some special handling though, due to
86+
the lack of RPATH support:
87+
88+
.. literalinclude:: ../../tests/packages/sharedlib-in-package/mypkg/__init__.py
89+
:start-after: start-literalinclude
90+
:end-before: end-literalinclude
91+
92+
If an internal shared library is not only used as part of a Python package, but
93+
for example also as a regular shared library in a C/C++ project or as a
94+
standalone library, then the method shown above won't work - the library has to
95+
be installed to the default ``libdir`` location. In that case, ``meson-python``
96+
will detect that the library is going to be installed to ``libdir`` - which is
97+
not a recommended install location for wheels, and not supported by
98+
``meson-python``. Instead, ``meson-python`` will do the following *on platforms
99+
other than Windows*:
100+
101+
1. Install the shared library to ``<project-name>.mesonpy.libs`` (i.e., a
102+
top-level directory in the wheel, which on install will end up in
103+
``site-packages``).
104+
2. Rewrite RPATH entries for install targets that depend on the shared library
105+
to point to that new install location instead.
106+
107+
This will make the shared library work automatically, with no other action needed
108+
from the package author. *However*, currently an error is raised for this situation
109+
on Windows. This is documented also in :ref:`reference-limitations`.
110+
111+
112+
External shared libraries
113+
=========================
114+
115+
External shared libraries are installed somewhere on the build machine, and
116+
usually detected by a ``dependency()`` or ``compiler.find_library()`` call in a
117+
``meson.build`` file. When a Python extension module or executable uses the
118+
dependency, the shared library will be linked against at build time. On
119+
platforms other than Windows, an RPATH entry is then added to the built
120+
extension module or executable, which allows the shared library to be loaded at
121+
runtime.
122+
123+
.. note::
124+
125+
An RPATH entry alone is not always enough - if the directory that the shared
126+
library is located in is not on the loader search path, then the shared
127+
library may go missing at runtime. See, e.g., `meson#2121
128+
<https://github.com/mesonbuild/meson/issues/2121>`__ and
129+
`meson#13046 <https://github.com/mesonbuild/meson/issues/13046>`__ for
130+
issues this can cause.
131+
132+
If this happens, workarounds include adding the directory that the shared
133+
library is located in to the search path by amending the ``LD_LIBRARY_PATH``
134+
environment variable, or by using a compile flag ``-Wl,-rpath,/path/to/sharedlib/dir``.
135+
136+
On Windows, the shared library can either be preloaded, or vendored with
137+
``delvewheel`` in order to make the built Python package usable locally.
138+
139+
140+
Publishing wheels which depend on external shared libraries
141+
-----------------------------------------------------------
142+
143+
On all platforms, wheels which depend on external shared libraries usually need
144+
post-processing to make them usable on machines other than the one on which
145+
they were built. This is because the RPATH entry for an external shared library
146+
contains a path specific to the build machine. This post-processing is done by
147+
tools like ``auditwheel`` (Linux), ``delvewheel`` (Windows), ``delocate``
148+
(macOS) or ``repair-wheel`` (any platform, wraps the other tools).
149+
150+
Running any of those tools on a wheel produced by ``meson-python`` will vendor
151+
the external shared library into the wheel and rewrite the RPATH entries (it
152+
may also do some other things, like symbol mangling).
153+
154+
On Windows, the package author may also have to add the preloading like shown
155+
above with ``_enable_sharedlib_loading()`` to the main ``__init__.py`` of the
156+
package, ``delvewheel`` may or may not take care of this (please check its
157+
documentation if your shared library goes missing at runtime).
158+
159+
Note that we don't cover using shared libraries contained in another wheel
160+
and depending on such a wheel at runtime in this guide. This is inherently
161+
complex and not recommended (you need to be in control of both packages, or
162+
upgrades may be impossible/breaking).
163+
164+
165+
Using libraries from a Meson subproject
166+
=======================================
167+
168+
It can often be useful to build a shared library in a
169+
`Meson subproject <https://mesonbuild.com/Subprojects.html>`__, for example as
170+
a fallback in case an external dependency isn't detected. There are two main
171+
strategies for folding a library built in a subproject into a wheel built with
172+
``meson-python``:
173+
174+
1. Build the library as a static library instead of a shared library, and
175+
link it into a Python extension module that needs it.
176+
2. Build the library as a shared library, and either change its install path
177+
to be within the Python package's tree, or rely on ``meson-python`` to fold
178+
it into the wheel when it'd otherwise be installed to ``libdir``.
179+
180+
Option (1) tends to be easier, so unless the library of interest cannot be
181+
built as a static library or it would inflate the wheel size too much because
182+
it's needed by multiple Python extension modules, we recommend trying option
183+
(1) first.
184+
185+
A typical C or C++ project providing a library to link against tends to provide
186+
(a) one or more ``library()`` targets, which can be built as shared, static, or both,
187+
and (b) headers, pkg-config files, tests and perhaps other development targets
188+
that are needed to use the ``library()`` target(s). One of the challenges to use
189+
such projects as a subproject is that the headers and other installable targets
190+
are targeting system locations (e.g., ``<prefix>/include/``) which isn't supported
191+
by wheels and hence ``meson-python`` errors out when it encounters such an install
192+
target. This is perhaps the main issue one encounters with subproject usage,
193+
and the following two sections discuss how options (1) and (2) can work around
194+
that.
195+
196+
Static library from subproject
197+
------------------------------
198+
199+
The major advantage of building a library target as static and folding it directly
200+
into an extension module is that no targets from the subproject need to be installed.
201+
To configure the subproject for this use case, add the following to the
202+
``pyproject.toml`` file of your package:
203+
204+
.. code-block:: toml
205+
206+
[tool.meson-python.args]
207+
setup = ['--default-library=static']
208+
install = ['--skip-subprojects']
209+
210+
This ensures that ``library`` targets are built as static, and nothing gets installed.
211+
212+
To then link against the static library in the subproject, say for a subproject
213+
named ``bar`` with the main library target contained in a ``bar_dep`` dependency,
214+
add this to your ``meson.build`` file:
215+
216+
.. code-block:: meson
217+
218+
bar_proj = subproject('bar')
219+
bar_dep = bar_proj.get_variable('bar_dep')
220+
221+
py.extension_module(
222+
'_example',
223+
'_examplemod.c',
224+
dependencies: bar_dep,
225+
install: true,
226+
)
227+
228+
That is all!
229+
230+
Shared library from subproject
231+
------------------------------
232+
233+
If we can't use the static library approach from the section above and we need
234+
a shared library, then we must have ``install: true`` for that shared library
235+
target. This can only work if we can pass some build option to the subproject
236+
that tells it to *only* install the shared library and not headers or other
237+
targets that we don't need. Install tags don't work per subproject, so
238+
this will look something like:
239+
240+
.. code-block:: meson
241+
242+
foo_subproj = subproject('foo',
243+
default_options: {
244+
# This is a custom option - if it doesn't exist, can you add it
245+
# upstream or in WrapDB?
246+
'only_install_main_lib': true,
247+
})
248+
foo_dep = foo_subproj.get_variable('foo_dep')
249+
250+
Now we can use ``foo_dep`` like a normal dependency, ``meson-python`` will
251+
include it into the wheel in ``<project-name>.mesonpy.libs`` just like an
252+
internal shared library that targets ``libdir`` (see
253+
:ref:`internal-shared-libraries`).
254+
*Remember: this method doesn't support Windows (yet)!*

‎docs/index.rst‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -82,6 +82,7 @@ the use of ``meson-python`` and Meson for Python packaging.
8282
how-to-guides/config-settings
8383
how-to-guides/meson-args
8484
how-to-guides/debug-builds
85+
how-to-guides/shared-libraries
8586
reference/limitations
8687
projects-using-meson-python
8788

0 commit comments

Comments
 (0)