New implementation of dependecy resolution for bionic packages #26910
Conversation
|
Hey @termux/termux-packages-maintainers, please provide your insights and decide whether seperation of Personally, I do not wish to maintain code we aren't concerned with. |
|
I would like to suggest the names |
|
I’d prefer not to see the implementations for bionic and glibc diverge, since maintaining separate branches would make things harder to keep in sync. |
| @@ -0,0 +1,50 @@ | |||
| # Documentation for `new_buildorder.py` | |||
|
|
|||
| This script is used to determine the order in which to build the dependencies of a given package. It performs a topological sort (post-order Depth First Traversal) on the dependency graph to generate a linear order. | |||
There was a problem hiding this comment.
Regarding the docs folder and this .md file here, maybe this should be carefully formatted to match the directory structure and markdown syntax template used by most of the files in this other site/pages/en/projects/docs folder PR, here?
Personally I think it's ok for the glibc-specific |
| __log() { | ||
| local msg="$1" | ||
| [[ "$TERMUX_QUIET_BUILD" != "true" ]] && echo "$msg" | ||
| } |
There was a problem hiding this comment.
Seems like it could be moved to https://github.com/termux/termux-packages/blob/master/scripts/utils/termux/package/termux_package.sh
| ( | ||
| cd "$TERMUX_COMMON_CACHEDIR-$DEP_ARCH" | ||
| if [[ "$TERMUX_REPO_PKG_FORMAT" == "debian" ]]; then | ||
| # Ignore topdir `.`, to avoid possible permission errors from tar | ||
| ar p "${PKG}_${DEP_VERSION}_${DEP_ARCH}.deb" "data.tar.xz" | \ | ||
| tar xJ --no-overwrite-dir --transform='s#^.$#data#' -C / | ||
| elif [[ "$TERMUX_REPO_PKG_FORMAT" == "pacman" ]]; then | ||
| tar -xJf "${PKG}-${DEP_VERSION_PAC}-${DEP_ARCH}.pkg.tar.xz" \ | ||
| --anchored --exclude=.{BUILDINFO,PKGINFO,MTREE,INSTALL} \ | ||
| --force-local --no-overwrite-dir -C / | ||
| fi | ||
| ) |
There was a problem hiding this comment.
In the case if you split get_dependencies step into two separate implementations you could at least move common code blocks into separate termux_step_get_dependencies_common.sh file to avoid duplication and make maintaining easier.
|
Oh, actually it seems like you took some parts of #24698. |
What do you mean by codes that don't concern you? |
According to my calculations, when I consider all code in the termux project and apply my opinion to the situation, I believe that we can think of those as "pre-build-dependencies", i.e. any build-only dependencies listed in them are considered to be mandatory prerequisites for building any and all termux-packages both on-device and in the termux-package-builder, and should actually not be specified unnecessarily in termux-packages/scripts/setup-ubuntu.sh Line 83 in 94d4d3b termux-packages/scripts/setup-termux.sh Line 46 in 94d4d3b (@Maxython this might be a good time for me to mention that As a result, after performing the above calculations of considering the status of --- a/packages/librsvg/build.sh
+++ b/packages/librsvg/build.sh
@@ -7,7 +7,7 @@ TERMUX_PKG_SRCURL=https://download.gnome.org/sources/librsvg/${TERMUX_PKG_VERSIO
TERMUX_PKG_SHA256=bc1bbcd419120b098db28bea55335d9de2470d4e6a9f6ee97207b410fc15867d
TERMUX_PKG_AUTO_UPDATE=true
TERMUX_PKG_DEPENDS="fontconfig, freetype, gdk-pixbuf, glib, harfbuzz, libcairo, libdav1d, libpng, libxml2, pango"
-TERMUX_PKG_BUILD_DEPENDS="g-ir-scanner, valac"
+TERMUX_PKG_BUILD_DEPENDS="g-ir-scanner"
TERMUX_PKG_BREAKS="librsvg-dev"
TERMUX_PKG_REPLACES="librsvg-dev"
TERMUX_PKG_VERSIONED_GIR=false |
|
With that explanation out of the way, I should mention that this PR currently would fail to build but this is in fact, in my opinion, not a real problem with this PR itself, but is actually this --- a/packages/librsvg/build.sh
+++ b/packages/librsvg/build.sh
@@ -7,7 +7,7 @@ TERMUX_PKG_SRCURL=https://download.gnome.org/sources/librsvg/${TERMUX_PKG_VERSIO
TERMUX_PKG_SHA256=bc1bbcd419120b098db28bea55335d9de2470d4e6a9f6ee97207b410fc15867d
TERMUX_PKG_AUTO_UPDATE=true
TERMUX_PKG_DEPENDS="fontconfig, freetype, gdk-pixbuf, glib, harfbuzz, libcairo, libdav1d, libpng, libxml2, pango"
-TERMUX_PKG_BUILD_DEPENDS="g-ir-scanner"
+TERMUX_PKG_BUILD_DEPENDS="g-ir-scanner, xorgproto"
TERMUX_PKG_BREAKS="librsvg-dev"
TERMUX_PKG_REPLACES="librsvg-dev"
TERMUX_PKG_VERSIONED_GIR=false(most distros, for example Arch Linux https://archlinux.org/packages/extra/x86_64/libx11/ resolve this by simply making |
|
@MrAdityaAlok I continued testing and I can report what seems like might be an actual problem with the PR in its current form, but which might hopefully be possible to fix with relatively few changes, but I'm not sure. This command: will currently produce this error if this PR is used: The presence of Also, more precisely, when building reverse dependencies of this should mean that in the Do you know what the correct changes to support this test case might be? |
Thanks! That was handled in the previous implementation of mine but I added wrong check in the |
|
|
Personally, I do not wish to maintain code that isn’t relevant to our scope. By “code that doesn’t concern us,” I mean packages outside the Termux infrastructure (specifically, non-bionic packages). If glibc packages are officially merged into This is just my personal opinion. Others might differ. |
This is an utterly meaningless justification for refusing to implement the features and capabilities of the original/full-fledged
I have a more ideal proposal that would make the package builder "neutral." You're unhappy with the lack of glibc packages in |
|
Either it's governed as part of the Org and we maintain it as such, which is not currently the case. Or glibc-packages and termux-pacman are second party projects, like the TUR, and you need to make sure you have what build infrastructure you need for it in the separate project. We can certainly make some accommodations to make things easier for you. This isn't a matter of anyone being against Glibc support, it's your project and you've been mostly running it yourself separately from the Org. |
|
@Maxython It's not that I do not want to implement it. It's just that this repo isn't concerned with your project. The build system we maintain should only be designed for our purpose or can accommodate changes that allows others to implement their own. We shouldn't implement the needs of everyone. Understand that you want me to accommodate changes for a packaging system we do not handle. Suppose someday someone else decides to support some other packaging format and they too want us to accommodate their changes. It is not feasible and it will add unnecessary burden to us. That said, I would like to propose that you keep a copy of the current build system in your project and only merge changes that concern your project. This way we both have less of a thing to worry about. Also, this will make your project independent of changes we do here. |
|
I have a good idea, I will try to build some glibc-packages using this new buildorder.py without initially changing very much in either glibc-packages or the new buildorder.py, and I will report back what the first problem seems to me to be. That might help to narrow down the specific problems which would prevent the use of this buildorder.py as-is for glibc-packages. |
Why do you think the
Okay, I don't argue with you on this point. Just answer me one question: why is your purpuse more important than the issue of preserving capabilities and ensuring equality between building bionic and glibc packages? (a response like "
Which changes??? I just ask you to keep some of the functionality that is in the original
What do you mean by "unnecessary burden to us"? It's just that, as someone (i.e., a termux and termux-pacman employee) who maintains pacman, glibc, and also ensures that previous implementations and features are compatible with new ones, I'm not familiar with that concept.
I'm sorry, but I have to tell you no. I don't support the continued support of glibc and bionic packages in this format. So, expect me to make the necessary changes to your new |
|
Seems like @MrAdityaAlok has made sure building packages is still possible for bionic and glibc, so as I see it the PR does not break anything or make performance worse than today. Personally I never build for glibc, and I don't think we want to make it mandatory for package maintainers and infrastructure maintainers to always test their changes for both bionic and glibc builds, so the type of split as done here seems fine to me.
Capabilities are preserved here, right? Aditya has preserved current script for glibc to use.
For someone who is not used to building glibc packages: which functionality are you referring to specifically that is missing in new buildoder.py, that glibc-packages needs?
Sounds great, simplifying scripts and improving maintainability in termux-packages is what we all want, lets review that PR when it is ready. Would be great if glibc could use the new buildorder.py script as well, but it is probably best if the people that regularly build for glibc implements and tests it. Sounds like @robertkirkman has already started testing, so I'm sure we'll have something to review in the near future! |
|
@Maxython I tried implementing support for Cycles
Cycle: ['gcc-libs-glibc', 'doxygen-glibc', 'gcc-libs-glibc']
Cycle: ['gcc-libs-glibc', 'libisl-glibc', 'libgmp-glibc', 'gcc-libs-glibc']
Cycle: ['gcc-libs-glibc', 'binutils-glibc', 'libbz2-glibc', 'bash-glibc', 'ncurses-glibc', 'gcc-libs-glibc']
Cycle: ['gcc-libs-glibc', 'binutils-glibc', 'libdebuginfod-glibc', 'libcurl-glibc', 'krb5-glibc', 'libverto-glibc', 'libevent-glibc', 'openssl-glibc', 'gcc-libs-glibc']
Cycle: ['gcc-libs-glibc', 'binutils-glibc', 'libdebuginfod-glibc', 'libcurl-glibc', 'libpsl-glibc', 'libidn2-glibc', 'gcc-libs-glibc']
Cycle: ['gcc-libs-glibc', 'binutils-glibc', 'libdebuginfod-glibc', 'libcurl-glibc', 'brotli-glibc', 'gcc-libs-glibc']
Cycle: ['gcc-libs-glibc', 'binutils-glibc', 'libdebuginfod-glibc', 'libelf-glibc', 'zstd-glibc', 'gcc-libs-glibc']
Cycle: ['attr-glibc', 'gettext-glibc', 'libacl-glibc', 'attr-glibc']
Cycle: ['glib-glibc', 'gobject-introspection-glibc', 'libgirepository-glibc', 'glib-glibc']Note: Above cycles exists when building in like: ❯ ./scripts/new_buildorder.py python-glibc
Cycle: ['python-glibc', 'llvm-glibc', 'python-glibc']
Cycle: ['python-glibc', 'llvm-glibc', 'libxml2-glibc', 'python-glibc']
Cycle: ['python-glibc', 'llvm-glibc', 'binutils-libs-glibc', 'libelf-glibc', 'libcurl-glibc', 'krb5-glibc', 'e2fsprogs-glibc', 'util-linux-glibc', 'python-glibc']
Cycle: ['libacl-glibc', 'attr-glibc', 'gettext-glibc', 'libacl-glibc'] |
|
Topological sorting was implemented to actually catch these real cycles so that they could be fixed and packages be buildable from scratch, like for bootstraps. I don't know the issues for glibc and why these cycles need to exist, and wasn't testing them. |
MrAdityaAlok
force-pushed
the
new-buildorder
branch
3 times, most recently
from
74885df to
ecffd6b
Compare
Signed-off-by: Aditya Alok <[email protected]>
Signed-off-by: Aditya Alok <[email protected]>
Signed-off-by: Aditya Alok <[email protected]>
Signed-off-by: Aditya Alok <[email protected]>
Signed-off-by: Aditya Alok <[email protected]>
Signed-off-by: Aditya Alok <[email protected]>
Signed-off-by: Aditya Alok <[email protected]>
Signed-off-by: Aditya Alok <[email protected]>
Signed-off-by: Aditya Alok <[email protected]>
Signed-off-by: Aditya Alok <[email protected]>
Signed-off-by: Aditya Alok <[email protected]>
Signed-off-by: Aditya Alok <[email protected]>
Signed-off-by: Aditya Alok <[email protected]>
MrAdityaAlok
force-pushed
the
new-buildorder
branch
from
ecffd6b to
c8a6c19
Compare
@robertkirkman it's fixed now. You may test further. |
basically I know about that too, but I found them using a slightly different method. there is technically a way to find and debug them without actually editing the buildorder.py and just using the current one, so I am doing it that way first and when I finish I will make my PR to glibc-packages to fix everything. |
|
Because it was implemented for CI and in the case if single file is not available waiting for other jobs to be completed is pointless. |
|
Now |
Instead of adding a new variable you could have just returned a But anyway, that is not my point. This PR is to identify cyclic dependencies and fix them. Your check of |
No, I meant why can't we first download all packages and later build packages that failed to download, instead of killing the job in middle. |
Thanks for the advice, I will study your method and if it works, I will use it.
This circumvention of circular dependencies partially makes sense because the (root) package where the circular dependency "begins" is guaranteed to compile. I'll note that I haven't fully explored the algorithms of your new
What do you mean by "fix them"? If you mean eliminating cyclic dependencies, then I have a question: do you have a roadmap for further development of your PR, and if so, what are the goals of that roadmap? I asked this question to properly understand your approach to cyclic dependencies and what you want to do about them.
If @robertkirkman can find and remove useless circular dependencies in glibc-packages without limiting packages or making them more complex, that would be great. However, I'll say right away that he definitely won't be able to remove all circular dependencies, as some of them are necessary or very complex/large to fix. |
How does it make sense and how is root package guaranteed to compile? Suppose a package A that depends upon another package B which in turn directly or indirectly depends upon A then how is the compilation of A guaranteed here?
Yes, I meant eliminate them. I do not think any further roadmap is needed for Cyclic dependencies are like chicken and egg problem. There will be a package from which everything is derived. Maybe that absolute package also depends upon some of the derived packages. So this can be resolved by first building the absolute package with some of it features disabled (called bootstrapping) and then building the fully functional package from it. This is generally how most compilers are compiled. |
|
The following question arose: have you studied my PR #20513 in detail (about the implementations I propose to solve problems due to cyclic dependencies and how they work)? |
I didn't originally want to make overly optimistic claims about my abilities, especially before I have finished implementing the necessary changes and before my PR is ready, but since the discussion has moved along rapidly here, I can provide assurance that I believe I will, in fact, be able to do it, at least in a way that satisfies me personally. I'm not completely sure whether my PR will be acceptable to everyone here once I finish it, but I hope it will be. |
No, I didn't and I don't think they bring any improvement for Besides, if you have to bypass circular dependency check in the new Also, I request you to not implement any further bypassing methods. Until and unless you have a proper way of topologically sorting |
I urge you to carefully review my PR, as it applies to both bionic and glibc packages. But most importantly, I see a conflict between us regarding the resolution of cyclic dependencies (your idea: prohibit cyclic dependencies, and if they arise, fix/eliminate them; my idea: exploit (or normalize) cyclic dependencies and analyze them so that packages can be built safely). I recommend immediately paying attention to the concept of "virtual packages" in my PR, which allow cyclic dependencies to be resolved by compiling an incomplete package - 51c089d |
But why add unnecessary complexity when it can be easily solved? I'm sure no bionic packages requires that and for glibc Robert has a way to fix them. So, I suggest to wait until he does and then switch to this new implementation. |
|
From discussion on matrix, Different build orders can be kept in our repo for now instead of glibc repo keeping its own variant in its repo. Just use I agree with @MrAdityaAlok that package manager and library specific code should be abstracted away so that whoever wants to implement a new format can manage their own implementations without us having to, and would be a good design. But that will require lot of work and can be looked into in future and I don't think it should be part of this pull, it should be worth it in the end, despite some duplication that will be required. |
|
I don't want to throw a wrench into agnostic-apollo's summary, which is mostly accurate, but since I have now been studying glibc-packages for a little while, I know a small detail that I feel that I should clarify in response:
I am now aware that there is at least one other glibc-package that currently has a dependency on a package from the termux-packages repository. It is |
|
Ah, thanks for that. But to be honest, glibc packages depending on bionic packages is not an issue anyways. Any bionic packages can be built beforehand and uploaded to repo, or if building bootstraps, just add build_package entries for any bionic packages first. Additionally, glibc algo, i.e current buildorder is already compatible for building bionic packages, it just doesn't catch cycles and is less efficient than new buildorder/topological sorting. So even if some glibc package depends on say bash, and glibc buildorder is engaged, bash should still get built successfully anyways. Mixing both orders would have only been necessary if a bionic package depended on a glibc package, which will never happen to be a concern. Additionally having separate files for both buildorders would mean that changes to glibc buildorder would not require approval or involvement of all the bionic devs, and only glibc devs, which will help prevent blockers for glibc and actually be good for Max/glibc devs. Of course code quality and proper documentation MUST BE maintained by both bionic and glibc devs for their variants. |
|
My repo.json changes pull also moves a lot of the apt and pacman specific code to their own files, I can look into abstracting and separating more of it away and use dynamically generated function names at runtime to invoke the current package manager function, similar changes should be possible for bionic and glibc separation too. |
It's highly debatable whether something is simpler or more complex. Comparing your PR with my PR, it's certainly true that your idea is simpler to implement. However, your idea also involves fixing cyclic dependencies by compiling incomplete packages within packages, which is a complication compared to my PR, as your idea doesn't provide algorithms for compiling packages within packages, meaning you'll have to do it all yourself. (Now imagine fixing 10-20 cyclic dependencies using this method; you'll probably be tempted to optimize it all.) Comparing my PR with yours, it's certainly true that my idea is more complex to implement, but at the same time, my idea offers tools that make it easy to fix/avoid/resolve cyclic dependencies.
Keep in mind that bionic packages evolve and grow in size, meaning they can also easily acquire cyclic dependencies, including large and complex ones. |
7 participants
Goals of new
buildorder.py:metaandsubpackagesANTI_DEPENDSONLINE(-I) modeCurrently only single cycle exists which is really a cyclic dependency:
This implementation significantly reduces the number of packages downloaded in CI.
Here is an example:
Closes: #19779
Closes: #20338