Can't cross compile

[europaul]

Hi,

I'm trying to cross-compile vector to ARM64
I enter the environment from the projet root via make environment
I run make build-aarch64-unknown-linux-musl, but get the following strange error:

root@build-server:/git/vectordotdev/vector# make build-aarch64-unknown-linux-musl
make -k cross-image-aarch64-unknown-linux-musl
make[1]: Entering directory '/git/vectordotdev/vector'
docker build \
	--tag vector-cross-env:aarch64-unknown-linux-musl \
	--file scripts/cross/aarch64-unknown-linux-musl.dockerfile \
	.
[+] Building 0.9s (9/9) FINISHED                                                                                                                                                                                                                              
 => [internal] load build definition from aarch64-unknown-linux-musl.dockerfile                                                                                                                                                                          0.0s
 => => transferring dockerfile: 616B                                                                                                                                                                                                                     0.0s
 => [internal] load metadata for ghcr.io/cross-rs/aarch64-unknown-linux-musl:0.2.5                                                                                                                                                                       0.6s
 => [internal] load .dockerignore                                                                                                                                                                                                                        0.0s
 => => transferring context: 145B                                                                                                                                                                                                                        0.0s
 => [internal] load build context                                                                                                                                                                                                                        0.0s
 => => transferring context: 211B                                                                                                                                                                                                                        0.0s
 => [1/4] FROM ghcr.io/cross-rs/aarch64-unknown-linux-musl:0.2.5@sha256:702154f52b2d8091671aa2c84d5582d849f949977228c735ff8462f93cc0e1e4                                                                                                                 0.0s
 => CACHED [2/4] COPY scripts/cross/bootstrap-ubuntu.sh scripts/environment/install-protoc.sh /                                                                                                                                                          0.0s
 => CACHED [3/4] RUN /bootstrap-ubuntu.sh && bash /install-protoc.sh                                                                                                                                                                                     0.0s
 => CACHED [4/4] RUN mkdir -p /lib/native-libs && cp /usr/local/aarch64-linux-musl/lib/libstdc++.a /lib/native-libs/                                                                                                                                     0.0s
 => exporting to image                                                                                                                                                                                                                                   0.0s
 => => exporting layers                                                                                                                                                                                                                                  0.0s
 => => writing image sha256:2763df290c781b9c5ecd6f443578a2079efaa8402d5533eb66095310b53d2abb                                                                                                                                                             0.0s
 => => naming to docker.io/library/vector-cross-env:aarch64-unknown-linux-musl                                                                                                                                                                           0.0s
make[1]: Leaving directory '/git/vectordotdev/vector'
cross build \
	--release \
	--target aarch64-unknown-linux-musl \
	--no-default-features \
	--features target-aarch64-unknown-linux-musl
info: syncing channel updates for '1.86-x86_64-unknown-linux-gnu'

  1.86-x86_64-unknown-linux-gnu unchanged - rustc 1.86.0 (05f9846f8 2025-03-31)

Error: 
   0: `docker inspect ''` failed with exit status: 1

Stderr:
   Error: No such object:

Stdout:
   []
make: *** [Makefile:286: target/aarch64-unknown-linux-musl/release/vector] Error 1

I tried it on two different machines - the error is the same. Is my setup maybe wrong?

I would really appreciate any hints 🙏

[thomasqueirozb]

Hi @europaul, this is likely due to using make environment since it drops you inside a docker container. cross is likely causing this issue since it's trying to run docker inspect. If you run make build-aarch64-unknown-linux-musl from your host machine you should be able to cross compile with no issues.

If you hit any issues running the make command, ./scripts/environment/prepare.sh will install any missing tools

[europaul]

Hi @thomasqueirozb, thank you for the answer! The make target you mentioned indeed works outside of the environment container. I found out by following the steps from the cross.yml GHA workflow.

I also found out what was causing the issue mentioned above: cross is relying on docker to fetch the cross compilation container, so the first thing it does is trying to find out if it's being run inside a container by looking at hostname. However in my case the code that's supposed to set it correctly in entrypoint.sh

#!/usr/bin/env bash

# set HOSTNAME to container id for `cross`
HOSTNAME="$(head -1 /proc/self/cgroup|cut -d/ -f3)"
export HOSTNAME

exec "$@"

doesn't work and sets HOSTNAME to an empty string. And of course docker inspect '' fails.

As far as I understood the environment container was also not build to support cross compiling vector. Or was it? In that case I think I can fix scripts/environment/entrypoint.sh to solve this bug.

[thomasqueirozb]

AFAIK nobody has tried to cross compile using the test environment in a while but I don't see a reason why it cannot be fixed. Thanks for pointing this out! I have done some work on this but it's still not compiling inside make environment but I'm going to make a PR for this either way and you can try to build on top of this and submit a follow up PR if you'd like

Note that cross compiling from/to certain architectures is not supported by cross so it may be that even if this is fully fixed make build-aarch64-unknown-linux-musl may not work in your machine, but this is a separate issue.

(I think that docker removed the container id from /proc/self/cgroup a while ago and this issue flew under the radar.)

Update: PR here

[europaul]

thanks @thomasqueirozb, I didn't know that! looks like a good fix. But tbh even after that I still cannot cross-compile from inside the container. Does that work for you?

[thomasqueirozb]

Nope, cross compiling from inside the container is still broken as I mentioned in my previous comment. I keep getting an wrap-rustc unused error. This is probably some weird cross behavior which is likely resolved by using an argument. Tracking this down is a bit of a nightmare and git bisecting would require cherry picking my partial fix commit and then trying to run make environment and then make build-aarch64-unknown-linux-musl from inside the container. And even when you find the offending commit (which may not exist due to different docker versions) the issue might live inside of cross source code. It's a rabbit hole.

I'd be happy to accept any contributions that fix this issue though so feel free to submit a PR.

[thomasqueirozb]

I also feel like we should close this discussion and create an issue from this since this is a bug that I can replicate locally

[thomasqueirozb]

Created #23479