Add CANN support

[ericcurtin]

We want to add CANN as a first class citizen of DMR.

Initial Dockerfile suggestion:

# syntax=docker/dockerfile:1

ARG CANN_VERSION=8.0.0-910b
ARG CANN_IMAGE_VARIANT=openeuler22.03

FROM quay.io/ascend/cann:{CANN_VERSION}-{CANN_IMAGE_VARIANT}-py3.10 AS builder

ARG TARGETARCH
ARG CANN_IMAGE_VARIANT

RUN apt-get update && apt-get install -y cmake ninja-build git build-essential curl

WORKDIR /llama-server

COPY .git .git
COPY native/CMakeLists.txt .
COPY native/src src
COPY native/vendor vendor

# Fix submodule .git file to point to correct location in container
RUN echo "gitdir: ../../.git/modules/native/vendor/llama.cpp" > vendor/llama.cpp/.git && \
    sed -i 's|worktree = ../../../../../native/vendor/llama.cpp|worktree = /llama-server/vendor/llama.cpp|' .git/modules/native/vendor/llama.cpp/config

RUN echo "-B build \
    -DCMAKE_BUILD_TYPE=Release \
    -DBUILD_SHARED_LIBS=ON \
    -DGGML_BACKEND_DL=ON \
    -DGGML_CPU_ALL_VARIANTS=ON \
    -DGGML_NATIVE=OFF \
    -DGGML_OPENMP=OFF \
    -DGGML_CANN=ON \
    -DLLAMA_CURL=OFF \
    -GNinja \
    -S ." > cmake-flags
RUN cmake $(cat cmake-flags)
RUN cmake --build build --config Release
RUN cmake --install build --config Release --prefix install

RUN rm install/bin/*.py
RUN rm -r install/lib/cmake
RUN rm -r install/lib/pkgconfig
RUN rm -r install/include

FROM scratch AS final

ARG TARGETARCH
ARG CANN_VERSION

COPY --from=builder /llama-server/install /com.docker.llama-server.native.linux.cann$CANN_VERSION.$TARGETARCH

[ericcurtin]

@leo-pony interested? Does this Dockerfile look correct? Is CANN both x86_64/aarch64?

[leo-pony]

@leo-pony interested? Does this Dockerfile look correct? Is CANN both x86_64/aarch64?

@ericcurtin Yes, I'm very happy to receive the development invitation. I'm starting CANN support today. This Dockerfile doesn't seem to have any major issues. I'll upgrade the CANN version in this Dockerfile and perform verification.

CANN has both x86_64 and aarch64 versions. We can support CANN aarch64 first, or support both simultaneously.

Also, I've observed that more people prefer Ubuntu, so I'm thinking of replacing the OpenEuler Docker image in the CANN Dockerfile with Ubuntu. What do you think?

[ericcurtin]

@leo-pony interested? Does this Dockerfile look correct? Is CANN both x86_64/aarch64?

@ericcurtin Yes, I'm very happy to receive the development invitation. I'm starting CANN support today. This Dockerfile doesn't seem to have any major issues. I'll upgrade the CANN version in this Dockerfile and perform verification.

CANN has both x86_64 and aarch64 versions. We can support CANN aarch64 first, or support both simultaneously.

Also, I've observed that more people prefer Ubuntu, so I'm thinking of replacing the OpenEuler Docker image in the CANN Dockerfile with Ubuntu. What do you think?

That's fine all the images in this project so far are Ubuntu-based, this is what they look like:

#298

So feel free to edit that one.

[ericcurtin]

@leo-pony this is merged now, so a base to work off:

#298

[ericcurtin]

Here's a good reference of ROCm added recently, (this is not everything though, grep -i cuda and rocm to catch all the places):

#273

[leo-pony]

Here's a good reference of ROCm added recently, (this is not everything though, grep -i cuda and rocm to catch all the places):

#273

Thanks! I will follow #273 ROCm's modification ideas to adapt CANN support and debug it with cann.Dockerfile.

[ericcurtin]

@leo-pony there's a CANN Dockerfile in the repo now cann.Dockerfile, you can probably do a better job

[leo-pony]

@leo-pony there's a CANN Dockerfile in the repo now cann.Dockerfile, you can probably do a better job

Thanks, get it

[doringeman]

Hey @leo-pony, can you PTAL at #323?

[leo-pony]

Hey @leo-pony, can you PTAL at #323?

@doringeman
Could you try the cann.Dockerfile in PR #307, I made issues fix in this pr, and tested in my local Ascend NPU host, everything is okay.

[leo-pony]

Hey @leo-pony, can you PTAL at #323?

@doringeman
Could you try the cann.Dockerfile in PR #307, I made issues fix in this pr, and tested in my local Ascend NPU host, everything is okay.

@doringeman
However, as I need to attend meetings all day these days, I'll address the CI issue of pr307 this Thursday or Friday.