Embedded Universal DSL: a good DSL for us, by us

This repository contains the source code for EUDSL, a toolkit for the construction of embedded DSLs, in arbitrary languages, for targeting MLIR.

FYI: this project is currently "alpha" quality.

Currently, there are five components:

1. eudsl-tblgen: Python bindings to LLVM's TableGen library;
2. eudsl-nbgen: A source-to-source translator that translates MLIR headers¹ into direct nanobind bindings;
3. eudsl-py: Direct Python bindings to MLIR, generated using eudsl-nbgen;
   • Currently only TableGen outputs (various *.h.inc files generated by mlir-tblgen) are automatically translated but the eudsl-nbgen tool can be manually run to translate any MLIR header (to varying degrees of success);
   • See projects/eudsl-py/tests for a few examples of what these bindings look like;
4. mlir-python-bindings: Scripts/CMake for nightly builds of the upstream MLIR Python bindings;
   • Available via pip install mlir-python-bindings -f https://llvm.github.io/eudsl.
5. eudsl-python-extras: Extra utilities for the upstream MLIR Python bindings;
   • Available via pip install eudsl-python-extras -f https://llvm.github.io/eudsl if used with the upstream bindings.
   • In reality, this package is meant to work in concert with any set of "host bindings" (some distribution of the actual MLIR Python bindings). So that means the pip install should be amended to

     $ EUDSL_PYTHON_EXTRAS_HOST_PACKAGE_PREFIX=<YOUR_HOST_MLIR_PYTHON_PACKAGE_PREFIX> \
           pip install eudsl-python-extras -f https://llvm.github.io/eudsl

     where EUDSL_PYTHON_EXTRAS_HOST_PACKAGE_PREFIX is (as it says) the package prefix for your chosen host bindings. When in doubt about this prefix, it is everything up until ir when you import your bindings, e.g., in import torch_mlir.ir, torch_mlir is the EUDSL_PYTHON_EXTRAS_HOST_PACKAGE_PREFIX for the torch-mlir bindings.

Getting started

Python wheels of all the tools are available at the eudsl release page. They are also pip install-able with .e.g

$ pip install eudsl-py -f https://llvm.github.io/eudsl

eudsl-py has a slowly growing set of tests @ projects/eudsl-py/tests.

If you don't want to install locally, here is a colab notebook minimal working example.

Developing/building

Strong recommendation: check the CI scripts @ .github/workflows - they do a fresh checkout and build on every commit and are written to be read by a non-CI expert.

Firstly, you need a distribution of LLVM. You can build LLVM from source using our submodule by doing (on Mac/Linux or mingw):

$ cd <EUDSL_CHECKOUT_DIR>
$ ./scripts/build_llvm.sh

Alternatively you can download a distribution for your platform from the llvm release page.

Then each of the above tools can both be built as a conventional, standalone, CMake project and as a Python wheel. The wheel build looks something like:

$ cd <EUDSL_CHECKOUT_DIR>
$ export CMAKE_PREFIX_PATH=$PWD/llvm-install 
$ pip wheel projects/eudsl-nbgen -w wheelhouse -v
$ pip wheel projects/eudsl-py -w wheelhouse -v --find-links $PWD/wheelhouse

Note, the trailing --find-links $PWD/wheelhouse on pip wheel projects/eudsl-py is because eudsl-nbgen is a dependency of eudsl-py (that can be satisfied using the eudsl-nbgen wheel).

If you want to build an individual tool via CMake you can do something like:

$ cd <EUDSL_CHECKOUT_DIR>
$ pip install -r requirements.txt
$ export CMAKE_PREFIX_PATH=$PWD/llvm-install 
$ cmake -B $PWD/eudsl-nbgen-build -S $PWD/projects/eudsl-nbgen -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=$PWD/eudsl-nbgen-install
$ cmake --build "$PWD/eudsl-build" --target install

If you want to build all the tools at once using CMake you can use the root CMakeLists.txt. Note, in this case, eudsl-nbgen will automatically be built prior to eudsl-py.

If you want to build all the tools at once using CMake directly against third_party/llvm-project then (roughly) the CMake invocation is:

cmake -GNinja -B build -S <EUDSL_CHECKOUT_DIR>/third_party/llvm-project/llvm \
  -DLLVM_EXTERNAL_PROJECTS=EUDSL \
  -DLLVM_EXTERNAL_EUDSL_SOURCE_DIR=<EUDSL_CHECKOUT_DIR> \
  -DLLVM_BUILD_LLVM_DYLIB=ON \
  -DLLVM_ENABLE_PROJECTS=clang;mlir

Footnotes

Footnotes

1. Yes C++ headers... ↩