remove j4rs dep by default

Author: wangxiaoying

.github/workflows/release.yml

@@ -64,33 +64,6 @@ jobs:
         env:
           SQLITE3_STATIC: 1
 
-      - name: Copy j4rs dependencies into dist
-        run: |
-          cp -rf connectorx-python/target/release/jassets connectorx-python/connectorx/dependencies
-
-      # rebuild the wheel to incorporate j4rs dependencies
-      - uses: PyO3/maturin-action@v1
-        with:
-          rust-toolchain: stable
-          command: build
-          args: -m connectorx-python/Cargo.toml -i python --release --manylinux 2_28 --features integrated-auth-gssapi
-          before-script-linux: |
-            # If we're running on rhel centos, install needed packages.
-            if command -v yum &> /dev/null; then
-                yum update -y && yum install -y perl-core openssl openssl-devel pkgconfig libatomic
-
-                # If we're running on i686 we need to symlink libatomic
-                # in order to build openssl with -latomic flag.
-                if [[ ! -d "/usr/lib64" ]]; then
-                    ln -s /usr/lib/libatomic.so.1 /usr/lib/libatomic.so
-                fi
-            else
-                # If we're running on debian-based system.
-                apt update -y && apt-get install -y libssl-dev openssl pkg-config
-            fi
-        env:
-          SQLITE3_STATIC: 1
-
       - uses: actions/upload-artifact@v4
         with:
           name: "ubuntu-latest-${{ matrix.python-version[1] }}"
@@ -139,19 +112,6 @@ jobs:
   #       env:
   #         SQLITE3_STATIC: 1
 
-  #     - name: Copy j4rs dependencies into dist
-  #       run: |
-  #         cp -rf connectorx-python/target/release/jassets connectorx-python/connectorx/dependencies
-
-  #     # rebuild the wheel to incorporate j4rs dependencies
-  #     - uses: PyO3/maturin-action@v1
-  #       with:
-  #         rust-toolchain: 1.71.1
-  #         command: build
-  #         args: -m connectorx-python/Cargo.toml --target aarch64-unknown-linux-gnu -i python --release --manylinux 2_28 --features integrated-auth-gssapi
-  #       env:
-  #         SQLITE3_STATIC: 1
-
   #     - uses: actions/upload-artifact@v3
   #       with:
   #         name: "aarch-${{ matrix.python-version[1] }}"
@@ -203,20 +163,6 @@ jobs:
         env:
           SQLITE3_STATIC: 1
 
-      - name: Copy j4rs dependencies into dist
-        run: |
-          cp -r connectorx-python/target/release/jassets connectorx-python/connectorx/dependencies
-
-      # rebuild the wheel to incorporate j4rs dependencies
-      - uses: PyO3/maturin-action@v1
-        with:
-          rust-toolchain: stable
-          maturin-version: v0.14.15
-          command: build
-          args: -m connectorx-python/Cargo.toml -i python --release ${{ matrix.features }}
-        env:
-          SQLITE3_STATIC: 1
-
       - uses: actions/upload-artifact@v4
         with:
           name: "${{ matrix.os }}-${{ matrix.python-version }}"
@@ -264,20 +210,6 @@ jobs:
         env:
           SQLITE3_STATIC: 1
 
-      - name: Copy j4rs dependencies into dist
-        run: |
-          cp -rf connectorx-python/target/aarch64-apple-darwin/release/jassets connectorx-python/connectorx/dependencies
-
-      # rebuild the wheel to incorporate j4rs dependencies
-      - uses: PyO3/maturin-action@v1
-        with:
-          rust-toolchain: stable
-          maturin-version: v0.14.15
-          command: build
-          args: -m connectorx-python/Cargo.toml --target aarch64-apple-darwin -i python --release  --features integrated-auth-gssapi
-        env:
-          SQLITE3_STATIC: 1
-
       - uses: actions/upload-artifact@v4
         with:
           name: "macos-arm-${{ matrix.python-version }}"

Federation.md

@@ -2,11 +2,16 @@
 
 We use [accio](https://github.com/sfu-db/accio) to rewrite a federated query into multiple single-source ones, and combine the result locally with [datafusion](https://github.com/apache/datafusion).
 
-To enable query federation for connectorx:
-1. Clone accio: `[email protected]:sfu-db/accio.git`.
-2. Build accio: `cd accio/rewriter && mvn package -Dmaven.test.skip=true`.
-3. Move the jar file to location `${YOUR_LOCAL_PYTHON_PATH}/site-packages/connectorx/dependencies/federated-rewriter.jar`
-4. Configure accio and set the configuration path as `FED_CONFIG_PATH`. Example configurations can be found [here](https://github.com/sfu-db/accio/tree/main/benchmark/config/tpch10_datafusion/10gbit).
-5. Run federated query using connectorx!
+
+Currently, this feature is only enabled by building connectorx from source code as follows:
+
+1. Clone connectorx: `git clone [email protected]:sfu-db/connector-x.git`.
+2. Build connectorx from source follows the [instruction](https://sfu-db.github.io/connector-x/install.html#build-from-source-code). Note, for the final step, build wheel with `build-python-wheel-fed` command instead of `build-python-wheel`.
+3. Install connectorx: `pip install ${YOUR_LOCAL_CONNECTORX_PATH}/connectorx-python/target/wheels/${YOUR_WHEEL_FILE}`.
+4. Clone accio: `git clone [email protected]:sfu-db/accio.git`.
+5. Build accio: `cd accio/rewriter && mvn package -Dmaven.test.skip=true`.
+6. Move the jar file to location `${YOUR_LOCAL_PYTHON_PATH}/site-packages/connectorx/dependencies/federated-rewriter.jar`
+7. Configure accio and set the configuration path as `FED_CONFIG_PATH`. Example configurations can be found [here](https://github.com/sfu-db/accio/tree/main/benchmark/config/tpch10_datafusion/10gbit).
+8. Run federated query using connectorx!
 
 Alternatively, accio provides wrappers that can directly run federated queries on various query engines. In particular, it uses connectorx as the data fetching method when datafusion or polars are the federation engine. For more details, check out [here](https://github.com/sfu-db/accio).

Justfile

@@ -100,6 +100,10 @@ benchmark-report: setup-python
 
 # releases
 build-python-wheel:
+    cd connectorx-python && maturin build --release -i python
+
+# release with federation enabled
+build-python-wheel-fed:
     # need to get the j4rs dependency first
     cd connectorx-python && maturin build --release -i python
     # copy files