Skip to content

Clone and run Dask-cuDF tests #8

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 8 commits into from
Feb 19, 2025
Merged

Clone and run Dask-cuDF tests #8

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
14e0860
update to 25.04
rjzamora Feb 18, 2025
bbed756
add dask-cudf tests
rjzamora Feb 18, 2025
6734e16
use --no-deps
rjzamora Feb 18, 2025
9c441f4
fix typo
rjzamora Feb 18, 2025
11ddc07
rol back --no-deps change for now
rjzamora Feb 18, 2025
4841888
Merge remote-tracking branch 'upstream/main' into test-dask-cudf
TomAugspurger Feb 19, 2025
c85bd73
Fixups
TomAugspurger Feb 19, 2025
898911c
Merge remote-tracking branch 'upstream/main' into test-dask-cudf
TomAugspurger Feb 19, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/cron.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Based off https://github.com/rapidsai/cudf/blob/branch-25.02/.github/workflows/pandas-tests.yaml
# Based off https://github.com/rapidsai/cudf/blob/branch-25.04/.github/workflows/pandas-tests.yaml
name: Test dask-upstream

on:
Expand Down Expand Up @@ -28,7 +28,7 @@ jobs:
needs: setup
# run the Dask and Distributed unit tests
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@branch-25.02
uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@branch-25.04
with:
# This selects "ARCH=amd64 + the latest supported Python + CUDA".
matrix_filter: map(select(.ARCH == "amd64")) | group_by(.CUDA_VER|split(".")|map(tonumber)|.[0]) | map(max_by([(.PY_VER|split(".")|map(tonumber)), (.CUDA_VER|split(".")|map(tonumber))]))
Expand Down
15 changes: 13 additions & 2 deletions scripts/run.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,16 @@ pip install --extra-index-url=https://pypi.anaconda.org/rapidsai-wheels-nightly/
"scipy" \
"dask-cuda"


# Clone cudf repo (need dask_cudf tests)
CUDF_VERSION="branch-25.04"
echo "Cloning cudf@{CUDF_VERSION}"

if [ ! -d "cudf" ]; then
git clone https://github.com/rapidsai/cudf.git --branch $CUDF_VERSION
Copy link
Contributor

@TomAugspurger TomAugspurger Feb 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Something I'm trying to think through: the commit we get the nightly wheel from will potentially not match the commit we test against, whenever there are commits merged between the time the nightly wheel build runs and the time this code runs.

If there are any changes to the tests that require new code in dask_cudf, then they'll fail here.

Ideally we'd checkout the exact commit the nightly wheel was built against, but I haven't found an easy way to do that yet.

Copy link
Member Author

@rjzamora rjzamora Feb 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If there are any changes to the tests that require new code in dask_cudf, then they'll fail here.

Aha - Great point. I don't think that's a "show stopper", but we definitely want to use the same commit if we can find a way to do that.

Copy link
Contributor

@TomAugspurger TomAugspurger Feb 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, not a blocker.

fi


echo "Installing dask@{DASK_VERSION}"

# depth needs to be sufficient to reach the last tag, so that the package
Expand All @@ -36,8 +46,9 @@ fi

# Install everything, including any new dependencies
pip uninstall dask distributed
pip install -e ./dask[test]
pip install -e ./distributed
pip install pandas[test] pytest pytest-cov pytest-mock pytest-rerunfailures pytest-timeout pytest-xdist pre-commit
pip install --no-deps -e ./dask
pip install --no-deps -e ./distributed

echo "[Setup done]"
pip list
Expand Down
8 changes: 7 additions & 1 deletion scripts/test.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,12 @@
#!/usr/bin/env bash
# SPDX-FileCopyrightText: Copyright (c) 2023-2025, NVIDIA CORPORATION & AFFILIATES.

echo "[testing dask-cudf]"
pushd cudf/python/dask_cudf
pytest dask -v
dask_cudf_status=$?
popd

echo "[testing dask]"
pushd dask
pytest dask -v -m gpu
Expand All @@ -13,7 +19,7 @@ pytest distributed -v -m gpu --runslow
distributed_status=$?
popd

if [ $dask_status -ne 0 ] || [ $distributed_status -ne 0 ]; then
if [ $dask_cudf_status -ne 0 ] || [ $dask_status -ne 0 ] || [ $distributed_status -ne 0 ]; then
echo "Tests faild"
exit 1
fi