[SPARK-54218][PYTHON][SQL][TESTS] Add integrated tests for Scalar Pandas Iterator UDF #52937
Conversation
dongjoon-hyun
changed the title
…calar Pandas Iterator UDF" This reverts commit 81be5fb.
zhengruifeng
force-pushed
the
reapply_pandas_iter_test
branch
from
fedb012 to
d201425
Compare
|
cc @pan3793 |
|
@zhengruifeng do you have any modifications on this? the CI seems to be stable after your previous reverting. |
no, the PR is exactly the same as the previous one |
dongjoon-hyun
left a comment
dongjoon-hyun
left a comment
There was a problem hiding this comment.
+1, LGTM.
Since this is only for Apache Spark 4.2.0, we can merge this PR according to the PR builder's success result.
For the other CIs, we can monitor the stability.
|
thanks, merged to master |
|
@pan3793 @dongjoon-hyun Not sure why it is related to this PR, I cannot reproduce it in my local, I am going to revert it again... |
|
reverted in 21c3122 |
|
this MIGHT (not sure) indicate the VM gets killed due to OOM
|
|
That's too bad. Anyway, thank you for testing and reverting. |
3 participants
Reapply #52916
What changes were proposed in this pull request?
Add integrated tests for Scalar Pandas Iterator UDF
Why are the changes needed?
to improve test coverage
Many UDF types are only tested in python side, and are missing in the SQL side. So
if the results (tested in python) are the same but the plan was unintentionally changed, we will not be aware of such change.
This PR adds SQL_SCALAR_PANDAS_ITER_UDF
Does this PR introduce any user-facing change?
no, test-only
How was this patch tested?
ci
Was this patch authored or co-authored using generative AI tooling?
no