[SPARK-54190][BUILD] Guava dependency governance #52873

pan3793 · 2025-11-04T07:52:49Z

What changes were proposed in this pull request?

Remove connect.guava.version and use the unified guava.version.

Strip the unused transitive dependencies of Guava:

as mentioned in https://github.com/google/guava/wiki/UseGuavaInYourBuild

Guava has one dependency that is needed for linkage at runtime:
com.google.guava:failureaccess:

Remove shaded Guava classes from spark-connect jar (reuse shaded Guava included in spark-network-common)

Fix the shading leaks of the spark-connect-jvm-client jar

Why are the changes needed?

Simplify Guava dependency management - now Spark uses a unified Guava version everywhere.

Reduce package size, spark-connect jar becomes smaller

before (master branch)

$ ll jars/spark-connect_2.13-4.2.0-SNAPSHOT.jar
-rw-r--r--@ 1 chengpan  staff    17M Nov  5 11:23 jars/spark-connect_2.13-4.2.0-SNAPSHOT.jar

after (this PR)

$ ll jars/spark-connect_2.13-4.2.0-SNAPSHOT.jar
-rw-r--r--@ 1 chengpan  staff    13M Nov  5 12:01 jars/spark-connect_2.13-4.2.0-SNAPSHOT.jar

Fix the shading leaks for spark-connect-jvm-client jar

before (master branch)

$ jar tf jars/connect-repl/spark-connect-client-jvm_2.13-4.2.0-SNAPSHOT.jar | grep '.class$' | grep -v 'org/apache/spark' | grep -v 'org/sparkproject' | grep -v 'META-INF'
javax/annotation/CheckForNull.class
javax/annotation/CheckForSigned.class
...

after (this PR)

$ jar tf jars/connect-repl/spark-connect-client-jvm_2.13-4.2.0-SNAPSHOT.jar | grep '.class$' | grep -v 'org/apache/spark' | grep -v 'org/sparkproject' | grep -v 'META-INF'
<no-output>

Does this PR introduce any user-facing change?

Reduce potential class conflict issues for users who use spark-connect-jvm-client.

How was this patch tested?

Manually checked, see the above section.

Also, manually tested the Connect Server, and Connect JVM client via BeeLine.

$ dev/make-distribution.sh --tgz --name guava -Pyarn -Pkubernetes -Phadoop-3 -Phive -Phive-thriftserver
$ cd dist
$ SPARK_NO_DAEMONIZE=1 sbin/start-connect-server.sh

$ SPARK_CONNECT_BEELINE=1 bin/beeline -u jdbc:sc://localhost:15002 -e "select 'Hello, Spark Connect!', version() as server_version;"
WARNING: Using incubator modules: jdk.incubator.vector
Connecting to jdbc:sc://localhost:15002
Connected to: Apache Spark Connect Server (version 4.2.0-SNAPSHOT)
Driver: Apache Spark Connect JDBC Driver (version 4.2.0-SNAPSHOT)
Error: Requested transaction isolation level REPEATABLE_READ is not supported (state=,code=0)
Using Spark's default log4j profile: org/apache/spark/log4j2-defaults.properties
25/11/05 13:30:03 WARN Utils: Your hostname, H27212-MAC-01.local, resolves to a loopback address: 127.0.0.1; using 10.242.159.140 instead (on interface en0)
25/11/05 13:30:03 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
+------------------------+-------------------------------------------------+
| Hello, Spark Connect!  |                 server_version                  |
+------------------------+-------------------------------------------------+
| Hello, Spark Connect!  | 4.2.0 0ea7f5599c5dcc169b0724caa48d5530c39dbefb  |
+------------------------+-------------------------------------------------+
1 row selected (0.09 seconds)
Beeline version 2.3.10 by Apache Hive
Closing: 0: jdbc:sc://localhost:15002

Was this patch authored or co-authored using generative AI tooling?

No.

pan3793 · 2025-11-05T05:36:36Z

cc @LuciferYang @dongjoon-hyun

LuciferYang · 2025-11-05T05:55:14Z

sql/connect/client/jvm/pom.xml

              <include>io.perfmark:*</include>
              <include>org.apache.arrow:*</include>
              <include>org.codehaus.mojo:*</include>
-              <include>org.checkerframework:*</include>


Are you sure that checkerframework won't be used? Although it will be removed after Guava version 33.5.

this is documented in Guava Wiki

Guava has one dependency that is needed for linkage at runtime: com.google.guava:failureaccess:1.0.3. We created it as part of making ListenableFuture available as a separate artifact from the rest of Guava.

https://github.com/google/guava/wiki/UseGuavaInYourBuild#what-about-guavas-own-dependencies

BTW, currently, in spark-network-common, it only shades the guava and failureaccess

LuciferYang · 2025-11-05T06:28:27Z

gh pr checkout 52873   
build/mvn clean install -DskipTests -Phive
build/mvn clean install -Phive -pl sql/connect/client/jvm

WARNING: Using incubator modules: jdk.incubator.vector
[INFO] Running org.apache.spark.sql.JavaEncoderSuite
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.340 s <<< FAILURE! -- in org.apache.spark.sql.JavaEncoderSuite
[ERROR] org.apache.spark.sql.JavaEncoderSuite -- Time elapsed: 0.340 s <<< ERROR!
java.lang.NoClassDefFoundError: org/sparkproject/guava/cache/CacheLoader
	at org.apache.spark.sql.connect.test.SparkConnectServerUtils$.createSparkSession(RemoteSparkSession.scala:208)
	at org.apache.spark.sql.connect.test.SparkConnectServerUtils$.createSparkSession(RemoteSparkSession.scala:190)
	at org.apache.spark.sql.connect.test.SparkConnectServerUtils.createSparkSession(RemoteSparkSession.scala)
	at org.apache.spark.sql.JavaEncoderSuite.setup(JavaEncoderSuite.java:45)
Caused by: java.lang.ClassNotFoundException: org.sparkproject.guava.cache.CacheLoader
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:525)
	... 4 more

It seems that the changes will have a negative impact on Maven testing.

pan3793 · 2025-11-05T06:33:35Z

@LuciferYang, thanks for checking Maven case, let me take a look

pan3793 · 2025-11-05T07:49:19Z

@LuciferYang, after a deeper look, I think it's a Maven-specific testing issue (the maven mixes the target/<artifact>.jar and target/classes into classes, then causes issues on modules that produce shaded jars), and I'm afraid I don't have an immediate solution ... for now, I have to withdraw the changes of removing shaded Guava from spark-connect

the above conclusion is obsolete after #52918

pan3793 · 2025-11-05T08:00:54Z

.github/workflows/build_and_test.yml

+    permissions:
+      packages: write
+    name: Run
+    uses: ./.github/workflows/maven_test.yml


will revert after passing maven tests

sql/connect/server/pom.xml

pan3793 · 2025-11-06T02:18:20Z

@LuciferYang now the Maven CI passes https://github.com/pan3793/spark/actions/runs/19113370178 except for the existing two failed cases, caused by #52496 (comment), could you please take another look?

This reverts commit 2a42a4e.

pan3793 · 2025-11-06T18:17:04Z

since #52918 fixes the maven testing issue, let me retry removing shaded guava from connect-server

pan3793 · 2025-11-07T03:00:00Z

@LuciferYang it's feasible to remove shaded guava from connect-server

Maven CI passes (except for those2 known failed cases) https://github.com/pan3793/spark/runs/54722616100
I also manually test the case you provided, passes too

LuciferYang · 2025-11-07T03:00:50Z

sql/connect/server/pom.xml

-              </includes>
+              <shadedPattern>${spark.shade.packageName}.guava.thirdparty</shadedPattern>
            </relocation>
+


Please delete this blank line.

this is intended, the added comment about guava classes relocation is only applied for the above blocks of this line

This reverts commit 2954854.

dongjoon-hyun · 2025-11-07T03:44:42Z

since #52918 fixes the maven testing issue, let me retry removing shaded guava from connect-server

Just a question~ Is this true? #52918 was merged 14 hours ago, but the Maven builds (on master) branch still failed in the morning, didn't it?

pan3793 · 2025-11-07T03:56:36Z

@dongjoon-hyun it's another independent maven issue, you can check the details in #52918 "How was this patch tested?" section.

dongjoon-hyun · 2025-11-07T04:23:12Z

Got it. Thank you for clarification~

### What changes were proposed in this pull request? Remove `connect.guava.version` and use the unified `guava.version`. Strip the unused transitive dependencies of Guava: as mentioned in https://github.com/google/guava/wiki/UseGuavaInYourBuild > Guava has one dependency that is needed for linkage at runtime: > com.google.guava:failureaccess:<version> Remove shaded Guava classes from `spark-connect` jar (reuse shaded Guava included in `spark-network-common`) Fix the shading leaks of the `spark-connect-jvm-client` jar ### Why are the changes needed? 1. Simplify Guava dependency management - now Spark uses a unified Guava version everywhere. 2. Reduce package size, spark-connect jar becomes smaller before (master branch) ``` $ ll jars/spark-connect_2.13-4.2.0-SNAPSHOT.jar -rw-r--r-- 1 chengpan staff 17M Nov 5 11:23 jars/spark-connect_2.13-4.2.0-SNAPSHOT.jar ``` after (this PR) ``` $ ll jars/spark-connect_2.13-4.2.0-SNAPSHOT.jar -rw-r--r-- 1 chengpan staff 13M Nov 5 12:01 jars/spark-connect_2.13-4.2.0-SNAPSHOT.jar ``` 2. Fix the shading leaks for `spark-connect-jvm-client` jar before (master branch) ``` $ jar tf jars/connect-repl/spark-connect-client-jvm_2.13-4.2.0-SNAPSHOT.jar | grep '.class$' | grep -v 'org/apache/spark' | grep -v 'org/sparkproject' | grep -v 'META-INF' javax/annotation/CheckForNull.class javax/annotation/CheckForSigned.class ... ``` after (this PR) ``` $ jar tf jars/connect-repl/spark-connect-client-jvm_2.13-4.2.0-SNAPSHOT.jar | grep '.class$' | grep -v 'org/apache/spark' | grep -v 'org/sparkproject' | grep -v 'META-INF' <no-output> ``` ### Does this PR introduce _any_ user-facing change? Reduce potential class conflict issues for users who use `spark-connect-jvm-client`. ### How was this patch tested? Manually checked, see the above section. Also, manually tested the Connect Server, and Connect JVM client via BeeLine. ``` $ dev/make-distribution.sh --tgz --name guava -Pyarn -Pkubernetes -Phadoop-3 -Phive -Phive-thriftserver $ cd dist $ SPARK_NO_DAEMONIZE=1 sbin/start-connect-server.sh ``` ``` $ SPARK_CONNECT_BEELINE=1 bin/beeline -u jdbc:sc://localhost:15002 -e "select 'Hello, Spark Connect!', version() as server_version;" WARNING: Using incubator modules: jdk.incubator.vector Connecting to jdbc:sc://localhost:15002 Connected to: Apache Spark Connect Server (version 4.2.0-SNAPSHOT) Driver: Apache Spark Connect JDBC Driver (version 4.2.0-SNAPSHOT) Error: Requested transaction isolation level REPEATABLE_READ is not supported (state=,code=0) Using Spark's default log4j profile: org/apache/spark/log4j2-defaults.properties 25/11/05 13:30:03 WARN Utils: Your hostname, H27212-MAC-01.local, resolves to a loopback address: 127.0.0.1; using 10.242.159.140 instead (on interface en0) 25/11/05 13:30:03 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address +------------------------+-------------------------------------------------+ | Hello, Spark Connect! | server_version | +------------------------+-------------------------------------------------+ | Hello, Spark Connect! | 4.2.0 0ea7f55 | +------------------------+-------------------------------------------------+ 1 row selected (0.09 seconds) Beeline version 2.3.10 by Apache Hive Closing: 0: jdbc:sc://localhost:15002 ``` ### Was this patch authored or co-authored using generative AI tooling? No. Closes #52873 from pan3793/guava-govern. Authored-by: Cheng Pan <[email protected]> Signed-off-by: yangjie01 <[email protected]> (cherry picked from commit a8d128c) Signed-off-by: yangjie01 <[email protected]>

LuciferYang · 2025-11-07T08:58:02Z

Merged into master and branch-4.1. Thanks @pan3793 and @dongjoon-hyun

github-actions bot added SQL BUILD CORE CONNECT labels Nov 4, 2025

pan3793 force-pushed the guava-govern branch 2 times, most recently from 0ea7f55 to cf86e73 Compare November 5, 2025 02:38

pan3793 changed the title ~~[WIP][SPARK-XXXXX][BUILD] Guava dependency governance~~ [SPARK-54190][BUILD] Guava dependency governance Nov 5, 2025

pan3793 marked this pull request as ready for review November 5, 2025 05:32

LuciferYang reviewed Nov 5, 2025

View reviewed changes

github-actions bot added the INFRA label Nov 5, 2025

pan3793 commented Nov 5, 2025

View reviewed changes

sql/connect/server/pom.xml Outdated Show resolved Hide resolved

pan3793 force-pushed the guava-govern branch 2 times, most recently from 201306c to 837b509 Compare November 5, 2025 19:10

pan3793 added 4 commits November 6, 2025 21:08

Guava dependency governance

caea08f

connect server shades guava

2a42a4e

rm LICENSE-check-qual.txt

bc430f0

mvn ci

2954854

pan3793 force-pushed the guava-govern branch from 837b509 to 70397ef Compare November 6, 2025 13:09

github-actions bot removed the INFRA label Nov 6, 2025

LuciferYang approved these changes Nov 6, 2025

View reviewed changes

Revert "connect server shades guava"

75f850c

This reverts commit 2a42a4e.

pan3793 force-pushed the guava-govern branch from 70397ef to 75f850c Compare November 6, 2025 18:22

github-actions bot added the INFRA label Nov 6, 2025

LuciferYang reviewed Nov 7, 2025

View reviewed changes

Revert "mvn ci"

420f355

This reverts commit 2954854.

github-actions bot removed the INFRA label Nov 7, 2025

remove unnecessary guava.version override

9f61749

LuciferYang approved these changes Nov 7, 2025

View reviewed changes

LuciferYang closed this in a8d128c Nov 7, 2025

[SPARK-54190][BUILD] Guava dependency governance #52873

[SPARK-54190][BUILD] Guava dependency governance #52873

Uh oh!

Conversation

pan3793 commented Nov 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

pan3793 commented Nov 5, 2025

Uh oh!

LuciferYang Nov 5, 2025

Choose a reason for hiding this comment

Uh oh!

pan3793 Nov 5, 2025

Choose a reason for hiding this comment

Uh oh!

pan3793 Nov 5, 2025

Choose a reason for hiding this comment

Uh oh!

LuciferYang commented Nov 5, 2025

Uh oh!

pan3793 commented Nov 5, 2025

Uh oh!

pan3793 commented Nov 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pan3793 Nov 5, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

pan3793 commented Nov 6, 2025

Uh oh!

pan3793 commented Nov 6, 2025

Uh oh!

pan3793 commented Nov 7, 2025

Uh oh!

LuciferYang Nov 7, 2025

Choose a reason for hiding this comment

Uh oh!

pan3793 Nov 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Nov 7, 2025

Uh oh!

pan3793 commented Nov 7, 2025

Uh oh!

dongjoon-hyun commented Nov 7, 2025

Uh oh!

LuciferYang commented Nov 7, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

pan3793 commented Nov 4, 2025 •

edited

Loading

pan3793 commented Nov 5, 2025 •

edited

Loading

pan3793 Nov 7, 2025 •

edited

Loading