Avoid Spark driver IP binding failure due to VPN setting #866

flyrain · 2025-01-24T02:11:37Z

Fixed Spark local run issue like this:

24/09/01 23:00:54 ERROR Inbox: Ignoring error
java.lang.NullPointerException: **Cannot invoke "org.apache.spark.storage.BlockManagerId.executorId()" because "idWithoutTopologyInfo"** is null
        at org.apache.spark.storage.BlockManagerMasterEndpoint.org$apache$spark$storage$BlockManagerMasterEndpoint$$register(BlockManagerMasterEndpoint.scala:677)    
        at org.apache.spark.storage.BlockManagerMasterEndpoint$$anonfun$receiveAndReply$1.applyOrElse(BlockManagerMasterEndpoint.scala:133)
        at org.apache.spark.rpc.netty.Inbox.$anonfun$process$1(Inbox.scala:103)
        at org.apache.spark.rpc.netty.Inbox.safelyCall(Inbox.scala:213)
        at org.apache.spark.rpc.netty.Inbox.process(Inbox.scala:100)
        at org.apache.spark.rpc.netty.MessageLoop.org$apache$spark$rpc$netty$MessageLoop$$receiveLoop(MessageLoop.scala:75)
        at org.apache.spark.rpc.netty.MessageLoop$$anon$1.run(MessageLoop.scala:41)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)

flyrain · 2025-01-24T02:21:41Z

regtests/run_spark_sql.sh


 ./setup.sh

 if [ -z "${SPARK_HOME}"]; then
  export SPARK_HOME=$(realpath ~/${SPARK_DISTRIBUTION})
 fi

-SPARK_BEARER_TOKEN="${REGTEST_ROOT_BEARER_TOKEN:-principal:root;realm:default-realm}"
+SPARK_BEARER_TOKEN="${REGTEST_ROOT_BEARER_TOKEN:-principal:root;realm:polaris}"


Looks like this change is optional. It passed without the change for me locally.

flyrain · 2025-01-24T04:05:23Z

The test failed due to it use the test profile in application.properties. There are two ways to fix this

Using a different profile or even different application.properties file for the tests.
Creating a new profile for local run.

jbonofre · 2025-01-24T08:12:12Z

When possible, we should use the same application.properties between the app and the tests.
Imho, it's fine to create a dedicated profile for test/local run.

adutra · 2025-01-24T09:27:07Z

quarkus/defaults/src/main/resources/application.properties

@@ -130,6 +131,8 @@ polaris.authentication.token-broker.max-token-generation=PT1H
 %test.quarkus.log.category."org.apache.polaris.service.storage.PolarisStorageIntegrationProviderImpl".level=ERROR
 %test.quarkus.http.limits.max-body-size=1000000
 %test.quarkus.otel.sdk.disabled=true
+%test.polaris.authentication.authenticator.type=test


My idea was to do exactly the opposite: make all tests (automated + regression) use the default authenticator. Cf #804.

That makes sense to me.

flyrain · 2025-01-24T22:03:36Z

Repurpose this PR as #804 will resolve the credential issue.

regtests/run_spark_sql.sh

adutra · 2025-01-25T11:01:28Z

regtests/run_spark_sql.sh

@@ -56,7 +56,7 @@ if [ -z "${SPARK_HOME}"]; then
  export SPARK_HOME=$(realpath ~/${SPARK_DISTRIBUTION})
 fi

-SPARK_BEARER_TOKEN="${REGTEST_ROOT_BEARER_TOKEN:-principal:root;realm:polaris}"
+SPARK_BEARER_TOKEN="${REGTEST_ROOT_BEARER_TOKEN:-principal:root;realm:default-realm}"


This change may not be needed anymore since I removed the default values for these environment variables in #804.

Tested again based on #804. This change is still needed for my local run. I think this is due a recent change of my local VPN setting. However I couldn't touch VPN settings by myself. Given this script is mainly used for local test, I think this change should be good.

Here is the error message without this change:

Exception in thread "main" java.io.IOException: Failed to send RPC /jars/software.amazon.awssdk_http-client-spi-2.23.19.jar to mac.lan/192.168.86.27:55366: io.netty.channel.StacklessClosedChannelException at org.apache.spark.network.client.TransportClient$2.handleFailure(TransportClient.java:164) at org.apache.spark.network.client.TransportClient$StdChannelListener.operationComplete(TransportClient.java:372) at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629) at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118) at io.netty.channel.AbstractChannel$AbstractUnsafe.safeSetFailure(AbstractChannel.java:999) at io.netty.channel.AbstractChannel$AbstractUnsafe.write(AbstractChannel.java:860) at io.netty.channel.DefaultChannelPipeline$HeadContext.write(DefaultChannelPipeline.java:1367) at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:877) at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:863) at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:968) at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:856) at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:113) at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:881) at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:863) at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:968) at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:856) at io.netty.handler.timeout.IdleStateHandler.write(IdleStateHandler.java:302) at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:879) at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:940) at io.netty.channel.AbstractChannelHandlerContext$WriteTask.run(AbstractChannelHandlerContext.java:1247) at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174) at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167) at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:566) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:1583) Caused by: io.netty.channel.StacklessClosedChannelException at io.netty.channel.AbstractChannel$AbstractUnsafe.write(Object, ChannelPromise)(Unknown Source)

eric-maynard · 2025-01-27T18:28:39Z

regtests/run.sh

@@ -26,6 +26,7 @@ if [ -z "${SPARK_HOME}" ]; then
  export SPARK_HOME=$(realpath ~/${SPARK_DISTRIBUTION})
 fi
 export PYTHONPATH="${SPARK_HOME}/python/:${SPARK_HOME}/python/lib/py4j-0.10.9.7-src.zip:$PYTHONPATH"
+export SPARK_LOCAL_HOSTNAME=localhost #avoid VPN messing up driver local IP address binding


nit: space after #

eric-maynard · 2025-01-27T18:28:47Z

regtests/run_spark_sql.sh

@@ -48,6 +48,7 @@ cd ${REGTEST_HOME}

 export SPARK_VERSION=spark-3.5.3
 export SPARK_DISTRIBUTION=${SPARK_VERSION}-bin-hadoop3
+export SPARK_LOCAL_HOSTNAME=localhost #avoid VPN messing up driver local IP address binding


nit: space after #

adutra · 2025-01-28T11:25:53Z

I also have a VPN on but wasn't having issues. I checked that this change does not introduce any regressions on my end, so LGTM.

flyrain · 2025-01-28T16:33:48Z

Thanks @eric-maynard and @adutra for the review

flyrain requested review from adutra, ashvina, dennishuo, dimas-b, eric-maynard, jackye1995, jbonofre, vvcephei, collado-mike, snazy, RussellSpitzer, takidau and ebyhr as code owners January 24, 2025 02:11

flyrain commented Jan 24, 2025

View reviewed changes

flyrain mentioned this pull request Jan 24, 2025

Failed to run run_spark_sql.sh #865

Closed

adutra reviewed Jan 24, 2025

View reviewed changes

adutra mentioned this pull request Jan 24, 2025

Make regression tests use default authentication #804

Merged

flyrain force-pushed the test-token-fix branch from f14cc67 to 79930c7 Compare January 24, 2025 22:00

flyrain changed the title ~~Fix the Polaris test credentials~~ Avoid Jan 24, 2025

flyrain changed the title ~~Avoid~~ Avoid Spark driver IP binding failure due to VPN setting Jan 24, 2025

adutra reviewed Jan 25, 2025

View reviewed changes

regtests/run_spark_sql.sh Outdated Show resolved Hide resolved

adutra reviewed Jan 25, 2025

View reviewed changes

Yufei Gu added 2 commits January 25, 2025 09:50

fix the same issue in run.sh

25d9c5c

fix the same issue in run.sh

7ff9a37

flyrain force-pushed the test-token-fix branch from 63cb810 to 7ff9a37 Compare January 25, 2025 17:52

eric-maynard reviewed Jan 27, 2025

View reviewed changes

eric-maynard approved these changes Jan 27, 2025

View reviewed changes

Resolve comments

122ad65

flyrain requested a review from MonkeyCanCode as a code owner January 27, 2025 19:30

adutra approved these changes Jan 28, 2025

View reviewed changes

flyrain merged commit 2f664a3 into apache:main Jan 28, 2025
5 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Avoid Spark driver IP binding failure due to VPN setting #866

Avoid Spark driver IP binding failure due to VPN setting #866

flyrain commented Jan 24, 2025 •

edited

Loading

flyrain Jan 24, 2025

flyrain commented Jan 24, 2025

jbonofre commented Jan 24, 2025

adutra Jan 24, 2025

flyrain Jan 24, 2025

flyrain commented Jan 24, 2025

adutra Jan 25, 2025

flyrain Jan 25, 2025

eric-maynard Jan 27, 2025

flyrain Jan 27, 2025

eric-maynard Jan 27, 2025

flyrain Jan 27, 2025

adutra commented Jan 28, 2025

flyrain commented Jan 28, 2025

Avoid Spark driver IP binding failure due to VPN setting #866

Avoid Spark driver IP binding failure due to VPN setting #866

Conversation

flyrain commented Jan 24, 2025 • edited Loading

Choose a reason for hiding this comment

flyrain commented Jan 24, 2025

jbonofre commented Jan 24, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

flyrain commented Jan 24, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adutra commented Jan 28, 2025

flyrain commented Jan 28, 2025

flyrain commented Jan 24, 2025 •

edited

Loading