[SPARK-2580] [PySpark] keep silent in worker if JVM close the socket #1625
Conversation
|
QA tests have started for PR 1625. This patch merges cleanly. |
|
QA results for PR 1625: |
|
QA tests have started for PR 1625. This patch merges cleanly. |
|
I've merged this into |
|
@JoshRosen did it include the last commit? I didn't find them in master nor branch-1.0. It's delayed? |
|
@davies Yeah, it included both commits. If you check the Apache repo, you should see the commits, but the GitHub mirroring is currently lagging behind. |
|
QA results for PR 1625: |
|
+1 lgtm, validated |
During rdd.take(n), JVM will close the socket if it had got enough data, the Python worker should keep silent in this case. In the same time, the worker should not print the trackback into stderr if it send the traceback to JVM successfully. Author: Davies Liu <[email protected]> Closes #1625 from davies/error and squashes the following commits: 4fbcc6d [Davies Liu] disable log4j during testing when exception is expected. cc14202 [Davies Liu] keep silent in worker if JVM close the socket (cherry picked from commit ccd5ab5) Signed-off-by: Josh Rosen <[email protected]>
During rdd.take(n), JVM will close the socket if it had got enough data, the Python worker should keep silent in this case. In the same time, the worker should not print the trackback into stderr if it send the traceback to JVM successfully. Author: Davies Liu <[email protected]> Closes apache#1625 from davies/error and squashes the following commits: 4fbcc6d [Davies Liu] disable log4j during testing when exception is expected. cc14202 [Davies Liu] keep silent in worker if JVM close the socket
During rdd.take(n), JVM will close the socket if it had got enough data, the Python worker should keep silent in this case.
In the same time, the worker should not print the trackback into stderr if it send the traceback to JVM successfully.