[SPARK-5307] SerializationDebugger to help debug NotSerializableException #4093
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…tion. This patch adds a SerializationDebugger that is used to add more information to a NotSerializableException. When a NotSerializableException is encountered, the debugger tries to serialize the object one more time through a DebugStream that hooks into the internals of ObjectOutputStream to get the serialization stack. An example output looks like this: org.apache.spark.serializer.NotSerializableClass Serialization stack (3): - org.apache.spark.serializer.NotSerializableClass@5e20dc10 (class org.apache.spark.serializer.NotSerializableClass) - org.apache.spark.serializer.SerializableClass2@521fb14e (class org.apache.spark.serializer.SerializableClass2) - org.apache.spark.serializer.SerializableClass1@5f54e92c (class org.apache.spark.serializer.SerializableClass1) Run the JVM with sun.io.serialization.extendedDebugInfo for more information.
| nse.getMessage + "\n" + | ||
| s"\tSerialization stack (${out.stack.size}):\n" + | ||
| out.stack.map(o => s"\t- $o (class ${o.getClass.getName})").mkString("\n") + "\n" + | ||
| "\tRun the JVM with sun.io.serialization.extendedDebugInfo for more information.") |
There was a problem hiding this comment.
Maybe -Dsun.io.serialization.extendedDebugInfo?
There was a problem hiding this comment.
It is actually "-Dsun.io.serialization.extendedDebugInfo=true". Kinda long ...
|
Test build #25716 has finished for PR 4093 at commit
|
|
This LGTM. |
|
Test build #25715 timed out for PR 4093 at commit |
|
Test build #25719 has finished for PR 4093 at commit
|
|
Closing this one in favor of #4098 |
|
could someone give a simple example of how to read the graph? I'm not sure where to start |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This patch adds a SerializationDebugger that is used to add more information to a NotSerializableException. When a NotSerializableException is encountered, the debugger tries to serialize the object one more time through a DebugStream that hooks into the internals of ObjectOutputStream to get the serialization stack. This ensures that there is no performance loss to run with SerializationDebugger, unlike setting the sun.io.serialization.extendedDebugInfo flag.
An example output looks like this:
When sun.io.serialization.extendedDebugInfo is turned on, this debugger no longer adds more information. Note that sun.io.serialization.extendedDebugInfo can show also the field name information, which is harder to get by the SerializationDebugger (technically possible with reflection but fairly convoluted).