[SPARK-3825] Log more detail when unrolling a block fails

[andrewor14]

Before:

14/10/06 16:45:42 WARN CacheManager: Not enough space to cache partition rdd_0_2
in memory! Free memory is 481861527 bytes.

After:

14/10/07 11:08:24 WARN MemoryStore: Not enough space to cache rdd_2_0 in memory!
(computed 68.8 MB so far)
14/10/07 11:08:24 INFO MemoryStore: Memory use = 1088.0 B (blocks) + 445.1 MB
(scratch space shared across 8 thread(s)) = 445.1 MB. Storage limit = 459.5 MB.

[andrewor14]

@pwendell @tdas

[SparkQA]

QA tests have started for PR 2688 at commit 5638c49.

• This patch merges cleanly.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21364/Test FAILed.

[andrewor14]

retest this please

[SparkQA]

QA tests have started for PR 2688 at commit 5638c49.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2688 at commit 5638c49.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21365/Test PASSed.

[SparkQA]

QA tests have finished for PR 2688 at commit 5638c49.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21366/Test PASSed.

[pwendell]

This is a little wordy. Throughout Spark the log messages are intentionally terse. This matters a lot when you are trying to read through hundreds of lines of logs. Something like below cuts down on words a lot but has just as much information.

4/10/06 21:40:36 WARN MemoryStore: Gave up caching rdd_0_2 after 68.8 MB!
4/10/06 21:40:36 INFO MemoryStore: Memory use: 1088.0 B (blocks)
+ 445.1 MB (scratch space for 8 thread(s)) = XX. Storage limit is 459.5 MB.

[andrewor14]

How about:

14/10/07 10:55:13 WARN MemoryStore: Not enough space to cache rdd_0_2 in memory!
(gave up after unrolling 38.7 MB)
14/10/07 10:55:13 INFO MemoryStore: Memory use = 1088.0 B (blocks) + 445.1 MB
(unrolling space shared across 8 thread(s)) = 445.1 MB. Storage limit = 459.5 MB.

[pwendell]

I'd prefer to use "unrolling" less but don't really mind. LGTM with whatever you want.

[andrewor14]

Ok, I updated it. Refer to the PR description for the latest state.

[SparkQA]

QA tests have started for PR 2688 at commit 28e33d6.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2688 at commit 28e33d6.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21389/Test PASSed.

[andrewor14]

Got a verbal LGTM from @pwendell. This is going into master and 1.1!