[SPARK-20084][Core] Remove internal.metrics.updatedBlockStatuses from history files.

[rdblue]

What changes were proposed in this pull request?

Remove accumulator updates for internal.metrics.updatedBlockStatuses from SparkListenerTaskEnd entries in the history file. These can cause history files to grow to hundreds of GB because the value of the accumulator contains all tracked blocks.

How was this patch tested?

Current History UI tests cover use of the history file.

[SparkQA]

Test build #75173 has finished for PR 17412 at commit 386846c.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[vanzin]

I'm fine with this, assuming there's no need for this information anywhere. (Basically, the question I asked in #16714).

[rdblue]

@vanzin, I looked through the listeners in the Spark UI for uses of it and didn't find any, plus I haven't seen anything in the UI that is obviously based on it. As I said on the issue, the updates duplicate data in metrics so it would be odd for a component to go to accumulators to get this info.

I'm deploying this internally today, maybe it makes sense to give it a couple days to see if we get complaints about something not working.

[rdblue]

@vanzin, we've been running this in production for a few days now and haven't had any problems. I think it should be safe to merge. Could you take another look? Thanks!

[vanzin]

LGTM. Merging to master / 2.1.

[zhouyejoe]

@rdblue Hi, why not the blockstatusupdates are not filtering out in executorMetricsUpdate? This line https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/util/JsonProtocol.scala#L245
While I am working on SPARK-21961, I filtered those blockstatusupdates while reading from logs in Spark History Server, but it causing some unit test failure.
Should it not be filtered out in both executorMetricsUpdateFromJson and executorMetricsUpdateToJson?
Thanks.