[SPARK-23275][SQL] hive/tests have been failing when run locally on the laptop (Mac) with OOM #20441

dilipbiswal · 2018-01-30T19:51:09Z

What changes were proposed in this pull request?

hive tests have been failing when they are run locally (Mac Os) after a recent change in the trunk. After running the tests for some time, the test fails with OOM with Error: unable to create new native thread.

I noticed the thread count goes all the way up to 2000+ after which we start getting these OOM errors. Most of the threads seem to be related to the connection pool in hive metastore (BoneCP-xxxxx-xxxx ). This behaviour change is happening after we made the following change to HiveClientImpl.reset()

 def reset(): Unit = withHiveState {
    try {
      // code
    } finally {
      runSqlHive("USE default")  ===> this is causing the issue
    }

I am proposing to temporarily back-out part of a fix made to address SPARK-23000 to resolve this issue while we work-out the exact reason for this sudden increase in thread counts.

How was this patch tested?

Ran hive/test multiple times in different machines.

(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)

Please review http://spark.apache.org/contributing.html before opening a pull request.

…ptop (Mac) with OOM

gatorsmile · 2018-01-30T21:31:19Z

@liufengdb You might be interested in this

SparkQA · 2018-01-30T21:56:48Z

Test build #86844 has finished for PR 20441 at commit 983aa18.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile

This is a test only PR. This is an interesting finding. We can dig it deeper in the future.

Thanks for reporting this issue.

LGTM

gatorsmile · 2018-01-30T22:11:34Z

Thanks! Merged to master/2.3

…he laptop (Mac) with OOM ## What changes were proposed in this pull request? hive tests have been failing when they are run locally (Mac Os) after a recent change in the trunk. After running the tests for some time, the test fails with OOM with Error: unable to create new native thread. I noticed the thread count goes all the way up to 2000+ after which we start getting these OOM errors. Most of the threads seem to be related to the connection pool in hive metastore (BoneCP-xxxxx-xxxx ). This behaviour change is happening after we made the following change to HiveClientImpl.reset() ``` SQL def reset(): Unit = withHiveState { try { // code } finally { runSqlHive("USE default") ===> this is causing the issue } ``` I am proposing to temporarily back-out part of a fix made to address SPARK-23000 to resolve this issue while we work-out the exact reason for this sudden increase in thread counts. ## How was this patch tested? Ran hive/test multiple times in different machines. (If this patch involves UI changes, please attach a screenshot; otherwise, remove this) Please review http://spark.apache.org/contributing.html before opening a pull request. Author: Dilip Biswal <[email protected]> Closes #20441 from dilipbiswal/hive_tests. (cherry picked from commit 58fcb5a) Signed-off-by: gatorsmile <[email protected]>

dilipbiswal · 2018-01-30T22:12:03Z

Many thanks @gatorsmile .

srowen · 2018-01-31T00:09:59Z

For the record, what was the error? I got this running on OS X from master with -Phive before this change, but was that same problem?

HiveWindowFunctionQuerySuite:
*** RUN ABORTED ***
  org.apache.spark.sql.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to fetch table part. Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient;

...

Exception in thread "block-manager-slave-async-thread-pool-23" java.lang.OutOfMemoryError: unable to create new native thread

dilipbiswal · 2018-01-31T00:54:55Z

@srowen Hello, yeah, i saw the same error. Quite a few errors like

java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient

with underlying reason

Caused by: java.lang.OutOfMemoryError: unable to create new native thread

liufengdb · 2018-02-09T21:23:07Z

@gatorsmile sorry for the late reply. I think the root cause is in hive metastore. I created one pr to bypass it: #20562

## What changes were proposed in this pull request? This is a follow up of #20441. The two lines actually can trigger the hive metastore bug: https://issues.apache.org/jira/browse/HIVE-16844 The two configs are not in the default `ObjectStore` properties, so any run hive commands after these two lines will set the `propsChanged` flag in the `ObjectStore.setConf` and then cause thread leaks. I don't think the two lines are very useful. They can be removed safely. ## How was this patch tested? (Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests) (If this patch involves UI changes, please attach a screenshot; otherwise, remove this) Please review http://spark.apache.org/contributing.html before opening a pull request. Author: Feng Liu <[email protected]> Closes #20562 from liufengdb/fix-omm. (cherry picked from commit 6d7c383) Signed-off-by: gatorsmile <[email protected]>

## What changes were proposed in this pull request? This is a follow up of apache#20441. The two lines actually can trigger the hive metastore bug: https://issues.apache.org/jira/browse/HIVE-16844 The two configs are not in the default `ObjectStore` properties, so any run hive commands after these two lines will set the `propsChanged` flag in the `ObjectStore.setConf` and then cause thread leaks. I don't think the two lines are very useful. They can be removed safely. ## How was this patch tested? (Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests) (If this patch involves UI changes, please attach a screenshot; otherwise, remove this) Please review http://spark.apache.org/contributing.html before opening a pull request. Author: Feng Liu <[email protected]> Closes apache#20562 from liufengdb/fix-omm.

[SPARK-23275] hive/tests have been failing when run locally on the la…

983aa18

…ptop (Mac) with OOM

dilipbiswal changed the title ~~[SPARK-23275] hive/tests have been failing when run locally on the laptop (Mac) with OOM~~ [SPARK-23275][SQL] hive/tests have been failing when run locally on the laptop (Mac) with OOM Jan 30, 2018

gatorsmile reviewed Jan 30, 2018

View reviewed changes

asfgit closed this in 58fcb5a Jan 30, 2018

liufengdb mentioned this pull request Feb 9, 2018

[SPARK-23275][SQL] fix the thread leaking in hive/tests #20562

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-23275][SQL] hive/tests have been failing when run locally on the laptop (Mac) with OOM #20441

[SPARK-23275][SQL] hive/tests have been failing when run locally on the laptop (Mac) with OOM #20441

dilipbiswal commented Jan 30, 2018

gatorsmile commented Jan 30, 2018

SparkQA commented Jan 30, 2018

gatorsmile left a comment

gatorsmile commented Jan 30, 2018

dilipbiswal commented Jan 30, 2018

srowen commented Jan 31, 2018 •

edited

Loading

dilipbiswal commented Jan 31, 2018

liufengdb commented Feb 9, 2018

[SPARK-23275][SQL] hive/tests have been failing when run locally on the laptop (Mac) with OOM #20441

[SPARK-23275][SQL] hive/tests have been failing when run locally on the laptop (Mac) with OOM #20441

Conversation

dilipbiswal commented Jan 30, 2018

What changes were proposed in this pull request?

How was this patch tested?

gatorsmile commented Jan 30, 2018

SparkQA commented Jan 30, 2018

gatorsmile left a comment

Choose a reason for hiding this comment

gatorsmile commented Jan 30, 2018

dilipbiswal commented Jan 30, 2018

srowen commented Jan 31, 2018 • edited Loading

dilipbiswal commented Jan 31, 2018

liufengdb commented Feb 9, 2018

srowen commented Jan 31, 2018 •

edited

Loading