Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CI] TransformIndexerStateTests.testStopAtCheckpointForThrottledTransform #65542

Closed
pugnascotia opened this issue Nov 26, 2020 · 6 comments · Fixed by #65721
Closed

[CI] TransformIndexerStateTests.testStopAtCheckpointForThrottledTransform #65542

pugnascotia opened this issue Nov 26, 2020 · 6 comments · Fixed by #65721
Labels
:ml/Transform Transform >test-failure Triaged test failures from CI

Comments

@pugnascotia
Copy link
Contributor

Build scan:

https://gradle-enterprise.elastic.co/s/cdnepzmdlrqr2/tests/failed

Repro line:

./gradlew ':x-pack:plugin:transform:test' --tests "org.elasticsearch.xpack.transform.transforms.TransformIndexerStateTests.testStopAtCheckpointForThrottledTransform" -Dtests.seed=2259493706BA5DDC -Dtests.security.manager=true -Dtests.locale=ar-OM -Dtests.timezone=Universal -Druntime.java=8

Reproduces locally?:

No

Applicable branches:

7.x

Failure history:

Failure excerpt:

java.lang.AssertionError: (No message provided)
<etc>
at org.elasticsearch.xpack.transform.transforms.TransformIndexerStateTests.testStopAtCheckpointForThrottledTransform(TransformIndexerStateTests.java:494)

#62996 seems related. @hendrikmuhs do you think this test needs a similar fix?
@pugnascotia pugnascotia added >test-failure Triaged test failures from CI :ml/Transform Transform labels Nov 26, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/ml-core (:ml/Transform)

@hendrikmuhs
Copy link

The log indicates a problem with a thread pool:

11:40:47 org.elasticsearch.xpack.transform.transforms.TransformIndexerStateTests > testStopAtCheckpointForThrottledTransform FAILED
11:40:47     java.lang.AssertionError
11:40:47         at __randomizedtesting.SeedInfo.seed([2259493706BA5DDC:DF350226037DAF0]:0)
11:40:47         at org.junit.Assert.fail(Assert.java:86)
11:40:47         at org.junit.Assert.assertTrue(Assert.java:41)
11:40:47         at org.junit.Assert.assertTrue(Assert.java:52)
11:40:47         at org.elasticsearch.xpack.transform.transforms.TransformIndexerStateTests.testStopAtCheckpointForThrottledTransform(TransformIndexerStateTests.java:494)
11:40:47 REPRODUCE WITH: ./gradlew ':x-pack:plugin:transform:test' --tests "org.elasticsearch.xpack.transform.transforms.TransformIndexerStateTests.testStopAtCheckpointForThrottledTransform" -Dtests.seed=2259493706BA5DDC -Dtests.security.manager=true -Dtests.locale=ar-OM -Dtests.timezone=Universal -Druntime.java=8
11:40:47 
11:40:47 Suite: Test class org.elasticsearch.xpack.transform.transforms.TransformIndexerStateTests
11:40:47   1> [2020-11-26T10:40:43,038][INFO ][o.e.x.t.t.TransformIndexerStateTests] [testStopAtCheckpointForThrottledTransform] before test
11:40:47   1> [2020-11-26T10:40:45,377][WARN ][o.e.x.t.t.TransformIndexer] [generic] [bMBXvFwkCU] transform encountered an exception: 
11:40:47   1> org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution of java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@cca552a on org.elasticsearch.threadpool.Scheduler$SafeScheduledThreadPoolExecutor@647862f7[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
11:40:47   1> 	at org.elasticsearch.common.util.concurrent.EsAbortPolicy.rejectedExecution(EsAbortPolicy.java:48) ~[elasticsearch-7.11.0-SNAPSHOT.jar:7.11.0-SNAPSHOT]
11:40:47   1> 	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) ~[?:1.8.0_241]
11:40:47   1> 	at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:326) ~[?:1.8.0_241]
11:40:47   1> 	at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:533) ~[?:1.8.0_241]
11:40:47   1> 	at org.elasticsearch.threadpool.ThreadPool.schedule(ThreadPool.java:365) ~[elasticsearch-7.11.0-SNAPSHOT.jar:7.11.0-SNAPSHOT]
11:40:47   1> 	at org.elasticsearch.xpack.core.indexing.AsyncTwoPhaseIndexer$ScheduledRunnable.<init>(AsyncTwoPhaseIndexer.java:83) ~[x-pack-core-7.11.0-SNAPSHOT.jar:7.11.0-SNAPSHOT]
11:40:47   1> 	at org.elasticsearch.xpack.core.indexing.AsyncTwoPhaseIndexer.nextSearch(AsyncTwoPhaseIndexer.java:577) ~[x-pack-core-7.11.0-SNAPSHOT.jar:7.11.0-SNAPSHOT]
11:40:47   1> 	at org.elasticsearch.xpack.core.indexing.AsyncTwoPhaseIndexer.lambda$onBulkResponse$11(AsyncTwoPhaseIndexer.java:550) ~[x-pack-core-7.11.0-SNAPSHOT.jar:7.11.0-SNAPSHOT]
11:40:47   1> 	at org.elasticsearch.xpack.transform.transforms.TransformIndexerStateTests$MockedTransformIndexer.doSaveState(TransformIndexerStateTests.java:208) ~[test/:?]
11:40:47   1> 	at org.elasticsearch.xpack.transform.transforms.TransformIndexerStateTests$MockedTransformIndexer.doSaveState(TransformIndexerStateTests.java:94) ~[test/:?]
11:40:47   1> 	at org.elasticsearch.xpack.core.indexing.AsyncTwoPhaseIndexer.onBulkResponse(AsyncTwoPhaseIndexer.java:550) ~[x-pack-core-7.11.0-SNAPSHOT.jar:7.11.0-SNAPSHOT]
11:40:47   1> 	at org.elasticsearch.xpack.core.indexing.AsyncTwoPhaseIndexer.lambda$onSearchResponse$9(AsyncTwoPhaseIndexer.java:517) ~[x-pack-core-7.11.0-SNAPSHOT.jar:7.11.0-SNAPSHOT]
11:40:47   1> 	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) ~[elasticsearch-7.11.0-SNAPSHOT.jar:7.11.0-SNAPSHOT]
11:40:47   1> 	at org.elasticsearch.xpack.transform.transforms.TransformIndexerStateTests$MockedTransformIndexer.lambda$doNextBulk$1(TransformIndexerStateTests.java:183) ~[test/:?]
11:40:47   1> 	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:678) [elasticsearch-7.11.0-SNAPSHOT.jar:7.11.0-SNAPSHOT]
11:40:47   1> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_241]
11:40:47   1> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_241]
11:40:47   1> 	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_241]
11:40:47   1> [2020-11-26T10:40:45,530][INFO ][o.e.x.t.t.TransformIndexerStateTests] [testStopAtCheckpointForThrottledTransform] after test
11:40:47   2> REPRODUCE WITH: ./gradlew ':x-pack:plugin:transform:test' --tests "org.elasticsearch.xpack.transform.transforms.TransformIndexerStateTests.testStopAtCheckpointForThrottledTransform" -Dtests.seed=2259493706BA5DDC -Dtests.security.manager=true -Dtests.locale=ar-OM -Dtests.timezone=Universal -Druntime.java=8
11:40:47   2> java.lang.AssertionError
11:40:47         at __randomizedtesting.SeedInfo.seed([2259493706BA5DDC:DF350226037DAF0]:0)
11:40:47         at org.junit.Assert.fail(Assert.java:86)
11:40:47         at org.junit.Assert.assertTrue(Assert.java:41)
11:40:47         at org.junit.Assert.assertTrue(Assert.java:52)
11:40:47         at org.elasticsearch.xpack.transform.transforms.TransformIndexerStateTests.testStopAtCheckpointForThrottledTransform(TransformIndexerStateTests.java:494)
11:40:47   1> [2020-11-26T10:40:45,840][INFO ][o.e.x.t.t.TransformIndexerStateTests] [testStopAtCheckpoint] before test
11:40:47   1> [2020-11-26T10:40:46,090][INFO ][o.e.x.c.i.AsyncTwoPhaseIndexer] [generic] Indexer job encountered [STOPPING] state, halting indexer.
11:40:47   1> [2020-11-26T10:40:46,091][INFO ][o.e.x.t.t.TransformIndexer] [generic] [wOnMnAJXNC] transform has stopped.
11:40:47   1> [2020-11-26T10:40:46,175][INFO ][o.e.x.c.i.AsyncTwoPhaseIndexer] [generic] Indexer job encountered [STOPPING] state, halting indexer.
11:40:47   1> [2020-11-26T10:40:46,175][INFO ][o.e.x.t.t.TransformIndexer] [generic] [wOnMnAJXNC] transform has stopped.
11:40:47   1> [2020-11-26T10:40:46,193][INFO ][o.e.x.c.i.AsyncTwoPhaseIndexer] [generic] Indexer job encountered [STOPPING] state, halting indexer.
11:40:47   1> [2020-11-26T10:40:46,194][INFO ][o.e.x.t.t.TransformIndexer] [generic] [wOnMnAJXNC] transform has stopped.
11:40:47   1> [2020-11-26T10:40:46,220][INFO ][o.e.x.c.i.AsyncTwoPhaseIndexer] [generic] Indexer job encountered [STOPPING] state, halting indexer.
11:40:47   1> [2020-11-26T10:40:46,220][INFO ][o.e.x.t.t.TransformIndexer] [generic] [wOnMnAJXNC] transform has stopped.
11:40:47   1> [2020-11-26T10:40:46,232][INFO ][o.e.x.t.t.TransformIndexerStateTests] [testStopAtCheckpoint] after test

The test uses the name of the "generic" threadpool, I wonder if that's the cause of the race. It should use a e.g. the test suite name as thread pool name.

@hendrikmuhs hendrikmuhs mentioned this issue Dec 2, 2020
Merged
hendrikmuhs pushed a commit that referenced this issue Dec 2, 2020
[Transform] Use test threadpool to avoid conflict with other tests (#…
95fda0a 
…65721)

use a test threadpool instead of the generic threadpool in test

fixes #65542
hendrikmuhs pushed a commit that referenced this issue Dec 2, 2020
[Transform] Use test threadpool to avoid conflict with other tests (#…
005bb81 
…65721)

use a test threadpool instead of the generic threadpool in test

fixes #65542
@hendrikmuhs hendrikmuhs mentioned this issue Dec 7, 2020
Merged
@henningandersen
Copy link
Contributor

This failed again today:

https://gradle-enterprise.elastic.co/s/3pnnqhzzbqkok

with the same message:

java.lang.AssertionError: (No message provided)
at org.elasticsearch.xpack.transform.transforms.TransformIndexerStateTests.testStopAtCheckpointForThrottledTransform(TransformIndexerStateTests.java:518)

repro line (did not reproduce for me)

./gradlew ':x-pack:plugin:transform:test' --tests "org.elasticsearch.xpack.transform.transforms.TransformIndexerStateTests.testStopAtCheckpointForThrottledTransform" \
  -Dtests.seed=E115F754B5954F77 \
  -Dtests.security.manager=true \
  -Dtests.locale=es-UY \
  -Dtests.timezone=America/Glace_Bay \
  -Druntime.java=11

history

Looks like it only failed once in the past 90 days.

@hendrikmuhs
Copy link

I had a look at the logs, no trace like in #65542 (comment), so this issue seems to have a different root cause.

@hendrikmuhs
Copy link

This might got fixed by #70461

I will keep this issue open for another week and see (however, even before the potential fix this happened extremely rare).

@jimczi jimczi mentioned this issue Mar 25, 2021
Merged
@hendrikmuhs
Copy link

No occurrence of this error in the last 90 days, closing

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
:ml/Transform Transform >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[Transform] Use test threadpool to avoid conflict with other tests hendrikmuhs/elasticsearch
4 participants
@hendrikmuhs @pugnascotia @elasticmachine @henningandersen