Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CI] FrozenIndexIT testRetryPointInTime failing #89231

Closed
albertzaharovits opened this issue Aug 10, 2022 · 3 comments · Fixed by #89429
Closed

[CI] FrozenIndexIT testRetryPointInTime failing #89231

albertzaharovits opened this issue Aug 10, 2022 · 3 comments · Fixed by #89429
Assignees
@grcevski
Labels
:Core/Infra/Core Core issues without another label Team:Core/Infra Meta label for core/infra team >test-failure Triaged test failures from CI

Comments

@albertzaharovits
Copy link
Contributor

Seems like a one-off failure on ARM : https://gradle-enterprise.elastic.co/scans/tests?search.timeZoneId=Europe%2FBucharest&tests.container=org.elasticsearch.index.engine.frozen.FrozenIndexIT&tests.test=testRetryPointInTime

Build scan:
https://gradle-enterprise.elastic.co/s/jgxaxl3gwjel2/tests/:x-pack:plugin:frozen-indices:internalClusterTest/org.elasticsearch.index.engine.frozen.FrozenIndexIT/testRetryPointInTime

Reproduction line:
./gradlew ':x-pack:plugin:frozen-indices:internalClusterTest' --tests "org.elasticsearch.index.engine.frozen.FrozenIndexIT.testRetryPointInTime" -Dtests.seed=991387B9FDE04ADC -Dtests.locale=pl-PL -Dtests.timezone=Africa/Algiers -Druntime.java=17

Applicable branches:
main

Reproduces locally?:
No

Failure history:
https://gradle-enterprise.elastic.co/scans/tests?tests.container=org.elasticsearch.index.engine.frozen.FrozenIndexIT&tests.test=testRetryPointInTime

Failure excerpt:

java.lang.IllegalStateException: unable to launch a new watch service

  at __randomizedtesting.SeedInfo.seed([991387B9FDE04ADC:EC6786CE761CF5F4]:0)
  at org.elasticsearch.reservedstate.service.FileSettingsService.startWatcher(FileSettingsService.java:201)
  at org.elasticsearch.reservedstate.service.FileSettingsService.setWatching(FileSettingsService.java:142)
  at org.elasticsearch.reservedstate.service.FileSettingsService.startIfMaster(FileSettingsService.java:136)
  at org.elasticsearch.reservedstate.service.FileSettingsService.doStart(FileSettingsService.java:111)
  at org.elasticsearch.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:43)
  at org.elasticsearch.node.Node.start(Node.java:1364)
  at org.elasticsearch.test.InternalTestCluster$NodeAndClient.startNode(InternalTestCluster.java:982)
  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
  at java.util.concurrent.FutureTask.run(FutureTask.java:264)
  at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:820)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  at java.lang.Thread.run(Thread.java:833)

  Caused by: java.io.IOException: User limit of inotify watches reached

    at sun.nio.fs.LinuxWatchService$Poller.implRegister(LinuxWatchService.java:264)
    at sun.nio.fs.AbstractPoller.processRequests(AbstractPoller.java:266)
    at sun.nio.fs.LinuxWatchService$Poller.run(LinuxWatchService.java:364)
    at java.lang.Thread.run(Thread.java:833)
@albertzaharovits albertzaharovits added :Distributed Indexing/Engine Anything around managing Lucene and the Translog in an open shard. >test-failure Triaged test failures from CI labels Aug 10, 2022
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-distributed (Team:Distributed)

@elasticsearchmachine elasticsearchmachine added the Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. label Aug 10, 2022
@arteam arteam added :Core/Infra/Core Core issues without another label and removed :Distributed Indexing/Engine Anything around managing Lucene and the Translog in an open shard. labels Aug 16, 2022
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-core-infra (Team:Core/Infra)

@elasticsearchmachine elasticsearchmachine added Team:Core/Infra Meta label for core/infra team and removed Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. labels Aug 16, 2022
@grcevski grcevski self-assigned this Aug 16, 2022
@grcevski grcevski mentioned this issue Aug 16, 2022
Closed
@grcevski
Copy link
Contributor

It looks like we are not cleaning up the watcher subscription correctly.

@grcevski grcevski mentioned this issue Aug 17, 2022
Merged
@idegtiarenko idegtiarenko mentioned this issue Aug 23, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@grcevski grcevski

Labels
:Core/Infra/Core Core issues without another label Team:Core/Infra Meta label for core/infra team >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Clean-up all file watcher keys in FileSettingsService grcevski/elasticsearch
4 participants
@arteam @albertzaharovits @grcevski @elasticsearchmachine