Guice Creation errors on java.time.Clock [BUG]

[gamebrigada]

Describe the bug
Elasticsearch anomaly detection plugin version 1.11.0.0 not functioning in upgraded cluster. 1.10.1.0 works fine.

Other plugins installed
All the other default plugins of OpenDistro also Wazuh

To Reproduce
Install anomaly detection plugin on existing cluster

Stacktrace

 [2020-10-29T11:39:24,432][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [master2.example.com] uncaught exception in thread [main]
 org.elasticsearch.bootstrap.StartupException: org.elasticsearch.common.inject.CreationException: Guice creation errors:
 1) A binding to java.time.Clock was already configured at _unknown_.
   at _unknown_
 1 error
         at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:174) ~[elasticsearch-7.9.1.jar:7.9.1]
         at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) ~[elasticsearch-7.9.1.jar:7.9.1]
         at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.9.1.jar:7.9.1]
         at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127) ~[elasticsearch-cli-7.9.1.jar:7.9.1]
         at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.9.1.jar:7.9.1]
         at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) ~[elasticsearch-7.9.1.jar:7.9.1]
         at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.9.1.jar:7.9.1]
 Caused by: org.elasticsearch.common.inject.CreationException: Guice creation errors:
 1) A binding to java.time.Clock was already configured at _unknown_.
   at _unknown_
 1 error
         at org.elasticsearch.common.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:361) ~[elasticsearch-7.9.1.jar:7.9.1]
         at org.elasticsearch.common.inject.InjectorBuilder.initializeStatically(InjectorBuilder.java:136) ~[elasticsearch-7.9.1.jar:7.9.1]
         at org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:92) ~[elasticsearch-7.9.1.jar:7.9.1]
         at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:96) ~[elasticsearch-7.9.1.jar:7.9.1]
         at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:70) ~[elasticsearch-7.9.1.jar:7.9.1]
         at org.elasticsearch.common.inject.ModulesBuilder.createInjector(ModulesBuilder.java:42) ~[elasticsearch-7.9.1.jar:7.9.1]
         at org.elasticsearch.node.Node.<init>(Node.java:669) ~[elasticsearch-7.9.1.jar:7.9.1]
         at org.elasticsearch.node.Node.<init>(Node.java:277) ~[elasticsearch-7.9.1.jar:7.9.1]
         at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:227) ~[elasticsearch-7.9.1.jar:7.9.1]
         at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[elasticsearch-7.9.1.jar:7.9.1]
         at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) ~[elasticsearch-7.9.1.jar:7.9.1]
         at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.9.1.jar:7.9.1]
         ... 6 more
 uncaught exception in thread [main]
 org.elasticsearch.common.inject.CreationException: Guice creation errors:
 1) A binding to java.time.Clock was already configured at _unknown_.
   at _unknown_
 1 error
         at <<<guice>>>
         at org.elasticsearch.node.Node.<init>(Node.java:669)
         at org.elasticsearch.node.Node.<init>(Node.java:277)
         at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:227)
         at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227)
         at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393)
         at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170)
         at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161)
         at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
         at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127)
         at org.elasticsearch.cli.Command.main(Command.java:90)
         at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126)
         at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92)

Desktop (please complete the following information):

• OS: CentOS
• Version 1.11.0

[khangcnttspkt]

I agree with you
[2020-10-29T09:45:43,602][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [elasticsearch2] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: org.elasticsearch.common.inject.CreationException: Guice creation errors:

1. A binding to java.time.Clock was already configured at org.elasticsearch.xpack.watcher.Watcher.lambda$createGuiceModules$3(Watcher.java:453).
   at com.amazon.opendistroforelasticsearch.ad.AnomalyDetectorPlugin.lambda$createGuiceModules$0(AnomalyDetectorPlugin.java:542)

1 error
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:174) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127) ~[elasticsearch-cli-7.9.1.jar:7.9.1]
at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.9.1.jar:7.9.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.9.1.jar:7.9.1]
Caused by: org.elasticsearch.common.inject.CreationException: Guice creation errors:

1. A binding to java.time.Clock was already configured at org.elasticsearch.xpack.watcher.Watcher.lambda$createGuiceModules$3(Watcher.java:453).
   at com.amazon.opendistroforelasticsearch.ad.AnomalyDetectorPlugin.lambda$createGuiceModules$0(AnomalyDetectorPlugin.java:542)

1 error
at org.elasticsearch.common.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:361) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.common.inject.InjectorBuilder.initializeStatically(InjectorBuilder.java:136) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:92) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:96) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:70) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.common.inject.ModulesBuilder.createInjector(ModulesBuilder.java:42) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.node.Node.(Node.java:669) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.node.Node.(Node.java:277) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:227) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) ~[elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.9.1.jar:7.9.1]
... 6 more

Do you fixed it? @gamebrigada

[wnbts]

thanks for using anomaly detection and reporting the issue!

There seems to be a dependency injection configuration issue. Feel free to try making a fix before the authors do.

[ylwu-amzn]

Seems the issue caused by https://github.com/opendistro-for-elasticsearch/anomaly-detection/blob/master/src/main/java/com/amazon/opendistroforelasticsearch/ad/AnomalyDetectorPlugin.java#L531,

Will fix and test it.

[ylwu-amzn]

@gamebrigada , @khangcnttspkt can you describe more details about your cluster like you install ODFE1.10.1 with RPM? And how did you install AD latest plugin?
I tried to install ODFE 1.10.1 with tarball and install AD 1.11.0, not able to reproduce this issue.

[gamebrigada]

@ylwu-amzn
Installed via standalone plugin install since we have a large cluster. Elasticsearch version 7.9.1. Had no problems with AD 1.10.1. I can see that codebase changed dramatically.

[ylwu-amzn]

I can't reproduce this issue by these steps:
1.Install ODFE1.10.1 with tarball and start a detector
2.Stop cluster and Install latest stoneAD version following this doc https://opendistro.github.io/for-elasticsearch-docs/docs/install/plugins/#anomaly-detection
3.Restart cluster, can't see the error "A binding to java.time.Clock was already configured at unknown"

Did you installed other plugins except AD?

[gamebrigada]

Wazuh plugin is also installed.

[gamebrigada]

If another module is interfering, wouldn't it be best practice to bind safely to prevent this kind of problems?

[ylwu-amzn]

Yeah, totally agree. Just want to know where the Clock is bound except AD.

[ylwu-amzn]

Removed Clock binding. PR merged, you can build latest AD plugin from source code. I have built one locally, you can download directly.
opendistro-anomaly-detection-1.11.0.0.zip

[gamebrigada]

You guys rock! Thanks!