After some time the serveric getting error

[roiromanoclick]

Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:nginx, :exception=>"LogStash::PluginLoadingError", :message=>"Couldn't find any output plugin named 'amazon_es'. Are you sure this is correct? Trying to load the amazon_es output plugin resulted in this error: no such file to load -- logstash/outputs/amazon_es", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/plugins/registry.rb:211:in lookup_pipeline_plugin'", "/usr/share/logstash/logstash-core/lib/logstash/plugin.rb:137:in lookup'", "org/logstash/plugins/PluginFactoryExt.java:200:in plugin'", "org/logstash/plugins/PluginFactoryExt.java:184:in plugin'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:71:in plugin'", "(eval):87:in initialize'", "org/jruby/RubyKernel.java:1061:in eval'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:49:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:90:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:43:in block in execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:96:in block in exclusive'", "org/jruby/ext/thread/Mutex.java:165:in synchronize'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:96:in exclusive'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:39:in execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:334:in `block in converge_state'"]}

After I'm reinstall the plugin, its staring to work for while.
Plugin version - 6.4.1

What can be the issue?

Thanks a lot.

[roiromanoclick]

The plugin is exist in the plugin list - "logstash-output-amazon_es
"

[blorby]

@roiromanoclick - was that resolved or found the source of the issue?

[davejennings]

We got this after an update to logstash was installed via apt and/or possibly when the server was rebooted shortly after that.

Ubuntu 18.04 installation, package upgrade:

$ grep logstash /var/log/dpkg.log.1
2021-02-26 06:21:07 upgrade logstash:amd64 1:7.11.0-1 1:7.11.1-1
2021-02-26 06:21:07 status half-configured logstash:amd64 1:7.11.0-1
2021-02-26 06:21:07 status unpacked logstash:amd64 1:7.11.0-1
2021-02-26 06:21:07 status half-installed logstash:amd64 1:7.11.0-1
2021-02-26 06:21:15 status half-installed logstash:amd64 1:7.11.0-1
2021-02-26 06:21:15 status unpacked logstash:amd64 1:7.11.1-1
2021-02-26 06:21:15 status unpacked logstash:amd64 1:7.11.1-1
2021-02-26 06:21:15 configure logstash:amd64 1:7.11.1-1 <none>
2021-02-26 06:21:15 status unpacked logstash:amd64 1:7.11.1-1
2021-02-26 06:21:15 status unpacked logstash:amd64 1:7.11.1-1
2021-02-26 06:21:15 status unpacked logstash:amd64 1:7.11.1-1
2021-02-26 06:21:15 status unpacked logstash:amd64 1:7.11.1-1
2021-02-26 06:21:15 status unpacked logstash:amd64 1:7.11.1-1
2021-02-26 06:21:15 status unpacked logstash:amd64 1:7.11.1-1
2021-02-26 06:21:15 status unpacked logstash:amd64 1:7.11.1-1
2021-02-26 06:21:15 status half-configured logstash:amd64 1:7.11.1-1
2021-02-26 06:21:19 status installed logstash:amd64 1:7.11.1-1

Then a reboot shortly after at 06:23 and on boot it fails to load the plugin. Excerpt from syslog:

Feb 26 06:24:49 utilities logstash[867]: [2021-02-26T06:24:49,852][ERROR][logstash.plugins.registry] Tried to load a plugin's code, but failed. {:exception=>#<LoadError: no such file to load -- logstash/outputs/amazon_es>, :path=>"logstash/outputs/amazon_es", :type=>"output", :name=>"amazon_es"}
Feb 26 06:24:49 utilities logstash[867]: [2021-02-26T06:24:49,864][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: (PluginLoadingError) Couldn't find any output plugin named 'amazon_es'. Are you sure this is correct? Trying to load the amazon_es output plugin resulted in this error: no such file to load -- logstash/outputs/amazon_es", :backtrace=>["org.logstash.config.ir.CompiledPipeline.<init>(CompiledPipeline.java:119)", "org.logstash.execution.JavaBasePipelineExt.initialize(JavaBasePipelineExt.java:83)", "org.logstash.execution.JavaBasePipelineExt$INVOKER$i$1$0$initialize.call(JavaBasePipelineExt$INVOKER$i$1$0$initialize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:837)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1169)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuperSplatArgs(IRRuntimeHelpers.java:1156)", "org.jruby.ir.targets.InstanceSuperInvokeSite.invoke(InstanceSuperInvokeSite.java:39)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$initialize$0(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:47)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:332)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:86)", "org.jruby.RubyClass.newInstance(RubyClass.java:939)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0(/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0$__VARARGS__(/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)", "usr.share.logstash.logstash_minus_core.lib.logstash.agent.RUBY$block$converge_state$2(/usr/share/logstash/logstash-core/lib/logstash/agent.rb:367)", "org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:138)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:52)", "org.jruby.runtime.Block.call(Block.java:139)", "org.jruby.RubyProc.call(RubyProc.java:318)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105)", "java.base/java.lang.Thread.run(Thread.java:834)"]}
Feb 26 06:24:49 utilities logstash[867]: warning: thread "Converge PipelineAction::Create<main>" terminated with exception (report_on_exception is true):
Feb 26 06:24:49 utilities logstash[867]: LogStash::Error: Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<main>`
Feb 26 06:24:49 utilities logstash[867]:           create at org/logstash/execution/ConvergeResultExt.java:129
Feb 26 06:24:49 utilities logstash[867]:              add at org/logstash/execution/ConvergeResultExt.java:57
Feb 26 06:24:49 utilities logstash[867]:   converge_state at /usr/share/logstash/logstash-core/lib/logstash/agent.rb:380
Feb 26 06:24:49 utilities logstash[867]: [2021-02-26T06:24:49,874][ERROR][logstash.agent           ] An exception happened when converging configuration {:exception=>LogStash::Error, :message=>"Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<main>`"}
Feb 26 06:24:49 utilities logstash[867]: [2021-02-26T06:24:49,925][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>#<LogStash::Error: Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<main>`>, :backtrace=>["org/logstash/execution/ConvergeResultExt.java:129:in `create'", "org/logstash/execution/ConvergeResultExt.java:57:in `add'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:380:in `block in converge_state'"]}
Feb 26 06:24:49 utilities logstash[867]: [2021-02-26T06:24:49,964][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (SystemExit) exit
Feb 26 06:24:49 utilities logstash[867]: org.jruby.exceptions.SystemExit: (SystemExit) exit
Feb 26 06:24:49 utilities logstash[867]: #011at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.13.0.jar:?]
Feb 26 06:24:49 utilities logstash[867]: #011at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.13.0.jar:?]
Feb 26 06:24:49 utilities logstash[867]: #011at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:89) ~[?:?]
Feb 26 06:24:50 utilities systemd[1]: logstash.service: Main process exited, code=exited, status=1/FAILURE
Feb 26 06:24:50 utilities systemd[1]: logstash.service: Failed with result 'exit-code'.
Feb 26 06:24:50 utilities systemd[1]: logstash.service: Service hold-off time over, scheduling restart.
Feb 26 06:24:50 utilities systemd[1]: logstash.service: Scheduled restart job, restart counter is at 1.
Feb 26 06:24:50 utilities systemd[1]: Stopped logstash.
Feb 26 06:24:50 utilities systemd[1]: Started logstash.

Works fine after reinstalling.