in_forward emits "expected loop to be an instance of Coolio::Loop" error

[skaji]

I use fluentd in the following situation:

• linux (centos 6.5)
• ruby 2.2.3
• fluentd 0.12.17
• cool.io 1.4.1

|--fluentd1 ----------|     |-- fluentd2 ------------|
|       (out_forward) | --> | (in_forward)           |
|---------------------|     |------------------------|

When shutting down fluentd2, I encountered the following error:

2015-11-15 11:48:30 +0000 [info]: shutting down fluentd
2015-11-15 11:48:30 +0000 [info]: shutting down input type="forward" plugin_id="object:3fe6b4520550"
2015-11-15 11:48:30 +0000 [error]: unexpected error error=#<ArgumentError: expected loop to be an instance of Coolio::Loop> error_class=ArgumentError
2015-11-15 11:48:30 +0000 [error]: self dump: #<Fluent::ForwardInput:0x007fcd68a40aa0 @log_level=nil, @port=10001, @bind="0.0.0.0", @backlog=nil, @linger_timeout=0, @blocking_timeout=0.5, @chunk_size_warn_limit=nil, @chunk_size_limit=nil, @log=#<Fluent::Log:0x007fcd6823dc18 @out=#<IO:<STDOUT>>, @level=2, @debug_mode=false, @self_event=true, @tag="fluent", @time_format="%Y-%m-%d %H:%M:%S %z ", @depth_offset=1, @color_trace="", @color_debug="", @color_info="", @color_warn="", @color_error="", @color_fatal="", @color_reset="", @threads_exclude_events=[#<Thread:0x007fcd682fdb30@/home/app/vendor/bundle/ruby/2.2.0/bundler/gems/fluentd-d5742bae0319/lib/fluent/engine.rb:151 sleep>], @suppress_repeated_stacktrace=true>, @router=#<Fluent::EventRouter:0x007fcd68a45c80 @match_rules=[#<Fluent::EventRouter::Rule:0x007fcd68a411f8 @pattern=#<Fluent::AllMatchPattern:0x007fcd68a410e0>, @pattern_str="**", @collector=#<Fluent::NullOutput:0x007fcd68a421e8 @log_level=nil, @log=#<Fluent::Log:0x007fcd6823dc18 @out=#<IO:<STDOUT>>, @level=2, @debug_mode=false, @self_event=true, @tag="fluent", @time_format="%Y-%m-%d %H:%M:%S %z ", @depth_offset=1, @color_trace="", @color_debug="", @color_info="", @color_warn="", @color_error="", @color_fatal="", @color_reset="", @threads_exclude_events=[#<Thread:0x007fcd682fdb30@/home/app/vendor/bundle/ruby/2.2.0/bundler/gems/fluentd-d5742bae0319/lib/fluent/engine.rb:151 sleep>], @suppress_repeated_stacktrace=true>, @router=#<Fluent::EventRouter:0x007fcd68a45c80 ...>, @id=nil, @config=name:match, arg:**, {"@type"=>"null"}, [], @config_root_section=<Fluent::Config::Section {"log_level":null}>>>], @match_cache=#<Fluent::EventRouter::MatchCache:0x007fcd68a45e88 @map={"hoge"=>#<Fluent::NullOutput:0x007fcd68a421e8 @log_level=nil, @log=#<Fluent::Log:0x007fcd6823dc18 @out=#<IO:<STDOUT>>, @level=2, @debug_mode=false, @self_event=true, @tag="fluent", @time_format="%Y-%m-%d %H:%M:%S %z ", @depth_offset=1, @color_trace="", @color_debug="", @color_info="", @color_warn="", @color_error="", @color_fatal="", @color_reset="", @threads_exclude_events=[#<Thread:0x007fcd682fdb30@/home/app/vendor/bundle/ruby/2.2.0/bundler/gems/fluentd-d5742bae0319/lib/fluent/engine.rb:151 sleep>], @suppress_repeated_stacktrace=true>, @router=#<Fluent::EventRouter:0x007fcd68a45c80 ...>, @id=nil, @config=name:match, arg:**, {"@type"=>"null"}, [], @config_root_section=<Fluent::Config::Section {"log_level":null}>>, "fluent.warn"=>#<Fluent::NullOutput:0x007fcd68a421e8 @log_level=nil, @log=#<Fluent::Log:0x007fcd6823dc18 @out=#<IO:<STDOUT>>, @level=2, @debug_mode=false, @self_event=true, @tag="fluent", @time_format="%Y-%m-%d %H:%M:%S %z ", @depth_offset=1, @color_trace="", @color_debug="", @color_info="", @color_warn="", @color_error="", @color_fatal="", @color_reset="", @threads_exclude_events=[#<Thread:0x007fcd682fdb30@/home/app/vendor/bundle/ruby/2.2.0/bundler/gems/fluentd-d5742bae0319/lib/fluent/engine.rb:151 sleep>], @suppress_repeated_stacktrace=true>, @router=#<Fluent::EventRouter:0x007fcd68a45c80 ...>, @id=nil, @config=name:match, arg:**, {"@type"=>"null"}, [], @config_root_section=<Fluent::Config::Section {"log_level":null}>>}, @keys=["hoge", "fluent.warn"]>, @default_collector=#<Fluent::Agent::NoMatchMatch:0x007fcd68a45a50 @log=#<Fluent::Log:0x007fcd6823dc18 @out=#<IO:<STDOUT>>, @level=2, @debug_mode=false, @self_event=true, @tag="fluent", @time_format="%Y-%m-%d %H:%M:%S %z ", @depth_offset=1, @color_trace="", @color_debug="", @color_info="", @color_warn="", @color_error="", @color_fatal="", @color_reset="", @threads_exclude_events=[#<Thread:0x007fcd682fdb30@/home/app/vendor/bundle/ruby/2.2.0/bundler/gems/fluentd-d5742bae0319/lib/fluent/engine.rb:151 sleep>], @suppress_repeated_stacktrace=true>, @count=0>, @emit_error_handler=#<Fluent::RootAgent:0x007fcd68a446f0 @context=nil, @outputs=[#<Fluent::NullOutput:0x007fcd68a421e8 @log_level=nil, @log=#<Fluent::Log:0x007fcd6823dc18 @out=#<IO:<STDOUT>>, @level=2, @debug_mode=false, @self_event=true, @tag="fluent", @time_format="%Y-%m-%d %H:%M:%S %z ", @depth_offset=1, @color_trace="", @color_debug="", @color_info="", @color_warn="", @color_error="", @color_fatal="", @color_reset="", @threads_exclude_events=[#<Thread:0x007fcd682fdb30@/home/app/vendor/bundle/ruby/2.2.0/bundler/gems/fluentd-d5742bae0319/lib/fluent/engine.rb:151 sleep>], @suppress_repeated_stacktrace=true>, @router=#<Fluent::EventRouter:0x007fcd68a45c80 ...>, @id=nil, @config=name:match, arg:**, {"@type"=>"null"}, [], @config_root_section=<Fluent::Config::Section {"log_level":null}>>], @filters=[], @started_outputs=[#<Fluent::NullOutput:0x007fcd68a421e8 @log_level=nil, @log=#<Fluent::Log:0x007fcd6823dc18 @out=#<IO:<STDOUT>>, @level=2, @debug_mode=false, @self_event=true, @tag="fluent", @time_format="%Y-%m-%d %H:%M:%S %z ", @depth_offset=1, @color_trace="", @color_debug="", @color_info="", @color_warn="", @color_error="", @color_fatal="", @color_reset="", @threads_exclude_events=[#<Thread:0x007fcd682fdb30@/home/app/vendor/bundle/ruby/2.2.0/bundler/gems/fluentd-d5742bae0319/lib/fluent/engine.rb:151 sleep>], @suppress_repeated_stacktrace=true>, @router=#<Fluent::EventRouter:0x007fcd68a45c80 ...>, @id=nil, @config=name:match, arg:**, {"@type"=>"null"}, [], @config_root_section=<Fluent::Config::Section {"log_level":null}>>], @started_filters=[], @log=#<Fluent::Log:0x007fcd6823dc18 @out=#<IO:<STDOUT>>, @level=2, @debug_mode=false, @self_event=true, @tag="fluent", @time_format="%Y-%m-%d %H:%M:%S %z ", @depth_offset=1, @color_trace="", @color_debug="", @color_info="", @color_warn="", @color_error="", @color_fatal="", @color_reset="", @threads_exclude_events=[#<Thread:0x007fcd682fdb30@/home/app/vendor/bundle/ruby/2.2.0/bundler/gems/fluentd-d5742bae0319/lib/fluent/engine.rb:151 sleep>], @suppress_repeated_stacktrace=true>, @event_router=#<Fluent::EventRouter:0x007fcd68a45c80 ...>, @error_collector=nil, @labels={}, @inputs=[#<Fluent::ForwardInput:0x007fcd68a40aa0 ...>], @started_inputs=[#<Fluent::ForwardInput:0x007fcd68a40aa0 ...>], @suppress_emit_error_log_interval=0, @next_emit_error_log_time=1447588107, @config=name:ROOT, arg:, {}, [name:source, arg:, {"@type"=>"forward", "port"=>"10001"}, [], name:match, arg:**, {"@type"=>"null"}, []], @config_root_section=<Fluent::Config::Section {}>>, @chain=#<Fluent::NullOutputChain:0x007fcd68a47620>>, @id=nil, @config=name:source, arg:, {"@type"=>"forward", "port"=>"10001"}, [], @config_root_section=<Fluent::Config::Section {"log_level":null,"port":10001,"bind":"0.0.0.0","backlog":null,"linger_timeout":0,"blocking_timeout":0.5,"chunk_size_warn_limit":null,"chunk_size_limit":null}>, @loop=#<Coolio::Loop:0x007fcd68a3b050 @watchers={}, @active_watchers=0, @loop=nil, @running=false>, @lsock=#<Coolio::TCPServer:0x007fcd68a3aab0 @block=nil, @args=[0, #<Fluent::Log:0x007fcd6823dc18 @out=#<IO:<STDOUT>>, @level=2, @debug_mode=false, @self_event=true, @tag="fluent", @time_format="%Y-%m-%d %H:%M:%S %z ", @depth_offset=1, @color_trace="", @color_debug="", @color_info="", @color_warn="", @color_error="", @color_fatal="", @color_reset="", @threads_exclude_events=[#<Thread:0x007fcd682fdb30@/home/app/vendor/bundle/ruby/2.2.0/bundler/gems/fluentd-d5742bae0319/lib/fluent/engine.rb:151 sleep>], @suppress_repeated_stacktrace=true>, #<Method: Fluent::ForwardInput#on_message>], @klass=Fluent::ForwardInput::Handler, @listen_socket=#<TCPServer:fd 8>>, @usock=#<UDPSocket:(closed)>, @hbr=#<Fluent::ForwardInput::HeartbeatRequestHandler:0x007fcd682fe288 @_io=#<UDPSocket:(closed)>, @_write_buffer=#<IO::Buffer:0x007fcd682fe260>, @_read_watcher=#<Coolio::IO::Watcher:0x007fcd682fe238 @coolio_io=#<Fluent::ForwardInput::HeartbeatRequestHandler:0x007fcd682fe288 ...>>, @_write_watcher=#<Coolio::IO::Watcher:0x007fcd682fe120 @coolio_io=#<Fluent::ForwardInput::HeartbeatRequestHandler:0x007fcd682fe288 ...>>, @io=#<UDPSocket:(closed)>, @callback=#<Method: Fluent::ForwardInput#on_heartbeat_request>>, @thread=#<Thread:0x007fcd682fde78@/home/app/vendor/bundle/ruby/2.2.0/bundler/gems/fluentd-d5742bae0319/lib/fluent/plugin/in_forward.rb:90 run>>
2015-11-15 11:48:30 +0000 [error]: loop dump: #<Coolio::Loop:0x007fcd68a3b050 @watchers={}, @active_watchers=0, @loop=nil, @running=false>
  2015-11-15 11:48:30 +0000 [error]: /home/app/vendor/bundle/ruby/2.2.0/gems/cool.io-1.4.1/lib/cool.io/io.rb:35:in `attach'
  2015-11-15 11:48:30 +0000 [error]: /home/app/vendor/bundle/ruby/2.2.0/gems/cool.io-1.4.1/lib/cool.io/io.rb:35:in `attach'
  2015-11-15 11:48:30 +0000 [error]: /home/app/vendor/bundle/ruby/2.2.0/gems/cool.io-1.4.1/lib/cool.io/socket.rb:38:in `attach'
  2015-11-15 11:48:30 +0000 [error]: /home/app/vendor/bundle/ruby/2.2.0/gems/cool.io-1.4.1/lib/cool.io/server.rb:40:in `on_connection'
  2015-11-15 11:48:30 +0000 [error]: /home/app/vendor/bundle/ruby/2.2.0/gems/cool.io-1.4.1/lib/cool.io/listener.rb:45:in `on_readable'
  2015-11-15 11:48:30 +0000 [error]: /home/app/vendor/bundle/ruby/2.2.0/gems/cool.io-1.4.1/lib/cool.io/loop.rb:88:in `run_once'
  2015-11-15 11:48:30 +0000 [error]: /home/app/vendor/bundle/ruby/2.2.0/gems/cool.io-1.4.1/lib/cool.io/loop.rb:88:in `run'
  2015-11-15 11:48:30 +0000 [error]: /home/app/vendor/bundle/ruby/2.2.0/bundler/gems/fluentd-d5742bae0319/lib/fluent/plugin/in_forward.rb:91:in `run'

The reproducing step is here: https://github.com/shoichikaji/fluentd-error

Do you have any ideas why this happened?

[repeatedly]

Thanks for the report.
I can't reproduce this problem on our environment.

If you have a time, could you try to reproduce this problem with cool.io v1.4.2?

socketry/cool.io@f4b14c8

v1.4.2 includes unexpected object information in error message.

[skaji]

@repeatedly Thank you. I will try cool.io v1.4.2.

[skaji]

with cool.io v1.4.2

2015-11-16 14:08:50 +0000 [info]: shutting down fluentd
2015-11-16 14:08:50 +0000 [info]: shutting down input type="forward" plugin_id="object:3f9cb30be160"
2015-11-16 14:08:50 +0000 [error]: unexpected error error=#<TypeError: wrong argument type nil (expected Data)> error_class=TypeError
2015-11-16 14:08:50 +0000 [error]: self dump: #<Fluent::ForwardInput:0x007f396617c2c0 @log_level=nil, @port=10001, @bind="0.0.0.0", @backlog=nil, @linger_timeout=0, @blocking_timeout=0.5, @chunk_size_warn_limit=nil, @chunk_size_limit=nil, @log=#<Fluent::Log:0x007f3965953aa8 @out=#<IO:<STDOUT>>, @level=2, @debug_mode=false, @self_event=true, @tag="fluent", @time_format="%Y-%m-%d %H:%M:%S %z ", @depth_offset=1, @color_trace="", @color_debug="", @color_info="", @color_warn="", @color_error="", @color_fatal="", @color_reset="", @threads_exclude_events=[#<Thread:0x007f3965a38c98@/home/app/vendor/bundle/ruby/2.2.0/bundler/gems/fluentd-d5742bae0319/lib/fluent/engine.rb:151 sleep>], @suppress_repeated_stacktrace=true>, @router=#<Fluent::EventRouter:0x007f396617f830 @match_rules=[#<Fluent::EventRouter::Rule:0x007f396617cae0 @pattern=#<Fluent::AllMatchPattern:0x007f396617c9f0>, @pattern_str="**", @collector=#<Fluent::NullOutput:0x007f396617d9e0 @log_level=nil, @log=#<Fluent::Log:0x007f3965953aa8 @out=#<IO:<STDOUT>>, @level=2, @debug_mode=false, @self_event=true, @tag="fluent", @time_format="%Y-%m-%d %H:%M:%S %z ", @depth_offset=1, @color_trace="", @color_debug="", @color_info="", @color_warn="", @color_error="", @color_fatal="", @color_reset="", @threads_exclude_events=[#<Thread:0x007f3965a38c98@/home/app/vendor/bundle/ruby/2.2.0/bundler/gems/fluentd-d5742bae0319/lib/fluent/engine.rb:151 sleep>], @suppress_repeated_stacktrace=true>, @router=#<Fluent::EventRouter:0x007f396617f830 ...>, @id=nil, @config=name:match, arg:**, {"@type"=>"null"}, [], @config_root_section=<Fluent::Config::Section {"log_level":null}>>>], @match_cache=#<Fluent::EventRouter::MatchCache:0x007f396617f7b8 @map={"hoge"=>#<Fluent::NullOutput:0x007f396617d9e0 @log_level=nil, @log=#<Fluent::Log:0x007f3965953aa8 @out=#<IO:<STDOUT>>, @level=2, @debug_mode=false, @self_event=true, @tag="fluent", @time_format="%Y-%m-%d %H:%M:%S %z ", @depth_offset=1, @color_trace="", @color_debug="", @color_info="", @color_warn="", @color_error="", @color_fatal="", @color_reset="", @threads_exclude_events=[#<Thread:0x007f3965a38c98@/home/app/vendor/bundle/ruby/2.2.0/bundler/gems/fluentd-d5742bae0319/lib/fluent/engine.rb:151 sleep>], @suppress_repeated_stacktrace=true>, @router=#<Fluent::EventRouter:0x007f396617f830 ...>, @id=nil, @config=name:match, arg:**, {"@type"=>"null"}, [], @config_root_section=<Fluent::Config::Section {"log_level":null}>>, "fluent.warn"=>#<Fluent::NullOutput:0x007f396617d9e0 @log_level=nil, @log=#<Fluent::Log:0x007f3965953aa8 @out=#<IO:<STDOUT>>, @level=2, @debug_mode=false, @self_event=true, @tag="fluent", @time_format="%Y-%m-%d %H:%M:%S %z ", @depth_offset=1, @color_trace="", @color_debug="", @color_info="", @color_warn="", @color_error="", @color_fatal="", @color_reset="", @threads_exclude_events=[#<Thread:0x007f3965a38c98@/home/app/vendor/bundle/ruby/2.2.0/bundler/gems/fluentd-d5742bae0319/lib/fluent/engine.rb:151 sleep>], @suppress_repeated_stacktrace=true>, @router=#<Fluent::EventRouter:0x007f396617f830 ...>, @id=nil, @config=name:match, arg:**, {"@type"=>"null"}, [], @config_root_section=<Fluent::Config::Section {"log_level":null}>>}, @keys=["hoge", "fluent.warn"]>, @default_collector=#<Fluent::Agent::NoMatchMatch:0x007f396617f880 @log=#<Fluent::Log:0x007f3965953aa8 @out=#<IO:<STDOUT>>, @level=2, @debug_mode=false, @self_event=true, @tag="fluent", @time_format="%Y-%m-%d %H:%M:%S %z ", @depth_offset=1, @color_trace="", @color_debug="", @color_info="", @color_warn="", @color_error="", @color_fatal="", @color_reset="", @threads_exclude_events=[#<Thread:0x007f3965a38c98@/home/app/vendor/bundle/ruby/2.2.0/bundler/gems/fluentd-d5742bae0319/lib/fluent/engine.rb:151 sleep>], @suppress_repeated_stacktrace=true>, @count=0>, @emit_error_handler=#<Fluent::RootAgent:0x007f3966180780 @context=nil, @outputs=[#<Fluent::NullOutput:0x007f396617d9e0 @log_level=nil, @log=#<Fluent::Log:0x007f3965953aa8 @out=#<IO:<STDOUT>>, @level=2, @debug_mode=false, @self_event=true, @tag="fluent", @time_format="%Y-%m-%d %H:%M:%S %z ", @depth_offset=1, @color_trace="", @color_debug="", @color_info="", @color_warn="", @color_error="", @color_fatal="", @color_reset="", @threads_exclude_events=[#<Thread:0x007f3965a38c98@/home/app/vendor/bundle/ruby/2.2.0/bundler/gems/fluentd-d5742bae0319/lib/fluent/engine.rb:151 sleep>], @suppress_repeated_stacktrace=true>, @router=#<Fluent::EventRouter:0x007f396617f830 ...>, @id=nil, @config=name:match, arg:**, {"@type"=>"null"}, [], @config_root_section=<Fluent::Config::Section {"log_level":null}>>], @filters=[], @started_outputs=[#<Fluent::NullOutput:0x007f396617d9e0 @log_level=nil, @log=#<Fluent::Log:0x007f3965953aa8 @out=#<IO:<STDOUT>>, @level=2, @debug_mode=false, @self_event=true, @tag="fluent", @time_format="%Y-%m-%d %H:%M:%S %z ", @depth_offset=1, @color_trace="", @color_debug="", @color_info="", @color_warn="", @color_error="", @color_fatal="", @color_reset="", @threads_exclude_events=[#<Thread:0x007f3965a38c98@/home/app/vendor/bundle/ruby/2.2.0/bundler/gems/fluentd-d5742bae0319/lib/fluent/engine.rb:151 sleep>], @suppress_repeated_stacktrace=true>, @router=#<Fluent::EventRouter:0x007f396617f830 ...>, @id=nil, @config=name:match, arg:**, {"@type"=>"null"}, [], @config_root_section=<Fluent::Config::Section {"log_level":null}>>], @started_filters=[], @log=#<Fluent::Log:0x007f3965953aa8 @out=#<IO:<STDOUT>>, @level=2, @debug_mode=false, @self_event=true, @tag="fluent", @time_format="%Y-%m-%d %H:%M:%S %z ", @depth_offset=1, @color_trace="", @color_debug="", @color_info="", @color_warn="", @color_error="", @color_fatal="", @color_reset="", @threads_exclude_events=[#<Thread:0x007f3965a38c98@/home/app/vendor/bundle/ruby/2.2.0/bundler/gems/fluentd-d5742bae0319/lib/fluent/engine.rb:151 sleep>], @suppress_repeated_stacktrace=true>, @event_router=#<Fluent::EventRouter:0x007f396617f830 ...>, @error_collector=nil, @labels={}, @inputs=[#<Fluent::ForwardInput:0x007f396617c2c0 ...>], @started_inputs=[#<Fluent::ForwardInput:0x007f396617c2c0 ...>], @suppress_emit_error_log_interval=0, @next_emit_error_log_time=1447682926, @config=name:ROOT, arg:, {}, [name:source, arg:, {"@type"=>"forward", "port"=>"10001"}, [], name:match, arg:**, {"@type"=>"null"}, []], @config_root_section=<Fluent::Config::Section {}>>, @chain=#<Fluent::NullOutputChain:0x007f396617f740>>, @id=nil, @config=name:source, arg:, {"@type"=>"forward", "port"=>"10001"}, [], @config_root_section=<Fluent::Config::Section {"log_level":null,"port":10001,"bind":"0.0.0.0","backlog":null,"linger_timeout":0,"blocking_timeout":0.5,"chunk_size_warn_limit":null,"chunk_size_limit":null}>, @loop=#<Coolio::Loop:0x007f3966176780 @watchers={}, @active_watchers=0, @loop=nil, @running=false>, @lsock=#<Coolio::TCPServer:0x007f39661761e0 @block=nil, @args=[0, #<Fluent::Log:0x007f3965953aa8 @out=#<IO:<STDOUT>>, @level=2, @debug_mode=false, @self_event=true, @tag="fluent", @time_format="%Y-%m-%d %H:%M:%S %z ", @depth_offset=1, @color_trace="", @color_debug="", @color_info="", @color_warn="", @color_error="", @color_fatal="", @color_reset="", @threads_exclude_events=[#<Thread:0x007f3965a38c98@/home/app/vendor/bundle/ruby/2.2.0/bundler/gems/fluentd-d5742bae0319/lib/fluent/engine.rb:151 sleep>], @suppress_repeated_stacktrace=true>, #<Method: Fluent::ForwardInput#on_message>], @klass=Fluent::ForwardInput::Handler, @listen_socket=#<TCPServer:fd 8>>, @usock=#<UDPSocket:(closed)>, @hbr=#<Fluent::ForwardInput::HeartbeatRequestHandler:0x007f3965a397d8 @_io=#<UDPSocket:(closed)>, @_write_buffer=#<IO::Buffer:0x007f3965a39760>, @_read_watcher=#<Coolio::IO::Watcher:0x007f3965a39670 @coolio_io=#<Fluent::ForwardInput::HeartbeatRequestHandler:0x007f3965a397d8 ...>>, @_write_watcher=#<Coolio::IO::Watcher:0x007f3965a39530 @coolio_io=#<Fluent::ForwardInput::HeartbeatRequestHandler:0x007f3965a397d8 ...>>, @io=#<UDPSocket:(closed)>, @callback=#<Method: Fluent::ForwardInput#on_heartbeat_request>>, @thread=#<Thread:0x007f3965a39288@/home/app/vendor/bundle/ruby/2.2.0/bundler/gems/fluentd-d5742bae0319/lib/fluent/plugin/in_forward.rb:90 run>>
2015-11-16 14:08:50 +0000 [error]: loop dump: #<Coolio::Loop:0x007f3966176780 @watchers={}, @active_watchers=0, @loop=nil, @running=false>
  2015-11-16 14:08:50 +0000 [error]: /home/app/vendor/bundle/ruby/2.2.0/gems/cool.io-1.4.2/lib/cool.io/io.rb:43:in `detach'
  2015-11-16 14:08:50 +0000 [error]: /home/app/vendor/bundle/ruby/2.2.0/gems/cool.io-1.4.2/lib/cool.io/io.rb:43:in `detach'
  2015-11-16 14:08:50 +0000 [error]: /home/app/vendor/bundle/ruby/2.2.0/gems/cool.io-1.4.2/lib/cool.io/io.rb:43:in `detach'
  2015-11-16 14:08:50 +0000 [error]: (eval):7:in `detach'
  2015-11-16 14:08:50 +0000 [error]: /home/app/vendor/bundle/ruby/2.2.0/gems/cool.io-1.4.2/lib/cool.io/io.rb:103:in `close'
  2015-11-16 14:08:50 +0000 [error]: /home/app/vendor/bundle/ruby/2.2.0/gems/cool.io-1.4.2/lib/cool.io/io.rb:129:in `rescue in on_readable'
  2015-11-16 14:08:50 +0000 [error]: /home/app/vendor/bundle/ruby/2.2.0/gems/cool.io-1.4.2/lib/cool.io/io.rb:122:in `on_readable'
  2015-11-16 14:08:50 +0000 [error]: /home/app/vendor/bundle/ruby/2.2.0/gems/cool.io-1.4.2/lib/cool.io/io.rb:186:in `on_readable'
  2015-11-16 14:08:50 +0000 [error]: /home/app/vendor/bundle/ruby/2.2.0/gems/cool.io-1.4.2/lib/cool.io/loop.rb:88:in `run_once'
  2015-11-16 14:08:50 +0000 [error]: /home/app/vendor/bundle/ruby/2.2.0/gems/cool.io-1.4.2/lib/cool.io/loop.rb:88:in `run'
  2015-11-16 14:08:50 +0000 [error]: /home/app/vendor/bundle/ruby/2.2.0/bundler/gems/fluentd-d5742bae0319/lib/fluent/plugin/in_forward.rb:91:in `run'

[repeatedly]

Weird. Error changed...

[repeatedly]

Anyway, I assume error in detach happen by race condition.

• Watcher#detach set loop to nil in main thread: https://github.com/tarcieri/cool.io/blob/68b203bdc16a0d31e0a00779370c9ff84b5e289f/ext/cool.io/watcher.c#L170
• Get loop after nil check in another thread: https://github.com/tarcieri/cool.io/blob/68b203bdc16a0d31e0a00779370c9ff84b5e289f/ext/cool.io/watcher.h#L35

I will check this problem without fluentd code.

[repeatedly]

Exception happens but fluentd continues to shutdown other plugins.
So this problem seems not a critical problem but should be fixed.

[skaji]

@repeatedly Thank you agiain.
I could reproduce "expected loop to be an instance of Coolio::Loop" error with cool.io 1.4.2 too.
https://github.com/shoichikaji/fluentd-error/blob/master/log/fluentd-only-error-coolio-1.4.2.log

[repeatedly]

@shoichikaji Thanks! I will try to reproduce this race condition issue later.

BTW, for detach, we need nil check before get loop object.

[gcs-github]

I stumbled upon this while running the latest git tip. What information can I give you which would be useful?

[tagomoris]

@repeatedly What's the current status of this issue?

[repeatedly]

v0.14 now ignores this exception so fluentd v0.14 doesn't have this problem.
We can close this issue on this repository.