Failure 'connection closed by remote host' happens when knife solo cook is executed under cygwin

[keeruline]

Failure 'connection closed by remote host' happens when knife solo cook is executed under cygwin. Last messages from command output are following:

DEBUG: Running processed command ver
DEBUG: ver stdout: bash: ver: command not found

DEBUG: Initial command mkdir -p -m 0700 ~/chef-solo
DEBUG: Running processed command mkdir -p -m 0700 ~/chef-solo
DEBUG: No chefignore file found at ./chefignore no files will be ignored
DEBUG: Initial command ver
DEBUG: Running processed command ver
DEBUG: ver stdout: bash: ver: command not found

DEBUG: ["rsync", "-rL", "-v", "--rsh=ssh user@lucid", "--delete-after", "--exclude=revision-deploys", "--exclude=tmp", "--exclude=.git", "--exclude=.hg", "--exclude=.svn", "--exclude=.bzr", "/usr/lib/ruby/gems/1.9.1/bundler/gems/knife-solo-1e1675f017e4/lib/knife-solo/resources/patch_cookbooks/", ":~/chef-solo/cookbooks-1"]
building file list ... done

sent 117 bytes  received 12 bytes  258.00 bytes/sec
total size is 0  speedup is 0.00
DEBUG: Initial command ver
DEBUG: Running processed command ver
/usr/lib/ruby/gems/1.9.1/gems/net-ssh-2.7.0/lib/net/ssh/transport/packet_stream.rb:87:in `next_packet': connection closed by remote host (Net::SSH::Disconnect)
        from /usr/lib/ruby/gems/1.9.1/gems/net-ssh-2.7.0/lib/net/ssh/transport/session.rb:172:in `block in poll_message'
        from /usr/lib/ruby/gems/1.9.1/gems/net-ssh-2.7.0/lib/net/ssh/transport/session.rb:167:in `loop'
        from /usr/lib/ruby/gems/1.9.1/gems/net-ssh-2.7.0/lib/net/ssh/transport/session.rb:167:in `poll_message'
        from /usr/lib/ruby/gems/1.9.1/gems/net-ssh-2.7.0/lib/net/ssh/connection/session.rb:460:in `dispatch_incoming_packets'
        from /usr/lib/ruby/gems/1.9.1/gems/net-ssh-2.7.0/lib/net/ssh/connection/session.rb:221:in `preprocess'
        from /usr/lib/ruby/gems/1.9.1/gems/net-ssh-2.7.0/lib/net/ssh/connection/session.rb:205:in `process'
        from /usr/lib/ruby/gems/1.9.1/gems/net-ssh-2.7.0/lib/net/ssh/connection/session.rb:169:in `block in loop'
        from /usr/lib/ruby/gems/1.9.1/gems/net-ssh-2.7.0/lib/net/ssh/connection/session.rb:169:in `loop'
        from /usr/lib/ruby/gems/1.9.1/gems/net-ssh-2.7.0/lib/net/ssh/connection/session.rb:169:in `loop'
        from /usr/lib/ruby/gems/1.9.1/gems/net-ssh-2.7.0/lib/net/ssh/connection/channel.rb:269:in `wait'
        from /usr/lib/ruby/gems/1.9.1/bundler/gems/knife-solo-1e1675f017e4/lib/knife-solo/ssh_connection.rb:46:in `run_command'
        from /usr/lib/ruby/gems/1.9.1/bundler/gems/knife-solo-1e1675f017e4/lib/knife-solo/ssh_command.rb:233:in `run_command'
        from /usr/lib/ruby/gems/1.9.1/bundler/gems/knife-solo-1e1675f017e4/lib/knife-solo/ssh_command.rb:181:in `windows_node?'
        from /usr/lib/ruby/gems/1.9.1/bundler/gems/knife-solo-1e1675f017e4/lib/chef/knife/solo_cook.rb:167:in `adjust_rsync_path_on_node'
        from /usr/lib/ruby/gems/1.9.1/bundler/gems/knife-solo-1e1675f017e4/lib/chef/knife/solo_cook.rb:253:in `rsync'
        from /usr/lib/ruby/gems/1.9.1/bundler/gems/knife-solo-1e1675f017e4/lib/chef/knife/solo_cook.rb:219:in `upload'
        from /usr/lib/ruby/gems/1.9.1/bundler/gems/knife-solo-1e1675f017e4/lib/chef/knife/solo_cook.rb:235:in `upload_to_provision_path'
        from /usr/lib/ruby/gems/1.9.1/bundler/gems/knife-solo-1e1675f017e4/lib/chef/knife/solo_cook.rb:114:in `block in sync_kitchen'
        from /usr/lib/ruby/gems/1.9.1/bundler/gems/knife-solo-1e1675f017e4/lib/chef/knife/solo_cook.rb:113:in `each'
        from /usr/lib/ruby/gems/1.9.1/bundler/gems/knife-solo-1e1675f017e4/lib/chef/knife/solo_cook.rb:113:in `each_with_index'
        from /usr/lib/ruby/gems/1.9.1/bundler/gems/knife-solo-1e1675f017e4/lib/chef/knife/solo_cook.rb:113:in `sync_kitchen'
        from /usr/lib/ruby/gems/1.9.1/bundler/gems/knife-solo-1e1675f017e4/lib/chef/knife/solo_cook.rb:88:in `block in run'
        from /usr/lib/ruby/gems/1.9.1/bundler/gems/knife-solo-1e1675f017e4/lib/chef/knife/solo_cook.rb:198:in `time'
        from /usr/lib/ruby/gems/1.9.1/bundler/gems/knife-solo-1e1675f017e4/lib/chef/knife/solo_cook.rb:71:in `run'
        from /usr/lib/ruby/gems/1.9.1/gems/chef-11.8.2/lib/chef/knife.rb:485:in `run_with_pretty_exceptions'
        from /usr/lib/ruby/gems/1.9.1/gems/chef-11.8.2/lib/chef/knife.rb:174:in `run'
        from /usr/lib/ruby/gems/1.9.1/gems/chef-11.8.2/lib/chef/application/knife.rb:133:in `run'
        from /usr/lib/ruby/gems/1.9.1/gems/chef-11.8.2/bin/knife:25:in `<top (required)>'
        from /usr/bin/knife:23:in `load'
        from /usr/bin/knife:23:in `<main>'

Looks that rsync --rsh='ssh user@lucid' brakes the SSH connection and it can not create the next session for execution of remote command. For example the following script executed under the cygwin ends up with similar result

command = "ver"
Net::SSH.start('lucid', 'user') do |ssh|
  2.times do
    ssh.open_channel do |channel|
      channel.request_pty
      channel.exec(command) do |ch, success|
        raise "ssh.channel.exec failure" unless success

        channel.on_data do |ch, data|  # stdout
          puts "#{command} stdout: #{data}"
        end

        channel.on_extended_data do |ch, type, data|
          next unless type == 1
          puts "#{command} stderr: #{data}"
        end

        channel.on_request("exit-status") do |ch, data|
          puts"#{command} exit: #{data.read_long}"
        end
      end
    end.wait
    system "rsync -rL -v --rsh='ssh user@lucid' --delete --exclude=revision-deploys --exclude=tmp --exclude=.git --exclude=.hg --exclude=.svn--exclude=.bzr Berksfile :~/chef-solo/cookbooks-1"
  end
end

[matschaffer]

Does that fix work? I don't want to take it out for all cases, but we should be able to skip it by adding a check on windows_client?. Or maybe specically cygwin_client? which isn't added yet, but I was planning to add to address some cygwin/mingw issues.

[keeruline]

It works. However I don't feel comfortable with this fix. I was not able root cause this issue. Can't explain why SSH behaves such way in cygwin. Therefore this fix is just a workaround.

[matschaffer]

Gotcha. Let me know if you find anything out. This may also be more reason to look into doing an SSH-tunnel-based chef zero implementation that's being tossed around.

[datacoda]

It oddly seems to be just the system() command. I've taken @keeruline 's sample script, and used backticks and open3.popen3. Those work without issue. Not sure if this helps or not. on Ruby 1.9.3p484 (i386-cygwin).