Prepare for td-agent 4 #24

repeatedly · 2020-04-08T07:02:05Z

Remove unused td-agent 2/3 code
Update bundled ruby and libraries

I build td-agent 4 for centos8 with this patch but error happens during installation.

$ sudo yum localinstall td-agent-4.0.0-1.el8.x86_64.rpm 
[sudo] password for repeatedly: 
Last metadata expiration check: 0:00:03 ago on Wed Apr  8 15:53:12 2020.
Error: 
 Problem: conflicting requests
  - nothing provides /usr/local/bin/ruby needed by td-agent-4.0.0-1.el8.x86_64
  - nothing provides libjemalloc.so.2()(64bit) needed by td-agent-4.0.0-1.el8.x86_64
  - nothing provides libzstd.so.1()(64bit) needed by td-agent-4.0.0-1.el8.x86_64
(try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)

Does anyone have workaround? td-agent seems to not need ruby.

ashie · 2020-04-08T08:14:13Z

nothing provides /usr/local/bin/ruby needed by td-agent-4.0.0-1.el8.x86_64

I've found the following unexpected shebang in the built package:

$ grep -rn "/usr/local"
...
opt/td-agent/lib/ruby/gems/2.7.0/gems/racc-1.4.16/bin/y2racc:1:#!/usr/local/bin/ruby
opt/td-agent/lib/ruby/gems/2.7.0/gems/racc-1.4.16/bin/racc2y:1:#!/usr/local/bin/ruby
...

ashie · 2020-04-08T08:38:47Z

I've found the following unexpected shebang in the built package:

opt/td-agent/lib/ruby/gems/2.7.0/gems/racc-1.4.16/bin/y2racc:1:#!/usr/local/bin/ruby
opt/td-agent/lib/ruby/gems/2.7.0/gems/racc-1.4.16/bin/racc2y:1:#!/usr/local/bin/ruby

They don't exist when I build it with Ruby-2.6.5.
Probably prefix setting isn't passed to racc which is bundled in Ruby-2.7.1.
I'll try to fix & report it.

ashie · 2020-04-08T09:16:07Z

nothing provides libjemalloc.so.2()(64bit) needed by td-agent-4.0.0-1.el8.x86_64

nothing provides libzstd.so.1()(64bit) needed by td-agent-4.0.0-1.el8.x86_64

These dependency should be added to our spec file.
Sorry, I already noticed it but forgot to add them.
I'll address it.

ashie · 2020-04-09T02:42:20Z

nothing provides /usr/local/bin/ruby needed by td-agent-4.0.0-1.el8.x86_64

I've confirmed that the following quick hack resolves it:

5e6e24e

ashie · 2020-04-09T09:43:17Z

rawhide's solution:

# These have wrong shebangs. Exclude them for now and let's see what upstream
# thinks about them.
# https://bugs.ruby-lang.org/issues/15982
%exclude %{_bindir}/{racc2y,y2racc}
%exclude %{gem_dir}/gems/racc-%{racc_version}/bin/{racc2y,y2racc}

Related links:

ashie · 2020-04-10T00:29:23Z

https://github.com/clear-code/td-agent-builder/pull/24/checks?check_run_id=569944780

25h25lent-plugin-windows-eventlog-0.5.3.gem:
        The last version of nokogiri (~> 1.10) to support your Ruby & RubyGems w
as 1.10.9. Try installing it with `gem install nokogiri -v 1.10.9` and then runn
ing the current command again
        nokogiri requires Ruby version >= 2.3, < 2.7.dev. The current ruby versi
on is 2.7.0.0.
rake aborted!

@cosmo0920 Do you have any idea to resolve it?

cosmo0920 · 2020-04-10T00:32:36Z

25h25lent-plugin-windows-eventlog-0.5.3.gem:
        The last version of nokogiri (~> 1.10) to support your Ruby & RubyGems w
as 1.10.9. Try installing it with `gem install nokogiri -v 1.10.9` and then runn
ing the current command again
        nokogiri requires Ruby version >= 2.3, < 2.7.dev. The current ruby versi
on is 2.7.0.0.
rake aborted!

@cosmo0920 Do you have any idea to resolve it?

How about using gem install nokogiri --platform ruby to avoid using fat gem?

cosmo0920 · 2020-04-10T00:37:42Z

Hmm, nokogiri 0.11.0.rc2 includes Ruby 2.7 binary....

They take along with unexpected dependency since they contains wrong shebang. In addition, they are already removed at upstream. See also: * https://bugs.ruby-lang.org/issues/15982 * ruby/racc#123 * https://src.fedoraproject.org/rpms/ruby/c/baf046a6a4d17fa309c9d20fa3db949f6c24aacf?branch=master * #24 Signed-off-by: Takuro Ashie <[email protected]>

They take along with unexpected dependency since they contain wrong shebang. In addition, they are already removed at upstream. See also: * https://bugs.ruby-lang.org/issues/15982 * ruby/racc#123 * https://src.fedoraproject.org/rpms/ruby/c/baf046a6a4d17fa309c9d20fa3db949f6c24aacf?branch=master * #24 Signed-off-by: Takuro Ashie <[email protected]>

ashie · 2020-04-10T01:52:03Z

rawhide's solution:

I also decide to remove them: #28

ashie · 2020-04-10T06:23:46Z

#28 has been merged.
@repeatedly please retry it on top of master.
In addition, please enable EPEL to install jemalloc & libzstd.

$ sudo dnf install epel-release -y
$ sudo dnf install /path/to/td-agent-4.0.0-1.el8.x86_64.rpm -y

ashie · 2020-04-10T07:27:03Z

How about using gem install nokogiri --platform ruby to avoid using fat gem?

When I add nokogoiri to plugin_gems.rb, it takes toooo long time to build (over 30min?).
I found that antimalware protection process of Windows eats 50% of CPU usage.
We need to turn-off it while building nokogiri...

Instead of building nokogiri we'll release a new version fluent-plugin-windows-eventlog which relaxes the required version of nokogiri.

cosmo0920 · 2020-04-10T07:52:33Z

Instead of building nokogiri we'll release a new version fluent-plugin-windows-eventlog which relaxes the required version of nokogiri.

I'd released the new versions of fluent-plugin-windows-eventlog and fluent-plugin-parser-winevt_xml. They will be able to use nokogiri 1.11.0.

repeatedly · 2020-04-13T07:32:25Z

@ashie How about including jemalloc in td-agent?
jemalloc changes internal memory management, so using same version is better for trouble-shooting.

BTW, why zstd is needed for td-agent?

ashie · 2020-04-13T08:55:22Z

@ashie How about including jemalloc in td-agent?
jemalloc changes internal memory management, so using same version is better for trouble-shooting.

I'll try it: #29

BTW, why zstd is needed for td-agent?

librdkafka.so depends on it:

$ find | grep "\.so" | xargs ldd
...
./opt/td-agent/lib/ruby/gems/2.6.0/gems/rdkafka-0.7.0/ext/librdkafka.so:
        linux-vdso.so.1 (0x00007ffc00020000)
        liblz4.so.1 => /usr/lib/x86_64-linux-gnu/liblz4.so.1 (0x00007efbff380000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007efbff231000)
        libzstd.so.1 => /usr/lib/x86_64-linux-gnu/libzstd.so.1 (0x00007efbff18a000)
        libssl.so.1.1 => /usr/lib/x86_64-linux-gnu/libssl.so.1.1 (0x00007efbff0f8000)
        libcrypto.so.1.1 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 (0x00007efbfee23000)
...

Since this compile switch is optional, this dependency can be removed easily.
But I'm not sure it's needed for td-agent or not. The first commit for it is done by @cosmo0920
Should we enable it? > @cosmo0920

cosmo0920 · 2020-04-13T09:01:47Z

I'd like to enable zstd.

zstd is used for compression on Kafka.
Yes, we can remove it.
But, users cannot use zstd compression on Kafka events.
librdkafka checks and uses zstd:
https://github.com/edenhill/librdkafka/blob/53bdb89b33f9823f38cb18d4f81a0e89f1e8473a/CMakeLists.txt#L60

If we remove it, librdkafka will not enable zstd compression.

ashie · 2020-04-14T02:19:59Z

I'd like to enable zstd.

I've found that it can be linked statically:

https://github.com/edenhill/librdkafka#requirements

STATIC_LIB_libzstd=/path/to/libzstd.a ./configure

(--enable-static doesn't seem required as far as I tried)

ashie · 2020-04-14T09:21:47Z

Hmm, CentOS doesn't seem provide static library of zstd.

ashie · 2020-04-15T01:24:46Z

librdkafka.so in official td-agent-3.7.0 doesn't support zstd:

./embedded/lib/ruby/gems/2.4.0/gems/rdkafka-0.7.0/ext/librdkafka.so:
        linux-vdso.so.1 =>  (0x00007ffbffffe000)
        libm.so.6 => /lib64/libm.so.6 (0x00007efbf77ae000)
        libsasl2.so.3 => /lib64/libsasl2.so.3 (0x00007efbf7591000)
        libssl.so.1.0.0 => /opt/td-agent/embedded/lib/libssl.so.1.0.0 (0x00007efbf7325000)
        libcrypto.so.1.0.0 => /opt/td-agent/embedded/lib/libcrypto.so.1.0.0 (0x00007efbf6eec000)
        libz.so.1 => /opt/td-agent/embedded/lib/libz.so.1 (0x00007efbf6cd5000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007efbf6ad1000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007efbf68b5000)
        librt.so.1 => /lib64/librt.so.1 (0x00007efbf66ad000)
        libc.so.6 => /lib64/libc.so.6 (0x00007efbf62df000)
        /lib64/ld-linux-x86-64.so.2 (0x00007efbf7ddd000)
        libresolv.so.2 => /lib64/libresolv.so.2 (0x00007efbf60c6000)
        libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007efbf5e8f000)
        libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007efbf5c42000)
        libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007efbf5959000)
        libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007efbf5726000)
        libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007efbf5522000)
        libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007efbf5312000)
        libfreebl3.so => /lib64/libfreebl3.so (0x00007efbf510f000)
        libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007efbf4f0b000)
        libselinux.so.1 => /lib64/libselinux.so.1 (0x00007efbf4ce4000)
        libpcre.so.1 => /lib64/libpcre.so.1 (0x00007efbf4a82000)

$ strings ./embedded/lib/ruby/gems/2.4.0/gems/rdkafka-0.7.0/ext/librdkafka.so | grep rd_kafka_zstd

cosmo0920 · 2020-04-15T09:40:59Z

fluent-plugin-windows-eventlog v0.6.0 removes nokogiri dependency by default.
Currently, nokogiri 1.11.0, which includes Ruby 2.7 binary, is not released.
We should bundle nokogiri 1.11.0 and fluent-plugin-parser-winevt_xml later.

repeatedly · 2020-04-15T15:33:22Z

If current td-agent doesn't support zstd, td-agent 4 also doesn't need zstd support.
Remove epel dependency is more important.

ashie · 2020-04-16T00:33:12Z

If current td-agent doesn't support zstd, td-agent 4 also doesn't need zstd support.
Remove epel dependency is more important.

I think so too. I'll remove the dependency.

ashie · 2020-05-29T03:42:10Z

@ashie I tested msi build on Mac. msi:build doesn't support Mac environment? Need to run command on windows?

Yes, currently it requires Windows host and Windows container.
I'm now considering to support any other host OS: #70

repeatedly · 2020-05-29T04:48:12Z

Ah, I see. I understood.

repeatedly · 2020-06-02T08:02:24Z

nokogiri now has rc2 for ruby 2.7.

https://rubygems.org/gems/nokogiri

Can we go td-agent 4 with this version?
or
Does this rc2 have the problem?

ashie · 2020-06-02T08:11:13Z

Can we go td-agent 4 with this version?

@cosmo0920 What do you think?
I think we can go with it.

cosmo0920 · 2020-06-02T08:14:22Z

Can we go td-agent 4 with this version?

@cosmo0920 What do you think?
I think we can go with it.

We should handle it with --pre option in gem command. Adding --pre code is no objection for me.

repeatedly · 2020-06-02T11:43:45Z

How about putting nokogiri rc2 into core_gems or plugin_gems?

ashie · 2020-06-03T03:17:53Z

How about putting nokogiri rc2 into core_gems or plugin_gems?

Adding the following line to plugin_gems.rb should resolve this issue:

download 'nokogiri', '1.11.0.rc2-x64-mingw32'

But I found a bug that platform field is ignored.
I've fixed it: #81
Please try with the current master.

ashie · 2020-06-03T06:13:29Z

FYI: We are now checking regressions in package scripts of RPM: #82

repeatedly · 2020-06-09T20:24:45Z

@ashie I re-checked this branch with centos7/8 and it depends on libruby. Could you check it?

$ rpm -qRp td-agent-4.0.0.rc2-1.el8.x86_64.rpm | grep libruby
libruby.so.2.7()(64bit)

ashie · 2020-06-10T00:09:02Z

I'll check it.

ashie · 2020-06-11T02:57:54Z

Sorry for that, #79 removes libruby.so.2.7.1 unexpectedly.
I've fixed it via #97

repeatedly · 2020-06-11T11:34:03Z

Thanks! Just confirmed rpm doesn't depend on ruby for installation.

Does anyone have a concern to merge this PR?

cosmo0920 · 2020-06-12T01:08:35Z

I have no concern to merge this PR. 👍

ashie · 2020-06-12T03:27:54Z

Does anyone have a concern to merge this PR?

Although some issues are still remained, I want to merge this ASAP.
We'll resolve remaining issues after we merge it.

The issues that I'm concerned about are:

Windows build is still failing event if both #24 (comment) and #95 are merged.

Fetching fluent-plugin-parser-winevt_xml-0.2.2.gem
Successfully installed mini_portile2-2.4.0
ERROR:  Error installing C:/td-agent-4.0.0.rc1/td-agent/downloads/plugin_gems/26-fluent-plugin-windows-eventlog-0.5.4.gem:
        The last version of nokogiri (>= 1.10, < 1.12) to support your Ruby & RubyGems was 1.10.9. Try installing it with `gem install nokogiri -v 1.10.9` and then running the current command again
        nokogiri requires Ruby version >= 2.3, < 2.7.dev. The current ruby version is 2.7.1.83.
rake aborted!

We should handle it with --pre option in gem command. Adding --pre code is no objection for me.

After all this seems to be necessary.

In addition, #98 seems critical, we should resolve it before releasing td-agent 4 (for windows).

ashie · 2020-06-14T23:55:22Z

We'll resolve remaining issues after merging this branch.

ashie · 2020-06-15T00:48:37Z

I've merged this pull request.

On this occasion, I want to transfer the ownership of this project to @treasure-data.
But nobody can't do it directly since permission is required on both organizations.
So, I'm attempting to transfer it via @fluent-plugins-nursery.

cosmo0920 · 2020-06-15T06:35:15Z

I tried to build with nokogiri 1.11.0.rc2 on Windows installer but I got failure:

diff --git a/td-agent/Rakefile b/td-agent/Rakefile
index 24f56f5..2cf5055 100755
--- a/td-agent/Rakefile
+++ b/td-agent/Rakefile
@@ -631,10 +631,18 @@ class BuildTask
     ENV["MAKEFLAGS"] = "-j#{Etc.nprocessors}"
     ENV["GEM_HOME"]  = gem_staging_dir

+    basename = File.basename(gem_path)
+
+    gem_option = if basename.include?("rc")
+                   [gem_path, "--pre"]
+                 else
+                   [gem_path]
+                 end
+
     sh(gem_command, "install",
        "--no-document",
        "--bindir", staging_bindir,
-       gem_path)
+       *gem_option)

     ENV["GEM_HOME"]  = gem_home
     ENV["MAKEFLAGS"] = makeflags

C:/opt/td-agent/bin/gem install --no-document --bindir C:/opt/td-agent/bin C:/td-agent-4.0.0.rc1/td-agent/downloads/plug
in_gems/26-fluent-plugin-windows-eventlog-0.5.4.gem
Fetching mini_portile2-2.4.0.gem
Fetching nokogiri-1.10.9-x64-mingw32.gem
Fetching fluent-plugin-parser-winevt_xml-0.2.2.gem
Successfully installed mini_portile2-2.4.0
ERROR:  Error installing C:/td-agent-4.0.0.rc1/td-agent/downloads/plugin_gems/26-fluent-plugin-windows-eventlog-0.5.4.ge
m:
        The last version of nokogiri (>= 1.10, < 1.12) to support your Ruby & RubyGems was 1.10.9. Try installing it wit
h `gem install nokogiri -v 1.10.9` and then running the current command again
        nokogiri requires Ruby version >= 2.3, < 2.7.dev. The current ruby version is 2.7.1.83.
rake aborted!
Command failed with status (1): [C:/opt/td-agent/bin/gem install --no-docum...]
C:/td-agent-4.0.0.rc1/td-agent/Rakefile:642:in `gem_install'
C:/td-agent-4.0.0.rc1/td-agent/Rakefile:653:in `block in install_gems'
C:/td-agent-4.0.0.rc1/td-agent/Rakefile:652:in `each'
C:/td-agent-4.0.0.rc1/td-agent/Rakefile:652:in `install_gems'
C:/td-agent-4.0.0.rc1/td-agent/Rakefile:272:in `block (2 levels) in define'
Tasks: TOP => msi:selfbuild => build:all => build:licenses => build:gems => build:plugin_gems
(See full trace by running task with --trace)
rake aborted!

We have to wait nokogiri 1.11.0 release....

ashie self-assigned this Apr 8, 2020

ashie mentioned this pull request Apr 10, 2020

Add patches to remove old executable of racc in Ruby 2.7 #28

Merged

ashie mentioned this pull request Apr 13, 2020

Bundle jemalloc #29

Closed

repeatedly force-pushed the td-agent-4-pre branch from 36eec19 to 142d40e Compare April 15, 2020 08:17

cosmo0920 mentioned this pull request Apr 16, 2020

yum: Remove EPEL dependecies #33

Merged

repeatedly force-pushed the td-agent-4-pre branch from 142d40e to 1db72e5 Compare May 1, 2020 00:04

repeatedly mentioned this pull request May 9, 2020

td-agent 4 treasure-data/omnibus-td-agent#209

Closed

repeatedly added 10 commits June 9, 2020 19:45

update GemParser for td-agent 4

e687c4e

update dependent gems

50e257b

update fluentd/ruby version for td-agent 4

bb084d7

fluentd v1.10.2

1e5cba5

fluentd: use latest commit

9e744e4

Update core version, fluentd, jemalloc and rubyinstaller

d2c96b8

td-agent 4.0.0.rc1

e89d6d7

Fix ruby versions to follow master change

1f6861c

fluentd v1.11.0

1b6a7ac

Add nokogiri to plugin_gems.rb

39401a7

repeatedly force-pushed the td-agent-4-pre branch from f543ab3 to 39401a7 Compare June 9, 2020 18:46

ashie merged commit daf9c82 into fluent:master Jun 14, 2020

kenhys mentioned this pull request Jun 23, 2020

MSI: Failed to execute rake msi:build #115

Closed

Prepare for td-agent 4 #24

Prepare for td-agent 4 #24

Conversation

repeatedly commented Apr 8, 2020 • edited Loading

ashie commented Apr 8, 2020 • edited Loading

ashie commented Apr 8, 2020

ashie commented Apr 8, 2020

ashie commented Apr 9, 2020 • edited Loading

ashie commented Apr 9, 2020

ashie commented Apr 10, 2020

cosmo0920 commented Apr 10, 2020

cosmo0920 commented Apr 10, 2020

ashie commented Apr 10, 2020

ashie commented Apr 10, 2020

ashie commented Apr 10, 2020

cosmo0920 commented Apr 10, 2020

repeatedly commented Apr 13, 2020

ashie commented Apr 13, 2020 • edited Loading

cosmo0920 commented Apr 13, 2020 • edited Loading

ashie commented Apr 14, 2020

ashie commented Apr 14, 2020 • edited Loading

ashie commented Apr 15, 2020 • edited Loading

cosmo0920 commented Apr 15, 2020

repeatedly commented Apr 15, 2020

ashie commented Apr 16, 2020

ashie commented May 29, 2020 • edited Loading

repeatedly commented May 29, 2020

repeatedly commented Jun 2, 2020

ashie commented Jun 2, 2020

cosmo0920 commented Jun 2, 2020 • edited Loading

repeatedly commented Jun 2, 2020

ashie commented Jun 3, 2020

ashie commented Jun 3, 2020

repeatedly commented Jun 9, 2020

ashie commented Jun 10, 2020

ashie commented Jun 11, 2020

repeatedly commented Jun 11, 2020

cosmo0920 commented Jun 12, 2020 • edited Loading

ashie commented Jun 12, 2020 • edited Loading

ashie commented Jun 14, 2020

ashie commented Jun 15, 2020 • edited Loading

cosmo0920 commented Jun 15, 2020

repeatedly commented Apr 8, 2020 •

edited

Loading

ashie commented Apr 8, 2020 •

edited

Loading

ashie commented Apr 9, 2020 •

edited

Loading

ashie commented Apr 13, 2020 •

edited

Loading

cosmo0920 commented Apr 13, 2020 •

edited

Loading

ashie commented Apr 14, 2020 •

edited

Loading

ashie commented Apr 15, 2020 •

edited

Loading

ashie commented May 29, 2020 •

edited

Loading

cosmo0920 commented Jun 2, 2020 •

edited

Loading

cosmo0920 commented Jun 12, 2020 •

edited

Loading

ashie commented Jun 12, 2020 •

edited

Loading

ashie commented Jun 15, 2020 •

edited

Loading