in_tail: readpartial maxlen is too small

[methane]

We use fluentd to transfer line based large log file; about 4GiB / hour and 1495 byte / line.

$ wc apache2.acc.21.log
   2949500  242309651 4410569207 apache2.acc.21.log

CPU usage of fluentd is very high. We tried sigdump and it shows this line every time:

fluentd/lib/fluent/plugin/in_tail.rb

Line 735 in 31cd8ca

@fifo << io.readpartial(2048, @iobuf)

I feel maxlen=2048 is too small, at least on our case. It reads only one or two lines on each call.

FYI, tail command in coreutils uses BUFSIZ in stdio.h, and BUFSIZ in Debian and Ubuntu is 8192.

In case of ruby, overhead of calling io.readpartial will be higher than read in C. Would you change the maxlen to at least 8192?

---

• fluentd or td-agent version: 1.3.3
• Environment information:
  • Operating system: cat /etc/os-release: "Debian GNU/Linux 9 (stretch)"
  • Kernel version: uname -r: 4.9.0-5-amd64
• Your configuration

<source>
  @type tail
  path /.../web/%Y-%m-%d/apache2.acc.%H.log
  pos_file /.../td-agent/web.acc.pos
  read_from_head
  tag web.acc
  format none
  enable_stat_watcher false
</source>

[repeatedly]

@nurse Do you see any concern for increasing this value?

[nurse]

@repeatedly Using 8192 sounds reasonable.

[repeatedly]

@methane Could you send a pull request?