Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

error occured 'INFO Read line error: file inactive ' and the registry_file do not update #1645

Closed
zxy12 opened this issue May 16, 2016 · 4 comments
Closed

error occured 'INFO Read line error: file inactive ' and the registry_file do not update #1645

zxy12 opened this issue May 16, 2016 · 4 comments
Labels
bug Filebeat Filebeat

Comments

@zxy12
Copy link

zxy12 commented May 16, 2016 •
edited
Loading

i use the filebeat with publish rolled logs, some times occur error, my filebeat.yml is:

filebeat:
  spool_size: 1024
  idle_timeout: "5s"
  registry_file: ".registry_file"
  prospectors:
    -
      paths:
          - /data/log/lbs/lbs.log.201*
      encoding: "utf-8"
      input_type: log
      ignore_older: 1h
      close_older: 2h
      harvester_buffer_size: 16384
      scan_frequency: "10s"
      backoff: "1s"
      fields:
          log_src: "lbs"
    -
      paths:
          - /data/log/ral/ral-worker.log
      input_type: log
      ignore_older: 1h
      close_older: 2h
      harvester_buffer_size: 16384
      scan_frequency: "10s"
      backoff: "1s"
      fields:
          log_src: "ral"



output:

    logstash:
        hosts:
          - "logstash1:5044"
          - "logstash1:5045"
          - "logstash2:5044"
shipper:

logging:
      to_files: true
      level: info
      files:
        path: "/data/log/filebeat/"
        name: filebeat.log
        rotateeverybytes: 10485760 # = 10MB

For confirmed bugs, please report:

  • Version: 1.2
  • Operating System: CentOS release 4.3
  • Steps to Reproduce:
    ./filebeat -c PATH-TO-CONFIG

i found this error in the log file

2016-05-11T16:29:31+08:00 INFO GeoIP disabled: No paths were set under output.geoip.paths
2016-05-11T16:29:31+08:00 INFO Max Retries set to: 3
2016-05-11T16:29:31+08:00 INFO Activated logstash as output plugin.
2016-05-11T16:29:31+08:00 INFO Publisher name: nj03-canary-app0366.nj03.baidu.com
2016-05-11T16:29:31+08:00 INFO Flush Interval set to: 1s
2016-05-11T16:29:31+08:00 INFO Max Bulk Size set to: 2048
2016-05-11T16:29:31+08:00 INFO Init Beat: filebeat; Version: 1.2.0
2016-05-11T16:29:31+08:00 INFO filebeat sucessfully setup. Start running.
2016-05-11T16:29:31+08:00 INFO Registry file set to: /data/script/.registry_file
2016-05-11T16:29:31+08:00 INFO Set ignore_older duration to 1h0m0s
2016-05-11T16:29:31+08:00 INFO Set close_older duration to 2h0m0s
2016-05-11T16:29:31+08:00 INFO Set scan_frequency duration to 10s
2016-05-11T16:29:31+08:00 INFO Input type set to: log
2016-05-11T16:29:31+08:00 INFO Set backoff duration to 1s
2016-05-11T16:29:31+08:00 INFO Set max_backoff duration to 10s
2016-05-11T16:29:31+08:00 INFO force_close_file is disabled
2016-05-11T16:29:31+08:00 INFO Set ignore_older duration to 1h0m0s
2016-05-11T16:29:31+08:00 INFO Set close_older duration to 2h0m0s
2016-05-11T16:29:31+08:00 INFO Set scan_frequency duration to 10s
2016-05-11T16:29:31+08:00 INFO Input type set to: log
2016-05-11T16:29:31+08:00 INFO Set backoff duration to 1s
2016-05-11T16:29:31+08:00 INFO Set max_backoff duration to 10s
2016-05-11T16:29:31+08:00 INFO force_close_file is disabled
2016-05-11T16:29:31+08:00 INFO Starting prospector of type: log
2016-05-11T16:29:31+08:00 INFO Starting prospector of type: log
2016-05-11T16:29:31+08:00 INFO Starting spooler: spool_size: 1024; idle_timeout: 5s
2016-05-11T16:29:31+08:00 INFO Harvester started for file: /data/log/ral/ral-worker.log
2016-05-11T16:29:31+08:00 INFO All prospectors initialised with 0 states to persist
2016-05-11T16:29:31+08:00 INFO Starting Registrar
2016-05-11T16:29:31+08:00 INFO Start sending events to output
2016-05-11T16:29:31+08:00 INFO Harvester started for file: /data/log/lbs/lbs.log.2016051115
2016-05-11T16:29:34+08:00 INFO Events sent: 1024
2016-05-11T16:29:34+08:00 INFO Registry file updated. 1 states written.
2016-05-11T16:29:40+08:00 INFO Events sent: 545
2016-05-11T16:29:40+08:00 INFO Registry file updated. 2 states written.
2016-05-11T16:30:29+08:00 INFO Events sent: 2
2016-05-11T16:30:29+08:00 INFO Registry file updated. 2 states written.
2016-05-11T16:30:31+08:00 INFO Harvester started for file: /data/log/lbs/lbs.log.2016051116
2016-05-11T16:30:36+08:00 INFO Events sent: 1
2016-05-11T16:30:36+08:00 INFO Registry file updated. 3 states written.
2016-05-11T16:33:41+08:00 INFO Events sent: 1
2016-05-11T16:33:41+08:00 INFO Registry file updated. 3 states written.
2016-05-11T16:33:46+08:00 INFO Events sent: 2
2016-05-11T16:33:46+08:00 INFO Registry file updated. 3 states written.
2016-05-11T18:29:36+08:00 INFO Read line error: file inactive
2016-05-11T18:33:41+08:00 INFO Read line error: file inactive
2016-05-11T18:33:46+08:00 INFO Read line error: file inactive
2016-05-13T11:54:16+08:00 INFO Harvester started for file: /data/log/lbs/lbs.log.2016051311
2016-05-13T11:54:19+08:00 INFO Events sent: 1
2016-05-13T11:54:19+08:00 INFO Registry file updated. 4 states written.
2016-05-13T11:56:34+08:00 INFO Events sent: 1
2016-05-13T11:56:34+08:00 INFO Registry file updated. 4 states written.
2016-05-13T11:56:34+08:00 INFO Harvester started for file: /data/log/ral/ral-worker.log
2016-05-13T11:56:41+08:00 INFO Events sent: 2
2016-05-13T11:56:41+08:00 INFO Registry file updated. 4 states written.
2016-05-13T11:56:54+08:00 INFO Events sent: 2
2016-05-13T11:56:54+08:00 INFO Registry file updated. 4 states written.
2016-05-13T11:57:01+08:00 INFO Events sent: 1
2016-05-13T11:57:01+08:00 INFO Registry file updated. 4 states written.
2016-05-13T11:57:09+08:00 INFO Events sent: 6
2016-05-13T11:57:09+08:00 INFO Registry file updated. 4 states written.
2016-05-13T11:57:21+08:00 INFO Events sent: 1
2016-05-13T11:57:21+08:00 INFO Registry file updated. 4 states written.
2016-05-13T11:57:26+08:00 INFO Events sent: 2
2016-05-13T11:57:26+08:00 INFO Registry file updated. 4 states written.
2016-05-13T11:57:46+08:00 INFO Events sent: 1
2016-05-13T11:57:46+08:00 INFO Registry file updated. 4 states written.
2016-05-13T11:57:54+08:00 INFO Events sent: 3
2016-05-13T11:57:54+08:00 INFO Registry file updated. 4 states written.
2016-05-13T11:58:06+08:00 INFO Events sent: 1
2016-05-13T11:58:06+08:00 INFO Registry file updated. 4 states written.
2016-05-13T11:58:19+08:00 INFO Events sent: 1
2016-05-13T11:58:19+08:00 INFO Registry file updated. 4 states written.
2016-05-13T11:58:26+08:00 INFO Events sent: 5
2016-05-13T11:58:26+08:00 INFO Registry file updated. 4 states written.
2016-05-13T12:07:56+08:00 INFO Harvester started for file: /data/log/lbs/lbs.log.2016051312
2016-05-13T12:07:59+08:00 INFO Events sent: 1
2016-05-13T12:07:59+08:00 INFO Registry file updated. 5 states written.
2016-05-13T12:08:24+08:00 INFO Events sent: 1
2016-05-13T12:08:24+08:00 INFO Registry file updated. 5 states written.
2016-05-13T12:11:11+08:00 INFO Events sent: 1
2016-05-13T12:11:11+08:00 INFO Registry file updated. 5 states written.
2016-05-13T12:11:26+08:00 INFO Events sent: 1
2016-05-13T12:11:26+08:00 INFO Registry file updated. 5 states written.
2016-05-13T12:11:56+08:00 INFO Events sent: 1
2016-05-13T12:11:56+08:00 INFO Registry file updated. 5 states written.
2016-05-13T12:15:34+08:00 INFO Events sent: 3
2016-05-13T12:15:34+08:00 INFO Registry file updated. 5 states written.
2016-05-13T13:21:26+08:00 INFO Harvester started for file: /data/log/lbs/lbs.log.2016051313
2016-05-13T13:21:26+08:00 INFO Events sent: 3
2016-05-13T13:21:26+08:00 INFO Registry file updated. 6 states written.
2016-05-13T13:22:11+08:00 INFO Events sent: 3
2016-05-13T13:22:11+08:00 INFO Registry file updated. 6 states written.
2016-05-13T13:29:06+08:00 INFO Events sent: 3
2016-05-13T13:29:06+08:00 INFO Registry file updated. 6 states written.
2016-05-13T13:30:04+08:00 INFO Events sent: 1
2016-05-13T13:30:04+08:00 INFO Registry file updated. 6 states written.
2016-05-13T13:30:11+08:00 INFO Events sent: 2
2016-05-13T13:30:11+08:00 INFO Registry file updated. 6 states written.
2016-05-13T13:42:29+08:00 INFO Events sent: 6
2016-05-13T13:42:29+08:00 INFO Registry file updated. 6 states written.
2016-05-13T13:58:31+08:00 INFO Read line error: file inactive
2016-05-13T14:15:36+08:00 INFO Read line error: file inactive
2016-05-13T14:55:17+08:00 INFO Harvester started for file: /data/log/lbs/lbs.log.2016051314
2016-05-13T14:55:19+08:00 INFO Events sent: 1
2016-05-13T14:55:19+08:00 INFO Registry file updated. 7 states written.
2016-05-13T14:55:26+08:00 INFO Events sent: 2
2016-05-13T14:55:26+08:00 INFO Registry file updated. 7 states written.
2016-05-13T14:55:49+08:00 INFO Events sent: 2
2016-05-13T14:55:49+08:00 INFO Registry file updated. 7 states written.
2016-05-13T14:55:56+08:00 INFO Events sent: 1
2016-05-13T14:55:56+08:00 INFO Registry file updated. 7 states written.
2016-05-13T14:56:29+08:00 INFO Events sent: 1
2016-05-13T14:56:29+08:00 INFO Registry file updated. 7 states written.
2016-05-13T14:56:36+08:00 INFO Events sent: 2
2016-05-13T14:56:36+08:00 INFO Registry file updated. 7 states written.
2016-05-13T14:57:16+08:00 INFO Events sent: 3
2016-05-13T14:57:16+08:00 INFO Registry file updated. 7 states written.
2016-05-13T14:58:34+08:00 INFO Events sent: 2
2016-05-13T14:58:34+08:00 INFO Registry file updated. 7 states written.
2016-05-13T14:58:39+08:00 INFO Events sent: 1
2016-05-13T14:58:39+08:00 INFO Registry file updated. 7 states written.
2016-05-13T15:08:56+08:00 INFO Events sent: 2
2016-05-13T15:08:56+08:00 INFO Registry file updated. 7 states written.
2016-05-13T15:08:57+08:00 INFO Harvester started for file: /data/log/lbs/lbs.log.2016051315
2016-05-13T15:09:04+08:00 INFO Events sent: 1
2016-05-13T15:09:04+08:00 INFO Registry file updated. 8 states written.
2016-05-13T15:12:14+08:00 INFO Events sent: 1
2016-05-13T15:12:14+08:00 INFO Registry file updated. 8 states written.
2016-05-13T15:12:26+08:00 INFO Events sent: 2
2016-05-13T15:12:26+08:00 INFO Registry file updated. 8 states written.
2016-05-13T15:24:26+08:00 INFO Events sent: 2
2016-05-13T15:24:26+08:00 INFO Registry file updated. 8 states written.
2016-05-13T15:24:31+08:00 INFO Events sent: 1
2016-05-13T15:24:31+08:00 INFO Registry file updated. 8 states written.
2016-05-13T15:24:44+08:00 INFO Events sent: 1
2016-05-13T15:24:44+08:00 INFO Registry file updated. 8 states written.
2016-05-13T15:24:51+08:00 INFO Events sent: 2
2016-05-13T15:24:51+08:00 INFO Registry file updated. 8 states written.
2016-05-13T15:26:59+08:00 INFO Events sent: 9
2016-05-13T15:26:59+08:00 INFO Registry file updated. 8 states written.
2016-05-13T15:27:19+08:00 INFO Events sent: 3
2016-05-13T15:27:19+08:00 INFO Registry file updated. 8 states written.
2016-05-13T15:28:09+08:00 INFO Events sent: 6
2016-05-13T15:28:09+08:00 INFO Registry file updated. 8 states written.
2016-05-13T15:31:04+08:00 INFO Events sent: 3
2016-05-13T15:31:04+08:00 INFO Registry file updated. 8 states written.
2016-05-13T15:31:44+08:00 INFO Events sent: 3
2016-05-13T15:31:44+08:00 INFO Registry file updated. 8 states written.
2016-05-13T15:32:54+08:00 INFO Events sent: 3
2016-05-13T15:32:54+08:00 INFO Registry file updated. 8 states written.
2016-05-13T15:33:06+08:00 INFO Events sent: 3
2016-05-13T15:33:06+08:00 INFO Registry file updated. 8 states written.
2016-05-13T15:38:56+08:00 INFO Events sent: 6
2016-05-13T15:38:56+08:00 INFO Registry file updated. 8 states written.
2016-05-13T15:39:04+08:00 INFO Events sent: 3
2016-05-13T15:39:04+08:00 INFO Registry file updated. 8 states written.
2016-05-13T15:40:29+08:00 INFO Events sent: 3
2016-05-13T15:40:29+08:00 INFO Registry file updated. 8 states written.
2016-05-13T15:42:31+08:00 INFO Read line error: file inactive
2016-05-13T15:45:49+08:00 INFO Events sent: 3
2016-05-13T15:45:49+08:00 INFO Registry file updated. 8 states written.
2016-05-13T15:47:14+08:00 INFO Events sent: 3
2016-05-13T15:47:14+08:00 INFO Registry file updated. 8 states written.
2016-05-13T15:50:11+08:00 INFO Events sent: 3
2016-05-13T15:50:11+08:00 INFO Registry file updated. 8 states written.
2016-05-13T16:58:42+08:00 INFO Read line error: file inactive
2016-05-13T17:50:13+08:00 INFO Read line error: file inactive
2016-05-13T17:50:14+08:00 INFO Read line error: file inactive

and the .registry_file seem not update now,but the logs are still publish successly.

cat .registry_file
{"/data/log/lbs/lbs.log.2016051115":{"source":"/data/log/lbs/lbs.log.2016051115","offset":57475,"FileStateOS":{"inode":2883970,"device":2054}},"/data/log/lbs/lbs.log.2016051116":{"source":"/data/log/lbs/lbs.log.2016051116","offset":28757,"FileStateOS":{"inode":2883976,"device":2054}},"/data/log/lbs/lbs.log.2016051311":{"source":"/data/log/lbs/lbs.log.2016051311","offset":56303,"FileStateOS":{"inode":2883979,"device":2054}},"/data/log/lbs/lbs.log.2016051312":{"source":"/data/log/lbs/lbs.log.2016051312","offset":32700,"FileStateOS":{"inode":2883981,"device":2054}},"/data/log/lbs/lbs.log.2016051313":{"source":"/data/log/lbs/lbs.log.2016051313","offset":26946,"FileStateOS":{"inode":2883983,"device":2054}},"/data/log/lbs/lbs.log.2016051314":{"source":"/data/log/lbs/lbs.log.2016051314","offset":22397,"FileStateOS":{"inode":2883985,"device":2054}},"/data/log/lbs/lbs.log.2016051315":{"source":"/data/log/lbs/lbs.log.2016051315","offset":95054,"FileStateOS":{"inode":2883987,"device":2054}},"/data/log/ral/ral-worker.log":{"source":"/data/log/ral/ral-worker.log","offset":899457,"FileStateOS":{"inode":2626942,"device":2054}}}

if i restart the filebeat, i think it will not remeber the last offset that it has published, so how can i do?
@ruflin
Copy link
Contributor

ruflin commented May 17, 2016

This sounds like issue #1528. This was fixed in #1532 and will be shipped with version 1.2.3. Unfortunately you are right and it will reship the rotated files on restart :-(

@ruflin ruflin added Filebeat Filebeat bug labels May 17, 2016
@ruflin
Copy link
Contributor

ruflin commented May 17, 2016

@zxy12 I will close this one, as I think it is already fixed. Please reopen if this is not the case. Let me know if you would like a SNAPSHOT build of 1.2.3

@ruflin ruflin closed this as completed May 17, 2016
@zxy12
Copy link
Author

zxy12 commented May 19, 2016

@ruflin ok, thanks a lot.

@draenor
Copy link

draenor commented Jul 6, 2016 •
edited
Loading

EDIT I created a new issue instead since it's similar, but not the same as this: #1974

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Filebeat Filebeat
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@ruflin @draenor @zxy12