Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support CRI log format #358

Closed
steven-sheehy opened this issue Feb 26, 2019 · 2 comments
Closed

Support CRI log format #358

steven-sheehy opened this issue Feb 26, 2019 · 2 comments

Comments

@steven-sheehy
Copy link
Contributor

steven-sheehy commented Feb 26, 2019

Is your feature request related to a problem? Please describe.
Containerd does not support JSON format, so after setting promtail's entry_parser: raw to make it work, I noticed that there were duplicate timestamps in the loki output. Containerd writes its logs in the CRI format <RFC3339Nano> <stream> <flags> <app output>\n. Other CRI runtimes like CRI-O may also support this same format, but I am not sure. Example log:

2019-02-23T05:30:45.598142519Z stdout F INFO 2019-02-23 05:30:45,596 [main] c.f.e.w.i.InventoryApplication: No active profile set

When I view the logs in Grafana, containerd's timestamp and other metadata isn't removed and the loki timestamp differs slightly from the containerd timestamp.

Describe the solution you'd like
promtail should be able to retrieve logs from containerd with the correct timestamp and remove all other data except the application stdout.

Describe alternatives you've considered
Instead of a containerd-specific solution, implement a generic regex-basedEntryParser similar to Fluent Bit's regex parser.

Or, instead of reading log files integrate with containerd directly via their API. This is the approach containerd's dev recommended via slack.

Additional context
In Grafana, if I enable all options I end up with 4 date/times being printed on screen, making it unuseable:

  1. Grafana timestamp
  2. Grafana local time
  3. Containerd timestamp
  4. Application timestamp

screen shot 2019-02-26 at 10 46 47

This issue deals with only item 3 above.

Containerd is the underlying runtime of Docker 18.09+, the runtime used in IBM Cloud, Pivotal Cloud Foundry and soon to be GKE and Azure so it's important that it is supported.

@steven-sheehy
Copy link
Contributor Author

I've confirmed with both containerd and CRI-O developers that they implement the CRI log format and that the format is stable. Updating title to reflect that.

@steven-sheehy steven-sheehy changed the title Support containerd log format Support CRI log format Mar 4, 2019
@NickLarsenNZ
Copy link

I've confirmed with both containerd and CRI-O developers that they implement the CRI log format and that the format is stable. Updating title to reflect that.

For anyone stumbling on this issue, the archived link is: https://github.com/kubernetes/design-proposals-archive/blob/main/node/kubelet-cri-logging.md

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants