Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace pipeline stage in docker driver #2274

Closed
andoks opened this issue Jun 29, 2020 · 1 comment
Closed

Replace pipeline stage in docker driver #2274

andoks opened this issue Jun 29, 2020 · 1 comment

Comments

@andoks
Copy link

andoks commented Jun 29, 2020

Is your feature request related to a problem? Please describe.
I am using the docker driver to consume postgresql logs, and I would like to rewrite postgresql's levels to more standard ones using the replace stage, but it seems like the replace stage does not work when configured as a pipeline stage with the loki driver.

Describe the solution you'd like
The replace pipeline stage in the docker driver too

Describe alternatives you've considered
None

Additional context
Loki driver info (docker plugin inspect loki)

[
    {
        "Config": {
            "Args": {
                "Description": "",
                "Name": "",
                "Settable": null,
                "Value": null
            },
            "Description": "Loki Logging Driver",
            "DockerVersion": "17.09.0-ce",
            "Documentation": "https://github.com/grafana/loki",
            "Entrypoint": [
                "/bin/docker-driver"
            ],
            "Env": [
                {
                    "Description": "Set log level to output for plugin logs",
                    "Name": "LOG_LEVEL",
                    "Settable": [
                        "value"
                    ],
                    "Value": "info"
                }
            ],
            "Interface": {
                "Socket": "loki.sock",
                "Types": [
                    "docker.logdriver/1.0"
                ]
            },
            "IpcHost": false,
            "Linux": {
                "AllowAllDevices": false,
                "Capabilities": null,
                "Devices": null
            },
            "Mounts": [
                {
                    "Description": "Optional external pipeline files",
                    "Destination": "/data",
                    "Name": "data",
                    "Options": [
                        "bind",
                        "ro"
                    ],
                    "Settable": [
                        "source",
                        "destination"
                    ],
                    "Source": "",
                    "Type": "none"
                }
            ],
            "Network": {
                "Type": "host"
            },
            "PidHost": false,
            "PropagatedMount": "",
            "User": {},
            "WorkDir": "",
            "rootfs": {
                "diff_ids": [
                    "sha256:b5a05d3a388293d9280239ff690aa8ff0b77586b690bc67ba88bef7a9994e098"
                ],
                "type": "layers"
            }
        },
        "Enabled": true,
        "Id": "1bbb0f13f71bcfad75af8dad94f506ae78b99fd86832f7d3d268f800f42c9267",
        "Name": "loki:latest",
        "PluginReference": "docker.io/grafana/loki-docker-driver:master-afd3daf",
        "Settings": {
            "Args": [],
            "Devices": [],
            "Env": [
                "LOG_LEVEL=info"
            ],
            "Mounts": [
                {
                    "Description": "Optional external pipeline files",
                    "Destination": "/data",
                    "Name": "data",
                    "Options": [
                        "bind",
                        "ro"
                    ],
                    "Settable": [
                        "source",
                        "destination"
                    ],
                    "Source": "",
                    "Type": "none"
                }
            ]
        }
    }
]

Logging configuration I use

 logging:                                                                                               
            driver: loki                                                                                       
            options:                                                                                           
                loki-url: *loki-url                                                                            
                loki-pipeline-stages: |                                                                        
                    - regex:                                                                                   
                        expression: ^(?P<timestamp>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\.\d{3} \w+) \[\d+\] (?P<level>\w+)                                                                                                     
                    - labels:                                                                                  
                        internal_level: level                                                                  
                        level: level                                                                           
                    - replace:                                                                                 
                        expression: "(\\S+)"                                                                   
                        source: "level"                                                                        
                        replace: "info"                                                                        
                    - timestamp:                                                                               
                        source: timestamp                                                                      
                        format: 2006-01-02 15:04:05.999 MST          

#2033
https://github.com/grafana/loki/tree/master/cmd/docker-driver#pipeline-stages

@andoks
Copy link
Author

andoks commented Jun 29, 2020

This was a PEBCAK, and replace is working in the loki driver too.

The issue was that the labels stage must run after the replace stage.

@andoks andoks closed this as completed Jun 29, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant