Promtail: CRI-O multi-lines introduces newline and breaks json-parsing

[david-jointech]

We are using Loki to manage logs of our application in a kubernetes cluster. Inside the cluster we're using grafana-agents logs-configuration to parse the logs. We only use the cri-pipeline in promtail/grafana-agent. Generally our app logs in json-format.
Sometimes log-messages get very long and are split into two line by kubernetes. Since the fix of #5854 promtail combines cri-o-multiline log-lines. But in the place of the splits it inserts a newline. This newline breaks json-parsing in LogQL (at least in Grafana).

To Reproduce
Steps to reproduce the behavior:

1. Loki is running in a different cluster from our application, using the loki-helm-chart (chart-version 4.6.0)
2. Grafana-Agent is running in the cluster with the application, using the grafana-agent-operator (chart-version 0.2.8)
3. A PodLogs Resource configured with the cri-pipeline is running in that cluster.
4. The application logs a line which gets split into multiple lines by kubernetes (>8192 Bytes). For our example lets say the log line looks like this: {"level":"demo","timestamp":"2023-02-14T14:01:25.807Z"} and gets split into {"level":"demo","timestamp":"2023-02- and 14T14:01:25.807Z"}.
5. In Grafana we use the following query to query the log-line: {cluster="demo", container="service"} | json

Actual behavior
The log-message can't get parsed in Grafana. Instead it produces a JSONParserErr with the following details: ReadString: invalid control character found: 10, error found in #10 byte of [...].
Taking a look at the message, it introduced a newline where the split is:

{"level":"demo","timestamp":"2023-02-
14T14:01:25.807Z"}

I'm assuming this newline introduces the error. This error might also be related to #3245.
This newline is probably the newline from the end of the split-message.

Expected behavior
The json-parser should parse {"level":"demo","timestamp":"2023-02-14T14:01:25.807Z"} and create the appropriate labels (meaning level and timestamp). In general, the multiline parsing of cri shoudln't introduce newlines into the message (where none were before).

Environment:

• Infrastructure: Kubernetes
• Deployment tool: helm

Screenshots, Promtail config, or terminal output

PodLogs-Config:

apiVersion: monitoring.grafana.com/v1alpha1
kind: PodLogs
metadata:
  labels:
    instance: primary
  name: kubernetes-pods
  namespace: {{ .Values.namespace }}
spec:
  selector:
    matchLabels: {}
  namespaceSelector:
    any: true
  pipelineStages:
  - cri: {}

[dbluxo]

Should be fixed with #7997. The mentioned PR is merged but not released yet.

[david-jointech]

Oh yes, seems like I overlooked #7996 when searching for existing issues. Sorry about the duplicate.