LogQL: Introduce distinct

[liguozhong]

What this PR does / why we need it:
Introduces distinct to LogQL. Usage:
{job="varlogs"} | distinct filename
It is similar to distinct in SQL and especially useful for sampling. Similar syntax exists in other log query languages.

Example
For the following log lines:

{"event": "access", "id": "1", "time": "2023-02-28 15:12:11"}
{"event": "access", "id": "1", "time": "2023-02-28 15:13:11"}
{"event": "access", "id": "2", "time": "2023-02-28 15:14:11"}
{"event": "access", "id": "2", "time": "2023-02-28 15:15:11"}

The query below:

{app="order"} | json | distinct id

Will return:

{"event": "access", `"id": "1",` "time": "2023-02-28 15:13:11"}
{"event": "access", `"id": "2", `"time": "2023-02-28 15:15:11"}

Example with multiple labels

{app="order"} | json | distinct id,time

Which issue(s) this PR fixes:
Fixes #8649

Special notes for your reviewer:
{job="varlogs"} |distinct filename
The test passed in a simple local development environment
snapshot.

Checklist

• Reviewed the CONTRIBUTING.md guide (required)
• Documentation added
• Tests updated
• CHANGELOG.md updated
• Changes that require user attention or interaction to upgrade are documented in docs/sources/upgrading/_index.md

[rkonfj]

@liguozhong Thank you very much for solving the problem that is bothering me, I just want to say cool!

Two questions:

1. Can multiple labels be specified?（Doesn't matter if this feature exists or not） | distinct label1, label2
2. Are the deduplicated log lines the latest? I don't know if LogQL needs to implement a deduplication strategy, at least the strategy I need is to keep only the latest log line.

[JStickler]

[Docs squad] One small comment, otherwise the docs LGTM.

[liguozhong]

• Can multiple labels be specified?（Doesn't matter if this feature exists or not） | distinct label1, label2

nice, I submitted a commit to complete this feature.

{app="order"} | json | distinct id,time

[liguozhong]

2. Are the deduplicated log lines the latest? I don't know if LogQL needs to implement a deduplication strategy, at least the strategy I need is to keep only the latest log line.

No, currently the earliest log is kept. because the log query is stateless now, so aggregation cannot be achieved.
If we want to realize this feature, we need to make a lot of changes, and we need to introduce a state for the log query. According to my analysis, this requirement is difficult to achieve.

[liguozhong]

The mysql's distinct was no keep the last one too, maybe we can keep the same logic with mysql's SQL now.

[rkonfj]

@liguozhong

nice, I submitted a commit to complete this feature.

Thanks.

No, currently the earliest log is kept. because the log query is stateless now, so aggregation cannot be achieved.
If we want to realize this feature, we need to make a lot of changes, and we need to introduce a state for the log query. According to my analysis, this requirement is difficult to achieve.

You are right, pipeline filtering is not possible, which conflicts with streaming log lines, since the latest log line is in the future.

However, when the date range and query conditions are determined, the log lines is no longer a stream, but a determined lines. Is it possible to reevaluate the log lines to be returned(if there is a suitable place to do that)?

[rkonfj]

The mysql's distinct was no keep the last one too, maybe we can keep the same logic with mysql's SQL now.

The result of the MySQL query is the column values, it doesn't matter whether it is earliest or latest. But for logs, the log content itself is important, the deduplication strategy become more important, right?

[liguozhong]

Metric query can do agg and do a commit after all the data is processed. But there is no such callback func in log query now.

And there is an important question we have to consider.
The log query always has a limit, which means that we don't need to pull all the logs to end the query, but if we promise that distinct can return the last line of logs, we will have to pull all the s3 files to end this query.

I'm afraid it's hard for loki to pull the last log.But I also think that the last line of log is the most important log. Maybe you can modify the start time continuously.

[liguozhong]

Anyway, your issue is very valuable, I asked our developers, they also want a feature in their daily oncall work.

[jeschkies]

What happens when r.datas[label] returns nil? E.g. | logfmt would add new labels that have not been seens when r.data was initialized.

[liguozhong]

If the label does not exist, it will be discarded, and the new label is based on the order of the pipeline. Such as the following example

log
{"a":"1", "b":2}

logql 1 :
{app="buy2"} | json | distinct a
logql 2 :
{app="buy2"} | distinct a | json

logql 1 can get log. logql 2 will not get log.

[jeschkies]

Hm, but you'd still have a nil pointer right?

[liguozhong]

good point.This code does look dangerous.
No, each label has been initialized in the NewDistinctFilter () function.

func NewDistinctFilter(labels []string) (Stage, error) {
	datas := make(map[string]map[string]int, 0)
	for _, label := range labels {
		datas[label] = make(map[string]int, 0)
	}
	...
}

If distinctFilter struct initialized through NewDistinctFilter(), the distinctFilter struct will never get nil point. But if we directly build a distinctFilter struct with &distinctFilter{}, it is possible that the nil point you mentioned will appear.

another way.

this for performance, it will not need to make a judgment like this for each log.

 _, ok = r.datas[label]
if !ok {
r.datas[label] = make(map[string]int, 0)
}

[jeschkies]

Is NewDestinctFilter called for each line? How is preserving the state?

[liguozhong]

NewDestinctFilter is generated for each logql, not for each line. So the state in NewDestinctFilter can be maintained continuously in one logql. The screenshot in the PR can also show that this state is valid

[jeschkies]

Thanks for your patience. I still don't understand how data[label] can be not nil if the label not known when NewDestinctFilter is called. I'll try out.

[jeschkies]

Ah, no I get it. The labels are not the labels seen in the line but the ones passed to distinct 🙈

[liguozhong]

thanks, ready for review.

[jeschkies]

@liguozhong I hope you don't mind me adding this test.

[jeschkies]

@DylanGuedes I'm fine with this change. What do you say?

[liguozhong]

thanks

[DylanGuedes]

@DylanGuedes I'm fine with this change. What do you say?

same 😄

@liguozhong thank you again for another awesome contribution ❤️

[xkkker]

Hi,what's the milestone of this feature? The lastest version 2.8.4 don't support distinct for now.