Skip to content

Commit

Permalink
Loki Query Frontend (#1442)
Browse files Browse the repository at this point in the history 
* Adds frontend to Loki.

Signed-off-by: Cyril Tovena <[email protected]>

* Improves tests.

Signed-off-by: Cyril Tovena <[email protected]>

* Fixes sneaky bug in entries sorting.

Signed-off-by: Cyril Tovena <[email protected]>

* Fixes the split by interval.

Signed-off-by: Cyril Tovena <[email protected]>

* Tweak jsonnet deployments and add a way to lint/fmt.

Signed-off-by: Cyril Tovena <[email protected]>

* lint.

Signed-off-by: Cyril Tovena <[email protected]>

* fix timezone issue.

Signed-off-by: Cyril Tovena <[email protected]>

* Improve tests and rollback change in loghttp package.

Signed-off-by: Cyril Tovena <[email protected]>

* Fixes a flaky test that might run one more goroutine.

Signed-off-by: Cyril Tovena <[email protected]>

* Fixes windows build.

Signed-off-by: Cyril Tovena <[email protected]>

* Improve tracing in the split by interval.

Signed-off-by: Cyril Tovena <[email protected]>

* Add test stream to proto conversion.

Signed-off-by: Cyril Tovena <[email protected]>

* Fixes flappy retry test.

Signed-off-by: Cyril Tovena <[email protected]>

* Remove err shadowing in stopQueryFrontend as it was confusing.

Signed-off-by: Cyril Tovena <[email protected]>

* Refactor grpc message size in the libsonnet config file.

Signed-off-by: Cyril Tovena <[email protected]>

* Don't check auth header for GRPC  TransferChunks.

Signed-off-by: Cyril Tovena <[email protected]>

* Query frontend (#3)

* frontend codec merging optimizations

* codec benchmarks

* removes unused bounds code in queryrange ordering

* [wip] splitby uses channels instead of sub batching intervals

* splitBy channel limit test

* single allocation for merging entries from a single stream

* skip merging loki responses when limit is already hit

* removes checks for unlimited queries in queryrange

* removes splitByInterval{,.interval} spans

* removes interval_batch_size from jsonnet lib

* moves benchmark utils to own file

* renames markers -> entries

* priority queue comments

* Removes unused logRequest.

Signed-off-by: Cyril Tovena <[email protected]>

* Sets the cache interval to the same split interval.

Signed-off-by: Cyril Tovena <[email protected]>

* Missing import libsonnet for the frontend.

Signed-off-by: Cyril Tovena <[email protected]>

* Frontend should not be a cluster IP.

Signed-off-by: Cyril Tovena <[email protected]>

Co-authored-by: Owen Diehl <[email protected]>
  • Loading branch information
@cyriltovena @owen-d
cyriltovena and owen-d authored Jan 7, 2020
1 parent 0b72ddb commit 4153740
Show file tree
Hide file tree
Showing 605 changed files with 153,205 additions and 93,758 deletions.
15 changes: 14 additions & 1 deletion Makefile
View file
Original file line number Diff line number Diff line change
Expand Up @@ -303,7 +303,7 @@ else
protoc -I ./vendor:./$(@D) --gogoslick_out=plugins=grpc:./vendor ./$(patsubst %.pb.go,%.proto,$@); \
;; \
*) \
protoc -I ./vendor:./$(@D) --gogoslick_out=Mgoogle/protobuf/timestamp.proto=github.com/gogo/protobuf/types,plugins=grpc:./$(@D) ./$(patsubst %.pb.go,%.proto,$@); \
protoc -I .:./vendor:./$(@D) --gogoslick_out=Mgoogle/protobuf/timestamp.proto=github.com/gogo/protobuf/types,plugins=grpc,paths=source_relative:./ ./$(patsubst %.pb.go,%.proto,$@); \
;; \
esac
endif
Expand Down Expand Up @@ -534,3 +534,16 @@ else
GO111MODULE=on GOPROXY=https://proxy.golang.org go mod vendor
endif
@git diff --exit-code -- go.sum go.mod vendor/


lint-jsonnet:
@RESULT=0; \
for f in $$(find . -name 'vendor' -prune -o -name '*.libsonnet' -print -o -name '*.jsonnet' -print); do \
jsonnetfmt -- "$$f" | diff -u "$$f" -; \
RESULT=$$(($$RESULT + $$?)); \
done; \
exit $$RESULT

fmt-jsonnet:
@find . -name 'vendor' -prune -o -name '*.libsonnet' -print -o -name '*.jsonnet' -print | \
xargs -n 1 -- jsonnetfmt -i
22 changes: 11 additions & 11 deletions go.mod
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,28 +3,26 @@ module github.com/grafana/loki
go 1.13

require (
github.com/Azure/go-ansiterm v0.0.0-20170929234023-d6e3b3328b78 // indirect
github.com/Microsoft/go-winio v0.4.12 // indirect
github.com/blang/semver v3.5.1+incompatible // indirect
github.com/bmatcuk/doublestar v1.1.1
github.com/containerd/containerd v1.3.2 // indirect
github.com/containerd/fifo v0.0.0-20190226154929-a9fb20d87448 // indirect
github.com/coreos/go-systemd v0.0.0-20190321100706-95778dfbb74e
github.com/cortexproject/cortex v0.3.1-0.20191122194007-ed7c302fd968
github.com/cortexproject/cortex v0.4.1-0.20191217132644-cd4009e2f8e7
github.com/davecgh/go-spew v1.1.1
github.com/docker/distribution v2.7.1+incompatible // indirect
github.com/docker/docker v0.0.0-20190607191414-238f8eaa31aa
github.com/docker/go-connections v0.4.0 // indirect
github.com/docker/docker v0.7.3-0.20190817195342-4760db040282
github.com/docker/go-metrics v0.0.0-20181218153428-b84716841b82 // indirect
github.com/docker/go-plugins-helpers v0.0.0-20181025120712-1e6269c305b8
github.com/dustin/go-humanize v1.0.0
github.com/fatih/color v1.7.0
github.com/fluent/fluent-bit-go v0.0.0-20190925192703-ea13c021720c
github.com/frankban/quicktest v1.7.2 // indirect
github.com/go-kit/kit v0.9.0
github.com/gocql/gocql v0.0.0-20181124151448-70385f88b28b // indirect
github.com/gogo/protobuf v1.3.0 // remember to update loki-build-image/Dockerfile too
github.com/golang/snappy v0.0.1
github.com/gorilla/mux v1.6.2
github.com/gorilla/mux v1.7.1
github.com/gorilla/websocket v1.4.0
github.com/grpc-ecosystem/grpc-gateway v1.9.6 // indirect
github.com/grpc-ecosystem/grpc-opentracing v0.0.0-20180507213350-8e809c8a8645
Expand All @@ -35,10 +33,8 @@ require (
github.com/json-iterator/go v1.1.9
github.com/klauspost/compress v1.9.4
github.com/mitchellh/mapstructure v1.1.2
github.com/morikuni/aec v0.0.0-20170113033406-39771216ff4c // indirect
github.com/morikuni/aec v1.0.0 // indirect
github.com/mwitkow/go-conntrack v0.0.0-20190716064945-2f068394615f
github.com/opencontainers/go-digest v1.0.0-rc1 // indirect
github.com/opencontainers/image-spec v1.0.1 // indirect
github.com/opentracing/opentracing-go v1.1.0
github.com/pierrec/lz4 v2.3.1-0.20191115212037-9085dacd1e1e+incompatible
github.com/pkg/errors v0.8.1
Expand All @@ -56,15 +52,15 @@ require (
go.opencensus.io v0.22.1 // indirect
golang.org/x/net v0.0.0-20190923162816-aa69164e4478
golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e // indirect
golang.org/x/time v0.0.0-20190921001708-c4c64cad1fd0
golang.org/x/sys v0.0.0-20191218084908-4a24b4065292 // indirect
golang.org/x/time v0.0.0-20191024005414-555d28b269f0
golang.org/x/tools v0.0.0-20190925134113-a044388aa56f // indirect
google.golang.org/appengine v1.6.3 // indirect
google.golang.org/genproto v0.0.0-20190916214212-f660b8655731 // indirect
google.golang.org/grpc v1.25.1
gopkg.in/alecthomas/kingpin.v2 v2.2.6
gopkg.in/fsnotify.v1 v1.4.7
gopkg.in/yaml.v2 v2.2.2
gotest.tools v2.2.0+incompatible // indirect
k8s.io/klog v0.4.0
)

Expand All @@ -75,3 +71,7 @@ replace k8s.io/client-go => k8s.io/client-go v0.0.0-20190620085101-78d2af792bab

// Override reference causing proxy error. Otherwise it attempts to download https://proxy.golang.org/golang.org/x/net/@v/v0.0.0-20190813000000-74dc4d7220e7.info
replace golang.org/x/net => golang.org/x/net v0.0.0-20190923162816-aa69164e4478

replace github.com/Azure/azure-sdk-for-go => github.com/Azure/azure-sdk-for-go v36.2.0+incompatible

replace github.com/Azure/go-autorest => github.com/Azure/go-autorest v13.3.0+incompatible
Loading

0 comments on commit 4153740

Please sign in to comment.