Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gateway: use go-micro store to cache #3250

Merged
merged 9 commits into from
Sep 23, 2022
Merged

gateway: use go-micro store to cache #3250

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
15a3a4c
gateway: use go-micro store to cache
butonic Sep 19, 2022
6df7d57
extract cache package
butonic Sep 20, 2022
73c00c5
refactor cache key generation
butonic Sep 20, 2022
4e33f26
split and reuse caches
butonic Sep 21, 2022
d7df2a9
bugfixes and cleanup
butonic Sep 21, 2022
5490a8f
lint
butonic Sep 21, 2022
4bb6fc6
update changelog
butonic Sep 21, 2022
4b0d436
go critic
butonic Sep 21, 2022
17181b9
also invalidate cache if events are disabled
butonic Sep 22, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions changelog/unreleased/micro-cache-store.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
Enhancement: allow sharing the gateway caches

We replaced the in memory implementation of the gateway with go-micro stores. The gateways `cache_store` defaults to `noop` and can be set to `memory`, `redis` or `etcd`. When setting it also set any dataproviders `datatxs.*.cache_store` new config option to the same values so they can invalidate the cache when a file has been uploadad.

Cache instances will be shared between handlers when they use the same configuration in the same process to allow the dataprovider to access the same cache as the gateway.

The `nats-js` implementation requires a limited set of characters in the key and is currently known to be broken.

The `etag_cache_ttl` was removed as it was not used anyway.

https://github.com/cs3org/reva/pull/3250
42 changes: 24 additions & 18 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ require (
github.com/CiscoM31/godata v1.0.5
github.com/Masterminds/sprig v2.22.0+incompatible
github.com/ReneKroon/ttlcache/v2 v2.11.0
github.com/aws/aws-sdk-go v1.44.61
github.com/aws/aws-sdk-go v1.44.94
github.com/beevik/etree v1.1.0
github.com/bluele/gcache v0.0.2
github.com/c-bata/go-prompt v0.2.5
Expand All @@ -29,6 +29,8 @@ require (
github.com/go-ldap/ldap/v3 v3.4.4
github.com/go-micro/plugins/v4/events/natsjs v1.1.0
github.com/go-micro/plugins/v4/server/http v1.1.0
github.com/go-micro/plugins/v4/store/nats-js v1.1.0
github.com/go-micro/plugins/v4/store/redis v1.1.0
github.com/go-sql-driver/mysql v1.6.0
github.com/gofrs/flock v0.8.1
github.com/golang-jwt/jwt v3.2.2+incompatible
Expand All @@ -49,30 +51,31 @@ require (
github.com/minio/minio-go/v7 v7.0.32
github.com/mitchellh/mapstructure v1.5.0
github.com/nats-io/nats-server/v2 v2.8.4
github.com/onsi/ginkgo/v2 v2.1.4
github.com/onsi/gomega v1.20.0
github.com/owncloud/ocis/v2 v2.0.0-beta2
github.com/nats-io/nats.go v1.17.0
github.com/onsi/ginkgo/v2 v2.1.6
github.com/onsi/gomega v1.20.2
github.com/owncloud/ocis/v2 v2.0.0-beta2.0.20220919072836-08a8ed20c18a
github.com/pkg/errors v0.9.1
github.com/pkg/xattr v0.4.7
github.com/prometheus/alertmanager v0.24.0
github.com/rs/cors v1.8.2
github.com/rs/zerolog v1.27.0
github.com/rs/zerolog v1.28.0
github.com/sciencemesh/meshdirectory-web v1.0.4
github.com/sethvargo/go-password v0.2.0
github.com/stretchr/testify v1.8.0
github.com/studio-b12/gowebdav v0.0.0-20220128162035-c7b1ff8a5e62
github.com/thanhpk/randstr v1.0.4
github.com/tus/tusd v1.9.0
github.com/tus/tusd v1.9.2
github.com/wk8/go-ordered-map v1.0.0
go-micro.dev/v4 v4.8.1
go.opencensus.io v0.23.0
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.33.0
go.opentelemetry.io/otel v1.8.0
go.opentelemetry.io/otel/exporters/jaeger v1.8.0
go.opentelemetry.io/otel/sdk v1.8.0
go.opentelemetry.io/otel/trace v1.8.0
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.34.0
go.opentelemetry.io/otel v1.10.0
go.opentelemetry.io/otel/exporters/jaeger v1.9.0
go.opentelemetry.io/otel/sdk v1.9.0
go.opentelemetry.io/otel/trace v1.10.0
golang.org/x/crypto v0.0.0-20220919173607-35f4265a4bc0
golang.org/x/oauth2 v0.0.0-20220722155238-128564f6959c
golang.org/x/oauth2 v0.0.0-20220822191816-0ebed06d0094
golang.org/x/sync v0.0.0-20220907140024-f12130a52804
golang.org/x/sys v0.0.0-20220919091848-fb04ddd9f9c8
golang.org/x/term v0.0.0-20220722155259-a9ba230a4035
Expand Down Expand Up @@ -100,6 +103,7 @@ require (
github.com/coreos/go-systemd/v22 v22.4.0 // indirect
github.com/cpuguy83/go-md2man/v2 v2.0.2 // indirect
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f // indirect
github.com/dustin/go-humanize v1.0.0 // indirect
github.com/emirpasic/gods v1.18.1 // indirect
github.com/fatih/color v1.13.0 // indirect
Expand All @@ -119,6 +123,7 @@ require (
github.com/go-micro/plugins/v4/registry/nats v1.1.0 // indirect
github.com/go-openapi/errors v0.20.2 // indirect
github.com/go-openapi/strfmt v0.21.2 // indirect
github.com/go-redis/redis/v8 v8.10.0 // indirect
github.com/go-stack/stack v1.8.1 // indirect
github.com/gogo/protobuf v1.3.2 // indirect
github.com/golang/glog v1.0.0 // indirect
Expand Down Expand Up @@ -156,7 +161,6 @@ require (
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
github.com/modern-go/reflect2 v1.0.2 // indirect
github.com/nats-io/jwt/v2 v2.2.1-0.20220330180145-442af02fd36a // indirect
github.com/nats-io/nats.go v1.17.0 // indirect
github.com/nats-io/nkeys v0.3.0 // indirect
github.com/nats-io/nuid v1.0.1 // indirect
github.com/nxadm/tail v1.4.8 // indirect
Expand All @@ -167,18 +171,18 @@ require (
github.com/pkg/term v1.1.0 // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect
github.com/pquerna/cachecontrol v0.1.0 // indirect
github.com/prometheus/client_golang v1.12.2 // indirect
github.com/prometheus/client_golang v1.13.0 // indirect
github.com/prometheus/client_model v0.2.0 // indirect
github.com/prometheus/common v0.32.1 // indirect
github.com/prometheus/procfs v0.7.3 // indirect
github.com/prometheus/common v0.37.0 // indirect
github.com/prometheus/procfs v0.8.0 // indirect
github.com/prometheus/statsd_exporter v0.22.4 // indirect
github.com/rivo/uniseg v0.2.0 // indirect
github.com/rs/xid v1.3.0 // indirect
github.com/rs/xid v1.4.0 // indirect
github.com/russross/blackfriday/v2 v2.1.0 // indirect
github.com/sergi/go-diff v1.2.0 // indirect
github.com/shurcooL/httpfs v0.0.0-20190707220628-8d4bc4ba7749 // indirect
github.com/shurcooL/vfsgen v0.0.0-20200824052919-0d455de96546 // indirect
github.com/sirupsen/logrus v1.8.1 // indirect
github.com/sirupsen/logrus v1.9.0 // indirect
github.com/stretchr/objx v0.4.0 // indirect
github.com/tidwall/pretty v1.2.0 // indirect
github.com/urfave/cli/v2 v2.16.3 // indirect
Expand All @@ -188,6 +192,8 @@ require (
go.etcd.io/etcd/client/pkg/v3 v3.5.5 // indirect
go.etcd.io/etcd/client/v3 v3.5.5 // indirect
go.mongodb.org/mongo-driver v1.8.3 // indirect
go.opentelemetry.io/otel/internal/metric v0.27.0 // indirect
go.opentelemetry.io/otel/metric v0.27.0 // indirect
go.uber.org/atomic v1.10.0 // indirect
go.uber.org/multierr v1.8.0 // indirect
go.uber.org/zap v1.23.0 // indirect
Expand Down
Loading