Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

nats-js cache not working #7049

Closed
wkloucek opened this issue Aug 16, 2023 · 9 comments · Fixed by #7793
Closed

nats-js cache not working #7049

wkloucek opened this issue Aug 16, 2023 · 9 comments · Fixed by #7793
Assignees
@kobergj
Labels
Priority:p3-medium Normal priority Type:Bug
Milestone
Release 5.0.0

Comments

@wkloucek
Copy link
Contributor

wkloucek commented Aug 16, 2023 •
edited
Loading

Describe the bug

The nats-js cache seems not to work properly for at least the storage-system service

Steps to reproduce

Steps to reproduce the behavior:

  1. run the oCIS Helm Chart with NATS example, where we use NATS as cache (use nats also as cache in the nats deployment example ocis-charts#378)
  2. try to use oCIS

Expected behavior

oCIS works correctly

Actual behavior

storage-system errors:

storagesystem-5f47ddfb78-d2t8j storagesystem {"level":"info","service":"storage-system","time":"2023-08-16T13:59:28.184615384Z","line":"github.com/owncloud/ocis/v2/ocis-pkg/registry/register.go:17","message":"registering external service com.owncloud.api.storage-system-a94fb2ac-c21d-4c70-a9d7-5293bc5578b8@10.244.3.101:9215"}
storagesystem-5f47ddfb78-d2t8j storagesystem {"level":"info","service":"storage-system","time":"2023-08-16T13:59:28.193060091Z","line":"github.com/owncloud/ocis/v2/ocis-pkg/registry/register.go:17","message":"registering external service com.owncloud.web.storage-system-65062824-ceac-4a2c-9dfd-7f60b3940173@10.244.3.101:9216"}
storagesystem-5f47ddfb78-d2t8j storagesystem {"level":"info","service":"storage-system","time":"2023-08-16T13:59:28.193699911Z","line":"github.com/cs3org/reva/[email protected]/cmd/revad/runtime/runtime.go:87","message":"host info: storagesystem-5f47ddfb78-d2t8j"}
storagesystem-5f47ddfb78-d2t8j storagesystem {"level":"info","service":"storage-system","time":"2023-08-16T13:59:28.193775829Z","line":"github.com/cs3org/reva/[email protected]/cmd/revad/runtime/runtime.go:180","message":"running on 12 cpus"}
storagesystem-5f47ddfb78-d2t8j storagesystem {"level":"info","service":"storage-system","pkg":"grace","time":"2023-08-16T13:59:28.194297547Z","line":"github.com/cs3org/reva/[email protected]/cmd/revad/internal/grace/grace.go:187","message":"pidfile saved at: /tmp/revad-storage-system-9873ebcc-da7b-4c78-95c2-cec00ef5e5bd.pid"}
storagesystem-5f47ddfb78-d2t8j storagesystem {"level":"warn","service":"storage-system","pkg":"rhttp","time":"2023-08-16T13:59:28.194725607Z","line":"github.com/cs3org/reva/[email protected]/internal/http/services/dataprovider/dataprovider.go:87","message":"missing or incomplete nats configuration. Events will not be published."}
storagesystem-5f47ddfb78-d2t8j storagesystem {"level":"info","service":"storage-system","pkg":"rgrpc","time":"2023-08-16T13:59:28.195499943Z","line":"github.com/cs3org/reva/[email protected]/pkg/rgrpc/rgrpc.go:227","message":"rgrpc: grpc service enabled: storageprovider"}
storagesystem-5f47ddfb78-d2t8j storagesystem {"level":"info","service":"storage-system","pkg":"rhttp","time":"2023-08-16T13:59:28.195601144Z","line":"github.com/cs3org/reva/[email protected]/pkg/rhttp/rhttp.go:220","message":"http service enabled: dataprovider@/data"}
storagesystem-5f47ddfb78-d2t8j storagesystem {"level":"info","service":"storage-system","pkg":"rhttp","time":"2023-08-16T13:59:28.195672103Z","line":"github.com/cs3org/reva/[email protected]/pkg/rhttp/rhttp.go:193","message":"http middleware enabled: prometheus"}
storagesystem-5f47ddfb78-d2t8j storagesystem {"level":"info","service":"storage-system","pkg":"rhttp","time":"2023-08-16T13:59:28.195692217Z","line":"github.com/cs3org/reva/[email protected]/pkg/rhttp/rhttp.go:273","message":"chaining http middleware prometheus with priority  100"}
storagesystem-5f47ddfb78-d2t8j storagesystem {"level":"info","service":"storage-system","pkg":"rhttp","time":"2023-08-16T13:59:28.195708281Z","line":"github.com/cs3org/reva/[email protected]/pkg/rhttp/rhttp.go:278","message":"unprotected URL: /data/tus"}
storagesystem-5f47ddfb78-d2t8j storagesystem {"level":"info","service":"storage-system","pkg":"rgrpc","time":"2023-08-16T13:59:28.195748859Z","line":"github.com/cs3org/reva/[email protected]/pkg/rgrpc/rgrpc.go:227","message":"rgrpc: grpc service enabled: gateway"}
storagesystem-5f47ddfb78-d2t8j storagesystem {"level":"info","service":"storage-system","pkg":"rhttp","time":"2023-08-16T13:59:28.195830574Z","line":"github.com/cs3org/reva/[email protected]/pkg/rhttp/rhttp.go:124","message":"http server listening at http://0.0.0.0:9216 '' ''"}
storagesystem-5f47ddfb78-d2t8j storagesystem {"level":"info","service":"storage-system","pkg":"rgrpc","time":"2023-08-16T13:59:28.195913546Z","line":"github.com/cs3org/reva/[email protected]/pkg/rgrpc/rgrpc.go:227","message":"rgrpc: grpc service enabled: userprovider"}
storagesystem-5f47ddfb78-d2t8j storagesystem {"level":"info","service":"storage-system","pkg":"rgrpc","time":"2023-08-16T13:59:28.195970746Z","line":"github.com/cs3org/reva/[email protected]/pkg/rgrpc/rgrpc.go:227","message":"rgrpc: grpc service enabled: authregistry"}
storagesystem-5f47ddfb78-d2t8j storagesystem {"level":"info","service":"storage-system","pkg":"rgrpc","time":"2023-08-16T13:59:28.196021312Z","line":"github.com/cs3org/reva/[email protected]/pkg/rgrpc/rgrpc.go:227","message":"rgrpc: grpc service enabled: authprovider"}
storagesystem-5f47ddfb78-d2t8j storagesystem {"level":"info","service":"storage-system","pkg":"rgrpc","time":"2023-08-16T13:59:28.196065522Z","line":"github.com/cs3org/reva/[email protected]/pkg/rgrpc/rgrpc.go:227","message":"rgrpc: grpc service enabled: permissions"}
storagesystem-5f47ddfb78-d2t8j storagesystem {"level":"info","service":"storage-system","pkg":"rgrpc","time":"2023-08-16T13:59:28.196137039Z","line":"github.com/cs3org/reva/[email protected]/pkg/rgrpc/rgrpc.go:227","message":"rgrpc: grpc service enabled: storageregistry"}
storagesystem-5f47ddfb78-d2t8j storagesystem {"level":"info","service":"storage-system","pkg":"rgrpc","time":"2023-08-16T13:59:28.196421784Z","line":"github.com/cs3org/reva/[email protected]/pkg/rgrpc/rgrpc.go:331","message":"rgrpc: chaining grpc unary interceptor prometheus with priority 100"}
storagesystem-5f47ddfb78-d2t8j storagesystem {"level":"info","service":"storage-system","pkg":"rgrpc","time":"2023-08-16T13:59:28.196633754Z","line":"github.com/cs3org/reva/[email protected]/pkg/rgrpc/rgrpc.go:191","message":"grpc server listening at tcp:0.0.0.0:9215"}
storagesystem-5f47ddfb78-d2t8j storagesystem {"level":"info","service":"storage-system","pkg":"rgrpc","traceid":"00000000000000000000000000000000","time":"2023-08-16T14:03:55.248261052Z","line":"github.com/cs3org/reva/[email protected]/internal/grpc/services/authprovider/authprovider.go:141","message":"user idp:\"internal\" opaque_id:\"6afdb672-44d1-4eda-b2ff-0c5bcee784ff\" type:USER_TYPE_PRIMARY  authenticated"}
storagesystem-5f47ddfb78-d2t8j storagesystem goroutine 409 [running]:
storagesystem-5f47ddfb78-d2t8j storagesystem runtime/debug.Stack()
storagesystem-5f47ddfb78-d2t8j storagesystem 	runtime/debug/stack.go:24 +0x65
storagesystem-5f47ddfb78-d2t8j storagesystem runtime/debug.PrintStack()
storagesystem-5f47ddfb78-d2t8j storagesystem 	runtime/debug/stack.go:16 +0x19
storagesystem-5f47ddfb78-d2t8j storagesystem github.com/cs3org/reva/v2/internal/grpc/interceptors/recovery.recoveryFunc({0x4520868, 0xc001354540}, {0x42bc300, 0xc00014b8d8})
storagesystem-5f47ddfb78-d2t8j storagesystem 	github.com/cs3org/reva/[email protected]/internal/grpc/interceptors/recovery/recovery.go:48 +0x3a
storagesystem-5f47ddfb78-d2t8j storagesystem github.com/grpc-ecosystem/go-grpc-middleware/recovery.recoverFrom({0x4520868?, 0xc001354540?}, {0x42bc300?, 0xc00014b8d8?}, 0xc001397e30?)
storagesystem-5f47ddfb78-d2t8j storagesystem 	github.com/grpc-ecosystem/[email protected]/recovery/interceptors.go:61 +0x36
storagesystem-5f47ddfb78-d2t8j storagesystem github.com/grpc-ecosystem/go-grpc-middleware/recovery.UnaryServerInterceptor.func1.1()
storagesystem-5f47ddfb78-d2t8j storagesystem 	github.com/grpc-ecosystem/[email protected]/recovery/interceptors.go:29 +0x7b
storagesystem-5f47ddfb78-d2t8j storagesystem panic({0x42bc300, 0xc00014b8d8})
storagesystem-5f47ddfb78-d2t8j storagesystem 	runtime/panic.go:884 +0x213
storagesystem-5f47ddfb78-d2t8j storagesystem github.com/go-micro/plugins/v4/store/nats-js.(*natsStore).Read(0xc000157600, {0xc00065e69f, 0x5b}, {0xc00163c8a0, 0x2, 0x437aa80?})
storagesystem-5f47ddfb78-d2t8j storagesystem 	github.com/go-micro/plugins/v4/store/[email protected]/nats.go:246 +0xac5
storagesystem-5f47ddfb78-d2t8j storagesystem github.com/cs3org/reva/v2/pkg/storage/cache.cacheStore.PullFromCache({{0x452ec20, 0xc000157600}, {0x2840066, 0x4}, {0x2906e47, 0xd}, 0x14f46b04000}, {0xc00065e69f, 0x5b}, {0x3e08b60, ...})
storagesystem-5f47ddfb78-d2t8j storagesystem 	github.com/cs3org/reva/[email protected]/pkg/storage/cache/cache.go:175 +0x1da
storagesystem-5f47ddfb78-d2t8j storagesystem github.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs/metadata.MessagePackBackend.loadAttributes({{0xc0015fc5c0?, 0x6?}, {0x4528f00?, 0xc00168e140?}}, {0x4520868?, 0xc001354db0?}, {0xc00065e680, 0x7a}, {0x0, 0x0})
storagesystem-5f47ddfb78-d2t8j storagesystem 	github.com/cs3org/reva/[email protected]/pkg/storage/utils/decomposedfs/metadata/messagepack_backend.go:213 +0x243
storagesystem-5f47ddfb78-d2t8j storagesystem github.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs/metadata.MessagePackBackend.All(...)
storagesystem-5f47ddfb78-d2t8j storagesystem 	github.com/cs3org/reva/[email protected]/pkg/storage/utils/decomposedfs/metadata/messagepack_backend.go:65
storagesystem-5f47ddfb78-d2t8j storagesystem github.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs/node.(*Node).Xattr(0xc001346870, {0x4520868, 0xc001354db0}, {0x29215be, 0x12})
storagesystem-5f47ddfb78-d2t8j storagesystem 	github.com/cs3org/reva/[email protected]/pkg/storage/utils/decomposedfs/node/xattrs.go:148 +0xd5
storagesystem-5f47ddfb78-d2t8j storagesystem github.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs/node.(*Node).XattrString(...)
storagesystem-5f47ddfb78-d2t8j storagesystem 	github.com/cs3org/reva/[email protected]/pkg/storage/utils/decomposedfs/node/xattrs.go:164
storagesystem-5f47ddfb78-d2t8j storagesystem github.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs/node.(*Node).readOwner(0xc001346870, {0x4520868, 0xc001354db0})
storagesystem-5f47ddfb78-d2t8j storagesystem 	github.com/cs3org/reva/[email protected]/pkg/storage/utils/decomposedfs/node/node.go:437 +0x5f
storagesystem-5f47ddfb78-d2t8j storagesystem github.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs/node.ReadNode({0x4520868?, 0xc001354870?}, {0x4528f48, 0xc001692108}, {0xc0003d0d20, 0x24}, {0xc0003d0d20, 0x24}, 0x1, 0x0, ...)
storagesystem-5f47ddfb78-d2t8j storagesystem 	github.com/cs3org/reva/[email protected]/pkg/storage/utils/decomposedfs/node/node.go:229 +0x1f6
storagesystem-5f47ddfb78-d2t8j storagesystem github.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs.(*Decomposedfs).CreateStorageSpace(0xc000566240, {0x4520868?, 0xc001354840?}, 0xc000df88a0)
storagesystem-5f47ddfb78-d2t8j storagesystem 	github.com/cs3org/reva/[email protected]/pkg/storage/utils/decomposedfs/spaces.go:91 +0x497
storagesystem-5f47ddfb78-d2t8j storagesystem github.com/cs3org/reva/v2/internal/grpc/services/storageprovider.(*service).CreateStorageSpace(0xc00168e2c0, {0x4520868, 0xc001354840}, 0xc000df88a0)
storagesystem-5f47ddfb78-d2t8j storagesystem 	github.com/cs3org/reva/[email protected]/internal/grpc/services/storageprovider/storageprovider.go:480 +0x49
storagesystem-5f47ddfb78-d2t8j storagesystem github.com/cs3org/go-cs3apis/cs3/storage/provider/v1beta1._ProviderAPI_CreateStorageSpace_Handler.func1({0x4520868, 0xc001354840}, {0x436a160?, 0xc000df88a0})
storagesystem-5f47ddfb78-d2t8j storagesystem 	github.com/cs3org/[email protected]/cs3/storage/provider/v1beta1/provider_api.pb.go:6454 +0x78
storagesystem-5f47ddfb78-d2t8j storagesystem go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1({0x4520868, 0xc001354780}, {0x436a160, 0xc000df88a0}, 0xc001718d60, 0xc000adbc80)
storagesystem-5f47ddfb78-d2t8j storagesystem 	go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/[email protected]/interceptor.go:341 +0x437
storagesystem-5f47ddfb78-d2t8j storagesystem github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x4520868?, 0xc001354780?}, {0x436a160?, 0xc000df88a0?})
storagesystem-5f47ddfb78-d2t8j storagesystem 	github.com/grpc-ecosystem/[email protected]/chain.go:25 +0x3a
storagesystem-5f47ddfb78-d2t8j storagesystem github.com/cs3org/reva/v2/internal/grpc/interceptors/prometheus.interceptorFromConfig.func1({0x4520868?, 0xc001354780?}, {0x436a160?, 0xc000df88a0?}, 0x3e91120?, 0x3c11201?)
storagesystem-5f47ddfb78-d2t8j storagesystem 	github.com/cs3org/reva/[email protected]/internal/grpc/interceptors/prometheus/prometheus.go:72 +0x43
storagesystem-5f47ddfb78-d2t8j storagesystem github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x4520868?, 0xc001354780?}, {0x436a160?, 0xc000df88a0?})
storagesystem-5f47ddfb78-d2t8j storagesystem 	github.com/grpc-ecosystem/[email protected]/chain.go:25 +0x3a
storagesystem-5f47ddfb78-d2t8j storagesystem github.com/cs3org/reva/v2/internal/grpc/interceptors/auth.NewUnary.func2({0x4520868, 0xc001354540}, {0x436a160, 0xc000df88a0}, 0xc001718d60, 0xc001718da0)
storagesystem-5f47ddfb78-d2t8j storagesystem 	github.com/cs3org/reva/[email protected]/internal/grpc/interceptors/auth/auth.go:160 +0xa06
storagesystem-5f47ddfb78-d2t8j storagesystem github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x4520868?, 0xc001354540?}, {0x436a160?, 0xc000df88a0?})
storagesystem-5f47ddfb78-d2t8j storagesystem 	github.com/grpc-ecosystem/[email protected]/chain.go:25 +0x3a
storagesystem-5f47ddfb78-d2t8j storagesystem github.com/grpc-ecosystem/go-grpc-middleware/recovery.UnaryServerInterceptor.func1({0x4520868?, 0xc001354540?}, {0x436a160?, 0xc000df88a0?}, 0xc000df5228?, 0x4965b7?)
storagesystem-5f47ddfb78-d2t8j storagesystem 	github.com/grpc-ecosystem/[email protected]/recovery/interceptors.go:33 +0xc4
storagesystem-5f47ddfb78-d2t8j storagesystem github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x4520868?, 0xc001354540?}, {0x436a160?, 0xc000df88a0?})
storagesystem-5f47ddfb78-d2t8j storagesystem 	github.com/grpc-ecosystem/[email protected]/chain.go:25 +0x3a
storagesystem-5f47ddfb78-d2t8j storagesystem github.com/cs3org/reva/v2/internal/grpc/interceptors/log.NewUnary.func1({0x4520868, 0xc001354540}, {0x436a160, 0xc000df88a0}, 0xc001718d60, 0xc001718de0)
storagesystem-5f47ddfb78-d2t8j storagesystem 	github.com/cs3org/reva/[email protected]/internal/grpc/interceptors/log/log.go:39 +0xa2
storagesystem-5f47ddfb78-d2t8j storagesystem github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x4520868?, 0xc001354540?}, {0x436a160?, 0xc000df88a0?})
storagesystem-5f47ddfb78-d2t8j storagesystem 	github.com/grpc-ecosystem/[email protected]/chain.go:25 +0x3a
storagesystem-5f47ddfb78-d2t8j storagesystem github.com/cs3org/reva/v2/internal/grpc/interceptors/useragent.NewUnary.func1({0x4520868, 0xc001354540}, {0x436a160, 0xc000df88a0}, 0x1?, 0xc001718e00)
storagesystem-5f47ddfb78-d2t8j storagesystem 	github.com/cs3org/reva/[email protected]/internal/grpc/interceptors/useragent/useragent.go:38 +0xf4
storagesystem-5f47ddfb78-d2t8j storagesystem github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x4520868?, 0xc001354540?}, {0x436a160?, 0xc000df88a0?})
storagesystem-5f47ddfb78-d2t8j storagesystem 	github.com/grpc-ecosystem/[email protected]/chain.go:25 +0x3a
storagesystem-5f47ddfb78-d2t8j storagesystem github.com/cs3org/reva/v2/internal/grpc/interceptors/token.NewUnary.func1({0x4520868, 0xc0013544b0}, {0x436a160, 0xc000df88a0}, 0x60643a0?, 0xc001718e20)
storagesystem-5f47ddfb78-d2t8j storagesystem 	github.com/cs3org/reva/[email protected]/internal/grpc/interceptors/token/token.go:44 +0x16d
storagesystem-5f47ddfb78-d2t8j storagesystem github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x4520868?, 0xc0013544b0?}, {0x436a160?, 0xc000df88a0?})
storagesystem-5f47ddfb78-d2t8j storagesystem 	github.com/grpc-ecosystem/[email protected]/chain.go:25 +0x3a
storagesystem-5f47ddfb78-d2t8j storagesystem github.com/cs3org/reva/v2/internal/grpc/interceptors/appctx.NewUnary.func1({0x4520868, 0xc000dfacc0}, {0x436a160, 0xc000df88a0}, 0x4096460?, 0xc001718e40)
storagesystem-5f47ddfb78-d2t8j storagesystem 	github.com/cs3org/reva/[email protected]/internal/grpc/interceptors/appctx/appctx.go:51 +0x73c
storagesystem-5f47ddfb78-d2t8j storagesystem github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x4520868?, 0xc000dfacc0?}, {0x436a160?, 0xc000df88a0?})
storagesystem-5f47ddfb78-d2t8j storagesystem 	github.com/grpc-ecosystem/[email protected]/chain.go:25 +0x3a
storagesystem-5f47ddfb78-d2t8j storagesystem github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1({0x4520868, 0xc000dfacc0}, {0x436a160, 0xc000df88a0}, 0xc0004b9a58?, 0x3fc5600?)
storagesystem-5f47ddfb78-d2t8j storagesystem 	github.com/grpc-ecosystem/[email protected]/chain.go:34 +0xbe
storagesystem-5f47ddfb78-d2t8j storagesystem github.com/cs3org/go-cs3apis/cs3/storage/provider/v1beta1._ProviderAPI_CreateStorageSpace_Handler({0x445b6a0?, 0xc00168e2c0}, {0x4520868, 0xc000dfacc0}, 0xc000a9ec40, 0xc000547230)
storagesystem-5f47ddfb78-d2t8j storagesystem 	github.com/cs3org/[email protected]/cs3/storage/provider/v1beta1/provider_api.pb.go:6456 +0x138
storagesystem-5f47ddfb78-d2t8j storagesystem google.golang.org/grpc.(*Server).processUnaryRPC(0xc0016da000, {0x4533780, 0xc0004f09c0}, 0xc0004e3440, 0xc000547500, 0x5dcc550, 0x0)
storagesystem-5f47ddfb78-d2t8j storagesystem 	google.golang.org/[email protected]/server.go:1337 +0xdf3
storagesystem-5f47ddfb78-d2t8j storagesystem google.golang.org/grpc.(*Server).handleStream(0xc0016da000, {0x4533780, 0xc0004f09c0}, 0xc0004e3440, 0x0)
storagesystem-5f47ddfb78-d2t8j storagesystem 	google.golang.org/[email protected]/server.go:1714 +0xa36
storagesystem-5f47ddfb78-d2t8j storagesystem google.golang.org/grpc.(*Server).serveStreams.func1.1()
storagesystem-5f47ddfb78-d2t8j storagesystem 	google.golang.org/[email protected]/server.go:959 +0x98
storagesystem-5f47ddfb78-d2t8j storagesystem created by google.golang.org/grpc.(*Server).serveStreams.func1
storagesystem-5f47ddfb78-d2t8j storagesystem 	google.golang.org/[email protected]/server.go:957 +0x18c
storagesystem-5f47ddfb78-d2t8j storagesystem {"level":"error","service":"storage-system","pkg":"rgrpc","traceid":"00000000000000000000000000000000","time":"2023-08-16T14:03:55.268993739Z","line":"github.com/cs3org/reva/[email protected]/internal/grpc/interceptors/recovery/recovery.go:50","message":"runtime error: slice bounds out of range [:1] with capacity 0; stack: goroutine 409 [running]:\nruntime/debug.Stack()\n\truntime/debug/stack.go:24 +0x65\ngithub.com/cs3org/reva/v2/internal/grpc/interceptors/recovery.recoveryFunc({0x4520868, 0xc001354540}, {0x42bc300, 0xc00014b8d8})\n\tgithub.com/cs3org/reva/[email protected]/internal/grpc/interceptors/recovery/recovery.go:50 +0x5b\ngithub.com/grpc-ecosystem/go-grpc-middleware/recovery.recoverFrom({0x4520868?, 0xc001354540?}, {0x42bc300?, 0xc00014b8d8?}, 0xc001397e30?)\n\tgithub.com/grpc-ecosystem/[email protected]/recovery/interceptors.go:61 +0x36\ngithub.com/grpc-ecosystem/go-grpc-middleware/recovery.UnaryServerInterceptor.func1.1()\n\tgithub.com/grpc-ecosystem/[email protected]/recovery/interceptors.go:29 +0x7b\npanic({0x42bc300, 0xc00014b8d8})\n\truntime/panic.go:884 +0x213\ngithub.com/go-micro/plugins/v4/store/nats-js.(*natsStore).Read(0xc000157600, {0xc00065e69f, 0x5b}, {0xc00163c8a0, 0x2, 0x437aa80?})\n\tgithub.com/go-micro/plugins/v4/store/[email protected]/nats.go:246 +0xac5\ngithub.com/cs3org/reva/v2/pkg/storage/cache.cacheStore.PullFromCache({{0x452ec20, 0xc000157600}, {0x2840066, 0x4}, {0x2906e47, 0xd}, 0x14f46b04000}, {0xc00065e69f, 0x5b}, {0x3e08b60, ...})\n\tgithub.com/cs3org/reva/[email protected]/pkg/storage/cache/cache.go:175 +0x1da\ngithub.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs/metadata.MessagePackBackend.loadAttributes({{0xc0015fc5c0?, 0x6?}, {0x4528f00?, 0xc00168e140?}}, {0x4520868?, 0xc001354db0?}, {0xc00065e680, 0x7a}, {0x0, 0x0})\n\tgithub.com/cs3org/reva/[email protected]/pkg/storage/utils/decomposedfs/metadata/messagepack_backend.go:213 +0x243\ngithub.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs/metadata.MessagePackBackend.All(...)\n\tgithub.com/cs3org/reva/[email protected]/pkg/storage/utils/decomposedfs/metadata/messagepack_backend.go:65\ngithub.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs/node.(*Node).Xattr(0xc001346870, {0x4520868, 0xc001354db0}, {0x29215be, 0x12})\n\tgithub.com/cs3org/reva/[email protected]/pkg/storage/utils/decomposedfs/node/xattrs.go:148 +0xd5\ngithub.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs/node.(*Node).XattrString(...)\n\tgithub.com/cs3org/reva/[email protected]/pkg/storage/utils/decomposedfs/node/xattrs.go:164\ngithub.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs/node.(*Node).readOwner(0xc001346870, {0x4520868, 0xc001354db0})\n\tgithub.com/cs3org/reva/[email protected]/pkg/storage/utils/decomposedfs/node/node.go:437 +0x5f\ngithub.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs/node.ReadNode({0x4520868?, 0xc001354870?}, {0x4528f48, 0xc001692108}, {0xc0003d0d20, 0x24}, {0xc0003d0d20, 0x24}, 0x1, 0x0, ...)\n\tgithub.com/cs3org/reva/[email protected]/pkg/storage/utils/decomposedfs/node/node.go:229 +0x1f6\ngithub.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs.(*Decomposedfs).CreateStorageSpace(0xc000566240, {0x4520868?, 0xc001354840?}, 0xc000df88a0)\n\tgithub.com/cs3org/reva/[email protected]/pkg/storage/utils/decomposedfs/spaces.go:91 +0x497\ngithub.com/cs3org/reva/v2/internal/grpc/services/storageprovider.(*service).CreateStorageSpace(0xc00168e2c0, {0x4520868, 0xc001354840}, 0xc000df88a0)\n\tgithub.com/cs3org/reva/[email protected]/internal/grpc/services/storageprovider/storageprovider.go:480 +0x49\ngithub.com/cs3org/go-cs3apis/cs3/storage/provider/v1beta1._ProviderAPI_CreateStorageSpace_Handler.func1({0x4520868, 0xc001354840}, {0x436a160?, 0xc000df88a0})\n\tgithub.com/cs3org/[email protected]/cs3/storage/provider/v1beta1/provider_api.pb.go:6454 +0x78\ngo.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1({0x4520868, 0xc001354780}, {0x436a160, 0xc000df88a0}, 0xc001718d60, 0xc000adbc80)\n\tgo.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/[email protected]/interceptor.go:341 +0x437\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x4520868?, 0xc001354780?}, {0x436a160?, 0xc000df88a0?})\n\tgithub.com/grpc-ecosystem/[email protected]/chain.go:25 +0x3a\ngithub.com/cs3org/reva/v2/internal/grpc/interceptors/prometheus.interceptorFromConfig.func1({0x4520868?, 0xc001354780?}, {0x436a160?, 0xc000df88a0?}, 0x3e91120?, 0x3c11201?)\n\tgithub.com/cs3org/reva/[email protected]/internal/grpc/interceptors/prometheus/prometheus.go:72 +0x43\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x4520868?, 0xc001354780?}, {0x436a160?, 0xc000df88a0?})\n\tgithub.com/grpc-ecosystem/[email protected]/chain.go:25 +0x3a\ngithub.com/cs3org/reva/v2/internal/grpc/interceptors/auth.NewUnary.func2({0x4520868, 0xc001354540}, {0x436a160, 0xc000df88a0}, 0xc001718d60, 0xc001718da0)\n\tgithub.com/cs3org/reva/[email protected]/internal/grpc/interceptors/auth/auth.go:160 +0xa06\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x4520868?, 0xc001354540?}, {0x436a160?, 0xc000df88a0?})\n\tgithub.com/grpc-ecosystem/[email protected]/chain.go:25 +0x3a\ngithub.com/grpc-ecosystem/go-grpc-middleware/recovery.UnaryServerInterceptor.func1({0x4520868?, 0xc001354540?}, {0x436a160?, 0xc000df88a0?}, 0xc000df5228?, 0x4965b7?)\n\tgithub.com/grpc-ecosystem/[email protected]/recovery/interceptors.go:33 +0xc4\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x4520868?, 0xc001354540?}, {0x436a160?, 0xc000df88a0?})\n\tgithub.com/grpc-ecosystem/[email protected]/chain.go:25 +0x3a\ngithub.com/cs3org/reva/v2/internal/grpc/interceptors/log.NewUnary.func1({0x4520868, 0xc001354540}, {0x436a160, 0xc000df88a0}, 0xc001718d60, 0xc001718de0)\n\tgithub.com/cs3org/reva/[email protected]/internal/grpc/interceptors/log/log.go:39 +0xa2\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x4520868?, 0xc001354540?}, {0x436a160?, 0xc000df88a0?})\n\tgithub.com/grpc-ecosystem/[email protected]/chain.go:25 +0x3a\ngithub.com/cs3org/reva/v2/internal/grpc/interceptors/useragent.NewUnary.func1({0x4520868, 0xc001354540}, {0x436a160, 0xc000df88a0}, 0x1?, 0xc001718e00)\n\tgithub.com/cs3org/reva/[email protected]/internal/grpc/interceptors/useragent/useragent.go:38 +0xf4\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x4520868?, 0xc001354540?}, {0x436a160?, 0xc000df88a0?})\n\tgithub.com/grpc-ecosystem/[email protected]/chain.go:25 +0x3a\ngithub.com/cs3org/reva/v2/internal/grpc/interceptors/token.NewUnary.func1({0x4520868, 0xc0013544b0}, {0x436a160, 0xc000df88a0}, 0x60643a0?, 0xc001718e20)\n\tgithub.com/cs3org/reva/[email protected]/internal/grpc/interceptors/token/token.go:44 +0x16d\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x4520868?, 0xc0013544b0?}, {0x436a160?, 0xc000df88a0?})\n\tgithub.com/grpc-ecosystem/[email protected]/chain.go:25 +0x3a\ngithub.com/cs3org/reva/v2/internal/grpc/interceptors/appctx.NewUnary.func1({0x4520868, 0xc000dfacc0}, {0x436a160, 0xc000df88a0}, 0x4096460?, 0xc001718e40)\n\tgithub.com/cs3org/reva/[email protected]/internal/grpc/interceptors/appctx/appctx.go:51 +0x73c\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x4520868?, 0xc000dfacc0?}, {0x436a160?, 0xc000df88a0?})\n\tgithub.com/grpc-ecosystem/[email protected]/chain.go:25 +0x3a\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1({0x4520868, 0xc000dfacc0}, {0x436a160, 0xc000df88a0}, 0xc0004b9a58?, 0x3fc5600?)\n\tgithub.com/grpc-ecosystem/[email protected]/chain.go:34 +0xbe\ngithub.com/cs3org/go-cs3apis/cs3/storage/provider/v1beta1._ProviderAPI_CreateStorageSpace_Handler({0x445b6a0?, 0xc00168e2c0}, {0x4520868, 0xc000dfacc0}, 0xc000a9ec40, 0xc000547230)\n\tgithub.com/cs3org/[email protected]/cs3/storage/provider/v1beta1/provider_api.pb.go:6456 +0x138\ngoogle.golang.org/grpc.(*Server).processUnaryRPC(0xc0016da000, {0x4533780, 0xc0004f09c0}, 0xc0004e3440, 0xc000547500, 0x5dcc550, 0x0)\n\tgoogle.golang.org/[email protected]/server.go:1337 +0xdf3\ngoogle.golang.org/grpc.(*Server).handleStream(0xc0016da000, {0x4533780, 0xc0004f09c0}, 0xc0004e3440, 0x0)\n\tgoogle.golang.org/[email protected]/server.go:1714 +0xa36\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1()\n\tgoogle.golang.org/[email protected]/server.go:959 +0x98\ncreated by google.golang.org/grpc.(*Server).serveStreams.func1\n\tgoogle.golang.org/[email protected]/server.go:957 +0x18c\n"}
storagesystem-5f47ddfb78-d2t8j storagesystem {"level":"error","service":"storage-system","pkg":"rgrpc","traceid":"00000000000000000000000000000000","user-agent":"grpc-go/1.56.2","from":"tcp://10.244.3.76:34974","uri":"/cs3.storage.provider.v1beta1.ProviderAPI/CreateStorageSpace","start":"16/Aug/2023:14:03:55 +0000","end":"16/Aug/2023:14:03:55 +0000","time_ns":17433429,"code":"Internal","time":"2023-08-16T14:03:55.26912113Z","line":"github.com/cs3org/reva/[email protected]/internal/grpc/interceptors/log/log.go:69","message":"rpc error: code = Internal desc = runtime error: slice bounds out of range [:1] with capacity 0"}

Setup

oCIS Helm Chart from owncloud/ocis-charts#378 with oCIS 3.1.0-rc.1

Additional context

Add any other context about the problem here.
@wkloucek
Copy link
Contributor Author

Also I wonder why we see:

{"level":"warn","service":"storage-system","pkg":"rhttp","time":"2023-08-16T13:59:28.194725607Z","line":"github.com/cs3org/reva/[email protected]/internal/http/services/dataprovider/dataprovider.go:87","message":"missing or incomplete nats configuration. Events will not be published."}

@wkloucek
Copy link
Contributor Author

This is the ENV section of that container:

        env:
        - name: MICRO_REGISTRY
          value: nats
        - name: MICRO_REGISTRY_ADDRESS
          value: nats.ocis-nats.svc.cluster.local:4222
        - name: STORAGE_SYSTEM_LOG_COLOR
          value: "false"
        - name: STORAGE_SYSTEM_LOG_LEVEL
          value: info
        - name: STORAGE_SYSTEM_LOG_PRETTY
          value: "false"
        - name: STORAGE_SYSTEM_TRACING_ENABLED
          value: "false"
        - name: STORAGE_SYSTEM_TRACING_TYPE
          value: jaeger
        - name: STORAGE_SYSTEM_TRACING_ENDPOINT
        - name: STORAGE_SYSTEM_TRACING_COLLECTOR
        - name: STORAGE_SYSTEM_DEBUG_PPROF
          value: "false"
        - name: STORAGE_SYSTEM_GRPC_ADDR
          value: 0.0.0.0:9215
        - name: STORAGE_SYSTEM_DEBUG_ADDR
          value: 0.0.0.0:9217
        - name: STORAGE_SYSTEM_HTTP_ADDR
          value: 0.0.0.0:9216
        - name: STORAGE_SYSTEM_DATA_SERVER_URL
          value: http://storagesystem:9216/data
        - name: STORAGE_SYSTEM_DRIVER
          value: ocis
        - name: STORAGE_SYSTEM_OCIS_METADATA_BACKEND
          value: messagepack
        - name: STORAGE_SYSTEM_CACHE_STORE
          value: nats-js
        - name: STORAGE_SYSTEM_CACHE_STORE_NODES
          value: nats.ocis-nats.svc.cluster.local:4222
        - name: STORAGE_SYSTEM_JWT_SECRET
          valueFrom:
            secretKeyRef:
              key: storage-system-jwt-secret
              name: storage-system-jwt-secret
        - name: OCIS_SYSTEM_USER_API_KEY
          valueFrom:
            secretKeyRef:
              key: api-key
              name: storage-system
        - name: OCIS_SYSTEM_USER_ID
          valueFrom:
            secretKeyRef:
              key: user-id
              name: storage-system

@wkloucek
Copy link
Contributor Author

Related: micro/plugins#121

@dj4oC
Copy link
Contributor

dj4oC commented Aug 21, 2023

@tbsbdr please schedule.

@tbsbdr tbsbdr added the Priority:p3-medium Normal priority label Aug 21, 2023
@wkloucek wkloucek mentioned this issue Aug 16, 2023
Closed
11 tasks
@wkloucek
Copy link
Contributor Author

oCIS 4.0.0-rc.1 is also affected

@micbar
Copy link
Contributor

micbar commented Sep 1, 2023

@wkloucek @kobergj We need to get that actionable.

What exactly needs to be done?

@wkloucek
Copy link
Contributor Author

wkloucek commented Sep 1, 2023

@wkloucek @kobergj We need to get that actionable.

What exactly needs to be done?

To me the nats-js cache / store / registry appear like they should be production ready because they are listed in the offical documentation on doc.owncloud.com.

Eg. the nats-js cache is not working. It don't know why exactly. But it should be working and production ready or discontinued.

@wkloucek
Copy link
Contributor Author

wkloucek commented Sep 7, 2023

please check also #7119 (comment) for NATS stream memory storage (benchmark and settings implications)

@wkloucek wkloucek mentioned this issue Sep 12, 2023
Closed
1 task
@wkloucek
Copy link
Contributor Author

will be supersed if #7272 is implemented

@kobergj kobergj mentioned this issue Nov 23, 2023
Merged
@kobergj kobergj moved this from Qualification to In progress in Infinite Scale Team Board Nov 23, 2023
@kobergj kobergj self-assigned this Nov 29, 2023
@github-project-automation github-project-automation bot moved this from In progress to Done in Infinite Scale Team Board Dec 14, 2023
@micbar micbar added this to the Release 5.0.0 milestone Jan 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@kobergj kobergj

Labels
Priority:p3-medium Normal priority Type:Bug
Projects
Infinite Scale Team Board
Archived in project
Milestone
Release 5.0.0
Development

Successfully merging a pull request may close this issue.

[full-ci] Use nats-js cache kobergj/ocis
5 participants
@micbar @kobergj @tbsbdr @wkloucek @dj4oC