Error no org id on endpoint https://xxxx/api/prom/push (ERROR 401)

[Serrvosky]

Hello everyone.

I'm trying to implement Cortex in a k8s cluster. I already set up all the components, but my prometheus is giving me error logs:

level=warn ts=2019-06-27T17:04:33.26167548Z caller=queue_manager.go:500 component=remote msg="Error sending samples to remote storage" count=100 err="server returned HTTP status 401 Unauthorized: no org id

This is my distributor log:

level=debug ts=2019-06-27T17:07:06.793307679Z caller=logging.go:44 traceID=7c6a25b2679d80bb msg="POST /api/prom/push (401) 56.07µs"

How can I resolve this issue? There are some Prometheus config missing? Or do I have to register my Prometheus deploy on distribute component? And if yes, how I do it?

If it helps this is my scenario:

Tanks for your help

[csmarchbanks]

Hello,

By default, Cortex requires you to set a tenant via the "X-Scope-OrgID" header. In our example k8s manifests, we use an nginx frontdoor to do this: https://github.com/cortexproject/cortex/blob/master/k8s/nginx-config.yaml#L28. I think Traefik is able to do something similar according to the docs.

If you do not care about what the tenant value is you can also set a flag for -auth.enabled=false which will add "fake" as the tenant everywhere.

[Serrvosky]

Hello @csmarchbanks,

Thanks for that tip. It works. But to implement a better solution latter, I can I send that OrgID?

However, now I have a new problem in my ingester. This is the log that I get:

level=error ts=2019-06-28T15:07:03.209285352Z caller=flush.go:234 org_id=0 msg="failed to flush user" err="ResourceNotFoundException: Cannot do operations on a non-existent table\n\tstatus code: 400, request id: 784f3b3a-9bc5-4cb4-a4b1-7a14f3d05ac5"

I think this is about DynamoDB. I'm using dynamodb-dep.yaml. Do I have to create some user?

[csmarchbanks]

But to implement a better solution latter, I can I send that OrgID?

You cannot have Prometheus directly send the appropriate header. You would have to write some sort of service that takes an authorization header or user/password and maps it to the appropriate Header value.

As far as the error goes it looks like the tables in dynamodb are not provisioned. Are you running the table-manager component?

[Serrvosky]

@csmarchbanks Yes I am

[csmarchbanks]

Are there any errors coming from it? Also, what does your storage config look like?

[Serrvosky]

This is my table-manager logs:

level=info ts=2019-06-28T16:20:35.78918298Z caller=cortex.go:196 msg=initialising module=server
level=info ts=2019-06-28T16:20:35.789516014Z caller=server.go:116 http=[::]:80 grpc=[::]:9095 msg="server listening on addresses"
level=info ts=2019-06-28T16:20:35.789793589Z caller=cortex.go:196 msg=initialising module=table-manager
level=info ts=2019-06-28T16:20:35.789943715Z caller=main.go:57 msg="Starting Cortex" version="(version=, branch=, revision=)"
level=info ts=2019-06-28T16:20:35.7913943Z caller=table_manager.go:206 msg="synching tables" num_expected_tables=247 expected_tables=247
level=info ts=2019-06-28T16:20:36.225525205Z caller=table_manager.go:349 msg="creating table" table=cortex_weekly_2458
level=info ts=2019-06-28T16:20:36.46440096Z caller=table_manager.go:349 msg="creating table" table=cortex_weekly_2520
level=info ts=2019-06-28T16:20:36.837354262Z caller=table_manager.go:349 msg="creating table" table=cortex_chunks_2517
level=info ts=2019-06-28T16:20:37.302797276Z caller=table_manager.go:349 msg="creating table" table=cortex_chunks_2541
level=info ts=2019-06-28T16:20:37.818558692Z caller=table_manager.go:349 msg="creating table" table=cortex_chunks_2507
level=info ts=2019-06-28T16:20:38.30979401Z caller=table_manager.go:349 msg="creating table" table=cortex_weekly_2477
....
level=info ts=2019-06-28T16:22:36.33709484Z caller=table_manager.go:349 msg="creating table" table=cortex_chunks_2519
level=info ts=2019-06-28T16:22:36.867632921Z caller=table_manager.go:349 msg="creating table" table=cortex_chunks_2531
level=info ts=2019-06-28T16:22:37.337944776Z caller=table_manager.go:349 msg="creating table" table=cortex_weekly_2491
level=info ts=2019-06-28T16:22:37.840159474Z caller=table_manager.go:349 msg="creating table" table=cortex_weekly_2525
level=info ts=2019-06-28T16:22:38.32599349Z caller=table_manager.go:349 msg="creating table" table=cortex_weekly_2479
level=info ts=2019-06-28T16:22:38.845804513Z caller=table_manager.go:349 msg="creating table" table=cortex_weekly_2480
level=info ts=2019-06-28T16:22:39.339310062Z caller=table_manager.go:206 msg="synching tables" num_expected_tables=247 expected_tables=247
level=error ts=2019-06-28T16:22:40.844994271Z caller=table_manager.go:174 msg="error syncing tables" err="UnknownOperationException: Tagging is not currently supported in DynamoDB Local.\n\tstatus code: 400, request id: a6ab9ba1-981c-4165-a1de-304838014431"

[Serrvosky]

I'm using all the files config. Didn't change anything