[DOC] Missing "ListObjects" permission in the S3 configuration doc

[chrisduong]

Describe the bug

When I deployed the Tempo with this S3 configuration doc https://grafana.com/docs/tempo/latest/configuration/s3/, I faced this error

{"caller": "main.go:105", "err": "failed to init module services error initialising module: usage-report: failed to initialize usage report: unexpected error from ListObjects on tempo-traces-foo: Access Denied","level":"error","msg":"error running Tempo","ts":"2023-05-18T10:09:09.908963824Z"}

Expected behavior

The S3 configuration doc should add s3:ListObjects action into the policy.

Environment:

• Infrastructure: K8s
• Deployment tool: Helm
• Tempo version 2.1.1

[joe-elliott]

Thanks for the report!

cc @knylander-grafana

[absolutemikex]

I am running into this now - however, I added the ListObjects and ListObjectsV2 and I am using IRSA. Is there anything else I can check?

level=info ts=2023-05-19T18:22:40.23471841Z caller=main.go:215 msg="initialising OpenTracing tracer" level=info ts=2023-05-19T18:22:40.243715669Z caller=main.go:102 msg="Starting Tempo" version="(version=2.1.1, branch=HEAD, revision=4157d7620)" level=error ts=2023-05-19T18:22:40.396802855Z caller=main.go:105 msg="error running Tempo" err="failed to init module services error initialising module: store: failed to create store unexpected error from ListObjects on BUCKET: Access Denied"

[absolutemikex]

I am dumb - disregard. I remove my SA config accidentally from values. Appreciate all your hard work.

[Eve832]

Hi @knylander-grafana , can you please update the status of this ticket in your view in https://github.com/orgs/grafana/projects/69/

[github-actions]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days.
The next time this stale check runs, the stale label will be removed if there is new activity. The issue will be closed after 15 days if there is no new activity.
Please apply keepalive label to exempt this Issue.

[r65535]

We're seeing this issue using ceph's S3 object store. I've given the Access/Secret key permission to s3:* but still getting the above error: err="failed to init module services error initialising module: store: failed to create store unexpected error from ListObjects on BUCKET: Access Denied"

I've tried setting forcepathstyle: true and signature_v2: true but neither works 😭

[zalegrala]

Double check the credentials @r65535 and perhaps test manually outside of Tempo to confirm.

[datsabk]

I am facing this issue as well - I am using S3 object storage, with s3:* permissions on the IAM role

[z0rc]

@datsabk if you updated to 2.2.2, most likely it's #2888.

[datsabk]

@z0rc Saved my day man! I had started to doubt myself !

Looks like a regression due to AWS SDK version upgrade. There's nothing much happening in the Tempo Code really.

[ekristen]

This is still an issue, #2888 doesn't seem to fix anything.

[knylander-grafana]

I'm getting ready to update the S3 config docs to address some of this content. Here are some related S3 issues we'll take into consideration for updating the doc:

• S3 IAM access via AWS Kubernetes IRSA does not work with 2.2.2 #2895
• 2.2.2 regression: S3 storage does not work with AWS IRSA authentication anymore #2888
• Storage S3 - AWS node role IMDSv1 does not work after upgrade to v2.2 #2743

To update the docs, we're going to:

• Review the S3 configuration and verify it works
• Add ListObjects

Draft PR: #3164

[bouk]

s3:ListObjects is not a valid permission, s3:ListBucket is the one needed for listing objects in a bucket

[sunrabbit123]

@joe-elliott

s3:ListObjects is not a valid permission, s3:ListBucket is the one needed for listing objects in a bucket

This issue is not closed, and we ask that you reopen it.

In fact, ListObjects does not exist.

aws said.

Actions – For each resource, Amazon S3 supports a set of operations. You identify resource operations that you will allow (or deny) by using action keywords.
For example, the s3:ListBucket permission allows the user to use the Amazon S3 GET Bucket (List Objects) operation. For more information about using Amazon S3 actions, see Amazon S3 actions. For a complete list of Amazon S3 actions, see Actions.

---

And I'm still getting the error that's attached to the issue.
Even though I gave S3 full permissions.

tempo_values.yaml

storage:
  trace:
    backend: s3
    s3:
      secret_key: secret
      access_key: key
      bucket: <bucket-name>
      endpoint: '<bucket-name>.s3.<region>.amazonaws.com'
      insecure: false
      region: <region>

I just excuted next command helm upgrade --values tempo_values.yaml --install tempo grafana/tempo-distributed -n <namespace>

[joe-elliott]

Are you sure that s3:ListObjects isn't a valid action?

If I go to the permissions doc here: https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-with-s3-actions.html

Then click here:

It takes me to this list of s3 actions: https://docs.aws.amazon.com/AmazonS3/latest/API/API_Operations.html

which includes ListObjects.

[bouk]

Confusingly, the action and permissions have different names. If you look at https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html you'll see:

General purpose bucket permissions - To use this operation, you must have READ access to the bucket. You must have permission to perform the s3:ListBucket action.

[joe-elliott]

Ah, thank you 🙏.

I'm going to leave this issue closed since it specifically discusses adding "ListObjects" to the docs. If you are having issues getting auth configured to work with S3 please update to 2.3.1 (we've made some fixes) and open an issue if things are still not working.

[stefanandres]

For what it's worth, I encountered the same error, but it turns out my aws_iam_policy_document was wrong.

The actual error message was from a pod with the same kubernetes serviceAccount:

bash-4.2# aws s3 ls s3://$bucket

An error occurred (AccessDenied) when calling the AssumeRoleWithWebIdentity operation: Not authorized to perform sts:AssumeRoleWithWebIdentity

After fixing the aws_iam_policy_document tempo spawned correctly.