Getting an error cannot create stream multipart_upload_metadata

[Dkairu]

I am trying to test the s3 output but getting this error cannot create stream multipart_upload_metadata

The command I am running is ./fluent-bit -i tail -p path=/var/log/apport.log -o s3 -p bucket=mybucket -p region=us-east-1 -v

Running this on an ubuntu ec2.

[2021/05/21 21:39:45] [debug] [fstore] [cio scan] opening path /tmp/fluent-bit/s3/mybucket
[2021/05/21 21:39:45] [debug] [fstore] created stream path /tmp/fluent-bit/s3/mybucket/2021-05-21T21:39:45
[2021/05/21 21:39:45] [debug] [fstore] [cio stream] new stream registered: 2021-05-21T21:39:45
[2021/05/21 21:39:45] [error] [fstore] cannot create stream multipart_upload_metadata
[2021/05/21 21:39:45] [error] [output:s3:s3.0] could not initialize multipart_upload stream
[2021/05/21 21:39:45] [debug] [fstore] [cio stream] delete stream path: /tmp/fluent-bit/s3/mybucket/2021-05-21T21:39:45
[2021/05/21 21:39:45] [error] [output:s3:s3.0] Failed to initialize S3 storage: /tmp/fluent-bit/s3
[2021/05/21 21:39:45] [error] [output] Failed to initialize 's3' plugin
[2021/05/21 21:39:45] [error] [lib] backend failed

Of course mybucket=myrealbucketname...

[PettitWesley]

[2021/05/21 21:39:45] [debug] [fstore] created stream path /tmp/fluent-bit/s3/mybucket/2021-05-21T21:39:45

This is the default buffer path that Fluent Bit S3 uses (/tmp/fluent-bit). It creates a directory for multipart upload metadata in that path called multipart_upload_metadata. That is what this error message is about.

You probably have some issue with file permissions which is preventing the buffer directory from being written to.

You can customize it with the store_dir parameter: https://docs.fluentbit.io/manual/pipeline/outputs/s3

[Dkairu]

Ok thank you @PettitWesley I will test that out. I also have a different issue to do with credentials, I didn't create a new issue but instead posted on an old issue. If you can please check this out for me, I would I appreciate it. #103

[matthewfala]

@Dkairu, another easy solution may be to just manually create the /tmp/fluent-bit/s3 folder on your instance. I'm closing this issue for now. Please open it if you are still facing problems regarding initialization.

Please take a look at the comment regarding your credentials issue: #258