aws s3store

[rodentskie]

Question
Is it possible to separate actual files from .info?

files/
info/

Setup works fine and as expected.

Setup details
Please provide following details, if applicable to your situation:

• Operating System: Ran it on AWS ECS Fargate
• Used tusd version: "github.com/tus/tusd/v2/pkg/handler" package I used on my go code.
• Used tusd data storage: AWS S3
• Used tus client library: uppy

[Acconut]

Yes, you can by configuring different ObjectPrefix and MetadataObjectPrefix values:

tusd/pkg/s3store/s3store.go

Lines 108 to 114 in fb5b620

	// ObjectPrefix is prepended to the name of each S3 object that is created
	// to store uploaded files. It can be used to create a pseudo-directory
	// structure in the bucket, e.g. "path/to/my/uploads".
	ObjectPrefix string
	// MetadataObjectPrefix is prepended to the name of each .info and .part S3
	// object that is created. If it is not set, then ObjectPrefix is used.
	MetadataObjectPrefix string

[rodentskie]

Yes, you can by configuring different ObjectPrefix and MetadataObjectPrefix values:

tusd/pkg/s3store/s3store.go

Lines 108 to 114 in fb5b620

	// ObjectPrefix is prepended to the name of each S3 object that is created
	// to store uploaded files. It can be used to create a pseudo-directory
	// structure in the bucket, e.g. "path/to/my/uploads".
	ObjectPrefix string
	// MetadataObjectPrefix is prepended to the name of each .info and .part S3
	// object that is created. If it is not set, then ObjectPrefix is used.
	MetadataObjectPrefix string

Copy, will try to implement now and update in a while.

[rodentskie]

Works as expected. Here is my full tusd handler.

package tushandler

import (
	"context"
	"fmt"

	env "library/go/env/library/go"

	"github.com/aws/aws-sdk-go-v2/config"
	"github.com/aws/aws-sdk-go-v2/credentials"
	"github.com/aws/aws-sdk-go-v2/service/s3"
	"go.uber.org/zap"

	logger "library/go/logger/library/go"

	tusd "github.com/tus/tusd/v2/pkg/handler"
	"github.com/tus/tusd/v2/pkg/s3store"
)

func TusdHandler() *tusd.Handler {
	l := logger.LoggerConfig()
	zapLog, _ := l.Build()

	defer zapLog.Sync()

	region := env.GetEnv("REGION", "xxx")
	bucket := env.GetEnv("BUCKET", "xxx")
	accessKey := env.GetEnv("ACCESS_KEY_ID", "")
	secretKey := env.GetEnv("SECRET_ACCESS_KEY", "")

	cfg, err := config.LoadDefaultConfig(context.TODO(),
		config.WithRegion(region),
		config.WithCredentialsProvider(credentials.NewStaticCredentialsProvider(
			accessKey,
			secretKey,
			"", // Session token (optional)
		)),
	)

	if err != nil {
		zapLog.Fatal("error load aws creds",
			zap.Error(err),
		)
	}

	client := s3.NewFromConfig(cfg)
	store := s3store.New(bucket, client)
	composer := tusd.NewStoreComposer()

	store.ObjectPrefix = "files/"
	store.MetadataObjectPrefix = "info/"

	store.UseIn(composer)
	handler, err := tusd.NewHandler(tusd.Config{
		BasePath:                "/uploads/",
		StoreComposer:           composer,
		RespectForwardedHeaders: true,
	})
	if err != nil {
		panic(fmt.Errorf("unable to create handler: %s", err))
	}

	return handler
}

Next question would be the Logger of the tusd.NewHandler. Is it possible to use zap on this one?

[Acconut]

Next question would be the Logger of the tusd.NewHandler. Is it possible to use zap on this one?

I don't use zap, but if it implements the slog.Logger interface, then yes. If not, you have to use or implement an adapter.

[rodentskie]

Hi @Acconut , I have a new feature to add.

The use-case is.

1. When we parse the JWT, there's a dynamic UUID as payload in there.
2. Same code above, the store.ObjectPrefix = "files/", I wanna make it dynamic and get the uuid from the request header. Like into store.ObjectPrefix = "files/(uuid here)".

I'm blocked on how to get the request header into the function handler above.
I did try also tusd.NewUnroutedHandler but to no avail.
Any tip will be a big help, if it is not doable hearing it from you is totally fine also.

[Acconut]

You can use the PreUploadCallback in the Config struct to customize the upload ID. In the callback, the request information is also available.

[rodentskie]

Thanks, maybe last concern is how the random ID was generated. I mean, I wanna retain it.

So it is like this.

files/(uuid folder)/(filename random ID from tus)

Is there a function to generate such ID?

PreUploadCreateCallback: func(hook tusd.HookEvent) (tusd.HTTPResponse, tusd.FileInfoChanges, error) {
	fmt.Println("==================================================")
	fmt.Println(hook.HTTPRequest.Header)
	fmt.Println("**************************************************")
	fmt.Println(hook.Upload.ID)

	return tusd.HTTPResponse{}, tusd.FileInfoChanges{
		ID: "dev",
	}, nil
},

The hook.Upload.ID is empty.

[Acconut]

As mentioned in https://tus.github.io/tusd/advanced-topics/hooks/#hook-requests-and-responses, the upload ID is null for the pre-create hook because tusd didn't generate one at this stage yet. You can just generate a random ID on your own. Tusd's default method for ID generation is not publicly exported but you can copy it if you want to: https://github.com/tus/tusd/blob/main/internal/uid/uid.go