Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix stream SSE uploads with S3 encrypt type #960

Merged
merged 4 commits into from
Apr 2, 2018

Conversation

jespino
Copy link
Contributor

@jespino jespino commented Mar 27, 2018

When you try to upload files stream based with encryption activated, minio-go
try to upload the parts using the ServerSideEncryption header, but it isn't
supported for uploadPart with encrypt type S3, only is needed for customer
keys. For S3 encryption we simply add the header at the creation, and in the
following uploadPart we don't pass the header.

I'm not sure, but maybe we need to exclude the encrypt KMS type too (I'm not
sure).

Here is an example of it failing before the patch, and working after it.

package main

import (
	"log"
	"os"

	minio "github.com/minio/minio-go"
	"github.com/minio/minio-go/pkg/encrypt"
)

func main() {
	s3Client, err := minio.New("s3.amazonaws.com", "API_KEY", "API_SECRET", true)
	if err != nil {
		log.Fatalln(err)
	}

	object, err := os.Open("test.txt")
	if err != nil {
		log.Fatalln(err)
	}
	defer object.Close()

	options := minio.PutObjectOptions{ServerSideEncryption: encrypt.NewSSE()}

	n, err := s3Client.PutObject("my-bucket", "my-objectname", object, -1, options)
	if err != nil {
		log.Fatalln(err)
	}

	log.Println("Uploaded", "my-objectname", " of size: ", n, "Successfully.")
}

@kannappanr kannappanr requested review from aead and donatello March 27, 2018 17:11
@aead
Copy link
Member

aead commented Mar 27, 2018

@jespino I think you're also right about KMS - but we should check to be sure...
Thanks for the PR 😄

@jespino
Copy link
Contributor Author

jespino commented Mar 27, 2018

You are welcome, I'm implementing the encrypt.NewSSEKMS to try it ;) If I realize that is needed for KMS too, I will send another PR, if you want :)

@aead
Copy link
Member

aead commented Mar 27, 2018

That would be awesome :) @jespino

func NewSSEKMS(keyId string) ServerSide { return kms{key: keyId, context: nil} }

// NewSSEKMSWithContext returns a new server-side-encryption using SSE-KMS and the provided Key Id and context.
func NewSSEKMSWithContext(keyId string, context *string) ServerSide {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

func parameter keyId should be keyID

@@ -90,6 +95,14 @@ type ServerSide interface {
// Using SSE-S3 the server will encrypt the object with server-managed keys.
func NewSSE() ServerSide { return s3{} }

// NewSSEKMS returns a new server-side-encryption using SSE-KMS and the provided Key Id.
func NewSSEKMS(keyId string) ServerSide { return kms{key: keyId, context: nil} }

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

func parameter keyId should be keyID

@@ -30,6 +30,11 @@ const (
// sseGenericHeader is the AWS SSE header used for SSE-S3 and SSE-KMS.
sseGenericHeader = "X-Amz-Server-Side-Encryption"

// sseKmsKeyId is the AWS SSE-KMS key id.
sseKmsKeyId = sseGenericHeader + "-Aws-Kms-Key-Id"

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

const sseKmsKeyId should be sseKmsKeyID

@jespino
Copy link
Contributor Author

jespino commented Mar 27, 2018

I have created what I think is needed, but I don't know too much about KMS, so I don't know if the context as an optional string is enough.

@jespino jespino force-pushed the fix-stream-sse-uploads branch from 24bcabc to ef97e05 Compare March 27, 2018 17:58
func NewSSEKMS(keyID string) ServerSide { return kms{key: keyID, context: nil} }

// NewSSEKMSWithContext returns a new server-side-encryption using SSE-KMS and the provided Key Id and context.
func NewSSEKMSWithContext(keyID string, context *string) ServerSide {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since::

The encryption context can be any value that you want, provided that the header adheres to the Base64-encoded JSON format.

I think we can follow this approach:

NewSSEKMS(keyID string, context interface{}) (ServerSide, error) {
      // if context is nil -> no context
      // if context implements json.Marshaler -> marshal
      // json.Marshal(context) 
}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like this approach and then we don't need the contextPresent proposed in the comment below. I'll do this changes tomorrow.

@@ -90,6 +95,14 @@ type ServerSide interface {
// Using SSE-S3 the server will encrypt the object with server-managed keys.
func NewSSE() ServerSide { return s3{} }

// NewSSEKMS returns a new server-side-encryption using SSE-KMS and the provided Key Id.
func NewSSEKMS(keyID string) ServerSide { return kms{key: keyID, context: nil} }
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we don't need two functions right now - removing a function breaks API while we can always add if requested.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, perfect, I'll leave one only.


type kms struct {
key string
context *string
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we use a bool var - like contextPresent instead of a *string and nil checks?

}
if serializedContext, err := json.Marshal(context); err != nil {
return nil, err
} else {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if block ends with a return statement, so drop this else and outdent its block (move short variable declaration to its own line if necessary)

@jespino jespino force-pushed the fix-stream-sse-uploads branch 2 times, most recently from 6216efc to 63401ff Compare March 28, 2018 06:54
Copy link
Member

@aead aead left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@deekoder deekoder merged commit 663902a into minio:master Apr 2, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants