PutObjectOperationDeserializerKt.throwPutObjectError caused by S3Exception - The provided token has expired

[yaroslav-v]

Before opening, please confirm:

• I have searched for duplicate or closed issues and discussions.

Language and Async Model

Java

Amplify Categories

Authentication, Storage

Gradle script dependencies

// Put output below this line
implementation "com.amplifyframework:core:2.14.5"
implementation "com.amplifyframework:aws-storage-s3:2.14.5"
implementation "com.amplifyframework:aws-auth-cognito:2.14.5"

Environment information

# Put output below this line
------------------------------------------------------------
Gradle 8.1.1
------------------------------------------------------------

Build time:   2023-04-21 12:31:26 UTC
Revision:     1cf537a851c635c364a4214885f8b9798051175b

Kotlin:       1.8.10
Groovy:       3.0.15
Ant:          Apache Ant(TM) version 1.10.11 compiled on July 10 2021
JVM:          1.8.0_201 (Oracle Corporation 25.201-b09)
OS:           Mac OS X 10.16 x86_64

Please include any relevant guides or documentation you're referencing

https://docs.amplify.aws/android/sdk/auth/

Describe the bug

Hi there!

A part of our users experience issues with uploading images to AWS. The problem has appeared after updating from v1.38.8 to v2.14.5. The issue appears on all Android version from 9 to 14, affected devices include Samsung, Motorola, Oneplus, Google etc.

As far as I can see, this issue appears randomly on part of the devices only. I've tried to reproduce the issue locally but couldn't find exact steps.

You can find a crashlog from Firebase in the attachment. I've checked Firebase reports for the previous versions as well (before the update) and there were no such cases.

The crash itself is caused either by
Caused by aws.sdk.kotlin.services.s3.model.S3Exception
The provided token has expired.

or by
Caused by aws.sdk.kotlin.services.s3.model.S3Exception
The provided token is malformed or otherwise invalid.

As far as I understand from the docs and from my previous experience, we don't need to refresh tokens anyhow in this case - "For Guest scenarios they will be automatically refreshed."

Reproduction steps (if applicable)

Code Snippet

// Initialization in Application class

Amplify.Logging.disable();

Amplify.addPlugin(new AWSCognitoAuthPlugin());
Amplify.addPlugin(new AWSS3StoragePlugin());

AmplifyConfiguration amplifyConfig = AmplifyConfiguration.builder(context)
		.devMenuEnabled(false)
		.build();
Amplify.configure(amplifyConfig, context);


// Uploading, somewhere in the app

DateFormat dateFormat = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ssZ", Locale.US);
dateFormat.setTimeZone(TimeZone.getTimeZone("UTC"));

Map<String, String> metadata = new ArrayMap<>(4);
metadata.put("date", dateFormat.format(date));
metadata.put("session_uuid", uid);
metadata.put("token", authToken);
metadata.put("service", serviceName);

StorageUploadFileOptions options =
		StorageUploadFileOptions.builder()
				.accessLevel(StorageAccessLevel.PUBLIC)
				.contentType(mimeType)
				.metadata(metadata)
				.build();

final File file = new File(path);
if (file.canRead()) {
	Amplify.Storage.uploadFile(
			file.getName(),
			file,
			options,
			result -> {
				// write success logs
			},
			failure -> {
				// write failure logs
			}
	);
}

Log output

// Put your logs below this line
Non-fatal Exception: com.amplifyframework.storage.StorageException
Something went wrong with your AWS S3 Storage upload file operation
	com.amplifyframework.storage.s3.operation.AWSS3StorageUploadFileOperation$UploadTransferListener.onError (AWSS3StorageUploadFileOperation.kt:228)
	com.amplifyframework.storage.s3.transfer.TransferStatusUpdater.updateOnError$lambda$11$lambda$10 (TransferStatusUpdater.kt:160)
	android.os.Handler.handleCallback (Handler.java:978)
	android.os.Handler.dispatchMessage (Handler.java:104)
	android.os.Looper.loopOnce (Looper.java:238)
	android.os.Looper.loop (Looper.java:357)
	android.app.ActivityThread.main (ActivityThread.java:8098)
	java.lang.reflect.Method.invoke (Method.java)
	com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run (RuntimeInit.java:548)
	com.android.internal.os.ZygoteInit.main (ZygoteInit.java:1026)


Caused by java.lang.Exception
aws.sdk.kotlin.services.s3.model.S3Exception: The provided token has expired.
	com.amplifyframework.storage.s3.transfer.worker.BaseTransferWorker.doWork$suspendImpl (BaseTransferWorker.kt:121)
	com.amplifyframework.storage.s3.transfer.worker.BaseTransferWorker$doWork$1.invokeSuspend (BaseTransferWorker.kt:11)
	kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith (ContinuationImpl.kt:33)
	kotlinx.coroutines.UndispatchedCoroutine.afterResume (CoroutineContext.kt:270)
	kotlinx.coroutines.AbstractCoroutine.resumeWith (AbstractCoroutine.kt:102)
	kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith (ContinuationImpl.kt:46)
	kotlinx.coroutines.UndispatchedCoroutine.afterResume (CoroutineContext.kt:270)
	kotlinx.coroutines.AbstractCoroutine.resumeWith (AbstractCoroutine.kt:102)
	kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith (ContinuationImpl.kt:46)
	kotlinx.coroutines.DispatchedTask.run (DispatchedTask.kt:108)
	kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely (CoroutineScheduler.kt:584)
	kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask (CoroutineScheduler.kt:793)
	kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker (CoroutineScheduler.kt:697)
	kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run (CoroutineScheduler.kt:684)


Caused by aws.sdk.kotlin.services.s3.model.S3Exception
The provided token has expired.
	aws.sdk.kotlin.services.s3.serde.PutObjectOperationDeserializerKt.throwPutObjectError (PutObjectOperationDeserializer.kt:69)
	aws.sdk.kotlin.services.s3.serde.PutObjectOperationDeserializerKt.access$throwPutObjectError (PutObjectOperationDeserializer.kt:1)
	aws.sdk.kotlin.services.s3.serde.PutObjectOperationDeserializerKt$throwPutObjectError$1.invokeSuspend (PutObjectOperationDeserializer.kt:10)
	kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith (ContinuationImpl.kt:33)
	kotlinx.coroutines.DispatchedTask.run (DispatchedTask.kt:108)
	kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely (CoroutineScheduler.kt:584)
	kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask (CoroutineScheduler.kt:793)
	kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker (CoroutineScheduler.kt:697)
	kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run (CoroutineScheduler.kt:684)

amplifyconfiguration.json

No response

GraphQL Schema

// Put your schema below this line

Additional information and screenshots

No response

[gpanshu]

Can you paste your gradle dependencies here please? My suspicion is that you are trying to use both AWS SDK for Android and Amplify V2. While V1 was dependent on AWS SDK for Android there is no such dependency for Amplify V2.

To add to that you definitely do need to refresh tokens even in guest scenarios, if you do a fetchAuthSession it should automatically do that for you.

[yaroslav-v]

Hi! The dependencies are available in the original post. There are only:

implementation "com.amplifyframework:core:2.14.5"
implementation "com.amplifyframework:aws-storage-s3:2.14.5"
implementation "com.amplifyframework:aws-auth-cognito:2.14.5"

The same dependencies were used with the previous v1.38.8. There were no code changes, no dependencies changes - only the version was updated.

Apart from that, we have some lines in the project build.gradle, but I don't think they make a difference in this case:

buildscript {
    ...
    dependencies {
            classpath 'com.amplifyframework:amplify-tools-gradle-plugin:1.0.1'
            ...
    }
}
...
apply plugin: 'com.amplifyframework.amplifytools'

Btw I did some more tests and currently I can say for sure that the problem appears only if you update the app from the version with AWS Amplify v1.38.8 to the new build with AWS Amplify v2.14.5. If I try to install a fresh new copy with AWS Amplify v2.14.5 all works as expected.

As far as I can see, the problem doesn't appear in a tick of time after updating the app, but after some period. I'm not sure about the exact length, but looks like it's something about 1h.

Anyway, a fresh new installation doesn't have the described issue and works well without any changes for longer periods of time. I've tried it for a couple of days and there were no such problems.

[gpanshu]

Yes that makes sense as the refresh token is expired in that case. Can you try one more thing and advise what happens in the following scenario:

1. You update your app from 1.38.8 to the one using 2.14.5
2. You do a fetchAuthSession before you do your operation
3. Add the logging plugin as the first plugin in your list : Amplify.addPlugin(AndroidLoggingPlugin(LogLevel.VERBOSE))
4. Paste the logs here

There could be an issue with migration and I would like to delve deeper to help you in this.

[yaroslav-v]

Thx for the answer.

I've added both fetchAuthSession and AndroidLoggingPlugin - the result is the same as previously. As far as I can see, fetchAuthSession receives credentials without any issues, nevertheless all uploads fail to finish.

Here is the log:

2023-12-16 13:13:48.904  8993-11324 APP          app.package  I  IdentityId: us-east-1:fa2f987b...

2023-12-16 13:13:49.005  8993-9152  amplify:aw...TransferDB app.package  I  update state for 138 to IN_PROGRESS

2023-12-16 13:13:22.814  8993-10940 amplify:aw...loadWorker app.package  E  SinglePartUploadWorker failed with exception: aws.sdk.kotlin.services.s3.model.S3Exception: The provided token has expired.
                                                                                        	at aws.sdk.kotlin.services.s3.serde.PutObjectOperationDeserializerKt.b(SourceFile:159)
                                                                                        	at aws.sdk.kotlin.services.s3.serde.PutObjectOperationDeserializerKt.a(SourceFile:1)
                                                                                        	at aws.sdk.kotlin.services.s3.serde.PutObjectOperationDeserializerKt$throwPutObjectError$1.invokeSuspend(Unknown Source:10)
                                                                                        	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(SourceFile:12)
                                                                                        	at kotlinx.coroutines.DispatchedTask.run(SourceFile:124)
                                                                                        	at kotlinx.coroutines.scheduling.CoroutineScheduler.s(SourceFile:1)
                                                                                        	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.d(SourceFile:15)
                                                                                        	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.p(SourceFile:29)
                                                                                        	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(SourceFile:1)
                                                                                        	
                                                                                        	
2023-12-16 13:13:50.949  8993-10940 amplify:aw...TransferDB app.package  I  update state for 138 to FAILED

2023-12-16 13:13:22.822  8993-8993                          app.package  E  AWS capture failed
                                                                                        StorageException{message=Something went wrong with your AWS S3 Storage upload file operation, cause=java.lang.Exception: aws.sdk.kotlin.services.s3.model.S3Exception: The provided token has expired., recoverySuggestion=See attached exception for more information and suggestions}
                                                                                        	at com.amplifyframework.storage.s3.operation.AWSS3StorageUploadFileOperation$UploadTransferListener.onError(SourceFile:33)
                                                                                        	at com.amplifyframework.storage.s3.transfer.TransferStatusUpdater.updateOnError$lambda$11$lambda$10(SourceFile:11)
                                                                                        	at com.amplifyframework.storage.s3.transfer.TransferStatusUpdater.c(SourceFile:1)
                                                                                        	at com.amplifyframework.storage.s3.transfer.g.run(SourceFile:1)
                                                                                        	at android.os.Handler.handleCallback(Handler.java:942)
                                                                                        	at android.os.Handler.dispatchMessage(Handler.java:99)
                                                                                        	at android.os.Looper.loopOnce(Looper.java:201)
                                                                                        	at android.os.Looper.loop(Looper.java:288)
                                                                                        	at android.app.ActivityThread.main(ActivityThread.java:7872)
                                                                                        	at java.lang.reflect.Method.invoke(Native Method)
                                                                                        	at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:548)
                                                                                        	at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:936)
                                                                                        Caused by: java.lang.Exception: aws.sdk.kotlin.services.s3.model.S3Exception: The provided token has expired.
                                                                                        	at com.amplifyframework.storage.s3.transfer.worker.BaseTransferWorker.doWork$suspendImpl(SourceFile:386)
                                                                                        	at com.amplifyframework.storage.s3.transfer.worker.BaseTransferWorker$doWork$1.invokeSuspend(Unknown Source:11)
                                                                                        	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(SourceFile:12)
                                                                                        	at kotlinx.coroutines.UndispatchedCoroutine.b1(SourceFile:60)
                                                                                        	at kotlinx.coroutines.AbstractCoroutine.resumeWith(SourceFile:16)
                                                                                        	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(SourceFile:48)
                                                                                        	at kotlinx.coroutines.UndispatchedCoroutine.b1(SourceFile:60)
                                                                                        	at kotlinx.coroutines.AbstractCoroutine.resumeWith(SourceFile:16)
                                                                                        	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(SourceFile:48)
                                                                                        	at kotlinx.coroutines.DispatchedTask.run(SourceFile:124)
                                                                                        	at kotlinx.coroutines.scheduling.CoroutineScheduler.s(SourceFile:1)
                                                                                        	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.d(SourceFile:15)
                                                                                        	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.p(SourceFile:29)
                                                                                        	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(SourceFile:1)
                                                                                        Caused by: aws.sdk.kotlin.services.s3.model.S3Exception: The provided token has expired.
                                                                                        	at aws.sdk.kotlin.services.s3.serde.PutObjectOperationDeserializerKt.b(SourceFile:159)
                                                                                        	at aws.sdk.kotlin.services.s3.serde.PutObjectOperationDeserializerKt.a(SourceFile:1)
                                                                                        	at aws.sdk.kotlin.services.s3.serde.PutObjectOperationDeserializerKt$throwPutObjectError$1.invokeSuspend(Unknown Source:10)
                                                                                        	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(SourceFile:12)
                                                                                        	at kotlinx.coroutines.DispatchedTask.run(SourceFile:124) 
                                                                                        	at kotlinx.coroutines.scheduling.CoroutineScheduler.s(SourceFile:1) 
                                                                                        	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.d(SourceFile:15) 
                                                                                        	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.p(SourceFile:29) 
                                                                                        	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(SourceFile:1) 
                                                                                        	
                                                                                        	
023-12-16 13:13:50.965  8993-9137  WM-WorkerWrapper        app.package  I  Worker result FAILURE for Work [ id=41e4ff8f-61d9-4172-997c-48aed81831b0, tags={ com.amplifyframework.storage.s3.transfer.worker.RouterWorker, awsS3StoragePlugin, 138, UPLOAD } ]

[gpanshu]

I am going to locally try to reproduce this and get back to you.

[gpanshu]

One question I had forgotten to ask before: is your device tracking enabled in your user pool ?

[yaroslav-v]

Hmm... I'm not sure about this. As far as I know, this setting should be set by default.

[gpanshu]

Gotcha. By default device tracking is not enabled but whenever the backend was configured if it was selected then it will be enabled. What happens in your upgrade if you sign out with forcesignout and then login ? Is the problem resolved ?

[yaroslav-v]

Well, we're using guest accounts at this stage, as I wrote earlier. I'm not sure that it's possible to do "forcesignout and then login" in such case. If it isn't correct, give me more details, please.

You can see the code snippet in the original message.

[yaroslav-v]

Btw I want to bring your attention to the fact that the error itself is caused either by
Caused by aws.sdk.kotlin.services.s3.model.S3Exception
The provided token has expired.

or by
Caused by aws.sdk.kotlin.services.s3.model.S3Exception
The provided token is malformed or otherwise invalid.

I can't reproduce the second cause, but I see it in our Firebase logs in quite big numbers. Probably, it's not just expiration issue rather something with the token handling.

[gpanshu]

Yes you can do a forcesignout and then a fetchauthsession which will get you fresh guest credentials. The force signout will clean out all existing tokens.

[yaroslav-v]

I couldn't find forcesignout, so I've used normal Amplify.Auth.signOut method. Looks like it works and locally I can't reproduce the issue anymore.

We'll publish an update for the app soon. I'll be able to give more details after that.

P.S. I'm not sure if it's related or not, but I've noticed this warning message in the logs after updating the app:

AndroidKeysetManager    app.package  W  keyset not found, will generate a new one
                                                                                        java.io.FileNotFoundException: can't read keyset; the pref value __androidx_security_crypto_encrypted_prefs_value_keyset__ does not exist
                                                                                        	at com.google.crypto.tink.integration.android.SharedPrefKeysetReader.b(SourceFile:33)
                                                                                        	at com.google.crypto.tink.integration.android.SharedPrefKeysetReader.a(SourceFile:1)
                                                                                        	at com.google.crypto.tink.KeysetHandle.j(SourceFile:1)
                                                                                        	at com.google.crypto.tink.integration.android.AndroidKeysetManager$Builder.e(SourceFile:7)
                                                                                        	at com.google.crypto.tink.integration.android.AndroidKeysetManager$Builder.f(SourceFile:1)
                                                                                        	at com.google.crypto.tink.integration.android.AndroidKeysetManager$Builder.d(SourceFile:12)
                                                                                        	at androidx.security.crypto.EncryptedSharedPreferences.a(SourceFile:93)
                                                                                        	at com.amplifyframework.core.store.EncryptedKeyValueRepository$sharedPreferences$2.invoke(SourceFile:7)
                                                                                        	at com.amplifyframework.core.store.EncryptedKeyValueRepository$sharedPreferences$2.invoke(SourceFile:1)
                                                                                        	at kotlin.SynchronizedLazyImpl.getValue(SourceFile:21)
                                                                                        	at com.amplifyframework.core.store.EncryptedKeyValueRepository.getSharedPreferences$com_amplifyframework_core_release(SourceFile:3)
                                                                                        	at com.amplifyframework.core.store.EncryptedKeyValueRepository$editor$2.invoke(SourceFile:2)
                                                                                        	at com.amplifyframework.core.store.EncryptedKeyValueRepository$editor$2.invoke(SourceFile:1)
                                                                                        	at kotlin.SynchronizedLazyImpl.getValue(SourceFile:21)
                                                                                        	at com.amplifyframework.core.store.EncryptedKeyValueRepository.getEditor$com_amplifyframework_core_release(SourceFile:3)
                                                                                        	at com.amplifyframework.core.store.EncryptedKeyValueRepository.put(SourceFile:6)
                                                                                        	at com.amplifyframework.auth.cognito.data.AWSCognitoAuthCredentialStore.saveCredential(SourceFile:18)
                                                                                        	at com.amplifyframework.auth.cognito.actions.CredentialStoreCognitoActions$migrateLegacyCredentialStoreAction$$inlined$invoke$1.execute(SourceFile:58)
                                                                                        	at com.amplifyframework.statemachine.ConcurrentEffectExecutor$execute$1$1.invokeSuspend(SourceFile:35)
                                                                                        	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(SourceFile:12)
                                                                                        	at kotlinx.coroutines.DispatchedTask.run(SourceFile:124)
                                                                                        	at kotlinx.coroutines.scheduling.CoroutineScheduler.s(SourceFile:1)
                                                                                        	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.d(SourceFile:15)
                                                                                        	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.p(SourceFile:29)
                                                                                        	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(SourceFile:1)

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[tjleing]

I saw @gpanshu closed this issue, but by the way @yaroslav-v the warning you mentioned is expected as we create an EncryptedSharedPreferences instance. If the issue persists please create a new issue.

[yaroslav-v]

FYI I've been tracking the state of this issue for a while in Firebase and see than the number of reports has fallen significantly after the fix was applied in the project. However, there are still some reports appear on a daily basis (using the latest version of the library 2.14.10).

I just want to point out that according to the docs AWS credentials for guest accounts should be refreshed automatically and the issue didn't exist for AWS Amplify v1 🤷‍♂️.

There's the mentioned doc https://docs.amplify.aws/android/sdk/auth/ - "AWS Credentials ... For Guest scenarios they will be automatically refreshed."