Rome crashes when downloading cache from remote - Zero bytes Archive

[thii]

Enhancement Suggestion / Bug Report

Rome crashes when downloading cache from remote.

Steps which explain the enhancement or reproduce the bug

I'll try to make a sample project that reproduces that crash, here is the crash log:

Error: could not find MyFramework in local cache at : /Users/admin/Library/Caches/Rome/swift-4.1.2-swiftlang-902.0.54-clang-902.0.39.2/MyFramework/iOS/MyFramework.framework-c57d3c0b3211274ffcda
b294a7fd0e5743130839.zip
Downloaded MyFramework from: swift-4.1.2-swiftlang-902.0.54-clang-902.0.39.2/MyFramework/iOS/MyFramework.framework-c57d3c0b3211274ffcdab294a7fd0e5743130839.zip
Copied MyFramework to: /Users/admin/Library/Caches/Rome/swift-4.1.2-swiftlang-902.0.54-clang-902.0.39.2/MyFramework/iOS/MyFramework.framework-c57d3c0b3211274ffcdab294a7fd0e5743130839.zip
rome: Data.Binary.Get.runGet at position 0: not enough bytes
CallStack (from HasCallStack):
  error, called at libraries/binary/src/Data/Binary/Get.hs:342:5 in binary-0.8.3.0:Data.Binary.Get

Current behavior

Rome crashes when downloading cache from remote.

Suggested behavior

Not crash.

Why would the enhancement be useful to most users

Rome version:

0.15.0.43

OS and version:

macOS High Sierra 10.13.4 (17E202)

[tmspzz]

Can you run with -v ? Is it possible you have a 0 byte archive?

From the messages it seems that the download was successful, the next step would be deleting the framework in Carthage/Build/<platform>/MyFramework.framework before copying the new one.

07:58:04 2018-05-30 - Error: could not find Tyro in local cache at : /Users/blender/Library/Caches/Rome/swift_4_0_3-swiftlang-900_0_74_1-clang-900_0_39_2/Tyro/iOS/Tyro.framework-v0.2.0.zip
07:58:04 2018-05-30 - Started downloading Tyro from: swift_4_0_3-swiftlang-900_0_74_1-clang-900_0_39_2/Tyro/iOS/Tyro.framework-v0.2.0.zip
07:58:05 2018-05-30 - Downloaded 1.01 MB of 1.88 MB for Tyro
07:58:06 2018-05-30 - Downloaded 1.88 MB of 1.88 MB for Tyro
07:58:06 2018-05-30 - Downloaded Tyro from: swift_4_0_3-swiftlang-900_0_74_1-clang-900_0_39_2/Tyro/iOS/Tyro.framework-v0.2.0.zip
07:58:06 2018-05-30 - Copying Tyro to: /Users/blender/Library/Caches/Rome/swift_4_0_3-swiftlang-900_0_74_1-clang-900_0_39_2/Tyro/iOS/Tyro.framework-v0.2.0.zip
07:58:06 2018-05-30 - Copied Tyro to: /Users/blender/Library/Caches/Rome/swift_4_0_3-swiftlang-900_0_74_1-clang-900_0_39_2/Tyro/iOS/Tyro.framework-v0.2.0.zip
07:58:06 2018-05-30 - Deleted: Carthage/Build/iOS/Tyro.framework
07:58:06 2018-05-30 - Staring to unzip Tyro.framework-v0.2.0.zip
07:58:06 2018-05-30 - Unzipped Tyro from: Tyro.framework-v0.2.0.zip

[thii]

One of the framework in local cache was a 0 byte archive, probably because of the upload command fails. Here is the log of the upload command.

[15:26:56]: ▸ 15:26:55 2018-05-30 - Staring to zip: Carthage/Build/iOS/MyFramework.framework
[15:26:56]: ▸ 15:26:55 2018-05-30 - Copying MyFramework.framework to: /Users/admin/Library/Caches/Rome/swift-4.1.2-swiftlang-902.0.54-clang-902.0.39.2/MyFramework/iOS/MyFramework.framework-1.0.2.zip
[15:26:56]: ▸ 15:26:55 2018-05-30 - Copied MyFramework.framework to: /Users/admin/Library/Caches/Rome/swift-4.1.2-swiftlang-902.0.54-clang-902.0.39.2/MyFramework/iOS/MyFramework.framework-1.0.2.zip
[15:26:56]: ▸ 15:26:55 2018-05-30 - Started uploading MyFramework to: swift-4.1.2-swiftlang-902.0.54-clang-902.0.39.2/MyFramework/iOS/MyFramework.framework-1.0.2.zip
[15:26:56]: ▸ 15:26:56 2018-05-30 - Error uploading MyFramework: ServiceError (ServiceError' {_serviceAbbrev = Abbrev "S3", _serviceStatus = Status {statusCode = 403, statusMessage = "Forbidden"}, _serviceHeaders = [("Server","VOS"),("Date","Wed, 30 May 2018 06:26:56 GMT"),("Content-Type","application/xml"),("Content-Length","198"),("Connection","keep-alive"),("x-amz-request-id","tx00000000000000061f841-005b0e442f-80aab6-defa
ult"),("Accept-Ranges","bytes"),("Access-Control-Allow-Origin","*"),("Access-Control-Allow-Methods","GET, POST, PUT, DELETE, HEAD, OPTIONS"),("Access-Control-Allow-Headers","Accept,Authorization,Cache-Control,Content-Type,DNT,If-Modified-Since,Keep-Alive,Origin,User-Agent,X-Requested-With,X-Amz-Date,X-Amz-User-Agent,Content-MD5,x-amz-acl,x-amz-copy-source,x-vos-token,x-vos-request-id"),("Access-Control-Expose-Headers","ET
ag, x-amz-request-id, X-RGW-Bytes-Used, X-RGW-Object-Count"),("Strict-Transport-Security","max-age=15768000")], _serviceCode = ErrorCode "SignatureDoesNotMatch", _serviceMessage = Nothing, _serviceRequestId = Just (RequestId "tx00000000000000061f841-005b0e442f-80aab6-default")})

[tmspzz]

I see.

To double check please run the download command with -v, it should try to show some download progress.
If it does show the progress, that then this might have to do with the unzipping.

I'm trying to nail down which part does not like a 0 byte archive.

[thii]

I uploaded all the cache to remote manually via the web interface, so now the download command works. But could you take a look at the error while uploading? There is an error on "SignatureDoesNotMatch", but I could upload with s3cmd using the same AWS key/secret.

[tmspzz]

I found this: https://stackoverflow.com/questions/10281557/signaturedoesnotmatch-amazon-s3-api

s3cmd is probably using a different profile or secret and id combination than rome

check the value of the following env variables:

$AWS_ACCESS_KEY_ID
$AWS_SECRET_ACCESS_KEY
$AWS_PROFILE

Edit:

There is really nothing I can do here as the part that talks to AWS is done via amazonka. Here are related issues that might shed some light: https://github.com/brendanhay/amazonka/issues?utf8=%E2%9C%93&q=is%3Aissue+SignatureDoesNotMatch

Also, on the JAVA sdk people report that regenerating the keys solved this for them: aws/aws-sdk-js#86

[tmspzz]

@thii I doubt something as important as Authorization is implemented differently between S3 and Ceph (assuming you're using Ceph)

Amazon: https://docs.aws.amazon.com/AmazonS3/latest/dev/RESTAuthentication.html#ConstructingTheAuthenticationHeader
Ceph: http://docs.ceph.com/docs/master/radosgw/s3/authentication/

[tmspzz]

Other possible causes aws/aws-cli#602 (comment) and what reported in aws/aws-cli#602

Also some characters in the keys seems to cause issues: aws/aws-cli#602 (comment) (+ / % )

[thii]

Does the download command use a different signature than the upload? Because the download command works, but the upload doesn't.

Download:

AWS_ENDPOINT=https://ourcorp.com AWS_REGION=us-east-1 AWS_ACCESS_KEY_ID=xxxx AWS_SECRET_ACCESS_KEY=yyyy Pods/Rome/rome download Framework --platform iOS --cache-prefix swift-4.1.2-swiftlang-902.0.54-clang-902.0.39.2

Upload:

AWS_ENDPOINT=https://ourcorp.com AWS_REGION=us-east-1 AWS_ACCESS_KEY_ID=xxxx AWS_SECRET_ACCESS_KEY=yyyy Pods/Rome/rome upload Framework --platform iOS --cache-prefix swift-4.1.2-swiftlang-902.0.54-clang-902.0.39.2

[tmspzz]

The signing procedure is the same I guess but obviously what is signed is different as the upload contains a ton more headers:

put: https://hackage.haskell.org/package/amazonka-s3-1.5.0/docs/Network-AWS-S3-PutObject.html
get: https://hackage.haskell.org/package/amazonka-s3-1.5.0/docs/Network-AWS-S3-PutObject.html

I suggest the following:

• try with a different key/secret pair. I have (+) in my secret and everything works fine for me however.
• check the clock of the machine. if it's too far off the the signature is invalid.

[tmspzz]

Relevant? https://ceph.com/planet/ceph-rgw-aws4-presigned-urls-working-with-the-minio-cloud-client/

[tmspzz]

Also list of issue in Ceph relating to SignatureDoesNotMatch http://tracker.ceph.com/search?utf8=%E2%9C%93&issues=1&q=SignatureDoesNotMatch

[thii]

Might be a Ceph issue. I'll report to the storage team and get you posted.

[tmspzz]

@thii any luck?

[thii]

@blender Still haven't figured out so I switched back to using s3cmd for the upload part.

[tmspzz]

That's a shame. I will see if I can manage to install ceph via docker and try myself.

[tmspzz]

@thii I have update the AWS library, can you try https://github.com/blender/Rome/releases/tag/v0.16.0.44

[thii]

@blender Still the same error.

[tmspzz]

@thii I just had this error with Minio, I updated the version I was using and It went away.

I strongly suspect this is a problem with Ceph and I think they share some common lib.

[erikkerber]

I just experienced this as well, and it was resolved after restarting my test Minio server. I'll follow up if I find anything new, but think it would be useful to anyone stumbling across this thread that it does reproduce (sometimes) with Minio 2019-02-20T22:44:29Z

[tmspzz]

@eskerber which part? The 0 byte archive or the failing upload?

[erikkerber]

Apologies - the 0 byte part. I got the error on rome download

[tmspzz]

ok, time to fix this.

[tmspzz]

@eskerber can you tell me if you encountered the problem while running the command in verbose mode? Does it happen even when rome is run without verbose flag?

[erikkerber]

@blender It was not verbose.

rome download --platform iOS

[thii]

@blender I tried with the build from master (commit f3ac8c3) but the failing upload error was still there. Should I create a new issue for it?

[tmspzz]

@thii yes. However I could never get your case to repo. If you could help me with that I can take a look.