ETag MD5 match issues #87
Comments
mzealey
added a commit
to moya-app/rp-archiver
that referenced
this issue
Sep 15, 2023
Per nyaruka#87, S3 often generates ETag differently from pure MD5 hash which means that database entries are not cleared.
Closed
mzealey
added a commit
to moya-app/rp-archiver
that referenced
this issue
Sep 15, 2023
Per nyaruka#87, S3 often generates ETag differently from pure MD5 hash which means that database entries are not cleared.
Merged
|
Addressed in #90 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Most of our S3 uploads are having issues with etag vs md5 calcs being incorrect, for example:
It seems from https://docs.aws.amazon.com/AmazonS3/latest/API/API_Object.html that ETag is calculated quite differently depending on upload size, S3 encryption setting etc. I suspect having our buckets by default encrypted is causing this issue.
#45 seems a related issue.
In short I'm not sure that MD5 ETag checks are the correct way to go here. Perhaps it's worth adding a flag to make this check optional? Because actually for us, the danger of having too much stuff in the DB and not archived (causing space usage, slowdowns etc) is likely greater than the cost of loosing old history.
The text was updated successfully, but these errors were encountered: