3855 - Check all files exist in S3 #4207
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
sorja
commented
Jan 13, 2025
| const batchResults = await Promises.each(batches, async (batch) => { | ||
| return Promises.each(batch, async ({ uuid: key }) => { | ||
| const exists = await FileStorage.fileExists({ key }) | ||
| if (!exists) Logger.info(`Missing file in S3: ${key}`) |
There was a problem hiding this comment.
Note: I deleted one file from S3 manually to see output from this command
All other files are there (and the file was there before manual deletion ;) )
0077fe28-....-....-....-5809eb1d364c
minotogna
reviewed
Jan 13, 2025
|
@minotogna I tried figuring out the best way to test this - but I think in the end, the most important part is that the files exist in S3 - everything else should be reduntant because we didn't change other logic. I couldn't cause other issues in the platform other than a timeout from S3 from spamming requests |
sorja
force-pushed
the
3855-test-migration
branch
from
72ef59e to
dbb1525
Compare
mergify bot
added a commit
that referenced
this pull request
Jan 20, 2025
* 3855 - file storage using s3 * 3855 - deprecate old static files * 3855 - migration step: files to s3 * 3855 - responses: add sendFileStream * 3855 - repository/file: don't return file data from postgres/db * 3855 - user profile picture get/put to s3 * 3855 - report get/put to s3 * 3855 - user profile picture get/put to s3 * 3855 - upload file to s3 - tool * 3855 - create and read files from s3 * eol .env.template * FileSize * add default path to uploads * 3855 - fileStorage: fileExists * 3855 - Add checks to avoid duplicate file uploads and ensure idempotent migration * remove double clause * type and migration fix * remove column public.file.file * Migrate all files * 3855 - Check all files exist in S3 (#4207) * 3855 - Add Arrays.chunk * 3855 - Add env var: authCookie * 3855 - Update test * 3855 - Remove authCookie env var * 3855 - Separate upload files (#4215) * rename testAllFilesUploaded * 3855 - Separate file uploading * 3855 - Move s3Files under tools * 3855 - Include public.file in db copy (#4217) * 3855 - minor fixes (#4224) * 3855 - delete file from s3 (#4225) * 3855 - Remove file from public.file (#4226) * 3855 - Remove unused files from S3 (#4231) * 3855 - Add transaction for public.file row removal (#4232) * 3855 - Show file size after uploading (#4233) * 3855 - Add transaction for public.file row removal * 3855 - Show file size after upload --------- Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> --------- Co-authored-by: Mino Togna <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
dev note: Avoid run many times back to back -
// Note: Running this script many times may cause runners IP to timeout as protection mechanism