-
-
Notifications
You must be signed in to change notification settings - Fork 381
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Propagate __exit__ call to underlying filestream #786
Merged
Merged
+51
−6
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
c401104
to
c50707a
Compare
0586dce
to
f0cca5d
Compare
ddelange
commented
Oct 5, 2023
@@ -909,7 +909,8 @@ def write(self, b): | |||
|
|||
def terminate(self): | |||
"""Cancel the underlying multipart upload.""" | |||
assert self._upload_id, "no multipart upload in progress" | |||
if self._upload_id is None: | |||
return |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
terminate() called before a multipart upload was initiated:
self = <smart_open.tests.test_smart_open.S3OpenTest testMethod=test_write_bad_encoding_strict>
@moto.mock_s3
def test_write_bad_encoding_strict(self):
"""Should open the file for writing with the correct encoding."""
s3 = _resource('s3')
s3.create_bucket(Bucket='bucket')
key = "s3://bucket/key.txt"
text = u'欲しい気持ちが成長しすぎて'
with self.assertRaises(UnicodeEncodeError):
with smart_open.open(key, 'w', encoding='koi8-r', errors='strict') as fout:
> fout.write(text)
5 tasks
mpenkov
approved these changes
Feb 21, 2024
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, thank you
ddelange
added a commit
to ddelange/smart_open
that referenced
this pull request
Feb 21, 2024
…open into patch-2 * 'develop' of https://github.com/RaRe-Technologies/smart_open: Propagate __exit__ call to underlying filestream (piskvorky#786) Retry finalizing multipart s3 upload (piskvorky#785) Fix `KeyError: 'ContentRange'` when received full content from S3 (piskvorky#789) Add support for SSH connection via aliases from `~/.ssh/config` (piskvorky#790) Make calls to smart_open.open() for GCS 1000x faster by avoiding unnecessary GCS API call (piskvorky#788) Add zstandard compression feature (piskvorky#801) Support moto 4 & 5 (piskvorky#802) Secure the connection using SSL when connecting to the FTPS server (piskvorky#793) upgrade dev status classifier to stable (piskvorky#798) Fix formatting of python code (piskvorky#795)
3 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Title
Propagate
__exit__
call to underlying filestream.No more need for
tweak_close
function: the double wrapping will now cause three calls to__exit__
, which in turn will callclose
, first on the outermost filestream, working inward.If a third party compression handler calls
close
on the inner filestream, the inner__exit__
call will do a second call toclose
on the inner filestream, which will be ignored.Motivation
Currently, the compression mechanic has a side-effect that
__exit__
is no longer called on the original filestream. Consequently, eg boto3 abort multipart upload is not called, causing lingering blobs that will be billed.Tests
✅
Work in progress
Checklist
Before you create the PR, please make sure you have:
Workflow
Please avoid rebasing and force-pushing to the branch of the PR once a review is in progress.
Rebasing can make your commits look a bit cleaner, but it also makes life more difficult from the reviewer, because they are no longer able to distinguish between code that has already been reviewed, and unreviewed code.