Storage: Add `Blob.from_string` and `Bucket.from_string`. #8424

HemangChothani · 2019-06-19T07:21:47Z

tseaver · 2019-06-19T16:56:37Z

storage/tests/unit/test_client.py

@@ -821,3 +821,49 @@ def dummy_response():
        self.assertEqual(page.remaining, 0)
        self.assertIsInstance(bucket, Bucket)
        self.assertEqual(bucket.name, blob_name)
+
+    def test_get_bucket_from_uri(self):


Please rename this testcase test_get_bucket_from_uri_w_valid_uri.

tseaver · 2019-06-19T16:56:58Z

storage/tests/unit/test_client.py

+        self.assertIs(bucket.client, client)
+        self.assertEqual(bucket.name, BUCKET_NAME)
+
+    def test_get_blob_from_uri(self):


Please rename this testcase test_get_blob_from_uri_w_valid_uri.

tseaver · 2019-06-19T16:57:15Z

storage/tests/unit/test_client.py

+        self.assertIs(blob.client, client)
+        self.assertEqual(blob.name, "/b/")
+
+    def test_get_bucket_with_invalid_uri(self):


Please rename this testcase test_get_bucket_from_uri_w_invalid_uri and move it above the test_get_object_w_valid_uri.

tseaver · 2019-06-19T16:57:30Z

storage/tests/unit/test_client.py

+            with pytest.raises(ValueError, match="URI scheme must be gs"):
+                client.get_bucket_from_uri("http://bucket_name")
+
+    def test_get_blob_with_invalid_uri(self):


Please rename this testcase test_get_blob_from_uri_w_invalid_uri.

tseaver · 2019-06-19T17:00:18Z

storage/google/cloud/storage/client.py

@@ -477,6 +477,37 @@ def list_buckets(
            extra_params=extra_params,
        )

+    def get_blob_from_uri(self, uri):


Please move this method below get_bucket_from_uri.

tseaver · 2019-06-19T17:02:11Z

storage/google/cloud/storage/client.py

+        """Get a constructor for blob object by URI.
+
+         :type uri: str
+         :param uri: The blob uri pass to get blob object.


Please add an example of the expected form of the uri, e.g.:

"""Construct a blob object from a URI. :type uri: str :param uri: The blob's URI, in the form "gs://<bucket-name>/<blob-name>". ... """

tseaver · 2019-06-19T17:02:59Z

storage/google/cloud/storage/client.py

+        """Get a constructor for bucket object by URI.
+
+        :type uri: str
+        :param uri: The bucket uri pass to get bucket object.


Please add an example of the expected form of the uri, e.g.:

"""Construct a bucket from a URI. :type uri: str :param uri: The bucket's URI, in the form "gs://<bucket-name>". ... """

tseaver · 2019-06-19T17:05:43Z

storage/google/cloud/storage/client.py

+            raise ValueError("URI scheme must be gs")
+
+        bucket = Bucket(self, name=netloc)
+        return Blob(path[1:], bucket)


@frankyn, @tswast Given that the names for these methods start with get_, should we be making the API calls to load the objects?

Yes. get_ implies actually making a GET request. This should call .reload().

tswast · 2019-06-19T18:39:16Z

Note: this doesn't actually fix #7693 yet. I'd prefer you refactor this to have a Blob.from_string(uri, client=None) method and similar for Bucket. The get_ methods do make sense, though, assuming they actually make a GET request.

tseaver · 2019-06-19T19:29:45Z

@tswast What is your expected usecase for having client default to None in a Blob.from_string / Bucket.from_string call? The objects themselves aren't very useful without a client.

tswast · 2019-06-20T00:07:45Z

What is your expected usecase for having client default to None in a Blob.from_string / Bucket.from_string call? The objects themselves aren't very useful without a client.

Per https://github.com/googleapis/google-cloud-python/issues/7762, the goal is to have Blob and Bucket usable without a client in more places, as methods like client.create_bucket should be able to take a Bucket object without a client object present. (I acknowledge that this isn't the case yet.)

"skipif" should be used instead of "skipIf", the latter is the thing from the unittest nodule.

Toward googleapis#8686.

Update Makefile test proto builder: - Use new 'conformance_tests' repo. - Handle updated file hierarchy, etc. Use new JSON format in 'test_cross_language.py'. - Copy in JSON testcase files from 'conformance-tests' repo.

…nt library generator. (googleapis#8712)

…googleapis#8701) Closes googleapis#7215.

Add undocumented-but-required '/documents' to the database path. Closes googleapis#8717.

…oogleapis#9087)

…protobuf files (via synth). (googleapis#9129)

tseaver · 2019-08-28T13:57:24Z

@HemangChothani please rebase your branch against the current master to pick up the unit test fixes merged yesterday in #9119. Also, can you please check your git configuration to ensure that the e-mail address used in the commits matches your CLA (one commit, 30617c2, has '[email protected]' as the e-mail address).

…9054)

…nd batch_predict() from AutoML Tables client (googleapis#9116)

…rame`. (googleapis#9084) * Specify the index data type in partial schema to `load_table_from_dataframe` to include it. If an index (or level of a multi-index) has a name and is present in the schema passed to `load_table_from_dataframe`, then that index will be serialized and written to the table. Otherwise, the index is omitted from the serialized table. * Don't include index if has same name as column name. * Move `load_table_dataframe` sample from `snippets.py` to `samples/`. Sample now demonstrates how to manually include the index with a partial schema definition. Update docs reference to new `load_table_dataframe` sample location.

* Make all system tests table columns NULL-able Autodetected schema assumes NULLABLE fields, thus use NULLABLE columns in test fixture. * Replace helper function with BQ client method The _add_rows() helper function is no longer needed, since client.load_table_from_json() essentially does the same. * Use explicit schema in data types decoding test * Adjust a comment related to partial schemas

…in a different project (googleapis#9139)

…loband bucket class

Doc string indention.

googlebot · 2019-08-29T05:57:02Z

All (the pull request submitter and all commit authors) CLAs are signed, but one or more commits were authored or co-authored by someone other than the pull request submitter.

We need to confirm that all authors are ok with their commits being contributed to this project. Please have them confirm that by leaving a comment that contains only @googlebot I consent. in this pull request.

Note to project maintainer: There may be cases where the author cannot leave a comment, or the comment is not properly detected as consent. In those cases, you can manually confirm consent of the commit author(s), and set the cla label to yes (if enabled on your project).

ℹ️ Googlers: Go here for more info.

HemangChothani · 2019-08-29T08:06:55Z

Closing this PR due to something irrelevant happens and unable to fix it, So replacing this PR with new PR 9143

HemangChothani requested review from crwilcox, frankyn and tseaver as code owners June 19, 2019 07:21

googlebot added the cla: yes This human has signed the Contributor License Agreement. label Jun 19, 2019

tseaver suggested changes Jun 19, 2019

View reviewed changes

tseaver added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Jun 19, 2019

yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Jun 19, 2019

sduskis added the api: storage Issues related to the Cloud Storage API. label Jun 19, 2019

tseaver changed the title ~~Storage: Add methods to parse URI for blob and bucket~~ Storage: Add 'Client.get_bucket_from_uri' and 'Client.get_blob_from_uri'. Jun 19, 2019

tseaver added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Jun 20, 2019

yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Jun 20, 2019

yoshi-automation added the 🚨 This issue needs some love. label Jun 26, 2019

yoshi-automation removed the 🚨 This issue needs some love. label Jul 3, 2019

plamut and others added 4 commits July 17, 2019 10:08

Fix several pytest "skip if" markers (googleapis#8694)

b9c06da

"skipif" should be used instead of "skipIf", the latter is the thing from the unittest nodule.

Add 'Cell.__repr__'. (googleapis#8683)

42bfb81

Update pins of 'googleapis-common-protos. (googleapis#8688)

d45bd71

Toward googleapis#8686.

AVaksman added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Jul 18, 2019

yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Jul 18, 2019

Ricardo Mendes and others added 8 commits July 18, 2019 11:53

Enable Sample Generator Tool for Data Catalog (googleapis#8708)

3df7cb0

Release google-api_core 1.14.0. (googleapis#8700)

74d76d2

Update docstrings (via synth). (googleapis#8711)

df09107

Bump minimum version for google-api-core to 1.14.0. (googleapis#8709)

2d0c094

[CHANGE ME] Re-generated talent to pick up changes in the API or clie…

7b662d2

…nt library generator. (googleapis#8712)

Avoid adding 'prefix' to update mask for transforms used in 'update'. (…

9299701

…googleapis#8701) Closes googleapis#7215.

Update Cloud Memorystore service link. (googleapis#8706)

ee472ad

Fix V1 'Client.collections' method. (googleapis#8718)

9a7d9f0

Add undocumented-but-required '/documents' to the database path. Closes googleapis#8717.

yoshi-automation added 2 commits August 28, 2019 09:10

Add 'batch_create_sessions' method to generated client (via synth). (g…

9d37d37

…oogleapis#9087)

Move 'BatchOperationMetadata' / 'JobOperationResult' messages to new …

041d89e

…protobuf files (via synth). (googleapis#9129)

tritone and others added 11 commits August 28, 2019 09:59

Expose 'HMACKeyMetadata.id' field. (googleapis#9115)

5640610

Add client_options to constructors for manual clients. (googleapis#…

79fba16

…9054)

Add client_options. (googleapis#9043)

362f664

Add client_options to logging v1 (googleapis#9046)

9a297e9

Release storage 1.19.0 (googleapis#9138)

b60cdea

Enable users to pass in Pandas Dataframe when calling import_data() a…

36e7179

…nd batch_predict() from AutoML Tables client (googleapis#9116)

Catch exceptions in GcsClient when a same name bucket already exists …

4d3a381

…in a different project (googleapis#9139)

Release automl 0.5.0 (googleapis#9141)

43ba6ee

Firestore: Add client_options to v1. (googleapis#9048)

519a545

HemangChothani force-pushed the feature/add_methods_fromuri_for_blob_bucket branch from 4b73404 to 58f2329 Compare August 29, 2019 05:54

HemangChothani requested review from anguillanneuf, busunkim96 and plamut as code owners August 29, 2019 05:54

HemangChothani requested a review from a team August 29, 2019 05:54

HemangChothani added 3 commits August 29, 2019 11:26

refactor from_uri method to from_string method from client class to b…

d6c48af

…loband bucket class

Edit doc string

2cb85e9

Update blob.py

a3c9c58

Doc string indention.

HemangChothani added 3 commits August 29, 2019 11:29

use from_string method for download file

bcb2436

resolve conflicts

8bf2e6b

add testscase of domain-named bucket

38a2c93

HemangChothani force-pushed the feature/add_methods_fromuri_for_blob_bucket branch from 58f2329 to 38a2c93 Compare August 29, 2019 06:01

HemangChothani closed this Aug 29, 2019

tseaver mentioned this pull request Aug 19, 2021

AttributeError: 'NoneType' object has no attribute '_http' when using Blob.upload_from_string googleapis/python-storage#540

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Storage: Add `Blob.from_string` and `Bucket.from_string`. #8424

Storage: Add `Blob.from_string` and `Bucket.from_string`. #8424

HemangChothani commented Jun 19, 2019

tseaver Jun 19, 2019

tseaver Jun 19, 2019

tseaver Jun 19, 2019

tseaver Jun 19, 2019

tseaver Jun 19, 2019

tseaver Jun 19, 2019

tseaver Jun 19, 2019

tseaver Jun 19, 2019

tswast Jun 19, 2019

tswast commented Jun 19, 2019

tseaver commented Jun 19, 2019

tswast commented Jun 20, 2019

tseaver commented Aug 28, 2019

googlebot commented Aug 29, 2019

HemangChothani commented Aug 29, 2019

Storage: Add Blob.from_string and Bucket.from_string. #8424

Storage: Add Blob.from_string and Bucket.from_string. #8424

Conversation

HemangChothani commented Jun 19, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tswast commented Jun 19, 2019

tseaver commented Jun 19, 2019

tswast commented Jun 20, 2019

tseaver commented Aug 28, 2019

googlebot commented Aug 29, 2019

HemangChothani commented Aug 29, 2019

Storage: Add `Blob.from_string` and `Bucket.from_string`. #8424

Storage: Add `Blob.from_string` and `Bucket.from_string`. #8424