Skip to content

Commit

Permalink
Follow up fix for merging errors from batch BigQuery inserts (#1348). (
Browse files Browse the repository at this point in the history
…#1350)

* Follow up fix for merging errors from batch BigQuery inserts (#1348).

* fix pylint false positive
  • Loading branch information
Dor1s authored Jan 17, 2020
1 parent 5e32deb commit 4b5e825
Showing 1 changed file with 12 additions and 2 deletions.
14 changes: 12 additions & 2 deletions src/python/google_cloud_utils/big_query.py
Original file line number Diff line number Diff line change
Expand Up @@ -370,7 +370,7 @@ def insert(self, inserts):
A json explained here:
https://cloud.google.com/bigquery/docs/reference/rest/v2/tabledata/insertAll
"""
result = None
result = {}
for i in range(0, len(inserts), INSERT_BATCH_SIZE):
response = self._insert_batch(
inserts[i:min(len(inserts), i + INSERT_BATCH_SIZE)])
Expand All @@ -379,8 +379,18 @@ def insert(self, inserts):
time.sleep(1)

if not result:
# Use result from the first batch, appending errors from the rest.
result = response
else:
result['insertErrors'].extend(response['insertErrors'])
# If there are new errors from the current batch, append to the result.
new_errors = response.get('insertErrors')
if not new_errors:
continue

# Apparently result may not have errors, be careful.
if result.get('insertErrors'):
result['insertErrors'].extend(new_errors)
else:
result['insertErrors'] = new_errors

return result

0 comments on commit 4b5e825

Please sign in to comment.