Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Return bq job id from biquery.run_job() #2957

Merged
merged 6 commits into from
Jun 8, 2020

Conversation

tomasaschan
Copy link
Contributor

This ensures that the job_id is accessible to client code after running a BQ job; this is useful for getting logs or other metadata from the job, and/or logging the job id itself to other places than just what's done by this class.

This enables client code to do further things with the job id, e.g. look
up logs, status or other information.
Copy link
Member

@honnix honnix left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor comments. Please also add a test case to cover the change. I see we already have a test file for this.

luigi/contrib/bigquery.py Show resolved Hide resolved
luigi/contrib/bigquery.py Outdated Show resolved Hide resolved
luigi/contrib/bigquery.py Show resolved Hide resolved
@tomasaschan
Copy link
Contributor Author

Thanks for the review!

Not sure how to properly test this, though - there are no tests for the run_job method currently, and I haven't found any docs on how to set up my local environment to properly run the tests (even the core ones; this would need running tests against GCP), so adding one is difficult.

However, a search through the project for run_job shows that all usages of this method (naturally) ignore its output, so I find it hard to imagine there would be any breakage due to this.

@tomasaschan
Copy link
Contributor Author

...added a couple of tests anyway, but I haven't been able to run these, so I have no idea what CI will say about them.

@tomasaschan tomasaschan requested a review from honnix June 8, 2020 12:08
@honnix
Copy link
Member

honnix commented Jun 8, 2020

It turns out we don't run anything for gcloud at all, not even contrib in general. Travis CI and test annotation seem to be a bit messy. Don't want to block the PR, so I'm fine we move on.

@dlstadther @Tarrasch What do you think? It seem some contrib related tests are not annotated properly and they run as part of core; also contrib is not enabled in TravisCI.

@dlstadther
Copy link
Collaborator

Some tests annotations were removed/excluded from Travis while i was trying to contain test runtime here. I definitely think we should try to include as many of the annotations as we can, but be considerate of the Travis resources we take and the runtime duration to wait for PR test response.

@honnix
Copy link
Member

honnix commented Jun 8, 2020

@dlstadther What you think of this PR? Should we merge as it is or we'd better remove unexecuted/non-verified test cases?

@dlstadther
Copy link
Collaborator

I think this is safe to merge as is. Then a separate PR can update Travis config and test annotations.

@honnix
Copy link
Member

honnix commented Jun 8, 2020

Cool. I will create an issue first for Travis CI. It doesn't feel like something we can quickly get into place.

@honnix
Copy link
Member

honnix commented Jun 8, 2020

Created #2959

@honnix honnix merged commit 3bccee3 into spotify:master Jun 8, 2020
@tomasaschan
Copy link
Contributor Author

Is there a time frame when we could expect a release that includes this?

No stress - we can solve it with inheritance and overriding in the meantime - but it would be nice to not have to carry that code longer than necessary.

@honnix
Copy link
Member

honnix commented Jun 17, 2020

@tomasaschan Sorry I missed this. We will cut a release next week, according to release cycle.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants