Permit Bigquery output to create jobs in a different project

[Roviluca]

Bigquery allows to have 3 different projects involved when querying Bigquery:

1. The project where the service account is created
2. The project where the job is created
3. The project where the data is stored.

The current implementation does not allow this flexibility.
if the project is set the job will be created in the Dataset's project, otherwise the client tries to use bigquery.DetectProjectID that returns the service account's project and searches the dataset in there.

With this PR I added the possibility to set JobProjectID so that users can have the flexibility to set the 3 parameters with different values.

[Roviluca]

Hello @Jeffail does this PR make sense or do you prefer me do approach this in another way?

[garsir]

Hey @Jeffail, is this something we can add to benthos? Currently benthos doesn't support a compute/storage project split that we are trying to enact in BQ.

dull feedback or comments appreciated

[oleksandr-kaleniuk]

Are there any updates on this? It's a very useful feature

[damjad]

Hello,

Can anyone please work on this?

This is a handy feature we need for the widespread adoption of Redpanda Connect in our tech stack.
Currently, it blocks us from using any of the Redpanda offerings in prod.

[mihaitodor]

@damjad @Roviluca Thanks for your patience with this! I took a stab at rebasing the code. Would you mind testing it to make sure it still works as expected both with job_project empty and also with job_project set to some project?

[damjad]

Thanks @mihaitodor!

We will test it on our side and keep you posted.

[damjad]

Hello,

We have tested it manually at our end.

We have tested with both happy scenarios mentioned above and it's working as expected.
Moreover, we have tested with some negative scenarios and it also gives errors as expected.

Please feel free to merge it.

[Jeffail]

LGTM, thanks @Roviluca (and all the testers!)