Dataflow standard template regional launch request failure

[nick-fenton]

Launching a standard template with regional endpoint europe-west3 is failing with error:

Grpc.Core.RpcException: Status(StatusCode="FailedPrecondition", Detail="(d5c84d5dd719994a): The workflow could not be created, since it was sent to an invalid regional endpoint (europe-west3) or the wrong API endpoint was used. Please resubmit to a valid Cloud Dataflow regional endpoint and ensure you are using dataflow.projects.locations.templates.launch API endpoint. The list of Cloud Dataflow regional endpoints is at https://cloud.google.com/dataflow/docs/concepts/regional-endpoints. ")

Here's how I'm trying to launch the job:

var r = await _templateClient.LaunchTemplateAsync(new GoogleDataflow.LaunchTemplateRequest()
            {
                GcsPath = launchRequest.TemplatePath,
                Location = launchRequest.Region, //"europe-west3"
                ProjectId = _googleOptions.ProjectId,
                LaunchParameters = launchParams
            });

It seems as if I need to change the endpoint using the client builder, but I'm not quite sure how to go about passing the resource name.

Edit: I'm using Dataflow Prime.

[amanda-tarafa]

See the Client Configuration documentation for an example of how to change the endpoint using the client builder.
The example is for the AI Platforma API, but the same principle will apply to Dataflow API using its default endoint dataflow.googleapis.com.

After creating the client, you still need to specify the region when creating resources (same as you are doing in your sample code), etc, and this region needs to match the one used for the client's endpoint.

[nick-fenton]

Ok I had a chance to run some code according to the way you described. Here's the code I've run.

          var region = "europe-west3";
          var client = new GoogleDataflow.TemplatesServiceClientBuilder()
          {
              Endpoint = $"{region}-dataflow.googleapis.com"
          }.Build();

          var r = await client.LaunchTemplateAsync(new GoogleDataflow.LaunchTemplateRequest()
          {

              GcsPath = launchRequest.TemplatePath,
              Location = region,
              ProjectId = _googleOptions.ProjectId,
              LaunchParameters = launchParams
          });

This results in the following error.

Grpc.Core.RpcException: Status(StatusCode="Unimplemented", Detail="Bad gRPC response. HTTP status code: 404")
   at Google.Api.Gax.Grpc.ApiCallRetryExtensions.<>c__DisplayClass0_0`2.<<WithRetry>b__0>d.MoveNext()

Any ideas?

[amanda-tarafa]

I'll try to reproduce and bring this to the attention of the Dataflow team. I would have expect your code to work.

[amanda-tarafa]

@nick-fenton I've reproduced your issues and I've raised it with the API team.

@meredithslota assigning to you for visibility: Although this doesn't seem exactly like a recurrence of #7038, it is possibly related.

[nick-fenton]

@amanda-tarafa Thanks. Yep, I saw #7038 and I suspect it could be related.

Just a side note here: I am able to launch a regional template job via rest API using the standard dataflow.googleapis.com base url, where regional constraints are passed on the route. This all works well and is what I'll be using for my service implementation in the meantime. Just wanted to point this out to help guide the issue to the right team.

Example request uri for launching a template job via rest:

$"https://dataflow.googleapis.com/v1b3/projects/{projectId}/locations/{location}/templates:launch?gcsPath={HttpUtility.UrlEncode(templatePath)}";

[meredithslota]

@amanda-tarafa Can you include me on the internal bug if there is one?

[bhisevishal]

Note: Dataflow currently does not support regional endpoints like $"{region}-dataflow.googleapis.com".

[nick-fenton]

@bhisevishal
Ok. Thanks for the response. How should I specify a regional endpoint? Setting the location prop on LaunchTemplateRequest doesn’t route to the correct regional resource. See my the first post in this thread for details on this.

[bhisevishal]

We are investigating bug internally once we able to root cause issue then someone from google client library team or dataflow team will update this thread.

[amanda-tarafa]

There seems to be an issue with the library or with the Grpc.Net.Client dependencies, but we haven't found what the problem is yet. For the time beeing, I can offer a workaround and that is to use Grpc.Core, for that, you need to:

• Build the client as follows

var client = new TemplatesServiceClientBuilder
{
    GrpcAdapter = GrpcCoreAdapter.Instance,
}.Build();

• Add a dependency to https://www.nuget.org/packages/Grpc.Core to your project.

The rest of your code should be the same as it is now.

We are actively looking into this and I'll come back here with updates when I know more.

[nick-fenton]

Thank you for the workaround!

[amanda-tarafa]

OK, we've found what the issue is, we'll now have to make a decision on how to fix it, there are several options with different pros and cons, etc.

I'll downgrade the priority of this issue to P2, given that there's a workaround and that we have found the root cause. But we continue to actively work on it. Again, I'll come back here when we know what we'll do and the ETA for that.

[amanda-tarafa]

@meredithslota assigning to you to figure out next steps. We can discuss on the internal document. Thanks.

[amanda-tarafa]

@nick-fenton We've added a workaround in Google.Api.Gax.Grpc so you don't have to force the use of Grpc.Core. Once that's been released, you can add the new version of Google.Api.Gax.Grpc explicitly to your project, or wait until we have released a new version of Google.Cloud.Dataflow.V1Beta3 that should depend on the newest Google.Api.Gax.Grpc.

[amanda-tarafa]

@nick-fenton We've released Google.Cloud.Dataflow.V1Beta3 v2.0.0-beta04 containing the library side fix. If you upgrade to that your original code should work as expected. Do let us know if you run into any other issues.

[nick-fenton]

@amanda-tarafa thank you!

[bhisevishal]

@amanda-tarafa thank you for quick fix!