(stepfunctions-tasks): AWS Batch integration with container overrides breaks with token values

[jstag711]

What is the problem?

I was eagerly awaiting 1.149 as it appeared to fix the issue that prevents container overrides of memory and cpu:

#18993
#19298

However, now I get the following error:

{
  "resourceType": "batch",
  "resource": "submitJob.sync",
  "error": "Batch.ClientException",
  "cause": "Value -1.888154589709072e+289 for type MEMORY in resourceRequirement is not valid. Provide a valid number as input. (Service: AWSBatch; Status Code: 400; Error Code: ClientException; Request ID: dcd8c27a-baac-414a-9b64-da5733cc1344; Proxy: null)"
}

Reproduction Steps

This is the code that sets up the container overrides

BatchContainerOverrides containerOverrides = BatchContainerOverrides.builder()
                .command(JsonPath.listAt("$." + SOLVER_TASK_PARAMETERS + "." + COMMAND))
                .vcpus(JsonPath.numberAt("$." + SOLVER_TASK_PARAMETERS + "." + CPU))
                .memory(Size.mebibytes(JsonPath.numberAt("$." + SOLVER_TASK_PARAMETERS + "." + MEMORY)))
                .build();

After I updated to 1.149 there is still no resourceRequirements field on BatchContainerOverrides. Based on the fix it appears that these values would just be copied from the top-level vcpus and memory.

Previously I was receiving the warning that my container overrides were being ignored, but the job would still execute.

What did you expect to happen?

I expected that with this update my container overrides would be respected.

What actually happened?

{
  "resourceType": "batch",
  "resource": "submitJob.sync",
  "error": "Batch.ClientException",
  "cause": "Value -1.888154589709072e+289 for type MEMORY in resourceRequirement is not valid. Provide a valid number as input. (Service: AWSBatch; Status Code: 400; Error Code: ClientException; Request ID: dcd8c27a-baac-414a-9b64-da5733cc1344; Proxy: null)"
}

CDK CLI Version

1.149.0

Framework Version

No response

Node.js Version

12.22.11

OS

MacOS 10.15.7

Language

Java

Language Version

Java 11

Other information

No response

[kaizencc]

Hi @jstag711, I'm hesitant to say this is a regression at the moment. Previously your values were being ignored. Now that they are not, it's hard to say if they were invalid before, or if there is a bug in the code.

It looks like the number provided to memory currently (-1.888154589709072e+289) is not what you are expecting. Can you tell me what you are expecting Size.mebibytes(JsonPath.numberAt("$." + SOLVER_TASK_PARAMETERS + "." + MEMORY) to be?

[jstag711]

So if you look at the first link I sent, the values were (emphasis on the quotation marks) "ignored" - they were in fact read and then flagged as ignored. This is very weird behavior IMO, the warning text essentially says "we see you setting this value, we read it, and we're not going to use it".

I expect the value to be 61440.

Does the fix not work if the values are JsonPath values? I can attempt to set fixed values there and see if the behavior remains the same.

[peterwoodworth]

Yeah what we need to figure out is what value is getting passed into here:

Size.mebibytes(JsonPath.numberAt("$." + SOLVER_TASK_PARAMETERS + "." + MEMORY))

If you can log both JsonPath.numberAt("$." + SOLVER_TASK_PARAMETERS + "." + MEMORY) and the full thing, then we'd be able to figure out at least what stage the bug is occuring

[jstag711]

The value is 61440. The input to the step has not changed.

Also I'm not sure what you mean by log, that's part of the step function definition. I can show you the input if you want to verify my json path is valid.

[peterwoodworth]

When I pass in a hardcoded 61440, this is the relevant part of the DefinitionString synthesized for the State Machine

"ContainerOverrides\":{\"ResourceRequirements\":[{\"Type\":\"MEMORY\",\"Value\":\"61440\"},{\"Type\":\"VCPU\",\"Value\":\"2\"}]}}}

What value are you seeing in your synthesized definition string?

[jstag711]

Pasting the relevant portion:

"ContainerOverrides":{"Command.$":"$.solverTaskParameters.cmd","ResourceRequirements":[{"Type":"MEMORY","Value":"-1.888154589709072e+289"},{"Type":"VCPU","Value":"-1.8881545897090718e+289"}]},"Timeout":{"AttemptDurationSeconds":900}}}}}

This is what it was prior to the upgrade:

"ContainerOverrides":{"Command.$":"$.solverTaskParameters.cmd","Memory.$":"$.solverTaskParameters.memory","Vcpus.$":"$.solverTaskParameters.cpu"},"Timeout":{"AttemptDurationSeconds":3300}}}}}

So can confirm the template is synthesized incorrectly. When I change the JsonPath params to raw numbers, I get the right thing:

"","ContainerOverrides":{"Command.$":"$.solverTaskParameters.cmd","ResourceRequirements":[{"Type":"MEMORY","Value":"10"},{"Type":"VCPU","Value":"10"}]},"Timeout":{"AttemptDurationSeconds":900}}}}}"

[peterwoodworth]

I think I see what's happening here.

Because you're using JsonPath.numberAt(), that will return a token. We're trying to call the direct value of a token, which won't work if we want JsonPath.

What's interesting is that the same applies for GpuCount

aws-cdk/packages/@aws-cdk/aws-stepfunctions-tasks/lib/batch/submit-job.ts

Line 299 in 8c60d4d

Value: `${containerOverrides.gpuCount}`,

, just no one has run into this error before with that?

We need to consider the case where the value is a token

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.