fix(test): wait enough time to Trigger Running Hook. Fixes: #12844

[shuangkun]

sleep 1s is sometimes too short to trigger the running hook

Fixes #12844

Motivation

Modifications

Verification

[shuangkun]

I saw there are many comment:

	// TODO: Temporarily comment out this assertion since it's flaky:
	// 	  The running hook is occasionally not triggered. Possibly because the step finishes too quickly
	//	  while the controller did not get a chance to trigger this hook.

This one we can sleep longer. Because it only verifies running hooks, unlike others that also verify other types of hooks.

[tczhao]

Could you let us how you tested it to make sure it doesn't flake

[shuangkun]

Could you let us how you tested it to make sure it doesn't flake

It is indeed not guaranteed, but now sleep 1s does often lead to a higher probability of failure of this test. In many other tests, the running hook check is skipped. If it is better for you to skip the task,I can skip it.

[tczhao]

I wonder if we can run https://argo-workflows.readthedocs.io/en/stable/running-locally/#running-e2e-tests-locally a few times to reproduce the original issue

[shuangkun]

Could you let us how you tested it to make sure it doesn't flake

It is indeed not guaranteed, but now sleep 1s does often lead to a higher probability of failure of this test. In many other tests, the running hook check is skipped. If it is better for you to skip the task, you can skip it.

I wonder if we can run https://argo-workflows.readthedocs.io/en/stable/running-locally/#running-e2e-tests-locally a few times to reproduce the original issue

It is difficult to reproduce, but it may happen in the test after submitting the PR. Like TestStoppedWorkflow and TestTemplateExecutor, It is almost difficult to reproduce locally when running a test alone. It is easy to occur only after the PR is submitted and several people trigger the pipeline at the same time.

[tczhao]

I think we can keep monitor and see if this change reduces flakiness

[shuangkun]

I think we can keep monitor and see if this change reduces flakiness

Yes, I think so. Thanks

[toyamagu-2021]

Thanks! We should observe this change resolves the issue.

I wonder if we can run argo-workflows.readthedocs.io/en/stable/running-locally/#running-e2e-tests-locally a few times to reproduce the original issue

It might be hard to reproduce this issue by local testing, because this fakiness may be caused by poor spec of CI runner (2core, 7mem).

Ref: #11534

@agilgur5 Thanks for mentioning me :)

[agilgur5]

poor spec of CI runner (2core, 7mem).

This did get doubled recently (see also Slack thread), it's now 4 vCPU & 16GB RAM. But 4 vCPU is still likely less vCPU than most local dev setups