Windows tests not run automatically

[mweibel]

Pre-requisites

• I have double-checked my configuration
• I can confirm the issues exists when I tested with :latest
• I'd like to contribute the fix myself (see contributing guide)

What happened/what you expected to happen?

While working on reproducing #11810 and contributing the fix in #11993 I noticed a lot of failing tests for windows. Especially also the bug in question would've triggered a failed run when CI would run windows tests automatically.
However it seems that this is not done currently.

Running the tests on a windows machine, I get quite a lot of errors, see attached logs.txt.

What is the reason for not running them automatically?
Was it always like this or did that change recently?

What can be done (besides providing windows fixes for those failing tests) to improve this in the future?

Version

latest

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

not applicable

Logs from the workflow controller

not applicable

Logs from in your workflow's wait container

not applicable

[terrytangyuan]

Seems like we only have ubuntu CI builds https://github.com/argoproj/argo-workflows/blob/master/.github/workflows/ci-build.yaml

We do have a build on Windows though. https://github.com/argoproj/argo-workflows/blob/master/.github/workflows/release.yaml

We should add Windows to CI builds. Would you like to help?

[mweibel]

Looks like, yeah.

I might be able to allocate some time next week for this. What would be the best way to go on with this? Would it work by just adding a windows spec?
Of course, fixing windows tests would be needed too, and that might need some more time - wondering a bit how to start best.
Could we mark the CI run as optional and for each failing test create an issue until we're relatively stable?

[terrytangyuan]

That should be fine

[agilgur5]

Hmm if this primarily needed for Executor unit tests, we could just add a second unit test run on Windows during CI. That would be a fairly simple addition

E2E tests would be more complicated. Still simple to add another host in CI to the existing matrix, but it could make the test flake situation and test run-time situation worse 😕

[mweibel]

See linked PR for a draft change. It's not ready yet since some tests are failing.

Hmm if this primarily needed for Executor unit tests, we could just add a second unit test run on Windows during CI. That would be a fairly simple addition

Yeah indeed. First I thought about testing only the executor related tests but that would exclude quite a few tests in packages which are used by the executor so I opted instead for testing all and excluding those which do not make sense.

E2E tests would be more complicated. Still simple to add another host in CI to the existing matrix, but it could make the test flake situation and test run-time situation worse 😕

From what I know about GitHub Actions it wouldn't even be that simple since we'd need to run k3s on a linux host and then run the e2e tests which make sense on windows, possibly rewriting them (or adding a few specific windows ones). Feels like an entirely separate and bigger task to me. Adding unit tests only will hopefully help avoid a whole bunch of issues.