DaemonSet should respect new scheduling constraints

[bgrant0607]

Node affinity / anti-affinity, at least.

It already respects nodeSelector.

See also #12744 (comment)

cc @davidopp @kevin-wangzefeng

[mikedanese]

I believe this is unblocked.

[davidopp]

Pod affinity is also in the pipeline, I just haven't had time to make the second pass review yet (#22985)

[davidopp]

This didn't make 1.3, moving to 1.4.

[lukaszo]

It looks like no one is working on it so I've looked at the code.
It should be easy to change nodeShouldRunDaemonPod in pkg/controller/daemon/controller.go to use GeneralPredicates.
Unfortunately I see one issue. In current implementation of nodeShouldRunDaemonPod when there are no free resource or there is a port conflict it generates Event. GeneralPredicates does not generate any events so moving to it would break backward compatibility.
What do you think how can it be solved? Can I add generating Events to GeneralPredicates?

[mikedanese]

@lukaszo that event is not part of the api so it wouldn't be breaking compatibility to change it. Is there any way to surface failure cause of GeneralPredicate to the user?

[lukaszo]

@mikedanese I ended up with logging the errors. Pls see my PR

[lukaszo]

Ups, I'm not sure if it should be closed. anti-affinity is still not supported.
cc @bgrant0607

[bgrant0607]

@lukaszo Could you please file a more specific issue for pod anti-affinity (assuming that's the only missing feature)? Once it exists, we can close this one.

[lukaszo]

@bgrant0607 done: #29276