Remove two-deployment mode

[TylerHelmuth]

Fixes #157

Removes the capability to install both the agent and deployment collector at the same time. values.yaml containing both agentCollector.enabled=true and standaloneCollector.enabled=true will fail helm install and helm upgrade commands.

The charts now support a simplified mode=daemonset or mode=deployment. The top-level config section in the values.yaml can now be modified and will be merged into the default .Values.Config. If there is a section of the default config you don't need you can remove it with null.

Single usage of agentCollector.enabled=true and standaloneCollector.enabled=true are still supported for backwards compatibility. If agentCollector.enabled=true or standaloneCollector.enabled=true are used, their settings will take precedence over mode.

[TylerHelmuth]

@dmitryax I think this branch is in a pretty good spot with this refactor but there are so many changes I need an extra set of eyes. Key things I tackled:

• added new mode and root-level configs from the agentCollector and standaloneCollector. root-level configs are never merged with the more specific version; if agentCollector or standaloneCollector is defined, their configs are always used in place of the new root-level ones.
• updated values.schema.json. Should prevent both mode but allow single mode and the new configurations.
• updated examples and makefile to support the new daemonset-and-deployment example that takes 2 values.yamls
• updated CI examples. I removed the both examples bc I couldn't find a way to use CT to test both values.yamls at the same time. don't love this

Still need to add upgrade.md and readme changes, but wanted more eyes before I wrote documentation in case something big changes.

[TylerHelmuth]

@dmitryax how should we handle this scenario:

A customer is depending on the default value of agentCollector.enabled=true, but has set agentCollector-specific values. Such as a values.yaml like:

agentCollector:
  resources:
    limits:
      cpu: 2
      memory: 4Gi

The way I've written this refactor so far will not use their resource.limits since the new default value in the values.yaml for agentCollector.enabled is false. I think I need to set the default value back to true, but allow mode: deployment to take precedence over agentCollector

If I do that I think it'll be tricky to deal with customers who want to use mode: daemonset and the new root-level values instead of the agentCollector values. Would probably need to have customers explicitly set agentCollector.enabled=false and but that in the UPGRADE.md.

[dmitryax]

The way I've written this refactor so far will not use their resource.limits since the new default value in the values.yaml for agentCollector.enabled is false. I think I need to set the default value back to true, but allow mode: deployment to take precedence over agentCollector

If I do that I think it'll be tricky to deal with customers who want to use mode: daemonset and the new root-level values instead of the agentCollector values. Would probably need to have customers explicitly set agentCollector.enabled=false and but that in the UPGRADE.md.

I agree that deployment mode is better default than daemonset, but we can handle it separately. For now let's keep mode=daemonset as default with a comment that it will be replaced with deployment in future so we encourage users to set it explicitly.

At some point in future we can make 2-step upgrade.

1. Remove default completely and fail if mode is unset.
2. Few versions later we set deployment as default.

How does it sound?

[TylerHelmuth]

The way I've written this refactor so far will not use their resource.limits since the new default value in the values.yaml for agentCollector.enabled is false. I think I need to set the default value back to true, but allow mode: deployment to take precedence over agentCollector

If I do that I think it'll be tricky to deal with customers who want to use mode: daemonset and the new root-level values instead of the agentCollector values. Would probably need to have customers explicitly set agentCollector.enabled=false and but that in the UPGRADE.md.

I agree that deployment mode is better default than daemonset, but we can handle it separately. For now let's keep mode=daemonset as default with a comment that it will be replaced with deployment in future so we encourage users to set it explicitly.

At some point in future we can make 2-step upgrade.

1. Remove default completely and fail if mode is unset.
2. Few versions later we set deployment as default.

How does it sound?

I can actually see us always leaving mode unset and forcing it to be set so that we force intentionality. I'm going back and forth on if we should ever default to deployment after forcing mode to be set, I'm torn.

All that said, I think the more pressing matter is the comment I just left in #157. Having to leave agentCollector.enabled=true is resulting in some gross values.yamls. Checkout the examples/CI folder in this PR so see what I mean.

[TylerHelmuth]

@dmitryax Updated charts to have mode not set by default, and to always take precedence if it is set.

[TylerHelmuth]

@dmitryax ready for official review. Lots of files, but 70% of them are examples/ or ci/.

[TylerHelmuth]

@dmitryax ready for round 2

[TylerHelmuth]

@dmitryax ready for round 3.

[dmitryax]

Thanks @TylerHelmuth. Nicely done!