Allow setting default execution group pod spec

[kdelee]

This will allow us to control the default container group created via settings, meaning
we could set this in the operator and the default container group would get created with it applied.

We need this for being able to apply things like node labels etc to the default container group pod spec at install time, which is one part of ansible/awx-operator#242

[kdelee]

@jlanda mentioned it may be easier to use extra_volumes instead of extra_settings for getting the formatting to work well in the operator
https://github.com/ansible/awx-operator#extra-settings

[AlanCoding]

the operator does have some pain points with extra_settings, but this PR seems to be perfectly correct on its own.

[kdelee]

Hypothetical CR to use w/ operator to make this work:

apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
  name: myawx
  namespace: mynamespace
spec:
  create_preload_data: true
  route_tls_termination_mechanism: Edge
  garbage_collect_secrets: false
  loadbalancer_port: 80
  image_pull_policy: IfNotPresent
  projects_storage_size: 8Gi
  task_privileged: false
  projects_storage_access_mode: ReadWriteMany
  projects_persistence: false
  replicas: 1
  admin_user: admin
  loadbalancer_protocol: http
  nodeport_port: 30080
  image: myregistry.io/myrepo/myawx
  image_version: myversion
  extra_settings: 
    - setting: DEFAULT_EXECUTION_QUEUE_POD_SPEC_OVERRIDE                                                                  
      value: |-                                                                                                           
"""
spec:
  affinity:
    nodeAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 1
        preference:
          matchExpressions:
          - key: another-node-label-key
            operator: In
            values:
            - another-node-label-value
"""

I'm realizing to specify a different namespace here in the pod spec, we would need to also create a k8s bearer token credential and associate that with the container group. But I think that can be addressed in a seperate PR

[kdelee]

Ok finally got this working. Turned out we needed to use the deepmerge function we were already using later on on the podspec. I'm not sure why this code is duplicated in tasks.py and in the kubernetes scheduler, but as it is -- this worked for me.

Creating my AWX Cr with this:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: containergroup-config
  namespace: elijah
data:
  containergroup.py: |+
    DEFAULT_EXECUTION_QUEUE_POD_SPEC_OVERRIDE = """spec:
      affinity:
        nodeAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 1
            preference:
              matchExpressions:
              - key: awx.awx.io/node-label
                operator: In
                values:
                - execution
    """

---
apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
  name: elijah-awx
  namespace: elijah
spec:
  create_preload_data: true
  service_type: nodeport
  ingress_type: route
  image_pull_policy: Always
  replicas: 1
  admin_user: admin
  image: quay.io/kdelee/awx
  image_version: elijah
  extra_volumes: |
    - name: containergroup-config
      configMap:
        name: containergroup-config
        items:
          - key: containergroup.py
            path: containergroup.py
  task_extra_volume_mounts: |
    - name: containergroup-config
      mountPath: /etc/tower/conf.d/containergroup.py
      subPath: containergroup.py
  web_extra_volume_mounts: |
    - name: containergroup-config
      mountPath: /etc/tower/conf.d/containergroup.py
      subPath: containergroup.py

And having previously added this label to a node on a cluster, I see that the job pod gets created with:

  namespace: elijah
  labels:
    ansible-awx: 921eac0e-9e59-4320-86b3-2506b743e96b
    ansible-awx-job-id: '17'
spec:
  restartPolicy: Never
  serviceAccountName: default
  imagePullSecrets:
    - name: default-dockercfg-dtsss
  priority: 0
  schedulerName: default-scheduler
  enableServiceLinks: true
  affinity:
    nodeAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 1
          preference:
            matchExpressions:
              - key: awx.awx.io/node-label
                operator: In
                values:
                  - execution
  terminationGracePeriodSeconds: 30
  preemptionPolicy: PreemptLowerPriority
  nodeName: tower-9tvvp-worker-a-b48ff.c.ansible-tower-engineering.internal

That node being the one where the label had been added.

🚀 Can now say I'm confident this PR allows adding node affinity/node selectors to the pod spec w/o having to also provide any other info that is normally set by AWX.

In a future PR I will consider what is necessary if we want to be able to use a different namespace + configure at deploy time a credential for using that namespace.

[rebeccahhh]

I like this a lot, but we've got to hold off (probably until tomorrow afternoon) on merging this. I'll make a note to come back when we're in the clear to merge.