[21.3.x] No Job logs for Container Groups other than default

[hurdigurdi]

Please confirm the following

• I agree to follow this project's code of conduct.
• I have checked the current issues for duplicates.
• I understand that AWX is open source software provided for free and that I might not receive a timely response.

Bug Summary

No logs showing for container groups other than default.

Similar issue reported in #13469 - however we are not using the "default" container group.

Authentication type used: Kubernetes Bearer Token including CA certificate created in target namespace for service account following steps from https://docs.ansible.com/automation-controller/4.3/html/userguide/credentials.html#openshift-or-kubernetes-api-bearer-token

Behaviour observed:

• jobs are starting in desired target namespace
• no stdout in AWX UI
• no output in pod logs
• stdout only shows ""Finished"
• final job status: Job terminated due to error
• runner pod directory /runner has the default folders created, but no files get populated

bash-5.1$ find . -type f -ls
bash-5.1$ ls -la
total 48
drwxrwxr-x. 6 root root 4096 Mar 10 12:12 .
drwxr-xr-x. 1 root root 4096 Mar 10 13:26 ..
drwxrwxr-x. 2 root root 4096 Mar 10 12:12 artifacts
drwxrwxr-x. 2 root root 4096 Mar 10 12:12 env
drwxrwxr-x. 2 root root 4096 Mar 10 12:12 inventory
drwxrwxr-x. 2 root root 4096 Mar 10 12:12 project
bash-5.1$ pwd
/runner

in awx-task:

bash-5.1$ /var/lib/awx/venv/awx/bin/receptorctl --socket /var/run/receptor/receptor.sock work list
Warning: receptorctl and receptor are different versions, they may not be compatible
{
    "hkrbBlUW": {
        "Detail": "Sending stdin to pod",
        "ExtraData": {
            "Command": "",
            "Image": "",
            "KubeConfig": "",
            "KubeNamespace": "ansible",
            "KubePod": "",
            "Params": "",
            "PodName": "automation-job-217-rd5kj"
        },
        "State": 0,
        "StateName": "Pending",
        "StdoutSize": 0,
        "WorkType": "kubernetes-runtime-auth"
    }
}
bash-5.1$ 

AWX version

21.3.0

Select the relevant components

• UI
• API
• Docs
• Collection
• CLI
• Other

Installation method

kubernetes

Modifications

yes

Ansible version

core 2.12.5.post0

Operating system

Garden Linux

Web browser

Chrome

Steps to reproduce

• Install AWX
• Create target namespace for container groups
• Create Service Account including containergroup-sa.yaml mappings for target container group namespace
• get CA cert and token details from service account
• Create Bearer Token for target container group including CA.crt and token
• Create Container Group
• Run any job --> no log output

Expected results

stdout of job logs

Actual results

no outoput of stdout, job goes into error after 25 minutes

Additional information

No response

[akus062381]

Hi @hurdigurdi,

Are you able to provide us with the event logs for the namespace of the container group you're trying to target? That may provide additional information that will help us provide insight.