Unexpected withAWS behaviour when assuming Role with useNode

[CameronMcAuley]

Jenkins and plugins versions report

Environment

Jenkins: 2.401.2
OS: Linux - 5.4.0-1089-aws
Java: 11.0.19 - Ubuntu (OpenJDK 64-Bit Server VM)
---
ace-editor:1.1
analysis-model-api:11.3.0
ansible:240.vc26740a_625c0
antisamy-markup-formatter:159.v25b_c67cd35fb_
apache-httpcomponents-client-4-api:4.5.14-150.v7a_b_9d17134a_5
authentication-tokens:1.53.v1c90fd9191a_b_
aws-credentials:191.vcb_f183ce58b_9
aws-java-sdk:1.12.481-392.v8b_291cfcda_09
aws-java-sdk-cloudformation:1.12.481-392.v8b_291cfcda_09
aws-java-sdk-codebuild:1.12.481-392.v8b_291cfcda_09
aws-java-sdk-ec2:1.12.481-392.v8b_291cfcda_09
aws-java-sdk-ecr:1.12.481-392.v8b_291cfcda_09
aws-java-sdk-ecs:1.12.481-392.v8b_291cfcda_09
aws-java-sdk-efs:1.12.481-392.v8b_291cfcda_09
aws-java-sdk-elasticbeanstalk:1.12.481-392.v8b_291cfcda_09
aws-java-sdk-iam:1.12.481-392.v8b_291cfcda_09
aws-java-sdk-kinesis:1.12.481-392.v8b_291cfcda_09
aws-java-sdk-logs:1.12.481-392.v8b_291cfcda_09
aws-java-sdk-minimal:1.12.481-392.v8b_291cfcda_09
aws-java-sdk-sns:1.12.481-392.v8b_291cfcda_09
aws-java-sdk-sqs:1.12.481-392.v8b_291cfcda_09
aws-java-sdk-ssm:1.12.481-392.v8b_291cfcda_09
azure-credentials:254.v64da_8176c83a
azure-sdk:132.v62b_48eb_6f32f
azure-vm-agents:859.v7213476e4fea_
blueocean:1.27.5
blueocean-autofavorite:1.2.5
blueocean-bitbucket-pipeline:1.27.5
blueocean-commons:1.27.5
blueocean-config:1.27.5
blueocean-core-js:1.27.5
blueocean-dashboard:1.27.5
blueocean-display-url:2.4.2
blueocean-events:1.27.5
blueocean-git-pipeline:1.27.5
blueocean-github-pipeline:1.27.5
blueocean-i18n:1.27.5
blueocean-jwt:1.27.5
blueocean-personalization:1.27.5
blueocean-pipeline-api-impl:1.27.5
blueocean-pipeline-editor:1.27.5
blueocean-pipeline-scm-api:1.27.5
blueocean-rest:1.27.5
blueocean-rest-impl:1.27.5
blueocean-web:1.27.5
bootstrap5-api:5.3.0-1
bouncycastle-api:2.29
branch-api:2.1122.v09cb_8ea_8a_724
build-timeout:1.31
caffeine-api:3.1.6-115.vb_8b_b_328e59d8
checks-api:2.0.0
cloud-stats:302.v45b_647b_90608
cloudbees-bitbucket-branch-source:825.va_6a_dc46a_f97d
cloudbees-folder:6.815.v0dd5a_cb_40e0e
cobertura:1.17
code-coverage-api:4.7.0
command-launcher:100.v2f6722292ee8
commons-lang3-api:3.12.0-36.vd97de6465d5b_
commons-text-api:1.10.0-36.vc008c8fcda_7b_
config-file-provider:952.va_544a_6234b_46
credentials:1271.v54b_1c2c6388a_
credentials-binding:604.vb_64480b_c56ca_
data-tables-api:1.13.5-1
disable-github-multibranch-status:1.2
display-url-api:2.3.7
docker-commons:439.va_3cb_0a_6a_fb_29
docker-workflow:563.vd5d2e5c4007f
durable-task:510.v324450f8dca_4
ec2-fleet:2.7.1
echarts-api:5.4.0-5
envinject:2.901.v0038b_6471582
envinject-api:1.199.v3ce31253ed13
extended-choice-parameter:375.v72e4b_a_d33d33
extended-read-permission:53.v6499940139e5
external-monitor-job:207.v98a_a_37a_85525
favorite:2.4.2
flaky-test-handler:1.2.2
font-awesome-api:6.4.0-2
forensics-api:2.3.0
ghprb:1.42.2
git:5.2.0
git-client:4.4.0
git-forensics:2.0.0
git-parameter:0.9.19
git-server:99.va_0826a_b_cdfa_d
github:1.37.1
github-api:1.314-431.v78d72a_3fe4c3
github-branch-source:1728.v859147241f49
github-checks:545.v79a_a_68b_ca_682
github-oauth:0.39
github-pr-comment-build:96.v9ff13b69dd66
github-scm-trait-notification-context:1.1
gradle:2.8.1
greenballs:1.15.1
groovy:453.vcdb_a_c5c99890
handlebars:3.0.8
handy-uri-templates-2-api:2.1.8-22.v77d5b_75e6953
htmlpublisher:1.31
instance-identity:173.va_37c494ec4e5
ionicons-api:56.v1b_1c8c49374e
jackson2-api:2.15.2-350.v0c2f3f8fc595
jacoco:3.3.4
jakarta-activation-api:2.0.1-3
jakarta-mail-api:2.0.1-3
javadoc:233.vdc1a_ec702cff
javax-activation-api:1.2.0-6
javax-mail-api:1.6.2-9
jaxb:2.3.8-1
jdk-tool:66.vd8fa_64ee91b_d
jenkins-design-language:1.27.5
jjwt-api:0.11.5-77.v646c772fddb_0
jnr-posix-api:3.1.17-1
jquery:1.12.4-1
jquery-detached:1.2.1
jquery3-api:3.7.0-1
jsch:0.2.8-65.v052c39de79b_2
junit:1217.v4297208a_a_b_ce
ldap:682.v7b_544c9d1512
lockable-resources:1172.v4b_8fc8eed570
log-parser:2.3.0
mailer:457.v3f72cb_e015e5
mask-passwords:150.vf80d33113e80
matrix-auth:3.1.10
matrix-project:789.v57a_725b_63c79
maven-plugin:3.22
mina-sshd-api-common:2.10.0-69.v28e3e36d18eb_
mina-sshd-api-core:2.10.0-69.v28e3e36d18eb_
momentjs:1.1.1
monitoring:1.94.1
multibranch-scan-webhook-trigger:1.0.9
naginator:1.19.2
node-iterator-api:49.v58a_8b_35f8363
nodejs:1.6.0
npm-yarn-wrapper-steps:0.4.0
okhttp-api:4.11.0-145.vcb_8de402ef81
pam-auth:1.10
pipeline-aws:1.43
pipeline-build-step:496.v2449a_9a_221f2
pipeline-github-lib:42.v0739460cda_c4
pipeline-githubnotify-step:49.vf37bf92d2bc8
pipeline-graph-analysis:202.va_d268e64deb_3
pipeline-groovy-lib:656.va_a_ceeb_6ffb_f7
pipeline-input-step:468.va_5db_051498a_4
pipeline-milestone-step:111.v449306f708b_7
pipeline-model-api:2.2144.v077a_d1928a_40
pipeline-model-definition:2.2144.v077a_d1928a_40
pipeline-model-extensions:2.2144.v077a_d1928a_40
pipeline-rest-api:2.33
pipeline-stage-step:305.ve96d0205c1c6
pipeline-stage-tags-metadata:2.2144.v077a_d1928a_40
pipeline-stage-view:2.33
pipeline-utility-steps:2.16.0
plain-credentials:143.v1b_df8b_d3b_e48
plugin-util-api:3.3.0
popper-api:1.16.1-3
popper2-api:2.11.6-2
prism-api:1.29.0-7
pubsub-light:1.17
resource-disposer:0.22
role-strategy:670.vc71a_a_c00039e
scm-api:676.v886669a_199a_a_
script-security:1251.vfe552ed55f8d
slack:664.vc9a_90f8b_c24a_
snakeyaml-api:1.33-95.va_b_a_e3e47b_fa_4
sse-gateway:1.26
ssh-agent:333.v878b_53c89511
ssh-credentials:305.v8f4381501156
ssh-slaves:2.877.v365f5eb_a_b_eec
sshd:3.303.vefc7119b_ec23
structs:324.va_f5d6774f3a_d
summary_report:1.15
swarm:3.40
throttle-concurrents:2.14
timestamper:1.25
token-macro:359.vb_cde11682e0c
trilead-api:2.84.v72119de229b_7
variant:59.vf075fe829ccb
warnings-ng:10.2.0
workflow-aggregator:596.v8c21c963d92d
workflow-api:1241.v4edc8b_44933b_
workflow-basic-steps:1017.vb_45b_302f0cea_
workflow-cps:3722.v85ce2a_c6240b_
workflow-durable-task-step:1247.v7f9dfea_b_4fd0
workflow-job:1316.vd2290d3341a_f
workflow-multibranch:756.v891d88f2cd46
workflow-scm-step:415.v434365564324
workflow-step-api:639.v6eca_cd8c04a_a_
workflow-support:848.v5a_383b_d14921
ws-cleanup:0.45

What Operating System are you using (both controller, and any agents involved in the problem)?

Master: Ubuntu 18.04.3 LTS
Agents: Ubuntu 18.04.6 LTS

Reproduction steps

1. Set up a role on AWS on Account A, with a policy allowing sts:AssumeRole for a role on Account B.
2. Set up a role on AWS on Account B with a trust policy allowing the role on Account A to assume it.
3. Assign role from Account A to Jenkins agent.
4. In declarative Jenkinsfile, create withAWS block, stating name of role in Account B. Set useNode to be true to ensure that Jenkinsfile uses the role attached to the agent instead of the role attached to the master. Add an AWS command within the withAWS block to ensure that resources displayed are from Account B (i.e the assumed credentials) and not Account A.

withAWS(role: 'ContainerPromotionRole', region: AWS_REGION, roleAccount: AWS_PRODUCTION_ACCOUNT_ID, useNode: true){
    sh """
        sudo aws ecr describe-repositories --region ${AWS_REGION}
    """
}

Expected Results

Jenkinsfile should list ECR repositories within account B due to having assumed the IAM role from Account B.

Actual Results

Jenkinsfile assumes the role correctly, but then lists ECR repositories from Account A instead of Account B.

Setting AWS region us-west-2 
 Requesting assume role
Assuming role ARN is arn:aws:iam::123456789098:role/ContainerPromotionRole role arn:aws:sts::123456789098:assumed-role/ContainerPromotionRole/Jenkins-TestServices_Multibranch_Pipeline-KS-12345-D-244 with id {KEY}:Jenkins-TestServices_Multibranch_Pipeline-KS-12345-D-244 
[Pipeline] {
[Pipeline] sh
Retrieving credentials from node.
+ sudo aws ecr describe-repositories --region us-west-2
(Wrong ECR repos are displayed, from Account A instead of Account B)

Anything else?

Initially I used a withAWS block without useNode. Jenkins failed to assume the role from Account B because it did not have permission. The reason was because the credentials from the Master were being used instead. As such, I added useNode to ensure that the Agent's IAM credentials would be used. After I added useNode, I successfully assumed the role from AWS Account B. However it doesn't look like it is completely working as it is still using the node/agent's credentials instead of the assumed role.