perma-red arm-fanned (Raspberry Pi) build failures

[Trott]

Happy holidays! Looks like ARM fanned builds have been failing for the last 12 hours. Error in console is complaining about the value of NODEJS_MAJOR_VERSION environment variable.

Example:

https://ci.nodejs.org/job/node-test-commit-arm-fanned/12149/console

03:00:48 Started by upstream project "node-test-commit" build number 34385
03:00:48 originally caused by:
03:00:48  Started by upstream project "node-daily-v12.x-staging" build number 63
03:00:48  originally caused by:
03:00:48   Started by timer
03:00:48 Running as SYSTEM
03:00:48 [EnvInject] - Loading node environment variables.
03:00:48 Building remotely on test-packetnet-ubuntu1604-x64-2 (Ubuntu Ubuntu-16.04 amd64-Ubuntu jenkins-workspace amd64-Ubuntu-16.04 16.04 amd64) in workspace /home/iojs/build/workspace/node-test-commit-arm-fanned
03:00:49 using credential 96d5f81c-e9ad-45f7-ba5d-bc8107c0ae2c
03:00:49  > git rev-parse --is-inside-work-tree # timeout=10
03:00:49 Fetching changes from the remote Git repository
03:00:49  > git config remote.origin.url [email protected]:nodejs/node.git # timeout=10
03:00:50 Cleaning workspace
03:00:50  > git rev-parse --verify HEAD # timeout=10
03:00:50 Resetting working tree
03:00:50  > git reset --hard # timeout=10
03:01:59  > git clean -fdx # timeout=10
03:02:01 Fetching upstream changes from [email protected]:nodejs/node.git
03:02:01  > git --version # timeout=10
03:02:01 using GIT_SSH to set credentials 
03:02:04  > git fetch --tags --force --progress -- [email protected]:nodejs/node.git +refs/heads/*:refs/remotes/origin/* +refs/heads/v12.x-staging:refs/remotes/origin/_jenkins_local_branch # timeout=20
03:02:42  > git rev-parse refs/remotes/origin/_jenkins_local_branch^{commit} # timeout=10
03:02:42  > git rev-parse refs/remotes/origin/refs/heads/_jenkins_local_branch^{commit} # timeout=10
03:02:42 Checking out Revision d96c76507fc718caaf36979104c7b3288b33d44b (refs/remotes/origin/_jenkins_local_branch)
03:02:42  > git config core.sparsecheckout # timeout=10
03:02:42  > git checkout -f d96c76507fc718caaf36979104c7b3288b33d44b # timeout=10
03:02:55 Commit message: "util: fix inspection of errors with tampered name or stack property"
03:02:55 Using 'Changelog to branch' strategy.
03:02:55 Cleaning workspace
03:02:55  > git rev-parse --verify HEAD # timeout=10
03:02:55 Resetting working tree
03:02:55  > git reset --hard # timeout=10
03:02:57  > git clean -fdx # timeout=10
03:02:58 Exporting environment variable NODEJS_VERSION with Node.js version '12.14.1'
03:02:58 Exporting environment variable NODEJS_MAJOR_VERSION with Node.js major version '12'
03:02:58 Exporting parameter NODEJS_VERSION with Node.js version '12.14.1'
03:02:58 Exporting parameter NODEJS_MAJOR_VERSION with Node.js major version '12'
03:02:58 Run condition [And] enabling prebuild for step [MultiJob Phase]
03:02:58 Run condition [Or] enabling prebuild for step [MultiJob Phase]
03:02:58 [node-test-commit-arm-fanned] $ /bin/bash -ex /tmp/jenkins1109565997767425910.sh
03:02:58 + echo
03:02:58 + '[' -n d96c76507fc718caaf36979104c7b3288b33d44b ']'
03:02:58 + TEMP_BRANCH=jenkins-node-test-commit-arm-fanned-d96c76507fc718caaf36979104c7b3288b33d44b
03:02:58 + echo 'Using TEMP_BRANCH name: jenkins-node-test-commit-arm-fanned-d96c76507fc718caaf36979104c7b3288b33d44b'
03:02:58 Using TEMP_BRANCH name: jenkins-node-test-commit-arm-fanned-d96c76507fc718caaf36979104c7b3288b33d44b
03:02:58 + echo TEMP_BRANCH=jenkins-node-test-commit-arm-fanned-d96c76507fc718caaf36979104c7b3288b33d44b
03:02:58 [EnvInject] - Injecting environment variables from a build step.
03:02:58 [EnvInject] - Injecting as environment variables the properties file path 'env.properties'
03:02:58 [EnvInject] - Variables injected successfully.
03:02:58 Waiting for the completion of git-rebase
03:09:52 git-rebase #59806 completed. Result was FAILURE
03:09:52 Build step 'Trigger/call builds on other projects' marked build as failure
03:09:52 Collecting metadata...
03:09:52 Metadata collection done.
03:09:52 Notifying upstream projects of job completion
03:09:52 Variable Existence Condition: checking "NODEJS_MAJOR_VERSION" variable 
03:09:52 Numerical comparison: [12] < Less than [12]
03:09:52 Run condition [And] preventing perform for step [Builder to mark whether executed]
03:09:52 Variable Existence Condition: checking "NODE_MAJOR_VERSION" variable 
03:09:52 Numerical comparison: [12] >= Greater than or equal to [12]
03:09:52 Run condition [Or] enabling perform for step [Builder to mark whether executed]
03:09:52 Finished: FAILURE

@rvagg (sorry)

[rvagg]

see #2116 (comment)

[rvagg]

https://ci.nodejs.org/job/node-test-commit/34429/ good except for what looks like some flakes

[Trott]

Thanks for taking care of it! Happy new year! (I did try to log into the host as root@ but could not. Re-ran the ansible playbook to rewrite my SSH config and still no luck. There's an entry in my .ssh/config file for it but it was still asking for a password.)

[rvagg]

odd, cause this one has the test key in it - when you do the root@IP make sure you -i ~/path/to/privkey too. In the inventory it's also listed as jenkins-workspace-3 so you can just ssh jenkins-workspace-3. Let me know if that doesn't work for you, there may be something deeper going on here, you should be able to ssh to all 3 of the jenkins-workspace machines.

[Trott]

ssh jenkins-workspace-3 worked. Thanks!