x/build: add LUCI linux-s390x builder

[dmitshur]

There currently isn't a LUCI builder that tests the linux/s390x port (other than the misc-compile builder, which tests only that the port compiles). This is the tracking issue for it.

The next steps that a builder owner will need to follow to make progress here are documented https://go.dev/wiki/DashboardBuilders#luci-builders.

[gopherbot]

Change https://go.dev/cl/617359 mentions this issue: crypto/internal/fips/sha3: reduce s390x divergence

[srinivas-pokala]

Hostname for the builder: linux-s390x-ibm
csr is attached:
linux-s390x-ibm.csr.txt

[dmitshur]

Thanks. CC @mknyszek.

[mknyszek]

Here's the certificate:
linux-s390x-ibm-1728069494.cert.txt

[gopherbot]

Change https://go.dev/cl/636055 mentions this issue: crypto/internal/cryptotest: skip TestAllocations on s390x

[srinivas-pokala]

@mknyszek I am facing issue while following steps mentioned above for LUCI builder setup. After step-03, I have done as follow.

Note: Default builder machines are running under user(linux1)

Step-04 : Created cron job for luci_machine_tokend using new user(a2)

[a2@go-s390x01 ~]$ crontab -l
*/10 * * * * luci_machine_tokend -backend luci-token-server.appspot.com -cert-pem /home/a2/linux-s390x-ibm-1728069494.cert.txt -pkey-pem /home/a2/linux-s390x-ibm.key -token-file=/var/lib/luci_machine_tokend/token.json

Step-05: Created system service to run "bootstrapswarm" as below using same user(a2).

Description=Bootstrapswarm Service
After=network.target

[Service]
User=a2
Group=a2
ExecStart=/home/a2/go/bin/bootstrapswarm -hostname linux-s390x-ibm
Restart=always
RestartSec=5
Environment="PATH=/usr/local/go/bin:/usr/bin:/bin:/home/a2/go/bin"

[Install]
WantedBy=multi-user.target

After this when I verify bot's start-up log, I am encountering below verbose

[a2@go-s390x01 ~]$ sudo journalctl -u bootstrapswarm -f
[sudo] password for a2: 
-- Logs begin at Mon 2025-01-20 04:52:31 EST. --
Jan 20 09:22:32 go-s390x01 systemd[1]: Stopped Bootstrapswarm Service.
Jan 20 09:22:32 go-s390x01 systemd[1]: Started Bootstrapswarm Service.
Jan 20 09:22:32 go-s390x01 systemd[1]: bootstrapswarm.service: Main process exited, code=exited, status=203/EXEC
Jan 20 09:22:32 go-s390x01 systemd[1]: bootstrapswarm.service: Failed with result 'exit-code'.
Jan 20 09:22:38 go-s390x01 systemd[1]: bootstrapswarm.service: Service RestartSec=5s expired, scheduling restart.
Jan 20 09:22:38 go-s390x01 systemd[1]: bootstrapswarm.service: Scheduled restart job, restart counter is at 3102.
Jan 20 09:22:38 go-s390x01 systemd[1]: Stopped Bootstrapswarm Service.
Jan 20 09:22:38 go-s390x01 systemd[1]: Started Bootstrapswarm Service.
Jan 20 09:22:38 go-s390x01 systemd[1]: bootstrapswarm.service: Main process exited, code=exited, status=203/EXEC
Jan 20 09:22:38 go-s390x01 systemd[1]: bootstrapswarm.service: Failed with result 'exit-code'.

But when I tried manually running "/home/a2/go/bin/bootstrapswarm -hostname linux-s390x-ibm" I am getting status code 401 error(authentication). I tried inspecting/addressing the issue but I could not make much progress.
Can you please help me in troubleshooting this issue.

[dmitshur]

@srinivas-pokala Thanks for working on this. I looked at the logs on our end for error details.

I'm seeing 403s that are failing because the bot ID being reported is "go-s390x01" instead of the expected "linux-s390x-ibm", which results in an "Bot ID doesn't match the token used" error. Can you check if the bootstrapswarm binary you're using is the latest version available in x/build? Looking at its code, if the -hostname linux-s390x-ibm is being propagated correctly, it should be sent to the server. As unlikely as it is, maybe you can check that the metadata.OnGCE() path isn't being taken somehow, since that does override hostname?

A 401 that I saw failed with the error that the token was expired 4 hrs earlier; perhaps /var/lib/luci_machine_tokend/token.json stopped being refreshed at the time you tried to run bootstrapswarm?

[srinivas-pokala]

@dmitshur Thank's for the reply.

Can you check if the bootstrapswarm binary you're using is the latest version available in x/build?

Yes I have checked it. It's latest only.

Looking at its code, if the -hostname linux-s390x-ibm is being propagated correctly, it should be sent to the server. As unlikely as it is, maybe you can check that the metadata.OnGCE() path isn't being taken somehow, since that does override hostname?
I have traced back metadata.OnGCE() which invoking testOnGCE() in which connectivity of GCE instance failing. I have cross checked the ping on this address, surprisingly packets are getting loss(100%).

[a2@go-s390x01 ~]$ ping google.com
PING google.com(lga15s49-in-x0e.1e100.net (2607:f8b0:4006:80d::200e)) 56 data bytes
^C
--- google.com ping statistics ---
11 packets transmitted, 0 received, 100% packet loss, time 10434ms

So, I am suspecting this could be causing bootstarpswarm to fail and not taking into if case of metadata.OnGCE() path.

`