Initial WASM/Cross build factory code

[emmatyping]

This is able to successfully cross compile Python on WASM, but fails on the make pythoninfo stage of the build, there need to be some fixes to CPython before this will work fully.

I figured that a general cross-compiler factory would probably be useful in future, so I tried to make the code a bit more generic.

Things I still want to do:

• Fix CPython so make pythoninfo works gh-84461: Add HOSTRUNNER for program to run Python executable cpython#91931
• Fix CPython so make buildbottest works gh-84461: Add ability for multiprocessed libregrtest to use a different Python executable cpython#91930
• Allow disabling those for crossbuilds (in case one wants to test just the cross compilation itself maybe?)
• consider better directory names instead of build/host? These are based on the names autoconf uses...
• Check that uploading results works?

[vstinner]

How many tests are skipped? Does the test suite pass?

[vstinner]

What is the goal of this work? Do you plan to add a buidbot worker? Which WASM target would it use? A browser? Node? Something else?

[emmatyping]

Hi Victor!

I am working with Brett and Christian (and several others) to make WebAssembly a supported platform for CPython eventually. Currently we run a nightly build and test job here: https://github.com/ethanhs/python-wasm/runs/6151353583?check_suite_focus=true

Of course, to be supported we need to add a buildbot, so these changes are to set up a nodejs based worker.

The current summary of the tests (just running node python.js -m test) is:

== Tests result: SUCCESS ==

342 tests OK.

92 tests skipped:
    test__xxsubinterpreters test_asyncgen test_asynchat test_asyncio
    test_asyncore test_check_c_globals test_clinic test_cmd_line
    test_concurrent_futures test_contextlib_async test_ctypes
    test_curses test_dbm_gnu test_dbm_ndbm test_devpoll test_doctest
    test_docxmlrpc test_embed test_epoll test_faulthandler test_fcntl
    test_file_eintr test_fork1 test_ftplib test_gdb test_grp
    test_httplib test_httpservers test_idle test_imaplib
    test_interpreters test_ioctl test_kqueue test_launcher test_lzma
    test_mmap test_msilib test_multiprocessing_fork
    test_multiprocessing_forkserver test_multiprocessing_main_handling
    test_multiprocessing_spawn test_nis test_openpty test_ossaudiodev
    test_pdb test_poll test_poplib test_pty test_pwd test_readline
    test_regrtest test_repl test_select test_selectors test_smtplib
    test_smtpnet test_socket test_socketserver test_spwd test_sqlite3
    test_ssl test_stable_abi_ctypes test_startfile test_subprocess
    test_sys_settrace test_syslog test_tcl test_telnetlib test_tix
    test_tk test_tools test_ttk_guionly test_ttk_textonly test_turtle
    test_urllib2 test_urllib2_localnet test_urllib2net test_urllibnet
    test_venv test_wait3 test_wait4 test_webbrowser test_winconsoleio
    test_winreg test_winsound test_wsgiref test_xmlrpc test_xmlrpc_net
    test_xxlimited test_zipfile64 test_zipimport_support test_zoneinfo

1 test run no tests:
    test_dtrace

Total duration: 9 min 24 sec
Tests result: SUCCESS

I believe that Microsoft has offered to donate a buildbot worker (if not I can likely donate one), and we will likely start with a nodejs based worker, since running the test suite in the browser would be difficult.

I also factored out the cross build code because I figured it could come in handy in future for other buildbots.

As I mention in my earlier comment, the WebAssembly buildbot will require a few tweaks to CPython itself. For example, some instances of calling python ... from the Makefile will need to be replaced by calling node python.js ....

[tiran]

At first we are going to use Node.JS with NODERAWFS for testing to ensure that Python compiles and works to wasm32-emscripten. Browser tests may follow in the future. The entire test suite was passing on Node.JS. Last time I tried one test_decimal test case was flaky. The test suite skips tests that don't work on Emscripten, e.g. socket tests or missing extension modules.

[vstinner]

Please make sure that test.pythoninfo collects all versions useful to debug, like the Node.js version. The bare minimum would be to log the newly added sys._emscripten_info attribute: python/cpython#91781

Is the ./configure --with-emscripten-target=TARGET value logged somewhere? I guess that it will be logged in the buildbot configure step.

[vstinner]

Node.JS with NODERAWFS

Is it possible to log Node.js options somehow in test.pythoninfo? It seems like there are many implementation for the filesystem: NODERAWFS, MEMFS, BrowserFS, IDB, etc. (depending on the target)

Also please write a documentation somewhere explaining how to replicate the buildbot worker configuration, to reproduce the issue locally. For example, I expect that the Node.js version will matter, and how Node.js is run (options).

[tiran]

The Node.JS version is not that useful. The relevant version is emsdk version. Since we are using node from emsdk, the emsdk version encodes all other relevant bits and pieces.

[vstinner]

The Node.JS version is not that useful. The relevant version is emsdk version. Since we are using node from emsdk, the emsdk version encodes all other relevant bits and pieces.

So please log it into test.pythoninfo ;-)

[tiran]

Also please write a documentation somewhere explaining how to replicate the buildbot worker configuration, to reproduce the issue locally. For example, I expect that the Node.js version will matter, and how Node.js is run (options).

This is already documented in the Tools/wasm directory. You only need the same emsdk, that's it.

[pablogsal]

Ping me when this is ready to review @ethanhs @tiran 🎉

Very excited about this ❤️

[tiran]

• make pythoninfo now works for emscripten-node and wasi targets.

[emmatyping]

Hey @pablogsal I think this is ready for review, huge thanks to @tiran for making some needed fixes to push this forward :)

[vstinner]

This change adds bcannon-wasm worker. Is it owned by Brett Cannon? Do you have a password for this worker?

[tiran]

This change adds bcannon-wasm worker. Is it owned by Brett Cannon? Do you have a password for this worker?

@brettcannon has acquired a buildbot host for WASM.

[tiran]

python/cpython#93844 adds documentation how to set up the SDKs and dependencies locally without a container.

[brettcannon]

This change adds bcannon-wasm worker. Is it owned by Brett Cannon? Do you have a password for this worker?

No password yet as I haven't asked for one. I've been waiting to be given the go-head from @tiran and @ethanhs that everything is in-place, then I'm going to acquire the appropriate Azure credits to set up a buildbot for this.

[tiran]

No password yet as I haven't asked for one. I've been waiting to be given the go-head from @tiran and @ethanhs that everything is in-place, then I'm going to acquire the appropriate Azure credits to set up a buildbot for this.

You are good to go! I was able to run WASI and Emscripten Node tests with a local buildbot instance inside my EMSDK container. python/cpython#93844 adds documentation how to set up EMSDK, WASI-SDK and other dependencies on an Ubuntu VM for your buildbot.

[brettcannon]

I'm going to be given a chunk of credits to use every month, so I've been asked to budget what we need. Is 8 GB RAM enough? 16 GB? How many CPUs to keep up with merges? @pablogsal @vstinner any ideas/suggestions?

People can play with https://azure.microsoft.com/en-us/pricing/calculator/ if you're curious or have specific input to provide since I will be getting credits which means we can use whatever Azure services we want, I just need to give them an estimate.

[tiran]

The instance will be testing Emscripten Node, Emscripten Browser (build-only, no tests yet), and WASI for 3.11 and main branch. I would go for at least 4 vCPUs. 8 vCPUs would be better, though. RAM wise, we are effectively testing 32bit programs. The extra memory consumption of NodeJS and wasmtime should not make a bit difference. So 8 GB RAM might be enough, 16 GB would not hurt, though.

IMHO the smallest useful instance is B4ms (4 vCPU, 16 GB RAM) + 64 GiB SSD for persistent data ($150 USD/month or ~$1000 USD/year upfront). This should be good enough until we have more branches with WASM support. I would not object F8sv2 instances (8 vCPU, 16 GB RAM, optimized for computational load). It's twice as expensive as the B4ms instance.

[tiran]

A B4ms instance runs the Emscripten Node tests with make buildbottest TESTOPTS=-j2 in 13:30 and WASI tests in 6 minutes. Good enough :)

[pablogsal]

I'm going to be given a chunk of credits to use every month, so I've been asked to budget what we need. Is 8 GB RAM enough? 16 GB? How many CPUs to keep up with merges? @pablogsal @vstinner any ideas/suggestions?

People can play with https://azure.microsoft.com/en-us/pricing/calculator/ if you're curious or have specific input to provide since I will be getting credits which means we can use whatever Azure services we want, I just need to give them an estimate.

8GB is enough but if we plan to have a bunch of builders it uses parallel tests (we should because otherwise the worker will be quite behind on tests on busy days) I would recommend 2 or 4 cores if that's possible.

[vstinner]

@brettcannon:

No password yet as I haven't asked for one. I've been waiting to be given the go-head from @tiran and @ethanhs that everything is in-place, then I'm going to acquire the appropriate Azure credits to set up a buildbot for this.

I sent you a password in private.

About the hardware, I would say that https://devguide.python.org/buildworker/#required-resources is outdated: a worker should have at least 1 GB. 4 GB should be better. More is good to have, but I don't think that it's required.

In my experience, 50 GB of disk is short if you want to have multiple configs (like regular, LTO, PGO, clang, etc.). 100 GB of disk should be more comfortable. A single build takes around 600 MB of disk (on sub-directory in /home/buildbot/buildarea).

Example with a RHEL8 worker:

$ lsblk 
NAME                                            MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
vda                                             252:0    0   50G  0 disk 
├─vda1                                          252:1    0  500M  0 part /boot
└─vda2                                          252:2    0 49,5G  0 part 
  ├─vg_root_python--builder--rhel8.osci.io-root 253:0    0 13,8G  0 lvm  /
  └─vg_root_python--builder--rhel8.osci.io-home 253:1    0 35,7G  0 lvm  /home

$ LANG= df -h / /home/buildbot/
Filesystem                                               Size  Used Avail Use% Mounted on
/dev/mapper/vg_root_python--builder--rhel8.osci.io-root   14G  4.8G  8.2G  37% /
/dev/mapper/vg_root_python--builder--rhel8.osci.io-home   36G   22G   12G  66% /home

On this RHEL8 machine, /tmp is part of /: it's mounted as ext4, not tmpfs.

[brettcannon]

Thanks for the suggestions, everyone! I will play with the proposed amounts and see what I can get funding for.

[tiran]

I have test instance running on an Azure VM using some free credits that will expire in a couple of weeks. Could you please merge the PR so we can give it a try?

/opt/buildbot/buildbot-worker/bin/pip install --upgrade buildbot-worker
/opt/buildbot/buildbot-worker/bin/buildbot-worker create-worker /opt/buildbot/bcannon-wasm buildbot.python.org bcannon-wasm SECRET_PASSWORD

[vstinner]

Merged, thanks.

@tiran: @brettcannon sent you the buildbot worker password?

[tiran]

Yes, Brett sent me the password