Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unify ways to run the Python test suite #109566

Closed
vstinner opened this issue Sep 19, 2023 · 1 comment
Closed

Unify ways to run the Python test suite #109566

vstinner opened this issue Sep 19, 2023 · 1 comment
Labels
tests Tests in the Lib/test dir

Comments

@vstinner
Copy link
Member

vstinner commented Sep 19, 2023 •
edited by bedevere-app bot
Loading

There are at least 15 known ways to run the Python test suite. IMO it's too much, it's hard to keep all these code paths consistent. IMO python -m test should be enough to fit all use cases.

Maybe we need a --ci, --buildbot, or --strict option which would run tests is "strict" mode.

Portable way to run tests:

./python -m test
./python -m test.regrtest
./python -m test.libregrtest
./python -m test.autotest

./python Lib/test/regrtest.py
./python Lib/test/autotest.py

./python -c 'from test import autotest'

The main drawback is that running ./python doesn't work when Python is built with ./configure --enable-shared, and it doesn't work when Python is cross-compiled (WASM/WASI buildbots). Moreover, on macOS, ./python.exe must be run instead. On Windows, it's just python.

Unix, from Python source code:

make buildbottest TESTOPTS="..."
make test
make testall
make testuniversal
make hostrunnertest

./python Tools/scripts/run_tests.py

Windows, from Python source code:

PCbuild\rt.bat
Tools\buildbot\test.bat

  • Lib/test/autotest.py just runs test.libregrtest.main().

  • Lib/test/regrtest.py runs test.libregrtest.main() and remove Lib/test/ from sys.path and make __file__ an absolute path.

  • PCbuild\rt.bat:

    • python: pass -u -Wd -E -bb options
    • regrtest: pass options passed to rt.bat (except of options specific to rt.bat)
    • most options are to get the path to the python.exe program in PCbuild/

  • Tools\buildbot\test.bat runs PCbuild\rt.bat:

    • rt.bat: pass -q -d options.
    • regrtest: pass -j1 -uall -rwW --slowest --timeout=1200 --fail-env-changed options.
    • ARM32 SSH pass -unetwork -udecimal -usubprocess -uurlfetch -utzdata -rwW --slowest --timeout=1200 --fail-env-changed options.

  • make buildbottest runs Tools/scripts/run_tests.py:

    • python: pass $(TESTPYTHONOPTS) options.
    • regrtest: pass -j 1 -u all -W --slowest --fail-env-changed --timeout=$(TESTTIMEOUT) $(TESTOPTS) options.

  • Tools/scripts/run_tests.py:

    • python: pass -u -W default -bb -E options -- also cross-compilation options, not detailed here
    • regrtest: pass -r -w -j0 -u all,-largefile,-audio,-gui options, add -n on Windows.

GitHub Action workflow uses, .github/workflows/build.yml:

  • Windows x86: .\PCbuild\rt.bat -p Win32 -d -q -uall -u-cpu -rwW --slowest --timeout=1200 -j0
  • Windows x64: .\PCbuild\rt.bat -p x64 -d -q -uall -u-cpu -rwW --slowest --timeout=1200 -j0
  • macOS: make buildbottest TESTOPTS="-j4 -uall,-cpu"
  • Ubuntu: xvfb-run make buildbottest TESTOPTS="-j4 -uall,-cpu"
  • Address Sanitizer: xvfb-run make buildbottest TESTOPTS="-j4 -uall,-cpu"

Buildbot use, master/custom/factories.py:

  • UnixBuild and UnixCrossBuild: make buildbottest TESTOPTS="..." TESTPYTHONOPTS="..." TESTTIMEOUT="..." with code to select TESTOPTS, TESTPYTHONOPTS and TESTTIMEOUT.
  • UnixInstalledBuild: /path/to/python -m test ... with code to select -m test options.
  • BaseWindowsBuild: Tools\buildbot\test.bat ... with code to select .... options

Linked PRs
vstinner added a commit to vstinner/cpython that referenced this issue Sep 19, 2023
@vstinner
pythongh-109566: Run buildbot tests with --fail-rerun
8dd5a56
@bedevere-app bedevere-app bot mentioned this issue Sep 19, 2023
Merged
vstinner added a commit to vstinner/cpython that referenced this issue Sep 19, 2023
@vstinner
pythongh-109566, regrtest: Add --fast-ci and --slow-ci options
bed5a77 
* Add "make fastcitest"
* --fast-ci has a default timeout to 10 minutes, --slow-ci has a
  default ot 20 minutes.
* "make buildbottest" now uses "--slow-ci". Remove now redundant
  options.
* GitHub Actions workflow now uses "make fastcitest". Remove now
  redundant options. Just use -j0: detect the number of CPUs.
vstinner added a commit to vstinner/cpython that referenced this issue Sep 19, 2023
@vstinner
pythongh-109566, regrtest: Add --fast-ci and --slow-ci options
e6fd818 
* Add "make fastcitest"
* --fast-ci has a default timeout to 10 minutes, --slow-ci has a
  default ot 20 minutes.
* "make buildbottest" now uses "--slow-ci". Remove now redundant
  options.
* GitHub Actions workflow now uses "make fastcitest". Remove now
  redundant options. Just use -j0: detect the number of CPUs.
vstinner added a commit to vstinner/cpython that referenced this issue Sep 19, 2023
@vstinner
pythongh-109566, regrtest: Add --fast-ci and --slow-ci options
fefb208 
* --fast-ci has a default timeout to 10 minutes, --slow-ci has a
  default ot 20 minutes.
* Add "make fastcitest": similar to "make buildbottest", but use
  --fast-ci option instead of --slow-ci, and don't use TESTTIMEOUT.
* "make buildbottest" now uses "--slow-ci". Remove now redundant
  options.
* GitHub Actions workflow now uses "make fastcitest". Remove now
  redundant options. Just use -j0: detect the number of CPUs.
vstinner added a commit to vstinner/cpython that referenced this issue Sep 19, 2023
@vstinner
pythongh-109566, regrtest: Add --fast-ci and --slow-ci options
dbbdc26 
* Add --fast-ci and --slow-ci options to libregrtest:

  * --fast-ci uses a default timeout of 10 minutes and "-u all,-cpu"
    (skip slowest tests).
  * --slow-ci uses a default timeout of 20 minues and "-u all" (run
    all tests).

* Add "make fastcitest" target: similar to "make buildbottest", but
  use --fast-ci option instead of --slow-ci, and don't use
  TESTTIMEOUT.
* "make buildbottest" now uses "--slow-ci". Remove now redundant
  options.
* GitHub Actions workflow now uses "make fastcitest". Remove now
  redundant options. Just use -j0: detect the number of CPUs.
@bedevere-app bedevere-app bot mentioned this issue Sep 19, 2023
Merged
vstinner added a commit that referenced this issue Sep 19, 2023
@vstinner
gh-109566: Run GHA and buildbot tests with --fail-rerun (#109567)
67d9363
@vstinner vstinner added the tests Tests in the Lib/test dir label Sep 20, 2023
vstinner added a commit to vstinner/cpython that referenced this issue Sep 22, 2023
@vstinner
pythongh-109566: Fix typo in PCbuild/rt.bat
5d1c46f
@bedevere-app bedevere-app bot mentioned this issue Sep 22, 2023
Merged
vstinner added a commit that referenced this issue Sep 22, 2023
@vstinner
gh-109566: Fix typo in PCbuild/rt.bat (#109701)
3cce6be
vstinner added a commit to vstinner/cpython that referenced this issue Sep 26, 2023
@vstinner
pythongh-109566, regrtest: Add --fast-ci and --slow-ci options
70d1c17 
* Add --fast-ci and --slow-ci options to libregrtest:

  * --fast-ci uses a default timeout of 10 minutes and "-u all,-cpu"
    (skip slowest tests).
  * --slow-ci uses a default timeout of 20 minues and "-u all" (run
    all tests).

* Add "make fastcitest" target: similar to "make buildbottest", but
  use --fast-ci option instead of --slow-ci, and don't use
  TESTTIMEOUT.
* "make buildbottest" now uses "--slow-ci". Remove now redundant
  options.
* GitHub Actions workflow now uses "make fastcitest". Remove now
  redundant options. Just use -j0: detect the number of CPUs.
vstinner added a commit to vstinner/cpython that referenced this issue Sep 26, 2023
@vstinner
pythongh-109566, regrtest: Add --fast-ci and --slow-ci options
a24c755 
* Add --fast-ci and --slow-ci options to libregrtest:

  * --fast-ci uses a default timeout of 10 minutes and "-u all,-cpu"
    (skip slowest tests).
  * --slow-ci uses a default timeout of 20 minues and "-u all" (run
    all tests).

* Add "make fastcitest" target: similar to "make buildbottest", but
  use --fast-ci option instead of --slow-ci, and don't use
  TESTTIMEOUT.
* "make buildbottest" now uses "--slow-ci". Remove now redundant
  options.
* GitHub Actions workflow now uses "make fastcitest". Remove now
  redundant options. Just use -j0: detect the number of CPUs.
* Set TESTTIMEOUT default to an empty string, since --slow-ci and
  --fast-ci use different default timeout. It's now accepted to pass
  "--timeout=" to regrtest: treated as not timeout.
vstinner added a commit to vstinner/cpython that referenced this issue Sep 26, 2023
@vstinner
pythongh-109566, regrtest: Add --fast-ci and --slow-ci options
699a0dd 
* Add --fast-ci and --slow-ci options to libregrtest:

  * --fast-ci uses a default timeout of 10 minutes and "-u all,-cpu"
    (skip slowest tests).
  * --slow-ci uses a default timeout of 20 minues and "-u all" (run
    all tests).

* regrtest header now lists test resources.
* Makefile changes:

  * "make test" now uses --fast-ci option and TESTTIMEOUT variable.
  * "make hostrunnertest" now uses --fast-ci option and TESTTIMEOUT
    variable.
  * "make buildbottest" now uses "--slow-ci". Remove options which
    became redundant with "--slow-ci".
  * "make coverage-report" now uses --fast-ci option and TESTTIMEOUT
    variable.
  * "make testall" and "make testuniversal" now use --slow-ci option
    and TESTTIMEOUT variable.
  * "make testall" now uses "find -exec rm ..." instead of
    "find ... -print|xargs rm ...", same as "make clean".

* GitHub Actions workflow now uses "make test". Remove options which
  became redundant with "--fast-ci". Just use -j0: detect the number
  of CPUs.
* Set TESTTIMEOUT default to an empty string, since --slow-ci and
  --fast-ci use different default timeout. It's now accepted to pass
  "--timeout=" to regrtest: treated as not timeout.
vstinner added a commit to vstinner/cpython that referenced this issue Sep 26, 2023
@vstinner
pythongh-109566, regrtest: Add --fast-ci and --slow-ci options
cf59c92 
* Add --fast-ci and --slow-ci options to libregrtest:

  * --fast-ci uses a default timeout of 10 minutes and "-u all,-cpu"
    (skip slowest tests).
  * --slow-ci uses a default timeout of 20 minues and "-u all" (run
    all tests).

* regrtest header now lists test resources.
* Makefile changes:

  * "make test" now uses --fast-ci option and TESTTIMEOUT variable.
  * "make hostrunnertest" now uses --fast-ci option and TESTTIMEOUT
    variable.
  * "make buildbottest" now uses "--slow-ci". Remove options which
    became redundant with "--slow-ci".
  * "make coverage-report" now uses --fast-ci option and TESTTIMEOUT
    variable.
  * "make testall" and "make testuniversal" now use --slow-ci option
    and TESTTIMEOUT variable.
  * "make testall" now uses "find -exec rm ..." instead of
    "find ... -print|xargs rm ...", same as "make clean".

* GitHub Actions workflow:

  * Ubuntu and Address Sanitizer jobs now use "make test". Remove
    options which became redundant with "--fast-ci".
  * Windows jobs now use --fast-ci option.
  * Use -j0 to detect the number of CPUs.

* Set Makefile TESTTIMEOUT default to an empty string, since
  --slow-ci and --fast-ci use different default timeout. It's now
  accepted to pass "--timeout=" to regrtest: treated as not timeout.
* Tools/scripts/run_tests.py now uses --fast-ci option.
* Tools/buildbot/test.bat now uses --slow-ci option. Remove
  --timeout=1200 option, redundant with --slow-ci.
vstinner added a commit to vstinner/cpython that referenced this issue Sep 26, 2023
@vstinner
pythongh-109566, regrtest: Add --fast-ci and --slow-ci options
be4c7be 
* Add --fast-ci and --slow-ci options to libregrtest:

  * --fast-ci uses a default timeout of 10 minutes and "-u all,-cpu"
    (skip slowest tests).
  * --slow-ci uses a default timeout of 20 minues and "-u all" (run
    all tests).

* regrtest header now lists test resources.
* Makefile changes:

  * "make test", "make hostrunnertest" and "make coverage-report" now
    use --fast-ci option and TESTTIMEOUT variable.
  * "make buildbottest" now uses "--slow-ci". Remove options which
    became redundant with "--slow-ci".
  * "make testall" and "make testuniversal" now use --slow-ci option
    and TESTTIMEOUT variable.
  * "make testall" now uses "find -exec rm ..." instead of
    "find ... -print|xargs rm ...", same as "make clean".

* GitHub Actions workflow:

  * Ubuntu and Address Sanitizer jobs now use "make test". Remove
    options which became redundant with "--fast-ci".
  * Windows jobs now use --fast-ci option.
  * Use -j0 to detect the number of CPUs.

* Set Makefile TESTTIMEOUT default to an empty string, since
  --slow-ci and --fast-ci use different default timeout. It's now
  accepted to pass "--timeout=" to regrtest: treated as not timeout.
* Tools/scripts/run_tests.py now uses --fast-ci option.
* Tools/buildbot/test.bat now uses --slow-ci option. Remove
  --timeout=1200 option, redundant with --slow-ci.
vstinner added a commit that referenced this issue Sep 26, 2023
@vstinner
gh-109566, regrtest: Add --fast-ci and --slow-ci options (#109570)
859618c 
* Add --fast-ci and --slow-ci options to libregrtest:

  * --fast-ci uses a default timeout of 10 minutes and "-u all,-cpu"
    (skip slowest tests).
  * --slow-ci uses a default timeout of 20 minues and "-u all" (run
    all tests).

* regrtest header now lists test resources.
* Makefile changes:

  * "make test", "make hostrunnertest" and "make coverage-report" now
    use --fast-ci option and TESTTIMEOUT variable.
  * "make buildbottest" now uses "--slow-ci". Remove options which
    became redundant with "--slow-ci".
  * "make testall" and "make testuniversal" now use --slow-ci option
    and TESTTIMEOUT variable.
  * "make testall" now uses "find -exec rm ..." instead of
    "find ... -print|xargs rm ...", same as "make clean".

* GitHub Actions workflow:

  * Ubuntu and Address Sanitizer jobs now use "make test". Remove
    options which became redundant with "--fast-ci".
  * Windows jobs now use --fast-ci option.
  * Use -j0 to detect the number of CPUs.

* Set Makefile TESTTIMEOUT default to an empty string, since
  --slow-ci and --fast-ci use different default timeout. It's now
  accepted to pass "--timeout=" to regrtest: treated as not timeout.
* Tools/scripts/run_tests.py now uses --fast-ci option.
* Tools/buildbot/test.bat now uses --slow-ci option. Remove
  --timeout=1200 option, redundant with --slow-ci.
vstinner added a commit to vstinner/cpython that referenced this issue Sep 26, 2023
@vstinner
pythongh-109566: regrtest reexecutes the process
b4e03bf 
regrtest now replaces the process with a new process to add options
to the Python command line. Add options: "-u -W default -bb". When
--fast-ci and --slow-ci options are used, the -E option is also
added.

The following methods to run the Python test suite don't replace the
process:

* "import test.autotest"
* "from test.regrtest import main; main()"

Changes:

* PCbuild/rt.bat and Tools/scripts/run_tests.py no longer need to add
  "-u -W default -bb -E" options to Python: it's now done by
  regrtest.
* Fix Tools/scripts/run_tests.py: flush stdout before replacing the
  process. Previously, buffered messages were lost.
vstinner added a commit to vstinner/cpython that referenced this issue Sep 26, 2023
@vstinner
pythongh-109566: regrtest reexecutes the process
4c3934a 
regrtest now replaces the process with a new process to add options
to the Python command line. Add options: "-u -W default -bb". When
--fast-ci and --slow-ci options are used, the -E option is also
added.

The following methods to run the Python test suite don't replace the
process:

* "import test.autotest"
* "from test.regrtest import main; main()"

Changes:

* PCbuild/rt.bat and Tools/scripts/run_tests.py no longer need to add
  "-u -W default -bb -E" options to Python: it's now done by
  regrtest.
* Fix Tools/scripts/run_tests.py: flush stdout before replacing the
  process. Previously, buffered messages were lost.
vstinner added a commit to vstinner/cpython that referenced this issue Sep 26, 2023
@vstinner
pythongh-109566: regrtest reexecutes the process
462c3e1 
regrtest now replaces the process with a new process to add options
to the Python command line. Add options: "-u -W default -bb". When
--fast-ci and --slow-ci options are used, the -E option is also
added.

The following methods to run the Python test suite don't replace the
process:

* "import test.autotest"
* "from test.regrtest import main; main()"

Changes:

* PCbuild/rt.bat and Tools/scripts/run_tests.py no longer need to add
  "-u -W default -bb -E" options to Python: it's now done by
  regrtest.
* Fix Tools/scripts/run_tests.py: flush stdout before replacing the
  process. Previously, buffered messages were lost.
vstinner added a commit to vstinner/cpython that referenced this issue Sep 26, 2023
@vstinner
pythongh-109566: regrtest reexecutes the process
7ddff04 
regrtest now replaces the process with a new process to add options
to the Python command line. Add options: "-u -W default -bb". When
--fast-ci or --slow-ci option is used, the "-E" option is also added.

The following methods to run the Python test suite don't replace the
process:

* "import test.autotest"
* "from test.regrtest import main; main()"

Changes:

* PCbuild/rt.bat and Tools/scripts/run_tests.py no longer need to add
  "-u -W default -bb -E" options to Python: it's now done by
  regrtest.
* Fix Tools/scripts/run_tests.py: flush stdout before replacing the
  process. Previously, buffered messages were lost.
@bedevere-app bedevere-app bot mentioned this issue Sep 26, 2023
Merged
vstinner added a commit to vstinner/cpython that referenced this issue Sep 26, 2023
@vstinner
pythongh-109566: regrtest reexecutes the process
62f359a 
When --fast-ci or --slow-ci option is used, regrtest now replaces the
current process with a new process to add "-u -W default -bb -E"
options to Python.

The following methods to run the Python test suite don't replace the
process:

* "import test.autotest"
* "from test.regrtest import main; main()"
* "from test.libregrtest.main import main; main()"

Changes:

* PCbuild/rt.bat and Tools/scripts/run_tests.py no longer need to add
  "-u -W default -bb -E" options to Python: it's now done by
  regrtest.
* Fix Tools/scripts/run_tests.py: flush stdout before replacing the
  process. Previously, buffered messages were lost.
vstinner added a commit to vstinner/cpython that referenced this issue Sep 26, 2023
@vstinner
pythongh-109566: regrtest reexecutes the process
f1bb3ad 
When --fast-ci or --slow-ci option is used, regrtest now replaces the
current process with a new process to add "-u -W default -bb -E"
options to Python.

Changes:

* PCbuild/rt.bat and Tools/scripts/run_tests.py no longer need to add
  "-u -W default -bb -E" options to Python: it's now done by
  regrtest.
* Fix Tools/scripts/run_tests.py: flush stdout before replacing the
  process. Previously, buffered messages were lost.
vstinner added a commit that referenced this issue Sep 26, 2023
@vstinner
gh-109566: regrtest reexecutes the process (#109909)
fbfec56 
When --fast-ci or --slow-ci option is used, regrtest now replaces the
current process with a new process to add "-u -W default -bb -E"
options to Python.

Changes:

* PCbuild/rt.bat and Tools/scripts/run_tests.py no longer need to add
  "-u -W default -bb -E" options to Python: it's now done by
  regrtest.
* Fix Tools/scripts/run_tests.py: flush stdout before replacing the
  process. Previously, buffered messages were lost.
vstinner added a commit to vstinner/cpython that referenced this issue Sep 26, 2023
@vstinner
pythongh-109566: Fix regrtest code adding Python options
3321352 
* On Windows, use subprocess.run() instead of os.execv().
* Only add needed options
* Rename reexec parameter to _add_python_opts.
* Rename --no-reexec option to --dont-add-python-opts.
@bedevere-app bedevere-app bot mentioned this issue Sep 26, 2023
Merged
vstinner added a commit to vstinner/cpython that referenced this issue Sep 26, 2023
@vstinner
pythongh-109566: Fix regrtest code adding Python options
3f82882 
* On Windows, use subprocess.run() instead of os.execv().
* Only add needed options
* Rename reexec parameter to _add_python_opts.
* Rename --no-reexec option to --dont-add-python-opts.
vstinner added a commit that referenced this issue Sep 26, 2023
@vstinner
gh-109566: Fix regrtest code adding Python options (#109926)
9abba71 
* On Windows, use subprocess.run() instead of os.execv().
* Only add needed options
* Rename reexec parameter to _add_python_opts.
* Rename --no-reexec option to --dont-add-python-opts.
csm10495 pushed a commit to csm10495/cpython that referenced this issue Sep 28, 2023
@vstinner @csm10495
pythongh-109566, regrtest: Add --fast-ci and --slow-ci options (pytho…
a3c4161 
…n#109570)

* Add --fast-ci and --slow-ci options to libregrtest:

  * --fast-ci uses a default timeout of 10 minutes and "-u all,-cpu"
    (skip slowest tests).
  * --slow-ci uses a default timeout of 20 minues and "-u all" (run
    all tests).

* regrtest header now lists test resources.
* Makefile changes:

  * "make test", "make hostrunnertest" and "make coverage-report" now
    use --fast-ci option and TESTTIMEOUT variable.
  * "make buildbottest" now uses "--slow-ci". Remove options which
    became redundant with "--slow-ci".
  * "make testall" and "make testuniversal" now use --slow-ci option
    and TESTTIMEOUT variable.
  * "make testall" now uses "find -exec rm ..." instead of
    "find ... -print|xargs rm ...", same as "make clean".

* GitHub Actions workflow:

  * Ubuntu and Address Sanitizer jobs now use "make test". Remove
    options which became redundant with "--fast-ci".
  * Windows jobs now use --fast-ci option.
  * Use -j0 to detect the number of CPUs.

* Set Makefile TESTTIMEOUT default to an empty string, since
  --slow-ci and --fast-ci use different default timeout. It's now
  accepted to pass "--timeout=" to regrtest: treated as not timeout.
* Tools/scripts/run_tests.py now uses --fast-ci option.
* Tools/buildbot/test.bat now uses --slow-ci option. Remove
  --timeout=1200 option, redundant with --slow-ci.
csm10495 pushed a commit to csm10495/cpython that referenced this issue Sep 28, 2023
@vstinner @csm10495
pythongh-109566: regrtest reexecutes the process (python#109909)
fdd4e67 
When --fast-ci or --slow-ci option is used, regrtest now replaces the
current process with a new process to add "-u -W default -bb -E"
options to Python.

Changes:

* PCbuild/rt.bat and Tools/scripts/run_tests.py no longer need to add
  "-u -W default -bb -E" options to Python: it's now done by
  regrtest.
* Fix Tools/scripts/run_tests.py: flush stdout before replacing the
  process. Previously, buffered messages were lost.
csm10495 pushed a commit to csm10495/cpython that referenced this issue Sep 28, 2023
@vstinner @csm10495
pythongh-109566: Fix regrtest code adding Python options (python#109926)
b46c07a 
* On Windows, use subprocess.run() instead of os.execv().
* Only add needed options
* Rename reexec parameter to _add_python_opts.
* Rename --no-reexec option to --dont-add-python-opts.
csm10495 pushed a commit to csm10495/cpython that referenced this issue Sep 28, 2023
@vstinner @csm10495
pythongh-109566: Fix regrtest Python options for WASM/WASI (python#10…
f13088c 
…9954)

WASM and WASI buildbots use multiple PYTHON environment variables
such as PYTHONPATH and _PYTHON_HOSTRUNNER. Don't use -E if the
--python=COMMAND option is used.
csm10495 pushed a commit to csm10495/cpython that referenced this issue Sep 28, 2023
@vstinner @csm10495
pythongh-109566: regrtest doesn't enable --rerun if --python is used (p…
5e4b590 
…ython#109969)

regrtest: --fast-ci and --slow-ci options no longer enable --rerun if
the --python option is used.
vstinner added a commit to vstinner/cpython that referenced this issue Sep 29, 2023
@vstinner
pythongh-109566: regrtest _add_python_opts() handles KeyboardInterrupt
d1c1a3a 
In the subprocess code path, wait until the child process completes
with a timeout of EXIT_TIMEOUT seconds.

Fix create_worker_process() regression: use start_new_session=True if
USE_PROCESS_GROUP is true.

WorkerThread.wait_stopped() uses a timeout of 60 seconds, instead of
30 seconds.
@bedevere-app bedevere-app bot mentioned this issue Sep 29, 2023
Merged
vstinner added a commit to vstinner/cpython that referenced this issue Sep 29, 2023
@vstinner
pythongh-109566: regrtest _add_python_opts() handles KeyboardInterrupt
126a502 
In the subprocess code path, wait until the child process completes
with a timeout of EXIT_TIMEOUT seconds.

Fix create_worker_process() regression: use start_new_session=True if
USE_PROCESS_GROUP is true.

WorkerThread.wait_stopped() uses a timeout of 60 seconds, instead of
30 seconds.
vstinner added a commit that referenced this issue Sep 29, 2023
@vstinner
gh-109566: regrtest _add_python_opts() handles KeyboardInterrupt (#11…
235aacd 
…0062)

In the subprocess code path, wait until the child process completes
with a timeout of EXIT_TIMEOUT seconds.

Fix create_worker_process() regression: use start_new_session=True if
USE_PROCESS_GROUP is true.

WorkerThread.wait_stopped() uses a timeout of 60 seconds, instead of
30 seconds.
@vstinner
Copy link
Member Author

Python options and regrtest options are now handled by new --fast-ci and --slow-ci options of regrtest.

vstinner added a commit to vstinner/cpython that referenced this issue Sep 29, 2023
@vstinner
pythongh-109566: PCbuild/rt.bat now uses --fast-ci
292fa12 
Replace "--fail-env-changed --fail-rerun" with "--fast-ci".

Tools/buildbot/test.bat pass --slow-ci which has the priority over
--fast-ci.
@bedevere-app bedevere-app bot mentioned this issue Sep 29, 2023
Merged
vstinner added a commit to vstinner/cpython that referenced this issue Sep 29, 2023
@vstinner
pythongh-109566: regrtest --fast-ci no longer enables --nowindow
6efcff1 
The --nowindow option is deprecated and does nothing but logs a
warning.
@bedevere-app bedevere-app bot mentioned this issue Sep 29, 2023
Merged
vstinner added a commit to vstinner/cpython that referenced this issue Sep 29, 2023
@vstinner
pythongh-109566: Remove make testall
8725c01 
Remove "make testall" target: use "make buildbottest" instead.
@bedevere-app bedevere-app bot mentioned this issue Sep 29, 2023
Merged
vstinner added a commit that referenced this issue Sep 29, 2023
@vstinner
gh-109566: PCbuild/rt.bat now uses --fast-ci (#110120)
f3df8fa 
Replace "--fail-env-changed --fail-rerun" with "--fast-ci".

Tools/buildbot/test.bat pass --slow-ci which has the priority over
--fast-ci.
vstinner added a commit that referenced this issue Sep 29, 2023
@vstinner
gh-109566: Remove make testall (#110122)
9c91141 
Remove "make testall" target: use "make buildbottest" instead.
vstinner added a commit that referenced this issue Sep 29, 2023
@vstinner
gh-109566: regrtest --fast-ci no longer enables --nowindows (#110121)
5ae6c6d 
The --nowindows option is deprecated and does nothing but logs a
warning.
Glyphack pushed a commit to Glyphack/cpython that referenced this issue Sep 2, 2024
@vstinner @Glyphack
pythongh-109566: Fix typo in PCbuild/rt.bat (python#109701)
4d56535
Glyphack pushed a commit to Glyphack/cpython that referenced this issue Sep 2, 2024
@vstinner @Glyphack
pythongh-109566, regrtest: Add --fast-ci and --slow-ci options (pytho…
693a65b 
…n#109570)

* Add --fast-ci and --slow-ci options to libregrtest:

  * --fast-ci uses a default timeout of 10 minutes and "-u all,-cpu"
    (skip slowest tests).
  * --slow-ci uses a default timeout of 20 minues and "-u all" (run
    all tests).

* regrtest header now lists test resources.
* Makefile changes:

  * "make test", "make hostrunnertest" and "make coverage-report" now
    use --fast-ci option and TESTTIMEOUT variable.
  * "make buildbottest" now uses "--slow-ci". Remove options which
    became redundant with "--slow-ci".
  * "make testall" and "make testuniversal" now use --slow-ci option
    and TESTTIMEOUT variable.
  * "make testall" now uses "find -exec rm ..." instead of
    "find ... -print|xargs rm ...", same as "make clean".

* GitHub Actions workflow:

  * Ubuntu and Address Sanitizer jobs now use "make test". Remove
    options which became redundant with "--fast-ci".
  * Windows jobs now use --fast-ci option.
  * Use -j0 to detect the number of CPUs.

* Set Makefile TESTTIMEOUT default to an empty string, since
  --slow-ci and --fast-ci use different default timeout. It's now
  accepted to pass "--timeout=" to regrtest: treated as not timeout.
* Tools/scripts/run_tests.py now uses --fast-ci option.
* Tools/buildbot/test.bat now uses --slow-ci option. Remove
  --timeout=1200 option, redundant with --slow-ci.
Glyphack pushed a commit to Glyphack/cpython that referenced this issue Sep 2, 2024
@vstinner @Glyphack
pythongh-109566: regrtest reexecutes the process (python#109909)
48a50c9 
When --fast-ci or --slow-ci option is used, regrtest now replaces the
current process with a new process to add "-u -W default -bb -E"
options to Python.

Changes:

* PCbuild/rt.bat and Tools/scripts/run_tests.py no longer need to add
  "-u -W default -bb -E" options to Python: it's now done by
  regrtest.
* Fix Tools/scripts/run_tests.py: flush stdout before replacing the
  process. Previously, buffered messages were lost.
Glyphack pushed a commit to Glyphack/cpython that referenced this issue Sep 2, 2024
@vstinner @Glyphack
pythongh-109566: Fix regrtest code adding Python options (python#109926)
8c9872c 
* On Windows, use subprocess.run() instead of os.execv().
* Only add needed options
* Rename reexec parameter to _add_python_opts.
* Rename --no-reexec option to --dont-add-python-opts.
Glyphack pushed a commit to Glyphack/cpython that referenced this issue Sep 2, 2024
@vstinner @Glyphack
pythongh-109566: Fix regrtest Python options for WASM/WASI (python#10…
7ffa11f 
…9954)

WASM and WASI buildbots use multiple PYTHON environment variables
such as PYTHONPATH and _PYTHON_HOSTRUNNER. Don't use -E if the
--python=COMMAND option is used.
Glyphack pushed a commit to Glyphack/cpython that referenced this issue Sep 2, 2024
@vstinner @Glyphack
pythongh-109566: regrtest doesn't enable --rerun if --python is used (p…
676ae9a 
…ython#109969)

regrtest: --fast-ci and --slow-ci options no longer enable --rerun if
the --python option is used.
Glyphack pushed a commit to Glyphack/cpython that referenced this issue Sep 2, 2024
@vstinner @Glyphack
pythongh-109566: regrtest _add_python_opts() handles KeyboardInterrupt (
262be29 
python#110062)

In the subprocess code path, wait until the child process completes
with a timeout of EXIT_TIMEOUT seconds.

Fix create_worker_process() regression: use start_new_session=True if
USE_PROCESS_GROUP is true.

WorkerThread.wait_stopped() uses a timeout of 60 seconds, instead of
30 seconds.
Glyphack pushed a commit to Glyphack/cpython that referenced this issue Sep 2, 2024
@vstinner @Glyphack
pythongh-109566: PCbuild/rt.bat now uses --fast-ci (python#110120)
3dc9aa1 
Replace "--fail-env-changed --fail-rerun" with "--fast-ci".

Tools/buildbot/test.bat pass --slow-ci which has the priority over
--fast-ci.
Glyphack pushed a commit to Glyphack/cpython that referenced this issue Sep 2, 2024
@vstinner @Glyphack
pythongh-109566: Remove make testall (python#110122)
f3ec973 
Remove "make testall" target: use "make buildbottest" instead.
Glyphack pushed a commit to Glyphack/cpython that referenced this issue Sep 2, 2024
@vstinner @Glyphack
pythongh-109566: regrtest --fast-ci no longer enables --nowindows (py…
421d151 
…thon#110121)

The --nowindows option is deprecated and does nothing but logs a
warning.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
tests Tests in the Lib/test dir
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@vstinner