Test if non CUDA builds are not added to accelorator path with jax #917

laraPPr · 2025-02-13T13:24:29Z

5 out of 86 required modules missing:



* absl-py/2.1.0-GCCcore-12.3.0 (absl-py-2.1.0-GCCcore-12.3.0.eb)

* pytest/7.4.2-GCCcore-12.3.0 (pytest-7.4.2-GCCcore-12.3.0.eb)

* pytest-xdist/3.3.1-GCCcore-12.3.0 (pytest-xdist-3.3.1-GCCcore-12.3.0.eb)

* ml_dtypes/0.3.2-gfbf-2023a (ml_dtypes-0.3.2-gfbf-2023a.eb)

* jax/0.4.25-gfbf-2023a-CUDA-12.1.1 (jax-0.4.25-gfbf-2023a-CUDA-12.1.1.eb)

eessi-bot · 2025-02-13T13:24:33Z

Instance eessi-bot-mc-aws is configured to build for:

architectures: x86_64/generic, x86_64/intel/haswell, x86_64/intel/sapphire_rapids, x86_64/intel/skylake_avx512, x86_64/amd/zen2, x86_64/amd/zen3, aarch64/generic, aarch64/neoverse_n1, aarch64/neoverse_v1
repositories: eessi.io-2023.06-software, eessi.io-2023.06-compat

eessi-bot · 2025-02-13T13:24:34Z

Instance eessi-bot-mc-azure is configured to build for:

architectures: x86_64/amd/zen4
repositories: eessi.io-2023.06-software, eessi.io-2023.06-compat

eessi-bot-casparvl · 2025-02-13T13:24:34Z

Instance eessi-bot-casparvl is configured to build for:

architectures: x86_64/amd/zen4, x86_64/amd/zen2
repositories: eessi.io-2023.06-software, eessi-hpc.org-2023.06-compat, eessi-hpc.org-2023.06-software, eessi.io-2023.06-compat

gpu-bot-ugent · 2025-02-13T13:24:34Z

Instance eessi-bot-vsc-ugent is configured to build for:

architectures: x86_64/amd/zen3
repositories: eessi-hpc.org-2023.06-compat, eessi.io-2023.06-software, eessi-hpc.org-2023.06-software, eessi.io-2023.06-compat

eessi-bot-trz42 · 2025-02-13T13:24:34Z

Instance trz42-GH200-jr is configured to build for:

architectures: aarch64/nvidia/grace
repositories: eessi.io-2023.06-software

laraPPr · 2025-02-13T13:25:30Z

bot: build instance:eessi-bot-vsc-ugent repo:eessi.io-2023.06-software accel:nvidia/cc80

eessi-bot · 2025-02-13T13:25:33Z

Updates by the bot instance eessi-bot-mc-aws (click for details)

received bot command build instance:eessi-bot-vsc-ugent repo:eessi.io-2023.06-software accel:nvidia/cc80 from laraPPr
- expanded format: build instance:eessi-bot-vsc-ugent repository:eessi.io-2023.06-software accelerator:nvidia/cc80
handling command build instance:eessi-bot-vsc-ugent repository:eessi.io-2023.06-software accelerator:nvidia/cc80 resulted in:
- no jobs were submitted

eessi-bot · 2025-02-13T13:25:34Z

Updates by the bot instance eessi-bot-mc-azure (click for details)

received bot command build instance:eessi-bot-vsc-ugent repo:eessi.io-2023.06-software accel:nvidia/cc80 from laraPPr
- expanded format: build instance:eessi-bot-vsc-ugent repository:eessi.io-2023.06-software accelerator:nvidia/cc80
handling command build instance:eessi-bot-vsc-ugent repository:eessi.io-2023.06-software accelerator:nvidia/cc80 resulted in:
- no jobs were submitted

eessi-bot-casparvl · 2025-02-13T13:25:34Z

Updates by the bot instance eessi-bot-casparvl (click for details)

account laraPPr has NO permission to send commands to the bot

gpu-bot-ugent · 2025-02-13T13:25:34Z

Updates by the bot instance eessi-bot-vsc-ugent (click for details)

received bot command build instance:eessi-bot-vsc-ugent repo:eessi.io-2023.06-software accel:nvidia/cc80 from laraPPr
- expanded format: build instance:eessi-bot-vsc-ugent repository:eessi.io-2023.06-software accelerator:nvidia/cc80
handling command build instance:eessi-bot-vsc-ugent repository:eessi.io-2023.06-software accelerator:nvidia/cc80 resulted in:
- submitted job 15445297, for details & status see Test if non CUDA builds are not added to accelorator path with jax #917 (comment)

eessi-bot-trz42 · 2025-02-13T13:25:34Z

Updates by the bot instance trz42-GH200-jr (click for details)

account laraPPr has NO permission to send commands to the bot

gpu-bot-ugent · 2025-02-13T13:25:56Z

New job on instance eessi-bot-vsc-ugent for CPU micro-architecture x86_64-amd-zen3 and accelerator nvidia/cc80 for repository eessi.io-2023.06-software in job dir /scratch/gent/vo/002/gvo00211/SHARED/jobs/2025.02/pr_917/15445297

date	job status	comment
Feb 13 13:25:54 UTC 2025	submitted	job id `15445297` awaits release by job manager
Feb 13 13:26:58 UTC 2025	released	job awaits launch by Slurm scheduler
Feb 13 13:29:02 UTC 2025	running	job `15445297` is running
Feb 13 15:05:06 UTC 2025	finished	😢 FAILURE (click triangle for details) Details ✅ job output file `slurm-15445297.out` ✅ no message matching `FATAL:` ❌ found message matching `ERROR:` ❌ found message matching `FAILED:` ❌ found message matching `required modules missing:` ❌ no message matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-x86_64-amd-zen3-1739456837.tar.gz` size: 6 MiB (6667595 bytes) entries: 1191 modules under 2023.06/software/linux/x86_64/amd/zen3/accel/nvidia/cc80/modules/all `absl-py/2.1.0-GCCcore-12.3.0.lua` `ml_dtypes/0.3.2-gfbf-2023a.lua` `pytest/7.4.2-GCCcore-12.3.0.lua` `pytest-xdist/3.3.1-GCCcore-12.3.0.lua` software under 2023.06/software/linux/x86_64/amd/zen3/accel/nvidia/cc80/software `absl-py/2.1.0-GCCcore-12.3.0` `ml_dtypes/0.3.2-gfbf-2023a` `pytest/7.4.2-GCCcore-12.3.0` `pytest-xdist/3.3.1-GCCcore-12.3.0` other under 2023.06/software/linux/x86_64/amd/zen3/accel/nvidia/cc80 no other files in tarball
Feb 13 15:05:06 UTC 2025	test result	😁 SUCCESS (click triangle for details) ReFrame Summary [ OK ] (1/1) EESSI_LAMMPS_lj %device_type=gpu %module_name=LAMMPS/2Aug2023_update2-foss-2023a-kokkos-CUDA-12.1.1 %scale=1_4_node /497af4b1 @BotBuildTests:x86_64_amd_zen3_nvidia_cc80+default P: perf: 4447.069 timesteps/s (r:0, l:None, u:None) [ PASSED ] Ran 1/1 test case(s) from 1 check(s) (0 failure(s), 0 skipped, 0 aborted) Details ✅ job output file `slurm-15445297.out` ❌ found message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`

laraPPr · 2025-02-13T13:41:32Z

@trz42 @ocaisa this looks like it is not doing what we expect it to do because it seems to be installing pytest-xdist in the accelerator path.

/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/Python/3.11.3-GCCcore-12.3.0/bin/python -m pip install --prefix=/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/accel/nvidia/cc80/software/pytest-xdist/3.3.1-GCCcore-12.3.0  --verbose  --no-deps  --ignore-installed  --no-index  --no-build-isolation  .

laraPPr · 2025-02-13T13:43:48Z

bot: build repo:eessi.io-2023.06-software arch:x86_64/amd/zen2 accel:nvidia/cc80

eessi-bot · 2025-02-13T13:43:51Z

Updates by the bot instance eessi-bot-mc-aws (click for details)

received bot command build repo:eessi.io-2023.06-software arch:x86_64/amd/zen2 accel:nvidia/cc80 from laraPPr
- expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2 accelerator:nvidia/cc80
handling command build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2 accelerator:nvidia/cc80 resulted in:
- submitted job 45927, for details & status see Test if non CUDA builds are not added to accelorator path with jax #917 (comment)

eessi-bot · 2025-02-13T13:43:51Z

Updates by the bot instance eessi-bot-mc-azure (click for details)

received bot command build repo:eessi.io-2023.06-software arch:x86_64/amd/zen2 accel:nvidia/cc80 from laraPPr
- expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2 accelerator:nvidia/cc80
handling command build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2 accelerator:nvidia/cc80 resulted in:
- no jobs were submitted

eessi-bot-casparvl · 2025-02-13T13:43:51Z

Updates by the bot instance eessi-bot-casparvl (click for details)

account laraPPr has NO permission to send commands to the bot

gpu-bot-ugent · 2025-02-13T13:43:51Z

Updates by the bot instance eessi-bot-vsc-ugent (click for details)

received bot command build repo:eessi.io-2023.06-software arch:x86_64/amd/zen2 accel:nvidia/cc80 from laraPPr
- expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2 accelerator:nvidia/cc80
handling command build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2 accelerator:nvidia/cc80 resulted in:
- no jobs were submitted

eessi-bot-trz42 · 2025-02-13T13:43:51Z

Updates by the bot instance trz42-GH200-jr (click for details)

account laraPPr has NO permission to send commands to the bot

eessi-bot · 2025-02-13T13:43:55Z

New job on instance eessi-bot-mc-aws for CPU micro-architecture x86_64-amd-zen2 and accelerator nvidia/cc80 for repository eessi.io-2023.06-software in job dir /project/def-users/SHARED/jobs/2025.02/pr_917/45927

date	job status	comment
Feb 13 13:43:54 UTC 2025	submitted	job id `45927` awaits release by job manager
Feb 13 13:44:41 UTC 2025	released	job awaits launch by Slurm scheduler
Feb 13 13:53:28 UTC 2025	running	job `45927` is running
Feb 13 14:18:15 UTC 2025	finished	😢 FAILURE (click triangle for details) Details ✅ job output file `slurm-45927.out` ✅ no message matching `FATAL:` ❌ found message matching `ERROR:` ❌ found message matching `FAILED:` ❌ found message matching `required modules missing:` ❌ no message matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-x86_64-amd-zen2-1739455531.tar.gz` size: 6 MiB (6661749 bytes) entries: 1191 modules under 2023.06/software/linux/x86_64/amd/zen2/accel/nvidia/cc80/modules/all `absl-py/2.1.0-GCCcore-12.3.0.lua` `ml_dtypes/0.3.2-gfbf-2023a.lua` `pytest/7.4.2-GCCcore-12.3.0.lua` `pytest-xdist/3.3.1-GCCcore-12.3.0.lua` software under 2023.06/software/linux/x86_64/amd/zen2/accel/nvidia/cc80/software `absl-py/2.1.0-GCCcore-12.3.0` `ml_dtypes/0.3.2-gfbf-2023a` `pytest/7.4.2-GCCcore-12.3.0` `pytest-xdist/3.3.1-GCCcore-12.3.0` other under 2023.06/software/linux/x86_64/amd/zen2/accel/nvidia/cc80 no other files in tarball
Feb 13 14:18:15 UTC 2025	test result	😢 FAILURE (click triangle for details) Reason EESSI test suite was not run, test step itself failed to execute. Details ✅ job output file `slurm-45927.out` ❌ found message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`

laraPPr · 2025-02-13T15:15:35Z

It failed building jax with this error:

FAILED: Installation ended unsuccessfully (build directory: /tmp/vsc48506/easybuild/build/jax/0.4.25/gfbf-2023a-CUDA-12.1.1): build failed (first 300 chars): Failed to determine installation prefix for binutils (took 39 mins 48 secs

and as you can see in the artifacts the non enabled cuda builds were build in the accelerator path.

Test is non CUDA builds are not add to accelorator path with jax

3a7498d

laraPPr added the accel:nvidia label Feb 17, 2025

laraPPr added 7 commits February 25, 2025 12:33

Create test-check_easystacks.yml

b008485

add the filter for changed easystack files

fb7ceae

fix syntax of yml file

c4b39d3

run check_missing_installation.sh in workflow

5b5434a

debugging checking missing installation step

b875d62

debugging getting changed file

9a981a5

debugging getting changed file

e8acea1

laraPPr and others added 19 commits February 25, 2025 14:07

trying to get the list of missing installations

10d2837

fix bash syntax

aedb5eb

debugging

951ad4a

debugging

561ee5e

debugging workflow

0151275

debugging

bff29c9

Don't stop workflow due to failed missing installations

a860642

get exit code of run check-missing-installations.sh

9d3f2fe

add script for parsing missing installations output

555a3d1

fix parse script

c7a7758

fix parse script

754faa4

fix parse script

41df446

add workflow to check the missing builds in the pr

6d1cd82

fix check easystack workflow

bf0e3fa

fix workflow

b4fff2e

debugging workflow

bbac63b

debugging workflow

e17f9da

clean-up workflow

9e29265

remove continue on error

8f15037

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Test if non CUDA builds are not added to accelorator path with jax #917

Test if non CUDA builds are not added to accelorator path with jax #917

laraPPr commented Feb 13, 2025

eessi-bot bot commented Feb 13, 2025

eessi-bot bot commented Feb 13, 2025

eessi-bot-casparvl bot commented Feb 13, 2025

gpu-bot-ugent bot commented Feb 13, 2025

eessi-bot-trz42 bot commented Feb 13, 2025

laraPPr commented Feb 13, 2025

eessi-bot bot commented Feb 13, 2025 •

edited

Loading

eessi-bot bot commented Feb 13, 2025 •

edited

Loading

eessi-bot-casparvl bot commented Feb 13, 2025

gpu-bot-ugent bot commented Feb 13, 2025 •

edited

Loading

eessi-bot-trz42 bot commented Feb 13, 2025

gpu-bot-ugent bot commented Feb 13, 2025 •

edited

Loading

laraPPr commented Feb 13, 2025 •

edited

Loading

laraPPr commented Feb 13, 2025

eessi-bot bot commented Feb 13, 2025 •

edited

Loading

eessi-bot bot commented Feb 13, 2025 •

edited

Loading

eessi-bot-casparvl bot commented Feb 13, 2025

gpu-bot-ugent bot commented Feb 13, 2025 •

edited

Loading

eessi-bot-trz42 bot commented Feb 13, 2025

eessi-bot bot commented Feb 13, 2025 •

edited

Loading

laraPPr commented Feb 13, 2025

Test if non CUDA builds are not added to accelorator path with jax #917

Are you sure you want to change the base?

Test if non CUDA builds are not added to accelorator path with jax #917

Conversation

laraPPr commented Feb 13, 2025

eessi-bot bot commented Feb 13, 2025

eessi-bot bot commented Feb 13, 2025

eessi-bot-casparvl bot commented Feb 13, 2025

gpu-bot-ugent bot commented Feb 13, 2025

eessi-bot-trz42 bot commented Feb 13, 2025

laraPPr commented Feb 13, 2025

eessi-bot bot commented Feb 13, 2025 • edited Loading

eessi-bot bot commented Feb 13, 2025 • edited Loading

eessi-bot-casparvl bot commented Feb 13, 2025

gpu-bot-ugent bot commented Feb 13, 2025 • edited Loading

eessi-bot-trz42 bot commented Feb 13, 2025

gpu-bot-ugent bot commented Feb 13, 2025 • edited Loading

laraPPr commented Feb 13, 2025 • edited Loading

laraPPr commented Feb 13, 2025

eessi-bot bot commented Feb 13, 2025 • edited Loading

eessi-bot bot commented Feb 13, 2025 • edited Loading

eessi-bot-casparvl bot commented Feb 13, 2025

gpu-bot-ugent bot commented Feb 13, 2025 • edited Loading

eessi-bot-trz42 bot commented Feb 13, 2025

eessi-bot bot commented Feb 13, 2025 • edited Loading

laraPPr commented Feb 13, 2025

eessi-bot bot commented Feb 13, 2025 •

edited

Loading

eessi-bot bot commented Feb 13, 2025 •

edited

Loading

gpu-bot-ugent bot commented Feb 13, 2025 •

edited

Loading

gpu-bot-ugent bot commented Feb 13, 2025 •

edited

Loading

laraPPr commented Feb 13, 2025 •

edited

Loading

eessi-bot bot commented Feb 13, 2025 •

edited

Loading

eessi-bot bot commented Feb 13, 2025 •

edited

Loading

gpu-bot-ugent bot commented Feb 13, 2025 •

edited

Loading

eessi-bot bot commented Feb 13, 2025 •

edited

Loading