add easyblock for RAxML #2180

lexming · 2020-09-16T17:45:41Z

This easyblock automatically builds those binaries of RAxML that

use optimizations available in the CPU micro-architecture of the host system (respects optarch settings)
have features compatible with the toolchain used in the build

Current easyconfigs of RAxML build one arbitrary binary of RAxML and create a symlink with the name raxmlHPC. This is a bad approach because the different binaries of RAxML have different requirements and features. For instance, binaries using MPI have a minimum number of trees to process

This is RAxML MPI Process Number: 0

WARNING: The number of threads is currently set to 0
You can specify the number of threads to run via -T numberOfThreads
NumberOfThreads must be set to an integer value greater than 1

RAxML, will now set the number of threads automatically to 2 !

Error: you are running the parallel MPI program but only want to compute one tree
For the MPI version you must specify a number of trees greater than 1 with the -# or -N option

lexming · 2020-09-16T17:50:58Z

Tests in easybuilders/easybuild-easyconfigs#11294

easybuild/easyblocks/r/raxml.py

lexming · 2023-02-27T09:45:17Z

Test reports

Auto-detection
Custom optarch

lexming · 2023-02-27T12:01:38Z

Test report by @lexming

Overview of tested easyconfigs (in order)

SUCCESS RAxML-8.2.12-GCC-11.3.0.eb
SUCCESS RAxML-8.2.12-gompi-2022a.eb

Build succeeded for 2 out of 2 (2 easyconfigs in total)
node300.hydra.os - Linux CentOS Linux 7.9.2009, x86_64, Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz (skylake_avx512), Python 3.6.8
See https://gist.github.com/c11b4d9e09219333dbd36fcf85a32ba3 for a full test report.

boegel · 2023-03-15T17:30:48Z

@lexming Some existing RAxML easyconfigs have raxmlHPC -v and raxmlHPC -h as sanity check commands, we can let this easyblock run those too?

boegel · 2023-03-16T09:37:05Z

easybuild/easyblocks/r/raxml.py

+from easybuild.tools.systemtools import get_cpu_features, HAVE_ARCHSPEC
+from easybuild.tools.toolchain.compiler import OPTARCH_GENERIC
+
+RAXML_BINARY_NAME = "raxmlHPC"


Not sure RAXML_BINARY_NAME is useful, since it's only used in one place currently?

boegel · 2023-03-16T09:37:23Z

easybuild/easyblocks/r/raxml.py

+
+RAXML_BINARY_NAME = "raxmlHPC"
+# Supported instruction sets ordered by priority (high to low) and related CPU features
+RAXML_INSTRUCT_SETS = ["AVX2", "AVX", "SSE3"]


What about AVX-512? If there's a reason no to include that, add a comment on it?
Where can the list of supported instruction sets be found (docs link? README?)

Also, out of curiosity, does it also support NEON (for aarch64)?
If someone cares enough the easyblock can be updated later to also cover that, doesn't need to be done here.

boegel · 2023-03-16T09:38:40Z

easybuild/easyblocks/r/raxml.py

+# Supported instruction sets ordered by priority (high to low) and related CPU features
+RAXML_INSTRUCT_SETS = ["AVX2", "AVX", "SSE3"]
+RAXML_CPU_FEATURES = {
+    "SSE3": ["sse3", "see4_1", "sse4_2"],


typo:

Suggested change

"SSE3": ["sse3", "see4_1", "sse4_2"],

"SSE3": ["sse3", "sse4_1", "sse4_2"],

boegel · 2023-03-16T09:40:17Z

easybuild/easyblocks/r/raxml.py

+        """RAxML easyblock constructor, define class variables."""
+        super(EB_RAxML, self).__init__(*args, **kwargs)
+
+        def filter_optarch_features(support_features):


Make this a function outside of the easyblock? Same for others below.

Advantage is that tests can be added easily to CI for these, and code becomes quite a bit easier to parse for humans

boegel · 2023-03-16T09:43:47Z

easybuild/easyblocks/r/raxml.py

+                return feature.lower() in host
+
+            try:
+                return any(f in get_cpu_features() for f in RAXML_CPU_FEATURES[feature])


Would be nicer with a single return as the end, I'm not a big fan of inline returns, see also https://realpython.com/python-return-statement/#remembering-the-return-value (but that's partially personal, I guess)

Maybe this function should be moved into framework, it's pretty generic... Same for filter_optarch_features

boegel · 2023-03-16T09:45:41Z

easybuild/easyblocks/r/raxml.py

+            # unrestricted optimization settings, set optimization level for host micro-architecture
+            cpu_features = [feat for feat in cpu_features if has_cpu_feature(feat)]
+            dbg_msg = "Enabling the following CPU optimizations for RAxML by autodetection: %s"
+            self.log.debug(dbg_msg, ", ".join(cpu_features))


Make this info log message?

boegel · 2023-03-16T09:46:12Z

easybuild/easyblocks/r/raxml.py

+            dbg_msg = "Enabling the following CPU optimizations for RAxML by autodetection: %s"
+            self.log.debug(dbg_msg, ", ".join(cpu_features))
+            # add generic build
+            cpu_features.append(None)


Can you clarify (in the comment) why this is done?

boegel · 2023-03-16T09:46:46Z

easybuild/easyblocks/r/raxml.py

+            parallel_features.extend(RAXML_PARALLEL_FEATURES["mpi"])
+
+        # List of builds to carry out
+        self.target_makefiles = list_filename_variants(cpu_features, parallel_features, "Makefile", "gcc", ".")


Is always using gcc OK? What if we're installing RAxML with Intel compilers?

boegel · 2023-03-16T09:48:04Z

easybuild/easyblocks/r/raxml.py

+            if not any(feature in mf for feature in RAXML_PARALLEL_FEATURES["mpi"]):
+                cc_opt = compiler_nompi
+            self.cfg["buildopts"] = '-f %s CC="%s" %s' % (mf, cc_opt, user_buildopts)
+            self.log.debug("Building RAxML makefile with %s: %s", cc_opt, mf)


log.info?

boegel · 2023-03-16T09:49:02Z

easybuild/easyblocks/r/raxml.py

+            "files": [os.path.join("bin", x) for x in self.target_bins],
+            "dirs": ["share/manual", "share/usefulScripts"],
+        }
+        super(EB_RAxML, self).sanity_check_step(custom_paths=custom_paths)


Also include a custom sanity check command like raxmlHPC -h?

akesandgren · 2023-10-23T08:52:26Z

@lexming this easyblock needs some attention...

add easyblock for RAxML

0e8b180

lexming added the new label Sep 16, 2020

lexming added this to the 4.x milestone Sep 16, 2020

remove easybuild.tools.environment from raxml

5b40659

lexming mentioned this pull request Sep 16, 2020

{bio}[GCC/9.3.0,gompi/2020a] RAxML v8.2.12 easybuilders/easybuild-easyconfigs#11294

Closed

boegel requested changes Sep 22, 2020

View reviewed changes

easybuild/easyblocks/r/raxml.py Outdated Show resolved Hide resolved

easybuild/easyblocks/r/raxml.py Outdated Show resolved Hide resolved

easybuild/easyblocks/r/raxml.py Outdated Show resolved Hide resolved

lexming added 4 commits September 23, 2020 17:09

fix formatting of raxml.py

f7afebd

Merge branch 'develop' into raxml

e2626a8

refactor CPU features detection and build variants in RAxML

71d922e

follow optarch settings in RAxML easyblock

e5f5114

lexming mentioned this pull request Feb 27, 2023

{bio}[GCC/11.3.0,gompi/2022a] RAxML v8.2.12 easybuilders/easybuild-easyconfigs#17433

Open

lexming added 2 commits February 27, 2023 01:05

fix optarch fallback chain in RAxML easyblock

8b33807

consider boolean optarch settings from toolchianopts in RAxML easyblock

7466ce8

boegel modified the milestones: 4.x, next release (4.7.1) Mar 1, 2023

boegel reviewed Mar 16, 2023

View reviewed changes

boegel modified the milestones: next release (4.7.1), release after 4.7.1 Mar 16, 2023

boegel modified the milestones: next release (4.7.2), release after 4.7.2 May 10, 2023

boegel modified the milestones: 4.7.3, release after 4.7.3 Jul 5, 2023

boegel modified the milestones: next release (4.8.1?), release after 4.8.1 Sep 3, 2023

boegel removed this from the next release (4.8.2) milestone Oct 27, 2023

boegel added this to the release after 4.8.2 milestone Oct 27, 2023

boegel modified the milestones: next release (4.9.0), release after 4.9.0 Dec 26, 2023

boegel modified the milestones: 4.9.1, release after 4.9.1 Apr 3, 2024

boegel modified the milestones: 4.9.2, release after 4.9.2 Jun 6, 2024

boegel modified the milestones: 4.9.3, release after 4.9.3 Sep 11, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add easyblock for RAxML #2180

add easyblock for RAxML #2180

lexming commented Sep 16, 2020 •

edited

Loading

lexming commented Sep 16, 2020

lexming commented Feb 27, 2023 •

edited

Loading

lexming commented Feb 27, 2023

boegel commented Mar 15, 2023

boegel Mar 16, 2023

boegel Mar 16, 2023

boegel Mar 16, 2023

boegel Mar 16, 2023

boegel Mar 16, 2023

boegel Mar 16, 2023

boegel Mar 16, 2023

boegel Mar 16, 2023

boegel Mar 16, 2023

boegel Mar 16, 2023

boegel Mar 16, 2023

akesandgren commented Oct 23, 2023

	"SSE3": ["sse3", "see4_1", "sse4_2"],
	"SSE3": ["sse3", "sse4_1", "sse4_2"],

add easyblock for RAxML #2180

Are you sure you want to change the base?

add easyblock for RAxML #2180

Conversation

lexming commented Sep 16, 2020 • edited Loading

lexming commented Sep 16, 2020

lexming commented Feb 27, 2023 • edited Loading

Test reports

lexming commented Feb 27, 2023

Overview of tested easyconfigs (in order)

boegel commented Mar 15, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

akesandgren commented Oct 23, 2023

lexming commented Sep 16, 2020 •

edited

Loading

lexming commented Feb 27, 2023 •

edited

Loading