Stampede2 User Guide (2022 under ACCESS)

Stampede2 is one of the Texas Advanced Computing Center (TACC), the University of Texas at Austin's flagship supercomputers. Stampede2 entered full production in the Fall of 2017 as an 18-petaflop national resource that builds on the successes of the original Stampede system it replaces. The first phase of the Stampede2 rollout featured the second generation of processors based on Intel's Many Integrated Core (MIC) architecture. Stampede2's 4,200 Knights Landing (KNL) nodes represent a radical break with the first-generation Knights Corner (KNC) MIC coprocessor. Unlike the legacy KNC, a Stampede2 KNL is not a coprocessor: each 68-core KNL is a stand-alone, self-booting processor that is the sole processor in its node. Phase 2 added to Stampede2 a total of 1,736 Intel Xeon Skylake (SKX) nodes. The final phase of Stampede2 features the replacement of 448 KNL nodes with 224 Ice Lake nodes. You may find useful information about Stapede2 in [Stampede2 User Guide]

1. Logging into Stampede2

Create an account

You will first need to create an ACCESS portal account here: ACCESS User Registration. If you already have an XSEDE or ACCESS account, please do not create another one. Select the “ACCESS CI (XSEDE)” identity provider to log on using your existing XSEDE/ACCESS account.

.. ... ...

2. Using Stampede2

Stampede2 nodes run Red Hat Enterprise Linux 7. Regardless of your research workflow, you'll need to master Linux basics and a Linux-based text editor (e.g. emacs, nano, gedit, or vi/vim) to use the system properly.

Configuring your account

Shell

The default login shell for your user account is Bash. To determine your current login shell, execute:

$ echo $SHELL

If you'd like to change your login shell to csh, sh, tcsh, or zsh, submit a ticket through the TACC User Portal. The "chsh" ("change shell") command will not work on TACC systems. When you start a shell on Stampede2, system-level startup files initialize your account-level environment and aliases before the system sources your own user-level startup scripts. You can use these startup scripts to customize your shell by defining your own environment variables, aliases, and functions. These scripts (e.g. .profile and .bashrc) are generally hidden files: so-called dotfiles that begin with a period, visible when you execute: "ls -a".

For more information see the Bash Users' Startup Files: Quick Start Guide that explain shell startup. To recover the originals that appear in a newly created account, execute "/usr/local/startup_scripts/install_default_scripts".

Environment Variables

Your environment includes the environment variables and functions defined in your current shell: those initialized by the system, those you define or modify in your account-level startup scripts, and those defined or modified by the modules that you load to configure your software environment. Use export (in Bash) or setenv (in csh) to define an environment variable.

Execute the "env" command to see the environment variables that define the way your shell and child shells behave.

Account-Level Diagnostics

TACC's _ sanitytool_ module loads an account-level diagnostic package that detects common account-level issues and often walks you through the fixes. You should certainly run the package's sanitycheck utility when you encounter unexpected behavior. To run sanitytool's account-level diagnostics, execute the following commands:

login1$ module load sanitytool

login1$ sanitycheck

Execute "module help sanitytool" for more information.

Accessing the Compute Nodes

You connect to Stampede2 through one of four "front-end" login nodes. The login nodes are shared resources: at any given time, there are many users logged into each of these login nodes, each preparing to access the "back-end" compute nodes. What you do on the login nodes affects other users directly because you are competing for the same memory and processing power. This is the reason you should not run your applications on the login nodes or otherwise abuse them. Think of the login nodes as a prep area where you can manage files and compile code before accessing the compute nodes to perform research computations. (See figure 1)
Figure 1. Login and compute node

You can use your command-line prompt, or the "hostname" command, to tell you whether you are on a login node or a compute node. The default prompt, or any custom prompt containing "\h", displays the short form of the hostname (e.g. c401-064). The hostname for a Stampede2 login node begins with the string "login" (e.g. login2.stampede2.tacc.utexas.edu), while compute node hostnames begin with the character "c" (e.g. c401-064.stampede2.tacc.utexas.edu)

There are three basic ways to access the compute nodes:

1. Submit a batch job using the sbatch command

2. Begin an interactive session using idev or srun

3. Begin an interactive session using ssh