Description
Stampede3 is generously funded through the National Science Foundation and is designed to serve today's researchers as well as support the research community on an evolutionary path toward many-core processors and accelerated technologies. Stampede 3 maintains the familiar programming model for all of today's users, and thus will be broadly useful for traditional simulation users, users performing data intensive computations, and emerging classes of new users.
RP Description
Stampede3 is a compute cluster consisting of Intel Sapphire Rapids high-bandwidth-memory nodes, Ice Lake CPU nodes, large-memory Ice Lake NVDIMM nodes, NVIDIA H100 GPU nodes, Intel Data Center GPU Max nodes, and Skylake CPU nodes. It is particularly well suited for large-scale simulation, high-throughput computing, artificial intelligence, GPU-accelerated workloads, and large-memory applications, and is often used for groundbreaking open science research projects.
Login Text
Sign in by following the [Stampede3 Accessing the System] guide. Before your first sign-in:
- You need an active ACCESS allocation on Stampede3. Without one your password is rejected.
- Log in with your TACC username, not your ACCESS ID. Once your allocation on Stampede3 is approved you will find it under Resource Provider Site Usernames on the [ACCESS Allocations] page.
- Create a TACC account and set its password at the [TACC Accounts Portal] before you connect for the first time.
- Pair a device for [Multi-Factor Authentication at TACC] before you connect. TACC requires MFA on all of its compute and storage resources.
Login nodes are shared, and are for editing, compiling, submitting and checking jobs - not for production work. Do not run ssh-keygen on Stampede3: the key pair it creates interferes with job scripts in the batch system.
If a connection fails with Corrupted MAC on input, reconnect with ssh -m hmac-sha2-512 <tacc-username>@stampede3.tacc.utexas.edu.
External Storage
Every Stampede3 allocation comes with 2 TB on Ranch, TACC's long-term archival tape system, at no additional cost, and it does not consume ACCESS credits. Ranch is for final data that is unlikely to change and does not need frequent access. Move data to it with Globus, or with scp myfile ${ARCHIVER}:${ARCHIVE}/myfilepath. See [Ranch User Guide].
Projects that need more space than $HOME and $WORK provide can apply for Corral, TACC's online project storage system, which Stampede3 reaches through the same transfer methods. See [Corral User Guide].
File Transfer Text
Use [Globus Data Transfer Guide at TACC] for datasets larger than about 200 GB, and the SSH tools scp, rsync and sftp for anything smaller - see [SSH-based Data Transfer].
TACC does not publish a separate data transfer node for Stampede3: command-line transfers go to stampede3.tacc.utexas.edu, the same host you log in to. Keep those transfers small, because the login nodes are shared, and move large data with Globus instead.
Storage Text
Stampede3 has three file systems: a small, backed-up $HOME for source code and configuration, $WORK on TACC's shared Stockyard file system for data you want to keep, and a large $SCRATCH for job output. Run your jobs out of $SCRATCH rather than $HOME.
Two things surprise people. The 1 TB $WORK quota is shared across every TACC system you use, not granted separately on each one, and files on $SCRATCH are purged once they have not been accessed for ten days. Only $HOME is backed up. See [Stampede3 Managing Your Files] for the details.
Jobs Information
You can run jobs at different sizes and durations on Stampede3. The following lists the different queues that you can submit to, describing how many nodes you get, how long you can run, the type of resources you get, and the average wait time.
Stampede3 uses the Slurm scheduler and supports multi-node and MPI jobs. Nothing is assumed for you: your job script must set --partition, --time, --account and --ntasks explicitly. See [Stampede3 Running Jobs] for submission commands, sample job scripts and job management.
TACC does not share nodes between users on any of its compute resources, so the smallest thing you can request is a whole node and a job that uses one core is charged for all of it. There is no shared or interactive partition; skx-dev is the development queue and is capped at 16 nodes and 2 hours, while every other queue allows up to 48 hours.
Stampede3 is five different node types behind one scheduler, so choosing a queue is choosing the hardware. The spr nodes carry 128 GB of on-package high-bandwidth memory and no DDR, so work that needs more memory per core belongs on icx or skx; nvdimm is single-node only and exists for jobs that fit nowhere else; and the two GPU queues take at most 4 nodes per job, with pvc providing Intel Data Center GPU Max GPUs rather than NVIDIA ones, so CUDA binaries will not run there. Per-queue limits are listed in [Stampede3 Production Queues].
Jobs are charged in node-hours on the nodes reserved, not on the cores used, and the rate differs by queue. Run /usr/local/etc/taccinfo on a login node to see your project balances and disk quotas, and see [Stampede3 Job Accounting] for the billing formula and current rates.