DeltaAI

Description

The DeltaAI resource comprises 114 NVIDIA quad Grace Hopper nodes interconnected by HPE's Slingshot interconnect. Each Grace Hopper node consists of four NVIDIA super chips with one ARM based CPU, 128 GB of LP-DDR5 RAM and one H100 GPU with 96GB of HBM. The four super chips are tightly coupled with NVLink and share a unified shared memory space.

Resource ID
2418
Global Resource ID
deltaai.ncsa.access-ci.org
Resource Type
Compute
Latest Status
production
Latest Status Begin
Latest Status End
Project Affiliation
ACCESS
Organization Name
National Center for Supercomputing Applications
RP Description

DeltaAI is a GPU compute cluster consisting of 152 nodes built on the NVIDIA GH200 superchip, each node holding four superchips that pair an NVIDIA H100-class GPU with a Grace ARM CPU. It is particularly well suited for GPU-bound work that benefits from high-bandwidth CPU-GPU memory, and is often used for large model training, fine-tuning, inference, and GPU-accelerated simulation. It includes a great deal of AI and machine learning software, with ARM-native builds of PyTorch, TensorFlow, JAX, CUDA and NGC containers.

MFA Required
On
External Storage

Helpful commands: quota shows your usage against each file system quota. accounts maps your ACCESS projects to local DeltaAI projects and shows your SU balance.

When an allocation ends you get 30 days of data-management-only access to move data off.

For long-term archival, the paired tape archive is Granite - see [Granite User Documentation].

File Transfer Text

For transfers under roughly 100 GB, use scp or rsync. For larger transfers, or transfers with many files, use Globus - the DeltaAI collection for ACCESS users is ACCESS Delta. DeltaAI has no separate data transfer node - command-line transfers go to the login host, dtai-login.delta.ncsa.illinois.edu.

File Transfer Methods
Transfer Method
Globus
Data Transfer Node / Globus Collection
ACCESS Delta
Transfer Method
SCP
Data Transfer Node / Globus Collection
dtai-login.delta.ncsa.illinois.edu
Transfer Method
RSYNC
Data Transfer Node / Globus Collection
dtai-login.delta.ncsa.illinois.edu
Storage Text

DeltaAI users have Home, Projects, Work HDD and Work NVMe file systems, plus node-local storage at /tmp. /work/hdd and /work/nvme are shared between Delta and DeltaAI; home directories are not.

DeltaAI does not provide a /scratch file system. Scripts that hard-code /scratch must be rewritten to use /work/hdd, /work/nvme, or /tmp.

Lustre and VAST handle very large numbers of small files poorly. Pack many small files into tar, HDF5, or similar archives before writing them to /work.

Jobs Information

You can run jobs at different sizes and durations on DeltaAI. The following lists the different queues that you can submit to, describing how many nodes you get, how long you can run, the type of resources you get, and the average wait time.

Jobs are submitted through the Slurm scheduler. Run batch jobs with sbatch, launch interactive work with srun or salloc, or access compute resources through Open OnDemand applications such as JupyterLab.

Node sharing is the default behavior. The smallest allocatable unit is a single GH200 superchip - roughly one GPU, 72 CPU cores, and a quarter of the node's memory - so jobs do not need to request a whole node. Users who need exclusive access can request all consumable resources for the node, or use Slurm options such as --exclusive --mem=0. All nodes are of a single type and any job can span any node.

Jobs are charged on the resources reserved, not the resources used, and are charged in GPU Hours. The charge is sized by the number of superchips needed to satisfy whichever of your GPU, CPU-core or memory request is largest, so asking for extra memory or cores alongside one GPU can raise the charge. The ghx4-interactive queue is charged at a higher rate than ghx4, takes at most 4 nodes per job, and limits each user to 8 node-hours of running jobs, so use it for development and testing rather than production runs. See [DeltaAI Job Accounting] for current rates.

DeltaAI's CPUs are ARM (aarch64), not x86. Binaries compiled for x86, including binaries built on Delta, will not run here - rebuild for aarch64 or use an ARM-native container. Containers run under Apptainer, not Docker, and images must be linux/arm64.

If you do not specify them, jobs default to a 30-minute wall-clock limit and 1000 MB of memory per core. Jobs are not automatically requeued or restarted by default; add --requeue to enable it.

For more information about running jobs on DeltaAI, see [DeltaAI Running Jobs].

Storage Filesystems
Directory
Home
File System Path
$HOME
Quota size
100
Quota inode amount
750000
Purge Policy
Never purged
Backup Policy
Snapshots only, retained 30 days
Notes
Code, scripts, and job files. Not intended for I/O during jobs. Snapshots live on the same hardware as the primary copy and are not a backup - back up anything you cannot lose.
Directory
Projects
File System Path
/projects/<project>
Quota size
500
Quota inode amount
300000
Purge Policy
Never purged
Backup Policy
Not backed up
Notes
Shared project data, common data sets, software, and results. Up to 25 TB by allocation request.
Directory
Work HDD
File System Path
/work/hdd/<project>/<ncsa-username>
Quota size
1000
Quota inode amount
500000
Purge Policy
Never purged
Backup Policy
Not backed up
Notes
Where job I/O should occur. Best for large sequential reads and writes. Up to 100 TB by allocation request. Shared with Delta.
Directory
Work NVMe
File System Path
/work/nvme
Quota size
1000
Quota inode amount
500000
Purge Policy
Never purged
Backup Policy
Not backed up
Notes
Where job I/O should occur. Best for many small reads and writes. Up to 100 TB by allocation request. Shared with Delta.
Directory
Node-local
File System Path
/tmp
Quota size
3900
Purge Policy
Purged at job end
Backup Policy
Not backed up
Notes
Locally attached NVMe for fast small-file I/O. 3.9 TB per node, with no quota enforced - the space is shared with any other jobs running on the same node, so less than 3.9 TB may be free. Copy anything you need to /work or /projects before the job ends.
Queue Specifications
Queue Name
ghx4
Purpose
Production runs. The default queue - use this unless you are testing.
CPU Type
NVIDIA Grace ARM (aarch64)
GPU Type
NVIDIA H100
GPU Count
4
GPU vRAM
96
CPU Count
288
Node RAM
480
Queue Name
ghx4-interactive
Purpose
Short development, debugging and testing runs. Capped per user and charged at a higher rate than ghx4.
CPU Type
NVIDIA Grace ARM (aarch64)
GPU Type
NVIDIA H100
GPU Count
4
GPU vRAM
96
CPU Count
288
Node RAM
480