Description
ACES is a Dell cluster with a rich accelerator testbed consisting of Intel Max GPUs (Graphics Processing Units), Intel FPGAs (Field Programmable Gate Arrays), NVIDIA H100 and A30 GPUs, NEC Vector Engines, NextSilicon co-processors, and Graphcore IPUs (Intelligence Processing Units). Researchers must be based in the US and associated with a US academic research institution.
The ACES cluster consists of compute nodes using a mix of the following processors:
Intel Xeon 8468 Sapphire Rapids processors
Intel Xeon Ice Lake 8352Y processors
Intel Xeon Cascade Lake 8268 processors
AMD Epyc Rome 7742 processors
The compute nodes are interconnected with NVIDIA NDR200 connections for MPI and access to the Lustre storage. The Intel Optane SSDs and all accelerators (except the Graphcore IPUs and NEC Vector Engines) are accessed using Liqid's composable infrustructre via PCIe (Peripheral Component Interconnect express) Gen4 and Gen5 fabrics.
RP Description
ACES is a composable compute testbed consisting of Intel Sapphire Rapids CPU nodes that draw accelerators on demand from a shared pool over a Liqid PCIe fabric (NVIDIA H100 and A30 GPUs, Intel Data Center GPU Max, FPGAs, and NextSilicon coprocessors), plus fixed nodes carrying Graphcore IPUs, NEC Vector Engines, and an NVIDIA Grace-Hopper superchip. It is particularly well suited for work that compares or combines accelerators rather than running on one fixed configuration, and is often used for AI and ML training, benchmarking on emerging hardware, and porting codes to new devices. It includes a broad AI/ML and scientific software stack, with interactive applications delivered through Open OnDemand.
Login Text
Sign in by following the [ACES Accessing] guide, or open the [ACES OnDemand Portal] web interface. Before your first sign-in:
- You need an active ACCESS account and an active ACES allocation.
- Log in with your ACCESS username - your ACCESS credentials are what you sign in with, and there is no separate Texas A&M account to create.
- Two-factor authentication applies.
- For the shell route, generate the certificate-signed key pair from the portal before you connect; a key you create locally is not accepted. That certificate is valid for 49 hours and has to be regenerated after it expires.
The ACES portal gives you browser-based access to files, a terminal, and interactive applications, and it needs no key setup; it is the route most users should take.
ACES does not accept a plain SSH connection. Traffic is routed through a jump host to the login node.
External Storage
Texas A&M HPRC offers storage beyond the ACES quotas: Google Drive and Microsoft OneDrive at 25 GB each, and HPRC Long Term Storage as a paid dedicated tier. The published terms are written for Texas A&M users, so confirm what you are eligible for if your only affiliation is an ACCESS allocation. Current options and rates are in [ACES Extra Storage Options].
What ACES itself keeps, and for how long, is set out in [ACES Key Policies].
File Transfer Text
Globus is the recommended route for large or many-file transfers: it retries and resumes on its own, and it runs against the ACES collection ACCESS TAMU ACES DTN rather than a login node.
For smaller copies, scp, sftp, rsync, rclone and gdown run on the ACES login node, login.aces.hprc.tamu.edu. ACES has no separate command-line data transfer node, and login nodes cut off long-running processes, so use Globus for anything large.
Setup instructions for every method are in [ACES File Transfer].
Storage Text
ACES gives you three spaces: a small Home for code, scripts, and compiling; Scratch for the files an active job reads and writes; and Project space shared across your group.
Only Home is backed up. Scratch and Project are not, and neither is meant for long-term storage, so keep your own copies of anything you cannot regenerate.
Every tier also carries a file-count limit alongside its capacity limit, and the file-count limit is often what you hit first.
For paths and quotas, see [ACES File Systems].
Jobs Information
You can run jobs at different sizes and durations on ACES. The following lists the different queues that you can submit to, describing how many nodes you get, how long you can run, the type of resources you get, and the average wait time.
Jobs are scheduled by Slurm. Each job picks a partition with --partition, which sets the hardware it runs on, and must request a core count and a walltime. Compute nodes in the main pool provide 96 cores and 512 GB of memory. If you give no walltime, the partition's default applies; run scontrol show partition <partition> on a login node to see the default and the maximum for that partition. For node and architecture details see the [ACES Hardware] page; for job files and submission commands see the [ACES Batch System] guide.
The pvc partition's nodes carry 2 to 8 Intel GPU Max cards, depending on the node.
The Grace-Hopper node (gh01) is not a batch partition. You reach it by connecting directly over SSH from an ACES login node.
The nec partition holds one node with 8 NEC Vector Engine Type 20B-P cards. Request cards with --gres=gpu:ve:N (1 to 8), interactively with srun --partition nec ... or in a batch script with #SBATCH --partition=nec.
ACES uses a composable Liqid PCIe fabric, so accelerators (H100, A30, PVC, FPGA, NextSilicon) are attached on demand to a shared pool of Sapphire Rapids compute nodes rather than being fixed to a partition. A node can belong to several partitions at once, so the per-queue node counts overlap and are not additive; they total more than the 110-node compute pool by design. For a live view, run sinfo -s -p <partition> or gpuavail on a login node.