Description
Voyager is a heterogeneous system designed to support complex deep learning AI workflows. The system features 42 Intel Habana Gaudi training nodes, each with 8 training processors (336 in total). Each training node has 512GB of memory and 6.4TB of node local NVMe storage. The Gaudi training processors feature specialized hardware units for AI, HBM2, and on-chip high-speed Ethernet. The on-chip ethernet ports are used in a non-blocking all-to-all network between processors on a node and the remaining ports are aggregated into 6 400G connections on each node that are plugged into a 400G Arista switch to provide scale out of network. Voyager also has two first-generation inference nodes, each with 8 inference processors (16 in total). In addition to the custom AI hardware, the system also has 36 Intel x86 processors compute nodes for general purpose computing and data processing. Voyager features 3PB of storage currently deployed as a Ceph filesystem.
RP Description
Voyager is an AI testbed system consisting of 42 Habana Gaudi training nodes with eight training processors each, two first-generation Habana inference nodes, 36 x86 compute nodes and a 400 gigabit-per-second Arista interconnect. It is particularly well suited for deep learning at scale, and is often used for training and inference in science and engineering research that depends on artificial intelligence. It includes a great deal of container-based machine learning software, with Habana-optimized TensorFlow and PyTorch images.
Login Text
Sign in by following the [Voyager System Access] guide. Before your first sign-in:
- You need to be on an approved Voyager project. Voyager is a testbed - SDSC gives access to selected projects, workshops and industry collaborations, requested from SDSC HPC Consulting at
consult@sdsc.edu, and PIs add and remove users by the same route. - Generate an SSH key pair before you connect and send the public key to
consult@sdsc.edu for your ~/.ssh/authorized_keys file. The guide above lists the key types SDSC accepts.
There is no Voyager password: the SSH key pair is the only credential SDSC documents, and there is no web portal and no Open OnDemand interface on Voyager.
Login nodes are for editing, compiling, submitting and checking jobs - not for production work and not for large or numerous file transfers. Run everything computationally demanding through Kubernetes.
File Transfer Text
Voyager has no data transfer node. SDSC lists Globus endpoints for /voyager/projects as coming soon and has not published an endpoint or collection name yet, so there is no Globus route today.
Until then, command-line transfers go through the login node login.voyager.sdsc.edu, which SDSC asks you not to use for large or numerous transfers. For a bulk move, contact SDSC HPC Consulting at consult@sdsc.edu first. See [Voyager Data Movement] for what SDSC has published so far.
Storage Text
Voyager has a small 200 GB Home for source code and configuration, a Ceph parallel file system at /voyager/ceph/users/ for active job data, and NFS project space at /voyager/projects/. Each node also has local NVMe scratch, which you mount as an emptyDir volume and which exists only while your pod is running on that node.
None of these file systems is backed up and there is no archival tier on Voyager, so anything you cannot lose has to be copied off the system yourself. SDSC limits each user to 2 million files on the Ceph file system. See [Voyager Storage] for the full description.
Jobs Information
You can run jobs at different sizes and durations on Voyager. The following lists the different queues that you can submit to, describing how many nodes you get, how long you can run, the type of resources you get, and the average wait time.
Voyager does not use Slurm. Work is scheduled by Kubernetes, so there is no sbatch and no squeue. Load the scheduler with module load kubernetes/voyager, describe the work in a YAML file, and submit it with kubectl apply -f. YAML is whitespace sensitive, so an example copied out of a web page often fails to parse.
The smallest unit you can ask for is a Pod, and accelerators are requested individually, so a job can take part of a node: ask for habana.ai/gaudi on the training nodes and habana.ai/goya on the inference nodes. Node type is chosen with the node selector brightcomputing.com/node-category, which takes goya or compute. A job that spans more than one node has to be an MPIJob, not a plain Job.
Work runs in containers, and the image has to carry the Habana drivers - a stock x86 container will not use the accelerators. SDSC publishes no charging rates for Voyager during its testbed phase.
There is no default node type; the pod spec decides. During the testbed phase every node category has a 48 hour maximum wall time, and a single job can use at most 42 Gaudi nodes, 2 inference nodes or 36 compute nodes. For the pod, job and MPIJob templates and the full kubectl reference see [Voyager Running Jobs].