Description Expanse is a Dell integrated compute cluster, with AMD Rome processors, 128 cores per node, interconnected with Mellanox HDR InfiniBand in a hybrid fat-tree topology. The compute node section of Expanse has a peak performance of 3.373 PF. Full bisection bandwidth is available at rack level (56 compute nodes) with HDR100 connectivity to each node. HDR200 switches are used at the rack level and 3:1 oversubscription cross-rack. Compute nodes feature 1TB of NVMe storage and 256GB of DRAM per node. The system also features 12PB of Lustre based performance storage (140GB/s aggregate), and 7PB of Ceph based object storage. Resource ID 637 Global Resource ID expanse.sdsc.access-ci.org Resource Type Compute Latest Status production Latest Status Begin Thu, 10/01/2020 - 12:00 Project Affiliation ACCESS Organization Name San Diego Supercomputer Center Organization URL https://www.sdsc.edu Organization Logo URL https://cider.access-ci.org/public/resource_providers/371/logo Detail View URL https://www.sdsc.edu/support/user_guides/expanse.html Raw Data URL https://operations-api.access-ci.org/wh2/cider/v1/cider_resource_id/637/ RP Description Expanse CPU is a compute cluster consisting of 728 nodes. It is particularly well suited for parallel workloads that scale across many nodes, and is often used for large MPI simulations and other tightly-coupled parallel work. MFA Required On RP Account Required Off Jobs Information Expanse uses the Simple Linux Utility for Resource Management (SLURM) batch environment. When you run in the batch mode, you submit jobs to be run on the compute nodes using the sbatch command as described in Expanse Running Jobs.Remember that computationally intensive jobs should be run only on the compute nodes and not the login nodes. Storage Filesystems Directory Scratch Lustre File System Path /expanse/lustre/scratch Quota (Deprecated) 10 TB Quota size 10000 Purge Policy 90 days after allocation expiration. Backup Policy Not backed up​ Notes This is not an archival file system, it is not backed up, and will be purged according to purge policy. Directory Scratch Compute Node File System Path /scratch/$USER/job_$SLURM_JOB_ID Quota (Deprecated) 1 TB Quota size 1000 Notes Users only have access to these SSDs during job execution at the local file system path to the compute node. Directory Home File System Path /home Quota (Deprecated) 100 GB Quota size 100 Purge Policy N/A Backup Policy 8 week rolling backup Notes The home directory is limited in space and should be used only for source code storage. Jobs should never be run from the home file system, as it is not set up for high performance throughput. Queue Specifications Queue Name compute Purpose Compute Node Usage CPU Type AMD EPYC 7742 RAM (Deprecated) 256 GB DDR4 DRAM CPU Count 128 Node RAM 256 Queue Name debug Purpose Priority access to shared nodes set aside for testing of jobs with short wall time and limited resources CPU Type AMD EPYC 7742 RAM (Deprecated) 256 GB DDR4 DRAM CPU Count 128 Node RAM 256 Queue Name preempt Purpose Non-refundable discounted jobs to run on free nodes that can be pre-empted by jobs submitted to any other queue CPU Type AMD EPYC 7742 RAM (Deprecated) 256 GB DDR4 DRAM CPU Count 128 Node RAM 256 Queue Name large-shared Purpose Single-node jobs using large memory up to 2 TB (minimum memory required 256G) CPU Type AMD EPYC 7742 RAM (Deprecated) 256 GB DDR4 DRAM CPU Count 128 Node RAM 2000 Queue Name shared Purpose Jobs using part of a single 256 GB node, sharing the node with other jobs. CPU Type AMD EPYC 7742 RAM (Deprecated) 256 GB DDR4 DRAM CPU Count 128 Node RAM 256 Datasets Dataset Name OceanTopography Dataset Description OpenTopography provides efficient, user-friendly access to high-resolution topography data, processing tools, and resources to advance understanding of the Earth's surface, vegetation, and built environment. Dataset Name OpenAltimetry Dataset Description OpenAltimetry is a web based data visualization and discovery tool for exploring surface elevation profiles over time using satellite altimetry data from NASA's ICESat and ICESat-2 missions. Dataset Name OpenForest4D Dataset Description OpenForest4D is a web-based platform that leverages multi-source remote sensing data and artificial intelligence to generate on-demand, research-grade estimates of forest structure and above-ground biomass in four dimensions for global forest monitoring. RP source documentation for Submitting Jobs https://www.sdsc.edu/systems/expanse/user_guide.html#narrow-wysiwyg-7 RP source documentation for Software https://www.sdsc.edu/systems/expanse/user_guide.html#narrow-wysiwyg-13