Description
A virtual HTCondor pool made up of resources from the OSG Consortium
RP Description
OSPool is a distributed high-throughput computing pool consisting of compute capacity contributed by campuses, national laboratories and research collaborations across the US and presented as a single virtual cluster. It is particularly well suited for workloads that split into many independent single-node jobs, each of which completes or checkpoints within 20 hours, and is often used for parameter sweeps, image and sequence processing, statistical resampling and simulation ensembles. It runs user software packaged in Apptainer or Docker containers, including a maintained set of OSG-supported containers for AI frameworks, bioinformatics tools and the common scripting languages.
Login Text
Sign in by following the [OSPool Log In to uw.osg-htc.org Access Points] guide, or open the OSPool Notebooks web interface. Before your first sign-in:
- You need an OSPool account, not an ACCESS allocation - apply through [Start Here: Overview of Requesting OSPool Access]; approval follows an orientation meeting with an OSG facilitator. ACCESS credits can charge OSPool usage but do not grant access.
- Log in with your OSPool username, not your ACCESS ID. It and your access point -
ap40, ap41 or ap43 at uw.osg-htc.org - arrive in your welcome email. - Web authentication needs only a browser; key-based access needs your public key in your COmanage profile first.
Access points are for preparing and submitting work, not for running it. OSG applies no MFA of its own and there is no separate OSPool password: you sign in with institutional credentials through CILogon, which may require MFA. OSPool Notebooks gives anyone without an account a temporary access point.
File Transfer Text
Move files between your own computer and your access point with scp, or with a GUI client such as WinSCP, FileZilla or Cyberduck. There is no separate data transfer node: the access point named in your welcome email, one of the ap4#.uw.osg-htc.org hosts, is also the transfer host. Getting data from there to your jobs is a second step - HTCondor carries files under 1 GB from /home, OSDF carries larger files and containers, and anything already on the web can be pulled straight into a job by URL. See [OSPool Use scp To Transfer Files To and From OSG Managed Access Points] and [OSPool Transfer HTTP-available Files up to 1GB In Size].
Storage Text
An OSPool access point gives you three places to keep data: your /home directory, which is also the only directory you may submit jobs from; an OSDF data directory for input, output and container files larger than 1 GB; and an OSDF projects directory for data shared with everyone on your project.
None of it is backed up. OSG treats access point storage as temporary space for active work and reserves the right to remove data without notice, so keep the copy you cannot lose somewhere else. OSDF also caches files across the pool, which means a file you edit in place may not reach your jobs - give each version its own name or path. See [OSPool Overview: Data Staging and Transfer to Jobs] for the quota table and [Policies for Using OSG Services and the OSPool] for what OSG guarantees. Quota increases are requested with the [OSPool quota request form].
Jobs Information
OSPool jobs are scheduled by HTCondor, not Slurm. You describe a job in a plain-text submit file - what to run, what to bring back, and how much CPU, memory and disk it needs - and submit it with condor_submit from your /home directory, the only directory you may submit from. See [Overview: Submit Jobs to the OSPool using HTCondor].
There are no queues or partitions to choose between. HTCondor matchmaking sends each job to whichever contributed machine satisfies what it asked for, so you select hardware by stating requirements in the submit file - request_cpus, request_memory, request_disk - not by naming a partition. Jobs are single-node, and a bigger request matches fewer machines and waits longer. Most of the capacity is opportunistic: the site that owns a machine can interrupt your job at any time and HTCondor requeues it to start again from the beginning, so work that cannot tolerate a restart must checkpoint.
There is no per-queue wall clock either. Each job declares a +JobDurationCategory instead - Medium, expected under 10 hours and held after 20, or Long, expected under 20 and held after 40 - and Medium applies if you set nothing. An overrun job is held, not killed, so you can resubmit it as Long. See [OSPool Job Duration Categories].
Two more traps. Keep no more than 10,000 jobs in the queue at once unless you add max_idle to the submit file. And bring your software with you, in a container or as a portable binary: the machines your jobs land on belong to other sites and are not yours to install on. OSPool access does not consume an ACCESS allocation by default. Coming from Slurm, start with [OSPool Convert your workflow from Slurm to HTCondor].