SLURM Training Jobs on HPC Clusters
An HPC training run combines a resource request, executable batch script, recorded environment, and durable outputs.
Combinatorial structure, time series, transit flows, and compute systems sharing one visual grammar.
Interactive model
Job eligibility under resource constraints
Change GPUs and wall time to see which jobs can fit a scheduler request.
Live HTML simulation · adjust the controls and watch the computed output respond.
Interactive
Schedulers fit jobs into finite GPU, memory, and wall-time budgets
This is a simplified teaching model. Its displayed values are computed from the controls; the article explains where the model stops.
Site connection
The SLM research project used Rutgers Amarel HPC and SLURM batch scripts to distribute training jobs across multiple nodes; the operational patterns here are general Slurm guidance, not a recovered project script.
Illustrative general Slurm script—not the project's original script:
#!/bin/bash
#SBATCH --job-name=slm-sft
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --gres=gpu:2
#SBATCH --time=04:00:00
#SBATCH --mem=64G
#SBATCH --output=logs/%x-%j.out
set -euo pipefail
module load cuda # cluster-specific
srun python train.py --config configs/gsm8k-sft.yml
Directive names, GPU syntax, modules, partitions, accounts, launchers, and storage paths are site-specific. Confirm them in the target cluster's documentation.
Allocation, Step, and Process
`sbatch` submits a script and requests a job allocation. Slurm evaluates that request against partitions, limits, priority, and available resources. Once allocated, the script runs; `srun` can launch one or more job steps inside the allocation and propagate task placement according to the site's configuration.
A node request is not the same as distributed training. The training framework must initialize ranks, coordinate workers, shard data or state, and communicate across nodes. Requesting two nodes while launching one ordinary Python process does not automatically use both nodes.
| Layer | Question | Typical evidence |
|---|---|---|
| Submission | What resources are requested? | Batch directives and `sbatch` output |
| Allocation | What did Slurm grant? | Job ID, node list, TRES, start/end state |
| Launch | Which processes ran where? | `srun`/launcher configuration and rank logs |
| Training | What experiment executed? | Commit, config, seed, data version |
| Artifacts | What survived the job? | Logs, metrics, checkpoints, provenance manifest |
Right-Sizing the Resource Request
Request CPUs, memory, accelerators, nodes, tasks, and wall time from measurements rather than aspiration. Too little memory or time can terminate a run; excessive requests can reduce scheduling opportunities. Slurm's official `sbatch` manual notes that a time request above a partition's limit can leave a job pending.
GPU flags and meanings depend on cluster configuration. `--gres=gpu:2` is common but not universally sufficient; partitions, GPU types, TRES settings, accounts, and binding rules may also be required. Measure utilization and distinguish per-node from per-task quantities.
Reproducibility and Durable State
A reproducible run records the Git commit and dirty-state patch, full resolved configuration, dataset identity and preprocessing, random seeds, container or package environment, hardware, Slurm job ID, and checkpoint/log destinations. Standard output alone is rarely a complete experiment record.
Use durable project or scratch storage according to local policy, and write checkpoints atomically when possible. Node-local temporary storage may be fast but can disappear after the allocation. Capture exit status and scheduler accounting so failed, timed-out, cancelled, or out-of-memory runs cannot be mistaken for completed experiments.
| Record | Why it matters |
|---|---|
| Git commit + dirty state | Reconstructs the executed code |
| Resolved config | Captures effective hyperparameters |
| Dataset/preprocessing version | Prevents silent input drift |
| Environment | Explains library, driver, and CUDA behavior |
| Job ID and accounting | Links the run to scheduler state and resources |
| Checkpoint manifest | Identifies durable recovery points |
Worked Example
Illustrative numbers: a one-GPU pilot processes 8 samples per second, uses 37 GB of a 40 GB GPU, peaks at 46 GB host memory, and finishes one epoch in 95 minutes. A production request might reserve one GPU, 56–64 GB host memory, and two hours plus measured checkpoint overhead. These numbers are pedagogical and do not describe the SLM project's actual run.
If the pilot exits at 01:54 under a two-hour limit, increasing directly to 12 hours hides uncertainty. First inspect where time was spent, checkpoint before the limit, and estimate a bounded margin. After completion, compare requested versus consumed resources through scheduler accounting and update the next request.
| Observation | Decision | Reason |
|---|---|---|
| 37/40 GB GPU memory | Keep one 40 GB-class GPU or reduce batch size | Little memory headroom |
| 46 GB host-memory peak | Request measured headroom, e.g. 56–64 GB | Avoid host OOM without gross overrequest |
| 95-minute epoch | Budget training plus validation/checkpoint time | Wall time covers the entire allocation |
| Exit near limit | Checkpoint and profile before a large time increase | Separates true compute from stalls |
Arrays, Dependencies, and Failure Recovery
Job arrays efficiently submit repeated runs that share initial resource options. Slurm sets `SLURM_ARRAY_TASK_ID` for each array element, which can index a manifest of seeds or configurations. A concurrency cap such as `--array=0-15%4` limits simultaneously running elements.
Arrays are not a replacement for multi-process distributed training: each element is a separate batch job. Dependencies can order preprocessing, training, and evaluation, while restartable checkpoints make timeout or preemption survivable. Each stage should validate required inputs before consuming expensive accelerators.
Project Context, Limits, and Common Misconceptions
The portfolio source supports these project claims: the work ran training and evaluation on Rutgers Amarel, used SLURM batch scripts across multiple nodes, adapted `nanochat`, and explored SFT, quantization, structural pruning, and GSM8k performance. It does not expose exact directives, node counts, GPU models, commands, numerical gains, or environment versions; the example script and numbers above are explicitly illustrative.
A queued job is not necessarily broken, a submitted job is not necessarily reproducible, and multiple GPUs do not guarantee a speedup. Queue time depends on site policy and availability; scaling depends on communication, workload, data pipeline, and framework configuration. Always follow the target cluster's documentation over generic examples.
Common Pitfalls
- Treating a resource allocation as proof that training uses every allocated device.
- Copying GPU, module, partition, or account directives without checking cluster-specific policy.
- Requesting resources without measuring actual consumption and runtime.
- Training from an unrecorded code, data, or environment state.
- Writing the only checkpoint to node-local temporary storage.
- Counting failed, cancelled, or timed-out runs as completed results.