5 個影響 HPC 叢集的 Slurm 設定參數

When building an HPC cluster, we often focus on compute nodes, CPUs, RAM, GPUs, networking and storage.

But there is another important part of the design: how Slurm manages those resources.

A cluster might have hundreds of CPU cores and several GPUs, but how effectively users access those resources depends partly on how Slurm is configured.

Let's explore five important configuration areas in slurm.conf and understand their impact on an HPC environment.

1. SelectType and SelectTypeParameters

These parameters control how Slurm selects resources and allocates them to jobs.

For example:

SelectType=select/cons_tres
SelectTypeParameters=CR_Core

select/cons_tres allows Slurm to track and allocate consumable resources, such as CPUs, memory and GPUs when configured appropriately.

SelectTypeParameters determines the granularity at which resources are allocated. For example, CR_Core allocates resources at the CPU core level, while CR_CPU allocates at the CPU level as Slurm defines it. CR_Socket allocates at socket granularity, and CR_Memory focuses on memory as a consumable resource. The choice affects how precisely resources can be allocated and shared, potentially influencing resource utilisation and job placement.

Infrastructure impact: This setting affects how compute node resources are shared between jobs. The best option depends on the CPU topology, workload requirements and intended allocation policy.

Technical note: CR_CPU, CR_Core and CR_Socket are not interchangeable on every system. Their effects depend on the CPU topology and Slurm configuration. CR_Memory is a memory focused allocation option, not simply another CPU allocation granularity. Supported combinations also depend on the selected resource plugin and Slurm version, so check the documentation for your installed version before changing production settings.

2. PriorityWeight and Job Scheduling Priority

When multiple jobs are waiting in the queue, Slurm needs to determine their scheduling priorities.

With multifactor priority scheduling, administrators can assign weights to different factors that influence job priority.

For example:

PriorityType=priority/multifactor
PriorityWeightAge=1000
PriorityWeightFairshare=10000
PriorityWeightJobSize=100

Here, the weights determine how much the corresponding factors contribute to a job's priority.

• PriorityWeightAge gives weight to how long a job has been waiting.
• PriorityWeightFairshare gives weight to fairshare usage, helping account for how much cluster resources users or projects have consumed.
• PriorityWeightJobSize gives weight to job size.

Why does this matter?

Imagine a research cluster shared by several teams. One team regularly submits large simulations, while another runs many smaller jobs.

Priority weights help administrators influence how waiting time, resource usage and job size affect scheduling decisions.

Infrastructure impact: Priority policies influence how users access shared compute capacity and can help balance competing workloads. They do not create additional resources or guarantee that a particular job runs first. Actual scheduling also depends on available resources, job constraints, reservations, QoS and other policies.

The weights should reflect the cluster's operational goals rather than being chosen arbitrarily.

3. PartitionName

A partition is a group of compute nodes that Slurm makes available under a particular set of scheduling policies.

For example:

PartitionName=cpu Nodes=cpu[01-08] \
    Default=YES MaxTime=24:00:00 State=UP

PartitionName=gpu Nodes=gpu[01-04] \
    Default=NO MaxTime=48:00:00 State=UP

In this example, the cluster has two partitions:

• cpu: eight CPU compute nodes.
• gpu: four GPU compute nodes.

A user can submit a job to the GPU partition with:

sbatch --partition=gpu job.sh

Why not put every node into one partition?

Different workloads have different requirements. A simulation might need many CPU cores, while an AI workload might require GPUs. Some applications need large amounts of RAM, and others have shorter execution times.

Partitions allow administrators to organise resources and apply different scheduling limits to different groups of nodes.

Infrastructure impact: Partition design influences which types of compute nodes are needed, how workloads are separated and how resources are made available to users.

A partition does not automatically guarantee a particular performance level or priority. Those behaviours depend on the wider scheduling configuration and policies.

4. MaxTime

Jobs can run for seconds, hours or several days. Slurm needs a way to limit how long a job is allowed to run.

For example:

PartitionName=cpu Nodes=cpu[01-08] \
    Default=YES MaxTime=24:00:00 State=UP

MaxTime=24:00:00 sets the maximum permitted job duration for this partition to 24 hours.

A job requesting a longer duration than the partition allows will not be accepted with that request.

Why does this matter?

Imagine a cluster used by two groups:

• Interactive users running short tests.
• Researchers running simulations that take several days.

Administrators could create separate partitions with different maximum durations, allowing short jobs to use one pool and longer jobs to use another.

Infrastructure impact: Runtime limits influence how administrators organise compute resources and balance job throughput against long running workloads.

However, a shorter MaxTime does not automatically make jobs finish faster or guarantee shorter queue times. Actual scheduling behaviour depends on resource availability, priority, backfill and other policies.

5. Accounting and Quality of Service (QoS)

An HPC cluster often serves multiple research teams, departments or projects. Administrators need to understand resource usage and control how resources are shared.

Two important concepts are accounting and Quality of Service.

An accounting configuration might include:

AccountingStorageType=accounting_storage/slurmdbd
AccountingStorageHost=headnode

These settings tell Slurm to use SlurmDBD for accounting data, with the accounting service hosted at headnode in this example. SlurmDBD must be installed and configured separately.

Accounting records help administrators answer questions such as:

• Which users and projects consume the most compute time?
• How much CPU or GPU time has a research team used?
• Which jobs failed, and how long did they run?

QoS policies can apply limits to jobs and allocations or influence their scheduling priority. For example, a research cluster might define a QoS that limits the number of running jobs or the resources a user can consume.

These policies are configured separately using Slurm's accounting tools, such as sacctmgr, along with the relevant associations and QoS settings.

Infrastructure impact: Accounting and QoS help administrators govern shared capacity, monitor demand and identify whether additional CPU nodes, GPU nodes or high memory nodes may be needed.

Accounting alone does not automatically enforce every usage limit. The relevant limits and policies must be configured explicitly.

Bringing it all together

These configuration areas address different parts of cluster management:

• SelectType and SelectTypeParameters determine how resources are selected and allocated.
• PriorityWeight settings influence how pending jobs are ranked.
• PartitionName organises compute resources into scheduling pools.
• MaxTime controls permitted job duration.
• Accounting and QoS support usage visibility and resource governance.

Consider an HPC cluster serving both AI researchers and traditional simulation users. The administrator may need GPU nodes, high memory nodes, separate partitions, suitable allocation policies and accounting controls.

Slurm cannot replace good infrastructure design. However, its configuration helps ensure that available resources are allocated according to the cluster's requirements and policies.

The key takeaway: Understanding Slurm configuration is not just about making jobs run. It is about designing an HPC environment that allocates resources effectively, supports different workloads and can grow as demand changes.