Scratch storages

Scratch storage is a storage for temporary files for running job:

...                                  # PBS params etc.
cp -r ~/input_data_dir/ $SCRATCHDIR/ # copy data into a scratch directory
cd $SCRATCHDIR                          
...                                  # run the computing job

This storage should be used only during computations and should be freed immediately after your job ends.

The location of scratch directory is defined by a system variable SCRATCHDIR.

Scratch types, the short versionCopy Anchor Link

Which scratch to useCopy Anchor Link

Is the data only needed DURING one job?
├── yes, data are for a single job
│     ├── one node
│     │     ├── ordinary I/O ....................... scratch_local   (default)
│     │     ├── many files / heavy I/O ............. scratch_ssd
│     │     └── ultra fast I/O, moderate data ...... scratch_shm (+mem)
│     └── several nodes, shared files .............. scratch_shared
└── no, must survive between jobs
      ├── needs to be fast, nodes limited to Brno or Plzeň  . multijob=brno / multijob=plzen
      └── slow access acceptable, available on all nodes .... multijob=global

Scratch types - detailed overviewCopy Anchor Link

Local scratchCopy Anchor Link

The default choice.

• PBS resource scratch_local

qsub -I -l walltime=01:00:00 -l select=1:ncpus=1:mem=1gb:scratch_local=400mb

• Availability: on every node.
• Speed: moderate (spinning HDD on most nodes).
• Lifetime: the job. The content can be erased any time after job’s end.
• Use it when: you have no special reason to use anything else — single-node jobs, ordinary data sizes, applications that read inputs once and write results once.

Fast SSD scratchCopy Anchor Link

When disk speed is the bottleneck.

• PBS resource scratch_ssd

qsub -I -l walltime=01:00:00 -l select=1:ncpus=1:mem=1gb:scratch_ssd=400mb

• Availability: on most but not all nodes; see https://my.metacentrum.cz/machines -> choose a node -> search for “Scratch SSD”.
• Speed: fast (SSD disc), smaller volume than local HDD scratch.
• Lifetime: the job. The content can be erased any time after job’s end.
• Use it when: the bottleneck is disc-related operations (applications that create/read a lot of files). If qstat -xf shows little CPU time used but long walltime, suspect I/O — try SSD.

Scratch in RAMCopy Anchor Link

Maximum speed, zero safety.

• PBS resource scratch_shm

qsub -I -l walltime=01:00:00 -l select=1:ncpus=1:mem=1gb:scratch_shm=true

• Availability: on every node; scratch directory in RAM.
• Size: maximum size of scratch is defined by the mem (memory) parameter. Remember to choose memory large enough (to hold both data in scratch and the actual memory requirements for the job)
• Speed: as fast as a scratch can possibly get.
• Lifetime: data on scratch do not survive the end/failure of the job.
• Use it when: you need ultra-fast scratch AND when you absolutely don’t care about data from failed/killed/ended jobs

Shared scratchCopy Anchor Link

A network volume shared between all nodes in a multi-node job.

• PBS resource scratch_shared

qsub -I -l walltime=01:00:00 -l select=1:ncpus=1:mem=1gb:scratch_shared=400mb

• Availability: not available on all computational nodes; to check for availability on a particular node, go to https://my.metacentrum.cz/machines -> choose a node -> search for “Scratch Shared”.
• Speed: read/write operation slower than on local scratch.
• Use it when: your job spans multiple nodes and the processes on different nodes must read/write the same files (e.g. MPI jobs with shared intermediate data, or job arrays where subjob N picks up what subjob N−1 left).

Multijob scratch fastCopy Anchor Link

Reuse big data across jobs, fast, limited nodes.

• PBS resource multijob=[ brno | plzen ]

qsub -I -l walltime=01:00:00 -l select=1:multijob=brno

• located in /scratch/multijob-[brno|plzen]/USERNAME/

• Availability: available only on some nodes located in Brno or Plzen with BeeGFS filesystem.

• Speed: fast (BeeGFS parallel filesystem).

• Lifetime: persistent scratch for independent consecutive jobs working on same data

• Use it when: large masses of data (databases, reference datasets) that take long to copy and are shared by many independent jobs.

Example

# Run 1st job

qsub -I -l walltime=01:00:00 -l select=1:multijob=brno

(TRIXIE)user123@bee4:~$ cd /scratch/multijob-brno/user_123/
(TRIXIE)user123@bee4:~$ mkdir my_new_multijob_project
(TRIXIE)user123@bee4:~$ SCRATCHDIR=/scratch/multijob-brno/user_123/my_new_multijob_project/
# cp whatever data to SCRATCHDIR
# run the job no.1
(TRIXIE)user123@bee4:~$ exit

# Run another job on the same data

qsub -I -l walltime=01:00:00 -l select=1:multijob=brno # MUST be brno this time

(TRIXIE)user123@bee4:~$ SCRATCHDIR=/scratch/multijob-brno/user_123/my_new_multijob_project/
(TRIXIE)user123@bee4:~$ cd $SCRATCHDIR
# run the job no.2
(TRIXIE)user123@bee4:~$ exit

Multijob scratch globalCopy Anchor Link

Reuse big data across jobs, slow, on all nodes.

• PBS resource multijob=global

• submit as: qsub -I -l walltime=01:00:00 -l select=1:multijob=global

• located in /scratch/multijob-global/USERNAME/

• Availability: on every node.

• Speed: low (CephFS filesystem).

• Lifetime: persistent scratch for independent consecutive jobs working on same data

• Use it when: suitable for jobs working on large masses of data (databases) which take long to copy to/from storage

Shared scratch on cluster BeeCopy Anchor Link

BeeGFS (Beyond Extensible Enterprise File System) is a parallel distributed filesystem designed specifically for the needs of high-performance computing (HPC). It is used in computing clusters, scientific simulations, machine learning, genomics, and everywhere large datasets and fast parallel access are essential.

At MetaCentrum, we’ve adopted BeeGFS to meet the increasing challenges of data-intensive computations on data used by multiple job and/or job arrays. BeeGFS is available as a temporary working directory via the scratch_shared resource on cluster bee.cerit-sc.cz.

Main usecases:

• PBS resource scratch_shared together with cl_bee=True,for example qsub -l walltime=1:0:0 -q default@pbs-m1.metacentrum.cz -l select=1:ncpus=1:mem=400mb:scratch_shared=400mb:cl_bee=True,
• jobs with large files or a huge number of small files; BeeGFS efficiently handles massive datasets,
• jobs with parallel I/O operation
• jobs spanning multiple compute nodes
• array jobs with intermediate results - BeeGFS is well-suited for workflows where subsequent computations can pick up intermediate results left in the scratch directory eliminating the need to copy data to permanent storage or run on the same machine as the previous step.

Read more on BeeGFS article on eInfra blog.

Cleaning scratchCopy Anchor Link

Directory SCRATCHDIR is not writable, only it’s content is. Therefore, you cannot, e.g. do rm -rf $SCRATCHDIR, but you can rm -rf $SCRATCHDIR/*.

Users should always clear the content of the scratch directory after the job ends to free disc space. Otherwise, this directory will be automatically deleted after 14 days at most (earlier if there is lack of space on disks).

ExamplesCopy Anchor Link

Submit batch job with 100 GB scratch on local disc:

qsub -l select=1:ncpus=1:mem=4gb:scratch_local=100gb

Submit the interactive job with 20 GB memory and scratch in RAM:

qsub -I -l select=1:ncpus=1:mem=20gb:scratch_shm=true

Submit batch job with 1 GB of scratch on SSD disc:

qsub -l select=1:ncpus=1:mem=4gb:scratch_ssd=1gb

System variables

SCRATCHDIR
    location of the scratch directory
    echo $SCRATCHDIR
SCRATCH_TYPE
    type of scratch directory
    echo $SCRATCH_TYPE
SCRATCH_VOLUME
    size of the scratch directory
    echo $SCRATCH_VOLUME