The job scheduler is SLURM (Simple Linux Utility for Resource Management). This software came from ShedMD. The version on Occigen, was updated by Bull/ATOS.

First, make a script with the sbatch commands, an example is given below. The first line of this file must always be a line giving the shell of the commands that will be used: # !/bin/bash. The lines beginning with an hyphen and containing the word SBATCH will be taken into account by the scheduler SLURM. <>><

They give instructions related to the reservation of resources you actually need and wall clock time limit.
Exemple de contenu d’un script SLURM pour un programme MPI s’exécutant sur 2 noeuds Broadwell et 56 coeurs

#!/bin/bash
#SBATCH -J job_name
#SBATCH --nodes=2
#SBATCH --constraint=BDW28
#SBATCH --ntasks=56
#SBATCH --ntasks-per-node=28
#SBATCH --threads-per-core=1
#SBATCH --mem=30GB
#SBATCH --time=00:30:00
#SBATCH --output job_name.output
module purge
module load intel
module load openmpi # n'hesitez pas a tester plusieurs versions
srun --mpi=pmi2 -K1 -n $SLURM_NTASKS ./mon_executable param1 param2 …

The interactive mode is forbidden on a login node.
On a compute node, first, you should type the following command (on the login node) to get access to some resources (compute nodes):
then when the resources become available, in the same windows, type :

salloc --constraint=BDW28 -N 1 -n 10 -t 10:00
srun mon_executable

If your executable is able to do checkpoint/restart files, please respect the fair play rules and submit a new job for continuation.>
If your application does not make a checkpoint/restart yet, please send a well argued demand to svp@cines.fr and it will be examined. Even if you are granted of such an authorization, you should you keep in mind that the resources that can host this queueing class are limited and the priority of these kind of jobs are lower. Your job is also largely exposed to the risk of node failure and a general advice is that you have to develop checkpointing in your code in order to avoid the waste of computational time in such a case.

Half of the 2106 nodes Haswell of the fist part of Occigen benefits from 128 GB of memory.
In order to tell the SLURM scheduler that your job actually need them, add the following line in your script :

#SBATCH --mem=118000

#SBATCH --constraint=HSW24

VASP offers both parallelization :

• over bands (see NPAR)
• and over k-points (see KPAR).

In your input file (INCAR) these parameters must be carefully chosen in order to obtain the best performance, taking into account of the Occigen architecture.

Parameters of the INCAR file
The KPAR value
Que représente ce paramètre?
The KPAR parameter manages the parallelization over the k-points,
it represents the number of kpoints groups created to parallelize on this dimension. The Occigen supercomputer has double-socket nodes with 12 cores per socket. Keep in mind that one should try to exploit the highest number of cores available on each node while complying with the constraints described below.
For example, if the study case owns 10 k-points and KPAR is set to 2, there will be 2 k-points groups each performing calculations on 5 k-points. Similarly, if KPAR is set to 5, there will be 5 groups each with 2 k-points.
Currently KPAR is limited to values that divide exactly both the total number of k-points and the total number of cores used by the job.

The NPAR value
Que représente ce paramètre ?
The NPAR parameter manages the parallelization over the bands,
It determines the number of bands that are handled in parallel, each band being distributed over some cores. The number of cores that are dedicated to a band is equal to the ratio : total number of cores divided by NPAR. If NPAR=1, all the cores will work together on every individual band.

NPAR must be adjusted in agreement with the KPAR value according to the following formula:

The best performance seems to be obtained on Occigen for NCORES_PAR_BAND ranging from 2 to 4.
Selecting the right number of cores for a calculation with VASP is very important because using too many cores will be inefficient. As a rule of thumb, you can choose the number of cores in the same order of magnitude as the number of atoms. This parameter needs further refinements. When the workload is split over several compute nodes, you have to ensure that a sufficient amount of work has been allocated to each core.
If you encounter « out Of Memory » problems, the following action should be taken :

• Select the high memory nodes by adding a special directive in your job script ( #SBATCH – mem=120GB)
• If this is not sufficient, consider again the parallelisation. Try to increase the number of cores and/or unpopulate the cores.

Following is an example of a submission script for Occigen

#!/bin/sh
#SBATCH --nodes=2
#SBATCH --constraint=HSW24
#SBATCH --ntasks-per-node=24
#SBATCH --ntasks=48
#SBATCH --mem=118000 # si les nœuds par défaut à 64 GB ne sont pas suffisants
#SBATCH --time=24:00:00
#SBATCH --exclusive
#SBATCH --output vasp5.3.3.out

module purge
module load intel
module load openmpi
ulimit -s  unlimited
srun --mpi=pmi2 -K1 /path/vasp-k-points