Using the Campus Cluster

Logging in

ssh username@cc-login.campuscluster.illinois.edu

Setting up your conda installation

Make sure your .bashrc is not loading another conda environment and that PYTHONPATH is not set. If it is, clear those things, log out, and log back in.

module load anaconda/3
conda create --name fast-mpi4py python=3.8
module load openmpi/3.1.1-gcc-7.2.0
export MPICC=$(which mpicc)
.  /usr/local/anaconda/5.2.0/python3/etc/profile.d/conda.sh
conda activate fast-mpi4py
pip install -v --no-binary mpi4py mpi4py
pip install -v --no-binary numpy numpy
pip install pyscf
module load git
pip install git+git://github.com/WagnerGroup/pyqmc --upgrade
pip install snakemake

Submission script

#!/bin/bash
#SBATCH --time=24:00:00                  # Job run time (hh:mm:ss)
#SBATCH --nodes=1                        # Number of nodes
#SBATCH --ntasks-per-node=40             # Number of task (cores/ppn) per node
#SBATCH --job-name=a120
#SBATCH --partition=wagner
#SBATCH --output=myjob.o%j              # Name of batch job output file

module load anaconda/3
module load openmpi/3.1.1-gcc-7.2.0
.  /usr/local/anaconda/5.2.0/python3/etc/profile.d/conda.sh
conda activate fast-mpi4py

srun python test.py 

snakemake setup

I would suggest starting with the setup here: https://github.com/WagnerGroup/Energy-Entropy

module load anaconda/3
module load openmpi/3.1.1-gcc-7.2.0
.  /usr/local/anaconda/5.2.0/python3/etc/profile.d/conda.sh
conda activate fast-mpi4py
pip install cookiecutter
cookiecutter https://github.com/Snakemake-Profiles/slurm.git

The profile will be installed in ~/.config/snakemake/slurm. I have changed my version of slurm-submit.py to include the partition:

RESOURCE_MAPPING = {
    "time": ("time", "runtime", "walltime"),
    "mem": ("mem", "mem_mb", "ram", "memory"),
    "mem-per-cpu": ("mem-per-cpu", "mem_per_cpu", "mem_per_thread"),
    "nodes": ("nodes", "nnodes"),
    "partition": ("partition",)
}

Then your snakemake can look like the following, if you’d like to run on qmc_threads cores on one node and the secondary partition. Note the resources section in the rule.

rule VMC_MF:
    input: hf="{dir1}/{dir2}/mf.chk", wffile="{dir1}/{dir2}/vmc_mf_{tol}_{orbs}_{nconfig}.chk"
    output: "{dir1}/{dir2}/eval_mf_{tol}_{orbs}.chk"
    threads: qmc_threads
    resources:
        walltime="4:00:00", partition="secondary"
    run:
        with concurrent.futures.ProcessPoolExecutor(max_workers=qmc_threads) as client:
            functions.evaluate_vmc(input.hf, None, input.wffile, output[0], 
                  slater_kws=None, nblocks=1000, client=client, npartitions=qmc_threads)

TODO: change this to work with MPI.

You can run by doing something like the following:

module load anaconda/3
module load openmpi/3.1.1-gcc-7.2.0
.  /usr/local/anaconda/5.2.0/python3/etc/profile.d/conda.sh
conda activate fast-mpi4py
hostname >> run.out
nohup nice snakemake --jobs=10  --profile=slurm [filenames] > vmc.out & 

You can run this on a login node and it will execute your jobs.

Summit

You should be able to follow similar instructions to set up summit; see for example

https://www.olcf.ornl.gov/wp-content/uploads/2019/02/STW_Feb_20190211_summit_workshop_python.pdf