Python Using Anaconda Environment

Step 1: Installing Python Packages with Anaconda

Load Environment Module: anaconda

Install Anaconda Package: Python 3.10, pip, Numpy

Installation Path (Anaconda Environment): /home/$USER/.conda/envs/<environment name>

To instal python packages with Anaconda, run below command in the SSH terminal (CLI), for tutorial in accessing CLI, please refer to SSH Shell Access to EdUHK HPC Platform and Cluster (Web-based Shell Access)

# Load Anaconda Environment Module
$ module load anaconda

# Add conda-forge Channels
$ conda config --add channels conda-forge

# Create Anaconda Virtual Environment
$ conda create –n <environment name> <packages to install>

# Create Anaconda Virtual Environment (Example)
$ conda create –n numpy_test_env python=3.10 pip numpy

# Purge Module
$ module purge

Step 2: Prepare Python Program Source Code

For the example source code please refer to Python Program Source Code (Example)

Step 3: Prepare Job Template Script

Pre-configured template script path → /home/$USER/job_template/slurm_job/numpy_test_conda.sh

Example template script

#!/bin/bash
#SBATCH --job-name=numpy_test_conda ## Job Name
#SBATCH --partition=shared_cpu ## Partition for Running Job
#SBATCH --nodes=1 ## Number of Compute Node
#SBATCH --ntasks-per-node=1 # Number of Task per Compute Node
#SBATCH --cpus-per-task=2 ## Number of CPU per task
#SBATCH --time=60:00 ## Job Time Limit (i.e. 60 Minutes)
#SBATCH --mem=10GB ## Total Memory for Job
#SBATCH --output=./%x%j.out ## Output File Path
#SBATCH --error=./%x%j.err ## Error Log Path

## Initiate Environment Module
source /usr/share/modules/init/profile.sh

## Reset the Environment Module components
module purge

## Load Module
module load anaconda

## Run user command in conda environemnt
## Do not use conda activate command, use absolute path instead
${CONDA_ENV_PATH}/numpy_test_env/bin/python3 \ /home/${USER}/job_template/python/numpy_test.py

## Clear Environment Module components
module purge

Step 4: Submit HPC Job

Guides for submitting HPC job, please refer to: HPC Job Submission (For CLI) and HPC Job Submission (For Web Portal)

Step 5: Remove User’s Conda (Python) Environment (Optional)

# Remove user’s Conda (Python) environment
$ rm –rf /home/$USER/.conda/envs/<environment name>

# Remove user’s Conda (Python) environment (Example)
$ rm –rf /home/$USER/.conda/envs/numpy_test_env