Using Virtual Environments
Have you ever wanted to install custom Python packages on a cluster but received permission denied errors when you tried? Or maybe you want to create separate environments for different workflows so they don’t interfere with one another. A good solution is to use
virtualenv. This is a way to create a local, unique environment that can be activated and deactivated as needed. For more in-depth information, see our online Python documentation.
In this example, we’ll set up a virtual environment, activate it, install a non-standard package, and use it in a job submission.
NOTE: This example uses python 3.6 to create the virtual environment. Python 3.8 uses different syntax. For more in-depth information, see our docs linked in the Overview.
To begin, start an interactive session. This can be easily done by running
interactive from one of the login nodes. Once it’s ready, load your desired version of python and create your virtual environment using
virtualenv --system-site-packages /path/to/env. The environment will be given the name included in the path. If only a name is provided without a path, your environment will be saved to your working directory.
NOTE: to run this example in your account, you’ll either want to create your virtual environment in the same directory where your python and SLURM scripts are located, or change the SLURM script to point to the correct location.
Once the process is complete, a directory with its own nested bin and lib directories will be available in the path specified. In the bin, there’s a script called
activate that you can source to load the environment. Once your environment is active, you can pip install custom packages. Once they’re installed, they will only be available while the environment is active. In this example, we’ll install the emoji package for demonstration purposes.
[netid@r2u05n1 ~]$ module load python/3.6 [netid@r2u05n1 ~]$ virtualenv --system-site-packages my_environment [netid@r2u05n1 ~]$ source my_environment/bin/activate (my_environment) [netid@r2u05n1 ~]$ pip install emoji
A simple python script that tests whether a virtual environment is active and reports back to the user.
#!/usr/bin/env python3 import os def in_venv(): try: os.environ["VIRTUAL_ENV"] return True except KeyError: return False virtualenv = in_venv() if virtualenv == True: try: import emoji print(emoji.emojize(":floppy_disk: Using a virtual environment! :floppy_disk:")) except ImportError: print("Virtual environment active! But it looks like you haven't installed the emoji package") else: print("Not using a virtual environment, let's check if we can import emoji") try: import emoji except ImportError: print("oops, can't import emoji!")
In this SLURM script, we’ll try running our python script with and without our virtual environment activated.
#!/bin/bash #SBATCH --job-name=Sample_venv_Job #SBATCH --ntasks=1 #SBATCH --nodes=1 #SBATCH --mem=1gb #SBATCH --time=00:01:00 #SBATCH --partition=standard #SBATCH --account=YOUR_GROUP # Load the necessary module module load python/3.6 echo -e "\nFirst, we'll test the script without activating our environment:\n" python3 myscript.py echo -e "\nNext, we'll activate our environment and rerun our script:\n" source my_environment/bin/activate python3 myscript.py
Submit the Job
[netid@junonia ~]$ sbatch myenv.slurm Submitted batch job 174168
[netid@wentletrap ~]$ cat slurm-174168.out First, we'll test the script without activating our environment: Not using a virtual environment, let's check if we can import emoji oops, can't import emoji! Next, we'll activate our environment and rerun our script: 💾 Using a virtual environment! 💾 Detailed performance metrics for this job will be available at https://metrics.hpc.arizona.edu/#job_viewer?action=show&realm=SUPREMM&resource_id=5&local_job_id=174168 by 8am on 2021/09/18.