Setting up a Testing Environment Using Conda

Summary

This section describes how to set up a running environment for developing WDLs using conda. Once the environment is set up, we can run a WDL directly using Cromwell which is a workflow engine that understands WDLs, and is what JAWS uses under the hood. You’ll want to develop and test your WDL with Cromwell and only run with JAWS once everything is working.

In this tutorial, you will:

  1. build a development environment

  2. run a WDL using cromwell

How to find packages you need for building environments

Common ways to install software into docker containers are:

1. apt-get:

apt-get is useful for installing dependencies and works for ubuntu, which is used for this docker build. For alpine or other operating systems, find the equivalent installer.

2. pip:

for installing python packages

3. conda:

installs packages from any language, but the available versions of a particular software may be limited

4. installing from source:

the old fashion way of running make && make install

For this tutorial, conda will be the primary way to install our requirements. The first step would be to:

  1. search for a software package at anaconda.org. bioconda or conda-forge are well trusted channels.

  2. If that fails, try to search pip packages. As mentioned, pip: installs only python packages but can be used to install stuff in your conda env. You can search for pip packages at pypi.

  3. If that fails, you may have to install from source.

Install Conda via Miniconda3

You can download minconda and choose the appropriate miniconda.

Installation

# pick the latest version, for Mac in this case.
bash Miniconda3-latest-MacOSX-x86_64.sh
# follow the directions for installation.

# Choose yes for: "Do you wish the installer to initialize Miniconda3 by running conda init?"
# This should add the miniconda installation to your PATH by modifying .bash_profile or .bashrc file.
# To disable the automatic base activation, please run:
conda config --set auto_activate_base false

# verify installation
which conda

Create your Conda Environment

We will create a conda environment called bbtools

# create bbtools env
conda create --name bbtools

# Now activate your environment.
conda activate bbtools
Create your Conda Environment on Dori

If you are creating the Conda Environment on Dori, please make sure to deactivate all the Conda environments before starting.

module load python
conda --version
## conda 4.12.0

Install necessary dependencies into your environment

# install dependencies
conda install -c bioconda bbmap==38.84
conda install -c bioconda samtools==1.11

# of course you can install Cromwell if you wanted to develop WDLs on your labtop
conda install -y -c bioconda cromwell
Create your Conda Environment on Dori

To install necessary dependencies into your environment, ensure the Java Version is not overwritten.

conda install -c bioconda bbmap==38.84 --update-all --force-reinstall
## Check Java Version
java --version
## openjdk 11.0.13 2021-10-19

Testing your Environment

Download the Example WDL repository

git clone https://code.jgi.doe.gov/official-jgi-workflows/wdl-specific-repositories/jaws-tutorial-examples.git
cd jaws-tutorial-examples/5min_example

The following command should run an alignment.

./script.sh ../data/sample.fastq.bz2 ../data/sample.fasta

This should create file called test.sorted.bam. We will use this script.sh to create a WDL later.

Run the WDL Workflow

You can run a WDL directly on the command-line (outside of JAWS) by using a Cromwell executable. Either you install your own, i.e using conda, or if you are on Dori, then there is an installation at /clusterfs/jgi/groups/dsi/homes/svc-jaws/jaws-install/dori-prod/lib/ which should point to the latest version used in JAWS.

Installing your own Cromwell
conda activate bbtools # make sure you are in a conda environment first
conda install cromwell
cromwell --version


Running with your own conda version

Make sure the bbtools conda environment is activated and you are in 5min_example.

# run with your installed version
cromwell run align.wdl -i inputs.json

Running with docker container

TODO