A Primer on the Cluster System at Hunter - Revision history

Wikiuser: /* From an environment.yml file */

2023-07-05T15:42:57Z

From an environment.yml file

← Older revision		Revision as of 15:42, 5 July 2023
Line 425:		Line 425:

	Conda users can share their environments in a fairly convenient fashion by dumping the list of packages:		Conda users can share their environments in a fairly convenient fashion by dumping the list of packages:
			<code>
	~~```bash~~
	~~# or mamba~~
	conda env export -n aligners > aligners_environment.yml		conda env export -n aligners > aligners_environment.yml
	~~```~~		</code>
			This will export the list of packages installed in the `aligners`, along with their exact version, into a YAML file called <code>aligners_environment.yml</code>. You can share this file with others and conda/mamba will attempt to recreate the environment on their machine.
	This will export the list of packages installed in the `aligners`, along with their exact version, into a YAML file called `aligners_environment.yml`. You can share this file with others and conda/mamba will attempt to recreate the environment on their machine.

	To import a YAML file you recieved, run:		To import a YAML file you recieved, run:

	~~```bash~~		<code>
	~~# or conda~~
	mamba env create -f environment.yml		mamba env create -f environment.yml
	~~```~~		</code>

	~~This will create a new environment with whatever name was specified in the environment.yml file. To choose your own, add `-n envname`.~~

	~~## Using conda~~/~~mamba~~		This will create a new environment with whatever name was specified in the environment.yml file. To choose your own, add <code>-n envname</code>.

			==Using conda/mamba==
	See the "From scratch" section above for a few examples.		See the "From scratch" section above for a few examples.

	## Installing Qiu-lab software		Installing Qiu-lab software

	~~```bash~~		<code>
	mamba create -n qiulab perl-bio-bpwrapper		mamba create -n qiulab perl-bio-bpwrapper
	~~```~~		</code>

Wikiuser: /* From scratch */

2023-07-05T15:40:57Z

From scratch

← Older revision		Revision as of 15:40, 5 July 2023
Line 398:		Line 398:
	===From scratch===		===From scratch===
	You can create a new enviroment using <code>mamba create</code> and supplying a name for the environment. Optionally (but strongly recommended), you can end the command with a list of packages. e.g.,		You can create a new enviroment using <code>mamba create</code> and supplying a name for the environment. Optionally (but strongly recommended), you can end the command with a list of packages. e.g.,

	<code>		<code>
	~~# or conda create~~
	mamba create aligners -n blast clustalo muscle bwa minimap2 mafft		mamba create aligners -n blast clustalo muscle bwa minimap2 mafft
	</code>		</code>
Line 418:		Line 416:
	You can also install specific versions of a software package:		You can also install specific versions of a software package:

	~~```bash~~		<code>
	mamba install -n aligners blast=2.10		mamba install -n aligners blast=2.10
	~~```~~		</code>

	The above will get conda/mamba to try to downgrade the package, if it is already installed. This may or may not work if the system cannot resolve any dependency issues that may occur with dependent packages.		The above will get conda/mamba to try to downgrade the package, if it is already installed. This may or may not work if the system cannot resolve any dependency issues that may occur with dependent packages.

	~~###~~ From an environment.yml file		===From an environment.yml file===

	Conda users can share their environments in a fairly convenient fashion by dumping the list of packages:		Conda users can share their environments in a fairly convenient fashion by dumping the list of packages:

Wikiuser: /* Creating a new environment */

2023-07-05T15:39:56Z

Creating a new environment

← Older revision		Revision as of 15:39, 5 July 2023
Line 394:		Line 394:
	</code>		</code>
	==Creating a new environment==		==Creating a new environment==
			You can create environments using <code>mamba create</code> (or <code>conda create</code>). An environment can be used to install all of the software needed to complete some task or workflow, without interfering with the system installation. You will still be able to access software installed to your operating system's default PATH, but software in an environment (usually) takes precedent, and is isolated to that environment. In other words, if you install a copy of <code>clustalw</code> in a conda environment, and you activate that environment, it will (should) mask whatever version of <code>clustalw</code> you may have installed in your system. Also, you can have more than one version of <code>clustalw</code> installed simultaneously, each one being installed in its own environment.

	You can create ~~environments~~ using `mamba create~~` (or `conda create`). An~~ environment ~~can be used to install all of the software needed to complete some task or workflow, without interfering with the system installation~~. ~~You will still be able to access software installed to your operating system's default PATH,~~ but ~~software in an environment (usually~~) ~~takes precedent~~, ~~and is isolated to that environment. In other words, if~~ you ~~install~~ a ~~copy~~ of ~~`clustalw` in a conda environment, and you activate that environment, it will (should) mask whatever version of `clustalw` you may have installed in your system~~. ~~Also~~, ~~you can have more than one version of `clustalw` installed simultaneously, each one being installed in its own environment.~~		===From scratch===
			You can create a new enviroment using <code>mamba create</code> and supplying a name for the environment. Optionally (but strongly recommended), you can end the command with a list of packages. e.g.,

	~~### From scratch~~		<code>

	~~You can create a new enviroment using `mamba create` and supplying a name for the environment. Optionally (but strongly recommended), you can end the command with a list of packages. e.g.,~~

	~~```bash~~
	# or conda create		# or conda create
	mamba create aligners -n blast clustalo muscle bwa minimap2 mafft		mamba create aligners -n blast clustalo muscle bwa minimap2 mafft
	~~```~~		</code>

	This will create a new environment called "aligners" with blast, clustalo, muscle, bwa, minimap2, and mafft available for use once activated. To activate an environment, do:		This will create a new environment called "aligners" with blast, clustalo, muscle, bwa, minimap2, and mafft available for use once activated. To activate an environment, do:

	~~```bash~~		<code>
	mamba activate aligners		mamba activate aligners
	~~```~~		</code>

	Post-environment creation, you can install packages using the `install` subcommand:		Post-environment creation, you can install packages using the <code>install</code> subcommand:

	~~```bash~~		<code>
	mamba install -n aligners diamond		mamba install -n aligners diamond
	~~```~~		</code>

	You can also install specific versions of a software package:		You can also install specific versions of a software package:

Wikiuser: /* Configuring bioconda */

2023-07-05T15:36:52Z

Configuring bioconda

← Older revision		Revision as of 15:36, 5 July 2023
Line 383:		Line 383:
	==Configuring bioconda==		==Configuring bioconda==

	[Bioconda](https://bioconda.github.io/) is a conda repository for software used in biology. You'll need to add this repository for most of the tools you will be using.		Bioconda(https://bioconda.github.io/) is a conda repository for software used in biology. You'll need to add this repository for most of the tools you will be using.

	Check the website for the latest instructions, but the commands to add the repository (as of Jun, 2023) are below:		Check the website for the latest instructions, but the commands to add the repository (as of Jun, 2023) are below:

	~~```bash~~		<code>
	conda config --add channels defaults		conda config --add channels defaults
	conda config --add channels bioconda		conda config --add channels bioconda
	conda config --add channels conda-forge		conda config --add channels conda-forge
	conda config --set channel_priority strict		conda config --set channel_priority strict
	~~```~~		</code>
			==Creating a new environment==
	## Creating a new environment

	You can create environments using `mamba create` (or `conda create`). An environment can be used to install all of the software needed to complete some task or workflow, without interfering with the system installation. You will still be able to access software installed to your operating system's default PATH, but software in an environment (usually) takes precedent, and is isolated to that environment. In other words, if you install a copy of `clustalw` in a conda environment, and you activate that environment, it will (should) mask whatever version of `clustalw` you may have installed in your system. Also, you can have more than one version of `clustalw` installed simultaneously, each one being installed in its own environment.		You can create environments using `mamba create` (or `conda create`). An environment can be used to install all of the software needed to complete some task or workflow, without interfering with the system installation. You will still be able to access software installed to your operating system's default PATH, but software in an environment (usually) takes precedent, and is isolated to that environment. In other words, if you install a copy of `clustalw` in a conda environment, and you activate that environment, it will (should) mask whatever version of `clustalw` you may have installed in your system. Also, you can have more than one version of `clustalw` installed simultaneously, each one being installed in its own environment.

Wikiuser: /* Setting up mambaforge */

2023-07-05T15:35:34Z

Setting up mambaforge

← Older revision		Revision as of 15:35, 5 July 2023
Line 365:		Line 365:
	<code>mkdir -p /lustre/projects/qiulab/$USER</code>		<code>mkdir -p /lustre/projects/qiulab/$USER</code>
	==Setting up mambaforge==		==Setting up mambaforge==

	Follow the instructions on the [miniforge git repository](https://github.com/conda-forge/miniforge#mambaforge) (partially reproduced below):		Follow the instructions on the [miniforge git repository](https://github.com/conda-forge/miniforge#mambaforge) (partially reproduced below):
			<code>
	~~```bash~~
	curl -L -O "https://github.com/conda-forge/miniforge/releases/latest/download/Mambaforge-$(uname)-$(uname -m).sh"		curl -L -O "https://github.com/conda-forge/miniforge/releases/latest/download/Mambaforge-$(uname)-$(uname -m).sh"
	bash Mambaforge-$(uname)-$(uname -m).sh		bash Mambaforge-$(uname)-$(uname -m).sh
	~~```~~		</code>

	During the setup wizard, you will be asked to enter the directory in which to install Mambaforge. Enter the directory you created above and add "/mambaforge" to the end. For example, if your username is "user", you would enter:		During the setup wizard, you will be asked to enter the directory in which to install Mambaforge. Enter the directory you created above and add "/mambaforge" to the end. For example, if your username is "user", you would enter:

	~~```bash~~		<code>
	/lustre/projects/qiulab/user/mambaforge		/lustre/projects/qiulab/user/mambaforge
	~~```~~		</code>

	Replace "user" above with your own username to use the directory created earlier.		Replace "user" above with your own username to use the directory created earlier.
Line 383:		Line 381:
	At the end of the installation, you will be asked whether or not you would like to run `conda init`. Answer "yes". When complete, close and reopen your shell (or disconnect/reconnect to the HPC node).		At the end of the installation, you will be asked whether or not you would like to run `conda init`. Answer "yes". When complete, close and reopen your shell (or disconnect/reconnect to the HPC node).

	## Configuring bioconda		==Configuring bioconda==

	[Bioconda](https://bioconda.github.io/) is a conda repository for software used in biology. You'll need to add this repository for most of the tools you will be using.		[Bioconda](https://bioconda.github.io/) is a conda repository for software used in biology. You'll need to add this repository for most of the tools you will be using.

Wikiuser at 15:33, 5 July 2023

2023-07-05T15:33:40Z

← Older revision		Revision as of 15:33, 5 July 2023
Line 359:		Line 359:
	</syntaxhighlight>		</syntaxhighlight>
	</div>		</div>

			=Setting up a conda/mamba environment on the HPC=
			==Create a space for your installation==
			Use the project space since the quota enforced on your home directory is quite small.
			<code>mkdir -p /lustre/projects/qiulab/$USER</code>
			==Setting up mambaforge==

			Follow the instructions on the [miniforge git repository](https://github.com/conda-forge/miniforge#mambaforge) (partially reproduced below):

			```bash
			curl -L -O "https://github.com/conda-forge/miniforge/releases/latest/download/Mambaforge-$(uname)-$(uname -m).sh"
			bash Mambaforge-$(uname)-$(uname -m).sh
			```

			During the setup wizard, you will be asked to enter the directory in which to install Mambaforge. Enter the directory you created above and add "/mambaforge" to the end. For example, if your username is "user", you would enter:

			```bash
			/lustre/projects/qiulab/user/mambaforge
			```

			Replace "user" above with your own username to use the directory created earlier.

			At the end of the installation, you will be asked whether or not you would like to run `conda init`. Answer "yes". When complete, close and reopen your shell (or disconnect/reconnect to the HPC node).

			## Configuring bioconda

			[Bioconda](https://bioconda.github.io/) is a conda repository for software used in biology. You'll need to add this repository for most of the tools you will be using.

			Check the website for the latest instructions, but the commands to add the repository (as of Jun, 2023) are below:

			```bash
			conda config --add channels defaults
			conda config --add channels bioconda
			conda config --add channels conda-forge
			conda config --set channel_priority strict
			```

			## Creating a new environment

			You can create environments using `mamba create` (or `conda create`). An environment can be used to install all of the software needed to complete some task or workflow, without interfering with the system installation. You will still be able to access software installed to your operating system's default PATH, but software in an environment (usually) takes precedent, and is isolated to that environment. In other words, if you install a copy of `clustalw` in a conda environment, and you activate that environment, it will (should) mask whatever version of `clustalw` you may have installed in your system. Also, you can have more than one version of `clustalw` installed simultaneously, each one being installed in its own environment.

			### From scratch

			You can create a new enviroment using `mamba create` and supplying a name for the environment. Optionally (but strongly recommended), you can end the command with a list of packages. e.g.,

			```bash
			# or conda create
			mamba create aligners -n blast clustalo muscle bwa minimap2 mafft
			```

			This will create a new environment called "aligners" with blast, clustalo, muscle, bwa, minimap2, and mafft available for use once activated. To activate an environment, do:

			```bash
			mamba activate aligners
			```

			Post-environment creation, you can install packages using the `install` subcommand:

			```bash
			mamba install -n aligners diamond
			```

			You can also install specific versions of a software package:

			```bash
			mamba install -n aligners blast=2.10
			```

			The above will get conda/mamba to try to downgrade the package, if it is already installed. This may or may not work if the system cannot resolve any dependency issues that may occur with dependent packages.

			### From an environment.yml file

			Conda users can share their environments in a fairly convenient fashion by dumping the list of packages:

			```bash
			# or mamba
			conda env export -n aligners > aligners_environment.yml
			```

			This will export the list of packages installed in the `aligners`, along with their exact version, into a YAML file called `aligners_environment.yml`. You can share this file with others and conda/mamba will attempt to recreate the environment on their machine.

			To import a YAML file you recieved, run:

			```bash
			# or conda
			mamba env create -f environment.yml
			```

			This will create a new environment with whatever name was specified in the environment.yml file. To choose your own, add `-n envname`.

			## Using conda/mamba

			See the "From scratch" section above for a few examples.

			## Installing Qiu-lab software

			```bash
			mamba create -n qiulab perl-bio-bpwrapper
			```

Wikiuser: /* Pre-written cluster scripts for various software */

2023-04-05T14:19:43Z

Pre-written cluster scripts for various software

← Older revision		Revision as of 14:19, 5 April 2023
Line 263:		Line 263:
	Rayees		Rayees

			=Conda Usage=
			<syntaxhighlight lang="bash">
			# to create an environment
			module load anaconda
			conda create --name gbs python=3.10 bwa samtools
			conda info --envs

			# to run
			module load anaconda
			source activate gbs
			</syntaxhighlight>
	=Pre-written cluster scripts for various software=		=Pre-written cluster scripts for various software=

Wikiuser: /* BWA & Samtools */

2023-04-05T14:16:34Z

BWA & Samtools

← Older revision		Revision as of 14:16, 5 April 2023
Line 269:		Line 269:

	==BWA & Samtools==		==BWA & Samtools==
			Use array option to loop through directories and files
	<div class="toccolours mw-collapsible">		<div class="toccolours mw-collapsible">
	<syntaxhighlight lang=bash">		<syntaxhighlight lang=bash">
Line 275:		Line 276:
	#SBATCH --job-name=gbs		#SBATCH --job-name=gbs
	#SBATCH --output=gbs.log		#SBATCH --output=gbs.log
	#SBATCH --array=1-5		#SBATCH --array=1-2
	#SBATCH -o output-%A_%a-%J.o		#SBATCH -o output-%A_%a-%J.o
	#SBATCH -n 1		#SBATCH -n 1
Line 305:		Line 306:
	</syntaxhighlight>		</syntaxhighlight>
	</div>		</div>

	==Interproscan==		==Interproscan==

Wikiuser: /* Pre-written cluster scripts for various software */

2023-04-05T14:14:35Z

Pre-written cluster scripts for various software

← Older revision		Revision as of 14:14, 5 April 2023
Line 269:		Line 269:

	==BWA & Samtools==		==BWA & Samtools==
			<div class="toccolours mw-collapsible">
	<syntaxhighlight lang=bash">		<syntaxhighlight lang=bash">
	#!/bin/bash		#!/bin/bash
Line 303:		Line 304:
	done		done
	</syntaxhighlight>		</syntaxhighlight>
			</div>
	==Interproscan==		==Interproscan==

Wikiuser: /* Pre-written cluster scripts for various software */

2023-04-05T14:13:35Z

Pre-written cluster scripts for various software

← Older revision		Revision as of 14:13, 5 April 2023
Line 268:		Line 268:
	Also please feel free to contribute to this page! Send me or Dr. Qiu an email to update this section with new example scripts for running software.		Also please feel free to contribute to this page! Send me or Dr. Qiu an email to update this section with new example scripts for running software.

			==BWA & Samtools==
			<syntaxhighlight lang=bash">
			#!/bin/bash
			#
			#SBATCH --job-name=gbs
			#SBATCH --output=gbs.log
			#SBATCH --array=1-5
			#SBATCH -o output-%A_%a-%J.o
			#SBATCH -n 1
			#

			# Reference for array operations: https://portal.supercomputing.wales/index.php/index/slurm/interactive-use-job-arrays/job-arrays/
			# formal doc: https://slurm.schedmd.com/job_array.html
			# %J: job identifier
			# %A: parent job
			# %a: array iteration index
			work_dir=/home/wqiu/project-home/GBS-April-2023
			module load anaconda
			source activate gbs
			echo SLURM_JOB_ID $SLURM_JOB_ID
			echo SLURM_ARRAY_JOB_ID $SLURM_ARRAY_JOB_ID
			echo SLURM_ARRAY_TASK_ID $SLURM_ARRAY_TASK_ID

			DIRS=(GBS100 GBS101)

			for i in ${DIRS[@]}
			do
			echo "Processing folder $i"
			dir=$work_dir/$i
			bwa mem $work_dir/ref.fas $dir/$i.R1.fq.gz $dir/$i.R2.fq.gz > $dir/$i.sam 2> /dev/null
			echo "sam file generated: $i.sam"
			samtools view -bT $work_dir/ref.fas $dir/${i}.sam \| samtools sort -o $dir/${i}.bam 2> /dev/null
			echo "bam file generated: $i.bam"
			done
			</syntaxhighlight>
	==Interproscan==		==Interproscan==