Skip to content

Commit 135803e

Browse files
committed
add toil installation and running instructions
1 parent b2ddf28 commit 135803e

File tree

1 file changed

+61
-2
lines changed

1 file changed

+61
-2
lines changed

docs/computing/running/toil-cwl-runner.md

Lines changed: 61 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -31,7 +31,20 @@ This page describes how run CWL worklflows on Puhti using `toil-cwl-runner`, inc
3131

3232
## Installing `toil-cwl-runner`
3333

34-
(and nodejs)
34+
Install `toil` with `CWL` plugin.
35+
```
36+
pip install -U setuptools wheel
37+
pip install toil[cwl]
38+
toil-cwl-runner --version
39+
```
40+
41+
Install `nodejs` which provides helpful tools for debugging `toil` internals.
42+
```
43+
cd /projappl/project_nnnnnnn
44+
wget https://nodejs.org/dist/v18.16.1/node-v18.16.1-linux-x64.tar.xz
45+
tar -xf node-v18.16.1-linux-x64.tar.xz
46+
export PATH=$PATH:/projappl/project_nnnnnnn/node-v18.16.1-linux-x64/bin
47+
```
3548

3649
## Defining CWL workflows
3750

@@ -41,6 +54,52 @@ Learning resources
4154

4255
## Running CWL workflows with `toil-cwl-runner`
4356

44-
(use sbatch to submit toil-cwl-runner job to submit further jobs)
57+
!!! Note
58+
Singularity containers can't be run in the **login node** or in an **interactive session** due to network constraints.
59+
60+
When you have defined a workflow with `CWL`, you can send it to the cluster using `sbatch`, and then `toil` will start new jobs for each item in the workflow description.
61+
62+
### Preliminary Steps
63+
Create working directories for `toil`.
64+
```
65+
mkdir /projappl/project_nnnnnnn/<job store name>
66+
mkdir /projappl/project_nnnnnnn/<work dir name>
67+
mkdir /projappl/project_nnnnnnn/<tmp name>
68+
```
69+
70+
### Creating the sbatch file
71+
The `sbatch` file `workflow.sh` will reference the `CWL` file `workflow.cwl` where you have described your workflow steps.
72+
73+
`workflow.sh`
74+
```bash
75+
#!/bin/sh
76+
#SBATCH --job-name=<job name here>
77+
#SBATCH --account=<project_number here>
78+
#SBATCH --time=01:00:00
79+
#SBATCH --mem-per-cpu=1G
80+
#SBATCH --nodes=1
81+
#SBATCH --cpus-per-task=2
82+
#SBATCH --partition=<partition name>
83+
84+
WORKDIR=/projappl/project_nnnnnnn
85+
SCRATCH=/scratch/project_nnnnnnn
86+
export TMPDIR=$WORKDIR/tmp
87+
export TOIL_SLURM_ARGS="--account=project_nnnnnnn --partition=small"
88+
export CWL_SINGULARITY_CACHE=$WORKDIR/singularity
89+
unset XDG_RUNTIME_DIR
90+
91+
TOIL_SLURM_ARGS="--account=<project_nnnnnnn> --partition=<partition name>" toil-cwl-runner \
92+
--jobStore $WORKDIR/<job store name> \
93+
--workDir $SCRATCH/<work dir name> \
94+
--tmpdir-prefix $TMPDIR/<tmp name> \
95+
--batchSystem slurm \
96+
$WORKDIR/workflow.cwl \
97+
--message "message for job"
98+
```
99+
100+
Send your workflow to the cluster.
101+
```
102+
sbatch workflow.sh
103+
```
45104

46105
## Monitoring a running workflow

0 commit comments

Comments
 (0)