Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
bf50cc1
Restructure the layout of the intro and outro for Side Quests
vdauwera Nov 25, 2025
fd98e1d
Fix formatting
vdauwera Nov 25, 2025
c07639f
Argh stupid formatting
vdauwera Nov 25, 2025
2539f0f
hmmmmppphhh
vdauwera Nov 25, 2025
3e50790
Experimenting some more with the list formatting
vdauwera Nov 25, 2025
fa362bb
it should be groovy not nextflow
vdauwera Nov 25, 2025
5f44e88
i swear it's like talking to a genie
vdauwera Nov 25, 2025
e99779f
i think I got it
vdauwera Nov 25, 2025
374fe00
Tone down the subheadings in the Get started section
vdauwera Nov 25, 2025
6c88fdb
streamlined
vdauwera Nov 25, 2025
0335b25
Metadata layout/intro/outro should be all set now
vdauwera Nov 25, 2025
6f2c5d3
Updated split & group
vdauwera Nov 25, 2025
e991638
Fix formatting
vdauwera Nov 25, 2025
7740bd7
The auto-style-correct keeps borking my spacing goddammit
vdauwera Nov 25, 2025
09043c3
THERE ARE FOUR SPACES
vdauwera Nov 25, 2025
a4b7221
it's like dark incantations
vdauwera Nov 25, 2025
ec9f5e2
metadata is correct, now fix split & group
vdauwera Nov 25, 2025
c4e86d7
list getting parsed ok, now let's get the groovy highlighting
vdauwera Nov 25, 2025
f1e9b9e
please let this work
vdauwera Nov 25, 2025
8c8db02
apply new layout to Files side quest
vdauwera Nov 25, 2025
234289c
Applied the new layout to WoW
vdauwera Nov 25, 2025
f4d4b01
Improve snippets in WwF
vdauwera Nov 25, 2025
e6aa009
small tweak
vdauwera Nov 25, 2025
c3e0cb2
Applied new layout to debugging SQ
vdauwera Nov 26, 2025
f3a0d9b
oops this got left out
vdauwera Nov 26, 2025
66da440
Applied layout to nf-test and first part of scripting patterns
vdauwera Nov 26, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
80 changes: 54 additions & 26 deletions docs/side_quests/debugging.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,15 +2,9 @@

Debugging is a critical skill that can save you hours of frustration and help you become a more effective Nextflow developer. Throughout your career, especially when you're starting out, you'll encounter bugs while building and maintaining your workflows. Learning systematic debugging approaches will help you identify and resolve issues quickly.

## What you should know first
### Learning goals

This guide assumes you've completed the [Hello Nextflow](../hello_nextflow/) training course and are comfortable with foundational Nextflow concepts including basic workflow structure, processes, channels, and configuration.

This guide focuses on debugging techniques and workflows. For comprehensive coverage of IDE features that support debugging (syntax highlighting, error detection, etc.), see the dedicated [IDE Features for Nextflow Development](ide_features.md) side quest. We recommend completing the IDE training beforehand.

## What you'll learn here

This guide focuses on **systematic debugging techniques** for Nextflow workflows:
In this side quest, we'll explore **systematic debugging techniques** for Nextflow workflows:

- **Syntax error debugging**: Using IDE features and Nextflow error messages effectively
- **Channel debugging**: Diagnosing data flow issues and channel structure problems
Expand All @@ -20,11 +14,29 @@ This guide focuses on **systematic debugging techniques** for Nextflow workflows

By the end, you'll have a robust debugging methodology that transforms frustrating error messages into clear roadmaps for solutions.

### Prerequisites

Before taking on this side quest, you should:

- Have completed the [Hello Nextflow](../hello_nextflow/README.md) tutorial or equivalent beginner's course.
- Be comfortable using basic Nextflow concepts and mechanisms (processes, channels, operators)

**Optional:** We recommend completing the [IDE Features for Nextflow Development](./ide_features.md) side quest first.
That covers comprehensive coverage of IDE features that support debugging (syntax highlighting, error detection, etc.), which we'll use heavily here.

---

## 0. Warmup
## 0. Get started

#### Open the training codespace

Let's move into the debugging exercise directory:
If you haven't yet done so, make sure to open the training environment as described in the [Environment Setup](../envsetup/index.md).

[![Open in GitHub Codespaces](https://github.com/codespaces/badge.svg)](https://codespaces.new/nextflow-io/training?quickstart=1&ref=master)

#### Move into the project directory

Let's move into the directory where the files for this tutorial are located.

```bash
cd side-quests/debugging
Expand All @@ -36,13 +48,11 @@ You can set VSCode to focus on this directory:
code .
```

The directory contains example workflows with various types of bugs that we'll use for practice:
#### Review the materials

```bash
tree .
```
You'll find a set of example workflows with various types of bugs that we'll use for practice:

```console title="Project structure"
```console title="Directory contents"
.
├── bad_bash_var.nf
├── bad_channel_shape.nf
Expand All @@ -67,19 +77,26 @@ tree .
├── missing_software_with_stub.nf
├── nextflow.config
└── no_such_var.nf

1 directory, 22 files
```

These files represent common debugging scenarios you'll encounter in real-world development.

### Takeaway
#### Review the assignment

You're set up with example files containing various types of bugs that we'll debug systematically throughout this guide.
Your challenge is to run each workflow, identify the error(s) and fix it.

### What's next?
<!-- TODO: give a bit more details, similar to how it's done in the Metadata side quest -->

#### Readiness checklist

Think you're ready to dive in?

- [ ] I understand the goal of this course and its prerequisites
- [ ] My codespace is up and running
- [ ] I've set my working directory appropriately
- [ ] I understand the assignment

Learn to identify and fix the most common type of error: syntax errors.
If you can check all the boxes, you're good to go.

---

Expand Down Expand Up @@ -2285,7 +2302,12 @@ Now it's time to put the systematic debugging approach into practice. The workfl

## Summary

In this side quest, we've learned:
In this side quest, you've learned a set of systematic techniques for debugging Nextflow workflows.
Applying these techniques in your own work will enable you to spend less time fighting your computer, solve problems faster and protect yourself from future issues.

### Key patterns

<!-- TODO: Can we add snippets of code below to illustrate? -->

**1. How to identify and fix syntax errors**:

Expand Down Expand Up @@ -2315,8 +2337,6 @@ In this side quest, we've learned:
- Applying resume for efficient debugging cycles
- Following a four-phase systematic debugging methodology

---

!!! tip "Quick Debugging Reference"

**Syntax errors?** → Check VSCode warnings, run `nextflow run workflow.nf -preview`
Expand All @@ -2333,7 +2353,11 @@ In this side quest, we've learned:

**Made fixes?** → Use `-resume` to save time testing: `nextflow run workflow.nf -resume`

### What's next?
---

### Additional resources

<!-- TODO: Are there any specific articles to call out / link to? -->

Check out the [Nextflow documentation](https://www.nextflow.io/docs/latest/) for more advanced debugging features and best practices. You might want to:

Expand All @@ -2343,6 +2367,10 @@ Check out the [Nextflow documentation](https://www.nextflow.io/docs/latest/) for
- Learn about other debugging tools like profiling and performance analysis
- Explore more advanced debugging techniques for complex workflows

Remember: Effective debugging is a skill that improves with practice. The systematic methodology and comprehensive toolkit you've developed here will serve you well throughout your Nextflow development journey.
**Remember:** Effective debugging is a skill that improves with practice. The systematic methodology and comprehensive toolkit you've acquired here will serve you well throughout your Nextflow development journey.

---

## What's next?

Return to the [menu of Side Quests](./index.md) or click the button in the bottom right of the page to move on to the next topic in the list.
56 changes: 40 additions & 16 deletions docs/side_quests/essential_scripting_patterns.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,10 @@ You can write a lot of Nextflow without venturing beyond basic syntax for variab

However, when you need to manipulate data, parse complex filenames, implement conditional logic, or build robust production workflows, it helps to think about two distinct aspects of your code: **dataflow** (channels, operators, processes, and workflows) and **scripting** (the code inside closures, functions, and process scripts). While this distinction is somewhat arbitrary—it's all Nextflow code—it provides a useful mental model for understanding when you're orchestrating your pipeline versus when you're manipulating data. Mastering both dramatically improves your ability to write clear, maintainable workflows.

This side quest takes you on a hands-on journey from basic concepts to production-ready patterns. We'll transform a simple CSV-reading workflow into a sophisticated bioinformatics pipeline, evolving it step-by-step through realistic challenges:
### Learning goals

This side quest takes you on a hands-on journey from basic concepts to production-ready patterns.
We'll transform a simple CSV-reading workflow into a sophisticated bioinformatics pipeline, evolving it step-by-step through realistic challenges:

- **Understanding boundaries:** Distinguish between dataflow operations and scripting, and understand how they work together
- **Data manipulation:** Extract, transform, and subset maps and collections using powerful operators
Expand All @@ -17,32 +20,40 @@ This side quest takes you on a hands-on journey from basic concepts to productio
- **Safe operations:** Handle missing data gracefully with null-safe operators and validate inputs with clear error messages
- **Configuration-based handlers:** Use workflow event handlers for logging, notifications, and lifecycle management

---
### Prerequisites

## 0. Warmup
Before taking on this side quest, you should:

### 0.1. Prerequisites
- Have completed the [Hello Nextflow](../hello_nextflow/README.md) tutorial or equivalent beginner's course.
- Be comfortable using basic Nextflow concepts and mechanisms (processes, channels, operators, working with files, meta data)
- Have basic familiarity with common programming constructs (variables, maps, lists)

Before taking on this side quest you should:
This tutorial will explain programming concepts as we encounter them, so you don't need extensive programming experience.
We'll start with fundamental concepts and build up to advanced patterns.

- Complete the [Hello Nextflow](../hello_nextflow/README.md) tutorial or have equivalent experience
- Understand basic Nextflow concepts (processes, channels, workflows)
- Have basic familiarity with common programming constructs (variables, maps, lists)
---

This tutorial will explain programming concepts as we encounter them, so you don't need extensive programming experience. We'll start with fundamental concepts and build up to advanced patterns.
## 0. Get started

### 0.2. Starting Point
#### Open the training codespace

Navigate to the project directory:
If you haven't yet done so, make sure to open the training environment as described in the [Environment Setup](../envsetup/index.md).

```bash title="Navigate to project directory"
[![Open in GitHub Codespaces](https://github.com/codespaces/badge.svg)](https://codespaces.new/nextflow-io/training?quickstart=1&ref=master)

#### Move into the project directory

Let's move into the directory where the files for this tutorial are located.

```bash
cd side-quests/essential_scripting_patterns
```

The `data` directory contains sample files and a main workflow file we'll evolve throughout.
#### Review the materials

You'll find a main workflow file and a `data` directory containing example data files.

```console title="Directory contents"
> tree
.
├── collect.nf
├── data
Expand All @@ -57,8 +68,6 @@ The `data` directory contains sample files and a main workflow file we'll evolve
│ ├── generate_report.nf
│ └── trimgalore.nf
└── nextflow.config

3 directories, 10 files
```

Our sample CSV contains information about biological samples that need different processing based on their characteristics:
Expand All @@ -72,6 +81,21 @@ SAMPLE_003,human,kidney,45000000,data/sequences/SAMPLE_003_S3_L001_R1_001.fastq,

We'll use this realistic dataset to explore practical programming techniques that you'll encounter in real bioinformatics workflows.

<!-- TODO: Can we make this more domain-agnostic? -->

<!-- TODO: add an assignment statement? #### Review the assignment -->

#### Readiness checklist

Think you're ready to dive in?

- [ ] I understand the goal of this course and its prerequisites
- [ ] My codespace is up and running
- [ ] I've set my working directory appropriately
<!-- - [ ] I understand the assignment -->

If you can check all the boxes, you're good to go.

---

## 1. Dataflow vs Scripting: Understanding the Boundaries
Expand Down
Loading