1.10.1 - Copper Camel [patch]

Patch release to fix the automatic template synchronisation, which failed in the v1.10 release.

• Improved logging: nf-core --log-file log.txt now saves a verbose log to disk.
• nf-core/tools GitHub Actions pipeline sync now uploads verbose log as an artifact.
• Sync - fixed several minor bugs, made logging less verbose.
• Python Rich library updated to >=4.2.1
• Hopefully fix git config for pipeline sync so that commit comes from @nf-core-bot
• Fix sync auto-PR text indentation so that it doesn't all show as code
• Added explicit flag --show-passed for nf-core lint instead of taking logging verbosity

1.10 - Copper Camel

Pipeline schema

This release of nf-core/tools introduces a major change / new feature: pipeline schema.
These are JSON Schema files that describe all of the parameters for a given
pipeline with their ID, a description, a longer help text, an optional default value, a variable type
(eg. string or boolean) and more.

The files will be used in a number of places:

• Automatic validation of supplied parameters when running pipelines
  • Pipeline execution can be immediately stopped if a required param is missing,
    or does not conform to the patterns / allowed values in the schema.
• Generation of pipeline command-line help
  • Running nextflow run <pipeline> --help will use the schema to generate a help text automatically
• Building online documentation on the nf-core website
• Integration with 3rd party graphical user interfaces

To support these new schema files, nf-core/tools now comes with a new set of commands: nf-core schema.

• Pipeline schema can be generated or updated using nf-core schema build - this takes the parameters from
  the pipeline config file and prompts the developer for any mismatch between schema and pipeline.
  • Once a skeleton Schema file has been built, the command makes use of a new nf-core website tool to provide
    a user friendly graphical interface for developers to add content to their schema: https://nf-co.re/pipeline_schema_builder
• Pipelines will be automatically tested for valid schema that describe all pipeline parameters using the
  nf-core schema lint command (also included as part of the main nf-core lint command).
• Users can validate their set of pipeline inputs using the nf-core schema validate command.

In addition to the new schema commands, the nf-core launch command has been completely rewritten from
scratch to make use of the new pipeline schema. This command can use either an interactive command-line
prompt or a rich web interface to help users set parameters for a pipeline run.

The parameter descriptions and help text are fully used and embedded into the launch interfaces to make
this process as user-friendly as possible. We hope that it's particularly well suited to those new to nf-core.

Whilst we appreciate that this new feature will add a little work for pipeline developers, we're excited at
the possibilities that it brings. If you have any feedback or suggestions, please let us know either here on
GitHub or on the nf-core #json-schema Slack channel.

Python code formatting

We have adopted the use of the Black Python code formatter.
This ensures a harmonised code formatting style throughout the package, from all contributors.
If you are editing any Python code in nf-core/tools you must now pass the files through Black when
making a pull-request. See .github/CONTRIBUTING.md for details.

Template

• Add --publish_dir_mode parameter #585
• Isolate R library paths to those in container #541
• Added new style of pipeline parameters JSON schema to pipeline template
• Add ability to attach MultiQC reports to completion emails when using mail
• Update output.md and add in 'Pipeline information' section describing standard NF and pipeline reporting.
• Build Docker image using GitHub Actions, then push to Docker Hub (instead of building on Docker Hub)
• Add Slack channel badge in pipeline README
• Allow multiple container tags in ci.yml if performing multiple tests in parallel
• Add AWS CI tests and full tests GitHub Actions workflows
• Update AWS CI tests and full tests secrets names
• Added macs_gsize for danRer10, based on this post
• Add information about config files used for workflow execution (workflow.configFiles) to summary
• Fix markdown_to_html.py to work with Python 2 and 3.
• Change params.reads -> params.input
• Change params.readPaths -> params.input_paths
• Added a .github/.dockstore.yml config file for automatic workflow registration with dockstore.org

Linting

• Refactored PR branch tests to be a little clearer.
• Linting error docs explain how to add an additional branch protecton rule to the branch.yml GitHub Actions workflow.
• Adapted linting docs to the new PR branch tests.
• Failure for missing the readme bioconda badge is now a warn, in case this badge is not relevant
• Added test for template {{ cookiecutter.var }} placeholders
• Fix failure when providing version along with build id for Conda packages
• New --json and --markdown options to print lint results to JSON / markdown files
• Linting code now automatically posts warning / failing results to GitHub PRs as a comment if it can
• Added AWS GitHub Actions workflows linting
• Fail if params.input isn't defined.
• Beautiful new progress bar to look at whilst linting is running and awesome new formatted output on the command line 😍
  • All made using the excellent rich python library - check it out!
• Tests looking for TODO strings should now ignore editor backup files. #477

nf-core/tools Continuous Integration

• Added CI test to check for PRs against master in tools repo
• CI PR branch tests fixed & now automatically add a comment on the PR if failing, explaining what is wrong
• Move some of the issue and PR templates into HTML <!-- comments --> so that they don't show in issues / PRs

Other

• Describe alternative installation method via conda with conda env create
• nf-core/tools version number now printed underneath header artwork
• Bumped Conda version shipped with nfcore/base to 4.8.2
• Added log message when creating new pipelines that people should talk to the community about their plans
• Fixed 'on completion' emails sent using the mail command not containing body text.
• Improved command-line help text for nf-core/tools
• nf-core list now hides archived pipelines unless --show_archived flag is set
• Command line tools now checks if there is a new version of nf-core/tools available
  • Disable this by setting the environment variable NFCORE_NO_VERSION_CHECK, eg. export NFCORE_NO_VERSION_CHECK=1
• Better command-line output formatting of nearly all nf-core commands using rich

1.9 - Platinum Pigeon

Continuous integration

• Travis CI tests are now deprecated in favor of GitHub Actions within the pipeline template.
  • nf-core bump-version support has been removed for .travis.yml
  • nf-core lint now fails if a .travis.yml file is found
• Ported nf-core/tools Travis CI automation to GitHub Actions
• Fixed the build for the nf-core/tools API documentation on the website

Template

• Rewrote the documentation markdown > HTML conversion in Python instead of R
• Removed the requirement for R in the conda environment
• Make params.multiqc_config give an additional MultiQC config file instead of replacing the one that ships with the pipeline
• Ignore only tests/ and testing/ directories in .gitignore to avoid ignoring test.config configuration file
• Rephrase docs to promote usage of containers over Conda to ensure reproducibility
• Stage the workflow summary YAML file within MultiQC work directory

Linting

• Removed linting for CircleCI
• Allow any one of params.reads or params.input or params.design before warning
• Added whitespace padding to lint error URLs
• Improved documentation for lint errors
• Allow either >= or !>= in nextflow version checks (the latter exits with an error instead of just warning) #506
• Check that manifest.version ends in dev and throw a warning if not
  • If running with --release check the opposite and fail if not
• Tidied up error messages and syntax for linting GitHub actions branch tests
• Add YAML validator
• Don't print test results if we have a critical error

Other

• Fix automatic synchronisation of the template after releases of nf-core/tools
• Improve documentation for installing nf-core/tools
• Replace preprint by the new nf-core publication in Nature Biotechnology 🍾
• Use stderr instead of stdout for header artwork
• Tolerate unexpected output from nextflow config command
• Add social preview image
• Added a release checklist for the tools repo

1.8 - Black Sheep

Continuous integration

• GitHub Actions CI workflows are now included in the template pipeline
  • Please update these files to match the existing tests that you have in .travis.yml
• Travis CI tests will be deprecated from the next tools release
• Linting will generate a warning if GitHub Actions workflows do not exist and if applicable to remove Travis CI workflow file i.e. .travis.yml.

Tools helper code

• Refactored the template synchronisation code to be part of the main nf-core tool
• nf-core bump-version now also bumps the version string of the exported conda environment in the Dockerfile
• Updated Blacklist of synced pipelines
• Ignore pre-releases in nf-core list
• Updated documentation for nf-core download
• Fixed typo in nf-core launch final command
• Handle missing pipeline descriptions in nf-core list

Linting

• Adjusted linting to enable patch branches from being tested
• Warn if GitHub Actions workflows do not exist, warn if .travis.yml and circleCI are there
• Lint for Singularity file and raise error if found #458
• Added linting of GitHub Actions workflows linting.yml, ci.yml and branch.yml
• Warn if pipeline name contains upper case letters or non alphabetical characters #85
• Make CI tests of lint code pass for releases

Template pipeline

• Fixed incorrect paths in iGenomes config as described in issue #418
• Fixed incorrect usage of non-existent parameter in the template #446
• Add UCSC genomes to igenomes.config and add paths to all genome indices
• Change maxMultiqcEmailFileSize parameter to max_multiqc_email_size
• Export conda environment in Docker file #349
• Change remaining parameters from camelCase to snake_case #39
  • --singleEnd to --single_end
  • --igenomesIgnore to --igenomes_ignore
  • Having the old camelCase versions of these will now throw an error
• Add autoMounts=true to default singularity profile
• Add in markdownlint checks that were being ignored by default
• Disable ansi logging in the travis CI tests
• Move paramssection from base.config to nextflow.config
• Use env scope to export PYTHONNOUSERSITE in nextflow.config to prevent conflicts with host Python environment
• Bump minimum Nextflow version to 19.10.0 - required to properly use env scope in nextflow.config
• Added support for nf-tower in the travis tests, using public mailbox [email protected]
• Add link to Keep a Changelog and Semantic Versioning to CHANGELOG
• Adjusted .travis.yml checks to allow for patch branches to be tested
• Add Python 3.7 dependency to the environment.yml file
• Remove awsbatch profile cf nf-core/configs#71
• Make scrape_software_versions.py compatible with Python3 to enable miniconda3 in base image PR
• Add GitHub Actions workflows and respective linting
• Add NXF_ANSI_LOG as global environment variable to template GitHub Actions CI workflow
• Fixed global environment variable in GitHub Actions CI workflow
• Add --awscli parameter
• Add README.txt path for genomes in igenomes.config nf-core/atacseq#75
• Fix buggy ANSI codes in pipeline summary log messages
• Add a TODO line in the new GitHub Actions CI test files

Base Docker image

• Use miniconda3 instead of miniconda for a Python 3k base environment
  • If you still need Python 2 for your pipeline, add conda-forge::python=2.7.4 to the dependencies in your environment.yml
• Update conda version to 4.7.12

Other

• Updated Base Dockerfile to Conda 4.7.10
• Entirely switched from Travis-Ci.org to Travis-Ci.com for template and tools
• Improved core documentation (-profile)

1.7 - Titanium Kangaroo

PyPI package description

• The readme should now be rendered properly on PyPI.

Tools helper code

• The tools create command now sets up a TEMPLATE and a dev branch for syncing
• Fixed issue 379
• nf-core launch now uses stable parameter schema version 0.1.0
• Check that PR from patch or dev branch is acceptable by linting
• Made code compatible with Python 3.7
• The download command now also fetches institutional configs from nf-core/configs
• When listing pipelines, a nicer message is given for the rare case of a detached HEAD ref in a locally pulled pipeline. #297
• The download command can now compress files into a single archive.
• nf-core create now fetches a logo for the pipeline from the nf-core website

Syncing

• Can now sync a targeted pipeline via command-line
• Updated Blacklist of synced pipelines
• Removed chipseq from Blacklist of synced pipelines
• Fixed issue #314

Linting

• If the container slug does not contain the nf-core organisation (for example during development on a fork), linting will raise a warning, and an error with release mode on

Template

• Add new code for Travis CI to allow PRs from patch branches too
• Fix small typo in central readme of tools for future releases
• Small code polishing + typo fix in the template main.nf file
• Header ANSI codes no longer print [2m to console when using -with-ansi
• Switched to yaml.safe_load() to fix PyYAML warning that was thrown because of a possible exploit
• Add nf-core citation
• Add proper nf-core logo for tools
• Add Quick Start section to main README of template
• Fix Docker RunOptions to get UID and GID set in the template
• Dockerfile now specifically uses the proper release tag of the nfcore/base image
• Use file instead of new File
  to avoid weird behavior such as making an s3:/ directory locally when using
  an AWS S3 bucket as the --outdir.
• Fix workflow.onComplete() message when finishing pipeline
• Update URL for joining the nf-core slack to https://nf-co.re/join/slack
• Increased default time limit to 4h
• Add direct link to the pipeline slack channel in the contribution guidelines
• Add contributions and support heading with links to contribution guidelines and link to the pipeline slack channel in the main README
• Fix Parameters JSON due to new versionized structure
• Added conda-forge::r-markdown=1.1 and conda-forge::r-base=3.6.1 to environment
• Plain-text email template now has nf-core ASCII artwork
• Template configured to use logo fetched from website
• New option --email_on_fail which only sends emails if the workflow is not successful
• Add file existence check when checking software versions
• Fixed issue [https://github.com//issues/165] - Use checkIfExists
• Consistent spacing for if statements
• Add sensible resource labels to base.config

Other

• Bump conda to 4.6.14 in base nf-core Dockerfile
• Added a Code of Conduct to nf-core/tools, as only the template had this before
• TravisCI tests will now also start for PRs from patch branches, to allow fixing critical issues without making a new major release

1.6 - Brass Walrus

Syncing

• Code refactoring to make the script more readable
• No travis build failure anymore on sync errors
• More verbose logging

Template pipeline

• awsbatch work-dir checking moved to nextflow itself. Removed unsatisfiable check in main.nf template.
• Fixed markdown linting
• Tools CI testing now runs markdown lint on compiled template pipeline
• Migrated large portions of documentation to the nf-core website
• Removed Gitter references in .github/ directories for tools/ and pipeline template.
• Changed scrape_software_versions.py to output .csv file
• Added export_plots parameter to multiqc config

Tools helper code

• Drop nf-core/rnaseq from blacklist.json to make template sync available
• Updated main help command to sort the subcommands in a more logical order
• Updated readme to describe the new nf-core launch command
• Fix bugs in nf-core download
  • The latest release is now fetched by default if not specified
  • Downloaded pipeline files are now properly executable.
• Fixed bugs in nf-core list
  • Sorting now works again
  • Output is partially coloured (better highlighting out of date pipelines)
  • Improved documentation
• Fixed bugs in nf-core lint
  • The order of conda channels is now correct, avoiding occasional erroneous errors that packages weren't found (#207)
• Add reporting of ignored errored process
  • As a solution for #103)

1.5 - Iron Shark

Template pipeline

• Dropped Singularity file
• Summary now logs details of the cluster profile used if from nf-core/configs
• Dockerhub is used in favor of Singularity Hub for pulling when using the Singularity profile
• Changed default container tag from latest to dev
• Brought the logo to life
• Change the default filenames for the pipeline trace files
• Remote fetch of nf-core/configs profiles fails gracefully if offline
• Remove params.container and just directly define process.container now
• Completion email now includes MultiQC report if not too big
• params.genome is now checked if set, to ensure that it's a valid iGenomes key
• Together with nf-core/configs, helper function now checks hostname and suggests a valid config profile

Tools helper code

• New nf-core launch command to interactively launch nf-core pipelines from command-line

  • Works with a parameters.settings.json file shipped with each pipeline
  • Discovers additional params from the pipeline dynamically

• Drop Python 3.4 support

• nf-core list now only shows a value for "is local latest version" column if there is a local copy.

• Lint markdown formatting in automated tests

  • Added markdownlint-cli for checking Markdown syntax in pipelines and tools repo

• Syncing now reads from a blacklist.json in order to exclude pipelines from being synced if necessary.

• Added nf-core tools API description to assist developers with the classes and functions available.

  • Docs are automatically built by Travis CI and updated on the nf-co.re website.

• Introduced test for filtering remote workflows by keyword.

• Build tools python API docs

  • Use Travis job for api doc generation and publish

• nf-core bump-version now stops before making changes if the linting fails

• Code test coverage

  • Introduced test for filtering remote workflows by keyword

• Linting updates

  • Now properly searches for conda packages in default channels
  • Now correctly validates version pinning for packages from PyPI
  • Updates for changes to process.container definition

Other

• Bump conda to 4.6.7 in base nf-core Dockerfile

1.4 - Tantalum Butterfly

Template pipeline

• Institutional custom config profiles moved to github nf-core/configs
  • These will now be maintained centrally as opposed to being shipped with the pipelines in conf/
  • Load base.config by default for all profiles
  • Removed profiles named standard and none
  • Added parameter --igenomesIgnore so igenomes.config is not loaded if parameter clashes are observed
  • Added parameter --custom_config_version for custom config version control. Can use this parameter to provide commit id for reproducibility. Defaults to master
  • Deleted custom configs from template in conf/ directory i.e. uzh.config, binac.config and cfc.config
• multiqc_config and output_md are now put into channels instead of using the files directly (see issue #222)
• Added local.md to cookiecutter template in docs/configuration/. This was referenced in README.md but not present.
• Major overhaul of docs to add/remove parameters, unify linking of files and added description for providing custom configs where necessary
• Travis: Pull the dev tagged docker image for testing
• Removed UPPMAX-specific documentation from the template.

Tools helper code

• Make Travis CI tests fail on pull requests if the CHANGELOG.md file hasn't been updated
• Minor bugfixing in Python code (eg. removing unused import statements)
• Made the web requests caching work on multi-user installations
• Handle exception if nextflow isn't installed
• Linting: Update for Travis: Pull the dev tagged docker image for testing

1.3 - Citreous Swordfish

Changes:

• nf-core create command line interface updated
  • Interactive prompts for required arguments if not given
  • New flag for workflow author
• Updated channel order for bioconda/conda-forge channels in environment.yaml
• Increased code coverage for sub command create and licenses
• Fixed nasty dependency hell issue between pytest and py package in Python 3.4.x
• Introduced .coveragerc for pytest-cov configuration, which excludes the pipeline template now from being reported
• Fix 189: Check for given conda and PyPi package dependencies, if their versions exist
• Added profiles for cfc,binac, uzh that can be synced across pipelines
  • Ordering alphabetically for profiles now
• Added pip install --upgrade pip to .travis.yml to update pip in the Travis CI environment

nf-core/tools version 1.2

• Updated the nf-core release command
  • Now called nf-core bump-versions instead
  • New flag --nextflow to change the required nextflow version instead
• Template updates
  • Simpler installation of the nf-core helper tool, now directly from PyPI
  • Bump minimum nextflow version to 0.32.0 - required for built in manifest.nextflowVersion check and access to workflow.manifest variables from within nextflow scripts
  • New withName syntax for configs
  • Travis tests fail if PRs come against the master branch, slightly refactored
  • Improved GitHub contributing instructions and pull request / issue templates
• New lint tests
  • .travis.yml test for PRs made against the master branch
  • Automatic --release option not used if the travis repo is nf-core/tools
  • Warnings if depreciated variables params.version and params.nf_required_version are found
• New nf-core licences subcommand to show licence for each conda package in a workflow
• nf-core list now has options for sorting pipeline nicely
• Latest version of conda used in nf-core base docker image
• Updated PyPI deployment to correctly parse the markdown readme (hopefully!)
• New GitHub contributing instructions and pull request template