diff --git a/README.md b/README.md index 40f2f6e..e66d1de 100644 --- a/README.md +++ b/README.md @@ -19,6 +19,24 @@ [![Get help on Slack](http://img.shields.io/badge/slack-nf--core%20%23seqsubmit-4A154B?labelColor=000000&logo=slack)](https://nfcore.slack.com/channels/seqsubmit)[![Follow on Bluesky](https://img.shields.io/badge/bluesky-%40nf__core-1185fe?labelColor=000000&logo=bluesky)](https://bsky.app/profile/nf-co.re)[![Follow on Mastodon](https://img.shields.io/badge/mastodon-nf__core-6364ff?labelColor=FFFFFF&logo=mastodon)](https://mstdn.science/@nf_core)[![Watch on YouTube](http://img.shields.io/badge/youtube-nf--core-FF0000?labelColor=000000&logo=youtube)](https://www.youtube.com/c/nf-core) +## Table of Contents + +- [Introduction](#introduction) +- [Requirements](#requirements) +- [Input samplesheets](#input-samplesheets) +- [Usage](#usage) + - [Submission study](#submission-study) + - [Data privacy](#data-privacy) + - [Database setup (`CheckM2` and `CAT_pack`)](#database-setup-checkm2-and-cat_pack) + - [Required parameters:](#required-parameters) + - [Optional parameters:](#optional-parameters) +- [Pipeline output](#pipeline-output) +- [Scalability](#scalability) +- [Limitations](#limitations) +- [Credits](#credits) +- [Contributions and Support](#contributions-and-support) +- [Citations](#citations) + ## Introduction **nf-core/seqsubmit** is a Nextflow pipeline for submitting sequence data to [ENA](https://www.ebi.ac.uk/ena/browser/home). @@ -35,9 +53,11 @@ Currently, the pipeline supports four submission modes, each routed to a dedicat - [Nextflow](https://www.nextflow.io/) `>=25.04.0` - Webin account registered at https://www.ebi.ac.uk/ena/submit/webin/login -- Raw reads used to assemble contigs submitted to [INSDC](https://www.insdc.org/) and associated accessions available +- For metagenomic assemblies submission the raw reads used to generate the assemblies have to be submitted to INSDC/ENA to obtain the corresponding run accessions. You can use the pipeline in mode `reads` to perform this. +- For MAGs or bins submission the raw reads only or raw reads and metagenomic assemblies used to generate the MAGs/bins have to be submitted to INSDC/ENA to obtain the corresponding run or assembly accessions. You can use the pipeline in mode `reads` or `metagenomic_assemblies` to perform this. +- For raw reads submission it is required to register source BioSamples and obtain the sample accessions. -Setup your environment secrets before running the pipeline: +Setup Webin credentials as Nextflow secrets before running the pipeline: `nextflow secrets set ENA_WEBIN "Webin-XXX"` @@ -47,42 +67,9 @@ Make sure you update commands above with your authorised credentials. ## Input samplesheets -For detailed descriptions of all samplesheet columns, see the [usage documentation](docs/usage.md#samplesheet-input). - -### `mags` and `bins` modes (`GENOMESUBMIT`) - -The input must follow `assets/schema_input_genome.json`. - -Required columns: - -- `sample` -- `fasta` (must end with `.fa.gz`, `.fasta.gz`, or `.fna.gz`) -- `accession` -- `assembly_software` -- `binning_software` -- `binning_parameters` -- `metagenome` -- `environmental_medium` -- `broad_environment` -- `local_environment` -- `co-assembly` - -At least one of the following must be provided per row: +### `mags` and `bins` modes -- reads (`fastq_1`, optional `fastq_2` for paired-end) -- `genome_coverage` - -Additional supported columns: - -- `stats_generation_software` -- `completeness` -- `contamination` -- `RNA_presence` -- `NCBI_lineage` - -If `genome_coverage`, `stats_generation_software`, `completeness`, `contamination`, `RNA_presence`, or `NCBI_lineage` are missing, the workflow can calculate or infer them when the required inputs are available. - -Those fields are metadata required for the [genome_uploader](https://github.com/EBI-Metagenomics/genome_uploader) package. +The input must follow [assets/schema_input_genome.json](assets/schema_input_genome.json). See the [corresponding samplesheet section](docs/usage.md#mags-and-bins-modes-genomesubmit) of the usage documentation for the full list of required and optional columns. Example `samplesheet_genomes.csv`: @@ -91,27 +78,9 @@ sample,fasta,accession,fastq_1,fastq_2,assembly_software,binning_software,binnin lachnospira_eligens,data/bin_lachnospira_eligens.fa.gz,SRR24458089,,,spades_v3.15.5,metabat2_v2.6,default,CheckM2_v1.0.1,61.0,0.21,32.07,sediment metagenome,No,marine,cable_bacteria,marine_sediment,No,d__Bacteria;p__Proteobacteria;s__unclassified_Proteobacteria ``` -> [!IMPORTANT] -> **Samplesheet column requirements**: All columns shown in the example above must be present in your samplesheet, even if some values are empty. Columns must be in exactly the same order as shown. - -### `metagenomic_assemblies` mode (`ASSEMBLYSUBMIT`) - -The input must follow `assets/schema_input_assembly.json`. +### `metagenomic_assemblies` mode -Required columns: - -- `sample` -- `fasta` (must end with `.fa.gz`, `.fasta.gz`, or `.fna.gz`) -- `run_accession` -- `assembler` -- `assembler_version` - -At least one of the following must be provided per row: - -- reads (`fastq_1`, optional `fastq_2` for paired-end) -- `coverage` - -If `coverage` is missing and reads are provided, the workflow calculates average coverage with `coverm`. +The input must follow [assets/schema_input_assembly.json](assets/schema_input_assembly.json). See the [corresponding samplesheet section](docs/usage.md#metagenomic_assemblies-mode-assemblysubmit) of the usage documentation for more details. Example `samplesheet_assembly.csv`: @@ -121,30 +90,9 @@ assembly_1,data/contigs_1.fasta.gz,data/reads_1.fastq.gz,data/reads_2.fastq.gz,, assembly_2,data/contigs_2.fasta.gz,,,42.7,ERR011323,MEGAHIT,1.2.9 ``` -> [!IMPORTANT] -> **Samplesheet column requirements**: All columns shown in the example above must be present in your samplesheet, even if some values are empty. Columns must be in exactly the same order as shown. - -### `reads` mode (`READSUBMIT`) - -The input must follow `assets/schema_input_reads.json`. +### `reads` mode -Required columns: - -- `sample` -- `sample_accession` -- `fastq_1` -- `fastq_2` -- `platform` -- `instrument` -- `library_source` -- `library_selection` -- `library_strategy` - -Optional columns: - -- `insert_size` -- `library_name` -- `description` +The input must follow [assets/schema_input_reads.json](assets/schema_input_reads.json). See the [corresponding samplesheet section](docs/usage.md#reads-mode-readsubmit) of the usage documentation for more details. Example `samplesheet_reads.csv`: @@ -153,9 +101,6 @@ sample,sample_accession,fastq_1,fastq_2,platform,instrument,library_source,libra illumina_run_001,SAMEA1234567,data/reads_R1.fastq.gz,data/reads_R2.fastq.gz,ILLUMINA,Illumina HiSeq 2000,GENOMIC,RANDOM,WGS,500,HiSeq_library_001,Illumina sequencing of sample XYZ ``` -> [!IMPORTANT] -> **Samplesheet column requirements**: All columns shown in the example above must be present in your samplesheet, even if some values are empty. Columns must be in exactly the same order as shown. - ## Usage > [!NOTE] @@ -163,30 +108,15 @@ illumina_run_001,SAMEA1234567,data/reads_R1.fastq.gz,data/reads_R2.fastq.gz,ILLU ### Submission study -All data submitted through this pipeline must be associated with an ENA study (project). You can either pass an accession of your existing study via `--submission_study`or provide a metadata file via `--study_metadata` and the pipeline will register the study with ENA before submitting your data. - -See the [usage documentation](docs/usage.md#submission-study) for more details. +All data submitted through this pipeline must be associated with an ENA study (project). You can either pass an accession of your existing study via `--submission_study`or provide a metadata file via `--study_metadata` and the pipeline will register the study with ENA before submitting your data. See the [usage documentation](docs/usage.md#submission-study) for more details. ### Data privacy -You can reference private ENA data if you have access to it via your Webin account (it was submitted previously under your credentials). To use private data accessions in your submission metadata, specify the `--is_private` flag. The pipeline will then use your provided Webin credentials to fetch the required metadata. - -You can also control the privacy of your submitted data: - -- If you provide an existing study via `--submission_study`, your submission will inherit the same privacy status as that study. -- If no study is provided, the pipeline will register a new one for you. To keep this new study private, you must specify a release date using `--release_date YYYY-MM-DD` (up to 2 years from the current date). - -| Example | Source Data | Submission Visibility | Required Arguments | Result | -| --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------- | --------------------- | ------------------------------------------------------ | ----------------------------------------------------- | -| Assemblies generated from public reads. Assemblies should be public immediately after submission | public | public | – | SUCCESS | -| Assemblies generated from public reads. Assemblies should remain private for 1 year after submission | public | private | `--release_date YYYY-MM-DD` (+1 year) | SUCCESS | -| Bins generated from private reads and assemblies. Bins should be public after submission. User **has access** to the reads and assemblies via Webin account | private | public | `--is_private` | SUCCESS | -| Bins generated from private reads and assemblies. Bins should remain private for 2 years after submission. User **has access** via Webin account | private | private | `--is_private`, `--release_date YYYY-MM-DD` (+2 years) | SUCCESS | -| MAGs generated from private reads and assemblies. MAGs should be public after submission. User **does not have access** to the reads and assemblies via Webin account | private | public | – | ERROR (submission can only reference accessible data) | +The pipeline supports private submissions. See the [usage documentation](docs/usage.md#data-privacy) for more details. ### Database setup (`CheckM2` and `CAT_pack`) -The `mags`/`bins` workflow requires databases for completeness/contamination estimation and taxonomy assignment. See [Usage documentation](usage.md) for details. +The `mags`/`bins` workflow requires databases for completeness/contamination estimation and taxonomy assignment. See [Usage documentation](usage.md) for more details. ### Required parameters: @@ -205,7 +135,7 @@ The `mags`/`bins` workflow requires databases for completeness/contamination est | `--upload_tpa` | Flag to control the type of assembly study (third party assembly or not). Default: false | | `--test_upload` | Upload to TEST ENA server instead of LIVE. Default: true | | `--webincli_mode` | Choose Webin-CLI mode: `submit` or `validate`. Default: `submit` | -| `--private` | Use that flag if you are referring to private data in ENA | +| `--is_private` | Use that flag if you are referring private data accessions in your submission | | `--release_date` | Use that flag if you want to keep your data private after submission until particular date | General command template: @@ -277,6 +207,44 @@ Key output locations in `--outdir`: For full details, see the [output documentation](https://nf-co.re/seqsubmit/output). +## Scalability + +Some processes in this pipeline are resource intensive — in particular coverage calculation (`coverm`), quality assessment (`CheckM2`), and taxonomic classification of MAGs/bins (`BAT`). For real life data these steps should be run on an HPC cluster or another compute environment with sufficient CPU, memory, and job scheduling capacity, rather than on a single local machine. + +Depending on the resources available on your machine or cluster, you will likely need to adapt the Nextflow configuration to limit the maximum number of concurrently running jobs and other resource settings (e.g. `max_cpus`, `max_memory`, `max_time`, or `executor.queueSize` in a custom config), so the pipeline doesn't try to claim more resources than are actually available. See the [nf-core configuration documentation](https://nf-co.re/docs/usage/getting_started/configuration) for guidance on writing a custom config for your infrastructure. + +When submitting a large number of records, be aware that ENA recommends limiting submissions to 5000 per day to avoid overfilling their processing queue. If you need to submit more than this, consider splitting your samplesheet and spreading the submission across several days. + +## Limitations + +nf-core/seqsubmit does not yet support the following data types and scenarios, grouped by the mode they affect: + +#### `mags` and `bins` modes (`GENOMESUBMIT`) + +- **Co-assemblies**: MAGs/bins generated from co-assemblies are not supported. The pipeline currently expects each MAG/bin to be derived from an assembly generated from a single run. Tracked in [issue #61](https://github.com/nf-core/seqsubmit/issues/61). + +- **Eukaryotic and viral bins/MAGs**: Submitting eukaryotic or viral genomes that require additional metadata generation is not supported. For such genomes tRNA and rRNA prediction, quality assessment (completeness/contamination estimation) and taxonomy assignment are not implemented. If all the required metadata (`completeness`, `contamination`, `stats_generation_software`, `RNA_presence` and `NCBI_lineage`) is already provided in the samplesheet, eukaryotic and viral bins/MAGs can still be submitted normally. Tracked in [issue #40](https://github.com/nf-core/seqsubmit/issues/40) (eukaryotic) and [issue #63](https://github.com/nf-core/seqsubmit/issues/63) (viral). + +- **Submission of a MAG/bin without submitting the assembly or raw reads**: If a MAG/bin cannot be traced back to an assembly accession or a reads accession — i.e. only a sample accession is available — it cannot currently be submitted, since the pipeline relies on that lineage to build the required metadata. Tracked in [issue #87](https://github.com/nf-core/seqsubmit/issues/87). + +- **Single-contig MAGs/bins**: MAGs/bins that consist of only one contig cannot be uploaded through this pipeline. ENA classifies these as "chromosomal assemblies", which follow a different submission procedure with different metadata requirements. If you have a single-contig MAG/bin to submit, please contact [ENA support](https://www.ebi.ac.uk/ena/browser/support) for guidance. + +- **Coverage calculation from long reads**: `genome_coverage` calculation via `coverm` has only been tested and validated with short reads (e.g. Illumina). It has not been tested with long reads (PacBio, Nanopore). Exercise caution when processing data generated from long reads, as issues may occur. + +#### `metagenomic_assemblies` mode (`ASSEMBLYSUBMIT`) + +- **Co-assemblies**: Metagenomic co-assemblies are not supported. The pipeline currently expects each assembly to be derived from a single run. Tracked in [issue #61](https://github.com/nf-core/seqsubmit/issues/61), addressed in [PR #66](https://github.com/nf-core/seqsubmit/pull/66). + +- **Submission of an assembly without submitting the raw reads**: Assemblies that have no associated reads accession (`run_accession` column in the samplesheet) are not supported, since the pipeline relies on that lineage to build the required metadata. Tracked in [issue #29](https://github.com/nf-core/seqsubmit/issues/29). + +- **Single-contig assemblies**: Assemblies that consist of only one contig cannot be uploaded through this pipeline. ENA classifies these as "chromosomal assemblies", which follow a different submission procedure with different metadata requirements. If you have a single-contig assembly to submit, please contact [ENA support](https://www.ebi.ac.uk/ena/browser/support) for guidance. + +- **Coverage calculation from long reads**: `coverage` calculation via `coverm` has only been tested and validated with short reads (e.g. Illumina). It has not been tested with long reads (PacBio, Nanopore). Exercise caution when processing data generated from long reads, as issues may occur. + +#### `reads` mode (`READSUBMIT`) + +- **Reads in non-FASTQ formats**: Only reads provided as FASTQ (`.fastq.gz`) can be submitted. Other formats (e.g. BAM) are not supported as input. + ## Credits nf-core/seqsubmit was originally written by [Martin Beracochea](https://github.com/mberacochea), [Ekaterina Sakharova](https://github.com/KateSakharova), [Sofia Ochkalova](https://github.com/ochkalova), [Evangelos Karatzas](https://github.com/vagkaratzas) and [Tim Rozday](https://github.com/timrozday-mgnify). diff --git a/assets/seqsubmit_schema.svg b/assets/seqsubmit_schema.svg new file mode 100644 index 0000000..2338fce --- /dev/null +++ b/assets/seqsubmit_schema.svg @@ -0,0 +1,114 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docs/CONTRIBUTING.md b/docs/CONTRIBUTING.md index 67a4370..7552e65 100644 --- a/docs/CONTRIBUTING.md +++ b/docs/CONTRIBUTING.md @@ -184,44 +184,12 @@ If you update images or graphics, follow the nf-core [style guidelines](https:// To ensure consistency and maintainability of the nf-core/seqsubmit pipeline, contributors should follow these guidelines when implementing new features or modifying existing functionality. -1. Understand Workflow Structure - -The pipeline consists of independent workflows, each corresponding to a certain data type: - -- GENOMESUBMIT (mags, bins) -- ASSEMBLYSUBMIT (metagenomic_assemblies) -- READSUBMIT (raw reads) - -When adding new functionality (e.g. a new submission mode such as `isolate_genomes`), ensure that: - -- It integrates cleanly into the appropriate workflow or introduces a well-structured new one. -- Existing workflows remain unaffected unless changes are explicitly required. -- Shared logic is abstracted into reusable modules or subworkflows where possible. - -⸻ - -2. Follow Samplesheet and Schema-Driven Design - -All inputs in seqsubmit are defined via strict samplesheet schemas. - -- Any new feature must: - - Extend or introduce a corresponding schema (assets/schema*input*\*.json) - - Clearly define required and optional fields -- Maintain: - - Column order consistency (critical for pipeline execution) - - Backward compatibility where possible -- Avoid duplicating metadata logic—reuse existing parsing and validation patterns. - -⸻ - -3. ENA Submission Requirements +### ENA Submission Requirements Submissions to [ENA](https://www.ebi.ac.uk/ena/browser/home) are governed by a well-defined data model and submission process. These include relationships between entities (e.g. studies, samples, experiments, runs, analyses), required submission steps, and accepted file types and formats. Contributors should refer directly to the [official ENA documentation](https://ena-docs.readthedocs.io/en/latest/submit/general-guide.html) to understand latest submission requirements and ensure that pipeline behaviour remains aligned with ENA standards. - Follow ENA-defined submission workflows and required steps for each data type - - Ensure that: - New pipeline workflows follow ENA-defined submission workflows and required steps for each data type - Metadata fields align with ENA requirements @@ -232,13 +200,13 @@ Contributors should refer directly to the [official ENA documentation](https://e ⸻ -4. Integration with Webin-CLI +### Integration with Webin-CLI All submissions are performed via [Webin-CLI](https://github.com/enasequence/webin-cli). Contributors should rely on official Webin-CLI documentation to understand expected inputs, parameters, and modes of operation. Make sure to always use the latest version of the tool. - Contributions affecting submission logic must: - - Respect --webincli_mode (validate vs submit) - - Support --test_upload for safe testing against ENA test server + - Respect `--webincli_mode` (validate vs submit) + - Support `--test_upload` for safe testing against ENA test server - Never introduce changes that: - Risk accidental submission to production without explicit user intent - Ensure compatibility with: @@ -246,9 +214,9 @@ All submissions are performed via [Webin-CLI](https://github.com/enasequence/web ⸻ -5. Testing Requirements +### Testing Requirements -All new functionality must be covered by tests. +All new functionality must be covered by tests. Running nf-tests that perform ENA submission locally requires ENA webin credentials. Contributors may ask for testing credentials on the nf-core Slack [#seqsubmit](https://nfcore.slack.com/channels/seqsubmit) channel. At minimum, include: @@ -268,36 +236,12 @@ Where applicable: ⸻ -6. Performance and Resource Awareness - -Some data processing steps may be resource-intensive (e.g. database preparation for CAT_pack). When introducing new workflows or features avoid unnecessary recomputation. Add option to reuse cached results or provide prebuilt databases when possible. - -⸻ - -7. Documentation and Usability - -Every contribution should improve usability: - -- Update: - - README.md - - docs/usage.md - - Schema descriptions -- Provide: - - Clear examples (e.g. samplesheets) - - Meaningful parameter descriptions - -The goal is to ensure that users can successfully submit data without needing deep knowledge of ENA submission requirements. - -⸻ - -8. Design Principles +### Performance and Resource Awareness -When contributing, prioritise: +Some data processing steps may be resource-intensive (e.g. database preparation for CAT_pack). -- Reproducibility (consistent results across environments) -- Automation (minimise manual user intervention) -- Clarity (explicit inputs and outputs) -- Extensibility (easy to add new submission types) +- When introducing new workflows or features avoid unnecessary recomputation. +- Add option to reuse cached results or provide prebuilt databases when possible. ⸻ diff --git a/docs/methods.md b/docs/methods.md index 0f15a33..acbf12f 100644 --- a/docs/methods.md +++ b/docs/methods.md @@ -1,15 +1,8 @@ # nf-core/seqsubmit: Methods -## Overview +This page provides a detailed overview of the methods and procedures that are executed throughout the nf-core/seqsubmit pipeline. -`nf-core/seqsubmit` currently contains two workflow implementations: - -- `GENOMESUBMIT` for `--mode mags` and `--mode bins` -- `ASSEMBLYSUBMIT` for `--mode metagenomic_assemblies` - -This page documents the methods that are currently implemented in the pipeline. - -## `GENOMESUBMIT` methods +## `GENOMESUBMIT` ### Overview @@ -17,22 +10,24 @@ The `GENOMESUBMIT` workflow: 1. Reads the samplesheet and associated genome FASTA files. 2. Validates genome FASTA files. -3. Reuses provided or calculates missing values for RNA genes presence, coverage, taxonomy, and genome quality metrics. +3. Reuses provided or calculates missing values for tRNA and rRNA genes presence, coverage, taxonomy, and genome quality metrics. 4. Collects genome metadata into the tabular format required by `genome_uploader`. 5. Generates submission manifests for ENA. 6. Performs submission to ENA. ### Genome FASTA validation -Genome FASTA files are validated with the `FASTAVALIDATOR` module before downstream processing. Each file is checked for FASTA format validity and contig count. A genome must contain at least two contigs to pass validation, which is an ENA requirement for contig-level submissions. +Genome FASTA files are validated with the `fa-lint` before downstream processing. Each file is checked for FASTA format validity and contig count. A genome must contain at least two contigs to pass validation, which is an ENA requirement for contig-level submissions. Only FASTA files that pass validation are retained for downstream processing and submission. -### RNA presence detection +### tRNA and rRNA genes detection -The workflow only runs internal RNA detection for entries where the `RNA_presence` column is empty. If a value is already supplied in the samplesheet, that value is passed through unchanged. +The workflow only runs tRNA and rRNA genes detection for entries where the `RNA_presence` column is empty. If a value is already supplied in the samplesheet, that value is passed through unchanged. -RNA detection is implemented through the `RNA_DETECTION` subworkflow and combines: +The resulting `RNA_presence` value is a decisive factor affecting MISAG/MIMAG assembly quality classification of a genome. + +tRNA and rRNA genes detection is implemented through the `RNA_DETECTION` subworkflow and combines: - `barrnap` for rRNA prediction - `tRNAscan-SE` for tRNA prediction @@ -42,11 +37,7 @@ RNA detection is implemented through the `RNA_DETECTION` subworkflow and combine `barrnap` is run in bacterial mode (`"bac"`). -The custom parser then scans the GFF output and keeps only the following ribosomal RNA subunits: - -- `16S_rRNA` -- `23S_rRNA` -- `5S_rRNA` +The custom parser then scans the GFF output and keeps only the following rRNA subunits: `16S_rRNA`, `23S_rRNA`, `5S_rRNA`. For each detected feature, the recovered length is calculated as: @@ -66,7 +57,7 @@ $$ If multiple hits are found for the same subunit, the workflow keeps the best recovered percentage for that subunit. -A subunit is considered present when its best recovered percentage is greater than or equal to `--rrna_limit`. The current default is `80`. +A subunit is considered present when its best recovered percentage is greater than or equal to `--rrna_limit`. The current default is `80`. This threshold is arbitrary, as no strict consensus length is defined by ENA or MISAG/MIMAG for considering an rRNA gene present. #### tRNA detection @@ -79,16 +70,16 @@ The parser sums the counts assigned to the 20 standard amino-acid isotypes: - `Leu`, `Lys`, `Met`, `Phe`, `Pro` - `Ser`, `Thr`, `Trp`, `Tyr`, `Val` -The total number of predicted tRNAs is compared against `params.trna_limit`. The current default is `18`. +The total number of predicted tRNAs is compared against `params.trna_limit`. The current default is `18`. This value is defined by the MISAG/MIMAG standard and must not be modified. -#### Final RNA presence decision +#### Final `RNA_presence` decision The final decision stored in `RNA_presence` is: - `Yes` when at least 18 tRNAs are detected and all three required rRNA subunits pass the recovery threshold - `No` otherwise -The result is written as a two-column TSV file containing the genome identifier and the final `Yes`/`No` decision, and this value is then merged back into the submission metadata. +The final `Yes`/`No` decision is then merged back into the submission metadata. ### Genome coverage calculation @@ -98,7 +89,7 @@ Genome coverage values from `coverm genome` output TSV are parsed and merged int ### Taxonomy assignment -If `NCBI_lineage` is already present in the input samlesheet, the value is retained. If it is missing, the workflow runs taxonomy classification using the `CAT_pack` tool. +If `NCBI_lineage` is already present in the input samplesheet, the value is retained. If it is missing, the workflow runs taxonomy classification using the `CAT_pack` tool. Before classification, input FASTA files are normalized to a `.fasta` suffix by `RENAME_FASTA_FOR_CATPACK`. Classification is then run in bin mode (`CAT_pack bins`, followed by `CAT_pack add_names`). @@ -112,7 +103,7 @@ The workflow checks three samplesheet fields: `completeness`, `contamination`, a For records that run `CheckM2`, completeness and contamination are extracted from the generated quality report (`quality_report.tsv`) and used `CheckM2` version is recorded as `stats_generation_software`. -## `ASSEMBLYSUBMIT` methods +## `ASSEMBLYSUBMIT` ### Overview @@ -120,14 +111,13 @@ The `ASSEMBLYSUBMIT` workflow: 1. Reads the samplesheet and associated assembly FASTA files. 2. Validates assembly FASTA files. -3. Reuses `coverage` values supplied in the samplesheet when they are already present. -4. Calculates `coverage` internally for entries where this field is missing and reads are available. -5. Builds the metadata table used for manifest generation. -6. Generates assembly manifests and submits the assemblies to ENA. +3. Reuses provided or calculates missing `coverage` values. +4. Builds the metadata table used for manifest generation. +5. Generates assembly manifests and submits the assemblies to ENA. ### Assembly FASTA validation -Assembly FASTA files are validated with `FASTAVALIDATOR` before downstream processing. Each file is checked for FASTA format validity and contig count. A genome must contain at least two contigs to pass validation, which is an ENA requirement for contig-level submissions. +Assembly FASTA files are validated with `fa-lint` before downstream processing. Each file is checked for FASTA format validity and contig count. A genome must contain at least two contigs to pass validation, which is an ENA requirement for contig-level submissions. Only assemblies with successful validation are forwarded to coverage estimation, metadata/manifest generation and submission. diff --git a/docs/usage.md b/docs/usage.md index 7d3b5ef..ad69cdb 100644 --- a/docs/usage.md +++ b/docs/usage.md @@ -4,6 +4,23 @@ > _Documentation of pipeline parameters is generated automatically from the pipeline schema and can no longer be found in markdown files._ +## Table of Contents + +- [Introduction](#introduction) +- [Before you start](#before-you-start) +- [Submission study](#submission-study) +- [Samplesheet input](#samplesheet-input) + - [`mags` and `bins` modes](#mags-and-bins-modes-genomesubmit) + - [`metagenomic_assemblies` mode](#metagenomic_assemblies-mode-assemblysubmit) + - [`reads` mode](#reads-mode-readsubmit) +- [Data privacy](#data-privacy) +- [Database preparation (`mags` / `bins` modes)](#database-preparation-mags--bins) +- [Running the pipeline](#running-the-pipeline) +- [Core Nextflow arguments](#core-nextflow-arguments) +- [Custom configuration](#custom-configuration) +- [Running in the background](#running-in-the-background) +- [Nextflow memory requirements](#nextflow-memory-requirements) + ## Introduction `nf-core/seqsubmit` is a Nextflow pipeline for submitting metagenomic assemblies, MAGs, bins, and raw reads to ENA. @@ -20,121 +37,17 @@ Before running the pipeline, make sure that: - Nextflow `>=25.04.0` is available. - You have a Webin account registered at . -- The raw reads used to generate the submitted assemblies have already been submitted to INSDC/ENA and the relevant accessions are available. - -Set your Webin credentials as Nextflow secrets: - -```bash -nextflow secrets set ENA_WEBIN "Webin-XXX" -nextflow secrets set ENA_WEBIN_PASSWORD "XXX" -``` - -## Samplesheet input - -You will need to create a samplesheet with information about the data entries you would like to process before running the pipeline. Use `--input` parameter to specify its location. It has to be a comma-separated file with the structure defined by the execution `--mode`. - -```bash ---input '[path to samplesheet.csv]' -``` +- For metagenomic assemblies submission the raw reads used to generate the assemblies have to be submitted to INSDC/ENA to obtain the corresponding run accessions. You can use the pipeline in mode `reads` to perform this. +- For MAGs or bins submission the raw reads only or raw reads and metagenomic assemblies used to generate the MAGs/bins have to be submitted to INSDC/ENA to obtain the corresponding run or assembly accessions. You can use the pipeline in mode `reads` or `metagenomic_assemblies` to perform this. +- For raw reads submission it is required to register source BioSamples and obtain the sample accessions. -### `mags` and `bins` modes (`GENOMESUBMIT`) +Setup Webin credentials as Nextflow secrets before running the pipeline: -Use this samplesheet structure for MAG and bin submission. The input format follows [assets/schema_input_genome.json](../assets/schema_input_genome.json). +`nextflow secrets set ENA_WEBIN "Webin-XXX"` -Example: - -```csv title="samplesheet_genomes.csv" -sample,fasta,accession,fastq_1,fastq_2,assembly_software,binning_software,binning_parameters,stats_generation_software,completeness,contamination,genome_coverage,metagenome,co-assembly,broad_environment,local_environment,environmental_medium,RNA_presence,NCBI_lineage -mag_001,data/mag_001.fasta.gz,SRR24458089,,,SPAdes 3.15.5,MetaBAT2 2.15,default,CheckM2 1.0.1,92.81,1.09,66.04,sediment metagenome,No,marine,cable bacteria,marine sediment,No,d__Bacteria;p__Proteobacteria;s__ -``` - -> [!IMPORTANT] -> **Samplesheet column requirements**: All columns shown in the example above must be present in your samplesheet, even if some values are empty. Columns must be in exactly the same order as shown. - -| Column | Description | -| --------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `sample` | A unique identifier for this data entry. Must be globally unique within the input dataset (used as `meta.id` throughout the pipeline). | -| `fasta` | Path to MAG/bin contigs in FASTA format compressed with `gzip`. All names of the FASTA files must be unique to prevent pipeline errors. | -| `accession` | ENA accession of the run or metagenomic assembly used to generate the MAG/bin. | -| `fastq_1` | Path to the read file in FASTQ format used to generate the source metagenomic assembly. Required if `genome_coverage` is not provided. | -| `fastq_2` | Path to the second read file in FASTQ format for paired-end data used to generate the source metagenomic assembly. Leave empty for single-end reads. | -| `assembly_software` | Tool name and version that were used to generate the source metagenomic assembly. | -| `binning_software` | Binning tool, including version, that was used to generate the bins. | -| `binning_parameters` | Arguments that were used during binning. | -| `stats_generation_software` | Tool, including version, that was used to calculate completeness and contamination. | -| `completeness` | Genome completeness value. | -| `contamination` | Genome contamination value. | -| `genome_coverage` | Estimated average sequencing depth across the genome. If the value is missing, it is computed automatically during pipeline execution when reads are provided. | -| `metagenome` | Registered metagenome taxonomic identifier or name that matches an existing ENA taxonomy entry. For more details see https://ena-docs.readthedocs.io/en/latest/faq/taxonomy.html | -| `co-assembly` | Whether a co-assembly strategy was used for the initial metagenomic assembly generation. Options: Yes or No. | -| `broad_environment` | Broad ecological context of the sample, for example 'marine biome', 'desert biome'. It is recommended to use subclasses of EnvO 'biome' class (http://purl.obolibrary.org/obo/ENVO_00000428) | -| `local_environment` | Local environmental context of the sample, for example 'tropical dry broadleaf forest biome', 'marine abyssal zone biome'. It is recommended to use EnvO terms which are of smaller spatial grain than your entry for "broad-scale environmental context". | -| `environmental_medium` | Material displaced by the sample, or the material in which the sample was embedded before sampling, for example 'mucus', 'lake water'. It is recommended to use subclasses of EnvO 'environmental material' class (http://purl.obolibrary.org/obo/ENVO_00010483). | -| `RNA_presence` | Presence or absence of the 23S, 16S, and 5S rRNA genes and at least 18 tRNAs. This is used for MISAG/MIMAG assembly quality classification. Options: Yes or No. | -| `NCBI_lineage` | NCBI taxonomy lineage of the genome. Can be composted of either numerical IDs or taxon names separated by ";". | - -> [!NOTE] -> More information about envioronment tags can be found at checklists [ERC000050](https://www.ebi.ac.uk/ena/browser/view/ERC000050) for bins and [ERC000047](https://www.ebi.ac.uk/ena/browser/view/ERC000047) for MAGs under the field names "broad-scale environmental context", "local environmental context", and "environmental medium". - -### `metagenomic_assemblies` mode (`ASSEMBLYSUBMIT`) - -Use this samplesheet structure for metagenomic assembly submission. The input format follows [assets/schema_input_assembly.json](../assets/schema_input_assembly.json). +`nextflow secrets set ENA_WEBIN_PASSWORD "XXX"` -Provide either read files (`fastq_1`, optionally `fastq_2`) or a `coverage` value for each row. If `coverage` is missing and reads are provided, the workflow calculates average coverage automatically. - -Example: - -```csv title="samplesheet_assembly.csv" -sample,fasta,fastq_1,fastq_2,coverage,run_accession,assembler,assembler_version -assembly_001,data/assembly_001.fasta.gz,data/assembly_001_R1.fastq.gz,data/assembly_001_R2.fastq.gz,,ERR011322,SPAdes,3.15.5 -assembly_002,data/assembly_002.fasta.gz,,,42.7,ERR011323,MEGAHIT,1.2.9 -``` - -> [!IMPORTANT] -> **Samplesheet column requirements**: All columns shown in the example above must be present in your samplesheet, even if some values are empty. Columns must be in exactly the same order as shown. - -| Column | Description | -| ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | -| `sample` | A unique identifier for this data entry. Must be globally unique within the input dataset (used as `meta.id` throughout the pipeline). | -| `fasta` | Path to assembly contigs in FASTA format compressed with `gzip`. | -| `fastq_1` | Path to the read file in FASTQ format used to generate the metagenomic assembly. Required if `coverage` is not provided. | -| `fastq_2` | Path to the second read file in FASTQ format for paired-end data used to generate the source metagenomic assembly. Leave empty for single-end reads. | -| `coverage` | Estimated sequencing depth of the assembly. If this value is missing, it is computed automatically during pipeline execution when reads are provided. | -| `run_accession` | ENA run accession for the reads used to generate the metagenomic assembly. Reads must already be submitted to ENA. | -| `assembler` | Name of the assembler software used to generate the assembly. | -| `assembler_version` | Version of the assembler software used to generate the assembly. | - -An example file is available at [assets/samplesheet_assembly.csv](../assets/samplesheet_assembly.csv). - -### `reads` mode (`READSUBMIT`) - -Use this samplesheet structure for raw sequencing reads submission. The input format follows [assets/schema_input_reads.json](../assets/schema_input_reads.json). - -Example: - -```csv title="samplesheet_reads.csv" -sample,sample_accession,fastq_1,fastq_2,platform,instrument,library_source,library_selection,library_strategy,insert_size,library_name,description -illumina_run_001,SAMEA1234567,data/reads_R1.fastq.gz,data/reads_R2.fastq.gz,ILLUMINA,Illumina HiSeq 2000,GENOMIC,RANDOM,WGS,500,HiSeq_library_001,Illumina sequencing of sample XYZ -pacbio_run_001,SAMEA7654321,data/pacbio_reads.fastq.gz,,PACBIO_SMRT,PacBio Sequel,GENOMIC,RANDOM,WGS,,PacBio_library_002,Long-read sequencing -``` - -> [!IMPORTANT] -> **Samplesheet column requirements**: All columns shown in the example above must be present in your samplesheet, even if some values are empty. Columns must be in exactly the same order as shown. - -| Column | Type | Required | Description | -| ------------------- | --------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `sample` | str | Yes | Unique identifier of this particular data entry. Used as an experiment name. | -| `sample_accession` | str | Yes | ENA sample accession (starting with SAMEA) of the sample used to generate raw reads. | -| `fastq_1` | file path | Yes | Path to forward reads in FASTQ format (optionally gzipped). | -| `fastq_2` | file path | No | Path to reverse reads for paired-end data. Leave empty for single-end reads. | -| `platform` | str | Yes | Sequencing platform. Supported values: `ILLUMINA`, `PACBIO_SMRT`, `OXFORD_NANOPORE`, `ION_TORRENT`, `CAPILLARY`, `DNBSEQ`, `ELEMENT`, `GENAPSYS`, `GENEMIND`, `HELICOS`, `LS454`, `BGISEQ`, `ULTIMA`, `VELA_DIAGNOSTICS`. See [ENA documentation](https://ena-docs.readthedocs.io/en/latest/submit/reads/webin-cli.html#metadata-validation) for complete list. | -| `instrument` | str | Yes | Sequencer model, e.g. "Illumina HiSeq 2000", "PacBio Sequel", "MinION". | -| `library_source` | str | Yes | Library source type. Options: `GENOMIC`, `METAGENOMIC`, `TRANSCRIPTOMIC`, `METAGENOMIC SINGLE CELL`, `TRANSCRIPTOMIC SINGLE CELL`, `SYNTHETIC`, `VIRAL RNA`, `OTHER`. | -| `library_selection` | str | Yes | Library selection method. Options: `RANDOM`, `PCR`, `RANDOM PCR`, `RT-PCR`, `MF`, `cDNA`, `cDNA_randomPriming`, `cDNA_oligo_dT`, `PolyA`, `Inverse rRNA`, `ChIP`, `MNase`, `DNase`, `Hybrid Selection`, etc. See [ENA documentation](https://ena-docs.readthedocs.io/en/latest/submit/reads/webin-cli.html#metadata-validation) for complete list. | -| `library_strategy` | str | Yes | Library strategy. Options: `WGS`, `WGA`, `WXS`, `RNA-Seq`, `miRNA-Seq`, `ncRNA-Seq`, `EST`, `Hi-C`, `ATAC-seq`, `WCS`, `RAD-Seq`, `CLONE`, `AMPLICON`, `POOLCLONE`, `etc`. See [ENA documentation](https://ena-docs.readthedocs.io/en/latest/submit/reads/webin-cli.html#metadata-validation) for complete list. | -| `insert_size` | number | No | Fragment/insert size for paired-end reads (e.g., 500 for 500 bp inserts). Leave empty if not applicable. | -| `library_name` | str | No | Descriptive library name (optional). | -| `description` | str | No | Free-text description of the experiment (optional). | +Make sure you update commands above with your authorised credentials. ## Submission study @@ -198,27 +111,149 @@ study-soil-2026 Soil microbiome study Survey of soil microbiota | `existing_study_type` | No | ENA study type (e.g. `Metagenomics`, `Other`). | | `new_study_type` | No | Custom study type. Only used when `existing_study_type` is set to `Other`. | +## Samplesheet input + +You will need to create a samplesheet with information about the data entries you would like to process before running the pipeline. Use `--input` parameter to specify its location. It has to be a comma-separated file with the structure defined by the execution `--mode`. + +```bash +--input '[path to samplesheet.csv]' +``` + +### `mags` and `bins` modes (`GENOMESUBMIT`) + +Use this samplesheet structure for MAG and bin submission. The input format follows [assets/schema_input_genome.json](../assets/schema_input_genome.json). + +Example: + +```csv title="samplesheet_genomes.csv" +sample,fasta,accession,fastq_1,fastq_2,assembly_software,binning_software,binning_parameters,stats_generation_software,completeness,contamination,genome_coverage,metagenome,co-assembly,broad_environment,local_environment,environmental_medium,RNA_presence,NCBI_lineage +mag_001,data/mag_001.fasta.gz,SRR24458089,,,SPAdes 3.15.5,MetaBAT2 2.15,default,CheckM2 1.0.1,92.81,1.09,66.04,sediment metagenome,No,marine,cable bacteria,marine sediment,No,d__Bacteria;p__Proteobacteria;s__ +``` + +> [!IMPORTANT] +> **Samplesheet column requirements**: All columns shown in the example above must be present in your samplesheet, even if some values are empty. Columns must be in exactly the same order as shown. + +| Column | Required | Description | +| --------------------------- | ----------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `sample` | Yes | A unique identifier for this data entry. Must be globally unique within the input dataset (used as `meta.id` throughout the pipeline). | +| `fasta` | Yes | Path to MAG/bin contigs in FASTA format compressed with `gzip`. All names of the FASTA files must be unique to prevent pipeline errors. | +| `accession` | Yes | ENA accession of the run or metagenomic assembly used to generate the MAG/bin. | +| `fastq_1` | Conditional | Path to the read file in FASTQ format used to generate the source metagenomic assembly. Required if `genome_coverage` is not provided. | +| `fastq_2` | No | Path to the second read file in FASTQ format for paired-end data used to generate the source metagenomic assembly. Leave empty for single-end reads. | +| `assembly_software` | Yes | Tool name and version that were used to generate the source metagenomic assembly. | +| `binning_software` | Yes | Binning tool, including version, that was used to generate the bins. | +| `binning_parameters` | Yes | Arguments that were used during binning. | +| `stats_generation_software` | No | Tool, including version, that was used to calculate completeness and contamination. | +| `completeness` | No | Genome completeness value. | +| `contamination` | No | Genome contamination value. | +| `genome_coverage` | Conditional | Estimated average sequencing depth across the genome. If the value is missing, it is computed automatically during pipeline execution when reads are provided. | +| `metagenome` | Yes | Registered metagenome taxonomic identifier or name that matches an existing ENA taxonomy entry. For more details see https://ena-docs.readthedocs.io/en/latest/faq/taxonomy.html | +| `co-assembly` | Yes | Whether a co-assembly strategy was used for the initial metagenomic assembly generation. Options: Yes or No. | +| `broad_environment` | Yes | Broad ecological context of the sample, for example 'marine biome', 'desert biome'. It is recommended to use subclasses of EnvO 'biome' class (http://purl.obolibrary.org/obo/ENVO_00000428) | +| `local_environment` | Yes | Local environmental context of the sample, for example 'tropical dry broadleaf forest biome', 'marine abyssal zone biome'. It is recommended to use EnvO terms which are of smaller spatial grain than your entry for "broad-scale environmental context". | +| `environmental_medium` | Yes | Material displaced by the sample, or the material in which the sample was embedded before sampling, for example 'mucus', 'lake water'. It is recommended to use subclasses of EnvO 'environmental material' class (http://purl.obolibrary.org/obo/ENVO_00010483). | +| `RNA_presence` | No | Presence or absence of the 23S, 16S, and 5S rRNA genes and at least 18 tRNAs. This is used for MISAG/MIMAG assembly quality classification. Options: Yes or No. | +| `NCBI_lineage` | No | NCBI taxonomy lineage of the genome. Can be composted of either numerical IDs or official NCBI taxon names separated by ";". | + +> [!NOTE] +> More information about environment tags can be found at checklists [ERC000050](https://www.ebi.ac.uk/ena/browser/view/ERC000050) for bins and [ERC000047](https://www.ebi.ac.uk/ena/browser/view/ERC000047) for MAGs under the field names "broad-scale environmental context", "local environmental context", and "environmental medium". + +If `genome_coverage`, `stats_generation_software`, `completeness`, `contamination`, `RNA_presence`, or `NCBI_lineage` are missing, the workflow can calculate or infer them when the required inputs are available. See the [Methods documentation](docs/methods.md) for more information on how metadata statistics are obtained. + +### `metagenomic_assemblies` mode (`ASSEMBLYSUBMIT`) + +Use this samplesheet structure for metagenomic assembly submission. The input format follows [assets/schema_input_assembly.json](../assets/schema_input_assembly.json). + +Example: + +```csv title="samplesheet_assembly.csv" +sample,fasta,fastq_1,fastq_2,coverage,run_accession,assembler,assembler_version +assembly_001,data/assembly_001.fasta.gz,data/assembly_001_R1.fastq.gz,data/assembly_001_R2.fastq.gz,,ERR011322,SPAdes,3.15.5 +assembly_002,data/assembly_002.fasta.gz,,,42.7,ERR011323,MEGAHIT,1.2.9 +``` + +> [!IMPORTANT] +> **Samplesheet column requirements**: All columns shown in the example above must be present in your samplesheet, even if some values are empty. Columns must be in exactly the same order as shown. + +| Column | Required | Description | +| ------------------- | ----------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | +| `sample` | Yes | A unique identifier for this data entry. Must be globally unique within the input dataset (used as `meta.id` throughout the pipeline). | +| `fasta` | Yes | Path to assembly contigs in FASTA format compressed with `gzip`. | +| `fastq_1` | Conditional | Path to the read file in FASTQ format used to generate the metagenomic assembly. Required if `coverage` is not provided. | +| `fastq_2` | No | Path to the second read file in FASTQ format for paired-end data used to generate the source metagenomic assembly. Leave empty for single-end reads. | +| `coverage` | Conditional | Estimated sequencing depth of the assembly. If this value is missing, it is computed automatically during pipeline execution when reads are provided. | +| `run_accession` | Yes | ENA run accession for the reads used to generate the metagenomic assembly. Reads must already be submitted to ENA. | +| `assembler` | Yes | Name of the assembler software used to generate the assembly. | +| `assembler_version` | Yes | Version of the assembler software used to generate the assembly. | + +Provide either read files (`fastq_1`, optionally `fastq_2`) or a `coverage` value for each row. If `coverage` is missing and reads are provided, the workflow calculates average coverage with `coverm`. See the [Methods documentation](docs/methods.md) for more information on how coverage is calculated. + +### `reads` mode (`READSUBMIT`) + +Use this samplesheet structure for raw sequencing reads submission. The input format follows [assets/schema_input_reads.json](../assets/schema_input_reads.json). + +Example: + +```csv title="samplesheet_reads.csv" +sample,sample_accession,fastq_1,fastq_2,platform,instrument,library_source,library_selection,library_strategy,insert_size,library_name,description +illumina_run_001,SAMEA1234567,data/reads_R1.fastq.gz,data/reads_R2.fastq.gz,ILLUMINA,Illumina HiSeq 2000,GENOMIC,RANDOM,WGS,500,HiSeq_library_001,Illumina sequencing of sample XYZ +pacbio_run_001,SAMEA7654321,data/pacbio_reads.fastq.gz,,PACBIO_SMRT,PacBio Sequel,GENOMIC,RANDOM,WGS,,PacBio_library_002,Long-read sequencing +``` + +> [!IMPORTANT] +> **Samplesheet column requirements**: All columns shown in the example above must be present in your samplesheet, even if some values are empty. Columns must be in exactly the same order as shown. + +| Column | Required | Description | +| ------------------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `sample` | Yes | Unique identifier of this particular data entry. Used as an experiment name. | +| `sample_accession` | Yes | ENA sample accession (starting with SAMEA) of the sample used to generate raw reads. | +| `fastq_1` | Yes | Path to forward reads in FASTQ format (optionally gzipped). | +| `fastq_2` | No | Path to reverse reads for paired-end data. Leave empty for single-end reads. | +| `platform` | Yes | Sequencing platform. Supported values: `ILLUMINA`, `PACBIO_SMRT`, `OXFORD_NANOPORE`, `ION_TORRENT`, `CAPILLARY`, `DNBSEQ`, `ELEMENT`, `GENAPSYS`, `GENEMIND`, `HELICOS`, `LS454`, `BGISEQ`, `ULTIMA`, `VELA_DIAGNOSTICS`. See [ENA documentation](https://ena-docs.readthedocs.io/en/latest/submit/reads/webin-cli.html#metadata-validation) for complete list. | +| `instrument` | Yes | Sequencer model, e.g. "Illumina HiSeq 2000", "PacBio Sequel", "MinION". See [ENA documentation](https://ena-docs.readthedocs.io/en/latest/submit/reads/webin-cli.html#permitted-values-for-instrument) | +| `library_source` | Yes | Library source type. Options: `GENOMIC`, `METAGENOMIC`, `TRANSCRIPTOMIC`, `METAGENOMIC SINGLE CELL`, `TRANSCRIPTOMIC SINGLE CELL`, `SYNTHETIC`, `VIRAL RNA`, `OTHER`. | +| `library_selection` | Yes | Library selection method. Options: `RANDOM`, `PCR`, `RANDOM PCR`, `RT-PCR`, `MF`, `cDNA`, `cDNA_randomPriming`, `cDNA_oligo_dT`, `PolyA`, `Inverse rRNA`, `ChIP`, `MNase`, `DNase`, `Hybrid Selection`, etc. See [ENA documentation](https://ena-docs.readthedocs.io/en/latest/submit/reads/webin-cli.html#metadata-validation) for complete list. | +| `library_strategy` | Yes | Library strategy. Options: `WGS`, `WGA`, `WXS`, `RNA-Seq`, `miRNA-Seq`, `ncRNA-Seq`, `EST`, `Hi-C`, `ATAC-seq`, `WCS`, `RAD-Seq`, `CLONE`, `AMPLICON`, `POOLCLONE`, `etc`. See [ENA documentation](https://ena-docs.readthedocs.io/en/latest/submit/reads/webin-cli.html#metadata-validation) for complete list. | +| `insert_size` | No | Fragment/insert size for paired-end reads (e.g., 500 for 500 bp inserts). Leave empty if not applicable. | +| `library_name` | No | Descriptive library name (optional). | +| `description` | No | Free-text description of the experiment (optional). | + +## Data privacy + +You can reference private ENA data if you have access to it via your Webin account (it was submitted previously under your credentials). To use private data accessions in your submission metadata, specify the `--is_private` flag. The pipeline will then use your provided Webin credentials to fetch the required metadata. + +You can also control the privacy of your submitted data: + +- If you provide an existing study via `--submission_study`, your submission will inherit the same privacy status as that study. +- If no study is provided, the pipeline will register a new one for you. To keep this new study private, you must specify a release date using `--release_date YYYY-MM-DD` (up to 2 years from the current date). + +| Example | Source Data | Submission Visibility | Required Arguments | Result | +| --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------- | --------------------- | ------------------------------------------------------ | ----------------------------------------------------- | +| Assemblies generated from public reads. Assemblies should be public immediately after submission | public | public | – | SUCCESS | +| Assemblies generated from public reads. Assemblies should remain private for 1 year after submission | public | private | `--release_date YYYY-MM-DD` (+1 year) | SUCCESS | +| Bins generated from private reads and assemblies. Bins should be public after submission. User **has access** to the reads and assemblies via Webin account | private | public | `--is_private` | SUCCESS | +| Bins generated from private reads and assemblies. Bins should remain private for 2 years after submission. User **has access** via Webin account | private | private | `--is_private`, `--release_date YYYY-MM-DD` (+2 years) | SUCCESS | +| MAGs generated from private reads and assemblies. MAGs should be public after submission. User **does not have access** to the reads and assemblies via Webin account | private | public | – | ERROR (submission can only reference accessible data) | + ## Database preparation (`mags` / `bins`) The `GENOMESUBMIT` workflow uses `CheckM2` and `CAT_pack` that require specialized databases for completeness/contamination assessment and taxonomy assignment. You can either provide pre-existing databases or let the pipeline prepare them during execution. +As databases preparation can take significant time, we strongly recommend downloading them locally and storing them in a local cache folder for reuse across runs. +In particular the `CAT_pack` database is extremely large and may take a long time to download and compute. + - `CheckM2`: - - provide the path to local database with `--checkm2_db`, otherwise the pipeline downloads version specified with `--checkm2_db_download_id` (by default `14897628`). + - provide the path to local database with `--checkm2_db`, otherwise the pipeline downloads version specified with `--checkm2_db_download_id` (by default Zenodo accession `14897628`). - `CAT_pack`: - provide the path to local database (containing `tax/` and `db/` folders or `tar.gz` archive) with `--cat_db`, otherwise the pipeline constructs version specified with `--cat_db_download_id` (by default `nr`). See [CAT_pack documentation](https://github.com/MGXlab/CAT_pack) and [CheckM2 documentation](https://github.com/chklovski/CheckM2) for more details on usage and creation of databases. -> [!IMPORTANT] -> `CAT_pack` database creation can take significant time. -> -> Reusing an existing database is strongly recommended for repeated runs. -> -> Databases created/downloaded by the pipeline are published under: -> `${params.outdir}/databases/` +Databases created/downloaded by the pipeline are published under: +`/databases/`. In the subsequent pipeline runs it is strongly recommended to reuse them using `--cat_db /databases/cat_pack/` and `--cat_db /databases/checkm2/`. ## Running the pipeline diff --git a/modules/local/create_genome_metadata_tsv/meta.yml b/modules/local/create_genome_metadata_tsv/meta.yml index 74752ad..870b889 100644 --- a/modules/local/create_genome_metadata_tsv/meta.yml +++ b/modules/local/create_genome_metadata_tsv/meta.yml @@ -35,7 +35,7 @@ input: - broad_environment: Broad environmental context - local_environment: Local environmental context - environmental_medium: Environmental medium description - - RNA_presence: RNA presence status ("Yes" or other) + - RNA_presence: status of tRNA and rRNA genes presence ("Yes" or other) - NCBI_lineage: NCBI taxonomic lineage - fasta: type: file diff --git a/nextflow_schema.json b/nextflow_schema.json index 59c6dc0..b7d7da5 100644 --- a/nextflow_schema.json +++ b/nextflow_schema.json @@ -43,11 +43,11 @@ } }, "rna_detection_options": { - "title": "RNA detection options", + "title": "rRNA and tRNA detection options", "type": "object", "fa_icon": "fas fa-university", - "description": "Parameters used to make a decision about presence/absence of RNA in data", - "help_text": "ENA requires a boolean value (True/False) defining RNA. Decision of presence/absence of RNA is based on number of rRNA and tRNA. If both values are bigger than their corresponding limits - RNA is present (True), otherwise - not (False).", + "description": "Parameters affecting a decision about presence/absence of tRNA and rRNA genes (RNA_presence) in a genome", + "help_text": "The value of RNA_presence is based on the number of rRNA and tRNA. If at least 18 tRNAs are detected and 16S, 23S, and 5S rRNA subunits pass the recovery threshold - RNA_presence is True, otherwise - False. The resulting value is a decisive factor affecting MISAG/MIMAG assembly quality classification of a genome.", "properties": { "rrna_limit": { "type": "number", @@ -57,7 +57,7 @@ }, "trna_limit": { "type": "number", - "description": "Minimum number of tRNA.", + "description": "Minimum number of tRNA genes detected to count tRNA as present. This value is defined by the MISAG/MIMAG standard and must not be modified.", "default": 18, "fa_icon": "fas fa-users-cog" }