Skip to content
Open
Show file tree
Hide file tree
Changes from 9 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 40 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,7 @@ Additional supported columns:
- `RNA_presence`
- `NCBI_lineage`

If `genome_coverage`, `stats_generation_software`, `completeness`, `contamination`, `RNA_presence`, or `NCBI_lineage` are missing, the workflow can calculate or infer them when the required inputs are available.
If `genome_coverage`, `stats_generation_software`, `completeness`, `contamination`, `RNA_presence`, or `NCBI_lineage` are missing, the workflow can calculate or infer them when the required inputs are available. See the [Methods documentation](docs/methods.md) for more information on how metadata statistics are obtained.

Those fields are metadata required for the [genome_uploader](https://github.com/EBI-Metagenomics/genome_uploader) package.

Expand Down Expand Up @@ -111,7 +111,7 @@ At least one of the following must be provided per row:
- reads (`fastq_1`, optional `fastq_2` for paired-end)
- `coverage`

If `coverage` is missing and reads are provided, the workflow calculates average coverage with `coverm`.
If `coverage` is missing and reads are provided, the workflow calculates average coverage with `coverm`. See the [Methods documentation](docs/methods.md) for more information on how coverage is calculated.

Example `samplesheet_assembly.csv`:

Expand Down Expand Up @@ -277,6 +277,44 @@ Key output locations in `--outdir`:

For full details, see the [output documentation](https://nf-co.re/seqsubmit/output).

## Scalability

Some processes in this pipeline are resource intensive — in particular coverage calculation (`coverm`), quality assessment (`CheckM2`), and taxonomic classification of MAGs/bins (`BAT`). For real life data these steps should be run on an HPC cluster or another compute environment with sufficient CPU, memory, and job scheduling capacity, rather than on a single local machine.

Depending on the resources available on your machine or cluster, you will likely need to adapt the Nextflow configuration to limit the maximum number of concurrently running jobs and other resource settings (e.g. `max_cpus`, `max_memory`, `max_time`, or `executor.queueSize` in a custom config), so the pipeline doesn't try to claim more resources than are actually available. See the [nf-core configuration documentation](https://nf-co.re/docs/usage/getting_started/configuration) for guidance on writing a custom config for your infrastructure.

When submitting a large number of records, be aware that ENA recommends limiting submissions to 5000 per day to avoid overfilling their processing queue. If you need to submit more than this, consider splitting your samplesheet and spreading the submission across several days.

## Limitations

nf-core/seqsubmit does not yet support the following data types and scenarios, grouped by the mode they affect:

#### `mags` and `bins` modes (`GENOMESUBMIT`)

- **Co-assemblies**: MAGs/bins generated from co-assemblies are not supported. The pipeline currently expects each MAG/bin to be derived from an assembly generated from a single run.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would point to your PR and say it is in development


- **Eukaryotic and viral bins/MAGs**: Submitting eukaryotic or viral genomes that require additional metadata generation is not supported. For such genomes tRNA and rRNA prediction, quality assessment (completeness/contamination estimation) and taxonomy assignment are not implemented. If all the required metadata (`completeness`, `contamination`, `stats_generation_software`, `RNA_presence` and `NCBI_lineage`) is already provided in the samplesheet, eukaryotic and viral bins/MAGs can still be submitted normally.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we should link issues here? We kind of target to fix those


- **Submission of a MAG/bin without submitting the assembly or raw reads**: If a MAG/bin cannot be traced back to an assembly accession or a reads accession — i.e. only a sample accession is available — it cannot currently be submitted, since the pipeline relies on that lineage to build the required metadata.

- **Single-contig MAGs/bins**: MAGs/bins that consist of only one contig cannot be uploaded through this pipeline. ENA classifies these as "chromosomal assemblies", which follow a different submission procedure with different metadata requirements. If you have a single-contig MAG/bin to submit, please contact [ENA support](https://www.ebi.ac.uk/ena/browser/support) for guidance.

- **Coverage calculation from long reads**: `genome_coverage` calculation via `coverm` has only been tested and validated with short reads (e.g. Illumina). It has not been tested with long reads (PacBio, Nanopore). Exercise caution when processing data generated from long reads, as issues may occur.

#### `metagenomic_assemblies` mode (`ASSEMBLYSUBMIT`)

- **Co-assemblies**: Metagenomic co-assemblies are not supported. The pipeline currently expects each assembly to be derived from a single run.

- **Submission of an assembly without submitting the raw reads**: Assemblies that have no associated reads accession (`run_accession` column in the samplesheet) are not supported, since the pipeline relies on that lineage to build the required metadata.

- **Single-contig assemblies**: Assemblies that consist of only one contig cannot be uploaded through this pipeline. ENA classifies these as "chromosomal assemblies", which follow a different submission procedure with different metadata requirements. If you have a single-contig assembly to submit, please contact [ENA support](https://www.ebi.ac.uk/ena/browser/support) for guidance.

- **Coverage calculation from long reads**: `coverage` calculation via `coverm` has only been tested and validated with short reads (e.g. Illumina). It has not been tested with long reads (PacBio, Nanopore). Exercise caution when processing data generated from long reads, as issues may occur.

#### `reads` mode (`READSUBMIT`)

- **Reads in non-FASTQ formats**: Only reads provided as FASTQ (`.fastq.gz`) can be submitted. Other formats (e.g. BAM) are not supported as input.

## Credits

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we have a quite big readme and need a table of contents in the beggining of file (it was also one of James's suggestions)


nf-core/seqsubmit was originally written by [Martin Beracochea](https://github.com/mberacochea), [Ekaterina Sakharova](https://github.com/KateSakharova), [Sofia Ochkalova](https://github.com/ochkalova), [Evangelos Karatzas](https://github.com/vagkaratzas) and [Tim Rozday](https://github.com/timrozday-mgnify).
Expand Down
114 changes: 114 additions & 0 deletions assets/seqsubmit_schema.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
76 changes: 10 additions & 66 deletions docs/CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -184,44 +184,12 @@ If you update images or graphics, follow the nf-core [style guidelines](https://

To ensure consistency and maintainability of the nf-core/seqsubmit pipeline, contributors should follow these guidelines when implementing new features or modifying existing functionality.

1. Understand Workflow Structure

The pipeline consists of independent workflows, each corresponding to a certain data type:

- GENOMESUBMIT (mags, bins)
- ASSEMBLYSUBMIT (metagenomic_assemblies)
- READSUBMIT (raw reads)

When adding new functionality (e.g. a new submission mode such as `isolate_genomes`), ensure that:

- It integrates cleanly into the appropriate workflow or introduces a well-structured new one.
- Existing workflows remain unaffected unless changes are explicitly required.
- Shared logic is abstracted into reusable modules or subworkflows where possible.


2. Follow Samplesheet and Schema-Driven Design

All inputs in seqsubmit are defined via strict samplesheet schemas.

- Any new feature must:
- Extend or introduce a corresponding schema (assets/schema*input*\*.json)
- Clearly define required and optional fields
- Maintain:
- Column order consistency (critical for pipeline execution)
- Backward compatibility where possible
- Avoid duplicating metadata logic—reuse existing parsing and validation patterns.


3. ENA Submission Requirements
### ENA Submission Requirements

Submissions to [ENA](https://www.ebi.ac.uk/ena/browser/home) are governed by a well-defined data model and submission process. These include relationships between entities (e.g. studies, samples, experiments, runs, analyses), required submission steps, and accepted file types and formats.

Contributors should refer directly to the [official ENA documentation](https://ena-docs.readthedocs.io/en/latest/submit/general-guide.html) to understand latest submission requirements and ensure that pipeline behaviour remains aligned with ENA standards.

Follow ENA-defined submission workflows and required steps for each data type

- Ensure that:
- New pipeline workflows follow ENA-defined submission workflows and required steps for each data type
- Metadata fields align with ENA requirements
Expand All @@ -232,23 +200,23 @@ Contributors should refer directly to the [official ENA documentation](https://e


4. Integration with Webin-CLI
### Integration with Webin-CLI

All submissions are performed via [Webin-CLI](https://github.com/enasequence/webin-cli). Contributors should rely on official Webin-CLI documentation to understand expected inputs, parameters, and modes of operation. Make sure to always use the latest version of the tool.

- Contributions affecting submission logic must:
- Respect --webincli_mode (validate vs submit)
- Support --test_upload for safe testing against ENA test server
- Respect `--webincli_mode` (validate vs submit)
- Support `--test_upload` for safe testing against ENA test server
- Never introduce changes that:
- Risk accidental submission to production without explicit user intent
- Ensure compatibility with:
- Credential handling via Nextflow secrets (ENA_WEBIN, ENA_WEBIN_PASSWORD)


5. Testing Requirements
### Testing Requirements

All new functionality must be covered by tests.
All new functionality must be covered by tests. Running nf-tests that perform ENA submission locally requires ENA webin credentials. Contributors may ask for testing credentials on the nf-core Slack [#seqsubmit](https://nfcore.slack.com/channels/seqsubmit) channel.

At minimum, include:

Expand All @@ -268,36 +236,12 @@ Where applicable:


6. Performance and Resource Awareness

Some data processing steps may be resource-intensive (e.g. database preparation for CAT_pack). When introducing new workflows or features avoid unnecessary recomputation. Add option to reuse cached results or provide prebuilt databases when possible.


7. Documentation and Usability

Every contribution should improve usability:

- Update:
- README.md
- docs/usage.md
- Schema descriptions
- Provide:
- Clear examples (e.g. samplesheets)
- Meaningful parameter descriptions

The goal is to ensure that users can successfully submit data without needing deep knowledge of ENA submission requirements.


8. Design Principles
### Performance and Resource Awareness

When contributing, prioritise:
Some data processing steps may be resource-intensive (e.g. database preparation for CAT_pack).

- Reproducibility (consistent results across environments)
- Automation (minimise manual user intervention)
- Clarity (explicit inputs and outputs)
- Extensibility (easy to add new submission types)
- When introducing new workflows or features avoid unnecessary recomputation.
- Add option to reuse cached results or provide prebuilt databases when possible.


Expand Down
50 changes: 20 additions & 30 deletions docs/methods.md
Original file line number Diff line number Diff line change
@@ -1,38 +1,33 @@
# nf-core/seqsubmit: Methods

## Overview
This page provides a detailed overview of the methods and procedures that are executed throughout the nf-core/seqsubmit pipeline.

`nf-core/seqsubmit` currently contains two workflow implementations:

- `GENOMESUBMIT` for `--mode mags` and `--mode bins`
- `ASSEMBLYSUBMIT` for `--mode metagenomic_assemblies`

This page documents the methods that are currently implemented in the pipeline.

## `GENOMESUBMIT` methods
## `GENOMESUBMIT`

### Overview

The `GENOMESUBMIT` workflow:

1. Reads the samplesheet and associated genome FASTA files.
2. Validates genome FASTA files.
3. Reuses provided or calculates missing values for RNA genes presence, coverage, taxonomy, and genome quality metrics.
3. Reuses provided or calculates missing values for tRNA and rRNA genes presence, coverage, taxonomy, and genome quality metrics.
4. Collects genome metadata into the tabular format required by `genome_uploader`.
5. Generates submission manifests for ENA.
6. Performs submission to ENA.

### Genome FASTA validation

Genome FASTA files are validated with the `FASTAVALIDATOR` module before downstream processing. Each file is checked for FASTA format validity and contig count. A genome must contain at least two contigs to pass validation, which is an ENA requirement for contig-level submissions.
Genome FASTA files are validated with the `fa-lint` before downstream processing. Each file is checked for FASTA format validity and contig count. A genome must contain at least two contigs to pass validation, which is an ENA requirement for contig-level submissions.

Only FASTA files that pass validation are retained for downstream processing and submission.

### RNA presence detection
### tRNA and rRNA genes detection

The workflow only runs internal RNA detection for entries where the `RNA_presence` column is empty. If a value is already supplied in the samplesheet, that value is passed through unchanged.
The workflow only runs tRNA and rRNA genes detection for entries where the `RNA_presence` column is empty. If a value is already supplied in the samplesheet, that value is passed through unchanged.

RNA detection is implemented through the `RNA_DETECTION` subworkflow and combines:
The resulting `RNA_presence` value is a decisive factor affecting MISAG/MIMAG assembly quality classification of a genome.

tRNA and rRNA genes detection is implemented through the `RNA_DETECTION` subworkflow and combines:

- `barrnap` for rRNA prediction
- `tRNAscan-SE` for tRNA prediction
Expand All @@ -42,11 +37,7 @@ RNA detection is implemented through the `RNA_DETECTION` subworkflow and combine

`barrnap` is run in bacterial mode (`"bac"`).

The custom parser then scans the GFF output and keeps only the following ribosomal RNA subunits:

- `16S_rRNA`
- `23S_rRNA`
- `5S_rRNA`
The custom parser then scans the GFF output and keeps only the following rRNA subunits: `16S_rRNA`, `23S_rRNA`, `5S_rRNA`.

For each detected feature, the recovered length is calculated as:

Expand All @@ -66,7 +57,7 @@ $$

If multiple hits are found for the same subunit, the workflow keeps the best recovered percentage for that subunit.

A subunit is considered present when its best recovered percentage is greater than or equal to `--rrna_limit`. The current default is `80`.
A subunit is considered present when its best recovered percentage is greater than or equal to `--rrna_limit`. The current default is `80`. This threshold is arbitrary, as no strict consensus length is defined by ENA or MISAG/MIMAG for considering an rRNA gene present.

#### tRNA detection

Expand All @@ -79,16 +70,16 @@ The parser sums the counts assigned to the 20 standard amino-acid isotypes:
- `Leu`, `Lys`, `Met`, `Phe`, `Pro`
- `Ser`, `Thr`, `Trp`, `Tyr`, `Val`

The total number of predicted tRNAs is compared against `params.trna_limit`. The current default is `18`.
The total number of predicted tRNAs is compared against `params.trna_limit`. The current default is `18`. This value is defined by the MISAG/MIMAG standard and must not be modified.

#### Final RNA presence decision
#### Final `RNA_presence` decision

The final decision stored in `RNA_presence` is:

- `Yes` when at least 18 tRNAs are detected and all three required rRNA subunits pass the recovery threshold
- `No` otherwise

The result is written as a two-column TSV file containing the genome identifier and the final `Yes`/`No` decision, and this value is then merged back into the submission metadata.
The final `Yes`/`No` decision is then merged back into the submission metadata.

### Genome coverage calculation

Expand All @@ -98,7 +89,7 @@ Genome coverage values from `coverm genome` output TSV are parsed and merged int

### Taxonomy assignment

If `NCBI_lineage` is already present in the input samlesheet, the value is retained. If it is missing, the workflow runs taxonomy classification using the `CAT_pack` tool.
If `NCBI_lineage` is already present in the input samplesheet, the value is retained. If it is missing, the workflow runs taxonomy classification using the `CAT_pack` tool.

Before classification, input FASTA files are normalized to a `.fasta` suffix by `RENAME_FASTA_FOR_CATPACK`. Classification is then run in bin mode (`CAT_pack bins`, followed by `CAT_pack add_names`).

Expand All @@ -112,22 +103,21 @@ The workflow checks three samplesheet fields: `completeness`, `contamination`, a

For records that run `CheckM2`, completeness and contamination are extracted from the generated quality report (`quality_report.tsv`) and used `CheckM2` version is recorded as `stats_generation_software`.

## `ASSEMBLYSUBMIT` methods
## `ASSEMBLYSUBMIT`

### Overview

The `ASSEMBLYSUBMIT` workflow:

1. Reads the samplesheet and associated assembly FASTA files.
2. Validates assembly FASTA files.
3. Reuses `coverage` values supplied in the samplesheet when they are already present.
4. Calculates `coverage` internally for entries where this field is missing and reads are available.
5. Builds the metadata table used for manifest generation.
6. Generates assembly manifests and submits the assemblies to ENA.
3. Reuses provided or calculates missing `coverage` values.
4. Builds the metadata table used for manifest generation.
5. Generates assembly manifests and submits the assemblies to ENA.

### Assembly FASTA validation

Assembly FASTA files are validated with `FASTAVALIDATOR` before downstream processing. Each file is checked for FASTA format validity and contig count. A genome must contain at least two contigs to pass validation, which is an ENA requirement for contig-level submissions.
Assembly FASTA files are validated with `fa-lint` before downstream processing. Each file is checked for FASTA format validity and contig count. A genome must contain at least two contigs to pass validation, which is an ENA requirement for contig-level submissions.

Only assemblies with successful validation are forwarded to coverage estimation, metadata/manifest generation and submission.

Expand Down
2 changes: 1 addition & 1 deletion modules/local/create_genome_metadata_tsv/meta.yml
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ input:
- broad_environment: Broad environmental context
- local_environment: Local environmental context
- environmental_medium: Environmental medium description
- RNA_presence: RNA presence status ("Yes" or other)
- RNA_presence: status of tRNA and rRNA genes presence ("Yes" or other)
- NCBI_lineage: NCBI taxonomic lineage
- fasta:
type: file
Expand Down
8 changes: 4 additions & 4 deletions nextflow_schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -43,11 +43,11 @@
}
},
"rna_detection_options": {
"title": "RNA detection options",
"title": "rRNA and tRNA detection options",
"type": "object",
"fa_icon": "fas fa-university",
"description": "Parameters used to make a decision about presence/absence of RNA in data",
"help_text": "ENA requires a boolean value (True/False) defining RNA. Decision of presence/absence of RNA is based on number of rRNA and tRNA. If both values are bigger than their corresponding limits - RNA is present (True), otherwise - not (False).",
"description": "Parameters affecting a decision about presence/absence of tRNA and rRNA genes (RNA_presence) in a genome",
"help_text": "The value of RNA_presence is based on the number of rRNA and tRNA. If at least 18 tRNAs are detected and 16S, 23S, and 5S rRNA subunits pass the recovery threshold - RNA_presence is True, otherwise - False. The resulting value is a decisive factor affecting MISAG/MIMAG assembly quality classification of a genome.",
"properties": {
"rrna_limit": {
"type": "number",
Expand All @@ -57,7 +57,7 @@
},
"trna_limit": {
"type": "number",
"description": "Minimum number of tRNA.",
"description": "Minimum number of tRNA genes detected to count tRNA as present. This value is defined by the MISAG/MIMAG standard and must not be modified.",
"default": 18,
"fa_icon": "fas fa-users-cog"
}
Expand Down
Loading