AIND Ephys Pipeline

aind-ephys-pipeline

Electrophysiology analysis pipeline with SpikeInterface.

News

[Jul 3rd 2026]: New release 1.3.0 is out! Check out release notes here

Overview

The pipeline is based on Nextflow and it includes the following steps:

Pipeline Architecture

flowchart LR
    %% Input/Output
    input[("📥 Input<br/>Ephys Data")]
    output[("📤 Output<br/>NWB + QC + Viz")]

    %% Deployment
    subgraph deploy["🚀 Deployment"]
        direction TB
        co["Code Ocean / SLURM / Local<br/><a href='pipeline/main.nf'>main.nf</a>"]
    end

    %% Infrastructure (link to detailed docs)
    containers["☁️ <b><a href='https://aind-ephys-pipeline.readthedocs.io/en/latest/architecture.html#infrastructure-components'>Containers</a></b><br/>4 images from GHCR"]
    models["🤗 <b><a href='https://aind-ephys-pipeline.readthedocs.io/en/latest/architecture.html#infrastructure-components'>ML Models</a></b><br/>UnitRefine classifiers"]

    %% Pipeline (simplified)
    subgraph pipeline["📊 <b><a href='https://aind-ephys-pipeline.readthedocs.io/en/latest/architecture.html'>Processing Pipeline</a></b> (11 steps)"]
        direction TB
        prep["1. Job Dispatch<br/>2. Preprocessing"]
        sort["3. Spike Sorting<br/>(KS2.5/KS4/SC2)"]
        post["4-6. Postprocessing<br/>Curation<br/>Visualization"]
        final["7-11. Results Collection<br/>Quality Control<br/>NWB Export"]

        prep --> sort --> post --> final
    end

    %% Connections
    input --> pipeline
    deploy -.->|"orchestrates"| pipeline
    containers -.->|"provide runtime"| pipeline
    models -.->|"used in curation"| pipeline
    pipeline --> output

    %% Styling
    classDef infra fill:#fff4e6,stroke:#ff9800,stroke-width:2px
    classDef data fill:#e8f5e9,stroke:#4caf50,stroke-width:3px

    class containers,models,deploy infra
    class input,output data

Loading

📖 View detailed architecture diagram with all infrastructure components, step details, and data flow.

Key Points:

• Single Pipeline: main.nf runs on Code Ocean (AWS Batch), SLURM, and local — sorter is selected via the --sorter parameter or SORTER environment variable
• 11 Processing Steps:
  • 1. Job dispatch
  • 2. Preprocessing
  • 3. Spike sorting (KS2.5/KS4/SC2)
  • 4-6. Postprocessing → Curation → Visualization
  • 7-11. Results collection → Quality control → NWB export
• Infrastructure: 4 container images from GHCR and UnitRefine ML models from Hugging Face
• Parallelization: Steps run in parallel per probe/shank; version controlled via process_versions.env

See the detailed architecture documentation for complete infrastructure details, data flow, and numbered step-by-step breakdown.

Documentation

The documentation is available at ReadTheDocs.

Code Ocean Deployment (AIND)

At AIND, the pipeline is deployed on the Code Ocean platform using a single main.nf pipeline. The sorter and parameters are configured via environment variables (SORTER, PARAMS_FILE) or a JSON parameter file, without requiring separate branches per sorter.