diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index ca0df5ac4..fbf9be1ee 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -1,6 +1,6 @@ # ARCHITECTURE.md -Last updated: 2026-03-11 +Last updated: 2026-09-08 ## Brand source @@ -111,9 +111,10 @@ Last updated: 2026-03-11 - Shared contracts live in `packages/shared-types` so the UI can evolve without importing Python internals. - Shared contracts should ultimately model section, role, cue, confidence, and export artifacts explicitly enough that desktop UI and analysis outputs do not invent their own parallel schemas. - The current shared-types baseline includes a rehearsal-domain fixture that exercises section, role, cue, confidence, provenance, and export-summary fields in the desktop shell before the full analysis pipeline lands. -- Local analysis orchestration uses typed Tauri IPC commands and a Python subprocess over stdin/stdout rather than a loopback HTTP listener. -- Local audio intake bootstraps a project by validating a user-selected file in Rust, creating app-owned temp/cache/project roots, and referencing the original source file rather than copying it in this phase. -- Those bootstrap roots should resolve from app-owned Tauri data/cache paths instead of the shared system temp namespace. +- Local analysis orchestration uses typed Tauri IPC commands and a Python subprocess over stdin/stdout rather than a loopback HTTP listener. Renderer-visible commands are synchronized across the invoke handler, `AppManifest::commands`, generated command permissions, and the window capability; the WebView never receives a PID or generic process handle. +- Local audio intake validates source metadata and encoded size before decode, stages admitted bytes into an app-owned project area, and commits the immutable project source with a path-free size/SHA-256 receipt. Unix publication uses a same-filesystem no-clobber hard link plus project-directory synchronization; Windows uses no-replace `MoveFileExW` with `MOVEFILE_WRITE_THROUGH`. A pre-existing project source is preserved rather than overwritten or deleted. +- Those project and temp/cache roots resolve from app-owned Tauri data/cache paths instead of the shared system temp namespace. +- Analysis cancellation is job-specific. Queued work can terminate as typed `cancelled`; accepted cancellation is serialized against terminal result publication. On Linux/macOS, `bandscope_desktop_core` is the single GUI-independent owner of subprocess process-group setup and termination. The analysis runner delegates pre-spawn containment plus cancellation/timeout/error cleanup to that owner, and the timed YouTube-import helper uses the same owner before spawning the allowlisted importer and joining inherited stdout/stderr readers. The real descendant-pipe regression requires the 50 ms timeout path to return in under one second despite a five-second child retaining those pipes, and the Tauri cancellation contract rejects reintroduction of local POSIX signalling primitives. This covers ordinary descendants that remain in the process group, not descendants that deliberately change group/session. Windows remains direct-child-only until a race-free Job Object boundary is implemented. Full-length real-audio cancellation latency, inherited pipe/handle release, temp-artifact cleanup, and peak-resource return remain acceptance work. - Product and UX decisions should prefer rehearsal-first simplicity while still maintaining high analytical accuracy. - Security decisions should prefer allowlisted narrow capabilities over generic convenience APIs. @@ -126,4 +127,4 @@ Last updated: 2026-03-11 - Security docs and checks are part of the default quickcheck path so design drift is caught early. - Supply-chain docs, workflow pinning, and lockfile verification are part of the default quickcheck path so dependency drift is caught early. - Quickcheck and CI are expected to verify dependency review, audit, supplemental inventory, and SBOM baseline presence as part of bootstrap. -- Cross-platform build workflow presence and trigger coverage are part of the default supply-chain verification path. +- Cross-platform build workflow presence and trigger coverage are part of the default supply-chain verification path so dependency drift is caught early. diff --git a/CHANGELOG.md b/CHANGELOG.md index 34331fb86..8a2be2528 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,49 @@ - Display the analyzed song tempo (BPM) as a badge in the rehearsal workspace. - 각 합주 역할(Role)별 개인 연습 진행도를 0~100% 범위로 기록 및 시각화할 수 있는 연습 진척도(`practiceProgress`) 트래커 기능 추가. UI 컨트롤(슬라이더 및 +/- 버튼)과 한/영 다국어 지원 포함. +### Fixed + +- Enforce one canonical local-audio resource policy across native local-file/YouTube bootstrap intake, the desktop bridge, Python request preflight, temporal decoding, and stem separation so oversized, overlong, malformed, wrong-rate, or non-finite input fails before bootstrap storage or expensive analysis/model work. +- Re-admit persisted stem-feature caches before MIR reuse: accept only a unique non-empty subset of the canonical vocals/bass/drums/other identities, require a second-read sidecar to remain on feature-cache schema version 1 while preserving the already-admitted `stemKeys` identity, sample rate, and exact role-key set, require persisted role metadata (when present) to preserve `vocals -> vocal` and `bass|drums|other -> instrument`, require every admitted stem to declare the same synchronized sample count, preflight the exact deflated NPY members, bounded headers, declared one-dimensional floating sample counts and bytes before NumPy materialization, then return only owned finite canonical float32 stems that pass the live `AudioResourcePolicy`; schema-generation substitution, invented or substituted stem identities, contradictory role metadata, desynchronized timelines, malformed, oversized, representation-drifted, or corrupted caches become misses rather than rehearsal evidence, while legacy metadata without `stemRoleTypes` reconstructs the canonical mapping. +- Bound both persisted feature-cache metadata reads before UTF-8/JSON materialization: require the already-open sidecar to be a non-empty regular file no larger than 1 MiB, reject growth or truncation relative to that descriptor snapshot, and treat special-file, malformed encoding/JSON, duplicate object member names at any depth, filesystem, allocator, or JSON integer-conversion-limit failures as cache misses; this bounds parser input and removes ambiguous first/last-wins cache authority but does not yet bind metadata, NPZ bytes, and admitted source identity into one immutable generation. +- Open persisted feature-cache NPZ archives with non-blocking/no-follow flags where the platform exposes them, then admit size and regular-file type from that same descriptor before snapshot copy; a substituted trailing symlink or blocking special-file path becomes a cache miss instead of redirecting or stalling rehearsal replay. +- Reject noncanonical persisted stem identities before canonical role-table lookup, so an explicit unknown mapping such as `guitar -> instrument` becomes a cache miss instead of escaping Resource Admission as an uncaught `KeyError`; this does not expand the current vocals/bass/drums/other separator contract. +- Require persisted `separation.duration_seconds` in both the first API metadata snapshot and the second-read archive sidecar before stem-cache replay, reject Boolean, non-numeric, non-finite, non-positive, or finite-float-overflowing duration metadata before a later sidecar replacement can repair missing first-read timing authority, and require the admitted duration to agree with the archive's common `sample_count / sample_rate` within half one sample before NumPy materialization; the two metadata reads, NPZ snapshot, and admitted source identity still require one versioned generation/digest binding. +- Copy the initially admitted persisted stem archive extent into one bounded private spooled snapshot before declaration preflight and NumPy materialization, so pathname changes or same-inode equal-size rewrites with restored timestamps cannot substitute unchecked samples between admission and MIR replay; metadata/archive/source digest generation binding remains separate work. +- Treat `MemoryError` or truncated EOF encountered while copying, preflighting, or opening an otherwise admitted persisted stem archive as a cache miss, so optional cache replay cannot turn allocator exhaustion or late archive truncation into an uncaught analysis-job failure; this does not claim a process-wide RSS ceiling or recovery from an operating-system OOM kill. +- Reject multidimensional source-separation model output instead of flattening it into mono rehearsal evidence; valid current Demucs output is already explicitly downmixed before NumPy conversion, while an upstream model/backend shape change now fails closed at the model-output boundary. +- Commit an admitted local source through a platform-specific no-clobber durability barrier before returning path-free project authority: Unix synchronizes the project directory after hard-link publication/stage removal, while Windows uses no-replace `MoveFileExW` with `MOVEFILE_WRITE_THROUGH`. +- Preflight source-container duration, sample rate, and channel count from the already-open audio handle before temporal, stem, or bass-transcription decoders resample, downmix, or truncate it; successful metadata probes rewind the handle and malformed probes fail closed. +- Advance the local-audio resource policy to v2 and bind the admitted canonical decoded mono buffer to the production float32 representation: 158,760,000 bytes for the existing 39,690,000-sample / 15-minute ceiling, preventing a wider floating buffer from silently consuming twice the intended canonical artifact memory while keeping the same sample count. +- Advance the local-audio resource policy to v3 and require the PCM artifact admitted to MIR to be native NumPy `float32`; noncanonical artifacts passed directly to policy validation fail closed with `decoded_dtype_unsupported`, while decoder-returned floating arrays are normalized only after their sample count and already-allocated bytes fit the shared policy. +- Pin the canonical local-audio decoder to explicit NumPy `float32` output and `soxr_hq` band-limited resampling instead of inheriting librosa defaults, so dependency-default changes cannot silently change the selected decode dtype or resampler; numerical output still requires versioned dependency and real-audio reproducibility evidence. +- Reject an oversized or over-budget decoder-returned array before float32 normalization can allocate a second canonical PCM buffer; decoder/resampler allocations made internally before `librosa.load` returns remain outside the artifact ceiling and require separate peak-RSS measurement. +- Detach an admitted non-owning decoder view into an owned canonical float32 PCM buffer before MIR handoff, so a returned artifact cannot retain a larger hidden backing allocation; decoder-internal transient memory remains part of the separate peak-RSS acceptance boundary. +- Map host allocator exhaustion during an otherwise policy-admitted canonical float32 copy to the stable payload-free `memory_budget_exceeded` rejection instead of surfacing a raw `MemoryError`; this preserves failure semantics without claiming that end-to-end peak RSS is bounded. +- Bound canonical PCM finiteness validation to 1 MiB temporary boolean-mask chunks instead of allocating a full-song NumPy mask, while preserving NaN and positive/negative infinity rejection. +- Keep native analysis cancellation job-scoped without exposing PIDs or generic process authority to the renderer, serialize accepted cancellation against terminal publication, and on Linux/macOS establish the Python engine as leader of a dedicated process group before `exec` so cancellation, timeout, and runner-error cleanup can signal the inherited group before reaping the direct child. Windows remains direct-child-only until the race-free Job Object boundary is implemented; real-audio cancellation latency, handle/temp cleanup, and peak-resource acceptance remain separate work. +- Bind analysis JSONL status to the BandScope-minted native job identity, stream only valid `Running` progress before native child exit, retain `Succeeded`/`Failed` for post-exit finalization, and fail closed on mismatched identities, helper-authored `Queued` state, or any status emitted after a terminal record; this prevents helper protocol output from re-keying concurrent jobs or publishing terminal rehearsal truth while its process is still running. +- Bind analysis `requestedAt` to the timestamp minted by the native job owner: pass it to the Python helper, preserve it unchanged in helper progress/terminal envelopes, and reject any helper status whose request timestamp differs before it can mutate native job truth. Standalone/manual CLI callers that omit the field retain the existing current-time fallback. +- Remove the CLI's temporary pre-orchestration `TemporalAnalyzer` pass for local audio, so one native analysis request delegates MIR work exactly once to `run_analysis_job` / `run_analysis_job_updates` instead of decoding and analyzing the same source a second time before the canonical orchestration path. +- Fail closed when analysis `--progress-jsonl` stdout cannot deserialize as the strict typed job-status schema, waking the existing single process-control owner instead of silently ignoring malformed/schema-invalid protocol lines and accepting a later terminal record. +- Validate typed analysis status semantics before native mutation: reject progress outside 0..=100, result/error payloads on non-terminal status, succeeded status without exactly one result payload, and failed status without exactly one error payload; invalid typed status wakes the existing process-control owner instead of reaching rehearsal job truth. +- Preserve fail-closed analysis protocol rejection in the stdout reader's join result as well as the MPSC wake-up path, so a fast successful child exit cannot outrun the failure notification and admit a retained terminal candidate after malformed, invalid, or post-terminal output. +- Preserve empty physical records in bounded analysis JSONL transport so a blank line reaches the strict JSON parser and fails closed instead of being silently erased below protocol validation. +- Put the timed YouTube import helper in a dedicated Linux/macOS process group before spawning `bandscope_analysis.youtube`; timeout and wait-error cleanup now terminate ordinary yt-dlp/FFmpeg descendants that retain the group before stdout/stderr readers are joined. Windows descendants and processes that deliberately leave the Unix group remain outside this claim. +- Terminate residual Linux/macOS same-group descendants after the directly owned import process reports terminal status and before stdout/stderr reader joins, so a successful parent cannot hang indefinitely behind an inherited pipe held by an outliving helper process. +- Apply that same terminal-status cleanup to the native analysis runner before it joins stdout/stderr readers, so a successfully exited Python analysis parent cannot leave an ordinary same-group descendant holding inherited pipes and stall terminal job publication. +- Bound captured stdout and stderr from BandScope-owned helper processes to 1 MiB per stream plus one overflow probe, failing closed before YouTube metadata parsing instead of allowing a buggy or hostile helper to grow parent-side output buffers without limit; this is a capture-memory bound, not a whole-process RSS/VRAM claim. +- Terminate the owned helper boundary as soon as the process-control loop observes stdout/stderr admission failure, so a helper that has already exceeded the 1 MiB stream ceiling cannot simply consume the remainder of the product timeout after its output reader has failed closed. +- Wake the single process-control owner immediately when stdout/stderr bounded readers report overflow or read failure, instead of delaying fail-closed termination until the next ordinary child-status poll; timeout waiting remains clamped to the monotonic deadline. +- Removed the compatibility module's stale unbounded helper-output implementation, leaving the bounded `process_output` module as the single execution owner while preserving the public crate-root API. +- Clamp helper-process polling sleeps to the monotonic time remaining before the configured deadline, so a coarse caller poll interval cannot extend a timed analysis/import helper by another full interval; this removes avoidable timeout overshoot without claiming hard real-time scheduling. +- Consolidate analysis-runner and timed-import Unix process containment into the GUI-independent desktop-core owner, so Tauri orchestration and YouTube import share the same pre-spawn process-group and group-termination semantics instead of carrying two security-sensitive POSIX implementations. +- Fail closed on malformed known YouTube duration metadata before `download=True`; Boolean, non-numeric, non-finite, zero, negative, and non-canonical numeric-subtype duration evidence can no longer authorize a media download through Python numeric coercion or subclass semantics. +- Align YouTube download admission with that same 100 MiB encoded-byte ceiling: abort in-flight with yt-dlp `max_filesize` and a progress hook, reject announced oversize before `download=True`, delete owned `.part` / `.ytdl` / ASCII-indexed `-Frag[0-9]+` siblings from that import directory on abort, reject a completed path that resolves outside the current import cache before post-download validation, cleanup, or success, and delete owned post-download artifacts that still exceed the policy. A 60 MiB import that the old 50 MB check rejected is now accepted; a file one byte over 100 MiB is not. +- Preserve legal `-Frag` text inside an 11-character YouTube video ID during abort cleanup: only a terminal ASCII-decimal yt-dlp fragment suffix (`-Frag[0-9]+` or `-Frag[0-9]+.part`) is normalized, so IDs such as `abc-Frag123` retain their identity while their real `.part-FragN` siblings are still removed and Unicode digit lookalikes do not acquire deletion authority. +- Bound native stored-score PDF reads to the 25 MiB product limit before heap allocation and revalidate PDF magic on the same opened descriptor, preventing an attached score that later grows from bypassing the local resource boundary. +- Treat every zero-element NumPy layout as empty chord input, including shapes whose first dimension is non-zero, before feature extraction. + ### Changed - Consolidated Bandit, dependency audits, supplemental secret checks, and Trivy into one trusted-branch security backstop, delegated CodeQL to GitHub default setup, and removed duplicate local PR security and release-preflight runs. @@ -23,56 +66,3 @@ - Published release assets through a tag-driven draft release flow so immutable GitHub Releases include desktop installers, checksums, SBOM, and supplemental inventory before publication. - Added a supply-chain regression guard that rejects post-publication release asset uploads. - -## [0.1.2] - 2026-04-29 - -### Changed - -- Aligned the packaged desktop app version with the release package metadata. - -### Fixed - -- Stabilized YouTube import fallback behavior in browser and desktop dev paths. -- Guarded OSSF Scorecard execution so release-branch pushes skip unsupported non-default branch runs cleanly. - -## [0.1.1] - 2026-04-28 - -### Added - -- Implemented rehearsal workspace design (Issue #107) -- Add capo and tuning detection heuristics (Issue #103) -- Add bandit security scan workflow - -### Fixed - -- Upgrade pytest to 9.0.3 to fix GHSA-6w46-j5rx-g56g -- Resolve npm audit vulnerabilities -- Fix ruff import sorting and formatting errors -- Add missing docstrings to tests -- Fix test configuration and typing issues - -## [0.1.0] - 2026-03-27 - -### Added - -- Issue #29: Defined core `song -> section -> role` rehearsal domain contracts -- Issue #38: Added cross-architecture build support (Windows/macOS arm64+amd64) -- Issue #40: Enforced 100% Python docstring and test coverage -- Issue #32: Implemented local analysis orchestration and secure IPC boundaries -- Issue #33: Implemented secure local audio intake and project bootstrap -- Issue #35: Engineered section, form, and cue anchor extraction pipeline -- Issue #34: Implemented role extraction targets and part graph -- Issue #31: Added role-specific harmony, range, overlap, and confidence metrics -- Issue #28: Delivered practical rehearsal workspace UI -- Issue #27: Supported manual overrides, provenance tracking, and local project persistence -- Issue #36: Implemented rehearsal priority calculation and cue-sheet (CSV) / chart (JSON) exports -- Issue #30: Added policy-constrained YouTube import with local fallback -- Issue #26: Finalized roadmap and prepared application for initial release - -## [0.1.4] - 2026-05-15 - -### 추가됨 (Added) - -- `ChordsFeature` (코드 분석) 화면에서 각 파트(Role)의 `transpositionPlan`(이조/조옮김 계획)을 표시하는 기능을 추가했습니다. -- `RangesFeature` (음역대 분석) 화면에서 겹침 경고(Overlap warning) 외에 해당 파트의 채보(Transcription) 가능 노드 수를 요약하여 보여주는 기능을 추가했습니다. -- 신규 UI 요소에 대한 단위 테스트를 추가했습니다 (`apps/desktop/src/features/chords/index.test.tsx`, `apps/desktop/src/features/ranges/index.test.tsx`). diff --git a/CLAUDE.md b/CLAUDE.md index b5a34c1fa..c549df37a 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -52,7 +52,7 @@ BandScope is a local-first desktop app for rehearsal prep: it turns a song into Three layers, decoupled through shared contracts: - `apps/desktop` — Tauri 2 + Vite + React 19 shell (Tailwind 4, Base UI, Storybook). Feature screens live in `src/features/` (home, workspace, chords, ranges, player, settings). The ready workspace names tonight's first playable range and the next instrument check. `src/lib/analysis.ts` and `src/lib/job_runner.ts` call typed Tauri IPC commands, with a browser fallback that serves demo data when not running inside Tauri. -- `apps/desktop/src-tauri/src/main.rs` — the Rust orchestration boundary. Tauri commands (`start_analysis_job`, `get_analysis_job_status`, `select_local_audio_source`, `import_youtube_url`) validate untrusted input (project IDs, file paths, URLs) and spawn the Python engine as a subprocess. There is no loopback HTTP listener and no network path for local analysis. +- `apps/desktop/src-tauri/src/main.rs` — the Rust orchestration boundary. Tauri commands cover analysis start/status/cancellation, local-audio selection and YouTube import, project save/load, and bounded score-PDF attach/read/remove operations. Each renderer-visible application command must remain synchronized across the invoke handler, `AppManifest::commands`, generated permission, and window capability. Commands validate untrusted input and spawn the allowlisted Python engine as a subprocess; no PID or generic process handle is exposed to the WebView. There is no loopback HTTP listener and no network path for local analysis. - `services/analysis-engine` — Python package `bandscope_analysis` (librosa/numpy). Entry point `cli.py` reads a JSON job request on stdin and prints a structured job-status JSON envelope on stdout (`--progress-jsonl` streams progress lines). `api.py` orchestrates the pipeline across the `separation`, `sections`, `roles`, `chords`, `ranges`, `temporal`, `transcription`, and `youtube` modules. Data flow: React UI → Tauri IPC command → Rust validation + Python subprocess over stdin/stdout → job status and progress events emitted back to the UI. diff --git a/apps/desktop/core/Cargo.toml b/apps/desktop/core/Cargo.toml index b01a537dc..44f482e73 100644 --- a/apps/desktop/core/Cargo.toml +++ b/apps/desktop/core/Cargo.toml @@ -7,7 +7,7 @@ publish = false [lib] name = "bandscope_desktop_core" -path = "src/lib.rs" +path = "src/root.rs" [lints.rust] unexpected_cfgs = { level = "warn", check-cfg = ['cfg(coverage)'] } diff --git a/apps/desktop/core/src/audio_resource.rs b/apps/desktop/core/src/audio_resource.rs new file mode 100644 index 000000000..a8383067b --- /dev/null +++ b/apps/desktop/core/src/audio_resource.rs @@ -0,0 +1,325 @@ +use crate::content_sha256::StreamingSha256; +use std::io::{ErrorKind, Read, Write}; + +/// Maximum encoded local-audio file size accepted by the desktop bootstrap boundary. +pub const MAX_LOCAL_AUDIO_FILE_BYTES: u64 = 100 * 1024 * 1024; + +const LOCAL_AUDIO_READ_ERROR: &str = "Could not read the selected audio file."; +const LOCAL_AUDIO_WRITE_ERROR: &str = "Could not prepare the local project workspace."; +const LOCAL_AUDIO_TOO_LARGE_ERROR: &str = + "Choose a shorter or smaller song file to start analysis."; + +/// Immutable identity evidence for one successfully staged local-audio byte stream. +#[derive(Clone, Debug, Eq, PartialEq)] +pub struct LocalAudioCopyReceipt { + /// Exact number of bytes written successfully to the staging writer. + pub file_size_bytes: u64, + /// SHA-256 of exactly the bytes written successfully, encoded as lowercase hexadecimal. + pub content_sha256: String, +} + +/// Validate a native local-audio file length before storing bootstrap metadata. +/// +/// The caller must obtain this length from the native filesystem descriptor or +/// metadata boundary rather than from renderer-controlled JSON. The function +/// intentionally returns only bounded product messages and never includes a +/// local path or payload content. +pub fn validate_local_audio_file_size(file_size_bytes: u64) -> Result { + if file_size_bytes == 0 { + return Err(LOCAL_AUDIO_READ_ERROR.to_string()); + } + if file_size_bytes > MAX_LOCAL_AUDIO_FILE_BYTES { + return Err(LOCAL_AUDIO_TOO_LARGE_ERROR.to_string()); + } + Ok(file_size_bytes) +} + +fn read_retrying_interrupted(reader: &mut impl Read, buffer: &mut [u8]) -> Result { + loop { + match reader.read(buffer) { + Ok(read) => return Ok(read), + Err(error) if error.kind() == ErrorKind::Interrupted => continue, + Err(_) => return Err(LOCAL_AUDIO_READ_ERROR.to_string()), + } + } +} + +fn copy_bounded_local_audio_with_limit( + mut reader: R, + writer: &mut W, + max_bytes: u64, +) -> Result { + let mut copied = 0_u64; + let mut buffer = [0_u8; 64 * 1024]; + let mut content_digest = StreamingSha256::default(); + + loop { + if copied == max_bytes { + let mut overflow_probe = [0_u8; 1]; + let read = read_retrying_interrupted(&mut reader, &mut overflow_probe)?; + if read == 0 { + break; + } + return Err(LOCAL_AUDIO_TOO_LARGE_ERROR.to_string()); + } + + let remaining = (max_bytes - copied).min(buffer.len() as u64) as usize; + let read = read_retrying_interrupted(&mut reader, &mut buffer[..remaining])?; + if read == 0 { + break; + } + writer + .write_all(&buffer[..read]) + .map_err(|_| LOCAL_AUDIO_WRITE_ERROR.to_string())?; + content_digest + .update(&buffer[..read]) + .map_err(|_| LOCAL_AUDIO_READ_ERROR.to_string())?; + copied += read as u64; + } + + if copied == 0 { + return Err(LOCAL_AUDIO_READ_ERROR.to_string()); + } + let content_sha256 = content_digest + .finalize_hex() + .map_err(|_| LOCAL_AUDIO_READ_ERROR.to_string())?; + Ok(LocalAudioCopyReceipt { + file_size_bytes: copied, + content_sha256, + }) +} + +/// Copy one admitted local-audio stream into a staging writer and return native content identity. +/// +/// Security Notes: callers must pass an already-open, OS-authorized source +/// descriptor and a private app-owned staging writer. The helper writes no more +/// than the 100 MiB ceiling, hashes exactly the bytes whose writes succeeded, +/// and, after reaching the ceiling exactly, reads only one probe byte to detect +/// source growth. Source-read and destination-write failures use distinct +/// bounded product errors so storage failures are not misdiagnosed as bad media. +/// The caller must discard the staging artifact on error, synchronize it before +/// publication, and bind the returned receipt only to the artifact that was +/// actually published. +pub fn copy_bounded_local_audio_with_receipt( + reader: R, + writer: &mut W, +) -> Result { + copy_bounded_local_audio_with_limit(reader, writer, MAX_LOCAL_AUDIO_FILE_BYTES) +} + +/// Re-read a published app-owned source and prove that it matches its staging receipt. +/// +/// Security Notes: the caller must pass an already-open descriptor for the +/// synchronized, published `source.` object. This helper opens no +/// path and grants no filesystem authority. The staging receipt is native +/// evidence from the prior bounded copy, so its byte length becomes the tighter +/// publication-read ceiling: the verifier hashes at most that many bytes and +/// reads one additional probe byte to reject growth. It then requires both size +/// and digest to equal the staging receipt. Any invalid expected length, read, +/// growth, truncation, or content mismatch is reported as a bounded +/// project-workspace failure because the selected source already passed +/// admission before publication. +pub fn verify_local_audio_publication_receipt( + reader: R, + expected: &LocalAudioCopyReceipt, +) -> Result { + if expected.file_size_bytes == 0 || expected.file_size_bytes > MAX_LOCAL_AUDIO_FILE_BYTES { + return Err(LOCAL_AUDIO_WRITE_ERROR.to_string()); + } + + let mut sink = std::io::sink(); + let actual = copy_bounded_local_audio_with_limit(reader, &mut sink, expected.file_size_bytes) + .map_err(|_| LOCAL_AUDIO_WRITE_ERROR.to_string())?; + if actual != *expected { + return Err(LOCAL_AUDIO_WRITE_ERROR.to_string()); + } + Ok(actual) +} + +/// Copy one admitted local-audio stream into a staging writer and return its byte count. +/// +/// This compatibility adapter preserves the existing desktop call boundary while +/// callers migrate to `copy_bounded_local_audio_with_receipt`. It uses the same +/// bounded copy and content-hash path and discards only the returned digest. +pub fn copy_bounded_local_audio(reader: R, writer: &mut W) -> Result { + copy_bounded_local_audio_with_receipt(reader, writer).map(|receipt| receipt.file_size_bytes) +} + +#[cfg(test)] +mod tests { + use super::*; + use std::io::{Cursor, Error}; + + struct FailingWriter; + + impl Write for FailingWriter { + fn write(&mut self, _buffer: &[u8]) -> std::io::Result { + Err(Error::new(ErrorKind::Other, "simulated destination failure")) + } + + fn flush(&mut self) -> std::io::Result<()> { + Ok(()) + } + } + + struct FailingReader; + + impl Read for FailingReader { + fn read(&mut self, _buffer: &mut [u8]) -> std::io::Result { + Err(Error::new(ErrorKind::Other, "simulated source failure")) + } + } + + struct InterruptedThenReader { + bytes: Cursor>, + interrupted: bool, + } + + impl Read for InterruptedThenReader { + fn read(&mut self, buffer: &mut [u8]) -> std::io::Result { + if !self.interrupted { + self.interrupted = true; + return Err(Error::from(ErrorKind::Interrupted)); + } + self.bytes.read(buffer) + } + } + + struct CountingReader { + bytes: Cursor>, + bytes_read: usize, + } + + impl Read for CountingReader { + fn read(&mut self, buffer: &mut [u8]) -> std::io::Result { + let read = self.bytes.read(buffer)?; + self.bytes_read += read; + Ok(read) + } + } + + #[test] + fn bounded_copy_rejects_stream_growth_without_staging_bytes_past_the_limit() { + let input = Cursor::new(vec![1_u8, 2, 3, 4, 5]); + let mut staged = Vec::new(); + + let error = copy_bounded_local_audio_with_limit(input, &mut staged, 4) + .expect_err("a source that grows beyond the admitted byte limit must fail closed"); + + assert_eq!(error, LOCAL_AUDIO_TOO_LARGE_ERROR); + assert_eq!(staged, vec![1_u8, 2, 3, 4]); + } + + #[test] + fn bounded_copy_accepts_the_exact_limit_and_reports_content_identity() { + let input = Cursor::new(vec![1_u8, 2, 3, 4]); + let mut staged = Vec::new(); + + let receipt = copy_bounded_local_audio_with_limit(input, &mut staged, 4) + .expect("the exact encoded-byte limit remains admissible"); + + assert_eq!(receipt.file_size_bytes, 4); + assert_eq!( + receipt.content_sha256, + "9f64a747e1b97f131fabb6b447296c9b6f0201e79fb3c5356e6c77e89b6a806a" + ); + assert_eq!(staged, vec![1_u8, 2, 3, 4]); + } + + #[test] + fn bounded_copy_reports_destination_failure_as_workspace_failure() { + let input = Cursor::new(vec![1_u8, 2, 3, 4]); + let mut staged = FailingWriter; + + let error = copy_bounded_local_audio_with_limit(input, &mut staged, 4) + .expect_err("a staging write failure must not be reported as a source read failure"); + + assert_eq!(error, LOCAL_AUDIO_WRITE_ERROR); + } + + #[test] + fn bounded_copy_keeps_source_failure_distinct_from_workspace_failure() { + let input = FailingReader; + let mut staged = Vec::new(); + + let error = copy_bounded_local_audio_with_limit(input, &mut staged, 4) + .expect_err("a source read failure must retain the media-read diagnosis"); + + assert_eq!(error, LOCAL_AUDIO_READ_ERROR); + assert!(staged.is_empty()); + } + + #[test] + fn bounded_copy_retries_interrupted_source_reads_without_changing_identity() { + let input = InterruptedThenReader { + bytes: Cursor::new(vec![1_u8, 2, 3, 4]), + interrupted: false, + }; + let mut staged = Vec::new(); + + let receipt = copy_bounded_local_audio_with_limit(input, &mut staged, 4) + .expect("an interrupted source read should be retried"); + + assert_eq!(receipt.file_size_bytes, 4); + assert_eq!( + receipt.content_sha256, + "9f64a747e1b97f131fabb6b447296c9b6f0201e79fb3c5356e6c77e89b6a806a" + ); + assert_eq!(staged, vec![1_u8, 2, 3, 4]); + } + + #[test] + fn publication_verification_maps_read_failure_to_workspace_failure() { + let expected = LocalAudioCopyReceipt { + file_size_bytes: 4, + content_sha256: + "9f64a747e1b97f131fabb6b447296c9b6f0201e79fb3c5356e6c77e89b6a806a" + .to_string(), + }; + + let error = verify_local_audio_publication_receipt(FailingReader, &expected) + .expect_err("published artifact read failure must be a workspace failure"); + + assert_eq!(error, LOCAL_AUDIO_WRITE_ERROR); + } + + #[test] + fn publication_verification_stops_after_expected_size_plus_one_probe_byte() { + let expected = LocalAudioCopyReceipt { + file_size_bytes: 4, + content_sha256: + "9f64a747e1b97f131fabb6b447296c9b6f0201e79fb3c5356e6c77e89b6a806a" + .to_string(), + }; + let mut published = CountingReader { + bytes: Cursor::new(vec![1_u8, 2, 3, 4, 5, 6, 7, 8]), + bytes_read: 0, + }; + + let error = verify_local_audio_publication_receipt(&mut published, &expected) + .expect_err("a grown published artifact must fail without scanning unrelated tail bytes"); + + assert_eq!(error, LOCAL_AUDIO_WRITE_ERROR); + assert_eq!(published.bytes_read, 5); + } + + #[test] + fn publication_verification_rejects_impossible_expected_lengths_without_reading() { + for file_size_bytes in [0, MAX_LOCAL_AUDIO_FILE_BYTES + 1] { + let expected = LocalAudioCopyReceipt { + file_size_bytes, + content_sha256: "00".repeat(32), + }; + let mut published = CountingReader { + bytes: Cursor::new(vec![1_u8, 2, 3, 4]), + bytes_read: 0, + }; + + let error = verify_local_audio_publication_receipt(&mut published, &expected) + .expect_err("an impossible native receipt length must fail before reading"); + + assert_eq!(error, LOCAL_AUDIO_WRITE_ERROR); + assert_eq!(published.bytes_read, 0); + } + } +} diff --git a/apps/desktop/core/src/content_sha256.rs b/apps/desktop/core/src/content_sha256.rs new file mode 100644 index 000000000..dbb109a49 --- /dev/null +++ b/apps/desktop/core/src/content_sha256.rs @@ -0,0 +1,293 @@ +//! Streaming SHA-256 for local content-identity receipts. +//! +//! The operations and constants follow NIST FIPS 180-4 SHA-256. The known-answer +//! tests below are correctness checks, not CAVP validation or a FIPS 140 claim. + +use std::io::{self, ErrorKind, Read}; + +const BLOCK_BYTES: usize = 64; +const DIGEST_BYTES: usize = 32; +const INITIAL_STATE: [u32; 8] = [ + 0x6a09_e667, + 0xbb67_ae85, + 0x3c6e_f372, + 0xa54f_f53a, + 0x510e_527f, + 0x9b05_688c, + 0x1f83_d9ab, + 0x5be0_cd19, +]; +const ROUND_CONSTANTS: [u32; 64] = [ + 0x428a_2f98, 0x7137_4491, 0xb5c0_fbcf, 0xe9b5_dba5, 0x3956_c25b, 0x59f1_11f1, + 0x923f_82a4, 0xab1c_5ed5, 0xd807_aa98, 0x1283_5b01, 0x2431_85be, 0x550c_7dc3, + 0x72be_5d74, 0x80de_b1fe, 0x9bdc_06a7, 0xc19b_f174, 0xe49b_69c1, 0xefbe_4786, + 0x0fc1_9dc6, 0x240c_a1cc, 0x2de9_2c6f, 0x4a74_84aa, 0x5cb0_a9dc, 0x76f9_88da, + 0x983e_5152, 0xa831_c66d, 0xb003_27c8, 0xbf59_7fc7, 0xc6e0_0bf3, 0xd5a7_9147, + 0x06ca_6351, 0x1429_2967, 0x27b7_0a85, 0x2e1b_2138, 0x4d2c_6dfc, 0x5338_0d13, + 0x650a_7354, 0x766a_0abb, 0x81c2_c92e, 0x9272_2c85, 0xa2bf_e8a1, 0xa81a_664b, + 0xc24b_8b70, 0xc76c_51a3, 0xd192_e819, 0xd699_0624, 0xf40e_3585, 0x106a_a070, + 0x19a4_c116, 0x1e37_6c08, 0x2748_774c, 0x34b0_bcb5, 0x391c_0cb3, 0x4ed8_aa4a, + 0x5b9c_ca4f, 0x682e_6ff3, 0x748f_82ee, 0x78a5_636f, 0x84c8_7814, 0x8cc7_0208, + 0x90be_fffa, 0xa450_6ceb, 0xbef9_a3f7, 0xc671_78f2, +]; + +#[derive(Clone)] +pub(crate) struct StreamingSha256 { + words: [u32; 8], + buffer: [u8; BLOCK_BYTES], + buffer_len: usize, + message_len_bytes: u64, +} + +impl Default for StreamingSha256 { + fn default() -> Self { + Self { + words: INITIAL_STATE, + buffer: [0; BLOCK_BYTES], + buffer_len: 0, + message_len_bytes: 0, + } + } +} + +impl StreamingSha256 { + /// Add the next contiguous admitted byte slice to this digest state. + pub(crate) fn update(&mut self, mut bytes: &[u8]) -> Result<(), ()> { + self.message_len_bytes = self + .message_len_bytes + .checked_add(bytes.len() as u64) + .ok_or(())?; + + if self.buffer_len != 0 { + let copied = (BLOCK_BYTES - self.buffer_len).min(bytes.len()); + self.buffer[self.buffer_len..self.buffer_len + copied] + .copy_from_slice(&bytes[..copied]); + self.buffer_len += copied; + bytes = &bytes[copied..]; + if self.buffer_len == BLOCK_BYTES { + let block = self.buffer; + self.compress(&block); + self.buffer_len = 0; + } + } + + while bytes.len() >= BLOCK_BYTES { + let block: &[u8; BLOCK_BYTES] = bytes[..BLOCK_BYTES].try_into().map_err(|_| ())?; + self.compress(block); + bytes = &bytes[BLOCK_BYTES..]; + } + + if !bytes.is_empty() { + self.buffer[..bytes.len()].copy_from_slice(bytes); + self.buffer_len = bytes.len(); + } + Ok(()) + } + + /// Finalize the digest as canonical lowercase hexadecimal. + pub(crate) fn finalize_hex(mut self) -> Result { + let message_len_bits = self.message_len_bytes.checked_mul(8).ok_or(())?; + + self.buffer[self.buffer_len] = 0x80; + self.buffer_len += 1; + if self.buffer_len > 56 { + self.buffer[self.buffer_len..].fill(0); + let block = self.buffer; + self.compress(&block); + self.buffer = [0; BLOCK_BYTES]; + self.buffer_len = 0; + } + self.buffer[self.buffer_len..56].fill(0); + self.buffer[56..].copy_from_slice(&message_len_bits.to_be_bytes()); + let block = self.buffer; + self.compress(&block); + + let mut digest = [0_u8; DIGEST_BYTES]; + for (index, word) in self.words.into_iter().enumerate() { + digest[index * 4..index * 4 + 4].copy_from_slice(&word.to_be_bytes()); + } + + let mut encoded = String::with_capacity(DIGEST_BYTES * 2); + const HEX: &[u8; 16] = b"0123456789abcdef"; + for byte in digest { + encoded.push(HEX[(byte >> 4) as usize] as char); + encoded.push(HEX[(byte & 0x0f) as usize] as char); + } + Ok(encoded) + } + + fn compress(&mut self, block: &[u8; BLOCK_BYTES]) { + let mut schedule = [0_u32; 64]; + for (index, chunk) in block.chunks_exact(4).enumerate() { + schedule[index] = u32::from_be_bytes( + chunk + .try_into() + .expect("SHA-256 message word always contains four bytes"), + ); + } + for index in 16..64 { + let small_sigma0 = schedule[index - 15].rotate_right(7) + ^ schedule[index - 15].rotate_right(18) + ^ (schedule[index - 15] >> 3); + let small_sigma1 = schedule[index - 2].rotate_right(17) + ^ schedule[index - 2].rotate_right(19) + ^ (schedule[index - 2] >> 10); + schedule[index] = schedule[index - 16] + .wrapping_add(small_sigma0) + .wrapping_add(schedule[index - 7]) + .wrapping_add(small_sigma1); + } + + let [mut a, mut b, mut c, mut d, mut e, mut f, mut g, mut h] = self.words; + for index in 0..64 { + let big_sigma1 = e.rotate_right(6) ^ e.rotate_right(11) ^ e.rotate_right(25); + let choose = (e & f) ^ ((!e) & g); + let temporary1 = h + .wrapping_add(big_sigma1) + .wrapping_add(choose) + .wrapping_add(ROUND_CONSTANTS[index]) + .wrapping_add(schedule[index]); + let big_sigma0 = a.rotate_right(2) ^ a.rotate_right(13) ^ a.rotate_right(22); + let majority = (a & b) ^ (a & c) ^ (b & c); + let temporary2 = big_sigma0.wrapping_add(majority); + + h = g; + g = f; + f = e; + e = d.wrapping_add(temporary1); + d = c; + c = b; + b = a; + a = temporary1.wrapping_add(temporary2); + } + + self.words[0] = self.words[0].wrapping_add(a); + self.words[1] = self.words[1].wrapping_add(b); + self.words[2] = self.words[2].wrapping_add(c); + self.words[3] = self.words[3].wrapping_add(d); + self.words[4] = self.words[4].wrapping_add(e); + self.words[5] = self.words[5].wrapping_add(f); + self.words[6] = self.words[6].wrapping_add(g); + self.words[7] = self.words[7].wrapping_add(h); + } +} + +/// Hash a caller-owned byte stream as canonical lowercase SHA-256. +/// +/// Security Notes: this helper never opens a path, logs bytes, or grants filesystem +/// authority. The caller must supply an already-authorized reader and decide how +/// the resulting digest is bound to a concrete artifact. `Interrupted` reads are +/// retried; other reader failures are returned unchanged. This is content identity, +/// not an authenticity primitive or a FIPS module-validation claim. +pub fn sha256_hex_reader(mut reader: impl Read) -> io::Result { + let mut digest = StreamingSha256::default(); + let mut chunk = [0_u8; 64 * 1024]; + loop { + match reader.read(&mut chunk) { + Ok(0) => break, + Ok(read_bytes) => digest + .update(&chunk[..read_bytes]) + .map_err(|_| io::Error::new(ErrorKind::InvalidData, "SHA-256 input too large"))?, + Err(error) if error.kind() == ErrorKind::Interrupted => continue, + Err(error) => return Err(error), + } + } + digest + .finalize_hex() + .map_err(|_| io::Error::new(ErrorKind::InvalidData, "SHA-256 input too large")) +} + +#[cfg(test)] +mod tests { + use super::*; + use std::io::{Cursor, Error}; + + fn digest_in_chunks(bytes: &[u8], chunk_size: usize) -> String { + let mut digest = StreamingSha256::default(); + for chunk in bytes.chunks(chunk_size) { + digest.update(chunk).expect("test vector length must fit SHA-256"); + } + digest + .finalize_hex() + .expect("test vector bit length must fit SHA-256") + } + + struct InterruptedShortReader { + bytes: Vec, + cursor: usize, + interrupted: bool, + } + + impl Read for InterruptedShortReader { + fn read(&mut self, output: &mut [u8]) -> io::Result { + if !self.interrupted { + self.interrupted = true; + return Err(Error::from(ErrorKind::Interrupted)); + } + if self.cursor == self.bytes.len() { + return Ok(0); + } + let copied = 7.min(output.len()).min(self.bytes.len() - self.cursor); + output[..copied].copy_from_slice(&self.bytes[self.cursor..self.cursor + copied]); + self.cursor += copied; + Ok(copied) + } + } + + struct FailingReader; + + impl Read for FailingReader { + fn read(&mut self, _output: &mut [u8]) -> io::Result { + Err(Error::new(ErrorKind::Other, "fixture read failure")) + } + } + + #[test] + fn matches_sha256_known_answer_vectors() { + for (message, expected) in [ + ( + &b""[..], + "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + ), + ( + &b"abc"[..], + "ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad", + ), + ( + &b"abcdbcdecdefdefgefghfghighijhijkijkljklmklmnlmnomnopnopq"[..], + "248d6a61d20638b8e5c026930c3e6039a33ce45964ff2167f6ecedd419db06c1", + ), + ] { + assert_eq!(digest_in_chunks(message, 7), expected); + } + } + + #[test] + fn shared_reader_retries_interrupted_short_reads() { + let bytes = (0..131_111) + .map(|index| (index % 251) as u8) + .collect::>(); + let expected = sha256_hex_reader(Cursor::new(&bytes)).expect("reference hash should succeed"); + let actual = sha256_hex_reader(InterruptedShortReader { + bytes, + cursor: 0, + interrupted: false, + }) + .expect("interrupted short reads should be retried"); + assert_eq!(actual, expected); + } + + #[test] + fn shared_reader_propagates_non_interrupted_failure() { + let error = sha256_hex_reader(FailingReader).expect_err("reader failure must propagate"); + assert_eq!(error.kind(), ErrorKind::Other); + } + + #[test] + fn matches_the_million_a_vector() { + assert_eq!( + digest_in_chunks(&vec![b'a'; 1_000_000], 64 * 1024), + "cdc76e5c9914fb9281a1c7e284d73e67f1809a48a497200e046d39ccc7112cd0" + ); + } +} diff --git a/apps/desktop/core/src/lib.rs b/apps/desktop/core/src/lib.rs index 200726570..eab65ea10 100644 --- a/apps/desktop/core/src/lib.rs +++ b/apps/desktop/core/src/lib.rs @@ -12,16 +12,26 @@ use std::{ collections::HashMap, io::Read, path::{Path, PathBuf}, - process::{Command, Stdio}, + process::{Child, Command}, sync::{ atomic::{AtomicU64, AtomicUsize, Ordering}, Arc, Mutex, }, - thread, - time::{Duration, Instant}, + time::Duration, }; +#[cfg(any(target_os = "linux", target_os = "macos"))] +use std::{ffi::c_int, os::unix::process::CommandExt}; use time::OffsetDateTime; +#[cfg(any(target_os = "linux", target_os = "macos"))] +const SIGKILL: c_int = 9; + +#[cfg(any(target_os = "linux", target_os = "macos"))] +extern "C" { + #[link_name = "kill"] + fn posix_kill(pid: c_int, signal: c_int) -> c_int; +} + #[derive(Clone)] pub struct AppState(pub Arc); @@ -79,6 +89,7 @@ pub enum AnalysisJobErrorCode { InvalidRequest, NotFound, EngineUnavailable, + Cancelled, } #[derive(Clone, Debug, Deserialize, Serialize)] @@ -423,101 +434,41 @@ pub fn youtube_missing_metadata_error(_parsed: &Value) -> String { "YouTube import reported ok but missing metadata.".to_string() } -pub fn wait_for_process_output( - mut command: Command, - timeout: Duration, - poll_interval: Duration, - timeout_message: &str, -) -> Result { - let mut child = command - .stdin(Stdio::null()) - .stdout(Stdio::piped()) - .stderr(Stdio::piped()) - .spawn() - .map_err(|_| "Failed to start YouTube import process.".to_string())?; - let stdout = child - .stdout - .take() - .expect("stdout should be piped for YouTube import process"); - let stderr = child - .stderr - .take() - .expect("stderr should be piped for YouTube import process"); - let stdout_reader = thread::spawn(move || { - let mut reader = stdout; - let mut buffer = Vec::new(); - reader.read_to_end(&mut buffer).map(|_| buffer) - }); - let stderr_reader = thread::spawn(move || { - let mut reader = stderr; - let mut buffer = Vec::new(); - reader.read_to_end(&mut buffer).map(|_| buffer) - }); - let deadline = Instant::now() + timeout; - - loop { - let process_status = { - #[cfg(coverage)] - { - child - .try_wait() - .expect("YouTube process status polling should not fail under coverage") - } - #[cfg(not(coverage))] - { - match child.try_wait() { - Ok(status) => status, - Err(_) => { - let _ = child.kill(); - let _ = child.wait(); - let _ = stdout_reader.join(); - let _ = stderr_reader.join(); - return Err("Failed to execute YouTube import process.".to_string()); - } - } - } - }; +/// Configure a BandScope-owned subprocess so ordinary Unix descendants share one process group. +/// +/// Security Notes: Linux and macOS create the group before `exec`. Windows is intentionally +/// direct-child-only until a race-free Job Object creation/assignment boundary is implemented. +pub fn configure_owned_process(command: &mut Command) { + #[cfg(any(target_os = "linux", target_os = "macos"))] + command.process_group(0); +} - match process_status { - Some(status) => { - #[cfg(coverage)] - let stdout = stdout_reader - .join() - .expect("stdout reader should not panic") - .expect("stdout reader should read process output"); - #[cfg(not(coverage))] - let stdout = stdout_reader - .join() - .map_err(|_| "Failed to execute YouTube import process.".to_string())? - .map_err(|_| "Failed to execute YouTube import process.".to_string())?; - #[cfg(coverage)] - let stderr = stderr_reader - .join() - .expect("stderr reader should not panic") - .expect("stderr reader should read process output"); - #[cfg(not(coverage))] - let stderr = stderr_reader - .join() - .map_err(|_| "Failed to execute YouTube import process.".to_string())? - .map_err(|_| "Failed to execute YouTube import process.".to_string())?; - return Ok(std::process::Output { - status, - stdout, - stderr, - }); - } - None => { - if Instant::now() >= deadline { - let _ = child.kill(); - let _ = child.wait(); - let _ = stdout_reader.join(); - let _ = stderr_reader.join(); - return Err(timeout_message.to_string()); - } - thread::sleep(poll_interval); - } - } +#[cfg(any(target_os = "linux", target_os = "macos"))] +fn kill_owned_process_group(child: &Child) -> bool { + let Ok(process_group_id) = c_int::try_from(child.id()) else { + return false; + }; + + // SAFETY: `configure_owned_process` establishes a fresh group whose id equals the child + // PID on supported Unix targets. A negative pid targets only that group. + unsafe { posix_kill(-process_group_id, SIGKILL) == 0 } +} + +/// Terminate a BandScope-owned subprocess boundary and reap the directly owned child. +/// +/// Security Notes: Linux and macOS signal the negative process-group id so ordinary descendants +/// that retain the inherited group terminate before reader threads are joined. If group signalling +/// fails, direct-child kill/reap remains the fail-closed fallback. This does not claim containment +/// for descendants that deliberately leave the group or for Windows descendants. +pub fn terminate_owned_process(child: &mut Child) { + #[cfg(any(target_os = "linux", target_os = "macos"))] + if kill_owned_process_group(child) { + let _ = child.wait(); + return; } + + let _ = child.kill(); + let _ = child.wait(); } pub fn is_youtube_video_id(value: &str) -> bool { @@ -955,7 +906,7 @@ mod tests { fn youtube_process_timeout_kills_and_reaps_child() { let command = long_sleep_command(); - let result = wait_for_process_output( + let result = crate::wait_for_process_output( command, Duration::from_millis(50), Duration::from_millis(5), @@ -972,7 +923,7 @@ mod tests { fn youtube_process_output_reports_spawn_failure() { let command = Command::new(unique_test_dir("missing-youtube-command").join("missing-tool")); - let result = wait_for_process_output( + let result = crate::wait_for_process_output( command, Duration::from_millis(50), Duration::from_millis(5), @@ -1022,10 +973,10 @@ mod tests { command .env("BANDSCOPE_TEST_CHILD_LARGE_OUTPUT", "1") .arg("--exact") - .arg("tests::youtube_process_output_drains_large_stdout_and_stderr_before_exit") + .arg("runtime_core::tests::youtube_process_output_drains_large_stdout_and_stderr_before_exit") .arg("--nocapture"); - let output = wait_for_process_output( + let output = crate::wait_for_process_output( command, Duration::from_secs(2), Duration::from_millis(5), diff --git a/apps/desktop/core/src/process_output.rs b/apps/desktop/core/src/process_output.rs new file mode 100644 index 000000000..1e1e8c707 --- /dev/null +++ b/apps/desktop/core/src/process_output.rs @@ -0,0 +1,327 @@ +use crate::runtime_core::{configure_owned_process, terminate_owned_process}; +use std::{ + io::{BufRead, BufReader, Error, ErrorKind, Read}, + process::{Command, Output, Stdio}, + sync::mpsc, + thread::{self, JoinHandle}, + time::{Duration, Instant}, +}; + +/// Maximum stdout or stderr bytes retained from one BandScope-owned helper process. +pub const MAX_PROCESS_OUTPUT_BYTES: usize = 1024 * 1024; + +const PROCESS_START_ERROR: &str = "Failed to start YouTube import process."; +const PROCESS_EXECUTION_ERROR: &str = "Failed to execute YouTube import process."; + +/// Read one helper stream into a bounded parent-side byte buffer. +/// +/// Security Notes: at most one probe byte beyond the product ceiling is read, +/// so callers can distinguish exact-limit EOF from overflow without permitting +/// an unbounded `Vec` allocation. This bounds captured bytes only; it does not +/// constrain memory allocated inside the helper process. +pub fn read_bounded_process_output(mut reader: impl Read) -> std::io::Result> { + let mut output = Vec::new(); + reader + .by_ref() + .take(MAX_PROCESS_OUTPUT_BYTES as u64 + 1) + .read_to_end(&mut output)?; + if output.len() > MAX_PROCESS_OUTPUT_BYTES { + return Err(Error::from(ErrorKind::InvalidData)); + } + Ok(output) +} + +/// Read newline-delimited helper output through the same bounded stream budget. +/// +/// Security Notes: the `Take` adapter permits at most the 1 MiB ceiling plus one +/// probe byte to enter the parent. A single unterminated line is therefore also +/// bounded. Only the transport CR/LF terminator is removed; every other decoded +/// character, including an empty transport record, is preserved so the protocol +/// parser rather than this resource boundary remains authoritative for validity. +/// The callback sees a line only after the cumulative stream remains within +/// policy, so an overflowing probe cannot become analysis-status input. +pub fn read_bounded_process_lines( + reader: impl Read, + mut on_line: impl FnMut(&str), +) -> std::io::Result<()> { + let mut reader = BufReader::new(reader).take(MAX_PROCESS_OUTPUT_BYTES as u64 + 1); + let mut total_bytes = 0usize; + let mut line = String::new(); + + loop { + line.clear(); + let read_bytes = reader.read_line(&mut line)?; + if read_bytes == 0 { + return Ok(()); + } + total_bytes = total_bytes + .checked_add(read_bytes) + .ok_or_else(|| Error::from(ErrorKind::InvalidData))?; + if total_bytes > MAX_PROCESS_OUTPUT_BYTES { + return Err(Error::from(ErrorKind::InvalidData)); + } + + let payload = line.strip_suffix('\n').unwrap_or(&line); + let payload = payload.strip_suffix('\r').unwrap_or(payload); + on_line(payload); + } +} + +fn join_process_output( + reader: JoinHandle>>, +) -> Result, String> { + reader + .join() + .map_err(|_| PROCESS_EXECUTION_ERROR.to_string())? + .map_err(|_| PROCESS_EXECUTION_ERROR.to_string()) +} + +/// Run one BandScope-owned helper with a deadline and bounded captured output. +/// +/// Security Notes: stdout and stderr are each admitted independently up to 1 MiB. +/// The reader consumes at most one probe byte beyond that ceiling and then drops +/// the pipe; oversized or unreadable output fails closed with a payload-free +/// product error and wakes the process-control owner without waiting for its next +/// ordinary poll. Poll waiting is clamped to the requested deadline, so a coarse +/// poll interval cannot silently extend helper lifetime. Process ownership and +/// descendant termination remain delegated to the shared runtime-core boundary. +/// This output ceiling limits parent-side capture memory only; it is not an +/// end-to-end RSS/VRAM or sandbox guarantee. +pub fn wait_for_process_output( + mut command: Command, + timeout: Duration, + poll_interval: Duration, + timeout_message: &str, +) -> Result { + configure_owned_process(&mut command); + let mut child = command + .stdin(Stdio::null()) + .stdout(Stdio::piped()) + .stderr(Stdio::piped()) + .spawn() + .map_err(|_| PROCESS_START_ERROR.to_string())?; + let stdout = child + .stdout + .take() + .expect("stdout should be piped for BandScope-owned helper process"); + let stderr = child + .stderr + .take() + .expect("stderr should be piped for BandScope-owned helper process"); + let (reader_failure_tx, reader_failure_rx) = mpsc::channel(); + let stdout_failure_tx = reader_failure_tx.clone(); + let stderr_failure_tx = reader_failure_tx.clone(); + let _reader_failure_guard = reader_failure_tx; + let stdout_reader = thread::spawn(move || { + let result = read_bounded_process_output(stdout); + if result.is_err() { + let _ = stdout_failure_tx.send(()); + } + result + }); + let stderr_reader = thread::spawn(move || { + let result = read_bounded_process_output(stderr); + if result.is_err() { + let _ = stderr_failure_tx.send(()); + } + result + }); + let deadline = Instant::now() + timeout; + + loop { + match child.try_wait() { + Ok(Some(status)) => { + // Preserve the direct child's observed status as product truth while the shared + // owner cleans any ordinary descendants before inherited pipes are joined. + terminate_owned_process(&mut child); + let stdout = join_process_output(stdout_reader)?; + let stderr = join_process_output(stderr_reader)?; + return Ok(Output { + status, + stdout, + stderr, + }); + } + Ok(None) if Instant::now() >= deadline => { + terminate_owned_process(&mut child); + let _ = stdout_reader.join(); + let _ = stderr_reader.join(); + return Err(timeout_message.to_string()); + } + Ok(None) => { + let wait_for = std::cmp::min( + poll_interval, + deadline.saturating_duration_since(Instant::now()), + ); + if reader_failure_rx.recv_timeout(wait_for).is_ok() { + terminate_owned_process(&mut child); + let _ = stdout_reader.join(); + let _ = stderr_reader.join(); + return Err(PROCESS_EXECUTION_ERROR.to_string()); + } + } + Err(_) => { + terminate_owned_process(&mut child); + let _ = stdout_reader.join(); + let _ = stderr_reader.join(); + return Err(PROCESS_EXECUTION_ERROR.to_string()); + } + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + use std::io::{Cursor, Error as IoError, Write}; + + struct FailingReader; + + impl Read for FailingReader { + fn read(&mut self, _buffer: &mut [u8]) -> std::io::Result { + Err(IoError::from(ErrorKind::Other)) + } + } + + #[test] + fn bounded_output_accepts_exact_limit() { + let output = read_bounded_process_output(Cursor::new(vec![b'x'; MAX_PROCESS_OUTPUT_BYTES])) + .expect("the exact process-output ceiling remains admissible"); + + assert_eq!(output.len(), MAX_PROCESS_OUTPUT_BYTES); + } + + #[test] + fn bounded_output_rejects_one_byte_over_limit() { + let error = read_bounded_process_output(Cursor::new(vec![ + b'x'; + MAX_PROCESS_OUTPUT_BYTES + 1 + ])) + .expect_err("one byte beyond the process-output ceiling must fail closed"); + + assert_eq!(error.kind(), ErrorKind::InvalidData); + } + + #[test] + fn bounded_output_preserves_reader_failure_as_internal_error() { + let error = read_bounded_process_output(FailingReader) + .expect_err("reader failure must remain an execution failure"); + + assert_eq!(error.kind(), ErrorKind::Other); + } + + #[test] + fn bounded_lines_preserve_empty_transport_records() { + let mut lines = Vec::new(); + read_bounded_process_lines(Cursor::new(b"first\n\n second \n"), |line| { + lines.push(line.to_string()); + }) + .expect("small newline-delimited output should remain admissible"); + + assert_eq!(lines, vec!["first", "", " second "]); + } + + #[test] + fn bounded_lines_preserve_non_line_ending_whitespace() { + let mut lines = Vec::new(); + let payload = " first \r\n\u{00a0}second\u{00a0}\n"; + read_bounded_process_lines(Cursor::new(payload.as_bytes()), |line| { + lines.push(line.to_string()); + }) + .expect("bounded JSONL transport must preserve payload whitespace outside line endings"); + + assert_eq!(lines, vec![" first ", "\u{00a0}second\u{00a0}"]); + } + + #[test] + fn bounded_lines_accept_exact_limit() { + let payload = vec![b'x'; MAX_PROCESS_OUTPUT_BYTES]; + let mut observed_bytes = 0usize; + read_bounded_process_lines(Cursor::new(payload), |line| { + observed_bytes = line.len(); + }) + .expect("the exact streaming output ceiling remains admissible"); + + assert_eq!(observed_bytes, MAX_PROCESS_OUTPUT_BYTES); + } + + #[test] + fn bounded_lines_reject_one_byte_over_limit_before_callback() { + let payload = vec![b'x'; MAX_PROCESS_OUTPUT_BYTES + 1]; + let mut callback_called = false; + let error = read_bounded_process_lines(Cursor::new(payload), |_| { + callback_called = true; + }) + .expect_err("one streaming byte beyond the process-output ceiling must fail closed"); + + assert_eq!(error.kind(), ErrorKind::InvalidData); + assert!(!callback_called); + } + + #[test] + fn oversized_process_output_wakes_before_coarse_poll_interval() { + if std::env::var_os("BANDSCOPE_TEST_CHILD_OVERSIZED_OUTPUT").is_some() { + std::thread::sleep(Duration::from_millis(100)); + let oversized_output = vec![b'x'; MAX_PROCESS_OUTPUT_BYTES + 1]; + let mut stdout = std::io::stdout(); + let _ = stdout.write_all(&oversized_output); + let _ = stdout.flush(); + std::thread::sleep(Duration::from_secs(5)); + return; + } + + let current_test_binary = std::env::current_exe().expect("test binary should resolve"); + let mut command = Command::new(current_test_binary); + command + .env("BANDSCOPE_TEST_CHILD_OVERSIZED_OUTPUT", "1") + .arg("--exact") + .arg("process_output::tests::oversized_process_output_wakes_before_coarse_poll_interval") + .arg("--nocapture"); + let started_at = Instant::now(); + + let error = wait_for_process_output( + command, + Duration::from_secs(4), + Duration::from_secs(2), + "YouTube import timed out.", + ) + .expect_err("known output overflow must wake the owner before the next coarse poll"); + + assert_eq!(error, PROCESS_EXECUTION_ERROR); + assert!( + started_at.elapsed() < Duration::from_secs(1), + "reader failure should wake process control instead of waiting for the poll interval" + ); + } + + #[test] + fn process_timeout_does_not_oversleep_poll_interval() { + if std::env::var_os("BANDSCOPE_TEST_CHILD_SLOW_PROCESS").is_some() { + std::thread::sleep(Duration::from_secs(5)); + return; + } + + let current_test_binary = std::env::current_exe().expect("test binary should resolve"); + let mut command = Command::new(current_test_binary); + command + .env("BANDSCOPE_TEST_CHILD_SLOW_PROCESS", "1") + .arg("--exact") + .arg("process_output::tests::process_timeout_does_not_oversleep_poll_interval") + .arg("--nocapture"); + let started_at = Instant::now(); + + let error = wait_for_process_output( + command, + Duration::from_millis(100), + Duration::from_secs(2), + "YouTube import timed out.", + ) + .expect_err("the helper deadline must not be extended by a coarse poll interval"); + + assert_eq!(error, "YouTube import timed out."); + assert!( + started_at.elapsed() < Duration::from_secs(1), + "the process owner must wake at the deadline rather than after the full poll interval" + ); + } +} diff --git a/apps/desktop/core/src/publication_identity.rs b/apps/desktop/core/src/publication_identity.rs new file mode 100644 index 000000000..63fddb190 --- /dev/null +++ b/apps/desktop/core/src/publication_identity.rs @@ -0,0 +1,117 @@ +use crate::{ + audio_resource::{LocalAudioCopyReceipt, MAX_LOCAL_AUDIO_FILE_BYTES}, + runtime_core::{is_valid_project_id, AUDIO_EXTENSIONS}, +}; +use serde::{de::Error as _, Deserialize, Deserializer, Serialize}; + +const LOCAL_AUDIO_PUBLICATION_IDENTITY_ERROR: &str = + "Could not prepare the local project workspace."; + +/// Path-free native identity for one verified app-owned local-audio publication. +/// +/// This value is suitable for Project Persistence handoff because it names only +/// a BandScope-owned artifact and carries the exact native size/digest evidence +/// produced by Resource Admission. It never contains an external or absolute +/// filesystem path. +#[derive(Clone, Debug, Eq, PartialEq, Serialize)] +#[serde(rename_all = "camelCase", deny_unknown_fields)] +pub struct LocalAudioPublicationIdentity { + /// Locally minted BandScope project id that owns the publication. + pub project_id: String, + /// Deterministic app-owned artifact name within that project. + pub artifact_name: String, + /// Canonical lowercase admitted audio extension. + pub extension: String, + /// Exact number of bytes in the verified publication. + pub file_size_bytes: u64, + /// Lowercase SHA-256 of the exact verified publication bytes. + pub content_sha256: String, +} + +#[derive(Deserialize)] +#[serde(rename_all = "camelCase", deny_unknown_fields)] +struct LocalAudioPublicationIdentityWire { + project_id: String, + artifact_name: String, + extension: String, + file_size_bytes: u64, + content_sha256: String, +} + +impl<'de> Deserialize<'de> for LocalAudioPublicationIdentity { + fn deserialize(deserializer: D) -> Result + where + D: Deserializer<'de>, + { + let wire = LocalAudioPublicationIdentityWire::deserialize(deserializer)?; + let receipt = LocalAudioCopyReceipt { + file_size_bytes: wire.file_size_bytes, + content_sha256: wire.content_sha256, + }; + let identity = build_local_audio_publication_identity( + &wire.project_id, + &wire.extension, + &receipt, + ) + .map_err(D::Error::custom)?; + if wire.artifact_name != identity.artifact_name { + return Err(D::Error::custom(LOCAL_AUDIO_PUBLICATION_IDENTITY_ERROR)); + } + Ok(identity) + } +} + +fn is_lowercase_sha256(value: &str) -> bool { + value.len() == 64 + && value + .bytes() + .all(|byte| byte.is_ascii_digit() || (b'a'..=b'f').contains(&byte)) +} + +/// Build the durable path-free identity for verified local-audio publication evidence. +/// +/// Security Notes: callers must supply a project id minted under BandScope's +/// existing project-id grammar and the canonical lowercase extension that was +/// admitted by Resource Admission. The receipt must come from the verified +/// publication path, not renderer input. Invalid ids, extensions, sizes, digest +/// encodings, or deserialized artifact-name mismatches fail closed with the +/// bounded project-workspace diagnosis. +pub fn build_local_audio_publication_identity( + project_id: &str, + extension: &str, + receipt: &LocalAudioCopyReceipt, +) -> Result { + if !is_valid_project_id(project_id) + || !AUDIO_EXTENSIONS.contains(&extension) + || extension.bytes().any(|byte| byte.is_ascii_uppercase()) + || receipt.file_size_bytes == 0 + || receipt.file_size_bytes > MAX_LOCAL_AUDIO_FILE_BYTES + || !is_lowercase_sha256(&receipt.content_sha256) + { + return Err(LOCAL_AUDIO_PUBLICATION_IDENTITY_ERROR.to_string()); + } + + Ok(LocalAudioPublicationIdentity { + project_id: project_id.to_string(), + artifact_name: format!("source.{extension}"), + extension: extension.to_string(), + file_size_bytes: receipt.file_size_bytes, + content_sha256: receipt.content_sha256.clone(), + }) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn lowercase_sha256_requires_exact_canonical_encoding() { + assert!(is_lowercase_sha256( + "9f64a747e1b97f131fabb6b447296c9b6f0201e79fb3c5356e6c77e89b6a806a" + )); + assert!(!is_lowercase_sha256(&"a".repeat(63))); + assert!(!is_lowercase_sha256(&"a".repeat(65))); + assert!(!is_lowercase_sha256(&"A".repeat(64))); + assert!(!is_lowercase_sha256(&"g".repeat(64))); + } +} diff --git a/apps/desktop/core/src/root.rs b/apps/desktop/core/src/root.rs new file mode 100644 index 000000000..667fb0ca6 --- /dev/null +++ b/apps/desktop/core/src/root.rs @@ -0,0 +1,30 @@ +//! Pure, GUI-independent logic for the BandScope desktop application. +//! +//! The historical desktop-core implementation remains in `lib.rs` as the +//! compatibility module while bounded resource boundaries are isolated in +//! auditable modules. Public symbols are re-exported so downstream callers keep +//! the same crate-root API. + +#[path = "lib.rs"] +mod runtime_core; +mod audio_resource; +mod content_sha256; +mod process_output; +mod publication_identity; +mod score_pdf; + +pub use audio_resource::{ + copy_bounded_local_audio, copy_bounded_local_audio_with_receipt, + validate_local_audio_file_size, verify_local_audio_publication_receipt, + LocalAudioCopyReceipt, MAX_LOCAL_AUDIO_FILE_BYTES, +}; +pub use content_sha256::sha256_hex_reader; +pub use process_output::{ + read_bounded_process_lines, read_bounded_process_output, wait_for_process_output, + MAX_PROCESS_OUTPUT_BYTES, +}; +pub use publication_identity::{ + build_local_audio_publication_identity, LocalAudioPublicationIdentity, +}; +pub use runtime_core::*; +pub use score_pdf::read_validated_score_pdf; diff --git a/apps/desktop/core/src/score_pdf.rs b/apps/desktop/core/src/score_pdf.rs new file mode 100644 index 000000000..2b26744cc --- /dev/null +++ b/apps/desktop/core/src/score_pdf.rs @@ -0,0 +1,82 @@ +use crate::{MAX_SCORE_PDF_BYTES, PDF_MAGIC}; +use std::{fs::File, io::Read, path::Path}; + +const SCORE_READ_ERROR: &str = "Could not read the score PDF."; +const SCORE_TOO_LARGE_ERROR: &str = "Score PDF is too large (exceeds 25MB limit)."; +const SCORE_INVALID_PDF_ERROR: &str = "Stored score is not a valid PDF."; + +fn read_validated_pdf_stream( + reader: &mut impl Read, + expected_len: u64, +) -> Result, String> { + if expected_len > MAX_SCORE_PDF_BYTES { + return Err(SCORE_TOO_LARGE_ERROR.to_string()); + } + + // MAX_SCORE_PDF_BYTES is 25 MiB, which fits every supported Rust `usize`. + let mut bytes = vec![0_u8; expected_len as usize]; + reader + .read_exact(&mut bytes) + .map_err(|_| SCORE_READ_ERROR.to_string())?; + + let mut growth_probe = [0_u8; 1]; + if reader + .read(&mut growth_probe) + .map_err(|_| SCORE_READ_ERROR.to_string())? + != 0 + { + return Err(SCORE_TOO_LARGE_ERROR.to_string()); + } + + if !bytes.starts_with(PDF_MAGIC) { + return Err(SCORE_INVALID_PDF_ERROR.to_string()); + } + + Ok(bytes) +} + +/// Read one already-authorized stored score without allocating beyond the PDF limit. +/// +/// The caller remains responsible for path authority and containment. This helper +/// opens that resolved path once, snapshots the descriptor length, allocates only +/// that bounded size, reads exactly that many bytes, and then probes one additional +/// byte on the same descriptor. A file that was already oversized is rejected +/// before heap allocation; a file that grows after metadata inspection is rejected +/// by the one-byte probe without extending the heap buffer beyond the product cap. +/// Errors intentionally omit the local path and file content. +pub fn read_validated_score_pdf(path: &Path) -> Result, String> { + let mut file = File::open(path).map_err(|_| SCORE_READ_ERROR.to_string())?; + let metadata = file + .metadata() + .map_err(|_| SCORE_READ_ERROR.to_string())?; + if !metadata.is_file() { + return Err(SCORE_READ_ERROR.to_string()); + } + read_validated_pdf_stream(&mut file, metadata.len()) +} + +#[cfg(test)] +mod tests { + use super::*; + use std::io::Cursor; + + #[test] + fn stream_rejects_growth_after_the_metadata_length_snapshot() { + let mut reader = Cursor::new(b"%PDF-extra".to_vec()); + + let error = read_validated_pdf_stream(&mut reader, PDF_MAGIC.len() as u64) + .expect_err("bytes beyond the metadata snapshot must fail closed"); + + assert_eq!(error, SCORE_TOO_LARGE_ERROR); + } + + #[test] + fn stream_rejects_truncation_after_the_metadata_length_snapshot() { + let mut reader = Cursor::new(PDF_MAGIC.to_vec()); + + let error = read_validated_pdf_stream(&mut reader, (PDF_MAGIC.len() + 1) as u64) + .expect_err("truncation after the metadata snapshot must fail closed"); + + assert_eq!(error, SCORE_READ_ERROR); + } +} diff --git a/apps/desktop/core/tests/analysis_job_cancellation_error.rs b/apps/desktop/core/tests/analysis_job_cancellation_error.rs new file mode 100644 index 000000000..13006a4d6 --- /dev/null +++ b/apps/desktop/core/tests/analysis_job_cancellation_error.rs @@ -0,0 +1,14 @@ +use bandscope_desktop_core::{AnalysisJobError, AnalysisJobErrorCode}; + +#[test] +fn cancelled_analysis_error_uses_the_shared_snake_case_wire_code() { + let error = AnalysisJobError { + code: AnalysisJobErrorCode::Cancelled, + message: "Analysis was cancelled.".to_string(), + }; + + let value = serde_json::to_value(error).expect("analysis error should serialize"); + + assert_eq!(value["code"], "cancelled"); + assert_eq!(value["message"], "Analysis was cancelled."); +} diff --git a/apps/desktop/core/tests/audio_resource_next_action.rs b/apps/desktop/core/tests/audio_resource_next_action.rs new file mode 100644 index 000000000..9fa77d800 --- /dev/null +++ b/apps/desktop/core/tests/audio_resource_next_action.rs @@ -0,0 +1,9 @@ +use bandscope_desktop_core::{validate_local_audio_file_size, MAX_LOCAL_AUDIO_FILE_BYTES}; + +#[test] +fn oversized_local_audio_names_the_next_rehearsal_action() { + assert_eq!( + validate_local_audio_file_size(MAX_LOCAL_AUDIO_FILE_BYTES + 1), + Err("Choose a shorter or smaller song file to start analysis.".to_string()) + ); +} diff --git a/apps/desktop/core/tests/audio_resource_policy.rs b/apps/desktop/core/tests/audio_resource_policy.rs new file mode 100644 index 000000000..163e49d50 --- /dev/null +++ b/apps/desktop/core/tests/audio_resource_policy.rs @@ -0,0 +1,25 @@ +use bandscope_desktop_core::{validate_local_audio_file_size, MAX_LOCAL_AUDIO_FILE_BYTES}; + +#[test] +fn local_audio_size_policy_accepts_the_exact_native_bootstrap_ceiling() { + assert_eq!( + validate_local_audio_file_size(MAX_LOCAL_AUDIO_FILE_BYTES), + Ok(MAX_LOCAL_AUDIO_FILE_BYTES) + ); +} + +#[test] +fn local_audio_size_policy_rejects_an_empty_native_bootstrap_source() { + assert_eq!( + validate_local_audio_file_size(0), + Err("Could not read the selected audio file.".to_string()) + ); +} + +#[test] +fn local_audio_size_policy_rejects_a_native_source_above_the_canonical_ceiling() { + assert_eq!( + validate_local_audio_file_size(MAX_LOCAL_AUDIO_FILE_BYTES + 1), + Err("Choose a shorter or smaller song file to start analysis.".to_string()) + ); +} diff --git a/apps/desktop/core/tests/content_sha256_shared_kernel.rs b/apps/desktop/core/tests/content_sha256_shared_kernel.rs new file mode 100644 index 000000000..b8d9a0ebd --- /dev/null +++ b/apps/desktop/core/tests/content_sha256_shared_kernel.rs @@ -0,0 +1,10 @@ +use bandscope_desktop_core::sha256_hex_reader; +use std::io::Cursor; + +#[test] +fn shared_sha256_reader_matches_the_fips_180_4_abc_vector() { + assert_eq!( + sha256_hex_reader(Cursor::new(b"abc")).as_deref(), + Ok("ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad") + ); +} diff --git a/apps/desktop/core/tests/local_audio_content_identity.rs b/apps/desktop/core/tests/local_audio_content_identity.rs new file mode 100644 index 000000000..b43eb1bc5 --- /dev/null +++ b/apps/desktop/core/tests/local_audio_content_identity.rs @@ -0,0 +1,43 @@ +use bandscope_desktop_core::{ + copy_bounded_local_audio_with_receipt, verify_local_audio_publication_receipt, +}; +use std::io::Cursor; + +#[test] +fn local_audio_copy_receipt_hashes_exact_admitted_bytes() { + let input = vec![1_u8, 2, 3, 4]; + let mut staged = Vec::new(); + + let receipt = copy_bounded_local_audio_with_receipt(Cursor::new(&input), &mut staged) + .expect("bounded admission should return content identity for the bytes it stages"); + + assert_eq!(staged, input); + assert_eq!(receipt.file_size_bytes, 4); + assert_eq!( + receipt.content_sha256, + "9f64a747e1b97f131fabb6b447296c9b6f0201e79fb3c5356e6c77e89b6a806a" + ); +} + +#[test] +fn publication_receipt_requires_the_published_bytes_to_match_the_stage() { + let input = vec![1_u8, 2, 3, 4]; + let mut staged = Vec::new(); + let staged_receipt = copy_bounded_local_audio_with_receipt(Cursor::new(&input), &mut staged) + .expect("staging should produce native identity evidence"); + + let published_receipt = verify_local_audio_publication_receipt( + Cursor::new(&staged), + &staged_receipt, + ) + .expect("unchanged published bytes should retain the staging identity"); + + assert_eq!(published_receipt, staged_receipt); + + let mismatch = verify_local_audio_publication_receipt( + Cursor::new(vec![1_u8, 2, 3, 5]), + &staged_receipt, + ) + .expect_err("same-size mutation after staging must fail publication binding"); + assert_eq!(mismatch, "Could not prepare the local project workspace."); +} diff --git a/apps/desktop/core/tests/local_audio_publication_identity.rs b/apps/desktop/core/tests/local_audio_publication_identity.rs new file mode 100644 index 000000000..cd3b5fb63 --- /dev/null +++ b/apps/desktop/core/tests/local_audio_publication_identity.rs @@ -0,0 +1,77 @@ +use bandscope_desktop_core::{ + build_local_audio_publication_identity, LocalAudioCopyReceipt, +}; + +fn receipt() -> LocalAudioCopyReceipt { + LocalAudioCopyReceipt { + file_size_bytes: 4, + content_sha256: + "9f64a747e1b97f131fabb6b447296c9b6f0201e79fb3c5356e6c77e89b6a806a".to_string(), + } +} + +#[test] +fn publication_identity_is_path_free_and_deterministic() { + let identity = build_local_audio_publication_identity("project-1-1", "wav", &receipt()) + .expect("verified publication evidence should become a durable path-free identity"); + + assert_eq!(identity.project_id, "project-1-1"); + assert_eq!(identity.artifact_name, "source.wav"); + assert_eq!(identity.extension, "wav"); + assert_eq!(identity.file_size_bytes, 4); + assert_eq!( + identity.content_sha256, + "9f64a747e1b97f131fabb6b447296c9b6f0201e79fb3c5356e6c77e89b6a806a" + ); + + let json = serde_json::to_value(&identity).expect("publication identity should serialize"); + assert_eq!( + json, + serde_json::json!({ + "projectId": "project-1-1", + "artifactName": "source.wav", + "extension": "wav", + "fileSizeBytes": 4, + "contentSha256": "9f64a747e1b97f131fabb6b447296c9b6f0201e79fb3c5356e6c77e89b6a806a" + }) + ); + assert!(json.get("sourcePath").is_none()); + assert!(json.get("path").is_none()); +} + +#[test] +fn publication_identity_rejects_noncanonical_or_fabricated_evidence() { + for (project_id, extension, receipt) in [ + ("../project-1-1", "wav", receipt()), + ("project-1-1", "WAV", receipt()), + ("project-1-1", "exe", receipt()), + ( + "project-1-1", + "wav", + LocalAudioCopyReceipt { + file_size_bytes: 0, + content_sha256: "00".repeat(32), + }, + ), + ( + "project-1-1", + "wav", + LocalAudioCopyReceipt { + file_size_bytes: 4, + content_sha256: "AA".repeat(32), + }, + ), + ( + "project-1-1", + "wav", + LocalAudioCopyReceipt { + file_size_bytes: 4, + content_sha256: "not-a-sha256".to_string(), + }, + ), + ] { + let error = build_local_audio_publication_identity(project_id, extension, &receipt) + .expect_err("only canonical native publication evidence may cross persistence handoff"); + assert_eq!(error, "Could not prepare the local project workspace."); + } +} diff --git a/apps/desktop/core/tests/local_audio_publication_identity_deserialization.rs b/apps/desktop/core/tests/local_audio_publication_identity_deserialization.rs new file mode 100644 index 000000000..8551851db --- /dev/null +++ b/apps/desktop/core/tests/local_audio_publication_identity_deserialization.rs @@ -0,0 +1,69 @@ +use bandscope_desktop_core::LocalAudioPublicationIdentity; +use serde_json::{json, Value}; + +const CANONICAL_SHA256: &str = + "9f64a747e1b97f131fabb6b447296c9b6f0201e79fb3c5356e6c77e89b6a806a"; + +fn valid_identity_json() -> Value { + json!({ + "projectId": "project-1-1", + "artifactName": "source.wav", + "extension": "wav", + "fileSizeBytes": 4, + "contentSha256": CANONICAL_SHA256, + }) +} + +#[test] +fn publication_identity_deserialization_reuses_canonical_validation() { + let identity: LocalAudioPublicationIdentity = serde_json::from_value(valid_identity_json()) + .expect("canonical native publication identity should deserialize"); + + assert_eq!(identity.project_id, "project-1-1"); + assert_eq!(identity.artifact_name, "source.wav"); + assert_eq!(identity.extension, "wav"); + assert_eq!(identity.file_size_bytes, 4); + assert_eq!(identity.content_sha256, CANONICAL_SHA256); +} + +#[test] +fn publication_identity_deserialization_rejects_invalid_field_classes() { + let mut invalid_id = valid_identity_json(); + invalid_id["projectId"] = json!("../project-1-1"); + + let mut uppercase_extension = valid_identity_json(); + uppercase_extension["extension"] = json!("WAV"); + + let mut unsupported_extension = valid_identity_json(); + unsupported_extension["extension"] = json!("exe"); + + let mut zero_size = valid_identity_json(); + zero_size["fileSizeBytes"] = json!(0); + + let mut oversized = valid_identity_json(); + oversized["fileSizeBytes"] = json!(100 * 1024 * 1024_u64 + 1); + + let mut uppercase_digest = valid_identity_json(); + uppercase_digest["contentSha256"] = json!(CANONICAL_SHA256.to_uppercase()); + + let mut malformed_digest = valid_identity_json(); + malformed_digest["contentSha256"] = json!("not-a-sha256"); + + let mut mismatched_artifact = valid_identity_json(); + mismatched_artifact["artifactName"] = json!("source.mp3"); + + for candidate in [ + invalid_id, + uppercase_extension, + unsupported_extension, + zero_size, + oversized, + uppercase_digest, + malformed_digest, + mismatched_artifact, + ] { + let error = serde_json::from_value::(candidate) + .expect_err("unvalidated publication identity must fail closed"); + assert!(error.to_string().contains("Could not prepare the local project workspace.")); + } +} diff --git a/apps/desktop/core/tests/score_pdf_read.rs b/apps/desktop/core/tests/score_pdf_read.rs new file mode 100644 index 000000000..b70068931 --- /dev/null +++ b/apps/desktop/core/tests/score_pdf_read.rs @@ -0,0 +1,91 @@ +use bandscope_desktop_core::{read_validated_score_pdf, MAX_SCORE_PDF_BYTES}; +use std::io::Write; +use std::path::PathBuf; +use std::time::{SystemTime, UNIX_EPOCH}; + +fn unique_test_dir(name: &str) -> PathBuf { + let suffix = SystemTime::now() + .duration_since(UNIX_EPOCH) + .expect("system clock should be after epoch") + .as_nanos(); + std::env::temp_dir().join(format!("bandscope-{name}-{suffix}")) +} + +#[test] +fn score_pdf_read_returns_only_valid_bounded_pdf_bytes() { + let root = unique_test_dir("score-read-valid"); + std::fs::create_dir_all(&root).expect("test directory should be created"); + let path = root.join("score.pdf"); + let expected = b"%PDF-1.7\nvalidated body"; + std::fs::write(&path, expected).expect("valid PDF fixture should be written"); + + let actual = read_validated_score_pdf(&path).expect("valid stored PDF should be readable"); + + assert_eq!(actual, expected); + let _ = std::fs::remove_dir_all(root); +} + +#[test] +fn score_pdf_read_rejects_empty_short_and_wrong_magic_content() { + let root = unique_test_dir("score-read-invalid"); + std::fs::create_dir_all(&root).expect("test directory should be created"); + + for (name, content) in [ + ("empty.pdf", b"".as_slice()), + ("short.pdf", b"%PD".as_slice()), + ("wrong.pdf", b"PK\x03\x04 not a PDF".as_slice()), + ] { + let path = root.join(name); + std::fs::write(&path, content).expect("invalid PDF fixture should be written"); + let error = read_validated_score_pdf(&path).expect_err("invalid PDF must fail closed"); + assert!( + error == "Could not read the score PDF." || error == "Stored score is not a valid PDF.", + "unexpected payload-safe error: {error}" + ); + assert!(!error.contains(root.to_string_lossy().as_ref())); + } + + let _ = std::fs::remove_dir_all(root); +} + +#[test] +fn score_pdf_read_rejects_oversized_sparse_file_before_heap_allocation() { + let root = unique_test_dir("score-read-oversized"); + std::fs::create_dir_all(&root).expect("test directory should be created"); + let path = root.join("oversized.pdf"); + let mut file = std::fs::File::create(&path).expect("oversized PDF fixture should be created"); + file.write_all(b"%PDF-") + .expect("PDF magic should be written before extending sparse file"); + file.set_len(MAX_SCORE_PDF_BYTES + 1) + .expect("sparse PDF fixture should exceed the product limit"); + drop(file); + + let error = read_validated_score_pdf(&path).expect_err("oversized PDF must fail closed"); + + assert_eq!(error, "Score PDF is too large (exceeds 25MB limit)."); + let _ = std::fs::remove_dir_all(root); +} + +#[cfg(unix)] +#[test] +fn score_pdf_read_rejects_non_file_descriptor() { + let root = unique_test_dir("score-read-directory"); + std::fs::create_dir_all(&root).expect("test directory should be created"); + + let error = read_validated_score_pdf(&root).expect_err("directory must fail closed"); + + assert_eq!(error, "Could not read the score PDF."); + assert!(!error.contains(root.to_string_lossy().as_ref())); + let _ = std::fs::remove_dir_all(root); +} + +#[test] +fn score_pdf_read_rejects_missing_file_without_exposing_path() { + let root = unique_test_dir("score-read-missing"); + let path = root.join("private-score.pdf"); + + let error = read_validated_score_pdf(&path).expect_err("missing PDF must fail closed"); + + assert_eq!(error, "Could not read the score PDF."); + assert!(!error.contains("private-score.pdf")); +} diff --git a/apps/desktop/core/tests/youtube_process_containment.rs b/apps/desktop/core/tests/youtube_process_containment.rs new file mode 100644 index 000000000..8b19778af --- /dev/null +++ b/apps/desktop/core/tests/youtube_process_containment.rs @@ -0,0 +1,110 @@ +#[test] +fn process_output_has_one_canonical_implementation_owner() { + let compatibility_source = include_str!("../src/lib.rs"); + let process_output_source = include_str!("../src/process_output.rs"); + + assert!( + !compatibility_source.contains("pub fn wait_for_process_output("), + "the compatibility module must not retain a second public process-output implementation" + ); + assert_eq!( + process_output_source + .matches("pub fn wait_for_process_output(") + .count(), + 1, + "bounded helper execution must have one canonical implementation owner" + ); +} + +#[cfg(any(target_os = "linux", target_os = "macos"))] +#[test] +fn youtube_timeout_terminates_descendant_that_keeps_output_pipe_open() { + use bandscope_desktop_core::wait_for_process_output; + use std::{ + process::Command, + time::{Duration, Instant}, + }; + + let mut command = Command::new("sh"); + command.arg("-c").arg("sleep 5 & sleep 5"); + + let started = Instant::now(); + let result = wait_for_process_output( + command, + Duration::from_millis(50), + Duration::from_millis(5), + "YouTube import timed out.", + ); + let elapsed = started.elapsed(); + + assert_eq!( + result.expect_err("the import process should hit the product timeout"), + "YouTube import timed out." + ); + assert!( + elapsed < Duration::from_secs(1), + "timeout cleanup must terminate descendants that retain inherited stdout/stderr; elapsed={elapsed:?}" + ); +} + +#[cfg(any(target_os = "linux", target_os = "macos"))] +#[test] +fn youtube_success_terminates_descendant_that_keeps_output_pipe_open() { + use bandscope_desktop_core::wait_for_process_output; + use std::{ + process::Command, + time::{Duration, Instant}, + }; + + let mut command = Command::new("sh"); + command.arg("-c").arg("sleep 5 & exit 0"); + + let started = Instant::now(); + let output = wait_for_process_output( + command, + Duration::from_secs(2), + Duration::from_millis(5), + "YouTube import timed out.", + ) + .expect("the directly owned process should exit successfully"); + let elapsed = started.elapsed(); + + assert!(output.status.success()); + assert!( + elapsed < Duration::from_secs(1), + "successful parent exit must not block on an inherited descendant pipe; elapsed={elapsed:?}" + ); +} + +#[cfg(any(target_os = "linux", target_os = "macos"))] +#[test] +fn youtube_output_is_bounded_before_metadata_parse() { + use bandscope_desktop_core::wait_for_process_output; + use std::{ + process::Command, + time::{Duration, Instant}, + }; + + let mut command = Command::new("sh"); + command + .arg("-c") + .arg("dd if=/dev/zero bs=1048576 count=2 2>/dev/null"); + + let started = Instant::now(); + let result = wait_for_process_output( + command, + Duration::from_secs(5), + Duration::from_millis(5), + "YouTube import timed out.", + ); + let elapsed = started.elapsed(); + + assert_eq!( + result.expect_err("oversized subprocess output must fail closed"), + "Failed to execute YouTube import process." + ); + assert!( + elapsed < Duration::from_secs(2), + "output admission must not retain an unbounded pipe until the product timeout; elapsed={elapsed:?}" + ); +} diff --git a/apps/desktop/src-tauri/build.rs b/apps/desktop/src-tauri/build.rs index 0caf74c64..42de007c2 100644 --- a/apps/desktop/src-tauri/build.rs +++ b/apps/desktop/src-tauri/build.rs @@ -3,6 +3,7 @@ fn main() { tauri_build::AppManifest::new().commands(&[ "start_analysis_job", "get_analysis_job_status", + "cancel_analysis_job", "select_local_audio_source", "import_youtube_url", "save_project", diff --git a/apps/desktop/src-tauri/capabilities/main.json b/apps/desktop/src-tauri/capabilities/main.json index 8103420f4..a56517c22 100644 --- a/apps/desktop/src-tauri/capabilities/main.json +++ b/apps/desktop/src-tauri/capabilities/main.json @@ -8,6 +8,7 @@ "core:event:allow-unlisten", "allow-start-analysis-job", "allow-get-analysis-job-status", + "allow-cancel-analysis-job", "allow-select-local-audio-source", "allow-import-youtube-url", "allow-save-project", diff --git a/apps/desktop/src-tauri/gen/schemas/capabilities.json b/apps/desktop/src-tauri/gen/schemas/capabilities.json index 25d764245..86943b9a9 100644 --- a/apps/desktop/src-tauri/gen/schemas/capabilities.json +++ b/apps/desktop/src-tauri/gen/schemas/capabilities.json @@ -1 +1 @@ -{"main-capability":{"identifier":"main-capability","description":"Capability for the main BandScope window to use the analysis orchestration commands.","local":true,"windows":["main"],"permissions":["core:event:allow-listen","core:event:allow-unlisten","allow-start-analysis-job","allow-get-analysis-job-status","allow-select-local-audio-source","allow-import-youtube-url","allow-save-project","allow-load-project","allow-attach-score-pdf","allow-read-score-pdf","allow-remove-score-pdf"]}} \ No newline at end of file +{"main-capability":{"identifier":"main-capability","description":"Capability for the main BandScope window to use the analysis orchestration commands.","local":true,"windows":["main"],"permissions":["core:event:allow-listen","core:event:allow-unlisten","allow-start-analysis-job","allow-get-analysis-job-status","allow-cancel-analysis-job","allow-select-local-audio-source","allow-import-youtube-url","allow-save-project","allow-load-project","allow-attach-score-pdf","allow-read-score-pdf","allow-remove-score-pdf"]}} \ No newline at end of file diff --git a/apps/desktop/src-tauri/permissions/autogenerated/cancel_analysis_job.toml b/apps/desktop/src-tauri/permissions/autogenerated/cancel_analysis_job.toml new file mode 100644 index 000000000..31119de7f --- /dev/null +++ b/apps/desktop/src-tauri/permissions/autogenerated/cancel_analysis_job.toml @@ -0,0 +1,11 @@ +# Automatically generated - DO NOT EDIT! + +[[permission]] +identifier = "allow-cancel-analysis-job" +description = "Enables the cancel_analysis_job command without any pre-configured scope." +commands.allow = ["cancel_analysis_job"] + +[[permission]] +identifier = "deny-cancel-analysis-job" +description = "Denies the cancel_analysis_job command without any pre-configured scope." +commands.deny = ["cancel_analysis_job"] diff --git a/apps/desktop/src-tauri/src/local_audio_publication.rs b/apps/desktop/src-tauri/src/local_audio_publication.rs new file mode 100644 index 000000000..855b6a9ce --- /dev/null +++ b/apps/desktop/src-tauri/src/local_audio_publication.rs @@ -0,0 +1,187 @@ +use std::{io, path::Path}; + +/// Commit a synchronized local-audio stage into its immutable project-owned name. +/// +/// The caller must have already synchronized `stage`. Both paths must be direct +/// children of the same app-owned project root. The operation is no-clobber on +/// every supported platform: Unix publishes with a hard link, removes the +/// private stage name, then synchronizes the project directory; Windows uses +/// `MoveFileExW` without `MOVEFILE_REPLACE_EXISTING` and with +/// `MOVEFILE_WRITE_THROUGH` so the namespace move crosses the platform's +/// write-through boundary before authority is returned. +/// +/// This barrier covers the publication mutation inside an already-existing +/// project directory. It does not claim durability for creation or replacement +/// of higher ancestors, nor can it override storage that falsely acknowledges a +/// completed flush. +pub(crate) fn commit_local_audio_publication( + stage: &Path, + destination: &Path, + project_root: &Path, +) -> io::Result<()> { + if stage.parent() != Some(project_root) || destination.parent() != Some(project_root) { + return Err(io::Error::new( + io::ErrorKind::InvalidInput, + "local-audio publication paths must be direct project children", + )); + } + + commit_local_audio_publication_platform(stage, destination, project_root) +} + +#[cfg(unix)] +fn commit_local_audio_publication_platform( + stage: &Path, + destination: &Path, + project_root: &Path, +) -> io::Result<()> { + std::fs::hard_link(stage, destination)?; + if let Err(error) = std::fs::remove_file(stage) { + let _ = std::fs::remove_file(destination); + return Err(error); + } + + if let Err(error) = sync_project_directory(project_root) { + let _ = std::fs::remove_file(destination); + let _ = sync_project_directory(project_root); + return Err(error); + } + + Ok(()) +} + +#[cfg(unix)] +fn sync_project_directory(project_root: &Path) -> io::Result<()> { + std::fs::File::open(project_root)?.sync_all() +} + +#[cfg(windows)] +fn commit_local_audio_publication_platform( + stage: &Path, + destination: &Path, + _project_root: &Path, +) -> io::Result<()> { + use std::{iter, os::windows::ffi::OsStrExt}; + + const MOVEFILE_WRITE_THROUGH: u32 = 0x0000_0008; + + #[link(name = "Kernel32")] + unsafe extern "system" { + fn MoveFileExW( + existing_file_name: *const u16, + new_file_name: *const u16, + flags: u32, + ) -> i32; + } + + let stage_wide: Vec = stage + .as_os_str() + .encode_wide() + .chain(iter::once(0)) + .collect(); + let destination_wide: Vec = destination + .as_os_str() + .encode_wide() + .chain(iter::once(0)) + .collect(); + + // SAFETY: both UTF-16 buffers are NUL-terminated and remain alive for the + // call. No replacement flag is supplied, so an existing destination fails + // closed. The paths are already restricted to direct children of one + // app-owned project directory by the public boundary above. + let moved = unsafe { + MoveFileExW( + stage_wide.as_ptr(), + destination_wide.as_ptr(), + MOVEFILE_WRITE_THROUGH, + ) + }; + if moved == 0 { + Err(io::Error::last_os_error()) + } else { + Ok(()) + } +} + +#[cfg(test)] +mod tests { + use super::commit_local_audio_publication; + use std::{fs, io, path::PathBuf}; + + fn test_root() -> PathBuf { + std::env::temp_dir().join(format!( + "bandscope-publication-{}", + uuid::Uuid::new_v4() + )) + } + + #[test] + fn publication_commits_staged_bytes_without_leaving_the_private_name() { + let root = test_root(); + fs::create_dir_all(&root).expect("test project root must be created"); + let stage = root.join(".source-test.stage"); + let destination = root.join("source.wav"); + fs::write(&stage, b"bandscope-real-file-boundary") + .expect("test stage must be written"); + fs::OpenOptions::new() + .write(true) + .open(&stage) + .expect("test stage must reopen") + .sync_all() + .expect("test stage must synchronize"); + + commit_local_audio_publication(&stage, &destination, &root) + .expect("publication commit must succeed"); + + assert!(!stage.exists(), "private stage name must be removed"); + assert_eq!( + fs::read(&destination).expect("published bytes must remain readable"), + b"bandscope-real-file-boundary" + ); + fs::remove_dir_all(&root).expect("test project root must be removed"); + } + + #[test] + fn publication_never_clobbers_an_existing_destination() { + let root = test_root(); + fs::create_dir_all(&root).expect("test project root must be created"); + let stage = root.join(".source-test.stage"); + let destination = root.join("source.wav"); + fs::write(&stage, b"new").expect("test stage must be written"); + fs::write(&destination, b"existing").expect("test destination must be written"); + + let error = commit_local_audio_publication(&stage, &destination, &root) + .expect_err("existing destination must fail closed"); + + assert_ne!(error.kind(), io::ErrorKind::InvalidInput); + assert_eq!( + fs::read(&destination).expect("existing destination must remain readable"), + b"existing" + ); + assert_eq!( + fs::read(&stage).expect("failed publication must retain its stage"), + b"new" + ); + fs::remove_dir_all(&root).expect("test project root must be removed"); + } + + #[test] + fn publication_rejects_paths_outside_the_project_root() { + let root = test_root(); + let other = test_root(); + fs::create_dir_all(&root).expect("test project root must be created"); + fs::create_dir_all(&other).expect("other test root must be created"); + let stage = other.join(".source-test.stage"); + let destination = root.join("source.wav"); + fs::write(&stage, b"new").expect("test stage must be written"); + + let error = commit_local_audio_publication(&stage, &destination, &root) + .expect_err("cross-root publication must fail closed"); + + assert_eq!(error.kind(), io::ErrorKind::InvalidInput); + assert!(stage.exists(), "rejected stage must remain untouched"); + assert!(!destination.exists(), "rejected destination must not be created"); + fs::remove_dir_all(&root).expect("test project root must be removed"); + fs::remove_dir_all(&other).expect("other test root must be removed"); + } +} diff --git a/apps/desktop/src-tauri/src/main.rs b/apps/desktop/src-tauri/src/main.rs index ed4f967bd..6ed56b93d 100644 --- a/apps/desktop/src-tauri/src/main.rs +++ b/apps/desktop/src-tauri/src/main.rs @@ -1,10 +1,13 @@ #![cfg_attr(not(debug_assertions), windows_subsystem = "windows")] +mod local_audio_publication; + use bandscope_desktop_core::*; +use local_audio_publication::commit_local_audio_publication; use rfd::FileDialog; use serde_json::{json, Value}; use std::{ - io::{BufRead, BufReader, Read, Write}, + io::Write, path::{Path, PathBuf}, process::{Command, Stdio}, sync::{atomic::Ordering, mpsc}, @@ -14,6 +17,49 @@ use std::{ use tauri::{Emitter, Manager, Runtime}; use time::{format_description::well_known::Rfc3339, OffsetDateTime}; +/// Native-only cache of verified local-audio publication identities. +/// +/// Security Notes: entries are keyed only by BandScope-minted project ids and +/// contain the bounded path-free publication evidence emitted by Resource +/// Admission. User filesystem paths are never retained in this state. +#[derive(Default)] +struct LocalAudioPublicationIdentityState( + std::sync::Mutex>, +); + +/// Native owner for job-specific cancellation requests. +/// +/// Security Notes: the renderer may request cancellation only by an already +/// minted BandScope job id. It never receives a PID or generic process handle; +/// the worker that owns the child remains the only code allowed to terminate it. +#[derive(Clone, Default)] +struct AnalysisJobCancellationRegistry( + std::sync::Arc>>, +); + +impl AnalysisJobCancellationRegistry { + fn request(&self, job_id: &str) -> bool { + self.0 + .lock() + .map(|mut requests| requests.insert(job_id.to_string())) + .unwrap_or(false) + } + + fn is_requested(&self, job_id: &str) -> bool { + self.0 + .lock() + .map(|requests| requests.contains(job_id)) + .unwrap_or(true) + } + + fn take_requested(&self, job_id: &str) -> bool { + self.0 + .lock() + .map(|mut requests| requests.remove(job_id)) + .unwrap_or(true) + } +} + fn iso_timestamp_now() -> String { OffsetDateTime::now_utc() .format(&Rfc3339) @@ -141,7 +187,25 @@ fn app_owned_root( Ok(root) } -fn normalize_local_audio_source(path: &Path) -> Result { +/// Admit one OS-selected local audio file into a project-owned immutable source artifact. +/// +/// Security Notes: the external path is used only to canonicalize and open the +/// user-authorized source. Size is checked from that opened descriptor, bytes +/// are copied through the bounded Resource Admission helper into a private +/// same-project staging file. After the stage is synchronized, publication uses +/// a platform-specific no-clobber durability boundary: Unix links the stage, +/// removes the private name, and synchronizes the project directory; Windows +/// performs a no-replace `MoveFileExW` with `MOVEFILE_WRITE_THROUGH`. Only then +/// is the published object re-opened and required to reproduce the staging +/// size+SHA-256 receipt before path-free bootstrap/persistence identity is +/// minted. This does not claim durability for creation or replacement of +/// higher directory ancestors. Atomic no-follow descriptor acquisition remains +/// a separate platform-hardening requirement. +fn materialize_local_audio_source( + path: &Path, + project_root: &Path, + project_id: &str, +) -> Result<(LocalAudioSourcePayload, LocalAudioPublicationIdentity), String> { let canonical = path .canonicalize() .map_err(|_| "Could not read the selected audio file.".to_string())?; @@ -153,22 +217,104 @@ fn normalize_local_audio_source(path: &Path) -> Result receipt, + Err(error) => { + drop(staged); + let _ = std::fs::remove_file(&stage); + return Err(error); + } + }; + if staged.sync_all().is_err() { + drop(staged); + let _ = std::fs::remove_file(&stage); + return Err("Could not prepare the local project workspace.".to_string()); + } + drop(staged); - Ok(LocalAudioSourcePayload { - source_path: canonical.to_string_lossy().into_owned(), - file_name: file_name.to_string(), - extension, - file_size_bytes: metadata.len(), - }) + if commit_local_audio_publication(&stage, &destination, project_root).is_err() { + let _ = std::fs::remove_file(&stage); + return Err("Could not prepare the local project workspace.".to_string()); + } + + let published_path_metadata = match std::fs::symlink_metadata(&destination) { + Ok(metadata) if metadata.is_file() && !metadata.file_type().is_symlink() => metadata, + _ => { + let _ = std::fs::remove_file(&destination); + return Err("Could not prepare the local project workspace.".to_string()); + } + }; + if published_path_metadata.len() != receipt.file_size_bytes { + let _ = std::fs::remove_file(&destination); + return Err("Could not prepare the local project workspace.".to_string()); + } + let published = match std::fs::File::open(&destination) { + Ok(file) => file, + Err(_) => { + let _ = std::fs::remove_file(&destination); + return Err("Could not prepare the local project workspace.".to_string()); + } + }; + let published_descriptor_metadata = match published.metadata() { + Ok(metadata) if metadata.is_file() && metadata.len() == receipt.file_size_bytes => metadata, + _ => { + drop(published); + let _ = std::fs::remove_file(&destination); + return Err("Could not prepare the local project workspace.".to_string()); + } + }; + if published_descriptor_metadata.len() != published_path_metadata.len() + || verify_local_audio_publication_receipt(published, &receipt).is_err() + { + let _ = std::fs::remove_file(&destination); + return Err("Could not prepare the local project workspace.".to_string()); + } + let published_path_metadata = match std::fs::symlink_metadata(&destination) { + Ok(metadata) if metadata.is_file() && !metadata.file_type().is_symlink() => metadata, + _ => { + let _ = std::fs::remove_file(&destination); + return Err("Could not prepare the local project workspace.".to_string()); + } + }; + if published_path_metadata.len() != receipt.file_size_bytes { + let _ = std::fs::remove_file(&destination); + return Err("Could not prepare the local project workspace.".to_string()); + } + + let publication_identity = + build_local_audio_publication_identity(project_id, &extension, &receipt)?; + Ok(( + LocalAudioSourcePayload { + source_path: destination.to_string_lossy().into_owned(), + file_name, + extension, + file_size_bytes: receipt.file_size_bytes, + }, + publication_identity, + )) } fn parse_request_payload(payload: Value) -> Result { @@ -283,6 +429,24 @@ fn failed_status( } } +fn cancelled_status(job_id: String, requested_at: String) -> AnalysisJobStatus { + AnalysisJobStatus { + job_id, + state: AnalysisJobState::Failed, + requested_at, + updated_at: iso_timestamp_now(), + progress_label: Some("Analysis cancelled".into()), + progress_stage: None, + progress_percent: None, + cache_status: None, + result: None, + error: Some(AnalysisJobError { + code: AnalysisJobErrorCode::Cancelled, + message: "Analysis was cancelled.".into(), + }), + } +} + fn store_status(state: &AppState, status: &AnalysisJobStatus) { if let Ok(mut jobs) = state.0.jobs.lock() { jobs.insert(status.job_id.clone(), status.clone()); @@ -298,12 +462,73 @@ fn store_status_and_emit( let _ = app.emit("analysis-job-updated", status); } +/// Commit one terminal job result while serializing late cancellation acceptance. +/// +/// Security Notes: `cancel_analysis_job` acquires the job-status lock before it +/// records a cancellation request. Holding that same lock while consuming the +/// request makes "accepted cancel" and terminal status publication one ordered +/// decision, so a renderer cannot receive a running acknowledgement and later +/// observe an unqualified success from the same job. +fn finalize_analysis_status_and_emit( + state: &AppState, + app: &tauri::AppHandle, + cancellation_state: &AnalysisJobCancellationRegistry, + finished: AnalysisJobStatus, +) { + let final_status = match state.0.jobs.lock() { + Ok(mut jobs) => { + let cancellation_requested = cancellation_state.take_requested(&finished.job_id); + let final_status = if cancellation_requested { + cancelled_status(finished.job_id.clone(), finished.requested_at.clone()) + } else { + finished.clone() + }; + jobs.insert(finished.job_id.clone(), final_status.clone()); + final_status + } + Err(_) => { + if cancellation_state.take_requested(&finished.job_id) { + cancelled_status(finished.job_id.clone(), finished.requested_at.clone()) + } else { + finished + } + } + }; + let _ = app.emit("analysis-job-updated", &final_status); +} + fn store_bootstrap_source(state: &AppState, summary: ProjectBootstrapSummaryPayload) { if let Ok(mut sources) = state.0.bootstrap_sources.lock() { sources.insert(summary.project_id.clone(), summary); } } +/// Retain path-free publication evidence before the renderer receives bootstrap authority. +fn store_local_audio_publication_identity( + state: &LocalAudioPublicationIdentityState, + identity: LocalAudioPublicationIdentity, +) -> Result<(), String> { + let project_id = identity.project_id.clone(); + let mut identities = state + .0 + .lock() + .map_err(|_| "Could not prepare the local project workspace.".to_string())?; + identities.insert(project_id, identity); + Ok(()) +} + +fn lookup_local_audio_publication_identity( + state: &LocalAudioPublicationIdentityState, + project_id: &str, +) -> Result { + state + .0 + .lock() + .ok() + .and_then(|identities| identities.get(project_id).cloned()) + .ok_or_else(|| "Analysis job source identity was not found. Choose local audio again.".to_string()) +} + fn lookup_bootstrap_source( state: &AppState, project_id: &str, @@ -317,6 +542,23 @@ fn lookup_bootstrap_source( .ok_or_else(|| "Analysis job source was not found. Choose local audio again.".to_string()) } +fn analysis_status_payload_is_valid(status: &AnalysisJobStatus) -> bool { + if status + .progress_percent + .is_some_and(|progress_percent| progress_percent > 100) + { + return false; + } + + match &status.state { + AnalysisJobState::Queued | AnalysisJobState::Running => { + status.result.is_none() && status.error.is_none() + } + AnalysisJobState::Succeeded => status.result.is_some() && status.error.is_none(), + AnalysisJobState::Failed => status.result.is_none() && status.error.is_some(), + } +} + fn drain_analysis_status_updates( state: &AppState, app: &tauri::AppHandle, @@ -332,10 +574,16 @@ fn drain_analysis_status_updates( fn run_analysis_engine( state: AppState, app: tauri::AppHandle, + cancellation_state: AnalysisJobCancellationRegistry, job_id: String, request: AnalysisJobRequest, + source_content_sha256: Option, requested_at: String, ) -> AnalysisJobStatus { + if cancellation_state.is_requested(&job_id) { + return cancelled_status(job_id, requested_at); + } + let (working_dir, program, mut args) = analysis_command(); if program == MISSING_ANALYSIS_PYTHON { @@ -348,14 +596,16 @@ fn run_analysis_engine( } args.push("--progress-jsonl".into()); - let mut process = match Command::new(program) + let mut command = Command::new(program); + command .args(args) .current_dir(working_dir) .stdin(Stdio::piped()) .stdout(Stdio::piped()) - .stderr(Stdio::piped()) - .spawn() - { + .stderr(Stdio::piped()); + configure_owned_process(&mut command); + + let mut process = match command.spawn() { Ok(process) => process, Err(_) => { return failed_status( @@ -367,13 +617,16 @@ fn run_analysis_engine( } }; - let payload = json!({ + let mut payload = json!({ "jobId": job_id.clone(), + "requestedAt": requested_at.clone(), "request": request, }); + if let Some(content_sha256) = source_content_sha256 { + payload["sourceContentSha256"] = Value::String(content_sha256); + } let Some(stdout) = process.stdout.take() else { - let _ = process.kill(); - let _ = process.wait(); + terminate_owned_process(&mut process); return failed_status( job_id, requested_at, @@ -382,8 +635,7 @@ fn run_analysis_engine( ); }; let Some(stderr) = process.stderr.take() else { - let _ = process.kill(); - let _ = process.wait(); + terminate_owned_process(&mut process); return failed_status( job_id, requested_at, @@ -392,37 +644,72 @@ fn run_analysis_engine( ); }; let (status_tx, status_rx) = mpsc::channel::(); + let (reader_failure_tx, reader_failure_rx) = mpsc::channel::<()>(); + let stdout_failure_tx = reader_failure_tx.clone(); + let stderr_failure_tx = reader_failure_tx.clone(); + let _reader_failure_guard = reader_failure_tx; + let expected_job_id = job_id.clone(); + let expected_requested_at = requested_at.clone(); let stdout_reader = thread::spawn(move || { - let reader = BufReader::new(stdout); let mut last_status = None; - for line in reader.lines() { - let Ok(line) = line else { - break; - }; - let trimmed = line.trim(); - if trimmed.is_empty() { - continue; + let mut protocol_rejected = false; + let mut terminal_status_seen = false; + let result = read_bounded_process_lines(stdout, |line| { + if protocol_rejected { + return; } - if let Ok(status) = serde_json::from_str::(trimmed) { - last_status = Some(status.clone()); - if status_tx.send(status).is_err() { - break; + match serde_json::from_str::(line) { + Ok(status) => { + if status.job_id != expected_job_id + || status.requested_at != expected_requested_at + || terminal_status_seen + || !analysis_status_payload_is_valid(&status) + { + protocol_rejected = true; + let _ = stdout_failure_tx.send(()); + return; + } + match &status.state { + AnalysisJobState::Succeeded | AnalysisJobState::Failed => { + terminal_status_seen = true; + last_status = Some(status); + } + AnalysisJobState::Running => { + let _ = status_tx.send(status); + } + _ => { + protocol_rejected = true; + let _ = stdout_failure_tx.send(()); + } + } + } + Err(_) => { + protocol_rejected = true; + let _ = stdout_failure_tx.send(()); } } + }); + let result = if protocol_rejected { + Err(std::io::Error::from(std::io::ErrorKind::InvalidData)) + } else { + result + }; + if result.is_err() { + let _ = stdout_failure_tx.send(()); } - last_status + (last_status, result) }); let stderr_reader = thread::spawn(move || { - let mut reader = stderr; - let mut buffer = Vec::new(); - let _ = reader.read_to_end(&mut buffer); - buffer + let result = read_bounded_process_output(stderr); + if result.is_err() { + let _ = stderr_failure_tx.send(()); + } + result }); if let Some(mut stdin) = process.stdin.take() { if stdin.write_all(payload.to_string().as_bytes()).is_err() { - let _ = process.kill(); - let _ = process.wait(); + terminate_owned_process(&mut process); let _ = stdout_reader.join(); let _ = stderr_reader.join(); return failed_status( @@ -442,15 +729,21 @@ fn run_analysis_engine( let exit_status; loop { drain_analysis_status_updates(&state, &app, &status_rx, &mut last_status); + if cancellation_state.is_requested(&job_id) { + terminate_owned_process(&mut process); + let _ = stdout_reader.join(); + let _ = stderr_reader.join(); + return cancelled_status(job_id, requested_at); + } match process.try_wait() { Ok(Some(status)) => { + terminate_owned_process(&mut process); exit_status = status; break; } Ok(None) => { if Instant::now() >= deadline { - let _ = process.kill(); - let _ = process.wait(); + terminate_owned_process(&mut process); let _ = stdout_reader.join(); let _ = stderr_reader.join(); return failed_status( @@ -463,11 +756,27 @@ fn run_analysis_engine( "Analysis engine timed out.", ); } - thread::sleep(ANALYSIS_WAIT_POLL); + let wait_for = std::cmp::min( + ANALYSIS_WAIT_POLL, + deadline.saturating_duration_since(Instant::now()), + ); + if reader_failure_rx.recv_timeout(wait_for).is_ok() { + terminate_owned_process(&mut process); + let _ = stdout_reader.join(); + let _ = stderr_reader.join(); + return failed_status( + payload["jobId"] + .as_str() + .unwrap_or("unknown-job") + .to_string(), + requested_at, + AnalysisJobErrorCode::EngineUnavailable, + "Analysis engine is unavailable.", + ); + } } Err(_) => { - let _ = process.kill(); - let _ = process.wait(); + terminate_owned_process(&mut process); let _ = stdout_reader.join(); let _ = stderr_reader.join(); return failed_status( @@ -482,12 +791,33 @@ fn run_analysis_engine( } } } - let reader_last_status = stdout_reader.join().unwrap_or(None); - let _ = stderr_reader.join(); - drain_analysis_status_updates(&state, &app, &status_rx, &mut last_status); - if last_status.is_none() { - last_status = reader_last_status; + let reader_last_status = match stdout_reader.join() { + Ok((last_status, Ok(()))) => last_status, + _ => { + return failed_status( + payload["jobId"] + .as_str() + .unwrap_or("unknown-job") + .to_string(), + requested_at, + AnalysisJobErrorCode::EngineUnavailable, + "Analysis engine is unavailable.", + ) + } + }; + if !matches!(stderr_reader.join(), Ok(Ok(_))) { + return failed_status( + payload["jobId"] + .as_str() + .unwrap_or("unknown-job") + .to_string(), + requested_at, + AnalysisJobErrorCode::EngineUnavailable, + "Analysis engine is unavailable.", + ); } + drain_analysis_status_updates(&state, &app, &status_rx, &mut last_status); + last_status = reader_last_status.or(last_status); if !exit_status.success() { return failed_status( @@ -501,8 +831,25 @@ fn run_analysis_engine( ); } - last_status.unwrap_or_else(|| { - failed_status( + match last_status { + Some(status) + if matches!( + &status.state, + AnalysisJobState::Succeeded | AnalysisJobState::Failed + ) => + { + status + } + Some(_) => failed_status( + payload["jobId"] + .as_str() + .unwrap_or("unknown-job") + .to_string(), + requested_at, + AnalysisJobErrorCode::EngineUnavailable, + "Analysis engine returned a non-terminal response.", + ), + None => failed_status( payload["jobId"] .as_str() .unwrap_or("unknown-job") @@ -510,8 +857,8 @@ fn run_analysis_engine( requested_at, AnalysisJobErrorCode::EngineUnavailable, "Analysis engine returned an invalid response.", - ) - }) + ), + } } #[tauri::command] @@ -519,6 +866,8 @@ fn start_analysis_job( request: Value, app: tauri::AppHandle, state: tauri::State<'_, AppState>, + publication_state: tauri::State<'_, LocalAudioPublicationIdentityState>, + cancellation_state: tauri::State<'_, AnalysisJobCancellationRegistry>, ) -> AnalysisJobStatus { let requested_at = iso_timestamp_now(); let mut parsed_request = match parse_request_payload(request) { @@ -533,6 +882,7 @@ fn start_analysis_job( } }; + let mut source_content_sha256 = None; if parsed_request.source_kind == "local_audio" { let Some(project_id) = parsed_request.project_id.clone() else { return failed_status( @@ -553,6 +903,33 @@ fn start_analysis_job( ) } }; + let publication_identity = + match lookup_local_audio_publication_identity(&publication_state, &project_id) { + Ok(identity) => identity, + Err(message) => { + return failed_status( + "invalid-job".into(), + requested_at, + AnalysisJobErrorCode::NotFound, + &message, + ) + } + }; + let source_artifact_name = Path::new(&bootstrap.source.source_path) + .file_name() + .and_then(|value| value.to_str()); + if publication_identity.file_size_bytes != bootstrap.source.file_size_bytes + || publication_identity.extension != bootstrap.source.extension + || source_artifact_name != Some(publication_identity.artifact_name.as_str()) + { + return failed_status( + "invalid-job".into(), + requested_at, + AnalysisJobErrorCode::NotFound, + "Analysis job source identity no longer matches the verified publication.", + ); + } + source_content_sha256 = Some(publication_identity.content_sha256); parsed_request.source_label = bootstrap.source.file_name.clone(); parsed_request.cache_root = Some(bootstrap.cache_root.clone()); parsed_request.temp_root = Some(bootstrap.temp_root.clone()); @@ -584,7 +961,19 @@ fn start_analysis_job( let app_state = state.inner().clone(); let worker_app_handle = app.clone(); + let worker_cancellation_state = cancellation_state.inner().clone(); std::thread::spawn(move || { + if worker_cancellation_state.is_requested(&job_id) { + let finished = cancelled_status(job_id.clone(), requested_at.clone()); + finalize_analysis_status_and_emit( + &app_state, + &worker_app_handle, + &worker_cancellation_state, + finished, + ); + release_job_slot(&app_state); + return; + } store_status_and_emit( &app_state, &worker_app_handle, @@ -604,11 +993,18 @@ fn start_analysis_job( let finished = run_analysis_engine( app_state.clone(), worker_app_handle.clone(), - job_id, + worker_cancellation_state.clone(), + job_id.clone(), parsed_request, + source_content_sha256, requested_at, ); - store_status_and_emit(&app_state, &worker_app_handle, &finished); + finalize_analysis_status_and_emit( + &app_state, + &worker_app_handle, + &worker_cancellation_state, + finished, + ); release_job_slot(&app_state); }); @@ -633,20 +1029,65 @@ fn get_analysis_job_status(job_id: String, state: tauri::State<'_, AppState>) -> }) } +#[tauri::command] +fn cancel_analysis_job( + job_id: String, + state: tauri::State<'_, AppState>, + cancellation_state: tauri::State<'_, AnalysisJobCancellationRegistry>, +) -> AnalysisJobStatus { + let jobs = match state.0.jobs.lock() { + Ok(jobs) => jobs, + Err(_) => { + return failed_status( + job_id, + iso_timestamp_now(), + AnalysisJobErrorCode::EngineUnavailable, + "Could not cancel the analysis job.", + ) + } + }; + let current = jobs.get(&job_id).cloned(); + let Some(current) = current else { + return failed_status( + job_id, + iso_timestamp_now(), + AnalysisJobErrorCode::NotFound, + "Analysis job was not found.", + ); + }; + + if !matches!(¤t.state, AnalysisJobState::Queued | AnalysisJobState::Running) { + return current; + } + if !cancellation_state.request(&job_id) && !cancellation_state.is_requested(&job_id) { + return failed_status( + job_id, + current.requested_at, + AnalysisJobErrorCode::EngineUnavailable, + "Could not cancel the analysis job.", + ); + } + + current +} + #[tauri::command] fn select_local_audio_source( app: tauri::AppHandle, state: tauri::State<'_, AppState>, + publication_state: tauri::State<'_, LocalAudioPublicationIdentityState>, ) -> Result { let path = FileDialog::new() .add_filter("Audio", &AUDIO_EXTENSIONS) .pick_file() .ok_or_else(|| "Choose a WAV, MP3, FLAC, or M4A file to start analysis.".to_string())?; - let source = normalize_local_audio_source(&path)?; let project_id = next_project_id(&state); let project_root = app_owned_root(&app, "projects", &project_id)?; let cache_root = app_owned_root(&app, "cache", &project_id)?; let temp_root = app_owned_root(&app, "temp", &project_id)?; + let (source, publication_identity) = + materialize_local_audio_source(&path, &project_root, &project_id)?; + store_local_audio_publication_identity(&publication_state, publication_identity)?; let summary = ProjectBootstrapSummaryPayload { project_id, @@ -712,6 +1153,7 @@ async fn import_youtube_url( if parsed.get("ok").and_then(|v| v.as_bool()) == Some(true) { if let Some(metadata) = parsed.get("metadata") { let source = youtube_source_from_metadata(metadata, &cache_root)?; + validate_local_audio_file_size(source.file_size_bytes)?; let summary = ProjectBootstrapSummaryPayload { project_id, @@ -826,7 +1268,9 @@ fn attach_score_pdf( /// Security Notes: no path crosses the IPC boundary. Both ids are validated /// against strict allowlist shapes, the path is rebuilt locally, and the /// canonicalize-plus-prefix guard in `resolve_existing_score_pdf` rejects any -/// escape from the app-owned scores root. +/// escape from the app-owned scores root. The resolved file is then read +/// through the bounded core helper so growth after attachment cannot trigger +/// an allocation beyond the 25 MiB product limit. #[tauri::command] fn read_score_pdf( project_id: String, @@ -838,7 +1282,7 @@ fn read_score_pdf( } let scores_root = scores_root_for_project(&app, &project_id)?; let path = resolve_existing_score_pdf(&scores_root, &score_id)?; - std::fs::read(path).map_err(|_| "Could not read the score PDF.".to_string()) + read_validated_score_pdf(&path) } /// Security Notes: same id validation and traversal guard as `read_score_pdf`; @@ -868,11 +1312,14 @@ fn remove_score_pdf( fn main() { tauri::Builder::default() .manage(AppState::default()) + .manage(LocalAudioPublicationIdentityState::default()) + .manage(AnalysisJobCancellationRegistry::default()) .invoke_handler(tauri::generate_handler![ select_local_audio_source, import_youtube_url, start_analysis_job, get_analysis_job_status, + cancel_analysis_job, save_project, load_project, attach_score_pdf, @@ -881,4 +1328,4 @@ fn main() { ]) .run(tauri::generate_context!()) .expect("error while running tauri application"); -} +} \ No newline at end of file diff --git a/apps/desktop/src-tauri/tests/analysis_job_cancellation_contract.rs b/apps/desktop/src-tauri/tests/analysis_job_cancellation_contract.rs new file mode 100644 index 000000000..b7e898f6c --- /dev/null +++ b/apps/desktop/src-tauri/tests/analysis_job_cancellation_contract.rs @@ -0,0 +1,280 @@ +fn capability_permissions_include(value: &serde_json::Value, expected: &str) -> bool { + value + .pointer("/main-capability/permissions") + .and_then(serde_json::Value::as_array) + .is_some_and(|permissions| { + permissions + .iter() + .any(|permission| permission.as_str() == Some(expected)) + }) +} + +fn desktop_schema_identifier_const(value: &serde_json::Value, expected: &str) -> bool { + value + .pointer("/definitions/Identifier/oneOf") + .and_then(serde_json::Value::as_array) + .is_some_and(|identifiers| { + identifiers.iter().any(|identifier| { + identifier + .get("const") + .and_then(serde_json::Value::as_str) + == Some(expected) + }) + }) +} + +#[test] +fn native_analysis_cancellation_contains_the_running_process_boundary() { + let source = include_str!("../src/main.rs"); + let runner_start = source + .find("fn run_analysis_engine(") + .expect("analysis process runner must remain present"); + let runner_tail = &source[runner_start..]; + let runner_end = runner_tail + .find("\n}\n\n#[tauri::command]\nfn start_analysis_job") + .expect("analysis process runner boundary must remain inspectable"); + let runner = &runner_tail[..runner_end]; + + assert!( + runner.contains("cancellation_state.is_requested(&job_id)"), + "the native runner must observe cancellation at the process boundary" + ); + assert!( + runner.contains("configure_owned_process(&mut command)"), + "the owned analysis process must use the shared core containment owner before spawn" + ); + assert!( + runner.contains("terminate_owned_process(&mut process)"), + "accepted cancellation and failure paths must use the shared core containment owner before reporting terminal status" + ); + assert!( + !runner.contains("process.kill()"), + "the runner must not bypass the containment helper with direct-child-only termination" + ); + assert!( + runner.contains("cancelled_status("), + "process cancellation must end in a dedicated machine-readable cancellation status" + ); +} + +#[test] +fn analysis_process_containment_uses_the_shared_core_owner() { + let source = include_str!("../src/main.rs"); + let core = include_str!("../../core/src/lib.rs"); + let core_runtime = include_str!("../../core/tests/youtube_process_containment.rs"); + let tauri_runtime = include_str!("youtube_process_containment_runtime.rs"); + let supported_guard = "#[cfg(any(target_os = \"linux\", target_os = \"macos\"))]"; + + assert!( + source.contains("configure_owned_process(&mut command)"), + "the Tauri analysis runner must delegate process-group configuration to desktop core" + ); + assert!( + source.contains("terminate_owned_process(&mut process)"), + "the Tauri analysis runner must delegate process termination to desktop core" + ); + assert!( + !source.contains("fn configure_analysis_process("), + "the Tauri adapter must not duplicate low-level process-group configuration" + ); + assert!( + !source.contains("fn terminate_analysis_process("), + "the Tauri adapter must not duplicate low-level process-group termination" + ); + assert!( + !source.contains("posix_kill(-process_group_id, SIGKILL)"), + "POSIX signalling belongs to the GUI-independent desktop-core owner" + ); + + assert!( + core.contains("pub fn configure_owned_process(command: &mut Command)"), + "desktop core must expose the canonical process configuration boundary" + ); + assert!( + core.contains("pub fn terminate_owned_process(child: &mut Child)"), + "desktop core must expose the canonical process termination boundary" + ); + assert!( + core.contains(&format!("{supported_guard}\n command.process_group(0);")), + "process-group setup must be enabled only where group termination is implemented" + ); + assert!( + core.contains("posix_kill(-process_group_id, SIGKILL)"), + "the canonical Unix owner must signal the owned process group" + ); + assert!( + core.contains("let _ = child.wait();"), + "the canonical owner must still reap the directly owned child" + ); + assert!( + core_runtime.contains(&format!( + "{supported_guard}\n#[test]\nfn youtube_timeout_terminates_descendant_that_keeps_output_pipe_open()" + )), + "the core descendant-containment runtime regression must run only on supported targets" + ); + assert!( + tauri_runtime.starts_with(supported_guard), + "the Tauri descendant-containment runtime regression must run only on supported targets" + ); +} + +#[test] +fn final_job_commit_serializes_cancellation_against_terminal_status() { + let source = include_str!("../src/main.rs"); + let finalizer_start = source + .find("fn finalize_analysis_status_and_emit<") + .expect("analysis worker needs one serialized terminal-status finalizer"); + let finalizer_tail = &source[finalizer_start..]; + let finalizer_end = finalizer_tail + .find("\n}\n\nfn store_bootstrap_source") + .expect("terminal-status finalizer boundary must remain inspectable"); + let finalizer = &finalizer_tail[..finalizer_end]; + + let jobs_lock = finalizer + .find("state.0.jobs.lock()") + .expect("finalization must hold the job-status lock"); + let cancellation_take = finalizer + .find("cancellation_state.take_requested(&finished.job_id)") + .expect("finalization must consume any accepted cancellation request"); + let terminal_store = finalizer + .find("jobs.insert(finished.job_id.clone(), final_status.clone())") + .expect("finalization must store exactly one terminal status while the job lock is held"); + + assert!( + jobs_lock < cancellation_take && cancellation_take < terminal_store, + "cancellation acceptance and terminal status publication must be serialized under the same job-status lock" + ); + + let worker_start = source + .find("let finished = run_analysis_engine(") + .expect("analysis worker must still delegate execution to the engine runner"); + let worker_tail = &source[worker_start..]; + let worker_end = worker_tail + .find("\n release_job_slot(&app_state);") + .expect("analysis worker must still release its in-flight slot"); + let worker_finalization = &worker_tail[..worker_end]; + + assert!( + worker_finalization.contains("finalize_analysis_status_and_emit("), + "the worker must route terminal publication through the serialized cancellation-aware finalizer" + ); + assert!( + !worker_finalization.contains("worker_cancellation_state.clear(&job_id)"), + "the worker must not clear cancellation before terminal status is serialized" + ); +} + +#[test] +fn queued_cancellation_uses_the_same_serialized_finalizer() { + let source = include_str!("../src/main.rs"); + let worker_start = source + .find("std::thread::spawn(move || {") + .expect("analysis worker must remain present"); + let worker_tail = &source[worker_start..]; + let running_publish = worker_tail + .find(" store_status_and_emit(\n &app_state,") + .expect("worker must still publish the running transition"); + let queued_cancel_path = &worker_tail[..running_publish]; + + assert!( + queued_cancel_path.contains("worker_cancellation_state.is_requested(&job_id)"), + "queued cancellation must still be observed before running work starts" + ); + assert!( + queued_cancel_path.contains("finalize_analysis_status_and_emit("), + "queued cancellation must use the same serialized terminal finalizer as running cancellation" + ); + assert!( + !queued_cancel_path.contains("worker_cancellation_state.clear(&job_id)"), + "queued cancellation must not clear the registry before terminal status is committed" + ); +} + +#[test] +fn cancel_command_holds_job_authority_while_accepting_a_request() { + let source = include_str!("../src/main.rs"); + let command_start = source + .find("fn cancel_analysis_job(") + .expect("job-specific cancellation command must remain present"); + let command_tail = &source[command_start..]; + let command_end = command_tail + .find("\n}\n\n#[tauri::command]\nfn select_local_audio_source") + .expect("cancellation command boundary must remain inspectable"); + let command = &command_tail[..command_end]; + + let jobs_guard = command + .find("let jobs = match state.0.jobs.lock()") + .expect("cancellation acceptance must retain the job-status lock guard"); + let request = command + .find("cancellation_state.request(&job_id)") + .expect("queued or running jobs must record a cancellation request"); + + assert!( + jobs_guard < request, + "the job-status lock must remain in scope until cancellation acceptance is recorded" + ); +} + +#[test] +fn cancellation_is_an_allowlisted_job_specific_tauri_command() { + let source = include_str!("../src/main.rs"); + + assert!( + source.contains("fn cancel_analysis_job("), + "the renderer boundary needs one job-specific cancellation command rather than generic process authority" + ); + assert!( + source.contains("cancel_analysis_job,\n save_project"), + "the cancellation command must be explicitly registered in the Tauri invoke handler" + ); + assert!( + source.contains("AnalysisJobCancellationRegistry::default()"), + "cancellation requests need a native state owner shared by commands and the worker" + ); + assert!( + source.contains("AnalysisJobErrorCode::Cancelled"), + "cancelled work must not be mislabeled as an unavailable engine" + ); +} + +#[test] +fn cancellation_is_declared_and_granted_in_the_tauri_capability_boundary() { + let build_manifest = include_str!("../build.rs"); + let main_capability = include_str!("../capabilities/main.json"); + let generated_permission = + include_str!("../permissions/autogenerated/cancel_analysis_job.toml"); + + assert!( + build_manifest.contains("\"cancel_analysis_job\""), + "the Tauri build manifest must generate a command permission for cancellation" + ); + assert!( + main_capability.contains("\"allow-cancel-analysis-job\""), + "the main window must be granted the generated cancellation permission" + ); + assert!( + generated_permission.contains("identifier = \"allow-cancel-analysis-job\"") + && generated_permission.contains("commands.allow = [\"cancel_analysis_job\"]"), + "the generated permission contract must bind only the cancellation command" + ); +} + +#[test] +fn generated_tauri_schemas_include_the_cancellation_permission() { + let generated_capabilities: serde_json::Value = + serde_json::from_str(include_str!("../gen/schemas/capabilities.json")) + .expect("generated capability snapshot must remain valid JSON"); + let desktop_schema: serde_json::Value = + serde_json::from_str(include_str!("../gen/schemas/desktop-schema.json")) + .expect("generated desktop schema must remain valid JSON"); + + assert!( + capability_permissions_include(&generated_capabilities, "allow-cancel-analysis-job"), + "the generated main capability must grant the cancellation permission in its permissions array" + ); + assert!( + desktop_schema_identifier_const(&desktop_schema, "allow-cancel-analysis-job") + && desktop_schema_identifier_const(&desktop_schema, "deny-cancel-analysis-job"), + "the generated desktop schema must enumerate the cancellation allow/deny identifiers as permission constants" + ); +} diff --git a/apps/desktop/src-tauri/tests/analysis_process_job_identity_contract.rs b/apps/desktop/src-tauri/tests/analysis_process_job_identity_contract.rs new file mode 100644 index 000000000..b39ed8f8e --- /dev/null +++ b/apps/desktop/src-tauri/tests/analysis_process_job_identity_contract.rs @@ -0,0 +1,45 @@ +#[test] +fn analysis_progress_cannot_rekey_the_native_job() { + let source = include_str!("../src/main.rs"); + let runner_start = source + .find("fn run_analysis_engine(") + .expect("analysis runner must remain present"); + let runner_end = source[runner_start..] + .find("\n#[tauri::command]\nfn start_analysis_job") + .map(|offset| runner_start + offset) + .expect("analysis runner must end before the start command"); + let runner = &source[runner_start..runner_end]; + + let stdout_reader_start = runner + .find("let stdout_reader = thread::spawn") + .expect("analysis stdout reader must remain present"); + let stdout_reader_end = runner[stdout_reader_start..] + .find("let stderr_reader = thread::spawn") + .map(|offset| stdout_reader_start + offset) + .expect("analysis stderr reader must follow stdout reader"); + let stdout_reader = &runner[stdout_reader_start..stdout_reader_end]; + + assert!( + runner[..stdout_reader_start].contains("let expected_job_id = job_id.clone()"), + "native analysis must capture the BandScope-minted job id before trusting helper progress" + ); + assert!( + stdout_reader.contains("status.job_id != expected_job_id"), + "helper JSONL for a different job id must fail closed instead of being stored or emitted" + ); + + let identity_guard = stdout_reader + .find("status.job_id != expected_job_id") + .expect("job-id guard must remain in the stdout reader"); + let status_publish = stdout_reader + .find("status_tx.send(status)") + .expect("valid helper progress must still reach the native status channel"); + assert!( + identity_guard < status_publish, + "job identity must be checked before helper progress can reach the native status channel" + ); + assert!( + stdout_reader[identity_guard..status_publish].contains("send(())"), + "job-id mismatch must wake the existing process-control owner rather than wait for helper exit" + ); +} diff --git a/apps/desktop/src-tauri/tests/analysis_process_output_admission_contract.rs b/apps/desktop/src-tauri/tests/analysis_process_output_admission_contract.rs new file mode 100644 index 000000000..0eb6bb333 --- /dev/null +++ b/apps/desktop/src-tauri/tests/analysis_process_output_admission_contract.rs @@ -0,0 +1,38 @@ +#[test] +fn analysis_process_streams_share_bounded_output_admission() { + let source = include_str!("../src/main.rs"); + let core_output = include_str!("../../core/src/process_output.rs"); + let runner_start = source + .find("fn run_analysis_engine(") + .expect("analysis process runner must remain present"); + let runner_tail = &source[runner_start..]; + let runner_end = runner_tail + .find("\n}\n\n#[tauri::command]\nfn start_analysis_job") + .expect("analysis process runner boundary must remain inspectable"); + let runner = &runner_tail[..runner_end]; + + assert!( + core_output.contains("pub fn read_bounded_process_output"), + "raw helper output admission must remain owned by desktop core" + ); + assert!( + core_output.contains("pub fn read_bounded_process_lines"), + "streaming line admission must remain owned by desktop core" + ); + assert!( + runner.contains("read_bounded_process_lines(stdout"), + "analysis stdout must use the shared bounded streaming owner before JSONL status parsing" + ); + assert!( + runner.contains("read_bounded_process_output(stderr"), + "analysis stderr must use the shared bounded byte owner instead of an unbounded Vec" + ); + assert!( + runner.contains("reader_failure_rx.recv_timeout"), + "analysis stream admission failure must wake the single process-control owner" + ); + assert!( + !runner.contains("reader.read_to_end(&mut buffer)"), + "analysis runner must not retain unbounded stderr before terminal publication" + ); +} diff --git a/apps/desktop/src-tauri/tests/analysis_process_progress_state_contract.rs b/apps/desktop/src-tauri/tests/analysis_process_progress_state_contract.rs new file mode 100644 index 000000000..81bb6c64b --- /dev/null +++ b/apps/desktop/src-tauri/tests/analysis_process_progress_state_contract.rs @@ -0,0 +1,189 @@ +#[test] +fn analysis_helper_cannot_publish_terminal_or_queued_state_before_native_exit() { + let source = include_str!("../src/main.rs"); + let runner_start = source + .find("fn run_analysis_engine(") + .expect("analysis process runner must remain present"); + let runner_end = source[runner_start..] + .find("\n#[tauri::command]\nfn start_analysis_job") + .map(|offset| runner_start + offset) + .expect("analysis process runner must end before the start command"); + let runner = &source[runner_start..runner_end]; + + let stdout_reader_start = runner + .find("let stdout_reader = thread::spawn") + .expect("analysis stdout reader must remain present"); + let stdout_reader_end = runner[stdout_reader_start..] + .find("let stderr_reader = thread::spawn") + .map(|offset| stdout_reader_start + offset) + .expect("analysis stderr reader must follow stdout reader"); + let stdout_reader = &runner[stdout_reader_start..stdout_reader_end]; + + assert!( + stdout_reader.contains("AnalysisJobState::Running"), + "only helper Running progress may be published before the owned process exits" + ); + assert!( + stdout_reader.contains("AnalysisJobState::Succeeded | AnalysisJobState::Failed"), + "helper terminal state must be recognized separately from streaming progress" + ); + + let running_guard = stdout_reader + .find("AnalysisJobState::Running") + .expect("running progress guard must remain present"); + let status_publish = stdout_reader + .find("status_tx.send(status)") + .expect("valid running progress must still reach the native status channel"); + assert!( + running_guard < status_publish, + "state must be validated before helper progress reaches the native status channel" + ); + + let terminal_guard = stdout_reader + .find("AnalysisJobState::Succeeded | AnalysisJobState::Failed") + .expect("terminal state guard must remain present"); + assert!( + terminal_guard < status_publish, + "terminal-state handling must be selected before any pre-exit status publication" + ); + assert!( + stdout_reader[terminal_guard..status_publish].contains("last_status = Some(status)"), + "terminal helper status must be retained for post-exit validation rather than emitted early" + ); + assert!( + stdout_reader.contains("_ =>") && stdout_reader.contains("send(())"), + "queued or otherwise invalid helper state must wake the existing process-control owner" + ); +} + +#[test] +fn malformed_analysis_jsonl_fails_closed_before_native_state_mutation() { + let source = include_str!("../src/main.rs"); + let runner_start = source + .find("fn run_analysis_engine(") + .expect("analysis runner must remain present"); + let runner_end = source[runner_start..] + .find("\n#[tauri::command]\nfn start_analysis_job") + .map(|offset| runner_start + offset) + .expect("analysis runner must end before the start command"); + let runner = &source[runner_start..runner_end]; + + let stdout_reader_start = runner + .find("let stdout_reader = thread::spawn") + .expect("analysis stdout reader must remain present"); + let stdout_reader_end = runner[stdout_reader_start..] + .find("let stderr_reader = thread::spawn") + .map(|offset| stdout_reader_start + offset) + .expect("analysis stderr reader must follow stdout reader"); + let stdout_reader = &runner[stdout_reader_start..stdout_reader_end]; + + assert!( + stdout_reader.contains("match serde_json::from_str::(line)"), + "analysis JSONL must handle deserialization failure explicitly instead of silently ignoring malformed protocol lines" + ); + assert!( + stdout_reader.contains("Err(_) =>") && stdout_reader.contains("stdout_failure_tx.send(())"), + "malformed analysis JSONL must wake the existing process-control owner and fail closed" + ); +} + +#[test] +fn analysis_status_payload_semantics_are_validated_before_native_state_mutation() { + let source = include_str!("../src/main.rs"); + let validator_start = source + .find("fn analysis_status_payload_is_valid(") + .expect("analysis status payload semantics must have one native validator"); + let validator_end = source[validator_start..] + .find("\nfn drain_analysis_status_updates(") + .map(|offset| validator_start + offset) + .expect("status payload validator must remain beside the native analysis protocol owner"); + let validator = &source[validator_start..validator_end]; + + assert!( + validator.contains("progress_percent") && validator.contains("> 100"), + "native analysis status admission must enforce the shared 0..=100 progress contract" + ); + assert!( + validator.contains("AnalysisJobState::Queued | AnalysisJobState::Running") + && validator.contains("status.result.is_none()") + && validator.contains("status.error.is_none()"), + "non-terminal helper status must not carry terminal result/error payloads" + ); + assert!( + validator.contains("AnalysisJobState::Succeeded") + && validator.contains("status.result.is_some()"), + "Succeeded helper status must carry a result before native acceptance" + ); + assert!( + validator.contains("AnalysisJobState::Failed") && validator.contains("status.error.is_some()"), + "Failed helper status must carry an error before native acceptance" + ); + + let runner_start = source + .find("fn run_analysis_engine(") + .expect("analysis runner must remain present"); + let runner_end = source[runner_start..] + .find("\n#[tauri::command]\nfn start_analysis_job") + .map(|offset| runner_start + offset) + .expect("analysis runner must end before the start command"); + let runner = &source[runner_start..runner_end]; + let stdout_reader_start = runner + .find("let stdout_reader = thread::spawn") + .expect("analysis stdout reader must remain present"); + let stdout_reader_end = runner[stdout_reader_start..] + .find("let stderr_reader = thread::spawn") + .map(|offset| stdout_reader_start + offset) + .expect("analysis stderr reader must follow stdout reader"); + let stdout_reader = &runner[stdout_reader_start..stdout_reader_end]; + + let semantic_guard = stdout_reader + .find("!analysis_status_payload_is_valid(&status)") + .expect("typed analysis status must pass semantic payload admission"); + let terminal_retention = stdout_reader + .find("last_status = Some(status)") + .expect("terminal candidate retention must remain present"); + let running_publish = stdout_reader + .find("status_tx.send(status)") + .expect("running progress publication must remain present"); + assert!( + semantic_guard < terminal_retention && semantic_guard < running_publish, + "semantic payload admission must happen before terminal retention or running status publication" + ); +} + +#[test] +fn analysis_protocol_rejection_survives_child_exit_race() { + let source = include_str!("../src/main.rs"); + let runner_start = source + .find("fn run_analysis_engine(") + .expect("analysis runner must remain present"); + let runner_end = source[runner_start..] + .find("\n#[tauri::command]\nfn start_analysis_job") + .map(|offset| runner_start + offset) + .expect("analysis runner must end before the start command"); + let runner = &source[runner_start..runner_end]; + + let stdout_reader_start = runner + .find("let stdout_reader = thread::spawn") + .expect("analysis stdout reader must remain present"); + let stdout_reader_end = runner[stdout_reader_start..] + .find("let stderr_reader = thread::spawn") + .map(|offset| stdout_reader_start + offset) + .expect("analysis stderr reader must follow stdout reader"); + let stdout_reader = &runner[stdout_reader_start..stdout_reader_end]; + + let rejection_to_join = stdout_reader + .rfind("if protocol_rejected") + .map(|offset| &stdout_reader[offset..]) + .expect("analysis stdout reader must retain explicit protocol rejection state"); + let reader_return = rejection_to_join + .find("(last_status, result)") + .expect("analysis stdout reader must return terminal candidate and transport result together"); + let rejection_result = &rejection_to_join[..reader_return]; + + assert!( + rejection_result.contains("std::io::ErrorKind::InvalidData") + && rejection_result.contains("Err("), + "protocol rejection must become a failing reader join result so a fast child exit cannot outrun the side-channel failure notification and admit a rejected terminal status" + ); +} diff --git a/apps/desktop/src-tauri/tests/analysis_process_requested_at_contract.rs b/apps/desktop/src-tauri/tests/analysis_process_requested_at_contract.rs new file mode 100644 index 000000000..2bddb9cb9 --- /dev/null +++ b/apps/desktop/src-tauri/tests/analysis_process_requested_at_contract.rs @@ -0,0 +1,45 @@ +#[test] +fn analysis_progress_cannot_replace_native_requested_at() { + let source = include_str!("../src/main.rs"); + let runner_start = source + .find("fn run_analysis_engine(") + .expect("analysis runner must remain present"); + let runner_end = source[runner_start..] + .find("\n#[tauri::command]\nfn start_analysis_job") + .map(|offset| runner_start + offset) + .expect("analysis runner must end before the start command"); + let runner = &source[runner_start..runner_end]; + + assert!( + runner.contains("\"requestedAt\": requested_at.clone()"), + "native analysis must pass its authoritative request timestamp to the helper envelope" + ); + assert!( + runner.contains("let expected_requested_at = requested_at.clone()"), + "native analysis must retain its request timestamp before reading helper progress" + ); + + let stdout_reader_start = runner + .find("let stdout_reader = thread::spawn") + .expect("analysis stdout reader must remain present"); + let stdout_reader_end = runner[stdout_reader_start..] + .find("let stderr_reader = thread::spawn") + .map(|offset| stdout_reader_start + offset) + .expect("analysis stderr reader must follow stdout reader"); + let stdout_reader = &runner[stdout_reader_start..stdout_reader_end]; + + let timestamp_guard = stdout_reader + .find("status.requested_at != expected_requested_at") + .expect("helper status must be bound to the native request timestamp"); + let status_publish = stdout_reader + .find("status_tx.send(status)") + .expect("valid helper progress must still reach the native status channel"); + assert!( + timestamp_guard < status_publish, + "requestedAt authority must be checked before helper progress can be stored or emitted" + ); + assert!( + stdout_reader[timestamp_guard..status_publish].contains("send(())"), + "requestedAt mismatch must wake the existing process-control owner and fail closed" + ); +} diff --git a/apps/desktop/src-tauri/tests/analysis_process_terminal_containment_contract.rs b/apps/desktop/src-tauri/tests/analysis_process_terminal_containment_contract.rs new file mode 100644 index 000000000..34719bc1f --- /dev/null +++ b/apps/desktop/src-tauri/tests/analysis_process_terminal_containment_contract.rs @@ -0,0 +1,47 @@ +#[test] +fn analysis_terminal_parent_exit_cleans_owned_descendants_before_reader_join() { + let source = include_str!("../src/main.rs"); + let core_runtime = include_str!("../../core/tests/youtube_process_containment.rs"); + let runner_start = source + .find("fn run_analysis_engine(") + .expect("analysis process runner must remain present"); + let runner_tail = &source[runner_start..]; + let runner_end = runner_tail + .find("\n}\n\n#[tauri::command]\nfn start_analysis_job") + .expect("analysis process runner boundary must remain inspectable"); + let runner = &runner_tail[..runner_end]; + + let terminal_arm_start = runner + .find("Ok(Some(status)) => {") + .expect("analysis runner must handle direct-parent terminal status"); + let terminal_arm_tail = &runner[terminal_arm_start..]; + let terminal_arm_end = terminal_arm_tail + .find("\n Ok(None) =>") + .expect("terminal-status arm must remain bounded before the running-process arm"); + let terminal_arm = &terminal_arm_tail[..terminal_arm_end]; + + assert!( + terminal_arm.contains("terminate_owned_process(&mut process)"), + "direct-parent terminal status must clean residual owned descendants before inherited stdout/stderr readers are joined" + ); + assert!( + terminal_arm.contains("exit_status = status;") && terminal_arm.contains("break;"), + "descendant cleanup must preserve the directly owned process exit status as product truth" + ); + + let cleanup = terminal_arm + .find("terminate_owned_process(&mut process)") + .expect("terminal cleanup must remain in the terminal arm"); + let terminal_reader_join = runner + .rfind("stdout_reader.join()") + .expect("analysis success path must still join its stdout reader after process control"); + assert!( + terminal_arm_start + cleanup < terminal_reader_join, + "owned descendant cleanup must happen before the analysis success path joins inherited output readers" + ); + + assert!( + core_runtime.contains("fn youtube_success_terminates_descendant_that_keeps_output_pipe_open()"), + "the shared process owner must retain an executable successful-parent descendant-pipe regression" + ); +} diff --git a/apps/desktop/src-tauri/tests/analysis_process_terminal_status_contract.rs b/apps/desktop/src-tauri/tests/analysis_process_terminal_status_contract.rs new file mode 100644 index 000000000..29ac133c9 --- /dev/null +++ b/apps/desktop/src-tauri/tests/analysis_process_terminal_status_contract.rs @@ -0,0 +1,30 @@ +#[test] +fn successful_analysis_process_requires_terminal_engine_status() { + let source = include_str!("../src/main.rs"); + let runner_start = source + .find("fn run_analysis_engine(") + .expect("analysis runner must remain present"); + let runner_end = source[runner_start..] + .find("\n#[tauri::command]\nfn start_analysis_job") + .map(|offset| runner_start + offset) + .expect("analysis runner must end before the start command"); + let runner = &source[runner_start..runner_end]; + let successful_exit_start = runner + .rfind("if !exit_status.success()") + .expect("analysis runner must reject a failed child exit"); + let successful_exit_completion = &runner[successful_exit_start..]; + + assert!( + successful_exit_completion.contains("AnalysisJobState::Succeeded") + && successful_exit_completion.contains("AnalysisJobState::Failed"), + "a zero exit code must not promote a queued/running progress record to terminal truth" + ); + assert!( + successful_exit_completion.contains("Analysis engine returned a non-terminal response."), + "a successful child exit without a terminal status must fail closed" + ); + assert!( + !successful_exit_completion.contains("last_status.unwrap_or_else"), + "the last progress record must not be accepted as completion merely because the child exited zero" + ); +} diff --git a/apps/desktop/src-tauri/tests/local_audio_publication_contract.rs b/apps/desktop/src-tauri/tests/local_audio_publication_contract.rs new file mode 100644 index 000000000..ea61483c1 --- /dev/null +++ b/apps/desktop/src-tauri/tests/local_audio_publication_contract.rs @@ -0,0 +1,150 @@ +#[test] +fn local_audio_materializer_consumes_publication_bound_receipt() { + let source = include_str!("../src/main.rs"); + let materializer_start = source + .find("fn materialize_local_audio_source(") + .expect("desktop materializer must remain present"); + let materializer_tail = &source[materializer_start..]; + let materializer_end = materializer_tail + .find("\n}\n\nfn parse_request_payload") + .expect("materializer boundary must remain inspectable"); + let materializer = &materializer_tail[..materializer_end]; + + assert!( + materializer.contains("copy_bounded_local_audio_with_receipt"), + "production materialization must retain native size+SHA-256 staging evidence" + ); + assert!( + materializer.contains("verify_local_audio_publication_receipt"), + "production materialization must re-read the published app-owned source and bind it to the staging receipt" + ); + assert!( + !materializer.contains("copy_bounded_local_audio(source"), + "the compatibility byte-count-only adapter must not remain on the production publication path" + ); +} + +#[test] +fn local_audio_publication_must_not_overwrite_an_existing_source_name() { + let source = include_str!("../src/main.rs"); + let materializer_start = source + .find("fn materialize_local_audio_source(") + .expect("desktop materializer must remain present"); + let materializer_tail = &source[materializer_start..]; + let materializer_end = materializer_tail + .find("\n}\n\nfn parse_request_payload") + .expect("materializer boundary must remain inspectable"); + let materializer = &materializer_tail[..materializer_end]; + + assert!( + materializer.contains("commit_local_audio_publication(&stage, &destination, project_root)"), + "publication must use the platform-correct no-clobber durable commit boundary" + ); + assert!( + !materializer.contains("destination.exists()"), + "a preflight existence check is racy and must not authorize a later overwrite-capable rename" + ); + assert!( + !materializer.contains("std::fs::rename(&stage, &destination)"), + "overwrite-capable portable rename must not publish the immutable project source" + ); +} + +#[test] +fn local_audio_materializer_commit_failure_must_preserve_existing_publication() { + let source = include_str!("../src/main.rs"); + let materializer_start = source + .find("fn materialize_local_audio_source(") + .expect("desktop materializer must remain present"); + let materializer_tail = &source[materializer_start..]; + let materializer_end = materializer_tail + .find("\n}\n\nfn parse_request_payload") + .expect("materializer boundary must remain inspectable"); + let materializer = &materializer_tail[..materializer_end]; + let commit_failure_start = materializer + .find("if commit_local_audio_publication(&stage, &destination, project_root).is_err()") + .expect("materializer must handle publication-commit failure explicitly"); + let commit_failure_tail = &materializer[commit_failure_start..]; + let commit_failure_end = commit_failure_tail + .find("\n }\n\n let published_path_metadata") + .expect("publication failure boundary must remain inspectable"); + let commit_failure = &commit_failure_tail[..commit_failure_end]; + + assert!( + commit_failure.contains("std::fs::remove_file(&stage)"), + "failed publication may clean up only the private stage it owns" + ); + assert!( + !commit_failure.contains("std::fs::remove_file(&destination)"), + "a no-clobber collision means destination may pre-exist; the materializer must never delete it on commit failure" + ); +} + +#[test] +fn local_audio_publication_commits_namespace_before_identity_authority() { + let source = include_str!("../src/main.rs"); + let materializer_start = source + .find("fn materialize_local_audio_source(") + .expect("desktop materializer must remain present"); + let materializer_tail = &source[materializer_start..]; + let materializer_end = materializer_tail + .find("\n}\n\nfn parse_request_payload") + .expect("materializer boundary must remain inspectable"); + let materializer = &materializer_tail[..materializer_end]; + + let durable_commit = materializer + .find("commit_local_audio_publication(&stage, &destination, project_root)") + .expect("publication must durably commit the project-owned namespace"); + let identity = materializer + .find("build_local_audio_publication_identity(project_id, &extension, &receipt)") + .expect("materializer must derive path-free publication identity"); + + assert!( + durable_commit < identity, + "bootstrap/persistence identity must not be minted before the publication namespace has crossed its platform durability barrier" + ); +} + +#[test] +fn local_audio_selection_retains_verified_path_free_identity_in_native_state() { + let source = include_str!("../src/main.rs"); + + assert!( + source.contains("struct LocalAudioPublicationIdentityState"), + "verified source identity must have a native-only state owner" + ); + assert!( + source.contains("build_local_audio_publication_identity(project_id, &extension, &receipt)"), + "the production materializer must derive persistence identity from the verified native receipt" + ); + assert!( + source.contains("store_local_audio_publication_identity(&publication_state, publication_identity)"), + "selection must retain native publication identity before returning bootstrap authority" + ); + assert!( + source.contains(".manage(LocalAudioPublicationIdentityState::default())"), + "the native publication identity state must be registered with the Tauri runtime" + ); +} + +#[test] +fn local_audio_analysis_handoff_consumes_verified_publication_identity() { + let source = include_str!("../src/main.rs"); + let start = source + .find("fn start_analysis_job(") + .expect("analysis start boundary must remain present"); + let tail = &source[start..]; + let end = tail + .find("\n}\n\n#[tauri::command]\nfn get_analysis_job_status") + .expect("analysis start boundary must remain inspectable"); + let analysis_start = &tail[..end]; + + assert!( + analysis_start.contains("lookup_local_audio_publication_identity"), + "local analysis must recover the verified native publication identity instead of relying on renderer-visible path/name/size metadata" + ); + assert!( + analysis_start.contains("source_content_sha256"), + "the analysis-engine handoff must carry the verified publication content digest for cache-generation binding" + ); +} diff --git a/apps/desktop/src-tauri/tests/youtube_process_containment_runtime.rs b/apps/desktop/src-tauri/tests/youtube_process_containment_runtime.rs new file mode 100644 index 000000000..27fa0b13c --- /dev/null +++ b/apps/desktop/src-tauri/tests/youtube_process_containment_runtime.rs @@ -0,0 +1,30 @@ +#[cfg(any(target_os = "linux", target_os = "macos"))] +#[test] +fn youtube_timeout_returns_after_terminating_descendant_pipe_holder() { + use bandscope_desktop_core::wait_for_process_output; + use std::{ + process::Command, + time::{Duration, Instant}, + }; + + let mut command = Command::new("sh"); + command.arg("-c").arg("sleep 5 & sleep 5"); + + let started = Instant::now(); + let result = wait_for_process_output( + command, + Duration::from_millis(50), + Duration::from_millis(5), + "YouTube import timed out.", + ); + let elapsed = started.elapsed(); + + assert_eq!( + result.expect_err("the import process should hit the product timeout"), + "YouTube import timed out." + ); + assert!( + elapsed < Duration::from_secs(1), + "timeout cleanup must release inherited output pipes promptly; elapsed={elapsed:?}" + ); +} diff --git a/apps/desktop/src/lib/analysis.audio-resource-next-action.test.ts b/apps/desktop/src/lib/analysis.audio-resource-next-action.test.ts new file mode 100644 index 000000000..8643ab84b --- /dev/null +++ b/apps/desktop/src/lib/analysis.audio-resource-next-action.test.ts @@ -0,0 +1,55 @@ +import { beforeEach, describe, expect, it, vi } from "vitest"; +import { + MAX_LOCAL_AUDIO_FILE_BYTES, + importYoutubeUrl, + selectLocalAudioSource +} from "./analysis"; + +type TauriWindow = Window & { + __TAURI_INTERNALS__?: unknown; + __TAURI_INVOKE__?: unknown; +}; + +const tauriWindow = window as TauriWindow; +const NEXT_ACTION = "Choose a shorter or smaller song file to start analysis."; + +function oversizedBootstrap(projectId: string) { + return { + projectId, + sourceMode: "reference", + projectRoot: `/tmp/bandscope/projects/${projectId}`, + cacheRoot: `/tmp/bandscope/cache/${projectId}`, + tempRoot: `/tmp/bandscope/temp/${projectId}`, + source: { + sourcePath: `/tmp/bandscope/${projectId}/input.wav`, + fileName: "input.wav", + extension: "wav", + fileSizeBytes: MAX_LOCAL_AUDIO_FILE_BYTES + 1 + } + }; +} + +describe("audio resource rejection next action", () => { + beforeEach(() => { + delete tauriWindow.__TAURI_INTERNALS__; + delete tauriWindow.__TAURI_INVOKE__; + }); + + it("names the next action for an oversized local selection", async () => { + tauriWindow.__TAURI_INVOKE__ = vi.fn().mockResolvedValue(oversizedBootstrap("local-project")); + + await expect(selectLocalAudioSource()).resolves.toEqual({ + ok: false, + error: { code: "invalid_request", message: NEXT_ACTION } + }); + }); + + it("names the same next action for an oversized imported selection", async () => { + tauriWindow.__TAURI_INVOKE__ = vi.fn().mockResolvedValue(oversizedBootstrap("youtube-project")); + + await expect(importYoutubeUrl("https://youtu.be/4ozX4yFUC34")).resolves.toEqual({ + ok: false, + error: { code: "invalid_request", message: NEXT_ACTION } + }); + }); +}); diff --git a/apps/desktop/src/lib/analysis.cancellation-bridge.test.ts b/apps/desktop/src/lib/analysis.cancellation-bridge.test.ts new file mode 100644 index 000000000..d7cb4bc2c --- /dev/null +++ b/apps/desktop/src/lib/analysis.cancellation-bridge.test.ts @@ -0,0 +1,93 @@ +import { beforeEach, describe, expect, it, vi } from "vitest"; +import { createDemoAnalysisJobRequest } from "@bandscope/shared-types"; +import { + cancelAnalysisJob, + getAnalysisJobStatus, + startAnalysisJob +} from "./analysis"; + +type TauriWindow = Window & { + __TAURI_INTERNALS__?: unknown; + __TAURI_INVOKE__?: unknown; +}; + +const tauriWindow = window as TauriWindow; + +describe("analysis cancellation bridge", () => { + beforeEach(() => { + delete tauriWindow.__TAURI_INTERNALS__; + delete tauriWindow.__TAURI_INVOKE__; + }); + + it("forwards a BandScope job id to the native cancellation command", async () => { + tauriWindow.__TAURI_INVOKE__ = vi.fn().mockResolvedValue({ + jobId: "job-42", + state: "running", + requestedAt: "2026-09-07T00:00:00.000Z", + updatedAt: "2026-09-07T00:00:01.000Z", + progressLabel: "Running analysis", + progressStage: "decode", + progressPercent: 10 + }); + + const status = await cancelAnalysisJob("job-42"); + + expect(tauriWindow.__TAURI_INVOKE__).toHaveBeenCalledWith("cancel_analysis_job", { + jobId: "job-42" + }); + expect(status).toMatchObject({ jobId: "job-42", state: "running" }); + }); + + it("rejects malformed native cancellation status through the shared parser boundary", async () => { + tauriWindow.__TAURI_INVOKE__ = vi.fn().mockResolvedValue({ + jobId: "job-malformed", + state: "cancelled", + requestedAt: "2026-09-07T00:00:00.000Z", + updatedAt: "2026-09-07T00:00:01.000Z" + }); + + await expect(cancelAnalysisJob("job-malformed")).rejects.toThrow( + "Invalid analysis job status response" + ); + }); + + it("keeps queued browser fallback cancellation terminal instead of later succeeding", async () => { + const queued = await startAnalysisJob(createDemoAnalysisJobRequest()); + + const acknowledged = await cancelAnalysisJob(queued.jobId); + expect(acknowledged).toMatchObject({ jobId: queued.jobId, state: "queued" }); + + const cancelled = await getAnalysisJobStatus(queued.jobId); + expect(cancelled).toMatchObject({ + jobId: queued.jobId, + state: "failed", + error: { + code: "cancelled", + message: "Analysis was cancelled." + } + }); + + await expect(getAnalysisJobStatus(queued.jobId)).resolves.toEqual(cancelled); + }); + + it("keeps running browser fallback cancellation terminal after a progress race", async () => { + const queued = await startAnalysisJob(createDemoAnalysisJobRequest()); + const running = await getAnalysisJobStatus(queued.jobId); + expect(running).toMatchObject({ jobId: queued.jobId, state: "running" }); + + const acknowledged = await cancelAnalysisJob(queued.jobId); + expect(acknowledged).toEqual(running); + + const cancelled = await getAnalysisJobStatus(queued.jobId); + expect(cancelled).toMatchObject({ + jobId: queued.jobId, + state: "failed", + error: { + code: "cancelled", + message: "Analysis was cancelled." + } + }); + + await expect(getAnalysisJobStatus(queued.jobId)).resolves.toEqual(cancelled); + }); +}); diff --git a/apps/desktop/src/lib/analysis.resource-policy.test.ts b/apps/desktop/src/lib/analysis.resource-policy.test.ts new file mode 100644 index 000000000..33ab81804 --- /dev/null +++ b/apps/desktop/src/lib/analysis.resource-policy.test.ts @@ -0,0 +1,58 @@ +import { beforeEach, describe, expect, it, vi } from "vitest"; +import { importYoutubeUrl, selectLocalAudioSource } from "./analysis"; + +type TauriWindow = Window & { + __TAURI_INTERNALS__?: unknown; + __TAURI_INVOKE__?: unknown; +}; + +const tauriWindow = window as TauriWindow; +const INVALID_RESOURCE_POLICY_MESSAGE = + "Selected audio file metadata violates the analysis resource policy."; + +function fractionalBootstrap(projectId: string) { + return { + projectId, + sourceMode: "reference", + projectRoot: `/tmp/bandscope/projects/${projectId}`, + cacheRoot: `/tmp/bandscope/cache/${projectId}`, + tempRoot: `/tmp/bandscope/temp/${projectId}`, + source: { + sourcePath: `/tmp/bandscope/${projectId}/input.wav`, + fileName: "input.wav", + extension: "wav", + fileSizeBytes: 1.5 + } + }; +} + +describe("analysis encoded-byte policy parity", () => { + beforeEach(() => { + delete tauriWindow.__TAURI_INTERNALS__; + delete tauriWindow.__TAURI_INVOKE__; + }); + + it("rejects fractional local-file metadata before project state", async () => { + tauriWindow.__TAURI_INVOKE__ = vi.fn().mockResolvedValue(fractionalBootstrap("local-project")); + + await expect(selectLocalAudioSource()).resolves.toEqual({ + ok: false, + error: { + code: "invalid_request", + message: INVALID_RESOURCE_POLICY_MESSAGE + } + }); + }); + + it("rejects fractional imported-file metadata before project state", async () => { + tauriWindow.__TAURI_INVOKE__ = vi.fn().mockResolvedValue(fractionalBootstrap("youtube-project")); + + await expect(importYoutubeUrl("https://youtu.be/4ozX4yFUC34")).resolves.toEqual({ + ok: false, + error: { + code: "invalid_request", + message: INVALID_RESOURCE_POLICY_MESSAGE + } + }); + }); +}); diff --git a/apps/desktop/src/lib/analysis.test.ts b/apps/desktop/src/lib/analysis.test.ts index e3347d1f5..4fb47f211 100644 --- a/apps/desktop/src/lib/analysis.test.ts +++ b/apps/desktop/src/lib/analysis.test.ts @@ -1,9 +1,11 @@ import { beforeEach, describe, expect, it, vi } from "vitest"; import { createDemoAnalysisJobRequest, createDemoRehearsalSong } from "@bandscope/shared-types"; import { + MAX_LOCAL_AUDIO_FILE_BYTES, MAX_YOUTUBE_URL_LENGTH, getAnalysisJobStatus, importYoutubeUrl, + selectLocalAudioSource, startAnalysisJob } from "./analysis"; @@ -13,6 +15,7 @@ type TauriWindow = Window & { }; const tauriWindow = window as TauriWindow; +const OVERSIZED_LOCAL_AUDIO_NEXT_ACTION = "Choose a shorter or smaller song file to start analysis."; describe("analysis bridge", () => { beforeEach(() => { @@ -20,6 +23,58 @@ describe("analysis bridge", () => { delete tauriWindow.__TAURI_INVOKE__; }); + it("rejects an oversized native local-audio selection before it becomes project state", async () => { + tauriWindow.__TAURI_INVOKE__ = vi.fn().mockResolvedValue({ + projectId: "native-local-project", + sourceMode: "reference", + projectRoot: "/tmp/bandscope/projects/native-local-project", + cacheRoot: "/tmp/bandscope/cache/native-local-project", + tempRoot: "/tmp/bandscope/temp/native-local-project", + source: { + sourcePath: "/tmp/bandscope/input.wav", + fileName: "input.wav", + extension: "wav", + fileSizeBytes: MAX_LOCAL_AUDIO_FILE_BYTES + 1 + } + }); + + const selection = await selectLocalAudioSource(); + + expect(selection).toEqual({ + ok: false, + error: { + code: "invalid_request", + message: OVERSIZED_LOCAL_AUDIO_NEXT_ACTION + } + }); + }); + + it("rejects an oversized native YouTube import before it becomes project state", async () => { + tauriWindow.__TAURI_INVOKE__ = vi.fn().mockResolvedValue({ + projectId: "native-youtube-project", + sourceMode: "reference", + projectRoot: "/tmp/bandscope/projects/native-youtube-project", + cacheRoot: "/tmp/bandscope/cache/native-youtube-project", + tempRoot: "/tmp/bandscope/temp/native-youtube-project", + source: { + sourcePath: "/tmp/bandscope/temp/native-youtube-project/youtube.wav", + fileName: "youtube.wav", + extension: "wav", + fileSizeBytes: MAX_LOCAL_AUDIO_FILE_BYTES + 1 + } + }); + + const selection = await importYoutubeUrl("https://youtu.be/4ozX4yFUC34"); + + expect(selection).toEqual({ + ok: false, + error: { + code: "invalid_request", + message: OVERSIZED_LOCAL_AUDIO_NEXT_ACTION + } + }); + }); + it("imports a standard YouTube URL through the browser fallback when Tauri is absent", async () => { const selection = await importYoutubeUrl("https://www.youtube.com/watch?v=4ozX4yFUC34"); @@ -99,6 +154,37 @@ describe("analysis bridge", () => { expect(selection.ok).toBe(true); }); + it.each([ + "Could not read the selected audio file.", + "Could not prepare the local project workspace.", + "Could not prepare the local cache workspace.", + "Could not prepare the local temp workspace." + ])("preserves an approved native local-audio string error: %s", async (message) => { + tauriWindow.__TAURI_INVOKE__ = vi.fn().mockRejectedValue(message); + + await expect(selectLocalAudioSource()).resolves.toEqual({ + ok: false, + error: { + code: "invalid_request", + message + } + }); + }); + + it("redacts an unapproved native local-audio string error", async () => { + tauriWindow.__TAURI_INVOKE__ = vi + .fn() + .mockRejectedValue("Could not read /Users/example/Music/private-demo.wav"); + + await expect(selectLocalAudioSource()).resolves.toEqual({ + ok: false, + error: { + code: "invalid_request", + message: "Choose a WAV, MP3, FLAC, or M4A file to start analysis." + } + }); + }); + it("normalizes legacy analysis job status responses before returning them", async () => { const legacyResult = createDemoRehearsalSong() as unknown as { sections: Array>; diff --git a/apps/desktop/src/lib/analysis.ts b/apps/desktop/src/lib/analysis.ts index bb750b34b..ce0c24ce3 100644 --- a/apps/desktop/src/lib/analysis.ts +++ b/apps/desktop/src/lib/analysis.ts @@ -35,8 +35,14 @@ const BROWSER_PROGRESS_STEPS = [ { progressLabel: "Saving reusable features", progressStage: "persist", progressPercent: 90 } ] as const; const UNSUPPORTED_LOCAL_AUDIO_MESSAGE = "Choose a WAV, MP3, FLAC, or M4A file to start analysis."; +const LOCAL_AUDIO_TOO_LARGE_MESSAGE = "Choose a shorter or smaller song file to start analysis."; +const LOCAL_AUDIO_POLICY_MESSAGE = + "Selected audio file metadata violates the analysis resource policy."; +const MAX_LOCAL_AUDIO_FILE_BYTES = 100 * 1024 * 1024; const SAFE_LOCAL_AUDIO_MESSAGES = new Set([ UNSUPPORTED_LOCAL_AUDIO_MESSAGE, + LOCAL_AUDIO_TOO_LARGE_MESSAGE, + LOCAL_AUDIO_POLICY_MESSAGE, "Could not read the selected audio file.", "Could not prepare the local project workspace.", "Could not prepare the local cache workspace.", @@ -45,7 +51,7 @@ const SAFE_LOCAL_AUDIO_MESSAGES = new Set([ const YOUTUBE_VIDEO_ID_PATTERN = /^[A-Za-z0-9_-]{11}$/; const MAX_YOUTUBE_URL_LENGTH = 2000; -export { MAX_YOUTUBE_URL_LENGTH }; +export { MAX_LOCAL_AUDIO_FILE_BYTES, MAX_YOUTUBE_URL_LENGTH }; /** Documented. */ export type LocalAudioSelectionResult = @@ -145,6 +151,9 @@ async function browserFallback(command: string, args?: Record): } }); } + if (existing.state === "failed" || existing.state === "succeeded") { + return existing; + } if (existing.state === "queued" || existing.state === "running") { const currentPercent = existing.progressPercent ?? 0; const nextStep = BROWSER_PROGRESS_STEPS.find((step) => step.progressPercent > currentPercent); @@ -176,6 +185,38 @@ async function browserFallback(command: string, args?: Record): return succeeded; } + if (command === "cancel_analysis_job") { + const jobId = String(args?.jobId ?? ""); + const existing = browserJobStore.get(jobId); + if (!existing) { + return createAnalysisJobStatus({ + jobId, + state: "failed", + error: { + code: "not_found", + message: "Analysis job was not found." + } + }); + } + if (existing.state === "failed" || existing.state === "succeeded") { + return existing; + } + browserJobStore.set( + jobId, + createAnalysisJobStatus({ + jobId, + state: "failed", + requestedAt: existing.requestedAt, + progressLabel: "Analysis cancelled", + error: { + code: "cancelled", + message: "Analysis was cancelled." + } + }) + ); + return existing; + } + if (command === "save_project") { return; } @@ -217,6 +258,26 @@ async function invokeAnalysis(command: string, args?: Record): return browserFallback(command, args); } +/** + * Parse a native/import bootstrap and enforce encoded-byte parity before the + * selection is allowed to become desktop project state. + * + * Python service and descriptor checks remain authoritative for analysis; this + * bridge check is defense in depth so local-file and imported-file intake fail + * at the same 100 MiB boundary instead of waiting for a later analysis stage. + */ +function parseBoundedAudioBootstrap(response: unknown): ProjectBootstrapSummary { + const bootstrap = parseProjectBootstrapSummary(response); + const fileSizeBytes = bootstrap.source.fileSizeBytes; + if (!Number.isSafeInteger(fileSizeBytes)) { + throw new Error(LOCAL_AUDIO_POLICY_MESSAGE); + } + if (fileSizeBytes > MAX_LOCAL_AUDIO_FILE_BYTES) { + throw new Error(LOCAL_AUDIO_TOO_LARGE_MESSAGE); + } + return bootstrap; +} + /** Documented. */ export function createDefaultAnalysisRequest(): AnalysisJobRequest { return createDemoAnalysisJobRequest(); @@ -228,7 +289,7 @@ export async function selectLocalAudioSource(): Promise { + const response = await invokeAnalysis("cancel_analysis_job", { jobId }); + try { + return parseAnalysisJobStatus(response); + } catch { + throw new Error("Invalid analysis job status response"); + } +} + /** Documented. */ export async function subscribeToAnalysisJobUpdates( jobId: string, @@ -328,7 +399,7 @@ export async function importYoutubeUrl(url: string): Promise one CLI dispatch -> one canonical analysis orchestration path. Existing CLI diagnostic configuration remains intact. Tests exercise supported orchestration behavior rather than patching the deleted pre-pass implementation. A future CLI-level preview or probe must be modeled as an explicit bounded domain operation with its own product status/resource contract rather than reintroducing a hidden pre-pass. \ No newline at end of file diff --git a/docs/doctoring/audio-resource-policy.md b/docs/doctoring/audio-resource-policy.md new file mode 100644 index 000000000..a19a35805 --- /dev/null +++ b/docs/doctoring/audio-resource-policy.md @@ -0,0 +1,64 @@ +# Audio resource policy evidence + +## Scope + +This note records the external evidence used by BandScope's versioned local-audio resource policy. It is implementation doctoring, not a claim that BandScope is certified against any external standard. + +The current Python policy v3 accepts at most 100 MiB of encoded local-audio input and at most 900 seconds of mono decoded audio at 44.1 kHz, which is 39,690,000 samples. The production decoder canonicalizes the admitted waveform to NumPy `float32`, so the canonical decoded-buffer ceiling is 158,760,000 bytes (39,690,000 samples × 4 bytes). Policy v1 derived that byte ceiling from `float64` and therefore allowed 317,520,000 bytes for the same sample count; v2 removed that byte-ceiling mismatch. Policy v3 closes the remaining representation gap by requiring the artifact admitted to MIR to have the canonical native NumPy `float32` dtype. A noncanonical artifact passed directly to policy validation fails closed with `decoded_dtype_unsupported`; the decode adapter may normalize a decoder-returned floating array to float32 only after its returned sample count and visible `nbytes` fit the policy, so a wide or oversized decoder result cannot trigger a second canonical-buffer allocation before rejection. If the decoder returns a non-owning NumPy view, BandScope detaches the admitted visible samples into a new owned canonical float32 array before MIR handoff. This prevents the returned artifact from retaining a larger hidden backing allocation through NumPy view ancestry without claiming that the decoder's transient backing allocation was itself bounded. Canonical finiteness validation then scans the PCM in chunks whose NumPy boolean mask is capped at 1 MiB, avoiding the previous full-song temporary mask while preserving rejection of NaN and positive/negative infinity. Request metadata is an early rejection hint only: the decoder boundary still checks the opened descriptor's actual size. After that byte extent is admitted, metadata parsing and decoding receive a seekable logical-EOF view fixed to the accepted byte count; `read`, `readinto`, and seek-from-end cannot expose bytes appended after admission. This closes post-admission growth as a resource-authority bypass without claiming that a privileged local actor cannot rewrite bytes in place within the admitted extent. Before any decode transformation, `soundfile.info` inspects that bounded already-open source view and the canonical policy rejects malformed headers, source rates below 8 kHz or above 192 kHz, source channel counts outside mono/stereo, and source duration beyond the path's limit; a successful probe rewinds the same bounded view. Decoder calls request one sample beyond the accepted duration and explicitly pass `dtype=np.float32` plus `res_type="soxr_hq"` to `librosa.load` along with the canonical target sample rate and mono downmix. Those arguments match librosa 0.11.0's documented defaults, but BandScope pins them because decoded dtype and resampler behavior are part of the canonical MIR input representation and must not drift with a dependency-default change. Librosa documents an `np.ndarray` return with shape and dtype semantics but does not promise NumPy `OWNDATA`; BandScope therefore accepts a valid non-owning decoder result only after its visible sample/byte bounds pass, then copies it into owned canonical PCM rather than treating ownership as a third-party API precondition. The resulting artifact is revalidated as a one-dimensional, non-empty, finite native float32 array at exactly the target rate and within both the accepted sample count and decoded-buffer byte budget before beat tracking, transcription, or Demucs inference. Policy construction also rejects byte, memory, rate, duration, or derived sample-count limits that cannot be represented within the host's bounded integer/sample-count model, so extreme integer configuration cannot escape through Python-to-float conversion overflow. YouTube import uses the same encoded-byte ceiling: yt-dlp `max_filesize`, a progress hook that aborts once `downloaded_bytes` / `total_bytes` / `total_bytes_estimate` exceed 100 MiB, a pre-download reject on announced `filesize` / `filesize_approx`, and a post-download `AudioResourcePolicy` check that deletes the artifact. Before post-download duration/size checks, cleanup, or success metadata can use the yt-dlp result, the completed path is canonicalized and required to remain strictly beneath the current import `out_dir`; a foreign or escaped path fails closed without being deleted. Native local-file and YouTube bootstrap then re-check the filesystem-observed length before storing project state. + +The native Tauri analysis runner now also narrows cancellation authority at the OS process boundary. On Linux and macOS, Rust's stable Unix `CommandExt::process_group(0)` is applied before spawn, so the Python analysis engine becomes leader of a fresh process group before `exec`. Cancellation, timeout, missing-pipe, stdin-write, and process-wait failure paths call one termination helper. That helper uses a narrow C ABI binding to POSIX `kill()` with a negative process-group id and `SIGKILL`, then reaps the directly owned child; if group signalling fails, it falls back to direct-child kill/reap. This covers ordinary descendants that retain the inherited process group. It does not claim containment for descendants that deliberately change group/session, and it does not claim Windows descendant containment: Windows still requires a race-free Job Object creation/assignment boundary. + +## Evidence-to-control mapping + +| Evidence | BandScope control | +| --- | --- | +| CWE-770 recommends explicit minimum/maximum resource expectations and limiting resources reachable by unprivileged actors. | `AudioResourcePolicy` makes encoded bytes, decoded samples, decoded mono-buffer bytes, sample rate, exact canonical dtype, shape, finiteness, and checked limit arithmetic explicit fail-closed invariants. The decode adapter rejects an over-budget visible decoder allocation before float32 normalization and detaches admitted non-owning views into an owned bounded artifact so hidden backing memory is not retained after handoff. Finiteness revalidation caps each temporary NumPy boolean mask at 1 MiB rather than allocating a mask proportional to the full admitted song. CWE-770 is the more specific mapping-friendly weakness beneath the broader CWE-400 resource-consumption class. | +| OWASP ASVS v5.0.0-5.1.1 requires file-handling documentation to define accepted types/extensions and maximum size; v5.0.0-5.2.1 requires accepting only file sizes that can be processed without performance loss or denial of service. | BandScope documents and enforces a finite encoded-byte ceiling before decode, while retaining authoritative descriptor checks at the actual file boundary. ASVS targets web applications/services, so BandScope uses these requirements as security-engineering guidance rather than claiming ASVS conformance for the desktop product. | +| librosa 0.11.0 documents `load(..., duration=..., dtype=np.float32, res_type="soxr_hq")`; its default `soxr_hq` mode is the high-quality band-limited SoXR resampler. The API documents an `np.ndarray` result but does not specify an ownership flag contract. | Temporal analysis and stem separation explicitly request canonical `float32` decode and `soxr_hq` resampling instead of inheriting mutable third-party defaults, while still requesting `max_duration + one sample` as a probe. The returned decoder array is checked for sample count and visible bytes before float32 normalization. A non-owning valid view is copied into an owned canonical array rather than rejected solely because a third-party API does not promise `OWNDATA`; the canonical artifact is then revalidated for exact decoded sample count, dtype, finiteness and in-memory byte size. The post-decode checks remain authoritative because decoder arguments alone are not treated as proof of resource-policy compliance. | +| NumPy documents `ndarray.nbytes` as the bytes consumed by an array's elements and exposes memory ownership through array flags/base relationships; `numpy.require(..., requirements=['O'])` documents that ownership can be ensured by copying when needed. | BandScope treats visible `nbytes` as the artifact-size gate, not as proof that a decoder view owns all memory it references. When ownership is absent, the admitted visible samples are copied to a new owned canonical float32 artifact, while the pre-copy decoder allocation remains part of separate peak-RSS acceptance. | +| python-soundfile 0.13.1 documents `soundfile.info(file)` for file-like objects and its virtual-I/O path relies on bounded read/seek/tell behavior rather than a path reopen. | `audio_metadata.preflight_audio_metadata` uses the bounded already-open source view for source metadata admission, applies the shared rate/channel/duration policy, and rewinds that same view before `librosa.load`. The decode adapter fixes logical EOF to the byte extent accepted before parsing, including `readinto` and `SEEK_END`, so later source growth cannot expand parser/decoder authority. Parser and rewind failures become the canonical payload-free policy error. | +| yt-dlp documents `max_filesize` as a FileDownloader parameter that skips files larger than the configured byte count, and `progress_hooks` as callbacks that receive `downloaded_bytes`, `total_bytes`, and `total_bytes_estimate`. | YouTube import sets `max_filesize` to `DEFAULT_MAX_ENCODED_FILE_BYTES`, aborts from the progress hook when those byte fields exceed the ceiling, validates that the completed path remains inside the per-import output directory, and still revalidates the written file with the canonical policy so a missing or lying size announcement cannot bypass admission. | +| yt-dlp `FragmentFD._download_fragment` constructs each fragment filename as `'%s-Frag%d' % (ctx['tmpfilename'], ctx['fragment_index'])`; the fragment index is therefore serialized with the ASCII decimal `%d` formatter. | BandScope cleanup grants transient deletion authority only to terminal `-Frag[0-9]+` / `-Frag[0-9]+.part` suffixes. Python's broader Unicode-aware `str.isdigit()` and `\d` semantics are deliberately not used for this authority boundary, so Unicode digit lookalikes cannot be mistaken for yt-dlp fragment names. | +| Tauri 2 runtime authority checks a WebView invoke against registered commands, capabilities, permissions, and scopes before the command is dispatched; application commands declared through `AppManifest::commands` receive generated allow/deny command permissions that are then granted through a capability. | `cancel_analysis_job` is declared in the desktop `AppManifest`, has a generated `allow-cancel-analysis-job` permission, and that permission is explicitly granted only through the main application capability. The renderer still sends only a BandScope job id; PID/process authority never crosses the IPC boundary. | +| Rust 1.98.1 exposes stable Unix `CommandExt::process_group(0)` so a spawned child can become leader of a new process group before `exec`. POSIX.1-2024 `kill()` defines a negative pid other than -1 as signalling processes whose process-group id is the absolute value of that pid. Rust's standard-library whole-group kill convenience remains unstable, so a nightly-only API is not an admissible production dependency. | On Linux/macOS, BandScope configures the analysis `Command` with `process_group(0)` before spawn and uses a reviewed narrow C ABI `kill()` binding to signal `-process.id()` with `SIGKILL`. The renderer cannot supply the pid/group id. The direct child is still reaped before terminal cancellation is returned, and direct-child kill/reap remains the fail-closed fallback if group signalling fails. The guarantee is deliberately limited to descendants that remain in the inherited group. | +| Windows Job Objects associate child processes with the job by default once the parent process is assigned, and `JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE` can terminate the processes associated with that job when its final handle is closed. `AssignProcessToJobObject` applies limits only after assignment. | The Windows containment boundary must own a dedicated analysis Job Object and close or terminate that job on cancellation. A post-spawn assignment is not treated as race-free proof if the Python process can create descendants before assignment; if ordinary `std::process::Command` cannot establish the job before execution, BandScope must use a creation path that prevents that pre-assignment window rather than claiming process-tree containment. | + +## Residual risk and follow-up + +Policy v3 bounds Python source-container admission, decode/model entry by decoded sample count, exact canonical float32 dtype, and the admitted canonical mono-buffer memory, plus native local-file bootstrap and YouTube download/bootstrap encoded-byte admission. In-flight YouTube abort also deletes owned `.part`, `.ytdl`, and ASCII-indexed `-Frag[0-9]+` / `-Frag[0-9]+.part` siblings that stay inside that import's output directory; paths that escape the directory are ignored. The completed-path containment check is a point-in-time canonical path check and does not claim descriptor/handle-level race freedom if a privileged local actor replaces filesystem entries after validation. The bounded Python decoder view closes growth beyond the accepted encoded EOF, but it is not a content-integrity snapshot: a privileged actor that can rewrite bytes inside the already-admitted range remains outside this narrow guarantee. Buyer workflows rely on BandScope's app-owned immutable project publication identity for that source-integrity boundary rather than treating one size observation as content identity. + +The production Tauri analysis boundary has a job-specific `cancel_analysis_job` command and a native cancellation registry. The command is part of Tauri's application manifest, the generated `allow-cancel-analysis-job` permission is granted by the main capability, and an exact contract test binds all three layers so registering the handler without granting runtime authority cannot silently ship a renderer-visible cancellation control that is denied at runtime. The WebView can request cancellation only by a BandScope-minted job id; no PID or generic process handle crosses IPC. A queued request is converted to the typed terminal `cancelled` error before the engine starts. Cancellation acceptance and terminal result publication are serialized: `cancel_analysis_job` retains the shared job-status lock while it records an accepted request, and the worker finalizer takes that same lock before consuming the request and storing exactly one terminal status. If cancellation wins that ordering, the finalizer publishes typed `cancelled`; if terminal publication wins first, the command observes a terminal job and does not accept cancellation. The in-flight slot is released after the serialized terminal publication. + +For running analysis on Linux and macOS, the process lifecycle is now stronger than the previous direct-child-only boundary. The worker establishes a fresh process group before `exec`, and cancellation/timeout/error cleanup targets the group before waiting for the directly owned Python child. This is appropriate for ordinary decoder/model helper descendants that inherit the parent's group. It is still narrower than sandbox/container containment: a child that intentionally calls `setsid()` or moves to another process group can leave this boundary, and the current implementation has not yet measured whether inherited pipes, child-created temp files, or process resources return within an acceptable latency on rights-cleared full-length audio. Windows remains direct-child-only in this slice; a race-free Job Object path is required before a Windows process-tree claim. + +This is also not an end-to-end resource-budget claim. The returned decoder array is checked against sample and visible-byte budgets before canonicalization; a valid non-owning view is detached into owned PCM so the returned artifact does not retain its decoder backing allocation; and the policy's own finiteness scan bounds its temporary boolean mask to 1 MiB. `librosa.load` can still allocate decode/resample intermediates and a large backing store before it returns, and the bounded canonical copy may temporarily coexist with that decoder allocation. Downstream temporary arrays, PyTorch tensors, model weights and accelerator allocations remain outside the 158,760,000-byte artifact ceiling. Remaining #781 work therefore includes the race-free Windows Job Object boundary; buyer-visible accessible cancellation; rights-cleared full-length rehearsal-audio measurement of cancellation latency, process/pipe/handle/temp cleanup and peak RSS/VRAM; explicit per-job CPU/GPU budgets; and whole-product CPU/GPU parity evidence. + +## References + +librosa development team. (2025). *librosa.load (librosa 0.11.0)* [Documentation]. https://librosa.org/doc/0.11.0/generated/librosa.load.html + +NumPy Developers. (2026). *numpy.ndarray and array ownership/requirements* [Documentation]. https://numpy.org/doc/stable/reference/generated/numpy.ndarray.html ; https://numpy.org/doc/stable/reference/generated/numpy.require.html + +python-soundfile contributors. (2025). *python-soundfile 0.13.1: `soundfile.info` and virtual I/O* [Documentation]. https://python-soundfile.readthedocs.io/en/latest/ + +MITRE Corporation. (2026, April 30). *CWE-770: Allocation of resources without limits or throttling (Version 4.20).* Common Weakness Enumeration. https://cwe.mitre.org/data/definitions/770.html + +OWASP Foundation. (2025, May). *OWASP Application Security Verification Standard 5.0.0.* https://github.com/OWASP/ASVS/tree/v5.0.0_release/5.0 + +Rust Project Developers. (2026). *CommandExt in std::os::unix::process (Rust 1.98.1)* [Documentation]. https://doc.rust-lang.org/stable/std/os/unix/process/trait.CommandExt.html + +IEEE & The Open Group. (2024). *The Open Group Base Specifications Issue 8 / IEEE Std 1003.1-2024: `kill()`.* https://pubs.opengroup.org/onlinepubs/9799919799/functions/kill.html + +Microsoft. (n.d.). *Job Objects.* Microsoft Learn. Retrieved September 7, 2026, from https://learn.microsoft.com/en-us/windows/win32/procthread/job-objects + +Microsoft. (2021, October 12). *AssignProcessToJobObject function (jobapi2.h).* Microsoft Learn. https://learn.microsoft.com/en-us/windows/win32/api/jobapi2/nf-jobapi2-assignprocesstojobobject + +Tauri Programme within The Commons Conservancy. (2026). *Permissions.* Tauri v2. https://v2.tauri.app/security/permissions/ + +Tauri Programme within The Commons Conservancy. (2025). *Runtime authority.* Tauri v2. https://v2.tauri.app/security/runtime-authority/ + +yt-dlp contributors. (2026). *FileDownloader parameters (`max_filesize`)* [Source documentation]. https://github.com/yt-dlp/yt-dlp/blob/master/yt_dlp/downloader/common.py + +yt-dlp contributors. (2026). *YoutubeDL `progress_hooks`* [Source documentation]. https://github.com/yt-dlp/yt-dlp/blob/master/yt_dlp/YoutubeDL.py + +yt-dlp contributors. (2026). *FragmentFD fragment filename construction (`FragmentFD._download_fragment`)* [Source code]. https://github.com/yt-dlp/yt-dlp/blob/master/yt_dlp/downloader/fragment.py diff --git a/docs/doctoring/feature-cache-archive-path-admission.md b/docs/doctoring/feature-cache-archive-path-admission.md new file mode 100644 index 000000000..8e28f678d --- /dev/null +++ b/docs/doctoring/feature-cache-archive-path-admission.md @@ -0,0 +1,43 @@ +# Feature-cache archive path admission + +## Problem + +Feature-cache metadata already opens an untrusted persisted sidecar with bounded, non-blocking/no-follow descriptor semantics where the platform exposes them. The NPZ archive path did not: `Path.open("rb")` resolved the pathname first and only then used `fstat()` to check that the resulting descriptor was a bounded regular file. + +That ordering let a local cache-path substitution redirect replay through a symbolic link to another regular NPZ. On Unix-like systems a FIFO at the archive pathname can also block during ordinary read-only open before the regular-file admission check runs. Both cases violate the persistence boundary: an app-owned cache path is not sufficient authority after crash recovery, restore, or local modification. + +## Decision + +The archive owner now opens the NPZ with `os.open()` using read-only plus `O_NONBLOCK` and `O_NOFOLLOW` when those constants are available, then performs `fstat()` and all size/type checks on that same descriptor before copying its admitted extent into the private spooled snapshot. + +The realistic regression uses a valid external NPZ as a symbolic-link target while keeping a valid BandScope sidecar at the cache pathname. On platforms exposing `O_NOFOLLOW`, replay must return a cache miss rather than materialize that target. The test is skipped where the runtime does not expose the flag; the production code keeps the existing portable `getattr(..., 0)` boundary instead of claiming identical filesystem semantics on every OS. + +RED: `1038bdbfec8606901561e0924050de55950b4403`. + +Production: `4ca8d02d856e0c13d9b2322268c6d79a8492d4e2`. + +## Alternatives considered + +Resolving the symlink and checking whether its target remains below the cache root was rejected. It retains a pathname-resolution race and would expand cache admission into path-containment policy even though replay only needs one already-open regular file. + +Calling `Path.resolve()` before `open()` was rejected for the same reason: a check on one pathname resolution does not make a later open operate on the same filesystem object. + +Removing `O_NONBLOCK` because it does not change ordinary regular-file reads was rejected. Persisted cache paths are untrusted at open time; on Unix, named-pipe open behavior can block before `fstat()` can reject the object. The flag is therefore a pre-admission liveness control, not a regular-file performance option. + +## Claim boundary + +This repair rejects a trailing-component symbolic link where `O_NOFOLLOW` is available and prevents ordinary FIFO open from becoming a blocking replay input where `O_NONBLOCK` is available. It does not provide Linux `openat2(2)`-style resolution constraints for every parent directory component, does not cryptographically bind the metadata and NPZ snapshots to one generation, and does not create durable source-content identity. The versioned immutable manifest remains separate work and must consume Project Persistence source identity after that owner reaches protected ancestry. + +## Traceability + +- CWE-59 — Improper Link Resolution Before File Access ('Link Following'): the weakness applies when a pathname associated with a symbolic link is used without ensuring that resolution stays within the intended resource policy. +- Python 3.14 `os` documentation: `O_NONBLOCK` is Unix-only, while `O_NOFOLLOW` is an extension exposed only when the underlying C library provides it; BandScope therefore feature-detects both flags. +- Linux `open(2)`: `O_NOFOLLOW` fails when the trailing pathname component is a symbolic link, and `O_NONBLOCK` avoids ordinary blocking open semantics where applicable, including FIFO handling. + +## References + +MITRE. (2026). *CWE-59: Improper link resolution before file access ('link following') (Version 4.20).* Common Weakness Enumeration. https://cwe.mitre.org/data/definitions/59.html + +Python Software Foundation. (2026). *os — Miscellaneous operating system interfaces (Python 3.14.7 documentation).* https://docs.python.org/3.14/library/os.html + +Kerrisk, M., & Linux man-pages contributors. (2026). *open(2) — Linux manual page* (Linux man-pages 6.18). https://man7.org/linux/man-pages/man2/open.2.html diff --git a/docs/doctoring/feature-cache-json-numeric-admission.md b/docs/doctoring/feature-cache-json-numeric-admission.md new file mode 100644 index 000000000..20c8fb56f --- /dev/null +++ b/docs/doctoring/feature-cache-json-numeric-admission.md @@ -0,0 +1,27 @@ +# Feature-cache JSON numeric admission + +## Problem + +BandScope bounds persisted feature-cache sidecars to 1 MiB before UTF-8 and JSON decoding, but the parser boundary previously caught `json.JSONDecodeError` without catching every failure that Python's default JSON numeric conversion can raise. A syntactically valid sidecar containing an integer longer than the interpreter's integer-string conversion limit causes the default `parse_int=int` path to raise `ValueError`. Because feature-cache replay is optional persistence, that exception must not escape as analysis-job failure; the cache must fail closed and be recomputed from the admitted local-audio source. + +Corrected RED `f667a9ebf3562dbc1c9ca2c3947d9a95daa4ebb2` writes a bounded JSON object whose `sampleRate` value contains 5,000 decimal digits and requires `read_bounded_feature_cache_metadata()` to return `None`. Production `be336746179567032ef0bdcdd811eb590a319326` keeps the existing descriptor, regular-file, size, UTF-8, and JSON boundaries and extends only the parser exception containment to `ValueError`. + +## Decision and alternatives + +Catch `ValueError` at the bounded metadata parser owner. This is narrower than changing Python's process-wide integer conversion limit, which would weaken the interpreter's denial-of-service protection, and narrower than supplying a custom `parse_int`, which is unnecessary because BandScope does not need arbitrary-precision persisted numeric authority. Catching the error later in `_load_cached_local_audio_features` was rejected because both the first metadata read and archive admission's second metadata read use the same parser and should share one cache-miss contract. + +The 1 MiB sidecar ceiling remains unchanged. This repair is exception containment, not permission for huge integers: the sidecar is rejected before its values can become sample-rate, duration, or rehearsal evidence. + +## Evidence and traceability + +Python Software Foundation. (2026). *json — JSON encoder and decoder* (Python 3.13.15 documentation). https://docs.python.org/3.13/library/json.html + +Python documents that untrusted JSON may consume substantial CPU and memory and recommends limiting input size. Since Python 3.11, the default `parse_int` delegates to `int()` with the interpreter's maximum integer-string conversion limit specifically to help avoid denial-of-service attacks. BandScope retains that protection and converts its resulting parser failure into a cache miss rather than letting it cross the optional-persistence boundary. + +MITRE. (2026). *CWE-248: Uncaught Exception* (CWE 4.20). https://cwe.mitre.org/data/definitions/248.html + +CWE-248 describes an exception escaping its intended handling boundary and potentially affecting availability. The mapping here is narrow: a bounded but adversarial/corrupt persisted sidecar can trigger Python's numeric conversion `ValueError`; BandScope now handles that exception at the cache parser. This does not claim that every possible Python, NumPy, ZIP, filesystem, or downstream MIR failure is contained. + +## Remaining claim boundary + +This repair does not complete the versioned immutable feature-cache generation. `.features.json` and `.features.npz` are still independent persisted objects, and the bounded metadata snapshot, private NPZ replay snapshot, and native verified source digest are not yet cryptographically committed as one generation. That remains the next #866 Project Persistence/Resource Admission integration step; Project Persistence #970 continues to own durable `sourceReference/contentSha256` semantics. diff --git a/docs/doctoring/feature-cache-metadata-sidecar-admission.md b/docs/doctoring/feature-cache-metadata-sidecar-admission.md new file mode 100644 index 000000000..14c0bf65c --- /dev/null +++ b/docs/doctoring/feature-cache-metadata-sidecar-admission.md @@ -0,0 +1,59 @@ +# Feature-cache metadata sidecar admission + +## Problem and boundary + +BandScope persists feature-cache metadata beside stem arrays so an interrupted or restarted rehearsal can reuse admitted local-audio work. That sidecar is app-owned persistence, but after a crash, restore, local modification, or partial publication its bytes are not trusted runtime state. + +The existing replay boundary bounded ZIP/NPY declarations before NumPy materialization, but both metadata reads still had a resource-policy asymmetry: the first API snapshot and the archive owner's independent second read used ordinary text `open()` followed by `json.load()`. A locally substituted sidecar could therefore consume memory in UTF-8/JSON materialization before BandScope applied a metadata-size admission rule. A special-file substitution could also make a persistence read behave differently from the regular file emitted by the canonical writer. + +The bounded parser originally still accepted duplicate JSON object member names through Python's default `json.loads` behavior. That is an authority ambiguity at a persistence boundary: a sidecar can physically contain two `schemaVersion` or nested `duration_seconds` members while a dictionary-shaped result exposes only one interpretation. RFC 8259 says object member names SHOULD be unique and warns that duplicate-name handling differs across implementations. Python exposes `object_pairs_hook` specifically at every decoded JSON object, so BandScope can reject duplicates before reducing the parsed representation to a dictionary. + +The repair introduces `read_bounded_feature_cache_metadata` in the Resource Admission & Decode owner. The helper opens the sidecar once, checks the already-open descriptor with `fstat`, requires a non-empty regular file no larger than 1 MiB, reads at most the admitted extent plus one probe byte, then performs UTF-8 decoding and JSON materialization. Growth or truncation during that read fails closed. Where the platform exposes them, `O_NONBLOCK` and `O_NOFOLLOW` are applied before the open descriptor is admitted. The 1 MiB ceiling is a BandScope product limit, not an external standard; the canonical writer emits only schema/source/sample-rate/separation/stem-key/role metadata, so the limit deliberately leaves substantial evolution headroom without accepting unbounded persistence. + +RED `2f6e73250368f58c25524a9a61510bb4bc58f1ec` exposed the broader unbounded first-read path by making JSON materialization fail if an oversized sidecar reached it. Production `a7531f54c5c84f74c5be45663948d918a3ea048d` establishes the bounded reusable sidecar reader and moves the archive owner's second read onto it. Test refinement `4a89c629c9280ee95ecdb612538fa1e1895c5201` isolates the new owner contract. Production descendant `02deb79bcdfbf0ac7db5e4bc30556d4b0c845536` routes `_load_cached_local_audio_features` first metadata snapshot through the same primitive, removing the remaining direct `json.load()` feature-cache path. Test descendant `cbfaf3b56fe9c16545c5c60314cf06045539b28f` proves both the primitive and the first API snapshot reject an oversized sidecar before `json.loads` can run. + +Duplicate-member RED `c7721839fb31dcb6cb5720f03bf24ee73250b05b` adds both top-level `schemaVersion` and nested `separation.duration_seconds` ambiguity cases. Production `10f7a369ef1277d8a50e6db2a80047185b35a6af` supplies an `object_pairs_hook` that materializes each JSON object only when all member names are unique; a repeated name raises the already-contained `ValueError` and becomes a cache miss. The hook applies recursively because Python invokes it for every decoded JSON object. + +## Invariants + +Every feature-cache metadata read used for stem replay must pass the same bounded primitive. A sidecar admitted by that helper must be a non-empty regular file whose observed byte extent is at most 1 MiB. The descriptor is opened before type and size checks, so admission and the bounded read refer to the same opened object rather than a later pathname lookup. The read must return exactly the admitted extent; an early EOF or an appended probe byte makes the cache a miss. UTF-8 decode failure, malformed JSON, duplicate JSON member names at any object depth, filesystem failure, allocator exhaustion, and integer-conversion-limit failures remain cache misses rather than rehearsal evidence. + +The helper does not make metadata, NPZ bytes, and source publication one generation. It also does not authorize new stem identities, relax timeline/resource rules, or establish scientific MIR accuracy. The existing archive owner still performs schema, canonical stem/role, sample-rate, duration/timeline, ZIP/NPY, dtype, sample-count, finite-signal, and live `AudioResourcePolicy` checks. + +## Alternatives and decision + +Keeping `json.load()` on an unbounded text file and validating fields afterwards was rejected because resource consumption precedes semantic validation. Reading the entire sidecar into an unconstrained `str` or `bytes` object was rejected for the same reason. + +Accepting Python's default duplicate-member behavior was rejected because it silently collapses multiple physical claims into one dictionary value. Selecting the first or last duplicate was also rejected: feature-cache metadata is optional acceleration state, so recomputation is safer than inventing precedence for contradictory persistence authority. The selected `object_pairs_hook` rejects any duplicate member at any nesting depth while retaining the standard parser and the existing bounded `ValueError` cache-miss path. + +Checking `Path.stat()` and reopening the pathname was rejected because the pathname can identify a different object between check and use. The selected helper opens first and uses `fstat()` plus the same descriptor for the bounded read. This is narrower than a cryptographic manifest: each metadata read is resource-bounded and descriptor-consistent, but two independently valid reads can still observe different publication generations. + +A second parser implementation in `api.py` was rejected. First-read and archive-read feature metadata now consume the same bounded primitive, so future size/type/encoding/duplicate-member policy changes have one owner. + +A very small schema-derived byte ceiling was also rejected. Cache schemas evolve, and an excessively tight limit would turn benign version growth into avoidable recomputation. One MiB is intentionally far above the current canonical sidecar while still putting a concrete ceiling before parser allocation. + +## Residual finding and follow-up + +Duplicate-member ambiguity is closed at the bounded JSON parser, but two calls to that parser do not create one transactional metadata generation. A sidecar can still be replaced between the first API snapshot and the archive owner's second read with another independently admissible schema-v1 object whose already-checked semantic fields happen to agree. The persistence design therefore still needs the planned versioned immutable manifest that binds one admitted metadata snapshot, the private NPZ snapshot, and the #970-owned `contentSha256` / `sourceReference` publication identity after protected ancestry. Do not replace that with pathname/timestamp heuristics or duplicate Project Persistence source hashing in #866. + +Windows process-tree containment remains a separate runtime gap. The current shared process owner establishes Unix process-group semantics, but Windows still requires race-free Job Object creation/assignment before whole-tree termination can be claimed. + +## Security traceability + +MITRE CWE-400 describes uncontrolled consumption of resources such as memory, CPU, storage, and other finite system capacity. The BandScope mapping here remains narrow: persistence-controlled metadata bytes were allowed to reach JSON materialization without a product byte ceiling. Both replay metadata reads now impose the product ceiling before parser allocation; this is not a claim of whole-process memory containment. + +RFC 8259 Section 4 requires interoperable JSON producers and consumers to treat unique object member names as the portable form and notes that duplicate-member behavior varies by implementation. BandScope chooses the stricter fail-closed interpretation because this JSON is persisted rehearsal evidence, not a user-authored document that benefits from permissive recovery. + +Python 3.14 documents `object_pairs_hook` as receiving the ordered member pairs for every decoded JSON object, which provides the parser boundary needed to detect duplicates before dictionary collapse. Python also documents the integer-string conversion limit applied by the default `parse_int=int`, which remains contained by the same `ValueError` cache-miss path. + +Python 3.14 documents `O_NONBLOCK` as available on Unix and `O_NOFOLLOW` as an extension that may be absent when the C library does not define it. BandScope therefore treats these flags as defense in depth and keeps regular-file/size admission on the opened descriptor as the portable semantic requirement. + +## References + +Bray, T. (2017). *The JavaScript Object Notation (JSON) Data Interchange Format* (RFC 8259). Internet Engineering Task Force. https://www.rfc-editor.org/rfc/rfc8259 + +MITRE. (2026). *CWE-400: Uncontrolled resource consumption* (CWE 4.20). Common Weakness Enumeration. https://cwe.mitre.org/data/definitions/400.html + +Python Software Foundation. (2026). *json — JSON encoder and decoder* (Python 3.14.7 documentation). https://docs.python.org/3/library/json.html + +Python Software Foundation. (2026). *os — Miscellaneous operating system interfaces* (Python 3.14.7 documentation). https://docs.python.org/3/library/os.html diff --git a/docs/doctoring/feature-cache-replay-admission.md b/docs/doctoring/feature-cache-replay-admission.md new file mode 100644 index 000000000..95a1997c7 --- /dev/null +++ b/docs/doctoring/feature-cache-replay-admission.md @@ -0,0 +1,53 @@ +# Persisted feature-cache replay admission + +## Problem and authority boundary + +BandScope persists separated stem arrays so a local rehearsal project can resume without repeating expensive separation work. That cache is app-owned persistence, not trusted in-memory state. Crash recovery, restore, local tampering, partial publication, or version drift can change metadata and array bytes independently of the original separator execution. + +Resource Admission & Decode therefore owns replay admission. Signal/MIR consumes only the admitted result; it does not accept new stem identities or role semantics from persisted cache metadata. + +The production separator contract currently exposes four stem identities: `vocals`, `bass`, `drums`, and `other`. Their canonical rehearsal role binding is `vocals -> vocal` and `bass|drums|other -> instrument`. Persisted metadata may omit `stemRoleTypes` for legacy compatibility, in which case the canonical mapping is reconstructed. The metadata sidecar itself is still required at replay because `api.py` has already admitted its schema, sample rate, stem set, and separation metadata; disappearance before archive admission is a concurrent persistence change, not a legacy-cache representation. When `stemRoleTypes` is present, a contradictory binding is a cache miss rather than rehearsal evidence. + +## Decision + +The replay boundary applies these checks before an archived stem can return to MIR: + +- `stemKeys` is a unique, non-empty subset of the canonical separator identities. +- The NPZ central directory contains exactly one deflated NPY member for every admitted stem and no hidden or duplicate member. +- NPY v1 headers, one-dimensional floating shape, declared sample count, visible bytes, sample rate, canonical owned `float32` conversion, and finiteness stay within the live `AudioResourcePolicy`. +- The sibling persisted metadata sidecar must remain readable at archive admission. Missing, malformed, or non-object replacement is a cache miss; only absence of the `stemRoleTypes` field inside an otherwise admitted legacy sidecar remains compatible. +- Persisted role metadata, when present, preserves the canonical identity-to-role binding. A valid `bass` signal cannot be relabelled `vocal` by cache metadata. +- Legacy metadata without `stemRoleTypes` remains readable by reconstructing the canonical mapping; malformed JSON, non-object metadata, non-map role metadata, invented stem identities, contradictory roles, and sidecar disappearance fail closed. + +The role check is deliberately an admission rule, not ontology expansion. BandScope's rehearsal semantics remain authoritative in BandScope; the cache does not mint a new domain vocabulary. + +## RED and repair evidence + +Regression `test_feature_cache_rejects_role_type_that_contradicts_canonical_stem_semantics` writes a real compressed NumPy `float32` bass stem. The same persisted array must be rejected with `{"bass":"vocal"}` and accepted with `{"bass":"instrument"}`. Companion coverage exercises malformed JSON, non-object metadata, omitted role metadata, non-map role metadata, and canonical binding. + +The initial semantic RED commit is `fd8ad6f51ebdbd39b6d736d4f105ef65621919d1`. Production role admission begins at `4d6441c898e358eda51898cf92f6640726710ebd`; subsequent ordinary descendants repair source-owned formatter findings without importing the independently owned protected-base Ruff change from PR #1176. Hosted failure for that RED commit is not claimed because its run was cancelled after the ordinary production descendant. + +A later concurrency-focused review found that the role helper treated a now-missing sidecar as legacy-compatible even though the production caller had necessarily parsed that sidecar moments earlier. RED `2eb807307bc76da007e4e6165d500fcfa3795bdd` changes the regression to require a missing sidecar to fail closed. Production `c7e8dfa96303ed46be2a07577b45cc50f4509233` removes the existence-as-legacy shortcut: the archive-admission read itself must succeed, while an existing legacy sidecar may still omit `stemRoleTypes`. This narrows the replacement race without claiming a transaction that does not yet exist. + +## Residual risk and follow-up + +The current repair still reads the sibling metadata sidecar twice overall: once in `api.py` for the authoritative cache payload and again in the archive-admission helper for role semantics. Missing, malformed, and contradictory second snapshots now fail closed, but two independently opened pathnames are not a race-free metadata/archive transaction. A privileged local actor can still replace one valid generation with another valid generation between reads. The next persistence hardening step remains one authoritative metadata snapshot bound to the already-open archive generation, or an equivalent immutable generation/digest contract, with a deterministic replacement-race regression before any crash/race-safe claim. + +This boundary also does not prove a whole-process RSS, CPU, decompression-time, or accelerator budget. NumPy/ZIP internals, decoder/resampler intermediates, PyTorch model memory, GPU/VRAM, and downstream MIR temporaries remain separate resource-acceptance work. Production scientific acceptance still requires rights-cleared decoded rehearsal audio and reproducible MIR evidence; synthetic arrays are used only as focused unit regressions for this persistence invariant. + +## Traceability + +- CWE-345 (Insufficient Verification of Data Authenticity) supports treating restored or mutable persisted evidence as untrusted until its expected semantics are re-established. BandScope does not claim cryptographic authenticity from this check; immutable project publication identity is a separate control. +- CWE-367 (Time-of-check Time-of-use Race Condition) maps to the remaining two-open metadata/archive generation race. The missing-sidecar repair closes one permissive outcome but does not claim the broader race resolved. +- CWE-770 (Allocation of Resources Without Limits or Throttling) maps to the bounded archive/member/sample/byte admission applied before NumPy materialization. +- NumPy's NPY/NPZ format and `ndarray` dtype/shape semantics define the representation inspected at this persistence boundary. BandScope additionally imposes its own canonical stem vocabulary and rehearsal-role invariant. + +## References + +MITRE Corporation. (2026). *CWE-345: Insufficient verification of data authenticity.* Common Weakness Enumeration. https://cwe.mitre.org/data/definitions/345.html + +MITRE Corporation. (2026). *CWE-367: Time-of-check time-of-use (TOCTOU) race condition.* Common Weakness Enumeration. https://cwe.mitre.org/data/definitions/367.html + +MITRE Corporation. (2026). *CWE-770: Allocation of resources without limits or throttling.* Common Weakness Enumeration. https://cwe.mitre.org/data/definitions/770.html + +NumPy Developers. (2026). *NumPy binary format (`.npy`) and `numpy.savez_compressed` documentation.* https://numpy.org/doc/stable/reference/generated/numpy.lib.format.html ; https://numpy.org/doc/stable/reference/generated/numpy.savez_compressed.html diff --git a/docs/doctoring/feature-cache-resource-admission.md b/docs/doctoring/feature-cache-resource-admission.md new file mode 100644 index 000000000..a907a364a --- /dev/null +++ b/docs/doctoring/feature-cache-resource-admission.md @@ -0,0 +1,128 @@ +# Persisted feature-cache resource admission + +## Scope and decision + +BandScope treats persisted stem features as a Resource Admission & Decode boundary, not trusted in-memory state. Cache bytes live below an app-owned path, but crash recovery, interrupted publication, restore, local modification, or version drift can make them disagree with the live decode/separation contract. Replaying them directly into MIR would let persistence bypass controls applied to freshly decoded audio. + +The single replay owner is `feature_cache_admission.load_bounded_stem_archive`. `_load_cached_local_audio_features` validates the first metadata/separation envelope and delegates archive admission with the live `DEFAULT_AUDIO_RESOURCE_POLICY`; archive admission independently re-reads the sidecar before replay. MIR, UI, and project code do not implement parallel cache validators. + +The current design copies exactly the initially admitted regular-file extent into a private `tempfile.SpooledTemporaryFile`, then performs ZIP/NPY declaration preflight and `numpy.load` against that same snapshot. Small archives remain in memory up to the explicit spool threshold; larger admitted archives roll over to `TemporaryFile` semantics. The snapshot is context-owned and is not a project artifact or a second persistent cache. Declaration preflight also returns the one synchronized non-zero stem sample count. Both the first API metadata snapshot and the second-read sidecar must carry `separation.duration_seconds`; replay requires a finite positive value, and archive admission additionally requires agreement with `sample_count / sample_rate` within half one sample before NumPy materialization. + +This replaces the earlier descriptor-metadata guard. The previous guard compared `(st_dev, st_ino, st_size, st_mtime_ns)` before preflight and after NumPy materialization. That detects ordinary writes, but it is not a content identity: an in-place writer can replace bytes with a same-size archive and restore the original modification timestamp. In that case every compared field can remain unchanged while `np.load` observes bytes that were never preflighted. + +RED `f94a4bf919561f407ae28aaef6a3b781a0b60989` reproduces that exact bypass. It admits a 16-sample `bass` archive containing negative zero, prepares a same-size archive containing ones, overwrites the same inode immediately before `np.load`, restores `st_mtime_ns`, and requires replay to retain the already-admitted negative-zero generation. Production `0cfc54a465b07d73c9d98ca94f49639b395b4006` introduces the bounded spooled snapshot. Test descendant `d2b3f4931f5cbf5a86d3bdba765285f4f3946a34` preserves the same-metadata substitution regression, changes the older post-preflight path-mutation test to the stronger snapshot semantics, and covers fail-closed short-copy behavior. + +RED refinement `ea23114061830753d02ba76142e7a0d9f0c5fd6b` covers a separate metadata boundary: Python's standard JSON decoder accepts the non-standard numeric token `Infinity` as `float('inf')`, so a persisted `separation.duration_seconds` can be non-finite even though RFC 8259 does not permit Infinity or NaN as JSON numbers. Production `8b3a512a3ea7b8f680b9eb787ac98f728d1e7929` rejects Boolean, non-numeric, non-finite, zero, and negative duration values when the field is present in the second-read sidecar. + +Review of that production delta found one conversion edge case before completion: a JSON integer can be valid Python `int` yet too large for `float(...)`, which raises `OverflowError`. RED `559a95f1d838b50737203104169c2ec57a19f49d` persists a 401-digit duration and requires a cache miss rather than an escaping exception. Production `1350df871942cf87910aab65bba243674065107c` now contains numeric conversion and rejects overflow before the value can become rehearsal timing authority. + +RED `dfbf288457a86f628e01488456c8f662093f8a60` closes the next stable-cache semantic gap. Its fixtures first make the valid cache duration sample-derived, then persist `17 / 44_100` seconds beside a synchronized 16-sample, 44.1 kHz stem archive and require a cache miss. The predecessor admitted that cache because both the duration and stem declarations were individually valid. Production `bbaf19a133b870c02a402b617fcabf7ed2f5b9aa` makes bounded NPY preflight return the synchronized sample count and checks second-read duration against `sample_count / sample_rate` with zero relative tolerance and an absolute half-sample tolerance. A one-sample timeline drift therefore fails before `np.load`; the product does not pad, truncate, stretch, or resample persistence to make contradictory timing appear valid. + +The half-sample tolerance is deliberate. Exact binary-float equality would couple cache validity to serialization details rather than the discrete waveform extent, while a full-sample tolerance would permit one complete sample of contradictory timeline authority. The selected bound accepts representation noise smaller than the nearest-sample decision boundary and rejects a persisted duration that denotes another discrete sample count. + +RED `aa9265c751a1219c123fed091a8260b27a82dfad` closes another second-read generation gap. The fixture now carries a sample-derived duration so every other replay invariant remains valid, then replaces only `schemaVersion: 1` with `schemaVersion: 2` between the API metadata read and archive admission. The predecessor accepted the second sidecar because the archive owner rechecked stem identity, sample rate, duration, and role semantics but did not re-admit the schema version. Production `e0fbb0307522b9901e19f1bac77a307ee0cfae32` requires the second-read sidecar to remain on feature-cache schema version 1 before any archive replay. This prevents a concurrent metadata replacement from crossing a schema-generation boundary while retaining values that happen to satisfy the old reader. + +Corrected RED `6123936f0ca4ad54f081d981b7e885cebbddf747` closes the stable missing-duration case. It removes `separation.duration_seconds` from an otherwise admissible 16-sample, 44.1 kHz cache and requires a cache miss. Production `ccda5980d8c40cab5a88c39148e19775c35e311d` requires the second-read sidecar to contain a `separation` mapping with a present, non-Boolean numeric, finite and positive duration before NPZ replay. This prevents an unchanged malformed cache from returning real stems with `duration_seconds=None` and later reaching rehearsal code that compares the missing value with zero. + +RED `20c606d3a36fcae7d1db8158200acf848f33e915` exercises the remaining first-read form of that defect. It starts with a first metadata snapshot whose duration is missing, restores a fully valid sidecar only when archive admission begins, and requires the cache to remain a miss. The predecessor could accept the repaired second sidecar and return the stale first snapshot's missing duration. Production `5d4ce528a04b3bcaf374a987f7f8d0656349e546` therefore validates first-read duration before archive admission. Review immediately caught that applying `numpy.isfinite` directly to a 401-digit Python integer could itself raise `TypeError`; descendant `0984f9333c4b18cc6f3f783508b8a6dd9439b335` keeps conversion inside the fail-closed boundary, preserving the existing oversized-integer regression and restoring the unrelated API docstring unchanged. + +RED `a1eaeadcf021665cc19461bfa4378f03de9ee6c8` closes the analogous first-read role-semantic gap. The first metadata snapshot persists `bass: vocal`, then a valid `bass: instrument` sidecar replaces it only when archive admission begins. The predecessor accepted both snapshots independently because the outer reader admitted any value from the broad `{vocal, instrument}` vocabulary while the second-read archive owner enforced the canonical stem mapping. Production `4fa56489e93b8b5596738460bb033ba4011178a2` validates first-read `stemRoleTypes` against the canonical mapping immediately. A later sidecar can no longer retroactively repair contradictory role truth retained by the outer loader; legacy caches that omit `stemRoleTypes` still derive the canonical mapping. + +RED `906bac5167815025b416571db6ecfe7c4d8205be` closes an exception-safety hole inside that second-read role boundary. A sidecar can be syntactically valid and explicitly declare `stemKeys=["guitar"]` with `stemRoleTypes={"guitar": "instrument"}`. The predecessor compared the explicit role to `_CANONICAL_STEM_ROLE_TYPES[stem_key]` before proving the key belonged to the canonical stem vocabulary, so the untrusted `guitar` key escaped cache-miss semantics as an uncaught `KeyError`. Production `c0bf2d3505b00ec3366eff82a6540805444bcce8` reuses the canonical member-name admission before role-table lookup. Unknown, duplicate, empty, non-identifier, or over-cardinality stem identities now fail closed before a dictionary lookup can occur. This repair is deliberately local: it does not expand the canonical separator vocabulary or invent a guitar stem contract. + +Earlier retained controls remain authoritative: canonical `vocals`, `bass`, `drums`, `other` stem identity; canonical role binding at both metadata reads; second-read schema/stem/sample-rate consistency; one synchronized non-zero sample timeline across cached stems; exact deflated NPY member set; bounded header/sample/decoded-byte declarations before materialization; `allow_pickle=False`; canonical owned finite NumPy `float32` re-admission; and `MemoryError`/truncation containment as a cache miss. These are persistence/resource controls, not evidence of source-separation accuracy. + +## Invariants + +A replayable cache must satisfy all of the following. + +- The archive path resolves to one non-empty regular file whose initially observed extent is within the aggregate archive ceiling derived from the live audio policy. +- The admitted source extent is copied in bounded chunks. A short read before the admitted byte count is reached fails closed. Bytes appended after that initial extent are not admitted into the snapshot. +- ZIP/NPY preflight and NumPy materialization use the same snapshot object. A pathname replacement, same-inode rewrite, timestamp restoration, or later growth of the original file cannot redirect those two phases to different bytes. +- Both the first metadata snapshot and the second metadata read must preserve the canonical role mapping for every named stem. The second read must also remain on feature-cache schema version 1 and preserve the caller-admitted `stemKeys` and sample rate. Metadata may name only a unique non-empty subset of `vocals`, `bass`, `drums`, and `other`, and membership is proven before any role-table lookup. Legacy absence of `stemRoleTypes` derives the canonical mapping; an explicit contradictory or unknown-stem role mapping fails closed. +- Persisted `separation.duration_seconds` is mandatory in both the first API metadata snapshot and the second-read sidecar, must convert to one finite positive real value, and must agree with the synchronized archive timeline within `0.5 / sample_rate` seconds. Missing duration, Python-specific `NaN`, `Infinity`, `-Infinity`, Boolean, zero, negative, integer values too large for finite-float conversion, and durations denoting another sample extent fail closed before authoritative replay. +- The ZIP central directory contains exactly the corresponding `stem_.npy` members: no extra/duplicate/directory/encrypted entries and no compression method other than the representation emitted by the current writer. +- Every member declares one non-empty floating one-dimensional array within the sample and visible-byte ceilings. All admitted stems declare the same sample count; replay never pads, truncates, stretches, or resamples malformed persistence into apparent synchronization. +- Loaded data is returned to MIR only after live `AudioResourcePolicy.validate_decoded_audio` rechecks sample rate, one-dimensional shape, canonical dtype, sample count, visible bytes, ownership/canonicalization, and finiteness. +- Optional-cache failures including malformed ZIP/NPY state, schema/identity/rate/role replacement, unknown stem identity, missing, invalid, or timeline-inconsistent persisted duration, allocator exhaustion, short snapshot copy, and unsupported representation become cache misses rather than authoritative rehearsal results or interpreter exceptions. + +## Alternatives considered + +Keeping only the pre/post `fstat` comparison was rejected because file metadata is not a content digest and can be restored after an in-place rewrite. Re-statting the pathname was rejected because a pathname can resolve to another object while an already-open descriptor still refers to the original file. + +Hashing the archive twice around `np.load` was rejected. It would still make preflight and materialization two reads from mutable source storage and would add two complete archive reads to an acceleration path. A single bounded byte snapshot is a smaller causal control: declaration inspection and use are performed on the same bytes. + +Reading the entire archive into `bytes` was rejected because the policy permits multiple long canonical stems and a full in-memory duplicate would make cache admission itself a large peak-RSS event. `SpooledTemporaryFile` provides a bounded in-memory threshold and rolls larger content to `TemporaryFile` behavior without creating a named project artifact. + +Direct `np.load` followed by post-load validation remains rejected because decompression/materialization would happen before BandScope checked the member declarations and resource bounds. Reimplementing NPY decoding is also rejected; BandScope parses only the declarations needed for admission and delegates actual decoding to NumPy with pickle disabled and a bounded header. + +Accepting Python's default `json.loads` numeric extensions as trusted duration evidence was rejected. Python deliberately accepts `NaN`, `Infinity`, and `-Infinity` even though they are outside the JSON specification. The semantic reader therefore validates the decoded domain value and contains finite-float conversion overflow. Replacing the entire metadata parser in this patch was unnecessary. A future manifest writer should additionally emit strict JSON (`allow_nan=False`) and use one bounded/versioned parser contract so invalid numbers cannot be persisted by the canonical writer in the first place. + +Treating `schemaVersion` as an API-only check was rejected. Archive admission intentionally performs a second metadata read to detect replacement between the outer metadata check and replay. Every field that defines the reader contract must therefore be re-admitted at that second boundary. Accepting another schema version because its current fields happen to look compatible would make future cache evolution depend on accidental structural overlap rather than an explicit migration contract. + +Comparing persisted duration to `sample_count / sample_rate` with exact floating-point equality was rejected because the invariant is a discrete-sample identity, not a requirement that every producer serialize an identical binary float. Accepting a full sample of error was also rejected because that permits a duration corresponding to another discrete sample count. Half one sample is the nearest-sample boundary for this persistence check; scientific timing accuracy remains a separate real-audio acceptance problem. + +Treating a missing duration as a legacy-compatible value was rejected. Persisted stems are timeline-bearing rehearsal evidence, and the current reader has no safe basis for inferring that the outer metadata generation and the archive generation are the same when timing authority is absent. Cache miss and recomputation preserve the source-of-truth boundary without fabricating a duration. A later valid sidecar is not allowed to retroactively repair a first-read snapshot that lacked timing authority. + +Treating any persisted `vocal` or `instrument` value as acceptable first-read role metadata was rejected. Stem names already define the canonical rehearsal role semantics at this boundary, so accepting `bass: vocal` or `vocals: instrument` would allow persistence to alter the separator contract. Relying only on the second-read sidecar was also rejected because the outer loader retains and returns its first metadata snapshot. Contradictory first-read role metadata therefore becomes a cache miss before the second read. + +Catching `KeyError` around the canonical role lookup was rejected as the primary repair. The persisted stem identity itself is outside the allowed domain, so membership must be rejected before the lookup rather than converted into a generic exception path after the invalid value has reached a canonical table. Expanding the role table with `guitar` was also rejected because the current separator contract does not emit that stem and persistence cannot create a new BandScope audio ontology by naming it. + +Repairing malformed synchronized timelines by padding, truncating, or resampling is rejected because it would fabricate rehearsal evidence after persistence corruption. Cache miss and recomputation from the admitted source are safer. + +## Security Notes and claim boundary + +MITRE CWE-367 describes TOCTOU as checking resource state and later using the resource after that state can change. The replay snapshot removes the concrete archive check/use split: one copied byte sequence is both checked and used. The second-read schema check additionally prevents a metadata replacement from moving replay into a different cache-format contract after the API already admitted schema version 1. First-read duration and role admission now prevent a later valid sidecar from repairing missing timing authority or contradictory role semantics retained by the outer loader. None of these controls makes metadata, archive, and source publication one cryptographic generation; a concurrently torn writer can still pair an individually admissible sidecar with an individually admissible archive from another generation if their declarations agree. + +MITRE CWE-248 describes an exception that escapes its intended handling boundary and can crash or otherwise destabilize the application. The unknown-stem regression maps narrowly to that failure mode: the persisted key was untrusted cache metadata, the product contract already defined a closed canonical vocabulary, and lookup occurred before membership validation. The repair restores cache-miss semantics by validating the closed vocabulary first; it is not a claim that every possible exception in NumPy, ZIP, filesystem, or downstream MIR execution is contained. + +Python 3.14 documents `SpooledTemporaryFile` as using memory until `max_size` is exceeded and then proceeding as `TemporaryFile`; the high-level temporary-file APIs support context-manager cleanup. BandScope uses that behavior only for an ephemeral replay snapshot. This does not claim immunity to operating-system termination, disk exhaustion, filesystem failure, or whole-process RSS pressure. + +Python 3.14 also documents that `json.dumps`/`json.loads` accept `NaN`, `Infinity`, and `-Infinity` by default and that this behavior is outside the JSON specification. RFC 8259 states that numeric values outside its grammar, including Infinity and NaN, are not permitted. The current reader therefore treats those decoded values as invalid domain metadata rather than relying on the permissive library default. Numeric conversion is itself inside the fail-closed boundary so an otherwise parseable oversized integer cannot turn admission into an uncaught `OverflowError` or NumPy type error. + +NumPy documents `numpy.load` as accepting seekable binary file-like objects, recommends `allow_pickle=False` for safer handling of untrusted data, and supports a `max_header_size` limit. `.npz` is ZIP-backed and arrays are loaded on access, so BandScope still performs its own exact-member and NPY-header admission before allowing materialization. Library defaults are not treated as a product resource policy. + +The first and second metadata reads can no longer omit duration or contradict the canonical stem-role mapping, and the second-read sidecar cannot change schema version, stem identity, sample rate, finite persisted duration/timeline semantics, or introduce an unknown stem identity without becoming a cache miss. This is still not a cryptographic metadata/archive/source generation. The cache does not bind its sidecar and private NPZ snapshot to the exact admitted source publication identity by digest, nor does it prove that two individually admissible metadata/archive generations with the same declarations belong together. The next persistence contract should version and bind the API metadata snapshot, archive snapshot, and admitted source identity in one immutable generation/manifest, while preserving atomic publication and cache-miss compatibility for older generations. + +Persisted cache controls do not prove MIR or separation accuracy. Canonical identities, finite float32 buffers, synchronized lengths, sample-derived duration consistency, schema consistency, canonical role semantics, and immutable replay bytes say nothing about bleed, interference, onset/section error, or model generalization. Production scientific acceptance still requires rights-cleared real decoded rehearsal audio, recognized MIR metrics, uncertainty/claim boundaries, and reproducible CPU/accelerator results. + +## Evidence-to-control traceability + +| Evidence | BandScope control | +| --- | --- | +| CWE-367: state checked before use can change and invalidate the check. | Archive declaration preflight and NumPy materialization consume one private copied snapshot rather than two reads from mutable source storage. | +| CWE-248: an exception escaping the intended handling boundary can crash or destabilize the application. | Persisted stem membership is validated against the closed canonical vocabulary before indexing the canonical role table; unknown stems become cache misses rather than `KeyError`. | +| Python 3.14 `tempfile`: `SpooledTemporaryFile` remains memory-backed to `max_size`, then rolls to `TemporaryFile`, and can be context-managed. | Replay uses an 8 MiB in-memory spool ceiling and lets larger admitted archives roll to an automatically cleaned temporary file instead of duplicating the full cache in RAM. | +| RFC 8259 forbids Infinity and NaN as JSON numbers; Python 3.14 `json` accepts and decodes them by default. | Persisted separation duration is explicitly re-admitted as a finite positive domain value instead of inheriting Python's permissive JSON numeric extension. | +| Python numeric conversion can raise `OverflowError` when an integer is outside finite-float range. | Duration conversion is contained inside cache admission; an oversized integer becomes a cache miss rather than escaping into the analysis job. | +| NumPy `numpy.load` accepts binary seekable file-like objects, exposes `allow_pickle=False`, and bounds header parsing with `max_header_size`. | The private snapshot is passed directly to bounded declaration inspection and then `np.load(..., allow_pickle=False, max_header_size=16 KiB)`. | +| The current BandScope/Demucs separation contract fits every canonical stem to the decoded source length and computes source duration as `audio.size / sample_rate`. | Replay requires persisted duration, requires all cached stems to share one sample count, and requires duration to resolve to that same discrete timeline within half a sample. | +| Exact-head regression `f94a4bf9…` keeps device, inode, size, and mtime unchanged while substituting equal-size sample bytes. | Metadata-only descriptor identity is no longer the authority for check/use consistency; immutable replay bytes are. | +| Exact-head regression `ea231140…` persists an infinite separation duration beside an otherwise admissible stem archive. | Second-read metadata admission rejects non-finite or non-positive duration before reusable stem replay. | +| Exact-head regression `559a95f1…` persists a 401-digit positive integer duration. | Finite-float conversion overflow is caught and converted to cache miss semantics. | +| Exact-head regression `dfbf2884…` persists `17 / 44_100` seconds beside a 16-sample, 44.1 kHz canonical stem archive. | Bounded NPY preflight returns the synchronized sample count, and replay rejects duration that falls outside the half-sample agreement bound before NumPy materialization. | +| Exact-head regression `aa9265c7…` changes only the second-read sidecar from schema version 1 to version 2 while all other current-schema fields and archive bytes remain admissible. | Archive admission independently requires the second-read sidecar to remain on feature-cache schema version 1 before replay. | +| RED `6123936f…` removes `separation.duration_seconds` from an otherwise admissible cached stem generation. | Second-read replay admission requires duration authority; an unchanged cache without it becomes a miss/recompute instead of returning stems with missing rehearsal timing. | +| RED `20c606d3…` starts from metadata with missing duration, then restores a valid sidecar only when archive admission begins. | First-read replay admission rejects missing or invalid duration before the second-read owner can observe a different generation. | +| RED `a1eaeadc…` starts with `bass: vocal`, then restores canonical `bass: instrument` only when archive admission begins. | First-read role admission rejects contradictory persisted semantics before a later sidecar can repair them. | +| RED `906bac51…` persists the noncanonical `guitar` key with an explicit role mapping and calls the second-read sidecar admission boundary directly. | Canonical stem membership is proven before role-table lookup, so untrusted identity cannot escape as `KeyError`. | + +## References + +Bray, T. (Ed.). (2017). *The JavaScript Object Notation (JSON) data interchange format* (RFC 8259). RFC Editor. https://doi.org/10.17487/RFC8259 + +Défossez, A., Usunier, N., Bottou, L., & Bach, F. (2021). *Music source separation in the waveform domain*. Transactions of the International Society for Music Information Retrieval, 4(1), 123–136. https://doi.org/10.5334/tismir.76 + +MITRE. (2026). *CWE-248: Uncaught exception* (Version 4.20). https://cwe.mitre.org/data/definitions/248.html + +MITRE. (2026). *CWE-367: Time-of-check time-of-use (TOCTOU) race condition* (Version 4.20). https://cwe.mitre.org/data/definitions/367.html + +NumPy Developers. (2025). *numpy.load — NumPy v2.3 manual*. https://numpy.org/doc/2.3/reference/generated/numpy.load.html + +Python Software Foundation. (2026). *json — JSON encoder and decoder* (Python 3.14.7 documentation). https://docs.python.org/3/library/json.html + +Python Software Foundation. (2026). *tempfile — Generate temporary files and directories* (Python 3.14.7 documentation). https://docs.python.org/3/library/tempfile.html + +## Follow-up acceptance + +The next persistence RED should bind the first API metadata snapshot, private NPZ replay snapshot, and exact admitted source publication identity to one versioned digest/generation rather than adding more pathname, timestamp, or independently re-read semantic checks. After that, Resource Admission still needs race-free Windows Job Object containment and rights-cleared full-length rehearsal-audio measurement for cancellation latency, inherited handle/pipe return, child temporary cleanup, decoder/resampler/downstream peak RSS/VRAM, and explicit per-job CPU/GPU budgets. diff --git a/docs/doctoring/local-audio-analysis-source-identity-handoff.md b/docs/doctoring/local-audio-analysis-source-identity-handoff.md new file mode 100644 index 000000000..7676e7d43 --- /dev/null +++ b/docs/doctoring/local-audio-analysis-source-identity-handoff.md @@ -0,0 +1,62 @@ +# Local-audio analysis source identity handoff + +## Problem + +BandScope already verifies the app-owned `source.` publication against the native staging receipt and retains a path-free `LocalAudioPublicationIdentity` containing the exact byte length and lowercase SHA-256 digest. Before this repair, `start_analysis_job` recovered only the renderer-hidden bootstrap path/name/size tuple. The verified content identity stayed in native state and never crossed the owned analysis-child boundary, so a future feature-cache generation could not bind itself to the exact publication that Resource Admission had verified. + +This is an authority-drop defect rather than a hash-algorithm defect. Path, display name, extension, and byte length are insufficient content identity: two different files can share all four values. + +A later review of the first Python consumer found a second boundary defect. The CLI disabled persisted caching when `sourceContentSha256` was absent by deleting a string `cacheRoot` before the canonical request validator ran. A malformed traversal value such as `../cache` could therefore disappear before `validate_analysis_job_request` had a chance to reject it. Disabling an unverified cache must not sanitize or erase an independently invalid request field. + +## Constraints + +- The renderer must not gain the content digest as a new authority. The digest is retrieved only from `LocalAudioPublicationIdentityState`, keyed by the BandScope-minted project id. +- Resource Admission remains the owner of the verified publication receipt. Project Persistence may consume that path-free identity, but this repair does not duplicate `sourceReference` or persistence schema owned by #970. +- A digest is integrity/identity evidence, not authenticity. An attacker who can replace both an artifact and an unauthenticated digest can recompute SHA-256. No MAC, signature, FIPS 140 validation, or tamper-proof cache claim is made here. +- The analysis child receives the digest in the native JSON job envelope. Python now validates the request before any digest-derived cache namespace transformation and scopes persisted cache lookup below `source-sha256-v1/` only when the verified digest is present. This is still not the versioned immutable metadata/NPZ manifest. +- Invalid request fields must remain observable to the canonical validator. Cache-disable logic is not a normalization or sanitization authority. + +## Decision + +`start_analysis_job` now requires the native publication identity for local-audio jobs before it grants analysis authority. It cross-checks the retained identity against the native bootstrap source size, extension, and app-owned artifact name. A missing or contradictory identity fails before the job slot is acquired. + +The owned child-process JSON envelope carries `sourceContentSha256` only when that verified native identity exists. Demo jobs remain unchanged. The field is deliberately outside the renderer-authored request object so JavaScript cannot nominate an arbitrary digest and have it treated as verified publication evidence. + +The Python CLI validates the renderer/native request object before `_bind_verified_source_cache_namespace` may remove an unverified cache root or append the verified digest namespace. A valid local request without native source identity still has persisted caching disabled, but malformed `cacheRoot`, `localSource`, project, and other request fields fail through the existing canonical validation path instead of being hidden by cache scoping. This preserves the rule that transformation must not make an invalid request appear valid. + +When the digest is present, persisted analysis and feature-cache paths are scoped below the verified source digest namespace. That prevents a path/name/size-addressed cache created for one verified byte sequence from being looked up under another verified source identity. The cache metadata and NPZ bytes are not yet cryptographically bound to one manifest, so this namespace is a prerequisite rather than completion of cache generation integrity. + +## Alternatives rejected + +Re-hashing `localSource.sourcePath` inside Python was rejected. It would create a second content-identity implementation below the already verified native publication boundary, add another full-file read on the analysis path, and reintroduce pathname replacement questions after native Resource Admission has already produced exact content evidence. + +Using file name, path, size, modification time, inode, or another filesystem tuple as the generation key was rejected. Those values are useful admission evidence but do not identify file contents and do not solve same-size replacement. + +Exposing `contentSha256` in renderer-visible `LocalAudioSource` was rejected. The browser-side request is not the authority that verified the app-owned publication, so reflecting the digest through that surface would make provenance easier to confuse without improving the native trust boundary. + +Teaching `_bind_verified_source_cache_namespace` to recognize and selectively preserve every malformed request shape was rejected. That would duplicate `validate_analysis_job_request` and let the two validation contracts drift. Canonical validation therefore runs first; cache namespace binding consumes only an already validated request. + +## Risks and follow-up + +The in-memory native identity state is session-scoped. Restart/re-admission and Project Persistence still need the canonical path-free durable identity owned by the persistence lane. The cache manifest must fail closed when the exact source identity is unavailable rather than falling back to path/name/size. + +The current digest namespace prevents cross-source lookup under a different verified digest, but it does not bind the existing `.features.json` and `.features.npz` pair into one crash-atomic generation, authenticate local cache contents, or prove MIR/source-separation accuracy. The next persistence change remains a versioned immutable manifest that binds the verified source identity, bounded metadata snapshot, and private NPZ replay snapshot. + +## Exact evidence + +- RED `be30f7156dc63371b4a9c19c377a2ac2a30fd144` requires local analysis startup to recover the native publication identity and carry a source-content digest toward the analysis boundary. +- Production `a9c1b436d2b26d7fcb5d0f58d515a5da0caee21c` recovers and cross-checks the retained identity, then adds its digest to the owned analysis-child JSON envelope. +- Intervening RED `6ce5e9587ea22cc84e589ba28c6146419a6f6f0b` and production `9472f99d10844f0cffdafbc8cb82be5d4b806362` scope persisted cache lookup to the verified source digest; `693ef8366ef62dec1235d86671b1da205dc8c545` adds fail-closed cache-scoping regressions. +- RED `d94f528ee3f73ea4808b3bb37dddc964870f2700` proves that cache-disable transformation could hide an invalid traversal `cacheRoot` before canonical request validation. +- Production `2edcfb71e98fbbc1a28954638351d230a7085cc9` validates the complete request before digest namespace binding, preserving canonical rejection while retaining cache disablement when verified source identity is absent. +- The pre-existing Resource Admission implementation computes SHA-256 over exactly the staged bytes, re-reads the published object, and requires the publication to reproduce the staging size/digest receipt before `LocalAudioPublicationIdentity` is minted. + +## TRACEABILITY + +National Institute of Standards and Technology. (2015). *Secure Hash Standard (SHS)* (FIPS PUB 180-4). https://doi.org/10.6028/NIST.FIPS.180-4 + +NIST specifies SHA-256 as part of the SHA-2 family and describes message digests as a means to detect whether message contents differ from the contents used to generate the digest. NIST announced in 2023 that FIPS 180-4 will be revised, chiefly to remove SHA-1 and incorporate applicable guidance; that revision decision does not invalidate the SHA-256 content-identity use here. BandScope does not claim CAVP/FIPS module validation from using the specified algorithm. + +MITRE. (2026). *CWE-22: Improper limitation of a pathname to a restricted directory (Path Traversal), CWE 4.20*. https://cwe.mitre.org/data/definitions/22.html + +CWE-22 describes relative-path traversal through special elements such as `..`. BandScope's canonical request validator rejects traversal segments for app-owned cache roots. The repair above preserves that rejection by ensuring cache-disable transformation does not erase the field first. diff --git a/docs/doctoring/local-audio-source-materialization.md b/docs/doctoring/local-audio-source-materialization.md new file mode 100644 index 000000000..848728069 --- /dev/null +++ b/docs/doctoring/local-audio-source-materialization.md @@ -0,0 +1,154 @@ +# Local audio source materialization + +## Problem + +BandScope originally validated an OS-selected local audio file and then let later analysis reopen the canonical external filesystem path. That left analysis and restart dependent on mutable host authority: the selected file could be moved, replaced, truncated, or grown after admission. Project Persistence #962 also needs a durable source identity that does not serialize an arbitrary user filesystem path. + +Resource Admission & Decode therefore owns creation and verification of the app-owned `source.` artifact and the native content identity for that publication. Project Persistence owns the later versioned project reference that consumes this evidence; it does not copy or hash user media itself. + +The hardening sequence exposed distinct defects: + +- source-read and app-owned destination-write failures were initially collapsed into one diagnosis; +- the one-byte over-limit probe was initially written into the disposable stage; +- the bounded copy returned only a byte count, so there was no native identity for the exact bytes written; +- SHA-256 existed in more than one security-sensitive implementation and initially had no reusable reader-only core port; +- a staging receipt alone did not prove that the final published object still contained the same bytes; +- publication verification initially read against the product-wide 100 MiB ceiling instead of the receipt's tighter expected length; +- the production Tauri materializer initially discarded the receipt and stayed on the byte-count-only adapter; +- publication initially used `destination.exists()` followed by overwrite-capable `rename`, creating a check-then-act clobber window; +- the no-clobber hard-link publication synchronized the staged file but did not explicitly cross a platform namespace-durability barrier after destination creation and private-stage removal; +- even after publication verification existed, Project Persistence still had no typed path-free handoff value for `projectId + artifactName + extension + fileSizeBytes + contentSha256`; +- after that type existed, the production selector still discarded the verified identity instead of retaining it in native state for the persistence owner. + +The canonical #866 branch now repairs those defects through native retention and a platform-specific publication commit. Production local-file materialization consumes the native receipt and synchronizes the stage. On Unix it creates the immutable destination with a same-filesystem no-clobber hard link, removes the private stage name, and synchronizes the project directory. On Windows it performs a no-replace `MoveFileExW` with `MOVEFILE_WRITE_THROUGH`. Only after that commit boundary does the materializer reopen and verify the published bytes, derive `LocalAudioPublicationIdentity` from the verified receipt, and retain the path-free value in native Tauri state keyed by the locally minted project id before returning bootstrap authority. The strict analysis-runtime `LocalAudioSource` wire remains unchanged. + +Project Persistence #970 has already adopted this Resource Admission ancestry in downstream Draft work: it consumes the retained identity into versioned `sourceReference`, re-admits only the app-owned artifact after restart, and snapshots the admitted bytes before analysis decode. The remaining cross-owner buyer gap is Active Player #1160's fresh audible Full mix/current-stem authority. Platform-atomic no-follow descriptor acquisition, durability of creation/replacement of higher directory ancestors, YouTube durable-source policy, and decoder licensing remain separate work. + +## Constraints and invariants + +- Local analysis remains local-first; this boundary adds no network authority. +- Renderer input never selects an arbitrary analysis or persistence path. +- The encoded-byte ceiling remains exactly 100 MiB. +- Metadata length before copying is not final evidence when the selected source can change during admission. +- Source-read failure and app-owned write/publication failure remain distinguishable without exposing paths or raw OS errors. +- `Interrupted` reads are retried. +- SHA-256 covers only byte slices whose staging writes succeeded. The one-byte growth probe is not admitted content and is not hashed into the receipt. +- SHA-256 is content-identity/correctness evidence only. This code does not claim CAVP validation, FIPS 140 validation, authenticity, or protection against an actor who can replace both artifact and stored digest. +- Reusable SHA-256 and publication-verification APIs accept caller-owned `Read` values and acquire no path authority. +- Publication verification consumes at most `expected.file_size_bytes + 1` bytes and rejects invalid expected lengths before reading. +- Publication must not overwrite an existing app-owned source name. +- Unix publication uses same-filesystem hard-link creation, private-stage unlink, then project-directory synchronization before the publication can mint persistence/bootstrap identity. +- Windows publication uses `MoveFileExW` without `MOVEFILE_REPLACE_EXISTING` and with `MOVEFILE_WRITE_THROUGH`; same-project staging keeps the move on one volume. +- Portable `std::fs::rename` is not the publication primitive because Rust's contract permits replacing an existing destination and platform semantics differ. +- A generic Windows directory `File::sync_all` is not treated as equivalent to Unix directory `fsync`; the Windows boundary uses the documented write-through move instead. +- The durability claim is scoped to the source publication mutation inside an already-existing app-owned project directory. Creation or replacement of higher ancestors and storage that falsely acknowledges flush completion remain outside this claim. +- The analysis-runtime `LocalAudioSource` contract remains `sourcePath + fileName + extension + fileSizeBytes`. `contentSha256` is not injected into that strict Rust/TypeScript/Python request without a versioned contract change. +- The persistence identity is a distinct contract. It contains exactly `projectId + artifactName + extension + fileSizeBytes + contentSha256`; it contains no `path` or `sourcePath` field. +- The persistence identity accepts only an existing BandScope project-id grammar, canonical lowercase admitted extension, byte size `1..=100 MiB`, and exactly 64 lowercase hexadecimal SHA-256 characters. `artifactName` is derived as `source.` rather than accepted from renderer input. +- Verified persistence identity is retained only in native Tauri state keyed by the minted project id. The renderer does not author or supply that evidence. +- If native identity state cannot be retained, local-source selection fails closed rather than returning bootstrap authority without persistence evidence. +- Portable `symlink_metadata` / open / re-check logic narrows linked-object substitution but does not claim atomic `O_NOFOLLOW` or Windows reparse-point-equivalent semantics. + +## Decision record + +1. Keep the external canonical path and revalidate before every analysis — rejected. Restart and persistence would still depend on mutable host authority. +2. Persist the absolute external path — rejected. It widens disclosure and violates #962's path-free direction. +3. Copy the selected file into app-owned `source.` — selected. Later analysis can use BandScope-owned authority. +4. Keep `std::io::copy` and one generic error — rejected. Explicit bounded read/write preserves the ceiling while distinguishing source and destination failures. +5. Hash later in the renderer or from the original path — rejected. Neither is authoritative for bytes actually staged into BandScope storage. +6. Add another SHA-256 implementation in persistence or Active Player — rejected. `bandscope_desktop_core::sha256_hex_reader` is the reader-only Shared Kernel. +7. Treat the staging receipt as publication truth without rereading — rejected. Same-size mutation would evade byte-count checks. +8. Re-read every published object up to 100 MiB — rejected. The native receipt gives a tighter expected length. +9. Leave the Tauri caller on `copy_bounded_local_audio -> u64` — rejected. Production publication must retain native size+digest evidence and verify the publication before bootstrap authority is returned. +10. Check `destination.exists()` and then rename the stage — rejected. On overwrite-capable rename semantics the sequence is racy. +11. Use portable `std::fs::rename` without a preflight check — rejected. Rust permits replacement of an existing destination and does not provide one cross-platform durability contract for the namespace mutation. +12. Use one generic directory-sync implementation on Unix and Windows — rejected. Unix directory synchronization and Windows directory-handle/flush contracts are not interchangeable enough to justify a portable-looking claim. +13. On Unix, create the destination with `std::fs::hard_link(stage, destination)`, remove the private stage name, then synchronize the project directory — selected. Destination creation is no-clobber and the final namespace state crosses an explicit directory durability barrier. +14. On Windows, use `MoveFileExW(stage, destination, MOVEFILE_WRITE_THROUGH)` without `MOVEFILE_REPLACE_EXISTING` — selected. Existing destinations fail closed and the documented write-through flag keeps the move from returning before the move reaches disk. +15. Add `contentSha256` to the existing analysis `LocalAudioSource` payload — rejected. Python admission is strict and this would mix persistence evidence with a narrower runtime request. +16. Define a separate path-free `LocalAudioPublicationIdentity` whose artifact name is derived from canonical native evidence — selected. This keeps Resource Admission as the copy/hash authority and gives #970 a serializable persistence input without absolute paths. +17. Return bootstrap authority while leaving the verified identity only in a local stack variable — rejected. The selector retains the typed identity in native Tauri state keyed by project id before returning; downstream persistence can adopt that native evidence without trusting renderer-authored digest/path data. + +## Implementation and exact evidence + +The cumulative hardening remains test-first where behavior changed: + +- `dbeee9c7407c72f999f584eb0eb9342ddc39fddd` adopted protected `develop@314ddeae7b775a4957594b599358c8255617eb2e` through ordinary non-force ancestry. +- RED `804a2867e877947feaffb1da6c6072e6a49049fe` and fix `0beee45b98e51ba46b571a82c6d0d93db61ea8d6` established exact-limit acceptance and one-byte-over rejection. +- `a2b1bd9e33a69be75f813f005abd37345200ce55` moved successful local-file intake to an app-owned same-project stage; `323a7fac00c4954af12b382802a9d6f8359ef4c5` exported the core port to Tauri. +- Diagnostics RED `131d6d7220985abd207559e6eb5dc122ac989cf4` and fix `ac4adfdb5df82f48aadd5e028433e3336d3ce2ae` separated source-read and destination-write failures and made the one-byte over-limit check read-only. +- Content-identity RED `dc413794fb84c736085ab77b763854ba0f58bdf1` and fix `566cd1f991296e7f3c288cb07a11c2d2effb258a` introduced `LocalAudioCopyReceipt { file_size_bytes, content_sha256 }`. +- Shared-kernel RED `373824c7bbb40f2df1bb2721316680378c104834` and fix `d1ba40683772019577fec4d8c767ff8b23294e38` exposed reader-only `sha256_hex_reader`. +- Publication RED `fdfdd7003b8a9162f846dcf22ffe66a3afd5f47e` and fix `a1c85cbfbdc7051169f097e8ad235e3bbac439d3` introduced `verify_local_audio_publication_receipt`; `20e7faaddd619c6cbd053876ca6de27b9933a4a2` exported it. +- Bounded-verification RED `6a0692ee288d3b126bd0598e07e03c88a702d567` and fix `c65a9fd312f4d67e6d1cad83b80b1213e692c8dd` changed publication verification to stop after expected bytes plus one growth probe. +- Production-integration RED `ed9fe7eba6261753dc0f68e820e2b642703fe2cd` and fix `bdf8f87d5e5c9db423537c7633e7ff4b92bec5b6` moved the Tauri materializer onto native receipt + publication verification. +- No-clobber RED `45b1f72abeded4e478775d31085244621f68c9f0` and fix `eb972e951ef090c92b595c752b18d66f11f6b96e` replaced check-then-rename with same-filesystem hard-link publication. +- Path-free handoff RED `bad908c83bfb89f545f0f2f637d96ac8fdfa3e0e` requires exact camelCase serialization of the five persistence fields, no path fields, and fail-closed rejection of invalid native evidence. +- Path-free handoff fix `87bdeea92d3bb6dc45eb666f422bd8a3d36f3872` adds `LocalAudioPublicationIdentity` and `build_local_audio_publication_identity`; export `344a9a39f32ac40b3e137c76e2cfd46243827bb5` makes the contract available from `bandscope_desktop_core` to the persistence owner. +- An earlier exploratory retention RED `cbfa967b16e94f2d84940665ce38537075a8ce41` was intentionally neutralized by `d8c57ce1d64d0bc9963219740aeaa83d9569a90b` rather than leaving a known failing head; those two commits add no production claim. +- Production native-retention RED `106ae75cad85553e56964a9844ea7a01f6ce456c` requires the materializer to derive the typed identity from the verified receipt, the selector to store it in native state, and Tauri to register that state. +- Native-retention fix `e4e2ba734bc80304a754ce2eb52e473fd9ee3631` returns `LocalAudioSourcePayload + LocalAudioPublicationIdentity` from materialization, stores the identity in `LocalAudioPublicationIdentityState` before bootstrap authority is returned, and registers the native state with the Tauri runtime. +- Publication-durability RED `ebc505504afd06bab55dfc4ba64aa312f7aa848e` requires a dedicated commit boundary and requires that boundary to occur before path-free identity is minted. It was followed immediately by implementation descendants, so no hosted RED failure is claimed. +- `d7945553c334fb192bf316ad21ddc790983952c9` introduced the platform-specific publication module; `4e2bccc5986070ac60937ff9ac481696ea898671` repaired its unit-test import before production integration. +- Production fix `94086edb9749cf82708718abc31a46fbbaaf7742` moves the Tauri materializer onto that commit boundary: Unix hard-link + stage unlink + project-directory sync; Windows no-replace `MoveFileExW` + `MOVEFILE_WRITE_THROUGH`. Publication verification and native identity follow the durability barrier rather than preceding it. + +The SHA-256 implementation is checked against standard known-answer vectors including the empty message, `abc`, the multi-block vector, and one million `a` bytes. Those are correctness regressions, not validation-module evidence. + +## Security Notes + +The selected audio path, file metadata, and media bytes are untrusted. The OS file dialog supplies initial user authority; BandScope uses that path only to canonicalize and open the source. The project-owned artifact is the authority after successful admission. + +The production Tauri materializer synchronizes the staged file before publication. Unix then creates the destination through a no-clobber same-filesystem hard link, removes the private stage name, and synchronizes the project directory. Windows uses a same-project `MoveFileExW` with `MOVEFILE_WRITE_THROUGH` and without a replace-existing flag. Failure at the publication boundary returns the existing path-redacted project-workspace diagnosis; no path-free source identity is minted on that failed call. + +After the platform commit, the materializer requires regular/non-symlink path observations, opens the publication, checks descriptor size, verifies exact receipt equality, and performs a post-verification path check. Publication mismatch or read failure is normalized to the bounded project-workspace diagnosis; source/destination paths, raw OS errors, and audio bytes are not exposed. + +`LocalAudioPublicationIdentity` does not acquire filesystem authority. It converts already verified native evidence into a deterministic, path-free value for the persistence boundary. Invalid project ids, extensions, byte counts, or digest encodings fail closed. Production local-file selection retains that value in native Tauri state before returning the ordinary bootstrap summary, so the renderer does not need to invent a digest or persist a host path. + +Residual risk remains explicit. The boundary does not claim platform-atomic no-follow acquisition; it does not claim that creation/replacement of higher project-root ancestors has been durably committed by this source-publication operation; and it cannot compensate for storage that reports successful flush/write-through before durable media persistence. + +No new logging, telemetry, network transfer, or raw-media export is introduced. The SHA-256 receipt and publication identity are non-secret content identity. + +## Test and acceptance points + +- exact 100 MiB encoded-byte limit accepted; one byte over rejected; +- empty source rejected; +- source-reader and destination-writer failures remain distinct and path-safe; +- `Interrupted` reads retry without changing identity; +- failed writes cannot return a partial receipt; +- the growth probe is neither staged nor hashed; +- unchanged published bytes reproduce the staging receipt; +- same-size mutation, truncation, growth, or publication-read failure fails closed; +- grown publication stops after expected bytes plus one probe; +- production publication cannot use existence-check plus overwrite-capable rename; +- an existing destination remains unchanged and publication fails closed; +- publication paths must remain direct children of the app-owned project root; +- Unix publication synchronizes the project directory after the final destination/stage namespace mutations; +- Windows publication requests no replacement and uses `MOVEFILE_WRITE_THROUGH`; +- production Tauri local-file materialization consumes receipt, durable publication-commit, and publication-verification boundaries, not the compatibility byte-count adapter; +- path-free identity is minted only after the platform publication commit; +- path-free identity serializes exactly the five persistence fields and cannot serialize `path`/`sourcePath`; +- invalid project ids, uppercase/unsupported extensions, zero/oversized byte counts, and noncanonical SHA-256 encodings are rejected; +- production local-file selection derives identity from the verified receipt and retains it in registered native Tauri state before returning bootstrap authority; +- hosted Rust/Tauri, Windows, macOS, security, SBOM, coverage/package, and independent-review evidence must be reacquired on the final exact #866 head. + +Synthetic test bytes exercise the filesystem unit boundary only. They do not substitute for production scientific acceptance. Rights-cleared real decoded audio still has to exercise the integrated Windows/macOS intake/decode/analysis/playback path where the relevant commercial claim is made. + +## Remaining risks and follow-up + +The local-file path now has separate native contracts for bytes, namespace publication and durable identity: `LocalAudioCopyReceipt` proves exact staged/published content; `commit_local_audio_publication` closes the supported-platform source-name publication barrier; retained `LocalAudioPublicationIdentity` represents path-free durable evidence for Project Persistence. + +Project Persistence #970 already consumes that evidence in downstream Draft code, re-admits the app-owned source on restart and snapshots admitted bytes for analysis. Active Player #1160 must still combine persisted `selectedPlaybackSource` intent with fresh native Full mix/current-stem availability; missing preferred stems fail closed to Full mix. When #866/#970 ancestry enters that stack, the private playable-stem SHA-256 implementation should be deleted in favor of `bandscope_desktop_core::sha256_hex_reader` while preserving stem identity/error tests. + +YouTube intake still uses its owned cache artifact and needs an explicit durable-source promotion decision. Platform-atomic no-follow acquisition and higher-ancestor crash durability remain Resource Admission/platform work. Issue #1129 remains the commercial decoder-dependency gate. + +## References + +Microsoft. (2023). *MoveFileExW function (winbase.h).* Microsoft Learn. https://learn.microsoft.com/windows/win32/api/winbase/nf-winbase-movefileexw + +Microsoft. (2025). *Directory handles.* Microsoft Learn. https://learn.microsoft.com/windows/win32/fileio/directory-handles + +National Institute of Standards and Technology. (2015). *Secure Hash Standard (SHS)* (FIPS PUB 180-4). https://doi.org/10.6028/NIST.FIPS.180-4 + +National Institute of Standards and Technology. (2023, March 7). *Decision to revise FIPS 180-4, Secure Hash Standard (SHS).* https://csrc.nist.gov/news/2023/decision-to-revise-fips-180-4 + +Rust Project Developers. (2026). *std::fs::rename.* Rust standard library documentation. https://doc.rust-lang.org/std/fs/fn.rename.html diff --git a/docs/doctoring/local-audio-stem-work-identity.md b/docs/doctoring/local-audio-stem-work-identity.md new file mode 100644 index 000000000..18100822c --- /dev/null +++ b/docs/doctoring/local-audio-stem-work-identity.md @@ -0,0 +1,86 @@ +# Local-audio stem-work identity + +## Problem + +BandScope admits concurrent local-audio analysis jobs. Native Resource Admission owns the immutable app-local source publication and verified SHA-256 identity; Python persisted cache and mutable stem work must not infer identity from renderer paths, names, or sizes. + +Retained #866 ancestry therefore scopes persisted cache below `source-sha256-v1/` and mutable stem work below the source digest plus `job-sha256-v1/`. Normal CLI completion may clean only that derived job namespace. + +The first cleanup implementation validated the lexical suffix and rejected pre-existing symlinks in the caller-selected base and generated source/job parents before calling pathname-based `shutil.rmtree(path)`. That was necessary but insufficient. A local filesystem actor could replace a checked ancestor after `Path.is_symlink()` returned and before `rmtree(path)` resolved it. A symlink-resistant `rmtree` protects traversal below the directory it opens; it does not bind every earlier pathname component to the object that was checked. + +## Constraints + +- `sourceContentSha256` remains native-owned Resource Admission evidence; renderer JSON cannot author it. +- Mutable stem work is per execution even when two jobs use the same verified source. +- Raw `jobId` is untrusted text and is never a filesystem segment. +- A validated `tempRoot` is not blanket recursive-delete authority. +- Cleanup may target only an absolute canonical `job-sha256-v1/<64-lowercase-hex>` namespace, optionally below canonical `source-sha256-v1/`. +- A safe temporary-file leak is preferable to deleting through ambiguous filesystem state. +- Project Persistence #970 remains the durable owner of `sourceReference/contentSha256`; this lane does not copy its aggregate or storage schema. + +## Decision + +`_cleanup_job_temp_namespace()` keeps the lexical shape and pre-existing-link checks as an early fail-closed filter, but deletion authority is descriptor anchored on supported Unix runtimes. + +The context-managed `_open_anchored_directory()` opens the filesystem anchor and then every parent component relative to the previously opened directory descriptor with `os.open(..., dir_fd=...)`, `O_DIRECTORY`, and `O_NOFOLLOW`, plus `O_CLOEXEC` where available. All opened descriptors remain live only for the `with` scope and are deterministically closed in the context manager's `finally` block, including partial-open failure. + +The caller then invokes `shutil.rmtree()` only for the final job-digest name with `dir_fd` set to the already-open `job-sha256-v1` parent. If an ancestor becomes a symlink before its descriptor-relative open, opening fails closed. If an already-open ancestor is renamed and its former pathname is replaced after the earlier lexical check, recursive deletion remains relative to the descriptor rather than resolving the replacement pathname. + +Cleanup is skipped when Python does not advertise symlink-attack-resistant `rmtree`, when `os.open` lacks `dir_fd` support, or when `O_DIRECTORY` / `O_NOFOLLOW` are unavailable. This is intentional portability behavior, not a fallback to weaker pathname deletion. + +## RED → production trace + +- `b8e55628d9cc35c210741ffbc6934cb415822a4b` → `a7f2af8718b5c6c3398f65dfbd31ee7484f31464`: verified source digest scopes temporary stem work. +- `2c0e4633bf9bea929f74551d1d97fd6ae0e35063` → `5d9e2fe23bdb2602c4446f11a0de1f9d32b9184b`: concurrent jobs for one source receive different job-digest namespaces. +- `adeb7afa8952fee242be4d44c34b316f1dd1263d` → `f870cbf273124ecf70cea267dcc2f5f69abc4174`: normal completion cleans only the derived job namespace. +- `8f23455d4a99ed7d2f08460e21e6bbcda35adbf6` → `8ac2733df64a55d8e506f0aa74f830fbd7e7cf51`: generated-parent symlink substitution fails closed and unsafe `rmtree` runtimes do not delete. +- `8adc4546667a7a701d5d260b998be60068fe2bc3` → `880e4b8c0b521270d886cef9198f9a6a313a4099`: caller-selected base-root symlink substitution also fails closed; `a7d7056157a640c0b6c5487c23613c14e2a7658d` covers the source-verified form. +- `3c0cec3f9fceef879806d4f3003563136149da36`: initial parent-swap test draft omitted the `shutil` import and is not acceptance evidence. +- `911957dcd8c10c5cae00d4cb3a17e55388c094c7`: corrected RED source replaces the checked base after the lexical link check and requires the outside sentinel to survive. +- `1c8ea0cdaf58b75ae3e35f446b337b124a41d109`: production introduces descriptor-relative no-follow parent-chain opens and final-name `rmtree(..., dir_fd=...)`. +- `de6d88c07e7302d893124185137db43a9ebea9c0`: regression edges require fail-closed behavior when `dir_fd` is unavailable and for absolute spellings containing `..`. +- `50ae703e932580a48ef821c9dfb0572e5a7498d0`: production refinement keeps partial-open and descriptor-close branches coverage-complete. +- `45b6c3d7e6206dcb1dcf3e99cbf8b963f1579697`: current review repair makes descriptor lifetime explicit through a context manager after code-quality correctly flagged the raw-descriptor-return shape as not locally proving closure. + +Hosted RED is not claimed because production descendants were pushed before an exact corrected-RED workflow failure was established. Exact-head workflow evidence remains authoritative. + +## Security Notes + +The untrusted boundary includes validated request roots plus cache/temp filesystem state that can change after a crash or local mutation. The parent-swap regression creates an intended temp tree and a separate outside tree with the same `job-sha256-v1/` suffix, then replaces the caller base after the lexical link check reports it as a normal directory. Descriptor-anchored cleanup must leave the outside sentinel untouched. Other tests cover normal verified/unverified cleanup, pre-existing generated/base symlinks, malformed source/job digests, unsupported `dir_fd`, unsafe `rmtree`, and `..` spellings. + +This repair materially narrows CWE-59/CWE-367 exposure on descriptor-capable Unix filesystems. It does not prove cleanup after process kill or OS crash; hostile mount/bind-mount replacement; filesystem semantics outside the descriptor assumptions; Windows descriptor-relative cleanup, because Python documents `dir_fd` support as Unix-only; or Windows Job Object process-tree containment. + +The fail-closed tradeoff is deliberate. Unsupported or ambiguous cleanup leaves temporary work for a separately admitted app-owned recovery path rather than broadening deletion authority. + +## Persistence and scientific claim boundary + +This cleanup work does not make the feature cache one crash-atomic generation. `.features.json` and `.features.npz` are still independent persisted objects. The next #866 persistence delta remains one versioned immutable manifest that binds the native verified source digest, one bounded metadata snapshot, and one private NPZ replay snapshot, and is published last as the commit marker. Missing, old, or unbound generations must be cache misses. + +SHA-256 is used here as content identity/integrity evidence. It is not an authenticity claim, MAC, signature, FIPS-module validation claim, or proof that local cache state is tamper-proof. + +After the immutable-manifest prerequisite, the next runtime gap is race-free Windows Job Object ownership. Rights-cleared full-length rehearsal audio must then establish cancellation latency, inherited pipe/handle return, abnormal-child temp cleanup, decoder/resampler/downstream peak RSS and VRAM, explicit per-job CPU/GPU budgets, and recognized MIR reproducibility metrics. Synthetic filesystem regressions are security/persistence evidence, not scientific MIR acceptance. + +## TRACEABILITY + +| Decision / invariant | Repository evidence | External basis | +| --- | --- | --- | +| Exact source bytes scope reusable cache and mutable stem work | `_bind_verified_source_cache_namespace`; source/job identity regressions | NIST FIPS 180-4 SHA-256 message digest | +| Concurrent jobs do not share mutable intermediate stem artifacts | job-digest namespace regression | CWE-362 shared-resource race guidance | +| Recursive cleanup cannot rely on a checked mutable pathname | parent-swap and static symlink regressions | CWE-59; CWE-367 | +| Parent identity remains bound through delete use | `_open_anchored_directory`; `_cleanup_job_temp_namespace`; context-managed descriptor lifetime | Python `os.open(..., dir_fd=...)`, `os.supports_dir_fd`, `O_DIRECTORY`, `O_NOFOLLOW`; `shutil.rmtree(..., dir_fd=...)` | +| Unsupported descriptor semantics fail closed | no-`dir_fd` and unsafe-`rmtree` regressions | Python documents platform-dependent `dir_fd` and symlink-resistant `rmtree` support | +| Persisted cache atomic generation remains open | #866 claim boundary and feature-cache doctoring | digest namespace isolation does not bind independent metadata/NPZ objects into one committed generation | + +## References + +National Institute of Standards and Technology. (2015). *Secure Hash Standard (SHS)* (FIPS PUB 180-4). U.S. Department of Commerce. https://doi.org/10.6028/NIST.FIPS.180-4 + +MITRE. (2026). *CWE-59: Improper link resolution before file access ('link following')* (CWE 4.20). Common Weakness Enumeration. https://cwe.mitre.org/data/definitions/59.html + +MITRE. (2026). *CWE-362: Concurrent execution using shared resource with improper synchronization (race condition)* (CWE 4.20). Common Weakness Enumeration. https://cwe.mitre.org/data/definitions/362.html + +MITRE. (2026). *CWE-367: Time-of-check time-of-use (TOCTOU) race condition* (CWE 4.20). Common Weakness Enumeration. https://cwe.mitre.org/data/definitions/367.html + +Python Software Foundation. (2026). *os — Miscellaneous operating system interfaces* (Python 3.12.14 documentation). https://docs.python.org/3.12/library/os.html + +Python Software Foundation. (2026). *shutil — High-level file operations* (Python 3.12.14 documentation). https://docs.python.org/3.12/library/shutil.html diff --git a/docs/doctoring/model-output-shape-admission.md b/docs/doctoring/model-output-shape-admission.md new file mode 100644 index 000000000..af1848d63 --- /dev/null +++ b/docs/doctoring/model-output-shape-admission.md @@ -0,0 +1,33 @@ +# Source-separation model-output shape admission + +Date: 2026-09-09 +Owner: Resource Admission & Decode / #781 / PR #866 + +## Problem + +The source-separation boundary already rejected empty, non-finite, non-numeric, and float32-overflowed model output, but `_as_float_array` used `np.ravel(...)`. A model/backend regression that returned an extra dimension could therefore be flattened into a one-dimensional stem and then trimmed/padded by `_fit_length`, turning malformed model output into rehearsal evidence instead of failing closed. + +This matters separately from decoder admission. The canonical decoder is required to produce mono float32 PCM before MIR, while Demucs output crosses another untrusted scientific boundary after model execution. Shape must remain explicit at that boundary as well. + +## Constraints and alternatives + +The fix must not change the four-stem contract, invent a new source-separation owner, add a dependency, broaden filesystem/network/model authority, or silently reinterpret multi-channel/model-batch output. + +Two alternatives were rejected: + +- Flattening and relying on `_fit_length`: preserves the defect because dimensional meaning is discarded before validation. +- Automatically downmixing any multidimensional model output: would make an undocumented scientific decision and could conceal an upstream model/backend contract change. + +## Decision + +Require model/decoder values entering `_as_float_array` to become a NumPy float32 array with exactly one dimension. Empty, multidimensional, non-finite, non-numeric, or float32-overflowed values retain the existing payload-free `Stem separation produced invalid audio.` failure. + +RED `2cb1e6485d68eaf360cf27432678ae7628ef489c` adds a two-channel-shaped regression that the predecessor flattened successfully. Production `2d256a68d7217098e070faa1d05aef9250a91123` removes the flattening operation and rejects `ndim != 1` before the output can reach `_fit_length` or downstream rehearsal analysis. The commits are consecutive ordinary descendants, so no hosted RED failure is claimed. + +## Risk and effect + +The intended Demucs path already performs its explicit channel mean before converting each source to NumPy, so valid current output remains one-dimensional. If a future model/backend changes that contract, BandScope now fails closed instead of silently changing the signal semantics. This fix does not bound total model RSS/VRAM, validate commercially admissible model rights, or establish real-audio separation accuracy. + +## Follow-up acceptance + +Exact-head Python/quickcheck/security checks must verify the repaired tree. Commercial scientific acceptance still requires rights-cleared real audio and recognized separation/MIR metrics, plus measured CPU/RSS/GPU/VRAM behavior. Windows descendant containment and the existing model distribution/rights owners remain separate prerequisites. diff --git a/docs/doctoring/request-path-security-diagnostics.md b/docs/doctoring/request-path-security-diagnostics.md new file mode 100644 index 000000000..670ff9b76 --- /dev/null +++ b/docs/doctoring/request-path-security-diagnostics.md @@ -0,0 +1,36 @@ +# Request-path security diagnostics + +## Decision + +BandScope rejects `..` path-traversal segments at the Python Resource Admission boundary before local-audio analysis. Rejection of `localSource.sourcePath` now emits the same kind of bounded security diagnostic already used for `projectId`, `cacheRoot`, and `tempRoot`, but the diagnostic contains only the fixed field identifier. The submitted path is never copied into the log message. + +This is deliberate. `localSource.sourcePath` crosses from the desktop request into the analysis engine and can contain user-local directory names, control characters, or attacker-chosen text. Logging that value verbatim, through `repr()`, or through a parameterized placeholder would retain attacker-controlled content in durable diagnostic output. BandScope therefore records the event class and field name, not the path payload. + +## RED → production → exact repair + +RED `0e313169299eb6112343ca7d8fbac612d6df5942` adds a request containing both a parent-traversal segment and a forged newline suffix. The regression requires the request to fail, requires exactly one warning containing only `localSource.sourcePath`, and asserts that the malicious path is absent from the logger call. + +Production `5546a17ccc848e2d043359586a2abc0bd198a66c` adds the bounded warning immediately before the existing traversal rejection. Review of that commit found two unrelated text-only drifts introduced while applying the one-line source repair. Descendant `7c0e858da327daa07b56c75b2a90a2761b4c4880` restores those lines. Comparing the pre-RED product head `4cb93a74b2668440479ed0ac884c60505acd7883` with `7c0e858d...` shows only one production-line addition plus the dedicated regression file. + +The earlier parallel Draft #1194 supplied the valid product finding. Its scanner-learning metadata is not a second runtime owner. Canonical implementation and regression ownership remain #866. + +## Alternatives rejected + +Logging the complete rejected path was rejected because the diagnostic does not require source identity and would persist user-local path data. Escaping the value with `repr()` was also rejected: escaping presentation characters does not remove the underlying attacker-controlled or privacy-sensitive content. Parameterizing the raw path with `%s` was rejected for the same reason. + +Suppressing all diagnostics was rejected because the other traversal fields already produce bounded security signals and `localSource.sourcePath` is the most direct local-file request boundary. The fixed field identifier preserves useful operational evidence without expanding log authority. + +## Claim boundary + +This control prevents the rejected path payload itself from entering this warning. It does not claim that every downstream library, operating-system error, decoder, or unrelated log statement is path-redacted. Those boundaries require their own tests and security review. The traversal predicate is unchanged by this repair; this change adds bounded observability to an already rejected request. + +## TRACEABILITY + +- Resource Admission source: `services/analysis-engine/src/bandscope_analysis/api.py::validate_analysis_job_request`. +- Regression: `services/analysis-engine/tests/test_request_security_diagnostics.py`. +- Canonical owner: #866; #1194 is a preservation/consolidation source only. +- Weakness model: CWE-117, Improper Output Neutralization for Logs. MITRE describes the weakness as constructing log output from external input without sufficient neutralization and notes log-forging consequences; BandScope avoids the source-to-log data flow entirely for this diagnostic. + +## Reference + +MITRE Corporation. (2026). *CWE-117: Improper Output Neutralization for Logs* (CWE List Version 4.20). Common Weakness Enumeration. https://cwe.mitre.org/data/definitions/117.html diff --git a/docs/doctoring/subprocess-containment.md b/docs/doctoring/subprocess-containment.md new file mode 100644 index 000000000..8e32e306d --- /dev/null +++ b/docs/doctoring/subprocess-containment.md @@ -0,0 +1,86 @@ +# Subprocess containment evidence + +Last reviewed: 2026-09-09 + +## Decision boundary + +BandScope treats native analysis and timed YouTube import as product-owned subprocess boundaries. The renderer may request work or cancellation only through BandScope-minted product identifiers; it does not receive a PID, process-group id, generic signal primitive, or arbitrary execution authority. + +On Linux and macOS, `bandscope_desktop_core` configures every owned analysis/import `Command` with `CommandExt::process_group(0)` before spawn. The child therefore leads a fresh process group before `exec`, and ordinary descendants inherit that group unless they deliberately change group or session. The shared termination path signals the negative group id with `SIGKILL`, then reaps the directly owned child when it is still running. Direct-child kill/reap is retained as the fail-closed fallback when group signalling is unavailable or fails. + +The output-draining boundary needs one additional invariant beyond timeout/cancellation cleanup. A directly owned process can exit successfully while a descendant still holds inherited stdout/stderr descriptors. Joining the reader threads immediately after observing the parent's terminal status can then block until that descendant exits, which defeats the product timeout even though the parent has already completed. The current implementation therefore terminates residual same-group descendants after direct-parent terminal status and before joining stdout/stderr readers. The returned `ExitStatus` remains the directly owned process's status; descendant cleanup is containment, not a replacement result. + +A fresh source-to-doctoring audit found that the timed-import helper already enforced that successful-parent invariant, but `run_analysis_engine` did not: its `Ok(Some(status))` arm preserved the direct parent's status and broke out of the poll loop before invoking the shared termination owner, then joined both reader threads. RED `1ae37e7836b0b8cb2be0ea258f0ecfc38b06de7d` adds a Tauri contract requiring terminal analysis cleanup before those joins and ties the adapter to the existing executable successful-parent descendant-pipe regression in desktop core. Production `f612c041bbf9a44331247daf4a964a9be563d462` adds only the missing shared-owner call in that terminal arm. No new PID/signal authority is introduced, and no hosted RED failure is claimed because the RED and fix were consecutive ordinary descendants. + +Captured helper output is also an admission boundary, not an unbounded diagnostics sink. stdout and stderr are each limited to 1 MiB plus one overflow-probe byte before metadata parsing or error presentation. Each reader reports a bounded-read failure to the single process-control owner through a standard-library MPSC channel. The process-control loop waits on that failure channel for at most the smaller of the configured poll interval and the remaining monotonic deadline. An oversized or unreadable stream therefore wakes the owner immediately instead of waiting for the next ordinary child-status poll; the same owner terminates/reaps the process boundary before joining readers and returns the existing payload-free process execution error. This bounds parent-side capture memory and removes avoidable post-detection helper lifetime; it does not constrain memory allocated inside Python, yt-dlp, FFmpeg, decoders, MIR/model code, or accelerator runtimes. + +A process timeout is also a deadline contract rather than a polling suggestion. The MPSC wait is clamped to the smaller of the configured poll interval and the monotonic time remaining until the deadline. A caller therefore cannot accidentally extend a 100 ms deadline by supplying a multi-second poll interval. This is intentionally separate from scheduler precision: BandScope does not claim hard real-time wake-up, but the implementation no longer adds a full poll interval of avoidable timeout overshoot or reader-failure latency. + +Analysis JSONL is a protocol-authority boundary in addition to a byte-admission boundary. The native worker mints both the authoritative job id and request timestamp before spawning the helper. The native process passes those values in the helper envelope, and helper status is accepted only when both `jobId` and `requestedAt` exactly match the native values; either mismatch wakes the existing process-control owner and fails closed before the status can reach the native job store. Before the directly owned child exits, only `Running` progress may be emitted. `Succeeded` or `Failed` is retained privately by the stdout reader and can become final native truth only after the child has exited successfully and both bounded readers have joined. Helper-authored `Queued` state, or any status after a terminal record, is treated as protocol failure. This prevents a helper from re-keying another concurrent job, replacing native request-time authority, making a still-running process appear complete, or making cancellation ineligible before native process ownership has actually settled. + +A fresh protocol review found that the Python CLI previously minted its own UTC `requestedAt` after receiving the native job envelope, while the native stdout admission checked only `jobId`. That split allowed untrusted helper time to replace the timestamp already minted by `start_analysis_job`. Rust RED `082cf63ee9007d0500b412e81a2910ac18b90e7a` requires the native envelope to carry `requestedAt` and requires exact timestamp admission before a helper status can enter the native status channel. Python RED `a253c099cb6781065f667e140968ab2a5fceeb91` requires `--progress-jsonl` to preserve a supplied native timestamp. Python production `9358035e96a091a9096e1760ed583a5a2dea190f` consumes and echoes that envelope timestamp while retaining current-time fallback only for standalone/manual callers that omit it. Native production `70e4d9da09ab4d05c07afde0ea38adddc3265886` sends the timestamp and rejects mismatched helper status before publication. RED and fixes were consecutive ordinary descendants, so no hosted RED failure is claimed for this timestamp slice. + +Typed deserialization alone is not the whole status contract. The TypeScript shared validator already constrains `progressPercent` to the integer range 0..=100, forbids `result`/`error` on queued or running state, requires `result` and forbids `error` on succeeded state, and requires `error` and forbids `result` on failed state. Rust's structural `AnalysisJobStatus` deserialization used `u8` plus optional payload fields, so values 101..=255 and contradictory/missing terminal payloads could deserialize successfully. Hosted RED `e3a6a20a69581dd16d08668162b391892ad55573` failed exactly because no native semantic validator existed. Production `2dbab44bf9a8a18b184a5371a6b7ad7d1efcd459` adds one native status-payload admission predicate and applies it before either running status publication or terminal-candidate retention. Invalid typed status now wakes the same reader-failure/process-control owner as malformed JSON; no raw helper payload is surfaced to the renderer or logs. + +Malformed JSONL remains fail-closed at that same boundary. `--progress-jsonl` owns stdout as one `AnalysisJobStatus` JSON object per line; Python logging is configured separately on stderr. The native reader therefore uses an explicit typed parse branch rather than treating a malformed or schema-invalid stdout line as ignorable diagnostics. Any deserialization failure marks the protocol rejected and wakes the existing reader-failure channel. The single process-control owner then terminates/reaps the helper and returns the existing payload-free unavailable result. A later valid terminal line cannot rehabilitate an earlier protocol violation. + +Protocol rejection must also survive direct-child exit ordering. The MPSC signal is the low-latency wake-up path while the child remains alive, but `try_wait()` and the reader notification are independent observations. Before `ea9428cc6c9f61c7448959ed04f22a52ba013deb`, a valid terminal candidate followed by a rejected record could leave the stdout thread with `(Some(candidate), Ok(()))`; if `try_wait()` observed the child's successful exit before the process-control loop consumed the failure channel, post-exit reader join could still present that rejected stream as successful. RED `64cb603b128af1ec362932383ddca6a1b73f1c8b` adds a deterministic Tauri source contract requiring rejection to remain in the join result. Production `ea9428cc6c9f61c7448959ed04f22a52ba013deb` maps `protocol_rejected` to `std::io::ErrorKind::InvalidData` before the stdout reader returns, while retaining the existing MPSC wake-up. The join therefore fails closed regardless of which observation wins the race. RED and fix were consecutive ordinary descendants, so no hosted RED failure is claimed. + +The bounded line transport must not normalize protocol bytes before that parser gets authority. RED `e444365970a273e19c6c134064108bc0a8d59152` proves that leading/trailing payload whitespace other than the JSONL line terminator is preserved, including U+00A0 NO-BREAK SPACE. The predecessor used Rust `str::trim()`, which is Unicode-aware and could strip characters that RFC 8259 does not admit as insignificant JSON whitespace, thereby converting malformed helper output into a parseable status. Production `6258b97cc5c0981c8617c53f8b27375f882e3631` removes only the transport LF and optional preceding CR. ASCII JSON whitespace remains for `serde_json` to judge under the JSON grammar, while non-JSON Unicode whitespace is no longer laundered by Resource Admission. + +Empty transport records need the same parser authority. JSON Lines requires each physical line to contain a valid JSON value and explicitly notes that a blank line is not a valid value. Before `f8135e0ab691f075c3bb057ea4f52732c036a6fd`, `read_bounded_process_lines` silently skipped an empty payload after removing the CR/LF terminator. That could make a malformed `--progress-jsonl` stream with an inserted blank record appear valid to the downstream typed parser. RED `161de8d6f2d9a1c9f0f6636983191953e07f3b57` changes the bounded-line regression to require `"first\n\n second \n"` to preserve the empty middle record. Production `f8135e0ab691f075c3bb057ea4f52732c036a6fd` removes only the empty-record filter and always invokes the callback after byte admission, so the analysis parser sees `""`, rejects it as non-JSON, wakes the existing process-control owner, and cannot be rehabilitated by a later terminal status. The RED workflow was superseded/cancelled by the ordinary production descendant before Tauri execution, so no hosted RED failure is claimed. + +Executable regressions cover the process-control terminal shapes on Linux/macOS and the output/time boundary: a parent that is still running when the product timeout expires, a parent that exits successfully while a five-second descendant retains inherited pipes, an oversized helper-output stream that must be rejected instead of being retained until JSON metadata parsing, and a helper that emits one byte beyond the output ceiling and then keeps running. The Tauri terminal-containment contract additionally requires `run_analysis_engine` to invoke that same cleanup owner in its direct-parent terminal arm before joining inherited stdout/stderr readers, so the adapter cannot silently diverge from the shared executable owner again. `oversized_process_output_wakes_before_coarse_poll_interval` deliberately delays overflow until after the parent has entered a two-second ordinary poll interval and requires fail-closed return in under one second; this proves reader failure is an event wake-up rather than merely a condition sampled at polling cadence. `process_timeout_does_not_oversleep_poll_interval` separately gives the helper a 100 ms deadline with a two-second poll interval and requires timeout return in under one second. Static Tauri protocol regressions require native job-id and request-time validation before status publication, prohibit helper terminal/queued state from entering the pre-exit progress channel, require malformed/schema-invalid JSONL to signal the existing process owner instead of being silently ignored, require state/payload semantics plus the 0..=100 progress range to pass native admission before status mutation, and require protocol rejection to remain a failed reader join result when direct-child exit wins the race with the side-channel notification. The Python CLI regression separately requires a native `requestedAt` envelope value to survive helper progress emission unchanged, while preserving manual fallback for callers outside the native product path. The bounded-line unit regression requires empty physical records to reach the protocol callback rather than being erased below the parser. These are process/pipe/output/deadline/protocol contracts, not proof of bounded whole-process RSS, temp-file cleanup, sandbox isolation, or hard real-time scheduling. + +## Authoritative evidence and traceability + +| Evidence | BandScope decision | +| --- | --- | +| Rust's Unix `CommandExt::process_group` configures a child's process group as part of command setup before execution. | Establish the BandScope-owned process group before `exec` rather than attempting to retrofit group membership after helper descendants may already exist. | +| POSIX.1-2024 `kill()` defines a negative PID other than `-1` as signalling processes whose process-group id equals the absolute value. | Use one narrow reviewed C ABI binding to signal the owned Linux/macOS group; no PID or signal value crosses IPC. | +| Rust 1.98.1 `std::io::Take` is a reader adapter that limits how many bytes can be read from the underlying reader. | Read at most the configured output ceiling plus one probe byte; reject when the probe demonstrates overflow instead of allowing `read_to_end` to grow an unbounded `Vec`. | +| Rust 1.98.1 `std::sync::mpsc` provides multi-producer/single-consumer channels, cloneable senders, and bounded waiting through `Receiver::recv_timeout`. | Let either stdout or stderr reader wake the single process-control owner immediately on an admission failure while ordinary child-status polling retains its configured cadence. The process-control owner remains the only code that terminates/reaps the helper. | +| Rust 1.98.1 `std::time::Instant` is a monotonic time measurement and `saturating_duration_since` yields the non-negative duration remaining between instants. `std::cmp::min` returns the smaller ordered value. | Compute the remaining deadline before each channel wait and wait for `min(poll_interval, remaining)` so configured poll granularity cannot itself extend the owned helper's deadline. | +| RFC 8259 defines a JSON text as `ws value ws` and insignificant JSON whitespace only as space (`%x20`), horizontal tab (`%x09`), line feed (`%x0A`) and carriage return (`%x0D`). | Remove only the JSONL transport CR/LF delimiter before typed parsing. Do not Unicode-trim payload content, because transport normalization must not turn a non-JSON character such as U+00A0 into valid protocol input. | +| JSON Lines requires each line to be a valid JSON value and explicitly distinguishes `null` as valid from a blank line as invalid. | Preserve empty physical records through bounded transport so the JSON parser, not Resource Admission, rejects the protocol violation. | +| Rust 1.98.1 documents `ChildExt` process-group signalling helpers but marks them as the unstable `unix_send_signal` API. | Do not make a nightly-only standard-library API a production dependency; retain the narrow stable-compatible `kill()` binding until a stable equivalent is available and adopted deliberately. | +| Microsoft documents that processes assigned to a Job Object are managed as a unit and that `JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE` terminates associated processes when the final job handle closes. | Windows requires a dedicated Job Object owner for analysis/import descendants before BandScope can claim process-tree containment. | +| `AssignProcessToJobObject` assigns an already-created process to a job. | A post-spawn assignment is not accepted as race-free proof when the Python process could create descendants before assignment. The Windows implementation must prevent execution through that pre-assignment window or use an equivalent creation boundary. | + +## Claim boundary + +The Linux/macOS guarantee covers ordinary descendants that retain the BandScope-owned process group. A descendant that deliberately invokes `setsid()` or changes process group escapes this narrow mechanism and is outside the claim. The mechanism is not a sandbox, container, seccomp/AppArmor/SELinux boundary, nor an egress policy owner. + +The 1 MiB stdout and 1 MiB stderr ceilings bound only bytes retained by BandScope's parent-side capture buffers. Once either reader detects overflow or a read failure, the channel wakes the process-control owner without waiting for the next ordinary poll; the owner then terminates the helper boundary and joins readers. The same wait is clamped to the monotonic deadline, so coarse caller polling cannot add an avoidable full-interval delay to timeout handling either. This narrows wasted post-violation and post-deadline compute time but still does not establish a whole-process memory limit and does not replace CPU, RAM, PID, disk, or accelerator controls. + +The native `jobId` and `requestedAt`, typed deserialization, state ordering and state/payload checks establish protocol authority only. Native admission enforces the shared progress range and terminal payload presence/exclusion rules, and helper status cannot replace the native request timestamp. Protocol rejection is retained both as an immediate process-owner wake-up and as a failed stdout-reader join result, so direct-child exit ordering cannot promote a stream already rejected by the parser/semantic boundary. Empty physical records and non-JSON Unicode whitespace are deliberately preserved through the bounded transport so parser rejection cannot be bypassed by transport normalization. None of these checks proves that helper-provided progress labels, `updatedAt`, cache metadata, rehearsal results, or error semantics are scientifically correct. Those values remain typed untrusted helper output subject to their domain validation and downstream real-audio acceptance. The checks also do not turn the Python helper into a trusted process; process, output, cancellation and resource boundaries remain independently required. + +Windows remains direct-child-only in the current production slice. The next platform prerequisite is race-free Job Object creation/assignment with kill-on-close semantics and tests that exercise an actual descendant, inherited handles, cancellation, normal parent exit, timeout, and cleanup. + +Commercial acceptance remains broader than these synthetic process regressions. Rights-cleared full-length rehearsal audio must still measure cancellation latency, inherited pipe/handle release, child-created temporary-artifact cleanup, decoder/resampler/downstream peak RSS and accelerator memory, and explicit per-job CPU/GPU budgets. Synthetic shell descendants establish the process-control invariant only; they do not substitute for real-audio scientific or buyer acceptance. + +## References + +Bray, T. (2017). *The JavaScript Object Notation (JSON) Data Interchange Format (RFC 8259).* Internet Engineering Task Force. https://www.rfc-editor.org/rfc/rfc8259 + +IEEE & The Open Group. (2024). *The Open Group Base Specifications Issue 8 / IEEE Std 1003.1-2024: kill().* https://pubs.opengroup.org/onlinepubs/9799919799/functions/kill.html + +JSON Lines. (n.d.). *JSON Lines.* Retrieved September 9, 2026, from https://jsonlines.org/ + +Microsoft. (2025). *Job Objects.* Microsoft Learn. https://learn.microsoft.com/en-us/windows/win32/procthread/job-objects + +Microsoft. (2022, July 26). *CreateJobObjectW function (jobapi2.h).* Microsoft Learn. https://learn.microsoft.com/en-us/windows/win32/api/jobapi2/nf-jobapi2-createjobobjectw + +Microsoft. (2021, October 12). *AssignProcessToJobObject function (jobapi2.h).* Microsoft Learn. https://learn.microsoft.com/en-us/windows/win32/api/jobapi2/nf-jobapi2-assignprocesstojobobject + +Rust Project Developers. (2026). *CommandExt in std::os::unix::process.* Rust standard library documentation. https://doc.rust-lang.org/stable/std/os/unix/process/trait.CommandExt.html + +Rust Project Developers. (2026). *Take in std::io (Rust 1.98.1).* Rust standard library documentation. https://doc.rust-lang.org/std/io/struct.Take.html + +Rust Project Developers. (2026). *std::sync::mpsc (Rust 1.98.1).* Rust standard library documentation. https://doc.rust-lang.org/std/sync/mpsc/ + +Rust Project Developers. (2026). *std::time (Rust 1.98.1).* Rust standard library documentation. https://doc.rust-lang.org/std/time/ + +Rust Project Developers. (2026). *min in core::cmp (Rust 1.98.0).* Rust standard library documentation. https://doc.rust-lang.org/stable/core/cmp/fn.min.html + +Rust Project Developers. (2026). *ChildExt in std::os::unix::process (Rust 1.98.1).* Rust standard library documentation. https://doc.rust-lang.org/std/os/unix/process/trait.ChildExt.html \ No newline at end of file diff --git a/docs/doctoring/youtube-process-containment.md b/docs/doctoring/youtube-process-containment.md new file mode 100644 index 000000000..55982300d --- /dev/null +++ b/docs/doctoring/youtube-process-containment.md @@ -0,0 +1,52 @@ +# YouTube subprocess containment evidence + +## Scope + +This note records the evidence and claim boundary for BandScope's timed YouTube-import subprocess lifecycle and the shared native subprocess-containment owner used by the analysis runner. It applies to the allowlisted `bandscope_analysis.youtube` process launched by the native desktop boundary, to the analysis engine process, and to ordinary descendants that retain the parent's process group. It is not a sandbox claim and does not establish Windows process-tree containment. + +## Problem and decision + +The existing `wait_for_process_output` timeout path killed and reaped only the directly spawned Python importer and then joined stdout/stderr reader threads. `bandscope_analysis.youtube` invokes yt-dlp with `FFmpegExtractAudio`, so a descendant could outlive the Python parent. If that descendant retained inherited output handles, the reader join could remain live after the product timeout. + +RED `65c94f72c1b87ce6a7be1c7f316409530330bf5e` starts a Unix shell that creates a five-second descendant while retaining inherited stdout/stderr, applies a 50 ms product timeout, and requires cleanup to return in under one second. Production `81a489026ce26fd14cf38bf1ea334e85d37346ee` configures the timed importer as leader of a fresh Unix process group before spawn and targets that group on timeout or wait failure before joining output readers. `9c36cf0a0b427e2383a128202db832b268d2f791` mirrors the regression through the Tauri shell test target so the native-shell test lane can execute the same boundary. + +Fresh DDD/SOLID review then found that the Tauri analysis runner and GUI-independent YouTube helper had equivalent low-level Unix process-group setup and termination implementations. RED `04e4d5c14d380ab97c246d5ecf501dece9f17c04` changes the native cancellation contract so the Tauri adapter must call the shared desktop-core process owner and explicitly rejects local POSIX signalling primitives. Production `25dd7a5c87bcd1b8d16ba316ce5d9cc1c53c24ff` removes the duplicate Tauri `CommandExt`/`kill(2)` implementation and routes analysis spawn/cancel/timeout/error cleanup through `configure_owned_process` and `terminate_owned_process` in `bandscope_desktop_core`. + +The chosen repair therefore has one GUI-independent OS mechanism: pre-`exec` process-group creation on Unix, negative-group signalling on Linux/macOS, direct-child reap, and direct-child kill/reap as fail-closed fallback if group signalling fails. The Tauri layer retains only orchestration and job-specific product authority. A generic renderer process-control API was rejected because the WebView must never receive PID, process-group, or arbitrary kill authority. + +No hosted RED failure is claimed for `04e4d5c1…`: the regression and production repair were consecutive ordinary descendants under cancel-in-progress. Exact-head hosted evidence must come from the final descendant, not from a predecessor run. + +## TRACEABILITY + +| Evidence | BandScope control | Claim boundary | +| --- | --- | --- | +| Rust 1.98.1 exposes Unix process extensions in `std::os::unix::process`; stable `CommandExt` is the supported pre-spawn extension point, while whole-process-group signalling on `ChildExt` remains experimental/nightly. | The GUI-independent desktop-core owner establishes the owned Unix process group before `spawn`/`exec`; Tauri and timed import consumers call that shared owner rather than carrying duplicate POSIX primitives. | Linux/macOS implementation evidence only; this is not Windows containment. | +| POSIX.1-2024 `kill()` defines a negative PID other than `-1` as targeting the process group whose id is the absolute value of that PID. | The owned child is made process-group leader, so signalling `-child.id()` targets ordinary descendants that retain that group. | Descendants that deliberately call `setsid()`/`setpgid()` or otherwise leave the group are outside the guarantee. | +| yt-dlp is configured with `FFmpegExtractAudio`, which may involve an external FFmpeg process during the import lifecycle. | The native timeout boundary contains the importer and ordinary helper descendants as one Unix process group before output-reader joins. | The regression uses an OS child process that retains pipes; it does not claim a rights-cleared real YouTube/FFmpeg commercial acceptance run. | +| Microsoft documents that Job Objects can associate child processes with a job and that `JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE` terminates associated processes when the final job handle closes. Microsoft also notes that resource operations performed before `AssignProcessToJobObject` are not retroactively covered. | The Windows design must create a dedicated Job Object and avoid an uncontained post-spawn window; if needed, create the process suspended, assign it, then resume. | Windows remains direct-child-only in current production code. | + +## Security Notes + +**Attack surface.** A public YouTube URL crosses the User Input, Network, Process, and Storage boundaries. The importer can invoke yt-dlp/FFmpeg and writes only inside the app-owned import workspace under the existing URL, byte-budget, lease, and path-containment controls. The analysis runner launches the local analysis engine under the same shared process-boundary primitive. + +**Threat.** A timeout that terminates only the Python parent can leave ordinary descendants consuming CPU, holding cache/temp files, or keeping stdout/stderr handles live. Separate low-level implementations also create drift risk: one path can receive a containment fix while another keeps weaker cancellation semantics. + +**Mitigation.** Establish containment before execution in one GUI-independent desktop-core owner, terminate the owned group before joining readers, reap the directly owned child, preserve direct-child fallback, and expose no process identifier or kill primitive to the renderer. Regression tests require the Tauri adapter to delegate to that owner and reject reintroduction of local POSIX signalling code. + +**Remaining risk.** Windows requires a race-free Job Object boundary. Unix descendants can intentionally escape the inherited process group. Real-audio/real-tool evidence is still required for cancellation latency, temp cleanup, pipe/handle release, and process-resource return. + +**Test points.** Execute the descendant-pipe regression on a Unix runner; run the Tauri shell cancellation-contract lane on the same exact head; verify timeout/error paths terminate before reader joins; retain cross-platform compilation; later add Windows Job Object descendant/handle tests and rights-cleared full-length real-audio/tool runs. + +## References + +Microsoft. (2022, July 26). *CreateJobObjectW function (jobapi2.h).* Microsoft Learn. https://learn.microsoft.com/en-us/windows/win32/api/jobapi2/nf-jobapi2-createjobobjectw + +Microsoft. (2021, October 12). *AssignProcessToJobObject function (jobapi2.h).* Microsoft Learn. https://learn.microsoft.com/en-us/windows/win32/api/jobapi2/nf-jobapi2-assignprocesstojobobject + +Microsoft. (2025). *Job Objects.* Microsoft Learn. https://learn.microsoft.com/en-us/windows/win32/procthread/job-objects + +Rust Project Developers. (2026). *std::os::unix::process (Rust 1.98.1)* [Documentation]. https://doc.rust-lang.org/std/os/unix/process/ + +Rust Project Developers. (2026). *ChildExt in std::os::unix::process (Rust 1.98.1)* [Documentation]. https://doc.rust-lang.org/std/os/unix/process/trait.ChildExt.html + +The IEEE & The Open Group. (2024). *The Open Group Base Specifications Issue 8 / IEEE Std 1003.1-2024: kill().* https://pubs.opengroup.org/onlinepubs/9799919799/functions/kill.html diff --git a/docs/security/app-security.md b/docs/security/app-security.md index a9983fb97..4d76f7251 100644 --- a/docs/security/app-security.md +++ b/docs/security/app-security.md @@ -95,6 +95,8 @@ Every boundary crossing requires validation, scope restriction, minimal logging, - Bind local backend only to `127.0.0.1` when a local HTTP surface exists. - Prefer direct IPC over a wider local HTTP surface when possible. - Allow only explicitly allowlisted IPC commands. +- For Tauri application commands, keep invoke-handler registration, `AppManifest::commands`, generated allow/deny permissions, and the window capability grant in sync. A registered handler without runtime-authority permission is not a usable product capability; a broad capability grant is not an acceptable repair. +- Job-control IPC must use BandScope-owned identifiers and typed operations. Do not expose PIDs, process handles, generic kill/exec commands, or arbitrary OS process authority to the WebView. - Validate all IPC and local backend payloads against strict schemas. - Reject unknown commands, unknown fields, and malformed payloads by default. - If a local HTTP service exists, consider per-session tokens or equivalent anti-cross-process protection. @@ -137,8 +139,11 @@ Every boundary crossing requires validation, scope restriction, minimal logging, - Cross-check extension, MIME, and actual decode behavior. - Prefer isolated worker processing for decode and analysis. - Guard against very large files, abnormal duration, and hostile metadata. +- Apply the versioned canonical local-audio resource policy consistently at request preflight and again at the opened-file/decoded-waveform boundary; request metadata is never authoritative for actual resource use. +- Before any decoder resamples, downmixes, or duration-truncates local audio, inspect source-container metadata from the already-open handle with `soundfile.info`, enforce the shared 8 kHz–192 kHz and mono/stereo source contract, reject overlong sources, and rewind the handle before `librosa.load`. +- In the Python analysis boundary, reject decoded audio that is empty, non-finite, wrong-rate, wrong-shaped, or over the accepted sample budget before beat tracking or model inference. Use the one-sample-over decode probe described in `docs/doctoring/audio-resource-policy.md` so an exact-boundary track remains accepted while excess decoded output is observable and fails closed. - Do not add arbitrary filesystem scanning just to find media files. -- When bootstrapping a project around local audio, prefer referencing the validated original file plus app-owned temp/cache/project roots over copying the file until persistence requirements justify the extra storage boundary. +- When bootstrapping a project around local audio, use the OS-selected external file only as untrusted admission input. Stage and sync admitted bytes under the app-owned project root, publish them as `source.`, then reopen and verify the published regular/non-symlink object against the bounded size and SHA-256 receipt before analysis or persistence. Do not persist an arbitrary external absolute path as authority. ### YouTube and remote URL import @@ -147,11 +152,17 @@ Every boundary crossing requires validation, scope restriction, minimal logging, - Validate scheme, host, path, and query before any fetch or handoff. - Do not widen URL intake into a generic remote downloader. - Sanitize remote metadata before display. +- Apply the same canonical 100 MiB encoded-byte ceiling during YouTube download as local-file intake. Abort with yt-dlp `max_filesize` and a progress hook, then delete owned `.part` / `.ytdl` / `-Frag*` siblings that stay inside that import directory. Do not keep a divergent post-download-only 50 MB limit that lets a large transfer fill the cache root first. +- Revalidate the filesystem-observed downloaded length before storing bootstrap state. Treat announced `filesize` / `filesize_approx` as a pre-download hint only. ### Subprocesses and native tools - Use fixed command templates plus allowlisted arguments. - Apply timeout, output path restriction, and resource bounds where possible. +- Cancellation and timeout must terminate and reap the owned execution boundary rather than merely flip a UI flag or kill one parent while inherited resources stay live. On Linux/macOS, `bandscope_desktop_core` is the single GUI-independent owner of process-group setup and termination. The analysis runner delegates pre-spawn configuration plus cancellation/timeout/error cleanup to that owner; the timed YouTube importer uses the same owner before spawning `bandscope_analysis.youtube`. Timeout/wait-error cleanup group-terminates ordinary yt-dlp/FFmpeg descendants before stdout/stderr reader joins, and a terminal direct-parent status also triggers residual same-group cleanup before those joins so a successful parent cannot hang behind an inherited pipe held by an outliving descendant. Executable regressions cover both the 50 ms timeout path and the successful-parent/five-second-descendant pipe-retention path. The Tauri cancellation contract also rejects reintroduction of local POSIX signalling primitives. These guarantees cover ordinary descendants that remain in the group, not descendants that deliberately call `setsid()`/`setpgid()` or otherwise leave it. +- Treat stdout and stderr from owned helper processes as untrusted resource input. Retain at most 1 MiB per stream plus one overflow-probe byte; overflow or read failure must fail closed before stdout is parsed as metadata. This caps BandScope's parent-side capture memory only and is not evidence of bounded helper RSS/VRAM. +- Windows remains direct-child-only until a race-free Job Object creation/assignment boundary is implemented in the shared process owner. Do not claim commercial process-tree cleanup until Windows containment and rights-cleared full-length real-audio evidence demonstrate inherited-handle/pipe release, temp cleanup, bounded cancellation latency, and resource return. +- Keep `docs/doctoring/subprocess-containment.md` synchronized with this boundary and the executable regressions; process-group control and bounded pipe capture are not sandbox/container isolation and do not replace whole-process resource measurement. - Redact sensitive paths and tokens from surfaced stderr or stdout. - Track tool versions and their supply chain source. @@ -213,6 +224,7 @@ Every boundary crossing requires validation, scope restriction, minimal logging, - Allow only required plugins and scopes. - Keep filesystem, network, and shell scopes minimal. +- Keep application-command manifests and capability permissions explicit and narrow; capability configuration is part of the IPC security boundary, not packaging-only metadata. - Validate command handler payloads with explicit types. - Do not load remote content into a privileged Tauri context. diff --git a/packages/shared-types/src/index.ts b/packages/shared-types/src/index.ts index cba4606a2..fa805afe4 100644 --- a/packages/shared-types/src/index.ts +++ b/packages/shared-types/src/index.ts @@ -238,7 +238,7 @@ export type AnalysisJobStage = "queued" | "decode" | "separate" | "analyze" | "p /** Documented. */ export type AnalysisCacheStatus = "disabled" | "miss" | "hit" | "stored"; /** Documented. */ -export type AnalysisJobErrorCode = "invalid_request" | "not_found" | "engine_unavailable"; +export type AnalysisJobErrorCode = "invalid_request" | "not_found" | "engine_unavailable" | "cancelled"; /** Documented. */ export type LocalAudioSource = { @@ -365,7 +365,7 @@ const ANALYSIS_SOURCE_KINDS = ["demo", "local_audio"] as const; const ANALYSIS_JOB_STATES = ["queued", "running", "succeeded", "failed"] as const; const ANALYSIS_JOB_STAGES = ["queued", "decode", "separate", "analyze", "persist", "ready"] as const; const ANALYSIS_CACHE_STATUSES = ["disabled", "miss", "hit", "stored"] as const; -const ANALYSIS_JOB_ERROR_CODES = ["invalid_request", "not_found", "engine_unavailable"] as const; +const ANALYSIS_JOB_ERROR_CODES = ["invalid_request", "not_found", "engine_unavailable", "cancelled"] as const; const PACK_STATES = ["queued", "analyzing", "ready", "failed"] as const; const HANDOFF_ASSET_STATUSES = ["referenced", "missing"] as const; diff --git a/packages/shared-types/test/analysis_job_cancellation.test.ts b/packages/shared-types/test/analysis_job_cancellation.test.ts new file mode 100644 index 000000000..5da00223f --- /dev/null +++ b/packages/shared-types/test/analysis_job_cancellation.test.ts @@ -0,0 +1,20 @@ +import { isAnalysisJobStatus, parseAnalysisJobStatus } from "../src/index"; + +describe("analysis cancellation contract", () => { + it("accepts the native cancelled error code as a terminal failed status", () => { + const cancelledStatus = { + jobId: "job-42", + state: "failed", + requestedAt: "2026-09-07T07:00:00Z", + updatedAt: "2026-09-07T07:00:01Z", + progressLabel: "Analysis cancelled", + error: { + code: "cancelled", + message: "Analysis was cancelled." + } + }; + + expect(isAnalysisJobStatus(cancelledStatus)).toBe(true); + expect(parseAnalysisJobStatus(cancelledStatus).error?.code).toBe("cancelled"); + }); +}); diff --git a/services/analysis-engine/src/bandscope_analysis/__init__.py b/services/analysis-engine/src/bandscope_analysis/__init__.py index 3867248e8..ce4beb801 100644 --- a/services/analysis-engine/src/bandscope_analysis/__init__.py +++ b/services/analysis-engine/src/bandscope_analysis/__init__.py @@ -1,6 +1,36 @@ """BandScope analysis engine package.""" -from .api import get_analysis_status +import logging +from importlib import import_module + from .health import build_health_report +_STEM_SAFE_FAILURE_LOG_MESSAGES = frozenset( + { + "Stem separation failed because the source file was missing.", + "Stem separation unavailable because Demucs or torch is not installed.", + "Stem separation rejected invalid audio source data.", + "Stem separation failed with a runtime error.", + "Stem separation failed unexpectedly.", + "Stem separation failed before analysis job completion.", + } +) + + +class _ApiDiagnosticPrivacyFilter(logging.Filter): + """Redact traceback payloads only for known stem safe-failure diagnostics.""" + + def filter(self, record: logging.LogRecord) -> bool: + """Preserve unrelated diagnostics while redacting owned safe-failure tracebacks.""" + if record.getMessage() in _STEM_SAFE_FAILURE_LOG_MESSAGES: + record.exc_info = None + record.exc_text = None + return True + + +_api_logger = logging.getLogger("bandscope_analysis.api") +_api_logger.addFilter(_ApiDiagnosticPrivacyFilter()) +_api_module = import_module(".api", __name__) +get_analysis_status = _api_module.get_analysis_status + __all__ = ["build_health_report", "get_analysis_status"] diff --git a/services/analysis-engine/src/bandscope_analysis/api.py b/services/analysis-engine/src/bandscope_analysis/api.py index b376de293..38c441f19 100644 --- a/services/analysis-engine/src/bandscope_analysis/api.py +++ b/services/analysis-engine/src/bandscope_analysis/api.py @@ -10,15 +10,20 @@ import time from contextlib import suppress from pathlib import Path -from typing import Any, Literal, NotRequired, TypedDict, cast +from typing import Any, Literal, NotRequired, TypedDict, cast, get_args import numpy as np +from bandscope_analysis.audio_resource_policy import DEFAULT_AUDIO_RESOURCE_POLICY +from bandscope_analysis.feature_cache_admission import ( + load_bounded_stem_archive, + read_bounded_feature_cache_metadata, +) from bandscope_analysis.health import HealthReport, build_health_report from bandscope_analysis.roles import RoleExtractor from bandscope_analysis.sections import extract_sections from bandscope_analysis.sections.segmenter import segment_with_boundaries -from bandscope_analysis.separation import AudioStemSeparator +from bandscope_analysis.separation import AudioStemName, AudioStemSeparator logger = logging.getLogger(__name__) @@ -26,6 +31,7 @@ ANALYSIS_CACHE_SCHEMA_VERSION = 1 FEATURE_CACHE_SCHEMA_VERSION = 1 STEM_SEPARATION_TIMEOUT_SECONDS = 20.0 +_CANONICAL_AUDIO_STEM_NAMES = frozenset(get_args(AudioStemName)) logger = logging.getLogger(__name__) @@ -86,7 +92,7 @@ class RangePayload(TypedDict): class HarmonyPayload(TypedDict): - """Typed harmony payload nested inside rehearsal results.""" + """Typed harmony payload nested inside rehearsal roles.""" chord: str functionLabel: str @@ -299,6 +305,7 @@ def validate_analysis_job_request(payload: object) -> AnalysisJobRequest: if not isinstance(source_path, str) or not source_path.strip(): raise ValueError("Invalid analysis job request: invalid field 'localSource.sourcePath'") if ".." in source_path.replace("\\", "/").split("/"): + logger.warning("Security: path traversal detected in localSource.sourcePath") raise ValueError( "Invalid analysis job request: path traversal detected in 'localSource.sourcePath'" ) @@ -306,8 +313,12 @@ def validate_analysis_job_request(payload: object) -> AnalysisJobRequest: raise ValueError("Invalid analysis job request: invalid field 'localSource.fileName'") if extension not in {"wav", "mp3", "flac", "m4a"}: raise ValueError("Invalid analysis job request: invalid field 'localSource.extension'") - if not isinstance(file_size_bytes, int) or file_size_bytes <= 0: - raise ValueError("Invalid analysis job request: invalid field 'localSource.fileSizeBytes'") + try: + file_size_bytes = DEFAULT_AUDIO_RESOURCE_POLICY.validate_encoded_file_bytes(file_size_bytes) + except ValueError as error: + raise ValueError( + "Invalid analysis job request: invalid field 'localSource.fileSizeBytes'" + ) from error normalized: AnalysisJobRequest = { "sourceKind": source_kind, @@ -702,31 +713,25 @@ def _default_stem_role_types(stem_keys: list[str]) -> dict[str, str]: def _normalize_stem_role_types( stem_role_types: object, stem_keys: list[str] ) -> dict[str, str] | None: - """Validate role metadata while preserving compatibility with older caches.""" + """Validate persisted role metadata against canonical stem semantics.""" + canonical = _default_stem_role_types(stem_keys) if stem_role_types is None: - return _default_stem_role_types(stem_keys) + return canonical if not isinstance(stem_role_types, dict): return None - - normalized: dict[str, str] = {} - for stem_key in stem_keys: - role_type = stem_role_types.get(stem_key) - if role_type not in ("vocal", "instrument"): - return None - normalized[stem_key] = role_type - return normalized + if set(stem_role_types) != set(stem_keys): + return None + if any(stem_role_types.get(stem_key) != canonical[stem_key] for stem_key in stem_keys): + return None + return canonical def _load_cached_local_audio_features( metadata_path: Path, arrays_path: Path ) -> dict[str, Any] | None: """Load cached stem/features payload, treating malformed files as cache misses.""" - try: - with metadata_path.open("r", encoding="utf-8") as metadata_file: - metadata_payload = json.load(metadata_file) - except (OSError, json.JSONDecodeError): - return None - if not isinstance(metadata_payload, dict): + metadata_payload = read_bounded_feature_cache_metadata(metadata_path) + if metadata_payload is None: return None if metadata_payload.get("schemaVersion") != FEATURE_CACHE_SCHEMA_VERSION: return None @@ -735,6 +740,15 @@ def _load_cached_local_audio_features( separation = metadata_payload.get("separation") if not isinstance(separation, dict): return None + duration_seconds = separation.get("duration_seconds") + if isinstance(duration_seconds, bool) or not isinstance(duration_seconds, (int, float)): + return None + try: + duration_value = float(duration_seconds) + except (OverflowError, ValueError): + return None + if not np.isfinite(duration_value) or duration_value <= 0: + return None stem_keys = metadata_payload.get("stemKeys") if not isinstance(stem_keys, list) or not stem_keys: return None @@ -744,18 +758,13 @@ def _load_cached_local_audio_features( if stem_role_types is None: return None - try: - with np.load(arrays_path, allow_pickle=False) as stems_archive: - stems: dict[str, np.ndarray] = {} - for stem_key in stem_keys: - archive_key = f"stem_{stem_key}" - if archive_key not in stems_archive: - return None - stem_array = stems_archive[archive_key] - if not isinstance(stem_array, np.ndarray): - return None - stems[stem_key] = stem_array - except (OSError, ValueError): + stems = load_bounded_stem_archive( + arrays_path, + stem_keys, + metadata_payload["sampleRate"], + policy_template=DEFAULT_AUDIO_RESOURCE_POLICY, + ) + if stems is None: return None return { @@ -763,7 +772,7 @@ def _load_cached_local_audio_features( "sr": metadata_payload["sampleRate"], "stem_role_types": stem_role_types, "separation": { - "duration_seconds": separation.get("duration_seconds"), + "duration_seconds": duration_value, "chunk_count": separation.get("chunk_count"), "notes": separation.get("notes"), }, @@ -781,6 +790,8 @@ def _serialize_stem_arrays(stems: object) -> dict[str, np.ndarray] | None: return None if not stem_name.isidentifier(): return None + if stem_name not in _CANONICAL_AUDIO_STEM_NAMES: + return None if not isinstance(stem_value, np.ndarray): return None serialized_stems[f"stem_{stem_name}"] = stem_value diff --git a/services/analysis-engine/src/bandscope_analysis/audio_decode.py b/services/analysis-engine/src/bandscope_analysis/audio_decode.py new file mode 100644 index 000000000..310c6764e --- /dev/null +++ b/services/analysis-engine/src/bandscope_analysis/audio_decode.py @@ -0,0 +1,185 @@ +"""Bounded PCM decode port for the Resource Admission & Decode context. + +The current adapter still delegates to ``librosa`` and therefore remains a +transitional boundary while #1129 removes the libsndfile-backed runtime graph. +Consumers must call this port rather than selecting decoder fallbacks +independently. + +Security Notes: +- The caller-authorized binary handle, container metadata, decoder output, and + third-party decoder exceptions are untrusted. +- Encoded byte size is measured and admitted from the caller-owned seekable + handle before metadata parsing or decode work begins. +- Metadata parsing and decoding receive a bounded view whose logical EOF is the + admitted byte count, so source growth after admission cannot widen decoder + read authority beyond the resource-policy ceiling. +- Source metadata is admitted before decode and the resulting PCM is revalidated + against the same versioned policy before it can enter MIR or model work. +- Decoder output must already be one-dimensional when ``mono=True``; a malformed + multi-channel shape is rejected rather than flattened into false mono PCM. +- Decoder output must remain floating-point before canonicalization. Integer, + boolean, object, or complex results are treated as a malformed decoder contract + rather than silently reinterpreted as real-valued rehearsal PCM. +- The decoder call explicitly pins canonical ``float32`` output and the + band-limited ``soxr_hq`` resampler so changes to third-party defaults cannot + silently change those selected decode parameters. Numerical output can still + change when the decoder/resampler implementation or dependency versions change. +- Decoder sample count, visible allocated bytes, and the predicted canonical + float32 byte count are checked before normalization. A non-owning NumPy view + is detached into an owned canonical buffer, so the returned MIR artifact cannot + retain a larger hidden backing allocation. If that bounded canonical copy + itself fails under host memory pressure, the port preserves the stable + ``memory_budget_exceeded`` resource-policy contract. Earlier decoder-output + materialization failures remain malformed-decoder failures rather than being + misclassified as an admitted canonical-buffer budget failure. Decoder-internal + peak memory before return remains a separate process-resource acceptance boundary. +- Decoder details remain exception causes only; the surfaced failure is the + payload-free canonical resource-policy error. +- This port adds no path, network, subprocess, or credential authority. +""" + +from __future__ import annotations + +import io +import warnings +from typing import BinaryIO, cast + +import librosa +import numpy as np +from numpy.typing import NDArray + +from bandscope_analysis.audio_metadata import preflight_audio_metadata +from bandscope_analysis.audio_resource_policy import ( + DEFAULT_AUDIO_RESOURCE_POLICY, + AudioResourcePolicy, + AudioResourcePolicyError, +) + +AudioMonoArray = NDArray[np.float32] +_CANONICAL_RESAMPLE_TYPE = "soxr_hq" +_CANONICAL_PCM_ITEMSIZE = np.dtype(np.float32).itemsize + + +class _BoundedEncodedSource(io.RawIOBase): + """Expose one seekable source with a fixed admitted logical end offset.""" + + def __init__(self, source: BinaryIO, admitted_bytes: int) -> None: + """Retain the caller-owned handle without acquiring path authority.""" + super().__init__() + self._source = source + self._admitted_bytes = admitted_bytes + + def readable(self) -> bool: + """Report the read capability required by decoder virtual I/O.""" + return True + + def seekable(self) -> bool: + """Report the seek capability required by container parsers.""" + return True + + def tell(self) -> int: + """Return the current position of the caller-owned handle.""" + return int(self._source.tell()) + + def seek(self, offset: int, whence: int = io.SEEK_SET) -> int: + """Seek relative to the admitted logical file rather than later growth.""" + if whence == io.SEEK_SET: + target = offset + elif whence == io.SEEK_CUR: + target = self.tell() + offset + elif whence == io.SEEK_END: + target = self._admitted_bytes + offset + else: + raise ValueError("invalid seek mode") + if target < 0: + raise OSError("invalid encoded source seek") + return int(self._source.seek(target, io.SEEK_SET)) + + def read(self, size: int = -1) -> bytes: + """Read no farther than the byte extent admitted before decode.""" + remaining = max(self._admitted_bytes - self.tell(), 0) + bounded_size = remaining if size < 0 else min(size, remaining) + return self._source.read(bounded_size) + + def readinto(self, buffer: bytearray | memoryview) -> int: + """Fill decoder-owned buffers without crossing the admitted EOF.""" + data = self.read(len(buffer)) + buffer[: len(data)] = data + return len(data) + + +def _malformed_decode_error() -> AudioResourcePolicyError: + """Build the stable payload-free decoder failure.""" + return AudioResourcePolicyError("malformed_header") + + +def _measure_encoded_source_bytes(source: BinaryIO) -> int: + """Measure one seekable encoded source and restore it to the decode origin.""" + try: + source.seek(0, 2) + file_size = source.tell() + source.seek(0) + except Exception as error: + raise _malformed_decode_error() from error + return file_size + + +def decode_mono_audio( + source: BinaryIO, + *, + policy: AudioResourcePolicy = DEFAULT_AUDIO_RESOURCE_POLICY, +) -> tuple[AudioMonoArray, int]: + """Admit and decode one caller-owned source to bounded mono float32 PCM.""" + admitted_bytes = policy.validate_encoded_file_bytes(_measure_encoded_source_bytes(source)) + admitted_source = cast(BinaryIO, _BoundedEncodedSource(source, admitted_bytes)) + preflight_audio_metadata(admitted_source, policy) + + try: + with warnings.catch_warnings(): + warnings.filterwarnings("ignore", category=DeprecationWarning, module=r"^audioread") + warnings.filterwarnings("ignore", category=FutureWarning, module=r"^audioread") + decoded, sample_rate = librosa.load( + admitted_source, + sr=policy.target_sample_rate, + mono=True, + duration=policy.decode_probe_duration_seconds, + dtype=np.float32, + res_type=_CANONICAL_RESAMPLE_TYPE, + ) + except AudioResourcePolicyError: + raise + except Exception as error: + raise _malformed_decode_error() from error + + try: + decoded_array = np.asarray(decoded) + if decoded_array.ndim != 1: + raise _malformed_decode_error() + if not np.issubdtype(decoded_array.dtype, np.floating): + raise _malformed_decode_error() + if decoded_array.size > policy.max_decoded_samples: + raise AudioResourcePolicyError("decoded_sample_count_exceeded") + if decoded_array.nbytes > policy.max_decoded_audio_bytes: + raise AudioResourcePolicyError("memory_budget_exceeded") + canonical_bytes = decoded_array.size * _CANONICAL_PCM_ITEMSIZE + if canonical_bytes > policy.max_decoded_audio_bytes: + raise AudioResourcePolicyError("memory_budget_exceeded") + if decoded_array.dtype == np.dtype(np.float32) and decoded_array.flags.owndata: + pcm = decoded_array + else: + try: + pcm = np.array(decoded_array, dtype=np.float32, copy=True) + except MemoryError as error: + raise AudioResourcePolicyError("memory_budget_exceeded") from error + except AudioResourcePolicyError: + raise + except (MemoryError, OverflowError, TypeError, ValueError) as error: + raise _malformed_decode_error() from error + + try: + policy.validate_decoded_audio(pcm, sample_rate) + except AudioResourcePolicyError: + raise + except Exception as error: + raise _malformed_decode_error() from error + return cast(AudioMonoArray, pcm), int(sample_rate) diff --git a/services/analysis-engine/src/bandscope_analysis/audio_metadata.py b/services/analysis-engine/src/bandscope_analysis/audio_metadata.py new file mode 100644 index 000000000..51e0a6176 --- /dev/null +++ b/services/analysis-engine/src/bandscope_analysis/audio_metadata.py @@ -0,0 +1,43 @@ +"""Bounded source-container metadata preflight for local audio decoders. + +Security Notes: +- The selected audio bytes and container headers are untrusted. +- This module reads metadata from an already-open caller-owned handle only; it + does not open paths, decode PCM, follow URLs, or allocate a waveform. +- Malformed headers, unsupported source rates/channels, and overlong sources + fail closed with the payload-free canonical policy error. +- A successful probe rewinds the handle so the downstream decoder receives the + same source from its beginning. +""" + +from __future__ import annotations + +from typing import BinaryIO + +import soundfile # type: ignore[import-untyped] # soundfile has no py.typed marker. + +from bandscope_analysis.audio_resource_policy import ( + DEFAULT_AUDIO_RESOURCE_POLICY, + AudioResourcePolicy, + AudioResourcePolicyError, +) + + +def preflight_audio_metadata( + fileobj: BinaryIO, + policy: AudioResourcePolicy = DEFAULT_AUDIO_RESOURCE_POLICY, +) -> None: + """Validate source metadata without decoding PCM and rewind the handle.""" + try: + fileobj.seek(0) + info = soundfile.info(fileobj) + fileobj.seek(0) + policy.validate_source_metadata( + frames=info.frames, + sample_rate=info.samplerate, + channels=info.channels, + ) + except AudioResourcePolicyError: + raise + except Exception as error: + raise AudioResourcePolicyError("malformed_header") from error diff --git a/services/analysis-engine/src/bandscope_analysis/audio_resource_policy.py b/services/analysis-engine/src/bandscope_analysis/audio_resource_policy.py new file mode 100644 index 000000000..c64feffe1 --- /dev/null +++ b/services/analysis-engine/src/bandscope_analysis/audio_resource_policy.py @@ -0,0 +1,299 @@ +"""Canonical resource admission policy for local audio analysis. + +The policy is intentionally independent of individual analyzers. Expensive +feature code consumes a decoded artifact only after encoded-file and decoded +output checks agree on the same versioned limits. This prevents temporal, +separation, chord, and register features from silently inventing incompatible +resource ceilings. + +Security Notes: +- Encoded byte counts are validated before decode/allocation work when the + opened file descriptor can provide an authoritative size. +- Decoded audio is revalidated because container metadata and decoder behavior + are untrusted; accepted artifacts are finite, mono, canonical float32, at the + configured sample rate, and within configured sample and memory budgets. +- The default decoded-memory ceiling is derived from the canonical float32 mono + representation used by the production decoder, rather than a wider host + default dtype that would silently double the admitted buffer footprint. +- Noncanonical floating buffers fail closed even when their byte count is below + the memory ceiling; downstream MIR therefore receives one reproducible PCM + representation rather than dtype-dependent numerical inputs. +- Finiteness validation scans the canonical PCM in chunks so the safety check + cannot allocate a second full-song boolean mask; each temporary finite mask + is capped at 1 MiB. +- Decoders receive a one-sample-over-budget probe duration so a longer source is + rejected instead of being silently truncated to the accepted duration. +- Policy arithmetic rejects unrepresentable limits before float/sample-count + conversion so malformed configuration cannot escape the stable failure mode. +- Resource rejections expose only a stable reason and policy version; messages + remain payload-free and never include source paths or audio content. +""" + +from __future__ import annotations + +import math +import sys +from dataclasses import dataclass +from typing import NoReturn, cast + +import numpy as np +from numpy.typing import NDArray + +AUDIO_RESOURCE_POLICY_VERSION = "3" +DEFAULT_TARGET_SAMPLE_RATE = 44_100 +DEFAULT_MIN_SOURCE_SAMPLE_RATE = 8_000 +DEFAULT_MAX_SOURCE_SAMPLE_RATE = 192_000 +DEFAULT_MIN_SOURCE_CHANNELS = 1 +DEFAULT_MAX_SOURCE_CHANNELS = 2 +DEFAULT_MAX_ENCODED_FILE_BYTES = 100 * 1024 * 1024 +DEFAULT_MAX_DURATION_SECONDS = 15 * 60 +DEFAULT_MAX_DECODED_AUDIO_BYTES = ( + DEFAULT_TARGET_SAMPLE_RATE * DEFAULT_MAX_DURATION_SECONDS * np.dtype(np.float32).itemsize +) +_POLICY_ERROR = "Audio input violates the audio resource policy." +_MAX_FINITE_CHECK_TEMP_BYTES = 1024 * 1024 + + +class AudioResourcePolicyError(ValueError): + """Payload-free resource rejection with stable machine-readable provenance.""" + + def __init__(self, reason: str) -> None: + """Record a stable rejection reason and the policy version that produced it.""" + super().__init__(_POLICY_ERROR) + self.reason = reason + self.policy_version = AUDIO_RESOURCE_POLICY_VERSION + + +def _reject(reason: str) -> NoReturn: + """Fail closed without echoing untrusted resource metadata.""" + raise AudioResourcePolicyError(reason) + + +def _all_samples_finite(audio: NDArray[np.float32]) -> bool: + """Check finiteness while bounding the temporary NumPy boolean mask.""" + finite_mask_itemsize = np.dtype(np.bool_).itemsize + chunk_samples = max(_MAX_FINITE_CHECK_TEMP_BYTES // finite_mask_itemsize, 1) + for start in range(0, audio.size, chunk_samples): + if not np.isfinite(audio[start : start + chunk_samples]).all(): + return False + return True + + +@dataclass(frozen=True) +class AudioResourcePolicy: + """Versioned limits applied before and after local audio decoding. + + Args: + max_encoded_file_bytes: Maximum non-empty encoded source size. + target_sample_rate: Required sample rate of the canonical decoded mono + artifact. + max_duration_seconds: Maximum decoded duration represented as a sample + ceiling at ``target_sample_rate``. + max_decoded_audio_bytes: Maximum in-memory byte size of the canonical + decoded mono NumPy buffer. + min_source_sample_rate: Minimum source-container sample rate accepted + before resampling. + max_source_sample_rate: Maximum source-container sample rate accepted + before resampling. + min_source_channels: Minimum source-container channel count accepted + before downmixing. + max_source_channels: Maximum source-container channel count accepted + before downmixing. + """ + + max_encoded_file_bytes: int = DEFAULT_MAX_ENCODED_FILE_BYTES + target_sample_rate: int = DEFAULT_TARGET_SAMPLE_RATE + max_duration_seconds: float = float(DEFAULT_MAX_DURATION_SECONDS) + max_decoded_audio_bytes: int = DEFAULT_MAX_DECODED_AUDIO_BYTES + min_source_sample_rate: int = DEFAULT_MIN_SOURCE_SAMPLE_RATE + max_source_sample_rate: int = DEFAULT_MAX_SOURCE_SAMPLE_RATE + min_source_channels: int = DEFAULT_MIN_SOURCE_CHANNELS + max_source_channels: int = DEFAULT_MAX_SOURCE_CHANNELS + + def __post_init__(self) -> None: + """Reject invalid policy configuration before it can weaken admission.""" + if ( + isinstance(self.max_encoded_file_bytes, bool) + or not isinstance(self.max_encoded_file_bytes, int) + or self.max_encoded_file_bytes <= 0 + or self.max_encoded_file_bytes > sys.maxsize - 1 + ): + raise ValueError(_POLICY_ERROR) + if ( + isinstance(self.target_sample_rate, bool) + or not isinstance(self.target_sample_rate, int) + or self.target_sample_rate <= 0 + or self.target_sample_rate > sys.maxsize - 1 + ): + raise ValueError(_POLICY_ERROR) + if isinstance(self.max_duration_seconds, bool) or not isinstance( + self.max_duration_seconds, int | float + ): + raise ValueError(_POLICY_ERROR) + if ( + isinstance(self.max_decoded_audio_bytes, bool) + or not isinstance(self.max_decoded_audio_bytes, int) + or self.max_decoded_audio_bytes <= 0 + or self.max_decoded_audio_bytes > sys.maxsize - 1 + ): + raise ValueError(_POLICY_ERROR) + for source_bound in ( + self.min_source_sample_rate, + self.max_source_sample_rate, + self.min_source_channels, + self.max_source_channels, + ): + if ( + isinstance(source_bound, bool) + or not isinstance(source_bound, int) + or source_bound <= 0 + or source_bound > sys.maxsize - 1 + ): + raise ValueError(_POLICY_ERROR) + if ( + self.min_source_sample_rate > self.max_source_sample_rate + or self.min_source_channels > self.max_source_channels + ): + raise ValueError(_POLICY_ERROR) + try: + duration_seconds = float(self.max_duration_seconds) + except (OverflowError, ValueError): + raise ValueError(_POLICY_ERROR) from None + if not math.isfinite(duration_seconds) or duration_seconds <= 0.0: + raise ValueError(_POLICY_ERROR) + decoded_samples = self.target_sample_rate * duration_seconds + if ( + not math.isfinite(decoded_samples) + or decoded_samples < 1.0 + or decoded_samples > sys.maxsize - 1 + ): + raise ValueError(_POLICY_ERROR) + + @property + def max_decoded_samples(self) -> int: + """Return the maximum mono sample count allowed after decoding.""" + return int(self.target_sample_rate * float(self.max_duration_seconds)) + + @property + def decode_probe_duration_seconds(self) -> float: + """Return a bounded decoder duration that includes one rejection probe sample.""" + return (self.max_decoded_samples + 1) / self.target_sample_rate + + def validate_encoded_file_bytes(self, file_size: object) -> int: + """Validate an authoritative encoded file size before decoding. + + Args: + file_size: Byte count obtained from the already-open source file. + + Returns: + The validated integer byte count. + + Raises: + AudioResourcePolicyError: If the value is not a positive integer + within policy. + """ + if isinstance(file_size, bool) or not isinstance(file_size, int) or file_size <= 0: + _reject("malformed_header") + if file_size > self.max_encoded_file_bytes: + _reject("encoded_file_too_large") + return file_size + + def validate_source_metadata( + self, + frames: object, + sample_rate: object, + channels: object, + ) -> None: + """Validate source-container metadata before any decode transformation. + + Args: + frames: Number of source frames reported by the container parser. + sample_rate: Source sample rate in Hz before resampling. + channels: Source channel count before downmixing. + + Raises: + AudioResourcePolicyError: If metadata is malformed or outside the + source bounds. + """ + if isinstance(frames, bool) or not isinstance(frames, int) or frames <= 0: + _reject("malformed_header") + if ( + isinstance(sample_rate, bool) + or not isinstance(sample_rate, int) + or sample_rate < self.min_source_sample_rate + or sample_rate > self.max_source_sample_rate + ): + _reject("sampling_rate_unsupported") + if ( + isinstance(channels, bool) + or not isinstance(channels, int) + or channels < self.min_source_channels + or channels > self.max_source_channels + ): + _reject("channel_count_unsupported") + try: + source_duration_seconds = float(frames) / float(sample_rate) + except (OverflowError, ValueError): + _reject("malformed_header") + if source_duration_seconds > float(self.max_duration_seconds): + _reject("duration_exceeded") + + def validate_decoded_audio( + self, + audio: object, + sample_rate: object, + ) -> NDArray[np.float32]: + """Revalidate the canonical decoded artifact before feature analysis. + + Args: + audio: Candidate mono NumPy array returned by the decoder. + sample_rate: Decoder-reported sample rate in Hz. + + Returns: + The original validated canonical float32 mono array without copying it. + + Raises: + AudioResourcePolicyError: If dtype, shape, sample rate, sample + count, memory use, or finiteness does not satisfy this policy. + """ + if ( + not isinstance(audio, np.ndarray) + or audio.ndim != 1 + or audio.size == 0 + or not np.issubdtype(audio.dtype, np.floating) + ): + _reject("malformed_header") + if ( + isinstance(sample_rate, bool) + or not isinstance(sample_rate, int) + or sample_rate != self.target_sample_rate + ): + _reject("sampling_rate_unsupported") + if audio.size > self.max_decoded_samples: + _reject("decoded_sample_count_exceeded") + if audio.nbytes > self.max_decoded_audio_bytes: + _reject("memory_budget_exceeded") + if audio.dtype != np.dtype(np.float32): + _reject("decoded_dtype_unsupported") + canonical_audio = cast(NDArray[np.float32], audio) + if not _all_samples_finite(canonical_audio): + _reject("malformed_header") + return canonical_audio + + +DEFAULT_AUDIO_RESOURCE_POLICY = AudioResourcePolicy() + +__all__ = [ + "AUDIO_RESOURCE_POLICY_VERSION", + "AudioResourcePolicy", + "AudioResourcePolicyError", + "DEFAULT_AUDIO_RESOURCE_POLICY", + "DEFAULT_MAX_DECODED_AUDIO_BYTES", + "DEFAULT_MAX_DURATION_SECONDS", + "DEFAULT_MAX_ENCODED_FILE_BYTES", + "DEFAULT_MAX_SOURCE_CHANNELS", + "DEFAULT_MAX_SOURCE_SAMPLE_RATE", + "DEFAULT_MIN_SOURCE_CHANNELS", + "DEFAULT_MIN_SOURCE_SAMPLE_RATE", + "DEFAULT_TARGET_SAMPLE_RATE", +] diff --git a/services/analysis-engine/src/bandscope_analysis/chords/chord_recognizer.py b/services/analysis-engine/src/bandscope_analysis/chords/chord_recognizer.py index 8f6466924..2d414afc8 100644 --- a/services/analysis-engine/src/bandscope_analysis/chords/chord_recognizer.py +++ b/services/analysis-engine/src/bandscope_analysis/chords/chord_recognizer.py @@ -396,7 +396,7 @@ def recognize(self, y: np.ndarray, sr: int = 22050) -> list[TrackedChord]: Returns: List of TrackedChord dicts with start_time, end_time, chord, and confidence. """ - if len(y) == 0: + if y.size == 0: return [] y_harmonic = self._separate_harmonic(y) diff --git a/services/analysis-engine/src/bandscope_analysis/cli.py b/services/analysis-engine/src/bandscope_analysis/cli.py index 6838ee711..5978eacaf 100644 --- a/services/analysis-engine/src/bandscope_analysis/cli.py +++ b/services/analysis-engine/src/bandscope_analysis/cli.py @@ -2,25 +2,43 @@ from __future__ import annotations +import hashlib import json import logging +import os +import re +import shutil import sys +from collections.abc import Iterator +from contextlib import contextmanager, suppress from datetime import UTC, datetime +from pathlib import Path -from bandscope_analysis.api import get_analysis_status, run_analysis_job, run_analysis_job_updates -from bandscope_analysis.temporal import TemporalAnalyzer +from bandscope_analysis.api import ( + get_analysis_status, + run_analysis_job, + run_analysis_job_updates, + validate_analysis_job_request, +) logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s") +_SOURCE_SHA256_PATTERN = re.compile(r"^[0-9a-f]{64}$") -def failed_cli_response(message: str) -> dict[str, object]: + +def failed_cli_response( + message: str, + *, + job_id: str = "unknown-job", + requested_at: str | None = None, +) -> dict[str, object]: """Return a typed CLI failure envelope for malformed stdin payloads.""" - requested_at = datetime.now(UTC).isoformat().replace("+00:00", "Z") + timestamp = requested_at or datetime.now(UTC).isoformat().replace("+00:00", "Z") return { - "jobId": "unknown-job", + "jobId": job_id, "state": "failed", - "requestedAt": requested_at, - "updatedAt": requested_at, + "requestedAt": timestamp, + "updatedAt": timestamp, "error": { "code": "invalid_request", "message": message, @@ -28,19 +46,140 @@ def failed_cli_response(message: str) -> dict[str, object]: } +def _bind_verified_source_cache_namespace( + request: object, + source_content_sha256: object, + job_id: object = None, +) -> object: + """Scope local cache and stem-work paths to verified source and job identities.""" + if not isinstance(request, dict): + return request + + source_kind = request.get("sourceKind") + if source_kind != "local_audio": + if source_content_sha256 is not None: + raise ValueError("Invalid analysis job request: invalid field 'sourceContentSha256'") + return request + + bound_request = dict(request) + job_digest = ( + hashlib.sha256(job_id.encode("utf-8")).hexdigest() + if isinstance(job_id, str) and job_id.strip() + else None + ) + if source_content_sha256 is None: + if isinstance(bound_request.get("cacheRoot"), str): + bound_request.pop("cacheRoot", None) + temp_root = bound_request.get("tempRoot") + if isinstance(temp_root, str) and temp_root.strip() and job_digest is not None: + bound_request["tempRoot"] = str(Path(temp_root) / "job-sha256-v1" / job_digest) + return bound_request + if not isinstance(source_content_sha256, str) or not _SOURCE_SHA256_PATTERN.fullmatch( + source_content_sha256 + ): + raise ValueError("Invalid analysis job request: invalid field 'sourceContentSha256'") + + cache_root = bound_request.get("cacheRoot") + if isinstance(cache_root, str) and cache_root.strip(): + bound_request["cacheRoot"] = str( + Path(cache_root) / "source-sha256-v1" / source_content_sha256 + ) + temp_root = bound_request.get("tempRoot") + if isinstance(temp_root, str) and temp_root.strip(): + scoped_temp_root = Path(temp_root) / "source-sha256-v1" / source_content_sha256 + if job_digest is not None: + scoped_temp_root = scoped_temp_root / "job-sha256-v1" / job_digest + bound_request["tempRoot"] = str(scoped_temp_root) + return bound_request + + +@contextmanager +def _open_anchored_directory( + path: Path, +) -> Iterator[int | None]: + """Yield one descriptor-anchored directory and close the complete chain on exit.""" + supports_dir_fd = getattr(os, "supports_dir_fd", set()) + if ( + not path.is_absolute() + or os.open not in supports_dir_fd + or not hasattr(os, "O_DIRECTORY") + or not hasattr(os, "O_NOFOLLOW") + ): + yield None + return + + flags = os.O_RDONLY | os.O_DIRECTORY | os.O_NOFOLLOW + flags |= getattr(os, "O_CLOEXEC", 0) + descriptors: list[int] = [] + try: + current_descriptor = os.open(path.anchor, flags) + descriptors.append(current_descriptor) + for component in path.parts[1:]: + if component in {"", ".", ".."}: + raise OSError + current_descriptor = os.open( + component, + flags, + dir_fd=current_descriptor, + ) + descriptors.append(current_descriptor) + except OSError: + yield None + else: + yield descriptors[-1] + finally: + for descriptor in reversed(descriptors): + with suppress(OSError): + os.close(descriptor) + + +def _cleanup_job_temp_namespace(request: object) -> None: + """Best-effort remove only a descriptor-anchored derived job namespace.""" + if not isinstance(request, dict): + return + temp_root = request.get("tempRoot") + if not isinstance(temp_root, str) or not temp_root.strip(): + return + path = Path(temp_root) + parts = path.parts + if not path.is_absolute() or len(parts) < 2 or parts[-2] != "job-sha256-v1": + return + if _SOURCE_SHA256_PATTERN.fullmatch(parts[-1]) is None: + return + + derived_parents = [path.parent] + if len(parts) >= 4 and parts[-4] == "source-sha256-v1": + if _SOURCE_SHA256_PATTERN.fullmatch(parts[-3]) is None: + return + derived_parents.extend((path.parents[1], path.parents[2], path.parents[3])) + else: + derived_parents.append(path.parents[1]) + if any(candidate.is_symlink() for candidate in derived_parents): + return + if not shutil.rmtree.avoids_symlink_attacks: + return + + with _open_anchored_directory(path.parent) as parent_descriptor: + if parent_descriptor is None: + return + shutil.rmtree( + path.name, + dir_fd=parent_descriptor, + ignore_errors=True, + ) + + def main() -> int: """Read a job payload from stdin and print a structured job response to stdout.""" - # Read all input from stdin first input_data = sys.stdin.read().strip() progress_jsonl = "--progress-jsonl" in sys.argv[1:] cli_args = [arg for arg in sys.argv[1:] if arg != "--progress-jsonl"] - # Check if there are command line arguments (fallback for manual testing) if cli_args: if cli_args[0] == "--status": json.dump(get_analysis_status(), sys.stdout) return 0 - elif cli_args[0] == "--job" and len(cli_args) > 1: + if cli_args[0] == "--job" and len(cli_args) > 1: input_data = cli_args[1] if not input_data.startswith("{"): try: @@ -73,41 +212,46 @@ def main() -> int: ) return 0 - request = payload.get("request") + requested_at_value = payload.get("requestedAt") + if requested_at_value is None: + requested_at = datetime.now(UTC).isoformat().replace("+00:00", "Z") + elif not isinstance(requested_at_value, str) or not requested_at_value.strip(): + json.dump( + failed_cli_response("Invalid analysis job request: invalid field 'requestedAt'"), + sys.stdout, + ) + return 0 + else: + requested_at = requested_at_value - # Temporary: Inject temporal analyzer call if it's a local file, just to prove it works - # before full orchestrator integration - if ( - isinstance(request, dict) - and request.get("sourceKind") == "local_audio" - and "localSource" in request - ): - local_source = request["localSource"] - audio_path = local_source.get("sourcePath") - file_name = local_source.get("fileName", "selected audio") - if audio_path: - logging.info("Extracting temporal features from %s...", file_name) - try: - temporal_analyzer = TemporalAnalyzer() - features = temporal_analyzer.analyze(audio_path) - logging.info(f"Extracted BPM: {features['bpm']}") - except Exception: - logging.warning( - "Temporal analysis failed for %s; continuing with safe fallback.", - file_name, - ) - - requested_at = datetime.now(UTC).isoformat().replace("+00:00", "Z") - if progress_jsonl: - for update in run_analysis_job_updates(job_id, request, requested_at): - json.dump(update, sys.stdout) - sys.stdout.write("\n") - sys.stdout.flush() + request = payload.get("request") + try: + request = validate_analysis_job_request(request) + request = _bind_verified_source_cache_namespace( + request, + payload.get("sourceContentSha256"), + job_id, + ) + except ValueError as error: + json.dump( + failed_cli_response(str(error), job_id=job_id, requested_at=requested_at), + sys.stdout, + ) return 0 - response = run_analysis_job(job_id, request, requested_at) - json.dump(response, sys.stdout) - return 0 + try: + if progress_jsonl: + for update in run_analysis_job_updates(job_id, request, requested_at): + json.dump(update, sys.stdout) + sys.stdout.write("\n") + sys.stdout.flush() + return 0 + + response = run_analysis_job(job_id, request, requested_at) + json.dump(response, sys.stdout) + return 0 + finally: + _cleanup_job_temp_namespace(request) if __name__ == "__main__": diff --git a/services/analysis-engine/src/bandscope_analysis/feature_cache_admission.py b/services/analysis-engine/src/bandscope_analysis/feature_cache_admission.py new file mode 100644 index 000000000..c82b94b2b --- /dev/null +++ b/services/analysis-engine/src/bandscope_analysis/feature_cache_admission.py @@ -0,0 +1,479 @@ +"""Bounded replay admission for persisted local-audio stem arrays. + +The feature cache is a persistence boundary, not trusted in-memory state. A +cached ``npz`` archive is therefore copied from one already-open regular file +into a bounded private snapshot before NumPy may materialize any member. Member +names, NPY headers, sample counts, visible bytes, floating-point representation, +and aggregate archive shape are admitted against the same audio resource budget +used by decode. + +Security Notes: +- The cache path is app-owned, but its bytes and metadata are untrusted after a + crash, local tampering, restore, or partial publication. +- Persisted metadata is admitted only from a regular sidecar no larger than 1 + MiB before UTF-8 decode or JSON materialization. Unix-like platforms also use + non-blocking/no-follow open flags when available so a substituted FIFO or + symlink cannot become an unbounded or blocking replay input. JSON decoder + numeric-limit failures and duplicate object members are treated as cache + misses rather than job failures or ambiguous cache authority. +- Persisted stem identities are admitted only from the canonical Demucs output + set (vocals, bass, drums, other); cache metadata cannot invent a new role. +- The persisted metadata sidecar must still be readable at archive admission; + its second-read schema version, stem identity, and sample rate must match the + caller's already-admitted metadata. Legacy caches may omit ``stemRoleTypes`` + inside that sidecar, but sidecar disappearance, schema/identity/rate + replacement, or malformed replacement fails closed. +- Persisted separation duration is required, finite, positive, and must agree + with the synchronized stem sample timeline within half one sample at the + admitted sample rate. Metadata cannot omit, stretch, or shrink rehearsal + timing away from the actual cached stem extent. Exact metadata/archive/source + generation binding remains a separate persistence contract. +- Persisted role metadata, when present beside the stem archive, must preserve + the canonical binding: vocals is vocal; bass, drums, and other are instruments. +- The archive pathname is opened with non-blocking/no-follow flags when the + platform exposes them before the same descriptor is admitted as a bounded + regular file, so a substituted FIFO or symlink cannot redirect or stall replay. +- The opened archive is copied exactly once into a bounded spooled snapshot. + ZIP/NPY declaration preflight and NumPy materialization consume that same + snapshot, so pathname or same-inode rewrites after the copy cannot substitute + different samples into the already-admitted rehearsal evidence. +- ZIP central-directory declarations and bounded NPY headers are checked before + ``np.load`` can decompress a stem member. Extra or duplicate members fail + closed rather than becoming hidden compressed payload. +- Every admitted stem must declare the same non-zero sample count so replay + preserves the synchronized timeline produced by source separation. +- Each member is one non-empty floating one-dimensional signal within the + configured sample and visible-byte ceilings. Loaded legacy floating dtypes + are converted to owned ``float32`` only after those pre-copy bounds pass. +- Canonical finiteness, dtype, sample-rate, sample-count, and memory checks are + reapplied before a replayed stem can return to MIR/rehearsal analysis. +- Allocator exhaustion or truncated archive state encountered while copying, + preflighting, or opening an otherwise admitted cache fails closed as a cache + miss instead of escaping the persistence boundary and crashing the analysis job. +- This creates one immutable replay byte snapshot; it does not bind that snapshot + cryptographically to the metadata sidecar or admitted source content, and it + does not claim a process-wide RSS ceiling for NumPy/ZIP or downstream MIR work. +""" + +from __future__ import annotations + +import json +import math +import os +import stat +import tempfile +import zipfile +from pathlib import Path +from typing import BinaryIO + +import numpy as np +from numpy.typing import NDArray + +from bandscope_analysis.audio_resource_policy import ( + DEFAULT_AUDIO_RESOURCE_POLICY, + AudioResourcePolicy, + AudioResourcePolicyError, +) + +_FEATURE_CACHE_SCHEMA_VERSION = 1 +_CANONICAL_STEM_ROLE_TYPES = { + "vocals": "vocal", + "bass": "instrument", + "drums": "instrument", + "other": "instrument", +} +_CANONICAL_STEM_KEYS = frozenset(_CANONICAL_STEM_ROLE_TYPES) +_MAX_STEM_MEMBERS = len(_CANONICAL_STEM_KEYS) +_MAX_NPY_HEADER_BYTES = 16 * 1024 +_MAX_ARCHIVE_CONTAINER_OVERHEAD_BYTES = 1024 * 1024 +_ARCHIVE_SNAPSHOT_MEMORY_BYTES = 8 * 1024 * 1024 +_ARCHIVE_SNAPSHOT_COPY_CHUNK_BYTES = 1024 * 1024 +_NPY_VERSION = (1, 0) +_CANONICAL_ITEMSIZE = np.dtype(np.float32).itemsize +MAX_FEATURE_CACHE_METADATA_BYTES = 1024 * 1024 + + +def _materialize_unique_json_object( + pairs: list[tuple[str, object]], +) -> dict[str, object]: + """Materialize one JSON object only when every member name is unique.""" + result: dict[str, object] = {} + for key, value in pairs: + if key in result: + raise ValueError("duplicate JSON object member") + result[key] = value + return result + + +def read_bounded_feature_cache_metadata( + metadata_path: Path, +) -> dict[str, object] | None: + """Read one bounded regular UTF-8 JSON sidecar from an already-open descriptor.""" + open_flags = os.O_RDONLY | getattr(os, "O_BINARY", 0) + open_flags |= getattr(os, "O_NONBLOCK", 0) + open_flags |= getattr(os, "O_NOFOLLOW", 0) + try: + descriptor = os.open(metadata_path, open_flags) + with os.fdopen(descriptor, "rb") as metadata_file: + metadata_stat = os.fstat(metadata_file.fileno()) + if ( + not stat.S_ISREG(metadata_stat.st_mode) + or metadata_stat.st_size <= 0 + or metadata_stat.st_size > MAX_FEATURE_CACHE_METADATA_BYTES + ): + return None + encoded_metadata = metadata_file.read(metadata_stat.st_size + 1) + if len(encoded_metadata) != metadata_stat.st_size: + return None + metadata = json.loads( + encoded_metadata.decode("utf-8"), + object_pairs_hook=_materialize_unique_json_object, + ) + except ( + MemoryError, + OSError, + UnicodeDecodeError, + ValueError, + ): + return None + if not isinstance(metadata, dict): + return None + return metadata + + +def _replay_policy( + sample_rate: object, + template: AudioResourcePolicy, +) -> AudioResourcePolicy | None: + """Derive a bounded replay policy for the cache-declared analysis rate.""" + if isinstance(sample_rate, bool) or not isinstance(sample_rate, int): + return None + if ( + sample_rate < template.min_source_sample_rate + or sample_rate > template.max_source_sample_rate + ): + return None + try: + max_samples = int(sample_rate * float(template.max_duration_seconds)) + canonical_bytes = max_samples * _CANONICAL_ITEMSIZE + return AudioResourcePolicy( + max_encoded_file_bytes=template.max_encoded_file_bytes, + target_sample_rate=sample_rate, + max_duration_seconds=template.max_duration_seconds, + max_decoded_audio_bytes=min( + template.max_decoded_audio_bytes, canonical_bytes + ), + min_source_sample_rate=template.min_source_sample_rate, + max_source_sample_rate=template.max_source_sample_rate, + min_source_channels=template.min_source_channels, + max_source_channels=template.max_source_channels, + ) + except (OverflowError, ValueError): + return None + + +def _expected_member_names(stem_keys: list[str]) -> set[str] | None: + """Return the exact NPY member set for one bounded canonical stem-key list.""" + if ( + not stem_keys + or len(stem_keys) > _MAX_STEM_MEMBERS + or len(set(stem_keys)) != len(stem_keys) + ): + return None + if not all( + stem_key and stem_key.isidentifier() and stem_key in _CANONICAL_STEM_KEYS + for stem_key in stem_keys + ): + return None + return {f"stem_{stem_key}.npy" for stem_key in stem_keys} + + +def _read_canonical_stem_role_metadata( + arrays_path: Path, + stem_keys: list[str], + *, + expected_sample_rate: object | None = None, +) -> dict[str, object] | None: + """Return one admitted second-read sidecar snapshot for archive replay.""" + metadata = read_bounded_feature_cache_metadata(arrays_path.with_suffix(".json")) + if metadata is None: + return None + if metadata.get("schemaVersion") != _FEATURE_CACHE_SCHEMA_VERSION: + return None + if metadata.get("stemKeys") != stem_keys: + return None + if _expected_member_names(stem_keys) is None: + return None + if expected_sample_rate is not None: + if metadata.get("sampleRate") != expected_sample_rate: + return None + + separation = metadata.get("separation") + if not isinstance(separation, dict): + return None + duration_seconds = separation.get("duration_seconds") + if duration_seconds is None or isinstance(duration_seconds, bool): + return None + if not isinstance(duration_seconds, (int, float)): + return None + try: + duration_value = float(duration_seconds) + except (OverflowError, ValueError): + return None + if not math.isfinite(duration_value) or duration_value <= 0.0: + return None + + stem_role_types = metadata.get("stemRoleTypes") + if stem_role_types is None: + return metadata + if not isinstance(stem_role_types, dict): + return None + if set(stem_role_types) != set(stem_keys): + return None + if not all( + stem_role_types.get(stem_key) == _CANONICAL_STEM_ROLE_TYPES[stem_key] + for stem_key in stem_keys + ): + return None + return metadata + + +def _has_canonical_stem_role_metadata( + arrays_path: Path, + stem_keys: list[str], + *, + expected_sample_rate: object | None = None, +) -> bool: + """Reject missing or persisted metadata that contradicts replay semantics.""" + return ( + _read_canonical_stem_role_metadata( + arrays_path, + stem_keys, + expected_sample_rate=expected_sample_rate, + ) + is not None + ) + + +def _duration_matches_sample_timeline( + duration_seconds: object, + sample_count: int, + sample_rate: object, +) -> bool: + """Return whether persisted duration agrees with the stem timeline to half a sample.""" + if duration_seconds is None: + return True + if isinstance(duration_seconds, bool) or not isinstance(duration_seconds, (int, float)): + return False + if isinstance(sample_rate, bool) or not isinstance(sample_rate, int) or sample_rate <= 0: + return False + try: + duration_value = float(duration_seconds) + except (OverflowError, ValueError): + return False + if not math.isfinite(duration_value) or duration_value <= 0.0: + return False + expected_duration = sample_count / sample_rate + return math.isclose( + duration_value, + expected_duration, + rel_tol=0.0, + abs_tol=0.5 / sample_rate, + ) + + +def _copy_exact_archive_snapshot( + source: BinaryIO, + destination: BinaryIO, + byte_count: int, +) -> bool: + """Copy exactly one admitted archive extent into a private replay snapshot.""" + remaining = byte_count + while remaining > 0: + chunk = source.read(min(remaining, _ARCHIVE_SNAPSHOT_COPY_CHUNK_BYTES)) + if not chunk: + return False + destination.write(chunk) + remaining -= len(chunk) + destination.seek(0) + return True + + +def _preflight_npz( + archive_file: BinaryIO, + stem_keys: list[str], + policy: AudioResourcePolicy, +) -> int | None: + """Return the synchronized sample count after bounded ZIP/NPY declaration admission.""" + expected_names = _expected_member_names(stem_keys) + if expected_names is None: + return None + max_member_bytes = policy.max_decoded_audio_bytes + _MAX_NPY_HEADER_BYTES + max_total_bytes = len(stem_keys) * max_member_bytes + + try: + with zipfile.ZipFile(archive_file, mode="r") as archive: + members = archive.infolist() + member_names = [member.filename for member in members] + if ( + len(members) != len(expected_names) + or set(member_names) != expected_names + ): + return None + if len(member_names) != len(set(member_names)): + return None + + expected_sample_count: int | None = None + total_declared_bytes = 0 + for member in members: + if ( + member.is_dir() + or member.flag_bits & 0x1 + or member.compress_type != zipfile.ZIP_DEFLATED + or member.file_size <= 0 + or member.file_size > max_member_bytes + ): + return None + total_declared_bytes += member.file_size + if total_declared_bytes > max_total_bytes: + return None + + with archive.open(member, mode="r") as npy_stream: + if np.lib.format.read_magic(npy_stream) != _NPY_VERSION: + return None + shape, fortran_order, dtype = np.lib.format.read_array_header_1_0( + npy_stream, + max_header_size=_MAX_NPY_HEADER_BYTES, + ) + if fortran_order or len(shape) != 1 or shape[0] <= 0: + return None + dtype = np.dtype(dtype) + if not np.issubdtype(dtype, np.floating): + return None + sample_count = int(shape[0]) + if expected_sample_count is None: + expected_sample_count = sample_count + elif sample_count != expected_sample_count: + return None + data_bytes = sample_count * dtype.itemsize + if ( + sample_count > policy.max_decoded_samples + or data_bytes > policy.max_decoded_audio_bytes + or npy_stream.tell() + data_bytes != member.file_size + ): + return None + except ( + EOFError, + MemoryError, + OSError, + ValueError, + zipfile.BadZipFile, + zipfile.LargeZipFile, + ): + return None + return expected_sample_count + + +def load_bounded_stem_archive( + arrays_path: Path, + stem_keys: list[str], + sample_rate: object, + *, + policy_template: AudioResourcePolicy = DEFAULT_AUDIO_RESOURCE_POLICY, +) -> dict[str, NDArray[np.float32]] | None: + """Load one admitted stem archive and return owned canonical float32 signals.""" + policy = _replay_policy(sample_rate, policy_template) + expected_names = _expected_member_names(stem_keys) + replay_metadata = _read_canonical_stem_role_metadata( + arrays_path, + stem_keys, + expected_sample_rate=sample_rate, + ) + if policy is None or expected_names is None or replay_metadata is None: + return None + + max_archive_bytes = ( + len(stem_keys) * (policy.max_decoded_audio_bytes + _MAX_NPY_HEADER_BYTES) + + _MAX_ARCHIVE_CONTAINER_OVERHEAD_BYTES + ) + open_flags = os.O_RDONLY | getattr(os, "O_BINARY", 0) + open_flags |= getattr(os, "O_NONBLOCK", 0) + open_flags |= getattr(os, "O_NOFOLLOW", 0) + try: + descriptor = os.open(arrays_path, open_flags) + with os.fdopen(descriptor, "rb") as archive_file: + file_stat = os.fstat(archive_file.fileno()) + if ( + not stat.S_ISREG(file_stat.st_mode) + or file_stat.st_size <= 0 + or file_stat.st_size > max_archive_bytes + ): + return None + with tempfile.SpooledTemporaryFile( + max_size=_ARCHIVE_SNAPSHOT_MEMORY_BYTES, + mode="w+b", + ) as snapshot_file: + if not _copy_exact_archive_snapshot( + archive_file, + snapshot_file, + file_stat.st_size, + ): + return None + sample_count = _preflight_npz(snapshot_file, stem_keys, policy) + if sample_count is None: + return None + separation = replay_metadata.get("separation") + if isinstance(separation, dict): + if not _duration_matches_sample_timeline( + separation.get("duration_seconds"), + sample_count, + sample_rate, + ): + return None + snapshot_file.seek(0) + with np.load( + snapshot_file, + allow_pickle=False, + max_header_size=_MAX_NPY_HEADER_BYTES, + ) as stems_archive: + stems: dict[str, NDArray[np.float32]] = {} + for stem_key in stem_keys: + archive_key = f"stem_{stem_key}" + if archive_key not in stems_archive: + return None + stem_array = stems_archive[archive_key] + if not isinstance(stem_array, np.ndarray): + return None + try: + with np.errstate(over="ignore", invalid="ignore"): + if ( + stem_array.dtype == np.dtype(np.float32) + and stem_array.flags.owndata + ): + canonical = stem_array + else: + canonical = np.array( + stem_array, dtype=np.float32, copy=True + ) + validated = policy.validate_decoded_audio( + canonical, sample_rate + ) + except ( + AudioResourcePolicyError, + MemoryError, + OverflowError, + TypeError, + ValueError, + ): + return None + stems[stem_key] = validated + except (EOFError, MemoryError, OSError, ValueError, zipfile.BadZipFile): + return None + return stems + + +__all__ = [ + "MAX_FEATURE_CACHE_METADATA_BYTES", + "load_bounded_stem_archive", + "read_bounded_feature_cache_metadata", +] diff --git a/services/analysis-engine/src/bandscope_analysis/separation/audio_separator.py b/services/analysis-engine/src/bandscope_analysis/separation/audio_separator.py index c36e0f1fc..a7e284b17 100644 --- a/services/analysis-engine/src/bandscope_analysis/separation/audio_separator.py +++ b/services/analysis-engine/src/bandscope_analysis/separation/audio_separator.py @@ -9,9 +9,16 @@ Security Notes: - Treats the selected audio file as untrusted input: the path is normalized and verified to be a file, and a maximum byte size is enforced before decode. -- Inference runs locally on CPU with no network access. The model weights are - loaded from the local Demucs cache or a configured bundled path; offline - weight bundling is tracked in the supplemental component inventory. +- Decoded audio is revalidated against the same versioned resource policy before + Demucs/model work so overlong, malformed, or non-finite decoder output fails + closed instead of being silently truncated or normalized. +- Empty, non-mono, non-finite, or float32-overflowed model stems fail closed before + they can become successful silence or downstream rehearsal evidence. +- Inference runs locally with no network access. Accelerator outputs cross back + to CPU before NumPy conversion so configured device execution cannot fail at + the device/host boundary. The model weights are loaded from the local Demucs + cache or a configured bundled path; offline weight bundling is tracked in the + supplemental component inventory. - Does not log or persist raw audio, separated stems, or full source paths. - Fails with bounded, filename-scoped errors so callers can surface a safe failure without leaking local directory structure. @@ -23,20 +30,18 @@ import logging import os import sys -import warnings from dataclasses import dataclass from pathlib import Path from typing import Any, cast -import librosa import numpy as np -from bandscope_analysis.temporal.analyzer import ( - KNOWN_LIBROSA_NUMBA_WARNING_FILTERS, - MAX_ANALYSIS_DURATION_SECONDS, - MAX_AUDIO_FILE_BYTES, - TARGET_SR, +from bandscope_analysis.audio_decode import decode_mono_audio +from bandscope_analysis.audio_resource_policy import ( + DEFAULT_MAX_DURATION_SECONDS, + AudioResourcePolicy, ) +from bandscope_analysis.temporal.analyzer import MAX_AUDIO_FILE_BYTES, TARGET_SR from .model import AudioSeparationResult, AudioStemArray, AudioStemName, AudioStemPayload @@ -45,6 +50,7 @@ # Demucs htdemucs emits these four sources; this is the canonical stem set. _STEM_ORDER: tuple[AudioStemName, ...] = ("vocals", "bass", "drums", "other") _EMPTY_RANGE_EPS = 1e-9 +_MODEL_OUTPUT_ERROR = "Stem separation produced invalid audio." def _contains_parent_path_segment(path: Path) -> bool: @@ -63,7 +69,7 @@ class AudioSeparationConfig: target_sample_rate: int = TARGET_SR max_file_bytes: int = MAX_AUDIO_FILE_BYTES - max_duration_seconds: float = float(MAX_ANALYSIS_DURATION_SECONDS) + max_duration_seconds: float = float(DEFAULT_MAX_DURATION_SECONDS) model_name: str = "htdemucs" device: str = "cpu" # Demucs splits long audio into overlapping segments internally, bounding @@ -75,8 +81,13 @@ class AudioStemSeparator: """Split a selected local mix into canonical stems for downstream analysis.""" def __init__(self, config: AudioSeparationConfig | None = None) -> None: - """Initialize the local stem separator (model is loaded lazily).""" + """Initialize the local stem separator and its canonical resource policy.""" self.config = config or AudioSeparationConfig() + self.resource_policy = AudioResourcePolicy( + max_encoded_file_bytes=self.config.max_file_bytes, + target_sample_rate=self.config.target_sample_rate, + max_duration_seconds=self.config.max_duration_seconds, + ) self._model: Any = None def separate(self, audio_path: str | Path) -> AudioSeparationResult: @@ -119,8 +130,8 @@ def _separate_signal( """Run the Demucs model on mono audio and return canonical mono stems. This is the single boundary to the neural model; it converts the mono - signal to the stereo tensor Demucs expects, applies the model on CPU, and - downmixes each source back to a mono float array. + signal to the stereo tensor Demucs expects, applies the model on the + configured device, and downmixes each source back to a mono host array. """ model = self._load_model() sources = self._apply_model(model, audio) @@ -172,7 +183,13 @@ def _apply_model(self, model: Any, audio: AudioStemArray) -> dict[str, np.ndarra progress=False, )[0] out = out * ref_std + ref_mean - return {name: out[i].mean(0).numpy() for i, name in enumerate(model.sources)} + stems: dict[str, np.ndarray[Any, Any]] = {} + for index, name in enumerate(model.sources): + stem = out[index].mean(0) + if self.config.device != "cpu": + stem = stem.cpu() + stems[name] = stem.numpy() + return stems def _resolve_audio_file(self, audio_path: str | Path) -> Path: """Normalize and validate the selected source path.""" @@ -190,39 +207,24 @@ def _resolve_audio_file(self, audio_path: str | Path) -> Path: return path def _load_audio(self, path: Path) -> tuple[AudioStemArray, int]: - """Load bounded mono audio without logging or exposing the full source path.""" + """Load bounded mono audio through the canonical decoder authority.""" try: with path.open("rb") as fileobj: file_size = os.fstat(fileobj.fileno()).st_size - if file_size > self.config.max_file_bytes: - raise ValueError( - "Audio file is too large for stem separation: " - f"{file_size} bytes (max {self.config.max_file_bytes} bytes)" - ) - - with warnings.catch_warnings(): - warnings.filterwarnings( - "ignore", category=DeprecationWarning, module=r"^audioread" - ) - warnings.filterwarnings("ignore", category=FutureWarning, module=r"^audioread") - for category, message, module in KNOWN_LIBROSA_NUMBA_WARNING_FILTERS: - warnings.filterwarnings( - "ignore", - category=category, - message=message, - module=module, - ) - y, sr = librosa.load( - fileobj, - sr=self.config.target_sample_rate, - mono=True, - duration=self.config.max_duration_seconds, - ) + if file_size <= 0: + raise ValueError(f"Stem separation decode failed for {path.name}") + try: + self.resource_policy.validate_encoded_file_bytes(file_size) + except ValueError as error: + raise ValueError("Audio file is too large for stem separation") from error + y, sr = decode_mono_audio(fileobj, policy=self.resource_policy) except ValueError: raise except Exception as error: raise ValueError(f"Stem separation decode failed for {path.name}") from error + if y.size == 0: + raise ValueError(f"Stem separation decode failed for {path.name}") return _as_float_array(y), int(sr) def _fit_length(self, audio: AudioStemArray, target_length: int) -> AudioStemArray: @@ -235,7 +237,12 @@ def _fit_length(self, audio: AudioStemArray, target_length: int) -> AudioStemArr def _as_float_array(values: object) -> AudioStemArray: - """Convert decoder and model output to a finite one-dimensional float array.""" - array = np.ravel(np.asarray(values, dtype=np.float32)) - finite = np.nan_to_num(array, copy=False, nan=0.0, posinf=0.0, neginf=0.0) - return cast(AudioStemArray, finite) + """Convert one finite, non-empty mono decoder/model output into float32 audio.""" + try: + with np.errstate(over="ignore", invalid="ignore"): + array = np.asarray(values, dtype=np.float32) + except (OverflowError, TypeError, ValueError) as error: + raise ValueError(_MODEL_OUTPUT_ERROR) from error + if array.ndim != 1 or array.size == 0 or not np.isfinite(array).all(): + raise ValueError(_MODEL_OUTPUT_ERROR) + return cast(AudioStemArray, array) diff --git a/services/analysis-engine/src/bandscope_analysis/temporal/analyzer.py b/services/analysis-engine/src/bandscope_analysis/temporal/analyzer.py index 7fe5ae6f7..ee8aec403 100644 --- a/services/analysis-engine/src/bandscope_analysis/temporal/analyzer.py +++ b/services/analysis-engine/src/bandscope_analysis/temporal/analyzer.py @@ -4,7 +4,6 @@ import logging import os -import warnings from pathlib import Path from typing import Any @@ -12,18 +11,36 @@ import numpy as np from numpy.typing import NDArray +from bandscope_analysis.audio_decode import decode_mono_audio +from bandscope_analysis.audio_resource_policy import ( + DEFAULT_AUDIO_RESOURCE_POLICY, + DEFAULT_MAX_DURATION_SECONDS, + AudioResourcePolicy, +) + from .model import TemporalFeatures logger = logging.getLogger(__name__) -# Standard sample rate for BandScope analysis -TARGET_SR = 44100 -MAX_AUDIO_FILE_BYTES = 100 * 1024 * 1024 # 100 MiB -MAX_ANALYSIS_DURATION_SECONDS = 15 * 60 # 15 minutes +# Compatibility aliases retained for callers/tests while the canonical values +# are owned by AudioResourcePolicy. The decode-duration alias intentionally +# includes one rejection-probe sample so an overlong source is detected rather +# than silently truncated at the accepted rehearsal duration. +TARGET_SR = DEFAULT_AUDIO_RESOURCE_POLICY.target_sample_rate +MAX_AUDIO_FILE_BYTES = DEFAULT_AUDIO_RESOURCE_POLICY.max_encoded_file_bytes +MAX_ANALYSIS_DURATION_SECONDS = DEFAULT_AUDIO_RESOURCE_POLICY.decode_probe_duration_seconds KNOWN_LIBROSA_NUMBA_WARNING_FILTERS = ( (DeprecationWarning, r".*pkg_resources is deprecated.*", r".*librosa.*"), (FutureWarning, r".*Numba.*", r".*numba.*"), ) +_SAFE_TEMPORAL_FAILURE_MESSAGES = frozenset( + { + "Audio file is too large for temporal analysis", + "Audio input violates the audio resource policy.", + } +) +_MISSING_AUDIO_MESSAGE = "Audio source is unavailable for temporal analysis." +_GENERIC_TEMPORAL_FAILURE_MESSAGE = "Temporal analysis failed." # ponytail: assumes 4/4; upgrade to meter estimation or a madmom DBN if other meters matter. BEATS_PER_BAR = 4 @@ -56,11 +73,34 @@ def _estimate_downbeats( return [float(bt) for i, bt in enumerate(beat_times) if (i - best_phase) % beats_per_bar == 0] +def _safe_temporal_failure_message(error: Exception) -> str: + """Return an allowlisted diagnostic without relaying decoder payload text.""" + message = str(error) + if message in _SAFE_TEMPORAL_FAILURE_MESSAGES: + return message + return _GENERIC_TEMPORAL_FAILURE_MESSAGE + + class TemporalAnalyzer: - """Analyzes temporal features (BPM, beats) from audio files.""" + """Analyze bounded temporal features (BPM and beat grids) from local audio.""" + + def __init__(self, resource_policy: AudioResourcePolicy | None = None) -> None: + """Create an analyzer bound to one canonical local-audio resource policy. + + Args: + resource_policy: Explicit policy for tests or specialized callers. + The default preserves the public module-level byte ceiling while + taking sample-rate and accepted rehearsal duration from the + canonical policy layer. + """ + self.resource_policy = resource_policy or AudioResourcePolicy( + max_encoded_file_bytes=MAX_AUDIO_FILE_BYTES, + target_sample_rate=TARGET_SR, + max_duration_seconds=DEFAULT_MAX_DURATION_SECONDS, + ) def analyze(self, audio_path: str | Path) -> TemporalFeatures: - """Decode audio and extract temporal features. + """Decode bounded audio and extract temporal features. Args: audio_path: Path to the audio file. @@ -71,54 +111,23 @@ def analyze(self, audio_path: str | Path) -> TemporalFeatures: path = Path(audio_path) path_str = str(path) if not path.exists() or not path.is_file(): - raise FileNotFoundError(f"Audio file not found: {path_str}") + raise FileNotFoundError(_MISSING_AUDIO_MESSAGE) - logger.info(f"Loading and decoding audio: {path_str}") + logger.info("Loading and decoding bounded local audio.") try: with path.open("rb") as fileobj: file_size = os.fstat(fileobj.fileno()).st_size - if file_size > MAX_AUDIO_FILE_BYTES: - raise ValueError( - f"Audio file is too large for temporal analysis: {file_size} bytes " - f"(max {MAX_AUDIO_FILE_BYTES} bytes)" - ) - - with warnings.catch_warnings(): - warnings.filterwarnings( - "ignore", category=DeprecationWarning, module=r"^audioread" - ) - warnings.filterwarnings("ignore", category=FutureWarning, module=r"^audioread") - - # Keep the loader's known third-party churn quiet without hiding - # unrelated decoder warnings that tests and callers should see. - for category, message, module in KNOWN_LIBROSA_NUMBA_WARNING_FILTERS: - warnings.filterwarnings( - "ignore", - category=category, - message=message, - module=module, - ) - # Load audio, converting to mono and standardizing sample rate - y, sr = librosa.load( - fileobj, - sr=TARGET_SR, - mono=True, - duration=MAX_ANALYSIS_DURATION_SECONDS, - ) - - # Ensure it's a 1D float array for librosa - if not isinstance(y, np.ndarray): - raise ValueError("Expected numpy array from librosa.load") - - y_array: NDArray[np.floating[Any]] = y + try: + self.resource_policy.validate_encoded_file_bytes(file_size) + except ValueError as error: + raise ValueError("Audio file is too large for temporal analysis") from error + y_array, sr = decode_mono_audio(fileobj, policy=self.resource_policy) + duration = float(librosa.get_duration(y=y_array, sr=sr)) logger.info("Extracting tempo and beat tracking...") - # Use librosa's robust beat tracker tempo, beat_frames = librosa.beat.beat_track(y=y_array, sr=sr) - - # Convert frame indices to time (seconds) beat_times: NDArray[np.floating[Any]] = librosa.frames_to_time(beat_frames, sr=sr) # Place downbeats on the strongest-onset bar phase (looks at the audio, @@ -139,6 +148,6 @@ def analyze(self, audio_path: str | Path) -> TemporalFeatures: "audio_path": path_str, } - except Exception as e: - logger.error(f"Failed to analyze audio {path_str}: {e}") - raise ValueError(f"Temporal analysis failed: {e}") from e + except Exception as error: + logger.error("Temporal analysis failed (%s).", type(error).__name__) + raise ValueError(_safe_temporal_failure_message(error)) from error diff --git a/services/analysis-engine/src/bandscope_analysis/transcription/api.py b/services/analysis-engine/src/bandscope_analysis/transcription/api.py index f2a732d31..3318da65e 100644 --- a/services/analysis-engine/src/bandscope_analysis/transcription/api.py +++ b/services/analysis-engine/src/bandscope_analysis/transcription/api.py @@ -3,13 +3,15 @@ from __future__ import annotations import io -import warnings from dataclasses import dataclass import librosa import numpy as np from numpy.typing import NDArray +from bandscope_analysis.audio_decode import decode_mono_audio +from bandscope_analysis.audio_resource_policy import AudioResourcePolicy + TARGET_SR = 22050 MAX_STEM_BYTES = 50 * 1024 * 1024 MAX_TRANSCRIPTION_DURATION_SECONDS = 120 @@ -17,6 +19,12 @@ HOP_LENGTH = 512 MIN_NOTE_DURATION_SECONDS = 0.05 MIN_SIGNAL_PEAK = 1e-5 +TRANSCRIPTION_RESOURCE_POLICY = AudioResourcePolicy( + max_encoded_file_bytes=MAX_STEM_BYTES, + target_sample_rate=TARGET_SR, + max_duration_seconds=MAX_TRANSCRIPTION_DURATION_SECONDS, + max_decoded_audio_bytes=(TARGET_SR * MAX_TRANSCRIPTION_DURATION_SECONDS + 1) * 8, +) @dataclass @@ -42,16 +50,8 @@ def transcribe_bass_stem(stem_data: bytes) -> list[NoteEvent]: if len(stem_data) > MAX_STEM_BYTES: raise ValueError("Stem data is too large for transcription.") - with warnings.catch_warnings(): - warnings.filterwarnings("ignore", category=DeprecationWarning, module=r"^audioread") - y, sr = librosa.load( - io.BytesIO(stem_data), - sr=TARGET_SR, - mono=True, - duration=MAX_TRANSCRIPTION_DURATION_SECONDS, - ) - - y_array = np.asarray(y, dtype=np.float32) + source = io.BytesIO(stem_data) + y_array, sr = decode_mono_audio(source, policy=TRANSCRIPTION_RESOURCE_POLICY) if y_array.size == 0 or float(np.max(np.abs(y_array))) < MIN_SIGNAL_PEAK: return [] diff --git a/services/analysis-engine/src/bandscope_analysis/youtube.py b/services/analysis-engine/src/bandscope_analysis/youtube.py index c98f4e513..e4f22e9b1 100644 --- a/services/analysis-engine/src/bandscope_analysis/youtube.py +++ b/services/analysis-engine/src/bandscope_analysis/youtube.py @@ -1,11 +1,47 @@ -""" -YouTube import capabilities for BandScope. +"""YouTube import capabilities for BandScope. This module provides a safe wrapper around yt-dlp to download audio from YouTube. + +Security Notes: + - URL intake remains host/path/query allowlisted before any network work. + - Each validated video ID acquires an atomic same-cache import lease before + yt-dlp starts. A second same-ID import cannot write or clean the first + import's predictable artifact names while the lease is held. A stale + lease fails closed rather than authorizing deletion. + - A completed artifact for the same video ID must not pre-exist the lease; + this prevents a later import from overwriting or claiming an older file. + - Encoded-byte admission uses the same canonical 100 MiB policy as local + audio. yt-dlp ``max_filesize`` and a progress hook abort in-flight + transfers so a multi-gigabyte download cannot fill the cache root before + the post-download check runs. + - Announced duration must be a finite positive non-Boolean number when + present; malformed known-duration metadata fails closed before download. + Download-result duration is revalidated before success so changed + metadata cannot bypass the same 15-minute admission boundary. + - Announced ``filesize`` / ``filesize_approx`` values over the policy + ceiling reject the import before ``download=True``. + - yt-dlp metadata must retain the requested video ID before and after the + download. A changed ID cannot redirect the current lease to another + import's predictable filenames. + - The completed download path must resolve beneath this import's ``out_dir`` + and match one canonical ``{video_id}{extension}`` final-artifact name + before post-download size checks, cleanup, or success metadata can use it. + - The opened-file size is revalidated with ``AudioResourcePolicy`` after + download; oversized artifacts and malformed zero-byte outputs are deleted + while retaining the correct buyer-facing rejection category. + - In-flight abort deletes only an exact canonical final artifact or the + leased video's explicitly transient ``.part``, ``.ytdl``, and ``-FragN`` + filenames beneath this ``out_dir``. Same-ID prefix alone never grants + deletion authority. + - Validation errors are payload-free and never include source paths, URLs, + cookies, or audio content. """ +from __future__ import annotations + import argparse import json +import math import os import re import sys @@ -14,49 +50,122 @@ import yt_dlp # type: ignore +from bandscope_analysis.audio_resource_policy import ( + DEFAULT_AUDIO_RESOURCE_POLICY, + DEFAULT_MAX_DURATION_SECONDS, + DEFAULT_MAX_ENCODED_FILE_BYTES, + AudioResourcePolicyError, +) + YOUTUBE_VIDEO_ID_PATTERN = re.compile(r"^[A-Za-z0-9_-]{11}$") MAX_YOUTUBE_URL_LENGTH = 2000 SUPPORTED_AUDIO_EXTENSIONS = (".opus", ".m4a", ".mp3", ".wav", ".aac", ".flac", ".ogg") +YOUTUBE_COMPLETED_AUDIO_EXTENSIONS = (*SUPPORTED_AUDIO_EXTENSIONS, ".webm") YOUTUBE_DOWNLOAD_FAILED_MESSAGE = ( "Failed to download audio from YouTube. Please use a local audio file instead." ) YOUTUBE_IMPORT_FAILED_MESSAGE = "YouTube import failed. Please use a local audio file instead." +YOUTUBE_SIZE_EXCEEDED_MESSAGE = "Selected audio file exceeds the 100 MiB analysis limit." -def validate_url(url: str) -> bool: - """ - Validate that a URL is a standard YouTube or youtu.be URL. +class YoutubeResourceLimitError(Exception): + """Fail-closed YouTube admission error that never includes payload paths.""" + + def __init__(self, code: str, message: str) -> None: + """Store a payload-safe public error code and next-action message. + + Args: + code: Stable machine-readable error code. + message: User-facing instruction that omits paths and URLs. + """ + super().__init__(message) + self.code = code + self.message = message + + +def _youtube_video_id(url: str) -> str | None: + """Return the one allowlisted YouTube video ID carried by ``url``. Args: - url: The URL to validate. + url: Candidate external URL. Returns: - True if the URL is valid, False otherwise. + The validated 11-character video ID, or ``None`` when the URL is outside + the supported HTTPS host/path/query contract. """ - # Pragmatic upper bound to avoid spending parser/downloader work on oversized user input. if len(url) > MAX_YOUTUBE_URL_LENGTH: - return False + return None try: parsed = urllib.parse.urlparse(url) if parsed.scheme != "https": - return False + return None host = parsed.netloc.lower().split(":")[0] if host == "youtu.be": path = parsed.path.strip("/") - return bool(YOUTUBE_VIDEO_ID_PATTERN.match(path)) + return path if YOUTUBE_VIDEO_ID_PATTERN.fullmatch(path) else None if host in {"youtube.com", "www.youtube.com"}: if parsed.path != "/watch": - return False + return None query = urllib.parse.parse_qs(parsed.query, keep_blank_values=True) video_ids = query.get("v", []) - return len(video_ids) == 1 and bool(YOUTUBE_VIDEO_ID_PATTERN.match(video_ids[0])) + if len(video_ids) != 1: + return None + video_id = video_ids[0] + return video_id if YOUTUBE_VIDEO_ID_PATTERN.fullmatch(video_id) else None - return False + return None except ValueError: - return False + return None + + +def validate_url(url: str) -> bool: + """Return whether ``url`` is one supported YouTube video URL.""" + return _youtube_video_id(url) is not None + + +def _import_lease_path(out_dir: str, video_id: str) -> str: + """Return the path-free-on-error lease directory for one video/cache pair.""" + return os.path.join(out_dir, f".bandscope-youtube-{video_id}.lock") + + +def _acquire_import_lease(out_dir: str, video_id: str) -> str | None: + """Atomically reserve one video ID inside a shared cache directory. + + The directory creation itself is the cross-process exclusion primitive. A + stale lease is intentionally not removed here: absence of current ownership + evidence must fail closed instead of deleting another process's state. + """ + lease_path = _import_lease_path(out_dir, video_id) + try: + os.mkdir(lease_path, 0o700) + except OSError: + return None + return lease_path + + +def _release_import_lease(lease_path: str) -> None: + """Release only the still-empty lease directory created by this import.""" + try: + os.rmdir(lease_path) + except OSError: + return + + +def _preexisting_final_artifact(out_dir: str, video_id: str) -> bool: + """Return whether this cache already contains a final artifact for ``video_id``. + + A previous completed import is not current ownership evidence. Refusing to + overwrite it keeps same-ID cache reuse fail-closed even after a lease ends. + Partial files are intentionally excluded because the active lease is the + authority that distinguishes current cleanup from a concurrent writer. + """ + return any( + os.path.lexists(os.path.join(out_dir, f"{video_id}{ext}")) + for ext in YOUTUBE_COMPLETED_AUDIO_EXTENSIONS + ) def _find_downloaded_file(actual_filepath: str) -> Optional[str]: @@ -72,9 +181,279 @@ def _find_downloaded_file(actual_filepath: str) -> Optional[str]: return actual_filepath +def _size_exceeded_result() -> Dict[str, Any]: + """Return the payload-safe oversize result shared by every admission path.""" + return { + "ok": False, + "error": { + "code": "size_exceeded", + "message": YOUTUBE_SIZE_EXCEEDED_MESSAGE, + }, + } + + +def _download_error_result() -> Dict[str, Any]: + """Return the payload-safe generic import failure result.""" + return { + "ok": False, + "error": {"code": "download_error", "message": YOUTUBE_IMPORT_FAILED_MESSAGE}, + } + + +def _reject_invalid_or_oversize_duration(info: dict[str, Any]) -> Dict[str, Any] | None: + """Validate announced duration before authorizing download work. + + Args: + info: Metadata dictionary from yt-dlp extraction. + + Returns: + A payload-safe failure for malformed/over-budget known duration, or + ``None`` when duration is absent or valid and within policy. + """ + duration = info.get("duration") + if duration is None: + return None + if type(duration) not in (int, float): + return _download_error_result() + duration_seconds = float(duration) + if not math.isfinite(duration_seconds) or duration_seconds <= 0.0: + return _download_error_result() + if duration_seconds > DEFAULT_MAX_DURATION_SECONDS: + return { + "ok": False, + "error": { + "code": "duration_exceeded", + "message": "Video exceeds the 15-minute limit.", + }, + } + return None + + +def _announced_size_exceeds_policy(announced: object) -> bool: + """Return whether yt-dlp metadata already reports an over-budget file. + + Args: + announced: Candidate ``filesize`` or ``filesize_approx`` value. + + Returns: + True when the value is a finite number strictly above the policy ceiling. + """ + if isinstance(announced, bool) or not isinstance(announced, int | float): + return False + if isinstance(announced, float) and not math.isfinite(announced): + return False + size_bytes: int | float = announced + return bool(size_bytes > DEFAULT_MAX_ENCODED_FILE_BYTES) + + +def _reject_announced_oversize(info: dict[str, Any]) -> Dict[str, Any] | None: + """Reject before download when extract_info already announced oversize bytes. + + Args: + info: Metadata dictionary from ``extract_info(..., download=False)``. + + Returns: + The size-exceeded result, or ``None`` when download may proceed. + """ + if _announced_size_exceeds_policy(info.get("filesize")) or _announced_size_exceeds_policy( + info.get("filesize_approx") + ): + return _size_exceeded_result() + return None + + +def _owned_file_path(path: object, out_dir: str) -> str | None: + """Return a real path only when it stays inside this import's output directory.""" + if not isinstance(path, str) or path == "": + return None + try: + resolved = os.path.realpath(path) + root = os.path.realpath(out_dir) + except OSError: + return None + if resolved == root or not resolved.startswith(root + os.sep): + return None + return resolved + + +def _owned_video_file_path(path: object, out_dir: str, video_id: str) -> str | None: + """Return a contained path only when its filename belongs to ``video_id``.""" + owned = _owned_file_path(path, out_dir) + if owned is None: + return None + if not os.path.basename(owned).startswith(f"{video_id}."): + return None + return owned + + +def _owned_completed_video_file_path( + path: object, + out_dir: str, + video_id: str, +) -> str | None: + """Return only one canonical completed artifact owned by the active lease.""" + owned = _owned_file_path(path, out_dir) + if owned is None: + return None + allowed_names = {f"{video_id}{ext}" for ext in YOUTUBE_COMPLETED_AUDIO_EXTENSIONS} + if os.path.basename(owned) not in allowed_names: + return None + return owned + + +def _owned_transient_video_file_path( + path: object, + out_dir: str, + video_id: str, +) -> str | None: + """Return only an explicitly transient yt-dlp artifact for the active lease.""" + owned = _owned_video_file_path(path, out_dir, video_id) + if owned is None: + return None + name = os.path.basename(owned) + artifact_name = name[len(video_id) + 1 :] + fragment_tail = artifact_name.rsplit("-Frag", maxsplit=1) + if len(fragment_tail) == 2: + if re.fullmatch(r"[0-9]+(?:\.part)?", fragment_tail[1]) is not None: + return owned + return None + if name.endswith((".part", ".ytdl")): + return owned + return None + + +def _remove_owned_file(path: object, out_dir: str) -> None: + """Delete one contained regular file, ignoring missing-path races. + + This generic containment helper remains useful for tests and non-video + cleanup. YouTube production cleanup uses ``_remove_video_owned_file`` so + directory containment alone never grants current-import deletion authority. + """ + owned = _owned_file_path(path, out_dir) + if owned is None: + return + try: + if os.path.isfile(owned): + os.remove(owned) + except OSError: + return + + +def _remove_video_owned_file(path: object, out_dir: str, video_id: str) -> None: + """Delete one regular file only when it carries the leased video identity.""" + owned = _owned_video_file_path(path, out_dir, video_id) + if owned is None: + return + try: + if os.path.isfile(owned): + os.remove(owned) + except OSError: + return + + +def _video_id_from_status(status: dict[str, Any]) -> str | None: + """Infer a validated video ID only for compatibility-focused helper tests.""" + for key in ("tmpfilename", "filename"): + candidate = status.get(key) + if not isinstance(candidate, str): + continue + name = os.path.basename(candidate) + video_id = name.split(".", maxsplit=1)[0] + if YOUTUBE_VIDEO_ID_PATTERN.fullmatch(video_id): + return video_id + return None + + +def _cleanup_stem(name: str) -> str: + """Return the canonical yt-dlp stem shared by one authorized transient path.""" + video_id, separator, artifact_name = name.partition(".") + if separator and YOUTUBE_VIDEO_ID_PATTERN.fullmatch(video_id): + fragment_tail = artifact_name.rsplit("-Frag", maxsplit=1) + if len(fragment_tail) == 2: + if re.fullmatch(r"[0-9]+(?:\.part)?", fragment_tail[1]) is None: + return name + name = f"{video_id}.{fragment_tail[0]}" + else: + fragment_match = re.search(r"-Frag[0-9]+(?:\.part)?$", name) + if fragment_match is not None: + name = name[: fragment_match.start()] + for suffix in (".part", ".ytdl"): + if name.endswith(suffix): + return name[: -len(suffix)] + return name + + +def _remove_download_artifacts( + status: dict[str, Any], + out_dir: str, + video_id: str | None = None, +) -> None: + """Delete only one leased video's canonical final or explicit transient files.""" + cleanup_video_id = video_id or _video_id_from_status(status) + if cleanup_video_id is None: + return + + stems: set[str] = set() + for key in ("tmpfilename", "filename"): + candidate = status.get(key) + owned = _owned_transient_video_file_path(candidate, out_dir, cleanup_video_id) + if owned is None: + owned = _owned_completed_video_file_path(candidate, out_dir, cleanup_video_id) + if owned is None: + continue + _remove_video_owned_file(owned, out_dir, cleanup_video_id) + stems.add(_cleanup_stem(os.path.basename(owned))) + if not stems: + return + try: + entries = os.listdir(out_dir) + except OSError: + return + for entry in entries: + transient = _owned_transient_video_file_path( + os.path.join(out_dir, entry), + out_dir, + cleanup_video_id, + ) + if transient is None: + continue + transient_name = os.path.basename(transient) + if any(_cleanup_stem(transient_name) == stem for stem in stems): + _remove_video_owned_file(transient, out_dir, cleanup_video_id) + + +def _abort_over_budget_download( + status: dict[str, Any], + out_dir: str, + video_id: str | None = None, +) -> None: + """Abort an in-flight leased download once encoded bytes exceed the policy ceiling.""" + if status.get("status") not in {"downloading", "finished"}: + return + for key in ("downloaded_bytes", "total_bytes", "total_bytes_estimate"): + candidate = status.get(key) + if isinstance(candidate, bool) or not isinstance(candidate, int): + continue + if candidate > DEFAULT_MAX_ENCODED_FILE_BYTES: + _remove_download_artifacts(status, out_dir, video_id) + raise YoutubeResourceLimitError("size_exceeded", YOUTUBE_SIZE_EXCEEDED_MESSAGE) + + +def _make_abort_hook(out_dir: str, video_id: str) -> Any: + """Bind the in-flight abort hook to the currently leased video identity.""" + + def _bound_abort_over_budget_download(status: dict[str, Any]) -> None: + """Abort and delete only this video's partials while its lease is held.""" + _abort_over_budget_download(status, out_dir, video_id) + + return _bound_abort_over_budget_download + + def _handle_download_error(e: yt_dlp.utils.DownloadError) -> Dict[str, Any]: """Map yt-dlp DownloadError to the public YouTube import error response.""" msg = str(e).lower() + if "max-filesize" in msg or "100 mib" in msg: + return _size_exceeded_result() if ( "sign in" in msg or "members-only" in msg @@ -102,17 +481,9 @@ def _handle_download_error(e: yt_dlp.utils.DownloadError) -> Dict[str, Any]: def download_youtube_audio(url: str, out_dir: str) -> Dict[str, Any]: - """ - Download audio from a YouTube URL to the specified directory. - - Args: - url: The YouTube URL to download. - out_dir: The directory to save the audio file. - - Returns: - A dictionary containing the result of the download. - """ - if not validate_url(url): + """Download one YouTube audio artifact under a same-video exclusive lease.""" + video_id = _youtube_video_id(url) + if video_id is None: return { "ok": False, "error": { @@ -121,76 +492,98 @@ def download_youtube_audio(url: str, out_dir: str) -> Dict[str, Any]: }, } - ydl_opts: Dict[str, Any] = { - "format": "bestaudio/best", - "outtmpl": os.path.join(out_dir, "%(id)s.%(ext)s"), - "quiet": True, - "no_warnings": True, - "noprogress": True, - "noplaylist": True, - "postprocessors": [{"key": "FFmpegExtractAudio"}], - "geo_bypass": False, - } + lease_path = _acquire_import_lease(out_dir, video_id) + if lease_path is None: + return _download_error_result() try: - with yt_dlp.YoutubeDL(ydl_opts) as ydl: - info = ydl.extract_info(url, download=False) - if info is None: - raise Exception("Failed to extract info") - duration = info.get("duration") - if duration is not None and duration > 15 * 60: - return { - "ok": False, - "error": { - "code": "duration_exceeded", - "message": "Video exceeds the 15-minute limit.", - }, - } + if _preexisting_final_artifact(out_dir, video_id): + return _download_error_result() - info = ydl.extract_info(url, download=True) - if info is None: - raise Exception("Failed to extract info") - actual_filepath = ydl.prepare_filename(info) + ydl_opts: Dict[str, Any] = { + "format": "bestaudio/best", + "outtmpl": os.path.join(out_dir, "%(id)s.%(ext)s"), + "quiet": True, + "no_warnings": True, + "noprogress": True, + "noplaylist": True, + "postprocessors": [{"key": "FFmpegExtractAudio"}], + "geo_bypass": False, + "max_filesize": DEFAULT_MAX_ENCODED_FILE_BYTES, + "progress_hooks": [_make_abort_hook(out_dir, video_id)], + } - actual_filepath = _find_downloaded_file(actual_filepath) + try: + with yt_dlp.YoutubeDL(ydl_opts) as ydl: + info = ydl.extract_info(url, download=False) + if info is None: + raise Exception("Failed to extract info") + if info.get("id") != video_id: + return _download_error_result() + duration_rejection = _reject_invalid_or_oversize_duration(info) + if duration_rejection is not None: + return duration_rejection + announced_rejection = _reject_announced_oversize(info) + if announced_rejection is not None: + return announced_rejection - if actual_filepath is None: - return { - "ok": False, - "error": { - "code": "file_not_found", - "message": "Downloaded file could not be found.", - }, - } + info = ydl.extract_info(url, download=True) + if info is None: + raise Exception("Failed to extract info") + if info.get("id") != video_id: + return _download_error_result() + actual_filepath = ydl.prepare_filename(info) + actual_filepath = _find_downloaded_file(actual_filepath) + + if actual_filepath is None: + return { + "ok": False, + "error": { + "code": "file_not_found", + "message": "Downloaded file could not be found.", + }, + } + + owned_filepath = _owned_completed_video_file_path( + actual_filepath, + out_dir, + video_id, + ) + if owned_filepath is None: + return _download_error_result() + actual_filepath = owned_filepath - if ( - os.path.exists(actual_filepath) - and os.path.getsize(actual_filepath) > 50 * 1024 * 1024 - ): - os.remove(actual_filepath) + duration_rejection = _reject_invalid_or_oversize_duration(info) + if duration_rejection is not None: + _remove_video_owned_file(actual_filepath, out_dir, video_id) + return duration_rejection + + try: + DEFAULT_AUDIO_RESOURCE_POLICY.validate_encoded_file_bytes( + os.path.getsize(actual_filepath) + ) + except AudioResourcePolicyError as error: + _remove_video_owned_file(actual_filepath, out_dir, video_id) + if error.reason == "encoded_file_too_large": + return _size_exceeded_result() + return _download_error_result() return { - "ok": False, - "error": { - "code": "size_exceeded", - "message": "Downloaded file exceeds the 50MB limit.", + "ok": True, + "metadata": { + "id": info.get("id"), + "title": info.get("title"), + "duration": info.get("duration"), + "filepath": actual_filepath, }, } - return { - "ok": True, - "metadata": { - "id": info.get("id"), - "title": info.get("title"), - "duration": info.get("duration"), - "filepath": actual_filepath, - }, - } - except yt_dlp.utils.DownloadError as e: - return _handle_download_error(e) - except Exception: - return { - "ok": False, - "error": {"code": "download_error", "message": YOUTUBE_IMPORT_FAILED_MESSAGE}, - } + except YoutubeResourceLimitError: + return _size_exceeded_result() + except yt_dlp.utils.DownloadError as e: + return _handle_download_error(e) + except Exception: + return _download_error_result() + finally: + _release_import_lease(lease_path) def main() -> None: diff --git a/services/analysis-engine/tests/test_audio_decode_backing_memory.py b/services/analysis-engine/tests/test_audio_decode_backing_memory.py new file mode 100644 index 000000000..114af7d87 --- /dev/null +++ b/services/analysis-engine/tests/test_audio_decode_backing_memory.py @@ -0,0 +1,42 @@ +"""Resource regressions for decoder-returned NumPy backing allocations.""" + +from __future__ import annotations + +import io + +import numpy as np +import pytest + +from bandscope_analysis import audio_decode +from bandscope_analysis.audio_resource_policy import AudioResourcePolicy + + +def test_decode_detaches_admitted_view_from_oversized_backing_array( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Return bounded owned PCM even when the decoder exposes a non-owning view.""" + policy = AudioResourcePolicy( + target_sample_rate=8, + max_duration_seconds=1.0, + max_decoded_audio_bytes=16, + ) + backing = np.array([0.1, -0.2, 0.3, -0.4] + [0.0] * 28, dtype=np.float32) + decoded_view = backing[:4] + assert decoded_view.nbytes == policy.max_decoded_audio_bytes + assert backing.nbytes > policy.max_decoded_audio_bytes + assert decoded_view.flags.owndata is False + + monkeypatch.setattr(audio_decode, "preflight_audio_metadata", lambda *_args: None) + monkeypatch.setattr( + audio_decode.librosa, + "load", + lambda *_args, **_kwargs: (decoded_view, policy.target_sample_rate), + ) + + decoded, sample_rate = audio_decode.decode_mono_audio(io.BytesIO(b"container"), policy=policy) + + np.testing.assert_array_equal(decoded, decoded_view) + assert decoded.flags.owndata is True + assert not np.shares_memory(decoded, backing) + assert decoded.nbytes == policy.max_decoded_audio_bytes + assert sample_rate == policy.target_sample_rate diff --git a/services/analysis-engine/tests/test_audio_decode_dtype_boundary.py b/services/analysis-engine/tests/test_audio_decode_dtype_boundary.py new file mode 100644 index 000000000..c0bd246ad --- /dev/null +++ b/services/analysis-engine/tests/test_audio_decode_dtype_boundary.py @@ -0,0 +1,53 @@ +"""Regression tests for decoder dtype admission at the canonical PCM boundary.""" + +from __future__ import annotations + +import io + +import numpy as np +import pytest + +from bandscope_analysis import audio_decode +from bandscope_analysis.audio_resource_policy import ( + AUDIO_RESOURCE_POLICY_VERSION, + DEFAULT_AUDIO_RESOURCE_POLICY, + AudioResourcePolicyError, +) + + +@pytest.mark.parametrize( + "decoder_output", + [ + np.array([1, -1], dtype=np.int16), + np.array([True, False], dtype=np.bool_), + np.array([0.25 + 0.5j, -0.25j], dtype=np.complex64), + ], +) +def test_decode_mono_audio_rejects_non_floating_decoder_dtype_before_canonicalization( + monkeypatch: pytest.MonkeyPatch, + decoder_output: np.ndarray, +) -> None: + """Reject malformed dtypes before allocating canonical rehearsal PCM.""" + monkeypatch.setattr(audio_decode, "preflight_audio_metadata", lambda *_args: None) + monkeypatch.setattr( + audio_decode.librosa, + "load", + lambda *_args, **_kwargs: ( + decoder_output, + DEFAULT_AUDIO_RESOURCE_POLICY.target_sample_rate, + ), + ) + monkeypatch.setattr( + audio_decode.np, + "array", + lambda *_args, **_kwargs: pytest.fail( + "non-floating decoder output must be rejected before canonical allocation" + ), + ) + + with pytest.raises(AudioResourcePolicyError) as caught: + audio_decode.decode_mono_audio(io.BytesIO(b"container")) + + assert caught.value.reason == "malformed_header" + assert caught.value.policy_version == AUDIO_RESOURCE_POLICY_VERSION + assert str(caught.value) == "Audio input violates the audio resource policy." diff --git a/services/analysis-engine/tests/test_audio_decode_port.py b/services/analysis-engine/tests/test_audio_decode_port.py new file mode 100644 index 000000000..fc8b6be44 --- /dev/null +++ b/services/analysis-engine/tests/test_audio_decode_port.py @@ -0,0 +1,350 @@ +"""Contract tests for the canonical local-audio decode port. + +These regressions keep resource admission, decoder failure redaction, and +decoded-output validation behind one owned boundary. +""" + +from __future__ import annotations + +import io + +import numpy as np +import pytest + +from bandscope_analysis import audio_decode +from bandscope_analysis.audio_resource_policy import ( + DEFAULT_AUDIO_RESOURCE_POLICY, + AudioResourcePolicy, + AudioResourcePolicyError, +) + + +def test_decode_mono_audio_rejects_encoded_size_before_metadata_or_decode( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Reject an oversized encoded handle before parser or decoder work begins.""" + source = io.BytesIO(b"oversized") + policy = AudioResourcePolicy(max_encoded_file_bytes=len(source.getvalue()) - 1) + monkeypatch.setattr( + audio_decode, + "preflight_audio_metadata", + lambda *_args: pytest.fail("metadata parser must not run for oversized encoded input"), + ) + monkeypatch.setattr( + audio_decode.librosa, + "load", + lambda *_args, **_kwargs: pytest.fail("decoder must not run for oversized encoded input"), + ) + + with pytest.raises(AudioResourcePolicyError) as caught: + audio_decode.decode_mono_audio(source, policy=policy) + + assert caught.value.reason == "encoded_file_too_large" + assert source.tell() == 0 + + +def test_decode_mono_audio_redacts_encoded_size_probe_failure( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """A handle that cannot expose a stable encoded size fails before parsing.""" + + class SizeProbeFails(io.BytesIO): + """Reject size inspection while retaining an otherwise readable handle.""" + + def tell(self) -> int: + """Simulate a local I/O failure without leaking its detail.""" + raise OSError("/private/rehearsal/source.wav size probe failed") + + monkeypatch.setattr( + audio_decode, + "preflight_audio_metadata", + lambda *_args: pytest.fail("metadata parser must not run after size-probe failure"), + ) + monkeypatch.setattr( + audio_decode.librosa, + "load", + lambda *_args, **_kwargs: pytest.fail("decoder must not run after size-probe failure"), + ) + + with pytest.raises(AudioResourcePolicyError) as caught: + audio_decode.decode_mono_audio(SizeProbeFails(b"container")) + + assert caught.value.reason == "malformed_header" + assert "/private/rehearsal/source.wav" not in str(caught.value) + assert isinstance(caught.value.__cause__, OSError) + + +def test_decode_mono_audio_preflights_then_validates_one_owned_decode( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Keep preflight, one decode, and decoded validation in strict order. + + The decode port must own the sequence so downstream analyzers cannot + bypass or duplicate resource admission. + """ + source = io.BytesIO(b"container") + calls: list[tuple[str, object]] = [] + decoder_output = np.array([0.25, -0.5], dtype=np.float64) + admitted_source: object | None = None + + def preflight(candidate: object, policy: object) -> None: + nonlocal admitted_source + calls.append(("preflight", candidate)) + assert policy is DEFAULT_AUDIO_RESOURCE_POLICY + assert candidate is not source + admitted_source = candidate + + def load(candidate: object, **kwargs: object) -> tuple[np.ndarray, int]: + calls.append(("decode", candidate)) + assert candidate is admitted_source + assert kwargs == { + "sr": DEFAULT_AUDIO_RESOURCE_POLICY.target_sample_rate, + "mono": True, + "duration": DEFAULT_AUDIO_RESOURCE_POLICY.decode_probe_duration_seconds, + "dtype": np.float32, + "res_type": "soxr_hq", + } + return decoder_output, DEFAULT_AUDIO_RESOURCE_POLICY.target_sample_rate + + def validate(self: AudioResourcePolicy, decoded: object, sample_rate: object) -> np.ndarray: + calls.append(("validate", decoded)) + assert self is DEFAULT_AUDIO_RESOURCE_POLICY + assert isinstance(decoded, np.ndarray) + assert decoded.dtype == np.float32 + assert decoded.shape == (2,) + assert sample_rate == DEFAULT_AUDIO_RESOURCE_POLICY.target_sample_rate + return decoded + + monkeypatch.setattr(audio_decode, "preflight_audio_metadata", preflight) + monkeypatch.setattr(audio_decode.librosa, "load", load) + monkeypatch.setattr(AudioResourcePolicy, "validate_decoded_audio", validate) + + decoded, sample_rate = audio_decode.decode_mono_audio( + source, + policy=DEFAULT_AUDIO_RESOURCE_POLICY, + ) + + assert calls[0][0] == "preflight" + assert calls[1][0] == "decode" + assert calls[2][0] == "validate" + np.testing.assert_array_equal(decoded, np.array([0.25, -0.5], dtype=np.float32)) + assert sample_rate == DEFAULT_AUDIO_RESOURCE_POLICY.target_sample_rate + + +def test_decode_mono_audio_bounds_growth_after_encoded_size_admission( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Do not let post-admission source growth expand parser or decoder authority.""" + + class GrowsAfterSizeProbe(io.BytesIO): + """Append bytes exactly when the initial size probe rewinds the source.""" + + def __init__(self, initial: bytes, growth: bytes) -> None: + """Retain deterministic initial and post-probe byte sequences.""" + super().__init__(initial) + self._growth = growth + self._grew = False + + def seek(self, offset: int, whence: int = 0) -> int: + """Grow once after the caller has measured the original end offset.""" + position = super().seek(offset, whence) + if not self._grew and whence == 0 and offset == 0: + self._grew = True + current = super().tell() + super().seek(0, 2) + super().write(self._growth) + super().seek(current) + return position + + initial = b"container" + source = GrowsAfterSizeProbe(initial, b"-post-admission-growth") + policy = AudioResourcePolicy(max_encoded_file_bytes=len(initial)) + admitted_source: object | None = None + + def preflight(candidate: object, _policy: object) -> None: + nonlocal admitted_source + admitted_source = candidate + assert candidate is not source + assert candidate.read() == initial # type: ignore[attr-defined] + candidate.seek(0) # type: ignore[attr-defined] + + def load(candidate: object, **_kwargs: object) -> tuple[np.ndarray, int]: + assert candidate is admitted_source + assert candidate.read() == initial # type: ignore[attr-defined] + return ( + np.array([0.1], dtype=np.float32), + DEFAULT_AUDIO_RESOURCE_POLICY.target_sample_rate, + ) + + monkeypatch.setattr(audio_decode, "preflight_audio_metadata", preflight) + monkeypatch.setattr(audio_decode.librosa, "load", load) + + decoded, sample_rate = audio_decode.decode_mono_audio(source, policy=policy) + + np.testing.assert_array_equal(decoded, np.array([0.1], dtype=np.float32)) + assert sample_rate == DEFAULT_AUDIO_RESOURCE_POLICY.target_sample_rate + assert source.getvalue() == initial + b"-post-admission-growth" + + +def test_bounded_encoded_source_readinto_stops_at_admitted_eof() -> None: + """Keep virtual-I/O buffer fills inside the originally admitted byte extent.""" + admitted = b"container" + source = io.BytesIO(admitted + b"-post-admission-growth") + bounded = audio_decode._BoundedEncodedSource(source, len(admitted)) + buffer = bytearray(len(admitted) + 8) + + read_count = bounded.readinto(buffer) + + assert read_count == len(admitted) + assert bytes(buffer[:read_count]) == admitted + assert bytes(buffer[read_count:]) == b"\x00" * 8 + assert bounded.tell() == len(admitted) + assert bounded.readinto(buffer) == 0 + + bounded.seek(-3, io.SEEK_END) + tail_buffer = bytearray(8) + tail_count = bounded.readinto(tail_buffer) + assert tail_count == 3 + assert bytes(tail_buffer[:tail_count]) == admitted[-3:] + assert bounded.tell() == len(admitted) + + +def test_bounded_encoded_source_seek_modes_fail_closed() -> None: + """Cover the virtual-I/O seek contract without exposing bytes past admitted EOF.""" + admitted = b"abcdef" + bounded = audio_decode._BoundedEncodedSource(io.BytesIO(admitted + b"extra"), len(admitted)) + + assert bounded.readable() is True + assert bounded.seekable() is True + assert bounded.seek(2, io.SEEK_SET) == 2 + assert bounded.seek(2, io.SEEK_CUR) == 4 + assert bounded.read(1) == b"e" + assert bounded.seek(0, io.SEEK_END) == len(admitted) + assert bounded.read() == b"" + + with pytest.raises(ValueError): + bounded.seek(0, 999) + with pytest.raises(OSError): + bounded.seek(-1, io.SEEK_SET) + + +def test_decode_mono_audio_rejects_non_mono_decoder_shape_before_normalization( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Reject multi-dimensional decoder output instead of flattening channels.""" + monkeypatch.setattr(audio_decode, "preflight_audio_metadata", lambda *_args: None) + monkeypatch.setattr( + audio_decode.librosa, + "load", + lambda *_args, **_kwargs: ( + np.array([[0.1, 0.2], [0.3, 0.4]], dtype=np.float32), + DEFAULT_AUDIO_RESOURCE_POLICY.target_sample_rate, + ), + ) + + with pytest.raises(AudioResourcePolicyError) as caught: + audio_decode.decode_mono_audio(io.BytesIO(b"container")) + + assert caught.value.reason == "malformed_header" + + +def test_decode_mono_audio_preserves_resource_policy_rejection( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Propagate the canonical preflight rejection without invoking a decoder. + + A rejected source must not consume additional decode resources or lose its typed policy reason. + """ + rejection = AudioResourcePolicyError("duration_exceeded") + + def reject(_source: object, _policy: object) -> None: + raise rejection + + monkeypatch.setattr(audio_decode, "preflight_audio_metadata", reject) + monkeypatch.setattr( + audio_decode.librosa, + "load", + lambda *_args, **_kwargs: pytest.fail("decoder must not run after rejected preflight"), + ) + + with pytest.raises(AudioResourcePolicyError) as caught: + audio_decode.decode_mono_audio(io.BytesIO(b"container")) + + assert caught.value is rejection + + +def test_decode_mono_audio_redacts_third_party_decoder_failure( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Map third-party decoder details to a payload-safe policy error. + + Native paths or token-shaped details may remain only in the exception + cause for local debugging, never in buyer-facing error text. + """ + secret_detail = "/Users/alice/Music/private.m4a token=secret" + monkeypatch.setattr(audio_decode, "preflight_audio_metadata", lambda *_args: None) + monkeypatch.setattr( + audio_decode.librosa, + "load", + lambda *_args, **_kwargs: (_ for _ in ()).throw(RuntimeError(secret_detail)), + ) + + with pytest.raises(AudioResourcePolicyError) as caught: + audio_decode.decode_mono_audio(io.BytesIO(b"container")) + + assert caught.value.reason == "malformed_header" + assert secret_detail not in str(caught.value) + assert isinstance(caught.value.__cause__, RuntimeError) + + +def test_decode_mono_audio_redacts_malformed_decoder_output( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Reject decoder output that cannot be normalized into bounded PCM. + + Malformed third-party values must fail at the decode boundary rather than + escaping into MIR analyzers. + """ + monkeypatch.setattr(audio_decode, "preflight_audio_metadata", lambda *_args: None) + monkeypatch.setattr( + audio_decode.librosa, + "load", + lambda *_args, **_kwargs: ([object()], DEFAULT_AUDIO_RESOURCE_POLICY.target_sample_rate), + ) + + with pytest.raises(AudioResourcePolicyError) as caught: + audio_decode.decode_mono_audio(io.BytesIO(b"container")) + + assert caught.value.reason == "malformed_header" + + +def test_decode_mono_audio_preserves_decoded_policy_rejection( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Preserve rejection identity from decoded-audio resource validation. + + The decode port must not collapse a precise post-decode budget failure + into a generic malformed-container error. + """ + rejection = AudioResourcePolicyError("decoded_sample_count_exceeded") + monkeypatch.setattr(audio_decode, "preflight_audio_metadata", lambda *_args: None) + monkeypatch.setattr( + audio_decode.librosa, + "load", + lambda *_args, **_kwargs: ( + np.array([0.1], dtype=np.float32), + DEFAULT_AUDIO_RESOURCE_POLICY.target_sample_rate, + ), + ) + + def reject_decoded( + _self: AudioResourcePolicy, _decoded: object, _sample_rate: object + ) -> np.ndarray: + raise rejection + + monkeypatch.setattr(AudioResourcePolicy, "validate_decoded_audio", reject_decoded) + + with pytest.raises(AudioResourcePolicyError) as caught: + audio_decode.decode_mono_audio(io.BytesIO(b"container")) + + assert caught.value is rejection diff --git a/services/analysis-engine/tests/test_audio_decode_preconversion_budget.py b/services/analysis-engine/tests/test_audio_decode_preconversion_budget.py new file mode 100644 index 000000000..e7a80ecaa --- /dev/null +++ b/services/analysis-engine/tests/test_audio_decode_preconversion_budget.py @@ -0,0 +1,162 @@ +"""Regression tests for decoder-output allocation before PCM normalization.""" + +from __future__ import annotations + +import io +from collections.abc import Callable +from typing import Any + +import numpy as np +import pytest + +from bandscope_analysis import audio_decode +from bandscope_analysis.audio_resource_policy import ( + AUDIO_RESOURCE_POLICY_VERSION, + AudioResourcePolicy, + AudioResourcePolicyError, +) + + +def _reject_float32_copy(monkeypatch: pytest.MonkeyPatch) -> None: + """Fail if an over-budget decoder result is copied into canonical float32 PCM.""" + original_array: Callable[..., np.ndarray[Any, Any]] = audio_decode.np.array + + def guarded_array(value: object, *args: object, **kwargs: object) -> np.ndarray[Any, Any]: + requested_dtype = kwargs.get("dtype") + if requested_dtype is None and args: + requested_dtype = args[0] + if requested_dtype is not None and np.dtype(requested_dtype) == np.dtype(np.float32): + pytest.fail("over-budget decoder output must be rejected before float32 normalization") + return original_array(value, *args, **kwargs) + + monkeypatch.setattr(audio_decode.np, "array", guarded_array) + + +def test_decode_rejects_sample_overflow_before_float32_copy( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """An oversized decoder result must not trigger a second canonical-buffer allocation.""" + decoder_output = np.zeros(3, dtype=np.float64) + policy = AudioResourcePolicy(target_sample_rate=2, max_duration_seconds=1.0) + monkeypatch.setattr(audio_decode, "preflight_audio_metadata", lambda *_args: None) + monkeypatch.setattr( + audio_decode.librosa, + "load", + lambda *_args, **_kwargs: (decoder_output, policy.target_sample_rate), + ) + _reject_float32_copy(monkeypatch) + + with pytest.raises(AudioResourcePolicyError) as caught: + audio_decode.decode_mono_audio(io.BytesIO(b"container"), policy=policy) + + assert caught.value.reason == "decoded_sample_count_exceeded" + assert caught.value.policy_version == AUDIO_RESOURCE_POLICY_VERSION + + +def test_decode_rejects_intermediate_memory_overflow_before_float32_copy( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """A wide decoder buffer above the byte budget must fail before normalization copies it.""" + decoder_output = np.zeros(3, dtype=np.float64) + policy = AudioResourcePolicy( + target_sample_rate=3, + max_duration_seconds=1.0, + max_decoded_audio_bytes=16, + ) + monkeypatch.setattr(audio_decode, "preflight_audio_metadata", lambda *_args: None) + monkeypatch.setattr( + audio_decode.librosa, + "load", + lambda *_args, **_kwargs: (decoder_output, policy.target_sample_rate), + ) + _reject_float32_copy(monkeypatch) + + with pytest.raises(AudioResourcePolicyError) as caught: + audio_decode.decode_mono_audio(io.BytesIO(b"container"), policy=policy) + + assert caught.value.reason == "memory_budget_exceeded" + assert caught.value.policy_version == AUDIO_RESOURCE_POLICY_VERSION + + +def test_decode_rejects_canonical_float32_expansion_before_copy( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Budget the future canonical float32 allocation, not only the decoder view.""" + decoder_output = np.zeros(8, dtype=np.float16) + policy = AudioResourcePolicy( + target_sample_rate=8, + max_duration_seconds=1.0, + max_decoded_audio_bytes=16, + ) + assert decoder_output.nbytes == policy.max_decoded_audio_bytes + assert decoder_output.size * np.dtype(np.float32).itemsize > policy.max_decoded_audio_bytes + monkeypatch.setattr(audio_decode, "preflight_audio_metadata", lambda *_args: None) + monkeypatch.setattr( + audio_decode.librosa, + "load", + lambda *_args, **_kwargs: (decoder_output, policy.target_sample_rate), + ) + _reject_float32_copy(monkeypatch) + + with pytest.raises(AudioResourcePolicyError) as caught: + audio_decode.decode_mono_audio(io.BytesIO(b"container"), policy=policy) + + assert caught.value.reason == "memory_budget_exceeded" + assert caught.value.policy_version == AUDIO_RESOURCE_POLICY_VERSION + + +def test_decode_maps_canonical_copy_memory_error_to_budget_rejection( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Allocator exhaustion must preserve the stable resource-policy failure contract.""" + decoder_output = np.zeros(2, dtype=np.float64) + policy = AudioResourcePolicy( + target_sample_rate=2, + max_duration_seconds=1.0, + max_decoded_audio_bytes=16, + ) + allocation_error = MemoryError("simulated allocator pressure") + monkeypatch.setattr(audio_decode, "preflight_audio_metadata", lambda *_args: None) + monkeypatch.setattr( + audio_decode.librosa, + "load", + lambda *_args, **_kwargs: (decoder_output, policy.target_sample_rate), + ) + + def exhausted_array(*_args: object, **_kwargs: object) -> np.ndarray[Any, Any]: + raise allocation_error + + monkeypatch.setattr(audio_decode.np, "array", exhausted_array) + + with pytest.raises(AudioResourcePolicyError) as caught: + audio_decode.decode_mono_audio(io.BytesIO(b"container"), policy=policy) + + assert caught.value.reason == "memory_budget_exceeded" + assert caught.value.policy_version == AUDIO_RESOURCE_POLICY_VERSION + assert caught.value.__cause__ is allocation_error + + +def test_decode_maps_array_materialization_memory_error_to_malformed_header( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Pre-canonical materialization failure must not masquerade as a budget rejection.""" + policy = AudioResourcePolicy(target_sample_rate=2, max_duration_seconds=1.0) + materialization_error = MemoryError("simulated pre-canonical materialization pressure") + monkeypatch.setattr(audio_decode, "preflight_audio_metadata", lambda *_args: None) + monkeypatch.setattr( + audio_decode.librosa, + "load", + lambda *_args, **_kwargs: ([0.0, 0.0], policy.target_sample_rate), + ) + + def exhausted_asarray(*_args: object, **_kwargs: object) -> np.ndarray[Any, Any]: + raise materialization_error + + monkeypatch.setattr(audio_decode.np, "asarray", exhausted_asarray) + + with pytest.raises(AudioResourcePolicyError) as caught: + audio_decode.decode_mono_audio(io.BytesIO(b"container"), policy=policy) + + assert caught.value.reason == "malformed_header" + assert caught.value.policy_version == AUDIO_RESOURCE_POLICY_VERSION + assert caught.value.__cause__ is materialization_error diff --git a/services/analysis-engine/tests/test_audio_decode_reproducibility.py b/services/analysis-engine/tests/test_audio_decode_reproducibility.py new file mode 100644 index 000000000..db62bbdc7 --- /dev/null +++ b/services/analysis-engine/tests/test_audio_decode_reproducibility.py @@ -0,0 +1,42 @@ +"""Reproducibility contracts for the canonical local-audio decoder.""" + +from __future__ import annotations + +import io + +import numpy as np +import pytest + +from bandscope_analysis import audio_decode +from bandscope_analysis.audio_resource_policy import DEFAULT_AUDIO_RESOURCE_POLICY + + +def test_decode_mono_audio_pins_canonical_dtype_and_resampler( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Do not let third-party decoder defaults redefine the MIR input signal.""" + source = io.BytesIO(b"container") + captured_kwargs: dict[str, object] = {} + + monkeypatch.setattr(audio_decode, "preflight_audio_metadata", lambda *_args: None) + + def load(_candidate: object, **kwargs: object) -> tuple[np.ndarray, int]: + captured_kwargs.update(kwargs) + return ( + np.array([0.25, -0.5], dtype=np.float32), + DEFAULT_AUDIO_RESOURCE_POLICY.target_sample_rate, + ) + + monkeypatch.setattr(audio_decode.librosa, "load", load) + + decoded, sample_rate = audio_decode.decode_mono_audio(source) + + assert captured_kwargs["dtype"] is np.float32 + assert captured_kwargs["res_type"] == "soxr_hq" + assert captured_kwargs["sr"] == DEFAULT_AUDIO_RESOURCE_POLICY.target_sample_rate + assert captured_kwargs["mono"] is True + assert ( + captured_kwargs["duration"] == DEFAULT_AUDIO_RESOURCE_POLICY.decode_probe_duration_seconds + ) + np.testing.assert_array_equal(decoded, np.array([0.25, -0.5], dtype=np.float32)) + assert sample_rate == DEFAULT_AUDIO_RESOURCE_POLICY.target_sample_rate diff --git a/services/analysis-engine/tests/test_audio_metadata.py b/services/analysis-engine/tests/test_audio_metadata.py new file mode 100644 index 000000000..990f12281 --- /dev/null +++ b/services/analysis-engine/tests/test_audio_metadata.py @@ -0,0 +1,109 @@ +"""Source-container metadata preflight regressions.""" + +from __future__ import annotations + +import io +from types import SimpleNamespace +from unittest.mock import patch + +import pytest + +from bandscope_analysis.audio_metadata import preflight_audio_metadata +from bandscope_analysis.audio_resource_policy import ( + AUDIO_RESOURCE_POLICY_VERSION, + AudioResourcePolicyError, +) + + +def _info(*, frames: int = 44_100, samplerate: int = 44_100, channels: int = 2) -> SimpleNamespace: + """Build the metadata subset consumed by the preflight boundary.""" + return SimpleNamespace(frames=frames, samplerate=samplerate, channels=channels) + + +@patch("bandscope_analysis.audio_metadata.soundfile.info") +def test_preflight_accepts_metadata_and_rewinds_the_caller_handle(mock_info: object) -> None: + """A successful metadata probe leaves the decoder handle at its beginning.""" + source = io.BytesIO(b"header-bytes") + + def inspect(handle: io.BytesIO) -> SimpleNamespace: + """Consume a small header before returning parsed metadata.""" + handle.read(3) + return _info() + + mock_info.side_effect = inspect # type: ignore[attr-defined] + + preflight_audio_metadata(source) + + assert source.tell() == 0 + + +@pytest.mark.parametrize( + ("info", "reason"), + [ + (_info(frames=44_100 * 901), "duration_exceeded"), + (_info(samplerate=7_999), "sampling_rate_unsupported"), + (_info(channels=3), "channel_count_unsupported"), + ], +) +@patch("bandscope_analysis.audio_metadata.soundfile.info") +def test_preflight_rejects_untrusted_source_metadata( + mock_info: object, + info: SimpleNamespace, + reason: str, +) -> None: + """Source duration, rate, and channel bounds fail before PCM decode.""" + mock_info.return_value = info # type: ignore[attr-defined] + + with pytest.raises(AudioResourcePolicyError, match="audio resource policy") as error: + preflight_audio_metadata(io.BytesIO(b"header")) + + assert error.value.reason == reason + + +@pytest.mark.parametrize( + "dependency_error", + [RuntimeError("decoder detail"), ValueError("decoder detail")], +) +def test_preflight_maps_parser_failures_to_payload_free_policy_error( + dependency_error: Exception, +) -> None: + """Container parser failures cannot masquerade as policy errors or leak decoder detail.""" + with patch( + "bandscope_analysis.audio_metadata.soundfile.info", + side_effect=dependency_error, + ): + with pytest.raises(AudioResourcePolicyError, match="audio resource policy") as error: + preflight_audio_metadata(io.BytesIO(b"bad-header")) + + assert error.value.reason == "malformed_header" + assert error.value.policy_version == AUDIO_RESOURCE_POLICY_VERSION + assert "decoder detail" not in str(error.value) + + +@patch("bandscope_analysis.audio_metadata.soundfile.info") +def test_preflight_maps_rewind_failures_to_payload_free_policy_error(mock_info: object) -> None: + """A handle that cannot rewind after probing cannot reach a decoder.""" + + class SeekFailsAfterProbe(io.BytesIO): + """Fail only when the metadata boundary tries to rewind the handle.""" + + def __init__(self) -> None: + """Initialize the caller-owned byte handle and seek counter.""" + super().__init__(b"header") + self.seek_count = 0 + + def seek(self, *args: object, **kwargs: object) -> int: + """Reject the second seek, which is the post-probe rewind.""" + self.seek_count += 1 + if self.seek_count == 2: + raise OSError("rewind failed") + return super().seek(*args, **kwargs) + + mock_info.return_value = _info() # type: ignore[attr-defined] + + with pytest.raises(AudioResourcePolicyError, match="audio resource policy") as error: + preflight_audio_metadata(SeekFailsAfterProbe()) + + assert error.value.reason == "malformed_header" + assert error.value.policy_version == AUDIO_RESOURCE_POLICY_VERSION + assert "rewind failed" not in str(error.value) diff --git a/services/analysis-engine/tests/test_audio_model_output_policy.py b/services/analysis-engine/tests/test_audio_model_output_policy.py new file mode 100644 index 000000000..df96b524c --- /dev/null +++ b/services/analysis-engine/tests/test_audio_model_output_policy.py @@ -0,0 +1,47 @@ +"""Regression tests for fail-closed source-separation model output.""" + +from __future__ import annotations + +import numpy as np +import pytest + +from bandscope_analysis.separation.audio_separator import _as_float_array + + +@pytest.mark.parametrize( + "values", + [ + np.array([], dtype=np.float32), + np.array([np.nan], dtype=np.float32), + np.array([np.inf], dtype=np.float32), + np.array([np.finfo(np.float64).max], dtype=np.float64), + ], +) +def test_model_output_rejects_empty_nonfinite_or_float32_overflow(values: np.ndarray) -> None: + """Malformed model stems must fail instead of becoming successful silence.""" + with pytest.raises(ValueError, match=r"^Stem separation produced invalid audio\.$"): + _as_float_array(values) + + +def test_model_output_rejects_non_mono_shape() -> None: + """Model output with an extra channel axis must not be flattened into rehearsal audio.""" + values = np.array([[0.25, -0.5], [0.75, -0.25]], dtype=np.float32) + + with pytest.raises(ValueError, match=r"^Stem separation produced invalid audio\.$"): + _as_float_array(values) + + +def test_model_output_wraps_non_numeric_conversion_errors() -> None: + """Non-numeric model output must fail with the stable payload-free error.""" + with pytest.raises(ValueError, match=r"^Stem separation produced invalid audio\.$"): + _as_float_array(object()) + + +def test_model_output_preserves_valid_finite_samples() -> None: + """Valid model samples remain finite float32 audio with their original values.""" + values = np.array([0.25, -0.5, 0.75], dtype=np.float64) + + result = _as_float_array(values) + + assert result.dtype == np.float32 + assert np.array_equal(result, values.astype(np.float32)) diff --git a/services/analysis-engine/tests/test_audio_resource_policy.py b/services/analysis-engine/tests/test_audio_resource_policy.py new file mode 100644 index 000000000..62653cbeb --- /dev/null +++ b/services/analysis-engine/tests/test_audio_resource_policy.py @@ -0,0 +1,237 @@ +"""Tests for the canonical local-audio resource policy.""" + +from __future__ import annotations + +import numpy as np +import pytest + +from bandscope_analysis.audio_resource_policy import ( + AUDIO_RESOURCE_POLICY_VERSION, + DEFAULT_AUDIO_RESOURCE_POLICY, + DEFAULT_MAX_SOURCE_CHANNELS, + DEFAULT_MAX_SOURCE_SAMPLE_RATE, + DEFAULT_MIN_SOURCE_CHANNELS, + DEFAULT_MIN_SOURCE_SAMPLE_RATE, + AudioResourcePolicy, + AudioResourcePolicyError, +) + + +def test_default_policy_has_stable_version_and_rehearsal_budget() -> None: + """The default policy exposes one versioned budget shared by analyzers.""" + assert AUDIO_RESOURCE_POLICY_VERSION == "3" + assert DEFAULT_AUDIO_RESOURCE_POLICY.max_encoded_file_bytes == 100 * 1024 * 1024 + assert DEFAULT_AUDIO_RESOURCE_POLICY.target_sample_rate == 44_100 + assert DEFAULT_AUDIO_RESOURCE_POLICY.max_duration_seconds == 15 * 60 + assert DEFAULT_AUDIO_RESOURCE_POLICY.max_decoded_samples == 44_100 * 15 * 60 + assert DEFAULT_AUDIO_RESOURCE_POLICY.max_decoded_audio_bytes == ( + 44_100 * 15 * 60 * np.dtype(np.float32).itemsize + ) + + +def test_oversized_encoded_file_exposes_stable_policy_reason() -> None: + """Encoded-size rejection carries a stable reason and policy version for UI/provenance.""" + policy = AudioResourcePolicy(max_encoded_file_bytes=100) + + with pytest.raises(AudioResourcePolicyError) as captured: + policy.validate_encoded_file_bytes(101) + + assert captured.value.reason == "encoded_file_too_large" + assert captured.value.policy_version == AUDIO_RESOURCE_POLICY_VERSION + assert "audio resource policy" in str(captured.value).lower() + + +def test_source_metadata_exposes_stable_policy_reasons() -> None: + """Container admission distinguishes duration, rate, and channel rejection reasons.""" + policy = AudioResourcePolicy(max_duration_seconds=1.0) + + with pytest.raises(AudioResourcePolicyError) as duration_rejection: + policy.validate_source_metadata(frames=44_101, sample_rate=44_100, channels=2) + assert duration_rejection.value.reason == "duration_exceeded" + assert duration_rejection.value.policy_version == AUDIO_RESOURCE_POLICY_VERSION + + with pytest.raises(AudioResourcePolicyError) as rate_rejection: + policy.validate_source_metadata( + frames=44_100, + sample_rate=DEFAULT_MAX_SOURCE_SAMPLE_RATE + 1, + channels=2, + ) + assert rate_rejection.value.reason == "sampling_rate_unsupported" + assert rate_rejection.value.policy_version == AUDIO_RESOURCE_POLICY_VERSION + + with pytest.raises(AudioResourcePolicyError) as channel_rejection: + policy.validate_source_metadata( + frames=44_100, + sample_rate=44_100, + channels=DEFAULT_MAX_SOURCE_CHANNELS + 1, + ) + assert channel_rejection.value.reason == "channel_count_unsupported" + assert channel_rejection.value.policy_version == AUDIO_RESOURCE_POLICY_VERSION + + +def test_decoded_memory_rejection_exposes_stable_policy_reason() -> None: + """Post-decode memory rejection remains machine-readable without exposing payload data.""" + policy = AudioResourcePolicy( + target_sample_rate=8, + max_duration_seconds=1.0, + max_decoded_audio_bytes=16, + ) + audio = np.zeros(4, dtype=np.float64) + + with pytest.raises(AudioResourcePolicyError) as captured: + policy.validate_decoded_audio(audio, 8) + + assert captured.value.reason == "memory_budget_exceeded" + assert captured.value.policy_version == AUDIO_RESOURCE_POLICY_VERSION + + +@pytest.mark.parametrize("file_size", [True, -1, 0, 101]) +def test_encoded_file_size_fails_closed_outside_policy(file_size: object) -> None: + """Invalid, empty, or oversized encoded inputs are rejected before decode.""" + policy = AudioResourcePolicy(max_encoded_file_bytes=100) + + with pytest.raises(ValueError, match="audio resource policy"): + policy.validate_encoded_file_bytes(file_size) + + +def test_encoded_file_size_accepts_exact_boundary() -> None: + """A non-empty encoded file exactly at the configured ceiling is accepted.""" + policy = AudioResourcePolicy(max_encoded_file_bytes=100) + + assert policy.validate_encoded_file_bytes(100) == 100 + + +def test_source_metadata_accepts_the_published_bounds() -> None: + """Source metadata accepts the inclusive rate, channel, and duration bounds.""" + policy = AudioResourcePolicy(max_duration_seconds=15 * 60) + + policy.validate_source_metadata( + frames=DEFAULT_MAX_SOURCE_SAMPLE_RATE * 15 * 60, + sample_rate=DEFAULT_MAX_SOURCE_SAMPLE_RATE, + channels=DEFAULT_MAX_SOURCE_CHANNELS, + ) + policy.validate_source_metadata( + frames=DEFAULT_MIN_SOURCE_SAMPLE_RATE, + sample_rate=DEFAULT_MIN_SOURCE_SAMPLE_RATE, + channels=DEFAULT_MIN_SOURCE_CHANNELS, + ) + + +@pytest.mark.parametrize( + ("frames", "sample_rate", "channels"), + [ + (DEFAULT_MAX_SOURCE_SAMPLE_RATE * (15 * 60 + 1), 44_100, 2), + (44_100, DEFAULT_MIN_SOURCE_SAMPLE_RATE - 1, 2), + (44_100, DEFAULT_MAX_SOURCE_SAMPLE_RATE + 1, 2), + (44_100, 44_100, DEFAULT_MAX_SOURCE_CHANNELS + 1), + (44_100, 44_100, DEFAULT_MIN_SOURCE_CHANNELS - 1), + (0, 44_100, 2), + (44_100, True, 2), + (44_100, 44_100, True), + (10**400, 44_100, 2), + ], +) +def test_source_metadata_fails_closed_before_decode( + frames: object, + sample_rate: object, + channels: object, +) -> None: + """Overlong and malformed source metadata cannot reach a decoder.""" + with pytest.raises(ValueError, match="audio resource policy"): + DEFAULT_AUDIO_RESOURCE_POLICY.validate_source_metadata(frames, sample_rate, channels) + + +@pytest.mark.parametrize( + ("audio", "sample_rate"), + [ + (np.zeros(8_001, dtype=np.float32), 8_000), + (np.zeros((2, 4_000), dtype=np.float32), 8_000), + (np.array([0.0, np.nan], dtype=np.float32), 8_000), + (np.array(["not-a-sample"], dtype=object), 8_000), + (np.zeros(10, dtype=np.int16), 8_000), + (np.zeros(10, dtype=np.float32), 0), + (np.zeros(10, dtype=np.float32), True), + ], +) +def test_decoded_audio_fails_closed_outside_policy( + audio: np.ndarray, + sample_rate: object, +) -> None: + """Decoded output is revalidated for type, shape, finiteness, rate, and sample budget.""" + policy = AudioResourcePolicy(target_sample_rate=8_000, max_duration_seconds=1.0) + + with pytest.raises(ValueError, match="audio resource policy"): + policy.validate_decoded_audio(audio, sample_rate) + + +def test_decoded_audio_rejects_buffer_above_memory_budget() -> None: + """A decoder cannot hide excessive memory behind an allowed sample count.""" + policy = AudioResourcePolicy( + target_sample_rate=8, + max_duration_seconds=1.0, + max_decoded_audio_bytes=16, + ) + audio = np.zeros(4, dtype=np.float64) + + with pytest.raises(ValueError, match="audio resource policy"): + policy.validate_decoded_audio(audio, 8) + + +def test_decoded_audio_accepts_exact_memory_boundary() -> None: + """A finite canonical buffer exactly at the memory ceiling is accepted.""" + policy = AudioResourcePolicy( + target_sample_rate=8, + max_duration_seconds=1.0, + max_decoded_audio_bytes=32, + ) + audio = np.zeros(8, dtype=np.float32) + + assert policy.validate_decoded_audio(audio, 8) is audio + + +def test_decoded_audio_accepts_exact_sample_boundary() -> None: + """A finite mono artifact exactly at the decoded-sample ceiling is accepted.""" + policy = AudioResourcePolicy(target_sample_rate=8_000, max_duration_seconds=1.0) + audio = np.zeros(8_000, dtype=np.float32) + + validated = policy.validate_decoded_audio(audio, 8_000) + + assert validated is audio + + +@pytest.mark.parametrize( + "kwargs", + [ + {"max_encoded_file_bytes": 0}, + {"target_sample_rate": 0}, + {"max_duration_seconds": 0.0}, + {"max_duration_seconds": float("inf")}, + {"max_decoded_audio_bytes": 0}, + {"max_decoded_audio_bytes": True}, + {"min_source_sample_rate": 0}, + {"max_source_channels": True}, + {"min_source_sample_rate": 48_000, "max_source_sample_rate": 44_100}, + {"min_source_channels": 2, "max_source_channels": 1}, + ], +) +def test_policy_configuration_itself_fails_closed(kwargs: dict[str, object]) -> None: + """Invalid policy construction cannot silently create an unbounded budget.""" + with pytest.raises(ValueError, match="audio resource policy"): + AudioResourcePolicy(**kwargs) # type: ignore[arg-type] + + +@pytest.mark.parametrize( + "kwargs", + [ + {"target_sample_rate": 10**400, "max_duration_seconds": 1.0}, + {"target_sample_rate": 1, "max_duration_seconds": 10**400}, + {"max_encoded_file_bytes": 10**400}, + {"max_decoded_audio_bytes": 10**400}, + ], +) +def test_policy_configuration_fails_closed_on_unrepresentable_limits( + kwargs: dict[str, object], +) -> None: + """Extreme integer limits cannot escape stable policy validation through overflow.""" + with pytest.raises(ValueError, match="audio resource policy"): + AudioResourcePolicy(**kwargs) # type: ignore[arg-type] diff --git a/services/analysis-engine/tests/test_audio_resource_policy_coverage_regressions.py b/services/analysis-engine/tests/test_audio_resource_policy_coverage_regressions.py new file mode 100644 index 000000000..8dfa3d688 --- /dev/null +++ b/services/analysis-engine/tests/test_audio_resource_policy_coverage_regressions.py @@ -0,0 +1,74 @@ +"""Coverage regressions for fail-closed audio resource admission branches.""" + +from __future__ import annotations + +from pathlib import Path + +import numpy as np +import pytest + +from bandscope_analysis.audio_resource_policy import AudioResourcePolicy +from bandscope_analysis.separation.audio_separator import ( + AudioSeparationConfig, + AudioStemSeparator, +) + + +def test_policy_rejects_boolean_duration_configuration() -> None: + """A Boolean duration must not be coerced into a one-second resource budget.""" + with pytest.raises(ValueError, match="audio resource policy"): + AudioResourcePolicy(max_duration_seconds=True) + + +def test_policy_rejects_less_than_one_decoded_sample_budget() -> None: + """A positive duration that represents less than one sample must fail closed.""" + with pytest.raises(ValueError, match="audio resource policy"): + AudioResourcePolicy(target_sample_rate=1, max_duration_seconds=0.5) + + +def test_separator_rejects_empty_internal_loader_result_before_model( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """An unexpected empty loader result must not reach Demucs inference.""" + audio_path = tmp_path / "unexpected-empty.wav" + audio_path.write_bytes(b"not-empty") + separator = AudioStemSeparator( + AudioSeparationConfig(target_sample_rate=8_000, max_file_bytes=1_000_000) + ) + monkeypatch.setattr( + separator, + "_load_audio", + lambda _path: (np.array([], dtype=np.float32), 8_000), + ) + + def fail_if_model_runs(_audio: np.ndarray, _sample_rate: int) -> dict[str, np.ndarray]: + raise AssertionError("empty decoded audio must be rejected before model inference") + + monkeypatch.setattr(separator, "_separate_signal", fail_if_model_runs) + + with pytest.raises(ValueError, match="Stem separation decode failed"): + separator.separate(audio_path) + + +def test_separator_rejects_zero_byte_file_before_decoder( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """A zero-byte selected source must fail before the decoder is invoked.""" + audio_path = tmp_path / "empty.wav" + audio_path.write_bytes(b"") + separator = AudioStemSeparator( + AudioSeparationConfig(target_sample_rate=8_000, max_file_bytes=1_000_000) + ) + + def fail_if_decoder_runs(*_args: object, **_kwargs: object) -> tuple[np.ndarray, int]: + raise AssertionError("zero-byte input must be rejected before decoder invocation") + + monkeypatch.setattr( + "bandscope_analysis.audio_decode.librosa.load", + fail_if_decoder_runs, + ) + + with pytest.raises(ValueError, match="Stem separation decode failed"): + separator.separate(audio_path) diff --git a/services/analysis-engine/tests/test_audio_resource_policy_dtype.py b/services/analysis-engine/tests/test_audio_resource_policy_dtype.py new file mode 100644 index 000000000..aad6a343d --- /dev/null +++ b/services/analysis-engine/tests/test_audio_resource_policy_dtype.py @@ -0,0 +1,34 @@ +"""Regression tests for the canonical decoded PCM dtype contract.""" + +from __future__ import annotations + +import numpy as np +import pytest + +from bandscope_analysis.audio_resource_policy import ( + AUDIO_RESOURCE_POLICY_VERSION, + AudioResourcePolicy, + AudioResourcePolicyError, +) + + +def test_canonical_decoded_dtype_contract_is_versioned() -> None: + """A stricter canonical PCM representation advances policy provenance.""" + assert AUDIO_RESOURCE_POLICY_VERSION == "3" + + +@pytest.mark.parametrize("dtype", [np.float16, np.float64]) +def test_noncanonical_floating_dtype_fails_closed(dtype: type[np.floating]) -> None: + """Floating buffers cannot bypass the canonical float32 PCM contract.""" + policy = AudioResourcePolicy( + target_sample_rate=8, + max_duration_seconds=1.0, + max_decoded_audio_bytes=64, + ) + audio = np.zeros(4, dtype=dtype) + + with pytest.raises(AudioResourcePolicyError) as captured: + policy.validate_decoded_audio(audio, 8) + + assert captured.value.reason == "decoded_dtype_unsupported" + assert captured.value.policy_version == AUDIO_RESOURCE_POLICY_VERSION diff --git a/services/analysis-engine/tests/test_audio_resource_policy_finiteness_memory.py b/services/analysis-engine/tests/test_audio_resource_policy_finiteness_memory.py new file mode 100644 index 000000000..5a4a99cf4 --- /dev/null +++ b/services/analysis-engine/tests/test_audio_resource_policy_finiteness_memory.py @@ -0,0 +1,43 @@ +"""Regression coverage for bounded decoded-audio finiteness validation.""" + +from __future__ import annotations + +import numpy as np +import pytest + +import bandscope_analysis.audio_resource_policy as resource_policy_module +from bandscope_analysis.audio_resource_policy import ( + AUDIO_RESOURCE_POLICY_VERSION, + AudioResourcePolicyError, +) + + +@pytest.mark.parametrize("nonfinite_value", [np.nan, np.inf, -np.inf]) +def test_finiteness_scan_caps_temporary_boolean_mask_and_preserves_rejection( + monkeypatch: pytest.MonkeyPatch, + nonfinite_value: float, +) -> None: + """Finiteness validation must not allocate one full-song boolean mask.""" + policy = resource_policy_module.AudioResourcePolicy( + target_sample_rate=44_100, + max_duration_seconds=24.0, + ) + audio = np.zeros(1_048_577, dtype=np.float32) + audio[-1] = nonfinite_value + observed_samples: list[int] = [] + original_isfinite = np.isfinite + + def tracking_isfinite(values: np.ndarray) -> np.ndarray: + observed_samples.append(values.size) + return original_isfinite(values) + + monkeypatch.setattr(resource_policy_module.np, "isfinite", tracking_isfinite) + + with pytest.raises(AudioResourcePolicyError) as captured: + policy.validate_decoded_audio(audio, 44_100) + + assert captured.value.reason == "malformed_header" + assert captured.value.policy_version == AUDIO_RESOURCE_POLICY_VERSION + assert observed_samples + assert max(observed_samples) * np.dtype(np.bool_).itemsize <= 1024 * 1024 + assert sum(observed_samples) == audio.size diff --git a/services/analysis-engine/tests/test_audio_resource_policy_integration.py b/services/analysis-engine/tests/test_audio_resource_policy_integration.py new file mode 100644 index 000000000..72ce8d0bd --- /dev/null +++ b/services/analysis-engine/tests/test_audio_resource_policy_integration.py @@ -0,0 +1,241 @@ +"""Cross-boundary regressions for canonical local-audio resource admission.""" + +from __future__ import annotations + +from types import SimpleNamespace +from unittest.mock import Mock + +import numpy as np +import pytest + +from bandscope_analysis.api import validate_analysis_job_request +from bandscope_analysis.audio_resource_policy import ( + DEFAULT_AUDIO_RESOURCE_POLICY, + AudioResourcePolicy, +) +from bandscope_analysis.separation.audio_separator import ( + AudioSeparationConfig, + AudioStemSeparator, +) +from bandscope_analysis.temporal.analyzer import TemporalAnalyzer + + +def _local_request(file_size_bytes: object) -> dict[str, object]: + """Build one local-audio request whose only variable is encoded byte metadata.""" + return { + "sourceKind": "local_audio", + "projectId": "policy-project", + "sourceLabel": "rehearsal.wav", + "roleFocus": [], + "localSource": { + "sourcePath": "/tmp/rehearsal.wav", + "fileName": "rehearsal.wav", + "extension": "wav", + "fileSizeBytes": file_size_bytes, + }, + } + + +@pytest.mark.parametrize( + "file_size_bytes", + [True, DEFAULT_AUDIO_RESOURCE_POLICY.max_encoded_file_bytes + 1], +) +def test_request_preflight_rejects_metadata_outside_canonical_policy( + file_size_bytes: object, +) -> None: + """Reject impossible/oversized metadata before orchestration starts expensive work.""" + with pytest.raises(ValueError, match="localSource.fileSizeBytes"): + validate_analysis_job_request(_local_request(file_size_bytes)) + + +def test_request_preflight_accepts_exact_encoded_byte_boundary() -> None: + """The service API accepts the same exact encoded-byte ceiling as the policy.""" + request = validate_analysis_job_request( + _local_request(DEFAULT_AUDIO_RESOURCE_POLICY.max_encoded_file_bytes) + ) + + assert ( + request["localSource"]["fileSizeBytes"] + == DEFAULT_AUDIO_RESOURCE_POLICY.max_encoded_file_bytes + ) + + +def test_temporal_decoder_probes_one_sample_past_duration_limit_and_fails_closed( + tmp_path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Temporal decode detects a one-sample-overlong source instead of silently truncating it.""" + import librosa + + policy = AudioResourcePolicy( + max_encoded_file_bytes=100, + target_sample_rate=8, + max_duration_seconds=1.0, + ) + source = tmp_path / "overlong.wav" + source.write_bytes(b"bounded") + monkeypatch.setattr( + "bandscope_analysis.audio_decode.preflight_audio_metadata", + lambda *_args, **_kwargs: None, + ) + captured: dict[str, object] = {} + + def fake_load(fileobj: object, **kwargs: object) -> tuple[np.ndarray, int]: + captured.update(kwargs) + return np.zeros(policy.max_decoded_samples + 1, dtype=np.float32), policy.target_sample_rate + + monkeypatch.setattr(librosa, "load", fake_load) + monkeypatch.setattr( + librosa.beat, + "beat_track", + lambda **_: (_ for _ in ()).throw( + AssertionError("analysis must not run after policy rejection") + ), + ) + + with pytest.raises(ValueError, match="audio resource policy"): + TemporalAnalyzer(resource_policy=policy).analyze(source) + + assert captured["duration"] == pytest.approx( + (policy.max_decoded_samples + 1) / policy.target_sample_rate + ) + assert captured["sr"] == policy.target_sample_rate + assert captured["mono"] is True + + +@pytest.mark.parametrize( + "metadata", + [ + SimpleNamespace(frames=44_100 * 901, samplerate=44_100, channels=2), + SimpleNamespace(frames=44_100, samplerate=7_999, channels=2), + SimpleNamespace(frames=44_100, samplerate=44_100, channels=3), + ], +) +def test_temporal_rejects_source_metadata_before_librosa_decode( + tmp_path, + monkeypatch: pytest.MonkeyPatch, + metadata: SimpleNamespace, +) -> None: + """Temporal analysis must inspect source metadata before resampling or truncation.""" + import librosa + + source = tmp_path / "source-metadata.wav" + source.write_bytes(b"bounded") + monkeypatch.setattr( + "bandscope_analysis.audio_metadata.soundfile.info", + lambda _fileobj: metadata, + ) + load_mock = Mock(side_effect=AssertionError("source metadata must be checked first")) + monkeypatch.setattr(librosa, "load", load_mock) + + with pytest.raises(ValueError, match="audio resource policy"): + TemporalAnalyzer().analyze(source) + + load_mock.assert_not_called() + + +def test_stem_decoder_probes_one_sample_past_duration_limit_and_fails_closed( + tmp_path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Stem separation consumes the same decoded-sample ceiling as temporal analysis.""" + import librosa + + config = AudioSeparationConfig( + target_sample_rate=8, + max_file_bytes=100, + max_duration_seconds=1.0, + ) + source = tmp_path / "overlong.wav" + source.write_bytes(b"bounded") + monkeypatch.setattr( + "bandscope_analysis.audio_decode.preflight_audio_metadata", + lambda *_args, **_kwargs: None, + ) + captured: dict[str, object] = {} + + def fake_load(fileobj: object, **kwargs: object) -> tuple[np.ndarray, int]: + captured.update(kwargs) + return np.zeros(9, dtype=np.float32), 8 + + monkeypatch.setattr(librosa, "load", fake_load) + monkeypatch.setattr( + AudioStemSeparator, + "_separate_signal", + lambda *_: (_ for _ in ()).throw( + AssertionError("model must not run after policy rejection") + ), + ) + + with pytest.raises(ValueError, match="audio resource policy"): + AudioStemSeparator(config).separate(source) + + assert captured["duration"] == pytest.approx(9 / 8) + assert captured["sr"] == 8 + assert captured["mono"] is True + + +@pytest.mark.parametrize( + "metadata", + [ + SimpleNamespace(frames=44_100 * 901, samplerate=44_100, channels=2), + SimpleNamespace(frames=44_100, samplerate=7_999, channels=2), + SimpleNamespace(frames=44_100, samplerate=44_100, channels=3), + ], +) +def test_stem_decoder_rejects_source_metadata_before_librosa_decode( + tmp_path, + monkeypatch: pytest.MonkeyPatch, + metadata: SimpleNamespace, +) -> None: + """Stem separation must inspect source metadata before mono conversion or model work.""" + import librosa + + source = tmp_path / "source-metadata.wav" + source.write_bytes(b"bounded") + monkeypatch.setattr( + "bandscope_analysis.audio_metadata.soundfile.info", + lambda _fileobj: metadata, + ) + load_mock = Mock(side_effect=AssertionError("source metadata must be checked first")) + monkeypatch.setattr(librosa, "load", load_mock) + + separator = AudioStemSeparator(AudioSeparationConfig(max_file_bytes=100)) + with pytest.raises(ValueError, match="audio resource policy"): + separator.separate(source) + + load_mock.assert_not_called() + + +def test_stem_decoder_rejects_nonfinite_decoded_output_before_model( + tmp_path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Decoder NaN/Inf values fail closed instead of being normalized into model input.""" + import librosa + + source = tmp_path / "nonfinite.wav" + source.write_bytes(b"bounded") + monkeypatch.setattr( + "bandscope_analysis.audio_decode.preflight_audio_metadata", + lambda *_args, **_kwargs: None, + ) + monkeypatch.setattr( + librosa, + "load", + lambda *args, **kwargs: (np.array([0.0, np.nan], dtype=np.float32), 8), + ) + monkeypatch.setattr( + AudioStemSeparator, + "_separate_signal", + lambda *_: (_ for _ in ()).throw(AssertionError("model must not receive non-finite audio")), + ) + + with pytest.raises(ValueError, match="audio resource policy"): + AudioStemSeparator( + AudioSeparationConfig( + target_sample_rate=8, + max_file_bytes=100, + max_duration_seconds=1.0, + ) + ).separate(source) diff --git a/services/analysis-engine/tests/test_audio_separator_device_boundary.py b/services/analysis-engine/tests/test_audio_separator_device_boundary.py new file mode 100644 index 000000000..e8545876b --- /dev/null +++ b/services/analysis-engine/tests/test_audio_separator_device_boundary.py @@ -0,0 +1,109 @@ +"""Device-boundary regressions for local Demucs separation.""" + +from __future__ import annotations + +import sys +from types import ModuleType + +import numpy as np +import pytest + +from bandscope_analysis.separation.audio_separator import AudioSeparationConfig, AudioStemSeparator + + +class _FakeModel: + """Expose the canonical Demucs source order used by production.""" + + sources = ["drums", "bass", "other", "vocals"] + + +class _DeviceTensor: + """Minimal tensor that refuses NumPy conversion until moved to CPU.""" + + def __init__(self, array: np.ndarray, *, on_cpu: bool) -> None: + self.array = np.asarray(array, dtype=np.float32) + self.on_cpu = on_cpu + + def float(self) -> "_DeviceTensor": + return _DeviceTensor(self.array.astype(np.float32), on_cpu=self.on_cpu) + + def mean(self, axis: int | None = None) -> float | "_DeviceTensor": + value = self.array.mean(axis=axis) + if axis is None: + return float(value) + return _DeviceTensor(np.asarray(value, dtype=np.float32), on_cpu=self.on_cpu) + + def std(self) -> float: + return float(self.array.std()) + + def cpu(self) -> "_DeviceTensor": + return _DeviceTensor(self.array, on_cpu=True) + + def numpy(self) -> np.ndarray: + if not self.on_cpu: + raise RuntimeError("can't convert cuda tensor to numpy") + return self.array + + def __getitem__(self, key: object) -> "_DeviceTensor": + return _DeviceTensor(self.array[key], on_cpu=self.on_cpu) + + def __add__(self, value: float) -> "_DeviceTensor": + return _DeviceTensor(self.array + value, on_cpu=self.on_cpu) + + def __sub__(self, value: float) -> "_DeviceTensor": + return _DeviceTensor(self.array - value, on_cpu=self.on_cpu) + + def __mul__(self, value: float) -> "_DeviceTensor": + return _DeviceTensor(self.array * value, on_cpu=self.on_cpu) + + def __truediv__(self, value: float) -> "_DeviceTensor": + return _DeviceTensor(self.array / value, on_cpu=self.on_cpu) + + +class _NoGrad: + def __enter__(self) -> None: + return None + + def __exit__(self, *args: object) -> None: + return None + + +def test_apply_model_moves_device_output_to_cpu_before_numpy( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """GPU-selected separation must cross the device boundary before NumPy conversion.""" + calls: dict[str, object] = {} + fake_torch = ModuleType("torch") + fake_torch.from_numpy = lambda array: _DeviceTensor(array, on_cpu=True) # type: ignore[attr-defined] + fake_torch.no_grad = _NoGrad # type: ignore[attr-defined] + + def fake_apply_model( + model: _FakeModel, + batch: _DeviceTensor, + *, + device: str, + split: bool, + overlap: float, + progress: bool, + ) -> _DeviceTensor: + calls.update(device=device, split=split, overlap=overlap, progress=progress) + source_values = np.arange(len(model.sources), dtype=np.float32).reshape(-1, 1, 1) + separated = np.broadcast_to(source_values, (len(model.sources), 2, 4)).copy() + return _DeviceTensor(separated[None], on_cpu=False) + + demucs_module = ModuleType("demucs") + apply_module = ModuleType("demucs.apply") + apply_module.apply_model = fake_apply_model # type: ignore[attr-defined] + demucs_module.apply = apply_module # type: ignore[attr-defined] + monkeypatch.setitem(sys.modules, "torch", fake_torch) + monkeypatch.setitem(sys.modules, "demucs", demucs_module) + monkeypatch.setitem(sys.modules, "demucs.apply", apply_module) + + audio = np.array([0.0, 1.0, -1.0, 0.5], dtype=np.float32) + separator = AudioStemSeparator(AudioSeparationConfig(device="cuda", overlap=0.375)) + + result = separator._apply_model(_FakeModel(), audio) + + assert calls == {"device": "cuda", "split": True, "overlap": 0.375, "progress": False} + assert set(result) == set(_FakeModel.sources) + assert all(stem.shape == (4,) for stem in result.values()) diff --git a/services/analysis-engine/tests/test_branch_coverage_contract.py b/services/analysis-engine/tests/test_branch_coverage_contract.py index 6198141c0..c7e3e83ff 100644 --- a/services/analysis-engine/tests/test_branch_coverage_contract.py +++ b/services/analysis-engine/tests/test_branch_coverage_contract.py @@ -95,10 +95,10 @@ def test_chord_segment_builder_handles_zero_frames_without_final_segment() -> No assert result == [] -def test_cli_skips_temporal_probe_when_local_source_path_is_empty( +def test_cli_delegates_empty_local_source_to_orchestration( monkeypatch: pytest.MonkeyPatch, ) -> None: - """Do not invoke the temporary temporal probe for an empty local source path.""" + """Keep local-source validation in the canonical analysis orchestration path.""" payload = { "jobId": "job-empty-source", "request": { @@ -111,17 +111,14 @@ def test_cli_skips_temporal_probe_when_local_source_path_is_empty( monkeypatch.setattr(cli.sys, "stdin", io.StringIO(json.dumps(payload))) monkeypatch.setattr(cli.sys, "stdout", stdout) - with ( - patch.object(cli, "TemporalAnalyzer") as temporal_analyzer, - patch.object( - cli, - "run_analysis_job", - return_value={"jobId": "job-empty-source", "state": "failed"}, - ), - ): + with patch.object( + cli, + "run_analysis_job", + return_value={"jobId": "job-empty-source", "state": "failed"}, + ) as run_analysis_job: assert cli.main() == 0 - temporal_analyzer.assert_not_called() + run_analysis_job.assert_called_once() assert json.loads(stdout.getvalue())["jobId"] == "job-empty-source" diff --git a/services/analysis-engine/tests/test_chord_recognizer.py b/services/analysis-engine/tests/test_chord_recognizer.py index 20a6dcf78..88ff6684c 100644 --- a/services/analysis-engine/tests/test_chord_recognizer.py +++ b/services/analysis-engine/tests/test_chord_recognizer.py @@ -3,6 +3,7 @@ from unittest.mock import patch import numpy as np +import pytest from bandscope_analysis.chords.chord_recognizer import ( ChordRecognizer, @@ -20,6 +21,14 @@ def test_chord_recognizer_empty_audio() -> None: assert result == [] +@pytest.mark.parametrize("shape", [(0, 2), (2, 0)]) +def test_chord_recognizer_empty_layouts(shape: tuple[int, int]) -> None: + """Every zero-element NumPy layout must short-circuit recognition.""" + recognizer = ChordRecognizer() + + assert recognizer.recognize(np.empty(shape), sr=22050) == [] + + def test_chord_recognizer_unvoiced_audio() -> None: """Test chord recognition with noise.""" recognizer = ChordRecognizer() diff --git a/services/analysis-engine/tests/test_cli.py b/services/analysis-engine/tests/test_cli.py index 057ef236b..ba6a755cd 100644 --- a/services/analysis-engine/tests/test_cli.py +++ b/services/analysis-engine/tests/test_cli.py @@ -320,93 +320,6 @@ def test_cli_main_job_arg_json_string(monkeypatch: pytest.MonkeyPatch) -> None: assert "job-raw" in stdout.getvalue() -def test_cli_main_temporal_analyzer_mock(monkeypatch: pytest.MonkeyPatch) -> None: - """Ensure the temporal analyzer injection block is covered and handles errors.""" - stdin = io.StringIO( - json.dumps( - { - "jobId": "job-audio", - "request": { - "sourceKind": "local_audio", - "projectId": "p1", - "sourceLabel": "test.wav", - "roleFocus": [], - "localSource": { - "sourcePath": "/invalid/path.wav", - "fileName": "test.wav", - "extension": "wav", - "fileSizeBytes": 100, - }, - }, - } - ) - ) - stdout = io.StringIO() - - class FakeAnalyzer: - def analyze(self, path): - raise RuntimeError("mocked failure") - - monkeypatch.setattr(cli, "TemporalAnalyzer", FakeAnalyzer) - monkeypatch.setattr(cli.sys, "stdin", stdin) - monkeypatch.setattr(cli.sys, "stdout", stdout) - monkeypatch.setattr(cli.sys, "argv", ["cli.py"]) - - assert cli.main() == 0 - res = json.loads(stdout.getvalue()) - assert res["jobId"] == "job-audio" - - -def test_cli_main_temporal_analyzer_mock_success( - monkeypatch: pytest.MonkeyPatch, - tmp_path, -) -> None: - """Ensure the temporal analyzer injection block succeeds.""" - audio_path = tmp_path / "test.wav" - write_short_wav(audio_path) - stdin = io.StringIO( - json.dumps( - { - "jobId": "job-audio-success", - "request": { - "sourceKind": "local_audio", - "projectId": "p1", - "sourceLabel": "test.wav", - "roleFocus": [], - "localSource": { - "sourcePath": str(audio_path), - "fileName": "test.wav", - "extension": "wav", - "fileSizeBytes": audio_path.stat().st_size, - }, - }, - } - ) - ) - stdout = io.StringIO() - - class FakeAnalyzerSuccess: - def analyze(self, path): - return {"bpm": 120.0, "beats": []} - - monkeypatch.setattr(cli, "TemporalAnalyzer", FakeAnalyzerSuccess) - monkeypatch.setattr( - "bandscope_analysis.ranges.pitch_tracker.PitchTracker.track", - lambda self, y, sr: None, - ) - monkeypatch.setattr( - "bandscope_analysis.chords.chord_recognizer.ChordRecognizer.recognize", - lambda self, y, sr: [], - ) - monkeypatch.setattr(cli.sys, "stdin", stdin) - monkeypatch.setattr(cli.sys, "stdout", stdout) - monkeypatch.setattr(cli.sys, "argv", ["cli.py"]) - - assert cli.main() == 0 - res = json.loads(stdout.getvalue()) - assert res["jobId"] == "job-audio-success" - - def test_cli_main_progress_jsonl_streams_status_updates( monkeypatch: pytest.MonkeyPatch, tmp_path, @@ -437,11 +350,6 @@ def test_cli_main_progress_jsonl_streams_status_updates( ) stdout = io.StringIO() - class FakeAnalyzerSuccess: - def analyze(self, path): - return {"bpm": 120.0, "beats": []} - - monkeypatch.setattr(cli, "TemporalAnalyzer", FakeAnalyzerSuccess) monkeypatch.setattr( "bandscope_analysis.ranges.pitch_tracker.PitchTracker.track", lambda self, y, sr: None, @@ -487,3 +395,56 @@ def fake_stem_separation(*args: Any, **kwargs: Any) -> dict[str, Any]: ] assert updates[-1]["state"] == "succeeded" assert updates[-1]["progressPercent"] == 100 + + +def test_cli_namespaces_local_cache_by_verified_source_digest( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + """Bind persisted local cache lookup to native verified source content identity.""" + digest = "ab" * 32 + captured_request: dict[str, Any] = {} + + def fake_run_analysis_job(job_id: str, request: object, requested_at: str) -> dict[str, Any]: + assert job_id == "job-bound-cache" + assert isinstance(request, dict) + captured_request.update(request) + return { + "jobId": job_id, + "state": "succeeded", + "requestedAt": requested_at, + "updatedAt": requested_at, + "result": {}, + } + + stdin = io.StringIO( + json.dumps( + { + "jobId": "job-bound-cache", + "sourceContentSha256": digest, + "request": { + "sourceKind": "local_audio", + "projectId": "p1", + "sourceLabel": "test.wav", + "roleFocus": [], + "localSource": { + "sourcePath": str(tmp_path / "source.wav"), + "fileName": "test.wav", + "extension": "wav", + "fileSizeBytes": 4, + }, + "cacheRoot": str(tmp_path / "cache"), + }, + } + ) + ) + stdout = io.StringIO() + monkeypatch.setattr(cli, "run_analysis_job", fake_run_analysis_job) + monkeypatch.setattr(cli.sys, "stdin", stdin) + monkeypatch.setattr(cli.sys, "stdout", stdout) + monkeypatch.setattr(cli.sys, "argv", ["cli.py"]) + + assert cli.main() == 0 + assert captured_request["cacheRoot"] == str( + tmp_path / "cache" / "source-sha256-v1" / digest + ) diff --git a/services/analysis-engine/tests/test_cli_requested_at_authority.py b/services/analysis-engine/tests/test_cli_requested_at_authority.py new file mode 100644 index 000000000..490abdf3f --- /dev/null +++ b/services/analysis-engine/tests/test_cli_requested_at_authority.py @@ -0,0 +1,126 @@ +"""Regression tests for native ownership of analysis request timestamps.""" + +from __future__ import annotations + +import io +import json +from typing import Any + +import pytest + +from bandscope_analysis import cli + + +def test_progress_jsonl_preserves_native_requested_at(monkeypatch: pytest.MonkeyPatch) -> None: + """Require the helper to echo the native request timestamp without re-minting it.""" + requested_at = "2026-09-09T00:57:21Z" + observed: dict[str, str] = {} + stdin = io.StringIO( + json.dumps( + { + "jobId": "job-native-time", + "requestedAt": requested_at, + "request": { + "sourceKind": "demo", + "sourceLabel": "Native Clock Authority", + "roleFocus": [], + }, + } + ) + ) + stdout = io.StringIO() + + def fake_updates( + job_id: str, + request: object, + helper_requested_at: str, + ) -> list[dict[str, Any]]: + del request + observed["requested_at"] = helper_requested_at + return [ + { + "jobId": job_id, + "state": "failed", + "requestedAt": helper_requested_at, + "updatedAt": helper_requested_at, + "error": { + "code": "engine_unavailable", + "message": "fixture terminal status", + }, + } + ] + + monkeypatch.setattr(cli, "run_analysis_job_updates", fake_updates) + monkeypatch.setattr(cli.sys, "stdin", stdin) + monkeypatch.setattr(cli.sys, "stdout", stdout) + monkeypatch.setattr(cli.sys, "argv", ["cli.py", "--progress-jsonl"]) + + assert cli.main() == 0 + assert observed["requested_at"] == requested_at + assert json.loads(stdout.getvalue())["requestedAt"] == requested_at + + +def test_progress_jsonl_delegates_local_audio_analysis_once( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Keep local-audio MIR work inside the orchestration owner instead of pre-running it in CLI.""" + requested_at = "2026-09-09T01:00:00Z" + observed = {"temporal_calls": 0, "orchestration_calls": 0} + stdin = io.StringIO( + json.dumps( + { + "jobId": "job-single-analysis-owner", + "requestedAt": requested_at, + "request": { + "sourceKind": "local_audio", + "projectId": "project-1-1", + "sourceLabel": "rehearsal.wav", + "roleFocus": [], + "localSource": { + "sourcePath": "/app-owned/project/source.wav", + "fileName": "rehearsal.wav", + "extension": "wav", + "fileSizeBytes": 1024, + }, + }, + } + ) + ) + stdout = io.StringIO() + + class CountingTemporalAnalyzer: + def __init__(self) -> None: + observed["temporal_calls"] += 1 + + def analyze(self, _audio_path: str) -> dict[str, float]: + return {"bpm": 120.0} + + def fake_updates( + job_id: str, + request: object, + helper_requested_at: str, + ) -> list[dict[str, Any]]: + del request + observed["orchestration_calls"] += 1 + return [ + { + "jobId": job_id, + "state": "failed", + "requestedAt": helper_requested_at, + "updatedAt": helper_requested_at, + "error": { + "code": "engine_unavailable", + "message": "fixture terminal status", + }, + } + ] + + monkeypatch.setattr(cli, "TemporalAnalyzer", CountingTemporalAnalyzer, raising=False) + monkeypatch.setattr(cli, "run_analysis_job_updates", fake_updates) + monkeypatch.setattr(cli.sys, "stdin", stdin) + monkeypatch.setattr(cli.sys, "stdout", stdout) + monkeypatch.setattr(cli.sys, "argv", ["cli.py", "--progress-jsonl"]) + + assert cli.main() == 0 + assert observed["orchestration_calls"] == 1 + assert observed["temporal_calls"] == 0 diff --git a/services/analysis-engine/tests/test_cli_source_identity_cache.py b/services/analysis-engine/tests/test_cli_source_identity_cache.py new file mode 100644 index 000000000..9477a6435 --- /dev/null +++ b/services/analysis-engine/tests/test_cli_source_identity_cache.py @@ -0,0 +1,83 @@ +"""Regression coverage for native source-identity cache scoping.""" + +from __future__ import annotations + +import io +import json + +import pytest + +from bandscope_analysis import cli +from bandscope_analysis.cli import _bind_verified_source_cache_namespace + + +def _local_request(cache_root: object = "/tmp/cache") -> dict[str, object]: + """Return the minimum local request surface relevant to cache scoping.""" + return { + "sourceKind": "local_audio", + "cacheRoot": cache_root, + } + + +def test_missing_verified_digest_disables_persisted_local_cache() -> None: + """Do not reuse path/name/size-addressed caches when exact source identity is absent.""" + bound = _bind_verified_source_cache_namespace(_local_request(), None) + + assert isinstance(bound, dict) + assert "cacheRoot" not in bound + + +def test_invalid_verified_digest_fails_closed() -> None: + """Reject a malformed digest instead of creating an attacker-shaped cache namespace.""" + with pytest.raises(ValueError, match="sourceContentSha256"): + _bind_verified_source_cache_namespace(_local_request(), "../not-a-digest") + + +def test_demo_request_rejects_local_source_digest_authority() -> None: + """Keep source-content authority exclusive to local-audio analysis.""" + with pytest.raises(ValueError, match="sourceContentSha256"): + _bind_verified_source_cache_namespace({"sourceKind": "demo"}, "ab" * 32) + + +def test_invalid_cache_root_type_remains_for_canonical_request_validation() -> None: + """Do not hide an independently malformed cacheRoot when no digest is present.""" + request = _local_request(cache_root=7) + + assert _bind_verified_source_cache_namespace(request, None) == request + + +def test_missing_digest_does_not_hide_invalid_cache_root( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Validate request fields before digest scoping can disable persisted caching.""" + stdin = io.StringIO( + json.dumps( + { + "jobId": "job-invalid-cache-root", + "request": { + "sourceKind": "local_audio", + "projectId": "project-1", + "sourceLabel": "source.wav", + "roleFocus": [], + "localSource": { + "sourcePath": "/tmp/source.wav", + "fileName": "source.wav", + "extension": "wav", + "fileSizeBytes": 4, + }, + "cacheRoot": "../cache", + }, + } + ) + ) + stdout = io.StringIO() + monkeypatch.setattr(cli.sys, "stdin", stdin) + monkeypatch.setattr(cli.sys, "stdout", stdout) + monkeypatch.setattr(cli.sys, "argv", ["cli.py"]) + + assert cli.main() == 0 + response = json.loads(stdout.getvalue()) + assert response["state"] == "failed" + assert response["error"]["message"] == ( + "Invalid analysis job request: path traversal detected in 'cacheRoot'" + ) diff --git a/services/analysis-engine/tests/test_cli_source_identity_temp_scope.py b/services/analysis-engine/tests/test_cli_source_identity_temp_scope.py new file mode 100644 index 000000000..5e3bfc24b --- /dev/null +++ b/services/analysis-engine/tests/test_cli_source_identity_temp_scope.py @@ -0,0 +1,338 @@ +"""Regression coverage for source-bound temporary stem-work paths.""" + +from __future__ import annotations + +import shutil +from pathlib import Path + +import pytest + +from bandscope_analysis import cli +from bandscope_analysis.cli import ( + _bind_verified_source_cache_namespace, + _cleanup_job_temp_namespace, +) + + +def test_verified_digest_scopes_temp_work_to_exact_source_identity() -> None: + """Prevent same-metadata replacement audio from sharing one stem-work path.""" + source_digest = "ab" * 32 + request = { + "sourceKind": "local_audio", + "cacheRoot": "/tmp/cache", + "tempRoot": "/tmp/work", + } + + bound = _bind_verified_source_cache_namespace(request, source_digest) + + assert isinstance(bound, dict) + assert Path(str(bound["cacheRoot"])).parts[-2:] == ( + "source-sha256-v1", + source_digest, + ) + assert Path(str(bound["tempRoot"])).parts[-2:] == ( + "source-sha256-v1", + source_digest, + ) + + +def test_same_source_concurrent_jobs_use_distinct_temp_work_namespaces() -> None: + """Keep the two admitted concurrent jobs from sharing one stem-work artifact path.""" + source_digest = "ab" * 32 + request = { + "sourceKind": "local_audio", + "cacheRoot": "/tmp/cache", + "tempRoot": "/tmp/work", + } + + first = _bind_verified_source_cache_namespace(request, source_digest, "job-1") + second = _bind_verified_source_cache_namespace(request, source_digest, "job-2") + + assert isinstance(first, dict) + assert isinstance(second, dict) + assert first["cacheRoot"] == second["cacheRoot"] + assert first["tempRoot"] != second["tempRoot"] + assert Path(str(first["tempRoot"])).parts[-4:-2] == ( + "source-sha256-v1", + source_digest, + ) + assert Path(str(first["tempRoot"])).parts[-2] == "job-sha256-v1" + assert len(Path(str(first["tempRoot"])).parts[-1]) == 64 + + +def test_unverified_job_disables_cache_but_still_isolates_temp_work() -> None: + """Keep manual local jobs isolated even when persisted cache reuse is disabled.""" + bound = _bind_verified_source_cache_namespace( + { + "sourceKind": "local_audio", + "cacheRoot": "/tmp/cache", + "tempRoot": "/tmp/work", + }, + None, + "manual-job", + ) + + assert isinstance(bound, dict) + assert "cacheRoot" not in bound + assert Path(str(bound["tempRoot"])).parts[-2] == "job-sha256-v1" + assert len(Path(str(bound["tempRoot"])).parts[-1]) == 64 + + +def test_job_temp_cleanup_removes_only_the_derived_execution_namespace(tmp_path: Path) -> None: + """Remove completed stem work without deleting the reusable source namespace.""" + source_digest = "ab" * 32 + bound = _bind_verified_source_cache_namespace( + { + "sourceKind": "local_audio", + "cacheRoot": str(tmp_path / "cache"), + "tempRoot": str(tmp_path / "work"), + }, + source_digest, + "job-cleanup", + ) + assert isinstance(bound, dict) + job_root = Path(str(bound["tempRoot"])) + source_root = job_root.parents[1] + stem_path = job_root / "stem-work-v1" / "stems.npz" + stem_path.parent.mkdir(parents=True) + stem_path.write_bytes(b"temporary-stems") + sibling = source_root / "keep.txt" + sibling.write_text("keep", encoding="utf-8") + + _cleanup_job_temp_namespace(bound) + + assert not job_root.exists() + assert sibling.read_text(encoding="utf-8") == "keep" + + +def test_unverified_job_temp_cleanup_removes_only_its_job_namespace(tmp_path: Path) -> None: + """Clean manual-job stem work without requiring persisted source identity.""" + bound = _bind_verified_source_cache_namespace( + { + "sourceKind": "local_audio", + "tempRoot": str(tmp_path / "work"), + }, + None, + "manual-cleanup", + ) + assert isinstance(bound, dict) + job_root = Path(str(bound["tempRoot"])) + stem_path = job_root / "stem-work-v1" / "stems.npz" + stem_path.parent.mkdir(parents=True) + stem_path.write_bytes(b"temporary-stems") + + _cleanup_job_temp_namespace(bound) + + assert not job_root.exists() + + +def test_job_temp_cleanup_refuses_symlinked_source_namespace(tmp_path: Path) -> None: + """Do not follow a substituted derived parent outside app-owned stem work.""" + source_digest = "ab" * 32 + job_digest = "cd" * 32 + outside_root = tmp_path / "outside" + outside_job = outside_root / "job-sha256-v1" / job_digest + outside_job.mkdir(parents=True) + sentinel = outside_job / "keep.txt" + sentinel.write_text("keep", encoding="utf-8") + + source_scope = tmp_path / "work" / "source-sha256-v1" + source_scope.mkdir(parents=True) + source_link = source_scope / source_digest + try: + source_link.symlink_to(outside_root, target_is_directory=True) + except OSError as error: + pytest.skip(f"directory symlink unavailable: {error}") + + _cleanup_job_temp_namespace( + {"tempRoot": str(source_link / "job-sha256-v1" / job_digest)} + ) + + assert sentinel.read_text(encoding="utf-8") == "keep" + + +def test_job_temp_cleanup_refuses_symlinked_temp_root(tmp_path: Path) -> None: + """A caller-selected temp root symlink cannot redirect recursive cleanup.""" + job_digest = "cd" * 32 + outside_root = tmp_path / "outside" + outside_job = outside_root / "job-sha256-v1" / job_digest + outside_job.mkdir(parents=True) + sentinel = outside_job / "keep.txt" + sentinel.write_text("keep", encoding="utf-8") + temp_root = tmp_path / "work-link" + try: + temp_root.symlink_to(outside_root, target_is_directory=True) + except OSError as error: + pytest.skip(f"directory symlink unavailable: {error}") + + _cleanup_job_temp_namespace( + {"tempRoot": str(temp_root / "job-sha256-v1" / job_digest)} + ) + + assert sentinel.read_text(encoding="utf-8") == "keep" + + +def test_job_temp_cleanup_refuses_verified_symlinked_temp_root(tmp_path: Path) -> None: + """Verified-source scoping cannot make a symlinked caller temp root deletable.""" + source_digest = "ab" * 32 + job_digest = "cd" * 32 + outside_root = tmp_path / "outside" + outside_job = ( + outside_root + / "source-sha256-v1" + / source_digest + / "job-sha256-v1" + / job_digest + ) + outside_job.mkdir(parents=True) + sentinel = outside_job / "keep.txt" + sentinel.write_text("keep", encoding="utf-8") + temp_root = tmp_path / "work-link" + try: + temp_root.symlink_to(outside_root, target_is_directory=True) + except OSError as error: + pytest.skip(f"directory symlink unavailable: {error}") + + _cleanup_job_temp_namespace( + { + "tempRoot": str( + temp_root + / "source-sha256-v1" + / source_digest + / "job-sha256-v1" + / job_digest + ) + } + ) + + assert sentinel.read_text(encoding="utf-8") == "keep" + + +def test_job_temp_cleanup_survives_parent_swap_after_lexical_check( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Do not let a post-check parent swap redirect cleanup outside the temp tree.""" + if not shutil.rmtree.avoids_symlink_attacks: + pytest.skip("symlink-resistant rmtree unavailable") + + job_digest = "cd" * 32 + temp_root = tmp_path / "work" + job_root = temp_root / "job-sha256-v1" / job_digest + job_root.mkdir(parents=True) + inside = job_root / "inside.txt" + inside.write_text("temporary", encoding="utf-8") + + outside_root = tmp_path / "outside" + outside_job = outside_root / "job-sha256-v1" / job_digest + outside_job.mkdir(parents=True) + sentinel = outside_job / "keep.txt" + sentinel.write_text("keep", encoding="utf-8") + parked_root = tmp_path / "parked-work" + + original_is_symlink = Path.is_symlink + swapped = False + + def swap_after_check(candidate: Path) -> bool: + nonlocal swapped + is_link = original_is_symlink(candidate) + if candidate == temp_root and not is_link and not swapped: + temp_root.rename(parked_root) + try: + temp_root.symlink_to(outside_root, target_is_directory=True) + except OSError as error: + pytest.skip(f"directory symlink unavailable: {error}") + swapped = True + return is_link + + monkeypatch.setattr(Path, "is_symlink", swap_after_check) + + _cleanup_job_temp_namespace({"tempRoot": str(job_root)}) + + assert swapped + assert sentinel.read_text(encoding="utf-8") == "keep" + + +def test_job_temp_cleanup_refuses_runtime_without_dir_fd( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Leave work intact when descriptor-relative directory opens are unavailable.""" + job_digest = "cd" * 32 + job_root = tmp_path / "work" / "job-sha256-v1" / job_digest + sentinel = job_root / "keep.txt" + sentinel.parent.mkdir(parents=True) + sentinel.write_text("keep", encoding="utf-8") + monkeypatch.setattr(cli.os, "supports_dir_fd", set()) + + _cleanup_job_temp_namespace({"tempRoot": str(job_root)}) + + assert sentinel.read_text(encoding="utf-8") == "keep" + + +def test_job_temp_cleanup_refuses_unsafe_rmtree_runtime( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Leave work for later cleanup when recursive deletion lacks symlink resistance.""" + bound = _bind_verified_source_cache_namespace( + { + "sourceKind": "local_audio", + "tempRoot": str(tmp_path / "work"), + }, + "ab" * 32, + "unsafe-runtime", + ) + assert isinstance(bound, dict) + job_root = Path(str(bound["tempRoot"])) + sentinel = job_root / "keep.txt" + sentinel.parent.mkdir(parents=True) + sentinel.write_text("keep", encoding="utf-8") + monkeypatch.setattr(cli.shutil.rmtree, "avoids_symlink_attacks", False) + + _cleanup_job_temp_namespace(bound) + + assert sentinel.read_text(encoding="utf-8") == "keep" + + +def test_job_temp_cleanup_refuses_unscoped_or_malformed_roots(tmp_path: Path) -> None: + """Do not grant recursive deletion authority outside a derived job namespace.""" + unsafe_root = tmp_path / "caller-root" + unsafe_root.mkdir() + sentinel = unsafe_root / "keep.txt" + sentinel.write_text("keep", encoding="utf-8") + malformed = tmp_path / "job-sha256-v1" / "not-a-digest" + malformed.mkdir(parents=True) + malformed_sentinel = malformed / "keep.txt" + malformed_sentinel.write_text("keep", encoding="utf-8") + malformed_source = ( + tmp_path + / "source-sha256-v1" + / "not-a-source-digest" + / "job-sha256-v1" + / ("ef" * 32) + ) + malformed_source.mkdir(parents=True) + malformed_source_sentinel = malformed_source / "keep.txt" + malformed_source_sentinel.write_text("keep", encoding="utf-8") + traversal_parent = tmp_path / "safe" + traversal_parent.mkdir() + traversal_target = tmp_path / "job-sha256-v1" / ("aa" * 32) + traversal_target.mkdir(parents=True) + traversal_sentinel = traversal_target / "keep.txt" + traversal_sentinel.write_text("keep", encoding="utf-8") + traversal_spelling = traversal_parent / ".." / "job-sha256-v1" / ("aa" * 32) + + _cleanup_job_temp_namespace(None) + _cleanup_job_temp_namespace({}) + _cleanup_job_temp_namespace({"tempRoot": "relative"}) + _cleanup_job_temp_namespace({"tempRoot": f"job-sha256-v1/{'ef' * 32}"}) + _cleanup_job_temp_namespace({"tempRoot": str(unsafe_root)}) + _cleanup_job_temp_namespace({"tempRoot": str(malformed)}) + _cleanup_job_temp_namespace({"tempRoot": str(malformed_source)}) + _cleanup_job_temp_namespace({"tempRoot": str(traversal_spelling)}) + + assert sentinel.read_text(encoding="utf-8") == "keep" + assert malformed_sentinel.read_text(encoding="utf-8") == "keep" + assert malformed_source_sentinel.read_text(encoding="utf-8") == "keep" + assert traversal_sentinel.read_text(encoding="utf-8") == "keep" diff --git a/services/analysis-engine/tests/test_feature_cache_archive_path_admission.py b/services/analysis-engine/tests/test_feature_cache_archive_path_admission.py new file mode 100644 index 000000000..209eb2cea --- /dev/null +++ b/services/analysis-engine/tests/test_feature_cache_archive_path_admission.py @@ -0,0 +1,49 @@ +"""Regressions for persisted feature-cache archive path admission.""" + +from __future__ import annotations + +import json +import os +from pathlib import Path + +import numpy as np +import pytest + +from bandscope_analysis.feature_cache_admission import load_bounded_stem_archive + + +def test_feature_cache_replay_rejects_symlinked_archive(tmp_path: Path) -> None: + """A cache pathname cannot redirect replay to a different filesystem object.""" + if not getattr(os, "O_NOFOLLOW", 0): + pytest.skip("platform does not expose O_NOFOLLOW") + + target_path = tmp_path / "outside-cache.npz" + arrays_path = tmp_path / "features.npz" + metadata_path = arrays_path.with_suffix(".json") + + np.savez_compressed( + target_path, + stem_bass=np.zeros(16, dtype=np.float32), + ) + metadata_path.write_text( + json.dumps( + { + "schemaVersion": 1, + "sampleRate": 44_100, + "separation": { + "duration_seconds": 16 / 44_100, + "chunk_count": 1, + "notes": "cached rehearsal stem", + }, + "stemKeys": ["bass"], + "stemRoleTypes": {"bass": "instrument"}, + } + ), + encoding="utf-8", + ) + try: + arrays_path.symlink_to(target_path) + except (NotImplementedError, OSError) as error: + pytest.skip(f"symlink creation unavailable: {error}") + + assert load_bounded_stem_archive(arrays_path, ["bass"], 44_100) is None diff --git a/services/analysis-engine/tests/test_feature_cache_json_number_admission.py b/services/analysis-engine/tests/test_feature_cache_json_number_admission.py new file mode 100644 index 000000000..7cbffea2d --- /dev/null +++ b/services/analysis-engine/tests/test_feature_cache_json_number_admission.py @@ -0,0 +1,16 @@ +"""Regression coverage for JSON numeric limits at the feature-cache boundary.""" + +from pathlib import Path + +from bandscope_analysis.feature_cache_admission import read_bounded_feature_cache_metadata + + +def test_feature_cache_metadata_huge_integer_is_a_cache_miss(tmp_path: Path) -> None: + """A bounded sidecar with an over-limit JSON integer must not escape as ValueError.""" + metadata_path = tmp_path / "features.json" + metadata_path.write_text( + '{"schemaVersion":1,"sampleRate":' + ("9" * 5_000) + "}", + encoding="utf-8", + ) + + assert read_bounded_feature_cache_metadata(metadata_path) is None diff --git a/services/analysis-engine/tests/test_feature_cache_metadata_admission.py b/services/analysis-engine/tests/test_feature_cache_metadata_admission.py new file mode 100644 index 000000000..e9a3615ac --- /dev/null +++ b/services/analysis-engine/tests/test_feature_cache_metadata_admission.py @@ -0,0 +1,72 @@ +"""Regression tests for bounded feature-cache metadata admission.""" + +from __future__ import annotations + +import importlib +from pathlib import Path + +import pytest + +from bandscope_analysis import feature_cache_admission + +api = importlib.import_module("bandscope_analysis.api") + +_MAX_ADMITTED_METADATA_BYTES = 1024 * 1024 + + +def _write_oversized_metadata(metadata_path: Path) -> None: + metadata_path.write_text( + '{"padding":"' + ("x" * _MAX_ADMITTED_METADATA_BYTES) + '"}', + encoding="utf-8", + ) + + +def test_oversized_feature_cache_metadata_fails_before_json_materialization( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Reject oversized sidecar bytes before the bounded reader allocates JSON.""" + metadata_path = tmp_path / "features.json" + _write_oversized_metadata(metadata_path) + + def fail_json_loads(*_args: object, **_kwargs: object) -> object: + raise MemoryError("oversized cache metadata must not reach json.loads") + + monkeypatch.setattr(feature_cache_admission.json, "loads", fail_json_loads) + + assert feature_cache_admission.read_bounded_feature_cache_metadata(metadata_path) is None + + +def test_first_feature_cache_snapshot_uses_bounded_metadata_reader( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Reject oversized first-read metadata before the API can materialize JSON.""" + metadata_path = tmp_path / "features.json" + arrays_path = tmp_path / "features.npz" + _write_oversized_metadata(metadata_path) + + def fail_json_loads(*_args: object, **_kwargs: object) -> object: + raise MemoryError("first cache snapshot must share bounded JSON admission") + + monkeypatch.setattr(feature_cache_admission.json, "loads", fail_json_loads) + + assert api._load_cached_local_audio_features(metadata_path, arrays_path) is None + + +@pytest.mark.parametrize( + "encoded_metadata", + [ + '{"schemaVersion":1,"schemaVersion":2}', + '{"separation":{"duration_seconds":1.0,"duration_seconds":2.0}}', + ], +) +def test_feature_cache_metadata_rejects_duplicate_json_members( + tmp_path: Path, + encoded_metadata: str, +) -> None: + """Reject ambiguous duplicate members at any JSON object depth.""" + metadata_path = tmp_path / "features.json" + metadata_path.write_text(encoded_metadata, encoding="utf-8") + + assert feature_cache_admission.read_bounded_feature_cache_metadata(metadata_path) is None diff --git a/services/analysis-engine/tests/test_feature_cache_metadata_generation.py b/services/analysis-engine/tests/test_feature_cache_metadata_generation.py new file mode 100644 index 000000000..9befc70fa --- /dev/null +++ b/services/analysis-engine/tests/test_feature_cache_metadata_generation.py @@ -0,0 +1,196 @@ +"""Regressions for persisted feature-cache metadata generation consistency.""" + +from __future__ import annotations + +import json +from pathlib import Path + +import numpy as np +import pytest + +from bandscope_analysis.api import _load_cached_local_audio_features +from bandscope_analysis.feature_cache_admission import load_bounded_stem_archive + + +def _write_cache(metadata_path: Path, arrays_path: Path) -> None: + """Write one admissible cached bass stem and its metadata sidecar.""" + metadata_path.write_text( + json.dumps( + { + "schemaVersion": 1, + "sampleRate": 44_100, + "separation": { + "duration_seconds": 16 / 44_100, + "chunk_count": 1, + "notes": "cached rehearsal stem", + }, + "stemKeys": ["bass"], + "stemRoleTypes": {"bass": "instrument"}, + } + ), + encoding="utf-8", + ) + np.savez_compressed(arrays_path, stem_bass=np.zeros(16, dtype=np.float32)) + + +def test_feature_cache_replay_rejects_second_read_sample_rate_substitution( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """A second-read sidecar cannot substitute a different analysis sample rate.""" + metadata_path = tmp_path / "features.json" + arrays_path = tmp_path / "features.npz" + _write_cache(metadata_path, arrays_path) + + real_loader = load_bounded_stem_archive + + def replace_sidecar_then_load( + archive_path: Path, + stem_keys: list[str], + sample_rate: object, + *, + policy_template, + ): + payload = json.loads(metadata_path.read_text(encoding="utf-8")) + payload["sampleRate"] = 48_000 + metadata_path.write_text(json.dumps(payload), encoding="utf-8") + return real_loader( + archive_path, + stem_keys, + sample_rate, + policy_template=policy_template, + ) + + monkeypatch.setattr( + "bandscope_analysis.api.load_bounded_stem_archive", + replace_sidecar_then_load, + ) + + assert _load_cached_local_audio_features(metadata_path, arrays_path) is None + + +def test_feature_cache_replay_rejects_second_read_schema_substitution( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """A second-read sidecar cannot substitute another cache schema generation.""" + metadata_path = tmp_path / "features.json" + arrays_path = tmp_path / "features.npz" + _write_cache(metadata_path, arrays_path) + + real_loader = load_bounded_stem_archive + + def replace_sidecar_then_load( + archive_path: Path, + stem_keys: list[str], + sample_rate: object, + *, + policy_template, + ): + payload = json.loads(metadata_path.read_text(encoding="utf-8")) + payload["schemaVersion"] = 2 + metadata_path.write_text(json.dumps(payload), encoding="utf-8") + return real_loader( + archive_path, + stem_keys, + sample_rate, + policy_template=policy_template, + ) + + monkeypatch.setattr( + "bandscope_analysis.api.load_bounded_stem_archive", + replace_sidecar_then_load, + ) + + assert _load_cached_local_audio_features(metadata_path, arrays_path) is None + + +def test_feature_cache_replay_rejects_missing_separation_duration(tmp_path: Path) -> None: + """A cached stem timeline without duration authority must be recomputed, not replayed.""" + metadata_path = tmp_path / "features.json" + arrays_path = tmp_path / "features.npz" + _write_cache(metadata_path, arrays_path) + + payload = json.loads(metadata_path.read_text(encoding="utf-8")) + del payload["separation"]["duration_seconds"] + metadata_path.write_text(json.dumps(payload), encoding="utf-8") + + assert _load_cached_local_audio_features(metadata_path, arrays_path) is None + + +def test_feature_cache_replay_rejects_first_read_missing_duration_after_replacement( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """A later valid sidecar cannot repair missing duration in the first metadata snapshot.""" + metadata_path = tmp_path / "features.json" + arrays_path = tmp_path / "features.npz" + _write_cache(metadata_path, arrays_path) + + valid_payload = json.loads(metadata_path.read_text(encoding="utf-8")) + first_payload = json.loads(json.dumps(valid_payload)) + del first_payload["separation"]["duration_seconds"] + metadata_path.write_text(json.dumps(first_payload), encoding="utf-8") + + real_loader = load_bounded_stem_archive + + def restore_valid_sidecar_then_load( + archive_path: Path, + stem_keys: list[str], + sample_rate: object, + *, + policy_template, + ): + metadata_path.write_text(json.dumps(valid_payload), encoding="utf-8") + return real_loader( + archive_path, + stem_keys, + sample_rate, + policy_template=policy_template, + ) + + monkeypatch.setattr( + "bandscope_analysis.api.load_bounded_stem_archive", + restore_valid_sidecar_then_load, + ) + + assert _load_cached_local_audio_features(metadata_path, arrays_path) is None + + +def test_feature_cache_replay_rejects_first_read_role_substitution_after_replacement( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """A later valid sidecar cannot repair contradictory first-read stem-role semantics.""" + metadata_path = tmp_path / "features.json" + arrays_path = tmp_path / "features.npz" + _write_cache(metadata_path, arrays_path) + + valid_payload = json.loads(metadata_path.read_text(encoding="utf-8")) + first_payload = json.loads(json.dumps(valid_payload)) + first_payload["stemRoleTypes"] = {"bass": "vocal"} + metadata_path.write_text(json.dumps(first_payload), encoding="utf-8") + + real_loader = load_bounded_stem_archive + + def restore_valid_sidecar_then_load( + archive_path: Path, + stem_keys: list[str], + sample_rate: object, + *, + policy_template, + ): + metadata_path.write_text(json.dumps(valid_payload), encoding="utf-8") + return real_loader( + archive_path, + stem_keys, + sample_rate, + policy_template=policy_template, + ) + + monkeypatch.setattr( + "bandscope_analysis.api.load_bounded_stem_archive", + restore_valid_sidecar_then_load, + ) + + assert _load_cached_local_audio_features(metadata_path, arrays_path) is None diff --git a/services/analysis-engine/tests/test_feature_cache_producer_admission.py b/services/analysis-engine/tests/test_feature_cache_producer_admission.py new file mode 100644 index 000000000..91ccfc130 --- /dev/null +++ b/services/analysis-engine/tests/test_feature_cache_producer_admission.py @@ -0,0 +1,77 @@ +"""Regression tests for feature-cache producer stem admission.""" + +import importlib + +import numpy as np + +api = importlib.import_module("bandscope_analysis.api") + + +_SAMPLE_RATE = 22_050 +_SAMPLE_COUNT = 4 + + +def _request() -> dict[str, object]: + """Return the minimal validated-shape local-audio request needed by cache storage.""" + return { + "sourceKind": "local_audio", + "projectId": "project-cache", + "sourceLabel": "track.wav", + "roleFocus": ["bass-guitar"], + "localSource": { + "sourcePath": "/app/project/source/track.wav", + "fileName": "track.wav", + "extension": "wav", + "fileSizeBytes": 4096, + }, + } + + +def _features(stem_name: str) -> dict[str, object]: + """Return one internally consistent stem feature payload for producer admission.""" + return { + "stems": {stem_name: np.zeros(_SAMPLE_COUNT, dtype=np.float32)}, + "sr": _SAMPLE_RATE, + "stem_role_types": {stem_name: "vocal" if stem_name == "vocals" else "instrument"}, + "separation": { + "duration_seconds": _SAMPLE_COUNT / _SAMPLE_RATE, + "chunk_count": 1, + "notes": "producer admission fixture", + }, + } + + +def test_feature_cache_producer_rejects_noncanonical_audio_stem_before_publication( + tmp_path, +) -> None: + """A producer cannot publish a cache the canonical replay boundary must reject.""" + metadata_path = tmp_path / "track.features.json" + arrays_path = tmp_path / "track.features.npz" + + stored = api._store_cached_local_audio_features( + metadata_path, + arrays_path, + _request(), + _features("guitar"), + ) + + assert stored is False + assert not metadata_path.exists() + assert not arrays_path.exists() + + +def test_feature_cache_producer_accepts_canonical_audio_stem_subset(tmp_path) -> None: + """A valid canonical audio stem subset remains publishable.""" + metadata_path = tmp_path / "track.features.json" + arrays_path = tmp_path / "track.features.npz" + + stored = api._store_cached_local_audio_features( + metadata_path, + arrays_path, + _request(), + _features("bass"), + ) + + assert stored is True + assert metadata_path.exists() + assert arrays_path.exists() diff --git a/services/analysis-engine/tests/test_feature_cache_resource_admission.py b/services/analysis-engine/tests/test_feature_cache_resource_admission.py new file mode 100644 index 000000000..e2d12cae0 --- /dev/null +++ b/services/analysis-engine/tests/test_feature_cache_resource_admission.py @@ -0,0 +1,316 @@ +"""Regressions for resource admission when replaying cached audio stems.""" + +from __future__ import annotations + +import json +import os +from io import BytesIO +from pathlib import Path +from unittest.mock import Mock + +import numpy as np +import pytest + +from bandscope_analysis.api import _load_cached_local_audio_features +from bandscope_analysis.audio_resource_policy import AudioResourcePolicy +from bandscope_analysis.feature_cache_admission import _copy_exact_archive_snapshot + + +def _write_metadata( + path: Path, + *, + sample_rate: int = 44_100, + stem_key: str = "bass", +) -> None: + """Write the smallest valid feature-cache metadata envelope for one stem.""" + path.write_text( + json.dumps( + { + "schemaVersion": 1, + "sampleRate": sample_rate, + "separation": { + "duration_seconds": 16 / sample_rate, + "chunk_count": 1, + "notes": "cached rehearsal stem", + }, + "stemKeys": [stem_key], + "stemRoleTypes": {stem_key: "vocal" if stem_key == "vocals" else "instrument"}, + } + ), + encoding="utf-8", + ) + + +def test_feature_cache_replay_canonicalizes_finite_float_stems(tmp_path: Path) -> None: + """Legacy floating caches become owned canonical float32 before MIR reuse.""" + metadata_path = tmp_path / "features.json" + arrays_path = tmp_path / "features.npz" + _write_metadata(metadata_path) + np.savez_compressed(arrays_path, stem_bass=np.linspace(-1.0, 1.0, 16, dtype=np.float64)) + + loaded = _load_cached_local_audio_features(metadata_path, arrays_path) + + assert loaded is not None + assert loaded["stems"]["bass"].dtype == np.dtype(np.float32) + assert loaded["stems"]["bass"].flags.owndata + + +def test_feature_cache_replay_rejects_multidimensional_stem(tmp_path: Path) -> None: + """A cached channel/batch axis cannot be flattened into rehearsal evidence.""" + metadata_path = tmp_path / "features.json" + arrays_path = tmp_path / "features.npz" + _write_metadata(metadata_path) + np.savez_compressed(arrays_path, stem_bass=np.zeros((2, 8), dtype=np.float32)) + + assert _load_cached_local_audio_features(metadata_path, arrays_path) is None + + +def test_feature_cache_replay_rejects_nonfinite_stem(tmp_path: Path) -> None: + """NaN or infinite cached samples fail closed before downstream MIR reuse.""" + metadata_path = tmp_path / "features.json" + arrays_path = tmp_path / "features.npz" + _write_metadata(metadata_path) + stem_bass = np.zeros(16, dtype=np.float32) + stem_bass[1] = np.nan + np.savez_compressed(arrays_path, stem_bass=stem_bass) + + assert _load_cached_local_audio_features(metadata_path, arrays_path) is None + + +def test_feature_cache_replay_rejects_unsupported_sample_rate(tmp_path: Path) -> None: + """Cache metadata cannot invent an analysis rate outside the audio policy bounds.""" + metadata_path = tmp_path / "features.json" + arrays_path = tmp_path / "features.npz" + _write_metadata(metadata_path, sample_rate=7_999) + np.savez_compressed(arrays_path, stem_bass=np.zeros(8, dtype=np.float32)) + + assert _load_cached_local_audio_features(metadata_path, arrays_path) is None + + +def test_feature_cache_replay_rejects_noncanonical_stem_identity(tmp_path: Path) -> None: + """Persisted metadata cannot invent a stem the canonical separator never emits.""" + metadata_path = tmp_path / "features.json" + arrays_path = tmp_path / "features.npz" + _write_metadata(metadata_path, stem_key="guitar") + np.savez_compressed(arrays_path, stem_guitar=np.zeros(8, dtype=np.float32)) + + assert _load_cached_local_audio_features(metadata_path, arrays_path) is None + + +def test_feature_cache_replay_rejects_oversized_member_before_decompression( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Declared cached PCM above policy is rejected before NumPy materializes the member.""" + metadata_path = tmp_path / "features.json" + arrays_path = tmp_path / "features.npz" + _write_metadata(metadata_path, sample_rate=8_000) + np.savez_compressed(arrays_path, stem_bass=np.zeros(16, dtype=np.float32)) + + tiny_policy = AudioResourcePolicy( + target_sample_rate=8_000, + max_duration_seconds=0.001, + max_decoded_audio_bytes=32, + ) + monkeypatch.setattr("bandscope_analysis.api.DEFAULT_AUDIO_RESOURCE_POLICY", tiny_policy) + load_mock = Mock(side_effect=AssertionError("oversized member must fail before np.load")) + monkeypatch.setattr("bandscope_analysis.api.np.load", load_mock) + + assert _load_cached_local_audio_features(metadata_path, arrays_path) is None + load_mock.assert_not_called() + + +def test_feature_cache_replay_treats_numpy_allocator_exhaustion_as_cache_miss( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Allocator exhaustion while opening an admitted NPZ cannot crash the analysis job.""" + metadata_path = tmp_path / "features.json" + arrays_path = tmp_path / "features.npz" + _write_metadata(metadata_path) + np.savez_compressed(arrays_path, stem_bass=np.zeros(16, dtype=np.float32)) + + load_mock = Mock(side_effect=MemoryError("allocator exhausted while opening cached stems")) + monkeypatch.setattr("bandscope_analysis.feature_cache_admission.np.load", load_mock) + + assert _load_cached_local_audio_features(metadata_path, arrays_path) is None + load_mock.assert_called_once() + + +def _write_two_stem_cache(metadata_path: Path, arrays_path: Path, *, drum_samples: int) -> None: + """Write canonical bass/drums cache data with a configurable drums timeline.""" + metadata_path.write_text( + json.dumps( + { + "schemaVersion": 1, + "sampleRate": 44_100, + "separation": { + "duration_seconds": 16 / 44_100, + "chunk_count": 1, + "notes": "cached rehearsal stems", + }, + "stemKeys": ["bass", "drums"], + "stemRoleTypes": {"bass": "instrument", "drums": "instrument"}, + } + ), + encoding="utf-8", + ) + np.savez_compressed( + arrays_path, + stem_bass=np.zeros(16, dtype=np.float32), + stem_drums=np.zeros(drum_samples, dtype=np.float32), + ) + + +def test_feature_cache_replay_accepts_aligned_stem_timelines(tmp_path: Path) -> None: + """Canonical persisted stems with one shared sample timeline remain reusable.""" + metadata_path = tmp_path / "features.json" + arrays_path = tmp_path / "features.npz" + _write_two_stem_cache(metadata_path, arrays_path, drum_samples=16) + + loaded = _load_cached_local_audio_features(metadata_path, arrays_path) + + assert loaded is not None + assert set(loaded["stems"]) == {"bass", "drums"} + assert loaded["stems"]["bass"].shape == loaded["stems"]["drums"].shape == (16,) + + +def test_feature_cache_replay_rejects_duration_that_disagrees_with_stem_timeline( + tmp_path: Path, +) -> None: + """Persisted duration cannot disagree with the synchronized stem sample timeline.""" + metadata_path = tmp_path / "features.json" + arrays_path = tmp_path / "features.npz" + _write_metadata(metadata_path) + metadata_payload = json.loads(metadata_path.read_text(encoding="utf-8")) + metadata_payload["separation"]["duration_seconds"] = 17 / 44_100 + metadata_path.write_text(json.dumps(metadata_payload), encoding="utf-8") + np.savez_compressed(arrays_path, stem_bass=np.zeros(16, dtype=np.float32)) + + assert _load_cached_local_audio_features(metadata_path, arrays_path) is None + + +def test_feature_cache_replay_rejects_nonfinite_metadata_duration(tmp_path: Path) -> None: + """Non-finite cached duration cannot become rehearsal timeline authority.""" + metadata_path = tmp_path / "features.json" + arrays_path = tmp_path / "features.npz" + _write_metadata(metadata_path) + metadata_payload = json.loads(metadata_path.read_text(encoding="utf-8")) + metadata_payload["separation"]["duration_seconds"] = float("inf") + metadata_path.write_text(json.dumps(metadata_payload), encoding="utf-8") + np.savez_compressed(arrays_path, stem_bass=np.zeros(16, dtype=np.float32)) + + assert _load_cached_local_audio_features(metadata_path, arrays_path) is None + + +def test_feature_cache_replay_rejects_integer_duration_too_large_for_float( + tmp_path: Path, +) -> None: + """Oversized integer duration fails as a cache miss instead of escaping admission.""" + metadata_path = tmp_path / "features.json" + arrays_path = tmp_path / "features.npz" + _write_metadata(metadata_path) + metadata_payload = json.loads(metadata_path.read_text(encoding="utf-8")) + metadata_payload["separation"]["duration_seconds"] = 10**400 + metadata_path.write_text(json.dumps(metadata_payload), encoding="utf-8") + np.savez_compressed(arrays_path, stem_bass=np.zeros(16, dtype=np.float32)) + + assert _load_cached_local_audio_features(metadata_path, arrays_path) is None + + +def test_feature_cache_replay_rejects_misaligned_stem_lengths_before_materialization( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Persisted stems must preserve one synchronized sample timeline before MIR reuse.""" + metadata_path = tmp_path / "features.json" + arrays_path = tmp_path / "features.npz" + _write_two_stem_cache(metadata_path, arrays_path, drum_samples=8) + + load_mock = Mock(side_effect=AssertionError("misaligned stems must fail before np.load")) + monkeypatch.setattr("bandscope_analysis.feature_cache_admission.np.load", load_mock) + + assert _load_cached_local_audio_features(metadata_path, arrays_path) is None + load_mock.assert_not_called() + + +def test_feature_cache_snapshot_copy_rejects_short_source() -> None: + """A cache truncated below its admitted extent cannot form a replay snapshot.""" + assert not _copy_exact_archive_snapshot(BytesIO(b"x"), BytesIO(), 2) + + +def test_feature_cache_replay_ignores_path_mutation_after_snapshot( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Mutation after snapshot creation cannot change the admitted replay bytes.""" + metadata_path = tmp_path / "features.json" + arrays_path = tmp_path / "features.npz" + _write_metadata(metadata_path) + np.savez_compressed(arrays_path, stem_bass=np.zeros(16, dtype=np.float32)) + + real_load = np.load + + def mutate_archive_then_load(archive_file, *args, **kwargs): + np.savez_compressed( + arrays_path, + stem_bass=np.arange(1024, dtype=np.float32), + ) + archive_file.seek(0) + return real_load(archive_file, *args, **kwargs) + + monkeypatch.setattr( + "bandscope_analysis.feature_cache_admission.np.load", + mutate_archive_then_load, + ) + + loaded = _load_cached_local_audio_features(metadata_path, arrays_path) + + assert loaded is not None + assert loaded["stems"]["bass"].shape == (16,) + assert np.count_nonzero(loaded["stems"]["bass"]) == 0 + + +def test_feature_cache_replay_materializes_one_admitted_archive_snapshot( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Replay uses one admitted byte snapshot even if file metadata is restored.""" + metadata_path = tmp_path / "features.json" + arrays_path = tmp_path / "features.npz" + replacement_path = tmp_path / "replacement.npz" + _write_metadata(metadata_path) + np.savez_compressed(arrays_path, stem_bass=np.full(16, -0.0, dtype=np.float32)) + np.savez_compressed(replacement_path, stem_bass=np.ones(16, dtype=np.float32)) + + admitted_stat = arrays_path.stat() + replacement_bytes = replacement_path.read_bytes() + assert len(replacement_bytes) == admitted_stat.st_size + real_load = np.load + + def replace_bytes_then_load(archive_file, *args, **kwargs): + arrays_path.write_bytes(replacement_bytes) + changed_stat = arrays_path.stat() + assert (changed_stat.st_dev, changed_stat.st_ino, changed_stat.st_size) == ( + admitted_stat.st_dev, + admitted_stat.st_ino, + admitted_stat.st_size, + ) + os.utime( + arrays_path, + ns=(changed_stat.st_atime_ns, admitted_stat.st_mtime_ns), + ) + restored_stat = arrays_path.stat() + assert restored_stat.st_mtime_ns == admitted_stat.st_mtime_ns + archive_file.seek(0) + return real_load(archive_file, *args, **kwargs) + + monkeypatch.setattr( + "bandscope_analysis.feature_cache_admission.np.load", + replace_bytes_then_load, + ) + + loaded = _load_cached_local_audio_features(metadata_path, arrays_path) + + assert loaded is not None + assert np.signbit(loaded["stems"]["bass"]).all() diff --git a/services/analysis-engine/tests/test_feature_cache_role_binding.py b/services/analysis-engine/tests/test_feature_cache_role_binding.py new file mode 100644 index 000000000..455282815 --- /dev/null +++ b/services/analysis-engine/tests/test_feature_cache_role_binding.py @@ -0,0 +1,182 @@ +"""Regression tests for persisted stem-role semantic admission.""" + +import importlib +import json + +import numpy as np + +api = importlib.import_module("bandscope_analysis.api") +feature_cache_admission = importlib.import_module("bandscope_analysis.feature_cache_admission") + + +_SAMPLE_RATE = 22_050 +_SAMPLE_COUNT = 4 +_DURATION_SECONDS = _SAMPLE_COUNT / _SAMPLE_RATE + + +def _sidecar( + *, + stem_keys: list[str], + stem_role_types: object = None, +) -> dict[str, object]: + """Build generation-valid sidecar metadata for focused role-admission tests.""" + payload: dict[str, object] = { + "schemaVersion": 1, + "sampleRate": _SAMPLE_RATE, + "separation": {"duration_seconds": _DURATION_SECONDS}, + "stemKeys": stem_keys, + } + if stem_role_types is not None: + payload["stemRoleTypes"] = stem_role_types + return payload + + +def test_feature_cache_rejects_role_type_that_contradicts_canonical_stem_semantics( + tmp_path, +) -> None: + """A persisted instrument stem cannot be replayed as vocal rehearsal evidence.""" + metadata_path = tmp_path / "track.features.json" + arrays_path = tmp_path / "track.features.npz" + np.savez_compressed( + arrays_path, + stem_bass=np.zeros(_SAMPLE_COUNT, dtype=np.float32), + ) + + metadata_path.write_text( + json.dumps(_sidecar(stem_keys=["bass"], stem_role_types={"bass": "vocal"})), + encoding="utf-8", + ) + + assert api._load_cached_local_audio_features(metadata_path, arrays_path) is None + + metadata_path.write_text( + json.dumps( + _sidecar(stem_keys=["bass"], stem_role_types={"bass": "instrument"}) + ), + encoding="utf-8", + ) + + replayed = api._load_cached_local_audio_features(metadata_path, arrays_path) + assert replayed is not None + assert replayed["stem_role_types"] == {"bass": "instrument"} + + +def test_stem_role_sidecar_admission_covers_legacy_and_malformed_metadata( + tmp_path, +) -> None: + """A required sidecar fails closed while legacy role-field absence stays compatible.""" + arrays_path = tmp_path / "track.features.npz" + metadata_path = arrays_path.with_suffix(".json") + + assert ( + feature_cache_admission._has_canonical_stem_role_metadata( + arrays_path, + ["bass"], + expected_sample_rate=_SAMPLE_RATE, + ) + is False + ) + + metadata_path.write_text("{", encoding="utf-8") + assert ( + feature_cache_admission._has_canonical_stem_role_metadata( + arrays_path, + ["bass"], + expected_sample_rate=_SAMPLE_RATE, + ) + is False + ) + + metadata_path.write_text("[]", encoding="utf-8") + assert ( + feature_cache_admission._has_canonical_stem_role_metadata( + arrays_path, + ["bass"], + expected_sample_rate=_SAMPLE_RATE, + ) + is False + ) + + metadata_path.write_text( + json.dumps(_sidecar(stem_keys=["bass"])), + encoding="utf-8", + ) + assert feature_cache_admission._has_canonical_stem_role_metadata( + arrays_path, + ["bass"], + expected_sample_rate=_SAMPLE_RATE, + ) + + metadata_path.write_text( + json.dumps(_sidecar(stem_keys=["bass"], stem_role_types=[])), + encoding="utf-8", + ) + assert ( + feature_cache_admission._has_canonical_stem_role_metadata( + arrays_path, + ["bass"], + expected_sample_rate=_SAMPLE_RATE, + ) + is False + ) + + +def test_stem_role_sidecar_rejects_unknown_stem_without_lookup_exception( + tmp_path, +) -> None: + """Unknown persisted stem identities fail closed before canonical-role lookup.""" + arrays_path = tmp_path / "track.features.npz" + metadata_path = arrays_path.with_suffix(".json") + metadata_path.write_text( + json.dumps( + _sidecar(stem_keys=["guitar"], stem_role_types={"guitar": "instrument"}) + ), + encoding="utf-8", + ) + + assert ( + feature_cache_admission._has_canonical_stem_role_metadata( + arrays_path, + ["guitar"], + expected_sample_rate=_SAMPLE_RATE, + ) + is False + ) + + +def test_stem_role_sidecar_rejects_replacement_with_different_stem_identity( + tmp_path, +) -> None: + """A second-read sidecar must describe the exact stem identity already admitted.""" + arrays_path = tmp_path / "track.features.npz" + metadata_path = arrays_path.with_suffix(".json") + + metadata_path.write_text( + json.dumps( + _sidecar(stem_keys=["bass"], stem_role_types={"bass": "instrument"}) + ), + encoding="utf-8", + ) + assert feature_cache_admission._has_canonical_stem_role_metadata( + arrays_path, + ["bass"], + expected_sample_rate=_SAMPLE_RATE, + ) + + metadata_path.write_text( + json.dumps( + _sidecar( + stem_keys=["drums"], + stem_role_types={"drums": "instrument"}, + ) + ), + encoding="utf-8", + ) + assert ( + feature_cache_admission._has_canonical_stem_role_metadata( + arrays_path, + ["bass"], + expected_sample_rate=_SAMPLE_RATE, + ) + is False + ) diff --git a/services/analysis-engine/tests/test_request_security_diagnostics.py b/services/analysis-engine/tests/test_request_security_diagnostics.py new file mode 100644 index 000000000..5e7dd4eb7 --- /dev/null +++ b/services/analysis-engine/tests/test_request_security_diagnostics.py @@ -0,0 +1,31 @@ +"""Regression tests for bounded Resource Admission security diagnostics.""" + +from unittest.mock import patch + +import pytest + +from bandscope_analysis.api import validate_analysis_job_request + + +def test_source_path_traversal_logs_only_bounded_field_name() -> None: + """Reject traversal while keeping attacker-controlled path bytes out of logs.""" + malicious_path = "/Users/test/../Music/late-night-set.wav\nFORGED-LOG-LINE" + payload = { + "sourceKind": "local_audio", + "projectId": "project-1", + "sourceLabel": "Late Night Set", + "roleFocus": [], + "localSource": { + "sourcePath": malicious_path, + "fileName": "late-night-set.wav", + "extension": "wav", + "fileSizeBytes": 1024, + }, + } + + with patch("bandscope_analysis.api.logger.warning") as warning: + with pytest.raises(ValueError, match="path traversal detected"): + validate_analysis_job_request(payload) + + warning.assert_called_once_with("Security: path traversal detected in localSource.sourcePath") + assert malicious_path not in repr(warning.call_args) diff --git a/services/analysis-engine/tests/test_separation.py b/services/analysis-engine/tests/test_separation.py index f8e098521..49ac97637 100644 --- a/services/analysis-engine/tests/test_separation.py +++ b/services/analysis-engine/tests/test_separation.py @@ -466,7 +466,11 @@ def test_audio_stem_separator_rejects_empty_decoder_output( audio_path = tmp_path / "empty.wav" audio_path.write_bytes(b"placeholder") monkeypatch.setattr( - "bandscope_analysis.separation.audio_separator.librosa.load", + "bandscope_analysis.audio_decode.preflight_audio_metadata", + lambda *_args, **_kwargs: None, + ) + monkeypatch.setattr( + "bandscope_analysis.audio_decode.librosa.load", lambda *args, **kwargs: (np.array([], dtype=np.float32), 8_000), ) separator = AudioStemSeparator(AudioSeparationConfig(target_sample_rate=8_000)) @@ -481,12 +485,16 @@ def test_audio_stem_separator_redacts_decoder_exceptions( """Ensure decoder failures are surfaced without full local paths.""" audio_path = tmp_path / "broken.wav" audio_path.write_bytes(b"placeholder") + monkeypatch.setattr( + "bandscope_analysis.audio_decode.preflight_audio_metadata", + lambda *_args, **_kwargs: None, + ) def fail_decode(*args, **kwargs): raise RuntimeError(f"decoder failed under {tmp_path}") monkeypatch.setattr( - "bandscope_analysis.separation.audio_separator.librosa.load", + "bandscope_analysis.audio_decode.librosa.load", fail_decode, ) separator = AudioStemSeparator(AudioSeparationConfig(target_sample_rate=8_000)) diff --git a/services/analysis-engine/tests/test_stem_separation_logging_privacy.py b/services/analysis-engine/tests/test_stem_separation_logging_privacy.py new file mode 100644 index 000000000..8d7d2d7b1 --- /dev/null +++ b/services/analysis-engine/tests/test_stem_separation_logging_privacy.py @@ -0,0 +1,124 @@ +"""Regression tests for stem-separation logging privacy.""" + +import logging + +import pytest + +import bandscope_analysis.api as analysis_api + + +class _ResultQueue: + """Capture the worker result without starting a multiprocessing queue.""" + + def __init__(self) -> None: + self.items: list[tuple[object, object]] = [] + + def put(self, item: tuple[object, object]) -> None: + """Record one result emitted by the worker.""" + self.items.append(item) + + +class _FailingSeparator: + """Raise dependency-controlled sensitive text from the separator boundary.""" + + def separate(self, source_path: str) -> dict[str, object]: + """Simulate a dependency failure after receiving an authorized source path.""" + raise RuntimeError( + f"decoder failed for {source_path} /Users/Alice/private-song.wav token=super-secret" + ) + + +def _local_audio_request() -> dict[str, object]: + """Return a valid local-audio request without cache or temporary-path authority.""" + return { + "sourceKind": "local_audio", + "projectId": "privacy-regression", + "sourceLabel": "private-song.wav", + "roleFocus": ["bass-guitar"], + "localSource": { + "sourcePath": "/private/customer/Alice/session.wav", + "fileName": "private-song.wav", + "extension": "wav", + "fileSizeBytes": 1024, + }, + } + + +def _assert_payload_free_log(caplog: pytest.LogCaptureFixture) -> None: + """Require routine logs to omit dependency payloads and exception tracebacks.""" + assert "/private/customer/Alice/session.wav" not in caplog.text + assert "/Users/Alice/private-song.wav" not in caplog.text + assert "private-song.wav token=super-secret" not in caplog.text + assert "super-secret" not in caplog.text + assert all(record.exc_info is None for record in caplog.records) + + +def test_stem_worker_failure_log_omits_dependency_payload_and_traceback( + monkeypatch: pytest.MonkeyPatch, + caplog: pytest.LogCaptureFixture, +) -> None: + """Routine worker diagnostics must not retain dependency payloads or tracebacks.""" + result_queue = _ResultQueue() + source_path = "/private/customer/Alice/session.wav" + + monkeypatch.setattr(analysis_api, "AudioStemSeparator", _FailingSeparator) + caplog.set_level(logging.ERROR, logger=analysis_api.__name__) + + analysis_api._stem_separation_worker(source_path, result_queue) + + assert result_queue.items == [ + ("runtime_error", "Runtime error occurred during stem separation.") + ] + assert "Stem separation failed with a runtime error." in caplog.text + _assert_payload_free_log(caplog) + + +def test_analysis_job_stem_failure_log_omits_dependency_payload_and_traceback( + monkeypatch: pytest.MonkeyPatch, + caplog: pytest.LogCaptureFixture, +) -> None: + """Parent orchestration failure logs must keep dependency details out of routine logs.""" + sensitive_detail = ( + "decode failed for /private/customer/Alice/session.wav " + "/Users/Alice/private-song.wav token=super-secret" + ) + + def fail_features(_request: analysis_api.AnalysisJobRequest) -> None: + raise ValueError(sensitive_detail) + + monkeypatch.setattr(analysis_api, "_build_local_audio_features", fail_features) + caplog.set_level(logging.ERROR, logger=analysis_api.__name__) + + updates = analysis_api.run_analysis_job_updates( + "job-privacy", + _local_audio_request(), + "2026-08-20T00:00:00Z", + ) + + assert updates[-1]["state"] == "failed" + assert updates[-1]["error"] == { + "code": "engine_unavailable", + "message": "Stem separation failed", + } + assert "Stem separation failed before analysis job completion." in caplog.text + _assert_payload_free_log(caplog) + + +def test_api_logger_preserves_unrelated_exception_traceback( + caplog: pytest.LogCaptureFixture, +) -> None: + """Privacy redaction must not erase traceback evidence from unrelated API diagnostics.""" + caplog.set_level(logging.ERROR, logger=analysis_api.__name__) + + try: + raise RuntimeError("non-sensitive diagnostic sentinel") + except RuntimeError: + analysis_api.logger.exception("Unrelated analysis API diagnostic.") + + records = [ + record + for record in caplog.records + if record.getMessage() == "Unrelated analysis API diagnostic." + ] + assert len(records) == 1 + assert records[0].exc_info is not None diff --git a/services/analysis-engine/tests/test_temporal.py b/services/analysis-engine/tests/test_temporal.py index 6ce90ae1c..16c7f7034 100644 --- a/services/analysis-engine/tests/test_temporal.py +++ b/services/analysis-engine/tests/test_temporal.py @@ -46,9 +46,9 @@ def test_temporal_analyzer_basic(dummy_audio_file: Path) -> None: def test_temporal_analyzer_file_not_found() -> None: - """Test that analyzer raises appropriate error for missing files.""" + """Test that analyzer raises a payload-safe error for missing files.""" analyzer = TemporalAnalyzer() - with pytest.raises(FileNotFoundError, match="Audio file not found"): + with pytest.raises(FileNotFoundError, match="Audio source is unavailable"): analyzer.analyze("nonexistent_file.wav") @@ -62,7 +62,7 @@ def test_temporal_analyzer_missing_file_does_not_call_decoder( monkeypatch.setattr(librosa, "load", load_mock) analyzer = TemporalAnalyzer() - with pytest.raises(FileNotFoundError, match="Audio file not found"): + with pytest.raises(FileNotFoundError, match="Audio source is unavailable"): analyzer.analyze("nonexistent_file.wav") load_mock.assert_not_called() @@ -77,7 +77,7 @@ def test_temporal_analyzer_directory_does_not_call_decoder( load_mock = Mock(side_effect=AssertionError("librosa.load should not be called")) monkeypatch.setattr(librosa, "load", load_mock) - with pytest.raises(FileNotFoundError, match="Audio file not found"): + with pytest.raises(FileNotFoundError, match="Audio source is unavailable"): TemporalAnalyzer().analyze(tmp_path) load_mock.assert_not_called() @@ -94,7 +94,7 @@ def fake_load(*args, **kwargs): monkeypatch.setattr(librosa, "load", fake_load) test_wav = tmp_path / "test.wav" - test_wav.write_bytes(b"dummy") + sf.write(test_wav, np.zeros(4_000, dtype=np.float32), 44_100) with pytest.raises(ValueError, match="Expected numpy array"): TemporalAnalyzer().analyze(test_wav) @@ -104,7 +104,7 @@ def test_temporal_analyzer_exception_handling( monkeypatch: pytest.MonkeyPatch, tmp_path: Path, ) -> None: - """Ensure temporal analyzer catches general exceptions and raises ValueError.""" + """Ensure arbitrary decoder exception payloads are not relayed to callers.""" import librosa from bandscope_analysis.temporal.analyzer import TemporalAnalyzer @@ -115,10 +115,11 @@ def fake_load(*args: object, **kwargs: object) -> tuple[np.ndarray, int]: monkeypatch.setattr(librosa, "load", fake_load) test_wav = tmp_path / "test.wav" - test_wav.write_bytes(b"dummy") + sf.write(test_wav, np.zeros(4_000, dtype=np.float32), 44_100) - with pytest.raises(ValueError, match="Temporal analysis failed: Mocked general error"): + with pytest.raises(ValueError, match=r"^Temporal analysis failed\.$") as exc_info: TemporalAnalyzer().analyze(test_wav) + assert "Mocked general error" not in str(exc_info.value) def test_temporal_analyzer_rejects_oversized_file(monkeypatch, tmp_path: Path) -> None: @@ -128,7 +129,7 @@ def test_temporal_analyzer_rejects_oversized_file(monkeypatch, tmp_path: Path) - from bandscope_analysis.temporal import analyzer as analyzer_module test_wav = tmp_path / "large.wav" - test_wav.write_bytes(b"1234") + sf.write(test_wav, np.zeros(4_000, dtype=np.float32), 44_100) monkeypatch.setattr(analyzer_module, "MAX_AUDIO_FILE_BYTES", 1) @@ -147,7 +148,7 @@ def test_temporal_analyzer_uses_duration_limit(monkeypatch, tmp_path: Path) -> N import librosa test_wav = tmp_path / "bounded.wav" - test_wav.write_bytes(b"1234") + sf.write(test_wav, np.zeros(4_000, dtype=np.float32), 44_100) captured_kwargs: dict[str, object] = {} def fake_load(path, **kwargs): @@ -178,7 +179,7 @@ def test_temporal_analyzer_does_not_suppress_unrelated_loader_warnings( import librosa test_wav = tmp_path / "test.wav" - test_wav.write_bytes(b"dummy") + sf.write(test_wav, np.zeros(4_000, dtype=np.float32), 44_100) def fake_load(*args: object, **kwargs: object) -> tuple[np.ndarray, int]: warnings.warn("unrelated downstream warning", FutureWarning, stacklevel=2) diff --git a/services/analysis-engine/tests/test_temporal_error_privacy.py b/services/analysis-engine/tests/test_temporal_error_privacy.py new file mode 100644 index 000000000..ea0c6519f --- /dev/null +++ b/services/analysis-engine/tests/test_temporal_error_privacy.py @@ -0,0 +1,57 @@ +"""Privacy regressions for temporal-analysis failure diagnostics.""" + +from __future__ import annotations + +import logging +from pathlib import Path + +import numpy as np +import pytest +import soundfile as sf + +from bandscope_analysis.temporal import TemporalAnalyzer + + +def test_missing_temporal_source_does_not_disclose_local_path(tmp_path: Path) -> None: + """Missing-file failures must not echo an absolute customer path to callers.""" + sensitive_path = tmp_path / "private-customer-session" / "unreleased-song.wav" + + with pytest.raises(FileNotFoundError) as exc_info: + TemporalAnalyzer().analyze(sensitive_path) + + message = str(exc_info.value) + assert message == "Audio source is unavailable for temporal analysis." + assert str(sensitive_path) not in message + assert "unreleased-song.wav" not in message + + +def test_decoder_failure_redacts_source_path_and_decoder_payload( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, + caplog: pytest.LogCaptureFixture, +) -> None: + """Decoder diagnostics must remain useful without logging customer path/payload data.""" + import librosa + + sensitive_path = tmp_path / "private-customer-session" / "unreleased-song.wav" + sensitive_path.parent.mkdir() + sf.write(sensitive_path, np.zeros(4_000, dtype=np.float32), 44_100) + decoder_payload = "decoder exposed /private/customer/token-shaped-audio-name.wav" + + def fail_decode(*args: object, **kwargs: object) -> tuple[object, int]: + raise RuntimeError(decoder_payload) + + monkeypatch.setattr(librosa, "load", fail_decode) + caplog.set_level(logging.INFO, logger="bandscope_analysis.temporal.analyzer") + + with pytest.raises(ValueError) as exc_info: + TemporalAnalyzer().analyze(sensitive_path) + + message = str(exc_info.value) + assert message == "Temporal analysis failed." + assert str(sensitive_path) not in message + assert decoder_payload not in message + assert str(sensitive_path) not in caplog.text + assert "unreleased-song.wav" not in caplog.text + assert decoder_payload not in caplog.text + assert "RuntimeError" in caplog.text diff --git a/services/analysis-engine/tests/test_transcription.py b/services/analysis-engine/tests/test_transcription.py index f9b55af93..80eb126ff 100644 --- a/services/analysis-engine/tests/test_transcription.py +++ b/services/analysis-engine/tests/test_transcription.py @@ -4,8 +4,11 @@ import io from dataclasses import dataclass +from types import SimpleNamespace +from unittest.mock import Mock import numpy as np +import pytest import soundfile as sf from bandscope_analysis.transcription import api as transcription_api @@ -62,6 +65,32 @@ def test_transcribe_bass_stem_rejects_oversized_input(monkeypatch) -> None: transcribe_bass_stem(b"abc") +@pytest.mark.parametrize( + "metadata", + [ + SimpleNamespace(frames=22050 * 121, samplerate=22050, channels=2), + SimpleNamespace(frames=22050, samplerate=7_999, channels=2), + SimpleNamespace(frames=22050, samplerate=22050, channels=3), + ], +) +def test_transcribe_bass_stem_rejects_source_metadata_before_decode( + monkeypatch: pytest.MonkeyPatch, + metadata: SimpleNamespace, +) -> None: + """Bass transcription must validate source duration, rate, and channels before librosa.""" + monkeypatch.setattr( + "bandscope_analysis.audio_metadata.soundfile.info", + lambda _fileobj: metadata, + ) + load_mock = Mock(side_effect=AssertionError("source metadata must be checked first")) + monkeypatch.setattr(transcription_api.librosa, "load", load_mock) + + with pytest.raises(ValueError, match="audio resource policy"): + transcribe_bass_stem(b"not-a-real-wav") + + load_mock.assert_not_called() + + def test_transcribe_bass_stem_wraps_pitch_tracking_parameter_errors(monkeypatch) -> None: """Return a stable ValueError when pYIN rejects decoded audio parameters.""" stem_data = _render_bass_sequence([ExpectedNote("E2", 0.0, 0.45)]) diff --git a/services/analysis-engine/tests/test_youtube.py b/services/analysis-engine/tests/test_youtube.py index 5531ac9d5..0ae449aa9 100644 --- a/services/analysis-engine/tests/test_youtube.py +++ b/services/analysis-engine/tests/test_youtube.py @@ -2,12 +2,22 @@ import importlib import sys +from pathlib import Path from unittest.mock import MagicMock, patch import pytest import yt_dlp # type: ignore -from bandscope_analysis.youtube import MAX_YOUTUBE_URL_LENGTH, download_youtube_audio, validate_url +from bandscope_analysis.audio_resource_policy import DEFAULT_MAX_ENCODED_FILE_BYTES +from bandscope_analysis.youtube import ( + MAX_YOUTUBE_URL_LENGTH, + YOUTUBE_SIZE_EXCEEDED_MESSAGE, + _owned_file_path, + _remove_download_artifacts, + _remove_owned_file, + download_youtube_audio, + validate_url, +) def test_validate_url() -> None: @@ -89,20 +99,23 @@ def test_download_youtube_audio_success( "id": "abc123DEF45", "title": "Test Video", "duration": 60, + "filesize": True, + "filesize_approx": float("nan"), } + out_dir = str(Path("/tmp").resolve()) mock_ydl.extract_info.return_value = mock_info - mock_ydl.prepare_filename.return_value = "/tmp/abc123DEF45.webm" + mock_ydl.prepare_filename.return_value = f"{out_dir}/abc123DEF45.webm" mock_exists.return_value = True mock_getsize.return_value = 10 * 1024 * 1024 input_url = "https://youtube.com/watch?v=abc123DEF45" - result = download_youtube_audio(input_url, "/tmp") + result = download_youtube_audio(input_url, out_dir) assert result["ok"] is True assert result["metadata"]["id"] == "abc123DEF45" assert result["metadata"]["title"] == "Test Video" assert result["metadata"]["duration"] == 60 - assert result["metadata"]["filepath"] == "/tmp/abc123DEF45.webm" + assert result["metadata"]["filepath"] == f"{out_dir}/abc123DEF45.webm" # Assert that YoutubeDL was initialized with the correct options mock_ydl_class.assert_called_once() @@ -114,6 +127,8 @@ def test_download_youtube_audio_success( assert called_opts["noplaylist"] is True assert called_opts["geo_bypass"] is False assert called_opts["postprocessors"] == [{"key": "FFmpegExtractAudio"}] + assert called_opts["max_filesize"] == DEFAULT_MAX_ENCODED_FILE_BYTES + assert called_opts["progress_hooks"] assert "%(id)s.%(ext)s" in called_opts["outtmpl"] # Verify extract_info was called twice correctly: once for metadata, once for download @@ -145,21 +160,22 @@ def test_download_youtube_audio_converted_extension( "title": "Test Video", "duration": 60, } + out_dir = str(Path("/tmp").resolve()) mock_ydl.extract_info.return_value = mock_info - mock_ydl.prepare_filename.return_value = "/tmp/abc123DEF45.webm" + mock_ydl.prepare_filename.return_value = f"{out_dir}/abc123DEF45.webm" # os.path.exists returns False for .webm, but True for the converted .opus. def exists_side_effect(path: str) -> bool: """Mock exists function to simulate converted extension file presence.""" - return path == "/tmp/abc123DEF45.opus" + return path == f"{out_dir}/abc123DEF45.opus" mock_exists.side_effect = exists_side_effect mock_getsize.return_value = 10 * 1024 * 1024 - result = download_youtube_audio("https://youtube.com/watch?v=abc123DEF45", "/tmp") + result = download_youtube_audio("https://youtube.com/watch?v=abc123DEF45", out_dir) assert result["ok"] is True - assert result["metadata"]["filepath"] == "/tmp/abc123DEF45.opus" + assert result["metadata"]["filepath"] == f"{out_dir}/abc123DEF45.opus" @patch("bandscope_analysis.youtube.os.path.exists") @@ -273,6 +289,50 @@ def test_download_youtube_audio_duration_exceeded(mock_ydl_class: MagicMock) -> assert result["error"]["code"] == "duration_exceeded" +@patch("bandscope_analysis.youtube.os.path.getsize") +@patch("bandscope_analysis.youtube.os.path.exists") +@patch("bandscope_analysis.youtube.yt_dlp.YoutubeDL") +def test_download_youtube_audio_accepts_size_between_legacy_and_canonical_ceiling( + mock_ydl_class: MagicMock, + mock_exists: MagicMock, + mock_getsize: MagicMock, +) -> None: + """A 60 MiB download that the old 50 MB check rejected is now accepted.""" + mock_ydl = MagicMock() + mock_ydl_class.return_value.__enter__.return_value = mock_ydl + out_dir = str(Path("/tmp").resolve()) + mock_ydl.extract_info.return_value = {"id": "abc123DEF45", "duration": 10 * 60} + mock_ydl.prepare_filename.return_value = f"{out_dir}/abc123DEF45.m4a" + mock_exists.return_value = True + mock_getsize.return_value = 60 * 1024 * 1024 + + result = download_youtube_audio("https://youtube.com/watch?v=abc123DEF45", out_dir) + + assert result["ok"] is True + assert result["metadata"]["filepath"] == f"{out_dir}/abc123DEF45.m4a" + + +@patch("bandscope_analysis.youtube.os.path.getsize") +@patch("bandscope_analysis.youtube.os.path.exists") +@patch("bandscope_analysis.youtube.yt_dlp.YoutubeDL") +def test_download_youtube_audio_accepts_exact_policy_ceiling( + mock_ydl_class: MagicMock, + mock_exists: MagicMock, + mock_getsize: MagicMock, +) -> None: + """An encoded YouTube file exactly at the 100 MiB ceiling is accepted.""" + mock_ydl = MagicMock() + mock_ydl_class.return_value.__enter__.return_value = mock_ydl + mock_ydl.extract_info.return_value = {"id": "abc123DEF45", "duration": 10 * 60} + mock_ydl.prepare_filename.return_value = "/tmp/abc123DEF45.m4a" + mock_exists.return_value = True + mock_getsize.return_value = DEFAULT_MAX_ENCODED_FILE_BYTES + + result = download_youtube_audio("https://youtube.com/watch?v=abc123DEF45", "/tmp") + + assert result["ok"] is True + + @patch("bandscope_analysis.youtube.os.path.getsize") @patch("bandscope_analysis.youtube.os.path.exists") @patch("bandscope_analysis.youtube.os.remove") @@ -283,18 +343,315 @@ def test_download_youtube_audio_size_exceeded( mock_exists: MagicMock, mock_getsize: MagicMock, ) -> None: - """Test download fails if size exceeds 50MB.""" + """Post-download files one byte over the canonical 100 MiB ceiling are deleted.""" + mock_ydl = MagicMock() + mock_ydl_class.return_value.__enter__.return_value = mock_ydl + out_dir = str(Path("/tmp").resolve()) + mock_ydl.extract_info.return_value = {"id": "abc123DEF45", "duration": 10 * 60} + mock_ydl.prepare_filename.return_value = f"{out_dir}/abc123DEF45.m4a" + mock_exists.return_value = True + mock_getsize.return_value = DEFAULT_MAX_ENCODED_FILE_BYTES + 1 + + result = download_youtube_audio("https://youtube.com/watch?v=abc123DEF45", out_dir) + assert result["ok"] is False + assert result["error"]["code"] == "size_exceeded" + assert result["error"]["message"] == YOUTUBE_SIZE_EXCEEDED_MESSAGE + mock_remove.assert_called_with(f"{out_dir}/abc123DEF45.m4a") + + +@patch("bandscope_analysis.youtube.os.path.getsize") +@patch("bandscope_analysis.youtube.os.path.exists") +@patch("bandscope_analysis.youtube.os.remove") +@patch("bandscope_analysis.youtube.yt_dlp.YoutubeDL") +def test_download_youtube_audio_oversize_skips_remove_when_file_already_gone( + mock_ydl_class: MagicMock, + mock_remove: MagicMock, + mock_exists: MagicMock, + mock_getsize: MagicMock, +) -> None: + """A vanished oversize artifact still fails closed without a remove race.""" mock_ydl = MagicMock() mock_ydl_class.return_value.__enter__.return_value = mock_ydl mock_ydl.extract_info.return_value = {"id": "abc123DEF45", "duration": 10 * 60} mock_ydl.prepare_filename.return_value = "/tmp/abc123DEF45.m4a" + mock_exists.side_effect = [True, False] + mock_getsize.return_value = DEFAULT_MAX_ENCODED_FILE_BYTES + 1 + + result = download_youtube_audio("https://youtube.com/watch?v=abc123DEF45", "/tmp") + + assert result["ok"] is False + assert result["error"]["code"] == "size_exceeded" + mock_remove.assert_not_called() + + +@patch("bandscope_analysis.youtube.yt_dlp.YoutubeDL") +def test_download_youtube_audio_rejects_announced_filesize_before_download( + mock_ydl_class: MagicMock, +) -> None: + """Announced filesize over the policy ceiling must not start the download.""" + mock_ydl = MagicMock() + mock_ydl_class.return_value.__enter__.return_value = mock_ydl + mock_ydl.extract_info.return_value = { + "id": "abc123DEF45", + "duration": 60, + "filesize": DEFAULT_MAX_ENCODED_FILE_BYTES + 1, + } + + result = download_youtube_audio("https://youtube.com/watch?v=abc123DEF45", "/tmp") + + assert result["ok"] is False + assert result["error"]["code"] == "size_exceeded" + assert result["error"]["message"] == YOUTUBE_SIZE_EXCEEDED_MESSAGE + mock_ydl.extract_info.assert_called_once_with( + "https://youtube.com/watch?v=abc123DEF45", + download=False, + ) + + +@pytest.mark.parametrize( + "info", + [ + { + "id": "abc123DEF45", + "duration": 60, + "filesize_approx": DEFAULT_MAX_ENCODED_FILE_BYTES + 1, + }, + { + "id": "abc123DEF45", + "duration": 60, + "filesize_approx": float(DEFAULT_MAX_ENCODED_FILE_BYTES) + 0.5, + }, + ], +) +@patch("bandscope_analysis.youtube.yt_dlp.YoutubeDL") +def test_download_youtube_audio_rejects_announced_approximate_oversize( + mock_ydl_class: MagicMock, + info: dict[str, object], +) -> None: + """Approximate oversize metadata rejects the import before download starts.""" + mock_ydl = MagicMock() + mock_ydl_class.return_value.__enter__.return_value = mock_ydl + mock_ydl.extract_info.return_value = info + + result = download_youtube_audio("https://youtube.com/watch?v=abc123DEF45", "/tmp") + + assert result["ok"] is False + assert result["error"]["code"] == "size_exceeded" + mock_ydl.extract_info.assert_called_once() + + +@patch("bandscope_analysis.youtube.yt_dlp.YoutubeDL") +def test_download_youtube_audio_progress_hook_aborts_over_budget( + mock_ydl_class: MagicMock, +) -> None: + """In-flight progress that crosses the encoded-byte ceiling fails closed.""" + mock_ydl = MagicMock() + mock_ydl_class.return_value.__enter__.return_value = mock_ydl + + def extract_info(_url: str, download: bool = False) -> dict[str, object]: + """Invoke the registered progress hook when the download starts.""" + if download: + hook = mock_ydl_class.call_args[0][0]["progress_hooks"][0] + hook( + { + "status": "downloading", + "downloaded_bytes": DEFAULT_MAX_ENCODED_FILE_BYTES + 1, + } + ) + return {"id": "abc123DEF45", "duration": 60} + + mock_ydl.extract_info.side_effect = extract_info + + result = download_youtube_audio("https://youtube.com/watch?v=abc123DEF45", "/tmp") + + assert result["ok"] is False + assert result["error"]["code"] == "size_exceeded" + assert result["error"]["message"] == YOUTUBE_SIZE_EXCEEDED_MESSAGE + + +@patch("bandscope_analysis.youtube.yt_dlp.YoutubeDL") +def test_download_youtube_audio_progress_hook_deletes_partial_artifacts( + mock_ydl_class: MagicMock, + tmp_path: Path, +) -> None: + """In-flight abort must delete written partials so they cannot fill the cache.""" + out_dir = tmp_path / "import-cache" + out_dir.mkdir() + outsider = tmp_path / "unrelated-youtube-partial.part" + partial = out_dir / "abc123DEF45.m4a.part" + fragment = out_dir / "abc123DEF45.m4a-Frag1" + control = out_dir / "abc123DEF45.m4a.ytdl" + keep = out_dir / "keep-me.txt" + partial.write_bytes(b"partial-cache-bytes") + fragment.write_bytes(b"hls-fragment-bytes") + control.write_bytes(b"ytdl-control-bytes") + keep.write_bytes(b"unrelated-cache-note") + outsider.write_bytes(b"must-not-delete") + mock_ydl = MagicMock() + mock_ydl_class.return_value.__enter__.return_value = mock_ydl + + def extract_info(_url: str, download: bool = False) -> dict[str, object]: + """Abort after yt-dlp has already written the current block to disk.""" + if download: + hook = mock_ydl_class.call_args[0][0]["progress_hooks"][0] + hook( + { + "status": "downloading", + "downloaded_bytes": DEFAULT_MAX_ENCODED_FILE_BYTES + 1, + "tmpfilename": str(partial), + "filename": str(out_dir / "abc123DEF45.m4a"), + } + ) + return {"id": "abc123DEF45", "duration": 60} + + mock_ydl.extract_info.side_effect = extract_info + + result = download_youtube_audio( + "https://youtube.com/watch?v=abc123DEF45", + str(out_dir), + ) + + assert result["ok"] is False + assert result["error"]["code"] == "size_exceeded" + assert result["error"]["message"] == YOUTUBE_SIZE_EXCEEDED_MESSAGE + assert not partial.exists() + assert not fragment.exists() + assert not control.exists() + assert keep.exists() + assert outsider.exists() + + +def test_owned_file_path_rejects_empty_foreign_and_unresolvable_paths( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Abort cleanup must not follow empty, escaped, or unresolvable paths.""" + out_dir = tmp_path / "import-cache" + out_dir.mkdir() + escaped = tmp_path / "outside.part" + escaped.write_bytes(b"keep") + + assert _owned_file_path(None, str(out_dir)) is None + assert _owned_file_path("", str(out_dir)) is None + assert _owned_file_path(str(out_dir), str(out_dir)) is None + assert _owned_file_path(str(escaped), str(out_dir)) is None + + def boom(_path: str) -> str: + """Simulate a filesystem error while resolving a candidate path.""" + raise OSError("realpath failed") + + monkeypatch.setattr("bandscope_analysis.youtube.os.path.realpath", boom) + assert _owned_file_path(str(out_dir / "clip.part"), str(out_dir)) is None + + +def test_remove_owned_file_ignores_missing_directories_and_remove_races( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Owned cleanup skips non-files and treats remove races as already gone.""" + out_dir = tmp_path / "import-cache" + nested = out_dir / "nested-dir" + nested.mkdir(parents=True) + _remove_owned_file(None, str(out_dir)) + _remove_owned_file(str(nested), str(out_dir)) + assert nested.is_dir() + + target = out_dir / "clip.part" + target.write_bytes(b"partial") + + def boom(_path: str) -> None: + """Simulate a disappearing file during abort cleanup.""" + raise OSError("remove failed") + + monkeypatch.setattr("bandscope_analysis.youtube.os.remove", boom) + _remove_owned_file(str(target), str(out_dir)) + assert target.exists() + + +def test_remove_download_artifacts_skips_empty_status_and_unlistable_cache( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Artifact sweep no-ops when yt-dlp omitted paths or the cache vanished.""" + out_dir = tmp_path / "import-cache" + out_dir.mkdir() + leftover = out_dir / "other-file.txt" + leftover.write_bytes(b"keep") + _remove_download_artifacts({"tmpfilename": None, "filename": 12}, str(out_dir)) + assert leftover.exists() + + partial = out_dir / "abc123DEF45.m4a.part" + partial.write_bytes(b"partial") + + def boom(_path: str) -> list[str]: + """Simulate the import cache disappearing after the first delete.""" + raise OSError("listdir failed") + + monkeypatch.setattr("bandscope_analysis.youtube.os.listdir", boom) + _remove_download_artifacts({"tmpfilename": str(partial)}, str(out_dir)) + assert leftover.exists() + + +@patch("bandscope_analysis.youtube.os.path.getsize") +@patch("bandscope_analysis.youtube.os.path.exists") +@patch("bandscope_analysis.youtube.yt_dlp.YoutubeDL") +def test_download_youtube_audio_progress_hook_ignores_non_budget_updates( + mock_ydl_class: MagicMock, + mock_exists: MagicMock, + mock_getsize: MagicMock, +) -> None: + """Unknown statuses and non-integer byte fields do not abort a valid download.""" + mock_ydl = MagicMock() + mock_ydl_class.return_value.__enter__.return_value = mock_ydl + mock_ydl.extract_info.return_value = {"id": "abc123DEF45", "duration": 60} + mock_ydl.prepare_filename.return_value = "/tmp/abc123DEF45.m4a" mock_exists.return_value = True - mock_getsize.return_value = 51 * 1024 * 1024 + mock_getsize.return_value = 10 * 1024 * 1024 result = download_youtube_audio("https://youtube.com/watch?v=abc123DEF45", "/tmp") + hook = mock_ydl_class.call_args[0][0]["progress_hooks"][0] + hook({"status": "error"}) + hook({"status": "downloading", "downloaded_bytes": True}) + hook({"status": "downloading", "downloaded_bytes": 12.5}) + hook({"status": "downloading", "downloaded_bytes": 10}) + hook({"status": "finished", "total_bytes": DEFAULT_MAX_ENCODED_FILE_BYTES}) + + assert result["ok"] is True + + +@patch("bandscope_analysis.youtube.yt_dlp.YoutubeDL") +def test_download_youtube_audio_maps_max_filesize_download_error( + mock_ydl_class: MagicMock, +) -> None: + """yt-dlp max-filesize aborts become the payload-safe size-exceeded result.""" + mock_ydl = MagicMock() + mock_ydl_class.return_value.__enter__.return_value = mock_ydl + mock_ydl.extract_info.side_effect = yt_dlp.utils.DownloadError( + "File is larger than max-filesize" + ) + + result = download_youtube_audio("https://youtube.com/watch?v=abc123DEF45", "/tmp") + assert result["ok"] is False assert result["error"]["code"] == "size_exceeded" - mock_remove.assert_called_with("/tmp/abc123DEF45.m4a") + assert result["error"]["message"] == YOUTUBE_SIZE_EXCEEDED_MESSAGE + assert "max-filesize" not in result["error"]["message"] + + +@patch("bandscope_analysis.youtube.yt_dlp.YoutubeDL") +def test_download_youtube_audio_maps_mib_limit_download_error( + mock_ydl_class: MagicMock, +) -> None: + """Download errors that mention the 100 MiB ceiling stay payload-safe.""" + mock_ydl = MagicMock() + mock_ydl_class.return_value.__enter__.return_value = mock_ydl + mock_ydl.extract_info.side_effect = yt_dlp.utils.DownloadError(YOUTUBE_SIZE_EXCEEDED_MESSAGE) + + result = download_youtube_audio("https://youtube.com/watch?v=abc123DEF45", "/tmp") + + assert result["ok"] is False + assert result["error"]["code"] == "size_exceeded" + assert result["error"]["message"] == YOUTUBE_SIZE_EXCEEDED_MESSAGE def test_main_block(monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str]) -> None: @@ -326,38 +683,35 @@ def test_main_block(monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixtu def test_module_execution( - monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str] + monkeypatch: pytest.MonkeyPatch, + capsys: pytest.CaptureFixture[str], + tmp_path: Path, ) -> None: - """Test the if __name__ == '__main__' block using runpy.""" + """Test module execution against a real owned output path without network I/O.""" import runpy import bandscope_analysis.youtube + downloaded_path = tmp_path / "abc123DEF45.m4a" + downloaded_path.write_bytes(b"test-audio") test_args = [ "youtube.py", "--url", "https://youtube.com/watch?v=abc123DEF45", "--out-dir", - "/tmp", + str(tmp_path), ] monkeypatch.setattr(sys, "argv", test_args) - # Mock yt_dlp so runpy doesn't actually download + # Mock only the downloader/network boundary. Real filesystem semantics are + # required so the completed-path ownership check remains exercised. mock_yt_dlp = MagicMock() mock_ydl = MagicMock() mock_yt_dlp.YoutubeDL.return_value.__enter__.return_value = mock_ydl mock_ydl.extract_info.return_value = {"id": "abc123DEF45"} - mock_ydl.prepare_filename.return_value = "/tmp/abc123DEF45.m4a" + mock_ydl.prepare_filename.return_value = str(downloaded_path) monkeypatch.setitem(sys.modules, "yt_dlp", mock_yt_dlp) - # Mock os to ensure runpy uses our mocked filesystem methods - mock_os = MagicMock() - # Keep some essential attributes - mock_os.path = MagicMock() - mock_os.path.exists.return_value = True - mock_os.path.getsize.return_value = 10 * 1024 * 1024 - monkeypatch.setitem(sys.modules, "os", mock_os) - with patch.object(sys, "exit") as mock_exit: runpy.run_path(bandscope_analysis.youtube.__file__, run_name="__main__") mock_exit.assert_called_with(0) diff --git a/services/analysis-engine/tests/test_youtube_downloaded_duration_revalidation.py b/services/analysis-engine/tests/test_youtube_downloaded_duration_revalidation.py new file mode 100644 index 000000000..6780203f6 --- /dev/null +++ b/services/analysis-engine/tests/test_youtube_downloaded_duration_revalidation.py @@ -0,0 +1,40 @@ +"""Post-download YouTube duration revalidation regressions.""" + +from pathlib import Path +from unittest.mock import MagicMock, patch + +from bandscope_analysis.youtube import download_youtube_audio + + +@patch("bandscope_analysis.youtube.os.path.exists") +@patch("bandscope_analysis.youtube.os.path.isfile") +@patch("bandscope_analysis.youtube.os.remove") +@patch("bandscope_analysis.youtube.yt_dlp.YoutubeDL") +def test_youtube_revalidates_downloaded_duration_before_returning_success( + mock_ydl_class: MagicMock, + mock_remove: MagicMock, + mock_isfile: MagicMock, + mock_exists: MagicMock, +) -> None: + """Changed download metadata must not bypass the 15-minute admission limit.""" + mock_ydl = MagicMock() + mock_ydl_class.return_value.__enter__.return_value = mock_ydl + out_dir = str(Path("/tmp").resolve()) + mock_ydl.extract_info.side_effect = [ + {"id": "abc123DEF45", "duration": 60}, + {"id": "abc123DEF45", "title": "Changed metadata", "duration": 16 * 60}, + ] + mock_ydl.prepare_filename.return_value = f"{out_dir}/abc123DEF45.m4a" + mock_exists.return_value = True + mock_isfile.return_value = True + + result = download_youtube_audio("https://youtube.com/watch?v=abc123DEF45", out_dir) + + assert result == { + "ok": False, + "error": { + "code": "duration_exceeded", + "message": "Video exceeds the 15-minute limit.", + }, + } + mock_remove.assert_called_once_with(f"{out_dir}/abc123DEF45.m4a") diff --git a/services/analysis-engine/tests/test_youtube_duration_contract.py b/services/analysis-engine/tests/test_youtube_duration_contract.py new file mode 100644 index 000000000..0cb168787 --- /dev/null +++ b/services/analysis-engine/tests/test_youtube_duration_contract.py @@ -0,0 +1,54 @@ +"""Fail-closed YouTube duration metadata admission contract.""" + +from __future__ import annotations + +from unittest.mock import MagicMock, patch + +import pytest + +from bandscope_analysis.youtube import download_youtube_audio + + +class _NonCanonicalFloat(float): + """Numeric subtype that must not cross the untrusted metadata boundary.""" + + +@pytest.mark.parametrize( + "duration", + [ + True, + 0, + -1, + float("nan"), + float("inf"), + "60", + object(), + _NonCanonicalFloat(60.0), + ], +) +@patch("bandscope_analysis.youtube.yt_dlp.YoutubeDL") +def test_youtube_rejects_malformed_announced_duration_before_download( + mock_ydl_class: MagicMock, + duration: object, +) -> None: + """Malformed known-duration metadata must not authorize a media download.""" + mock_ydl = MagicMock() + mock_ydl_class.return_value.__enter__.return_value = mock_ydl + mock_ydl.extract_info.return_value = { + "id": "abc123DEF45", + "duration": duration, + } + + result = download_youtube_audio("https://youtube.com/watch?v=abc123DEF45", "/tmp") + + assert result == { + "ok": False, + "error": { + "code": "download_error", + "message": "YouTube import failed. Please use a local audio file instead.", + }, + } + mock_ydl.extract_info.assert_called_once_with( + "https://youtube.com/watch?v=abc123DEF45", + download=False, + ) diff --git a/services/analysis-engine/tests/test_youtube_fragment_identity_cleanup.py b/services/analysis-engine/tests/test_youtube_fragment_identity_cleanup.py new file mode 100644 index 000000000..35ab64462 --- /dev/null +++ b/services/analysis-engine/tests/test_youtube_fragment_identity_cleanup.py @@ -0,0 +1,103 @@ +"""Regression coverage for yt-dlp fragment cleanup with valid ID-like tokens.""" + +from pathlib import Path + +from bandscope_analysis.youtube import _cleanup_stem, _remove_download_artifacts + +VIDEO_ID_WITH_FRAGMENT_TOKEN = "abc-Frag123" + + +def test_cleanup_stem_preserves_fragment_token_inside_valid_video_id() -> None: + """Only a terminal yt-dlp fragment suffix may be removed from the filename.""" + assert len(VIDEO_ID_WITH_FRAGMENT_TOKEN) == 11 + assert ( + _cleanup_stem(f"{VIDEO_ID_WITH_FRAGMENT_TOKEN}.m4a.part") + == f"{VIDEO_ID_WITH_FRAGMENT_TOKEN}.m4a" + ) + assert ( + _cleanup_stem(f"{VIDEO_ID_WITH_FRAGMENT_TOKEN}.m4a.part-Frag2") + == f"{VIDEO_ID_WITH_FRAGMENT_TOKEN}.m4a" + ) + assert ( + _cleanup_stem(f"{VIDEO_ID_WITH_FRAGMENT_TOKEN}.m4a.part-Frag2.part") + == f"{VIDEO_ID_WITH_FRAGMENT_TOKEN}.m4a" + ) + + +def test_cleanup_stem_requires_ascii_fragment_number() -> None: + """Unicode digits must not widen yt-dlp's ASCII ``-FragN`` cleanup syntax.""" + unicode_digit_name = f"{VIDEO_ID_WITH_FRAGMENT_TOKEN}.m4a.part-Frag２" + unicode_digit_part_name = f"{unicode_digit_name}.part" + + assert _cleanup_stem(unicode_digit_name) == unicode_digit_name + assert _cleanup_stem(unicode_digit_part_name) == unicode_digit_part_name + + +def test_abort_cleanup_removes_fragment_when_video_id_contains_fragment_token( + tmp_path: Path, +) -> None: + """A valid video ID containing '-Frag' must not strand its real fragment files.""" + out_dir = tmp_path / "import-cache" + out_dir.mkdir() + partial = out_dir / f"{VIDEO_ID_WITH_FRAGMENT_TOKEN}.m4a.part" + fragment = out_dir / f"{VIDEO_ID_WITH_FRAGMENT_TOKEN}.m4a.part-Frag2" + fragment_part = out_dir / f"{VIDEO_ID_WITH_FRAGMENT_TOKEN}.m4a.part-Frag2.part" + keep = out_dir / f"{VIDEO_ID_WITH_FRAGMENT_TOKEN}.keep.m4a" + partial.write_bytes(b"partial") + fragment.write_bytes(b"fragment") + fragment_part.write_bytes(b"fragment-part") + keep.write_bytes(b"preserve") + + _remove_download_artifacts( + { + "tmpfilename": str(partial), + "filename": str(out_dir / f"{VIDEO_ID_WITH_FRAGMENT_TOKEN}.m4a"), + }, + str(out_dir), + VIDEO_ID_WITH_FRAGMENT_TOKEN, + ) + + assert not partial.exists() + assert not fragment.exists() + assert not fragment_part.exists() + assert keep.exists() + + +def test_abort_cleanup_preserves_unicode_digit_fragment_decoy(tmp_path: Path) -> None: + """Cleanup must not delete same-ID files outside yt-dlp's ASCII fragment grammar.""" + out_dir = tmp_path / "import-cache" + out_dir.mkdir() + partial = out_dir / f"{VIDEO_ID_WITH_FRAGMENT_TOKEN}.m4a.part" + unicode_digit_decoy = out_dir / f"{VIDEO_ID_WITH_FRAGMENT_TOKEN}.m4a.part-Frag２" + partial.write_bytes(b"partial") + unicode_digit_decoy.write_bytes(b"preserve") + + _remove_download_artifacts( + {"tmpfilename": str(partial)}, + str(out_dir), + VIDEO_ID_WITH_FRAGMENT_TOKEN, + ) + + assert not partial.exists() + assert unicode_digit_decoy.exists() + + +def test_abort_cleanup_rejects_reported_unicode_digit_fragment_part( + tmp_path: Path, +) -> None: + """A reported Unicode lookalike must not gain direct or sibling deletion authority.""" + out_dir = tmp_path / "import-cache" + out_dir.mkdir() + unicode_digit_part = out_dir / f"{VIDEO_ID_WITH_FRAGMENT_TOKEN}.m4a.part-Frag２.part" + same_stem_ytdl = out_dir / f"{VIDEO_ID_WITH_FRAGMENT_TOKEN}.m4a.part-Frag２.ytdl" + unicode_digit_part.write_bytes(b"preserve-reported") + same_stem_ytdl.write_bytes(b"preserve-sibling") + + _remove_download_artifacts( + {"tmpfilename": str(unicode_digit_part)}, + str(out_dir), + VIDEO_ID_WITH_FRAGMENT_TOKEN, + ) + + assert unicode_digit_part.exists() + assert same_stem_ytdl.exists() diff --git a/services/analysis-engine/tests/test_youtube_post_download_admission_reason.py b/services/analysis-engine/tests/test_youtube_post_download_admission_reason.py new file mode 100644 index 000000000..314f242d0 --- /dev/null +++ b/services/analysis-engine/tests/test_youtube_post_download_admission_reason.py @@ -0,0 +1,47 @@ +"""Regression coverage for post-download YouTube resource-admission reasons.""" + +from __future__ import annotations + +from pathlib import Path +from unittest.mock import MagicMock, patch + +from bandscope_analysis.youtube import download_youtube_audio + + +@patch("bandscope_analysis.youtube.yt_dlp.YoutubeDL") +def test_zero_byte_download_is_malformed_not_oversize( + mock_ydl_class: MagicMock, + tmp_path: Path, +) -> None: + """Delete an empty artifact without telling the buyer it exceeded 100 MiB.""" + out_dir = tmp_path / "youtube-import" + out_dir.mkdir() + downloaded = out_dir / "abc123DEF45.m4a" + + mock_ydl = MagicMock() + mock_ydl_class.return_value.__enter__.return_value = mock_ydl + + def extract_info(_url: str, download: bool = False) -> dict[str, object]: + """Materialize the malformed artifact only after the import owns its lease.""" + if download: + downloaded.write_bytes(b"") + return { + "id": "abc123DEF45", + "title": "Empty postprocessor output", + "duration": 60, + } + + mock_ydl.extract_info.side_effect = extract_info + mock_ydl.prepare_filename.return_value = str(downloaded) + + result = download_youtube_audio( + "https://youtube.com/watch?v=abc123DEF45", + str(out_dir), + ) + + assert result["ok"] is False + assert result["error"] == { + "code": "download_error", + "message": "YouTube import failed. Please use a local audio file instead.", + } + assert not downloaded.exists() diff --git a/services/analysis-engine/tests/test_youtube_post_download_path_authority.py b/services/analysis-engine/tests/test_youtube_post_download_path_authority.py new file mode 100644 index 000000000..5bb103234 --- /dev/null +++ b/services/analysis-engine/tests/test_youtube_post_download_path_authority.py @@ -0,0 +1,153 @@ +"""Regression coverage for post-download YouTube path authority. + +The downloader owns only artifacts that resolve beneath the per-import output +directory and belong to its active same-video lease. Metadata returned by yt-dlp +must not turn an arbitrary or pre-existing filesystem path into a successful +import or deletion target. +""" + +from pathlib import Path +from unittest.mock import MagicMock, patch + +from bandscope_analysis.audio_resource_policy import DEFAULT_MAX_ENCODED_FILE_BYTES +from bandscope_analysis.youtube import ( + YOUTUBE_IMPORT_FAILED_MESSAGE, + _import_lease_path, + download_youtube_audio, +) + + +def _configure_download(mock_ydl_class: MagicMock, filepath: Path) -> None: + """Configure yt-dlp to report one completed download at ``filepath``.""" + mock_ydl = MagicMock() + mock_ydl_class.return_value.__enter__.return_value = mock_ydl + mock_ydl.extract_info.return_value = { + "id": "abc123DEF45", + "title": "Authority regression", + "duration": 60, + } + mock_ydl.prepare_filename.return_value = str(filepath) + + +@patch("bandscope_analysis.youtube.yt_dlp.YoutubeDL") +def test_download_rejects_foreign_completed_path( + mock_ydl_class: MagicMock, + tmp_path: Path, +) -> None: + """A completed path outside this import directory must never become success metadata.""" + out_dir = tmp_path / "import-cache" + out_dir.mkdir() + foreign = tmp_path / "foreign.m4a" + foreign.write_bytes(b"not-owned-by-this-import") + _configure_download(mock_ydl_class, foreign) + + result = download_youtube_audio( + "https://youtube.com/watch?v=abc123DEF45", + str(out_dir), + ) + + assert result == { + "ok": False, + "error": {"code": "download_error", "message": YOUTUBE_IMPORT_FAILED_MESSAGE}, + } + assert foreign.read_bytes() == b"not-owned-by-this-import" + + +@patch("bandscope_analysis.youtube.yt_dlp.YoutubeDL") +def test_oversize_foreign_completed_path_is_not_deleted( + mock_ydl_class: MagicMock, + tmp_path: Path, +) -> None: + """Oversize rejection must not delete a path outside this import's authority.""" + out_dir = tmp_path / "import-cache" + out_dir.mkdir() + foreign = tmp_path / "foreign-oversize.m4a" + with foreign.open("wb") as handle: + handle.truncate(DEFAULT_MAX_ENCODED_FILE_BYTES + 1) + _configure_download(mock_ydl_class, foreign) + + result = download_youtube_audio( + "https://youtube.com/watch?v=abc123DEF45", + str(out_dir), + ) + + assert result == { + "ok": False, + "error": {"code": "download_error", "message": YOUTUBE_IMPORT_FAILED_MESSAGE}, + } + assert foreign.exists() + assert foreign.stat().st_size == DEFAULT_MAX_ENCODED_FILE_BYTES + 1 + + +@patch("bandscope_analysis.youtube.yt_dlp.YoutubeDL") +def test_same_video_active_lease_blocks_second_import_without_deleting_owner_artifact( + mock_ydl_class: MagicMock, + tmp_path: Path, +) -> None: + """A rejecting same-ID import cannot enter another import's cleanup authority.""" + out_dir = tmp_path / "shared-import-cache" + out_dir.mkdir() + video_id = "abc123DEF45" + first_import_artifact = out_dir / f"{video_id}.m4a" + first_import_artifact.write_bytes(b"first-import-owned-audio") + Path(_import_lease_path(str(out_dir), video_id)).mkdir() + + result = download_youtube_audio( + f"https://youtube.com/watch?v={video_id}", + str(out_dir), + ) + + assert result == { + "ok": False, + "error": {"code": "download_error", "message": YOUTUBE_IMPORT_FAILED_MESSAGE}, + } + assert first_import_artifact.read_bytes() == b"first-import-owned-audio" + mock_ydl_class.assert_not_called() + + +@patch("bandscope_analysis.youtube.yt_dlp.YoutubeDL") +def test_preexisting_same_video_artifact_is_never_overwritten_or_claimed( + mock_ydl_class: MagicMock, + tmp_path: Path, +) -> None: + """An older same-ID final artifact is not ownership evidence for a new import.""" + out_dir = tmp_path / "shared-import-cache" + out_dir.mkdir() + existing = out_dir / "abc123DEF45.m4a" + existing.write_bytes(b"previous-import-audio") + + result = download_youtube_audio( + "https://youtube.com/watch?v=abc123DEF45", + str(out_dir), + ) + + assert result == { + "ok": False, + "error": {"code": "download_error", "message": YOUTUBE_IMPORT_FAILED_MESSAGE}, + } + assert existing.read_bytes() == b"previous-import-audio" + mock_ydl_class.assert_not_called() + + +@patch("bandscope_analysis.youtube.yt_dlp.YoutubeDL") +def test_same_video_prefixed_noncanonical_file_is_not_claimed_as_completed_artifact( + mock_ydl_class: MagicMock, + tmp_path: Path, +) -> None: + """A same-ID prefix alone must not authorize a noncanonical completed filename.""" + out_dir = tmp_path / "shared-import-cache" + out_dir.mkdir() + decoy = out_dir / "abc123DEF45.keep.m4a" + decoy.write_bytes(b"preexisting-noncanonical-audio") + _configure_download(mock_ydl_class, decoy) + + result = download_youtube_audio( + "https://youtube.com/watch?v=abc123DEF45", + str(out_dir), + ) + + assert result == { + "ok": False, + "error": {"code": "download_error", "message": YOUTUBE_IMPORT_FAILED_MESSAGE}, + } + assert decoy.read_bytes() == b"preexisting-noncanonical-audio" diff --git a/services/analysis-engine/tests/test_youtube_transient_cleanup_authority.py b/services/analysis-engine/tests/test_youtube_transient_cleanup_authority.py new file mode 100644 index 000000000..ecaaa29b1 --- /dev/null +++ b/services/analysis-engine/tests/test_youtube_transient_cleanup_authority.py @@ -0,0 +1,40 @@ +"""Regression coverage for lease-bound YouTube transient cleanup authority.""" + +from pathlib import Path + +from bandscope_analysis.youtube import _remove_download_artifacts + + +def test_transient_cleanup_preserves_same_id_nontransient_decoys(tmp_path: Path) -> None: + """An active video lease must not turn every same-ID filename into cleanup authority.""" + out_dir = tmp_path / "import-cache" + out_dir.mkdir() + video_id = "abc123DEF45" + partial = out_dir / f"{video_id}.m4a.part" + fragment = out_dir / f"{video_id}.m4a-Frag1" + part_fragment = out_dir / f"{video_id}.m4a.part-Frag2" + control = out_dir / f"{video_id}.m4a.ytdl" + reported_decoy = out_dir / f"{video_id}.keep.m4a" + stem_decoy = out_dir / f"{video_id}.m4a.keep" + partial.write_bytes(b"partial") + fragment.write_bytes(b"fragment") + part_fragment.write_bytes(b"part-fragment") + control.write_bytes(b"control") + reported_decoy.write_bytes(b"foreign-reported-file") + stem_decoy.write_bytes(b"foreign-stem-sibling") + + _remove_download_artifacts( + { + "tmpfilename": str(partial), + "filename": str(reported_decoy), + }, + str(out_dir), + video_id, + ) + + assert not partial.exists() + assert not fragment.exists() + assert not part_fragment.exists() + assert not control.exists() + assert reported_decoy.read_bytes() == b"foreign-reported-file" + assert stem_decoy.read_bytes() == b"foreign-stem-sibling"