diff --git a/.github/workflows/app-ci.yml b/.github/workflows/app-ci.yml index e13bba02f..f60749711 100644 --- a/.github/workflows/app-ci.yml +++ b/.github/workflows/app-ci.yml @@ -104,6 +104,21 @@ jobs: exit 1 fi + - name: Run repository-root governance contract tests + # tests/ (repo root) holds workflow/YAML contract tests, e.g. + # test_stacked_pr_workflow_contract.py. The backend job's own pytest + # invocation above runs from backend/ and never collects this + # directory, so it needs its own explicit step. Mirrors that step's + # own log scan: a warning-class message pytest prints without + # actually failing the run must not be accepted as clean evidence. + run: | + set -o pipefail + python -m pytest -q tests 2>&1 | tee root_pytest_output.log + if grep -qiE 'timeout|fatal|warn|denied' root_pytest_output.log; then + echo "::error::Root governance contract tests produced Timeout, Fatal, Warn, or Denied outputs" + exit 1 + fi + frontend: name: frontend runs-on: ubuntu-latest diff --git a/AGENTS.md b/AGENTS.md index c8fbb38e3..9104dd1f4 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -189,13 +189,6 @@ in this repo. ## PR automation and review defaults -- Stacked-PR trigger tests must parse the YAML event configuration, not search - source text for `**`: a comment can satisfy that search, and a later - `!feature/**` pattern can exclude the very stack being validated. Preserve the - Actions `on` key when choosing a YAML loader, inspect ordered branch patterns, - and retain rejection tests for both cases. The installed hash-locked PyYAML - dependency is sufficient; do not add a second parser for this contract. - - Follow `docs/development/merge-gate-policy.md` for PR gate interpretation. - PR Governance must stay metadata-only: no PR-head checkout, no admin merge, no review dismissal, and no security-check suppression. diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 9d2cbba18..97f19e71e 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -20,6 +20,21 @@ Runtime database connectivity is secret-injected: `backend/core/config.py` has no fallback `DATABASE_URL`, so missing database configuration fails at startup rather than silently using shared development credentials. +## Reply follow-up scheduling boundary (Proposed, PR #1486) + +`ReplySlaScheduler` binds its existing ORM session to one checked-out connection +for the entire PostgreSQL advisory-lease cycle, including per-workspace task +commits and rollbacks. Normal release requires boolean confirmation. Acquisition +uncertainty, cancellation, or failure invalidates that connection before session +cleanup; a disconnected backend must not be replaced inside the same sweep. +Healthy owner failures roll back and later owners are reloaded asynchronously. +Manual escalation requests can race this scheduler: the shared service refreshes +expired mail after a commit conflict and relies on savepoint rollback to remove +failed inserts. The scheduler reloads owner configuration between workspaces. +This retains source-linked task identity and the existing workspace queries. +It does not promise exactly-once execution, provider writeback, or support for +transaction-pooling proxies. See [decision and evidence](docs/doctoring/reply_sla_physical_lease.md). + ## Topic-intelligence boundary Naruon has no live Structural Topic Modeling endpoint, fitted topic artifact, diff --git a/CHANGELOG.md b/CHANGELOG.md index 7ec84c36f..64c9379a1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,4 +1,842 @@ ## [Unreleased] +- 답장 후속 작업을 저장한 뒤에도 다른 작업 공간의 처리가 이어지도록 예약 처리와 + 동시 요청 충돌 복구를 보완했습니다. DB 연결이 끊기면 해당 처리 회차를 중단하고 + 다음 회차에서 다시 확인합니다. PR #1486의 검증 중인 변경이며 아직 배포하지 않았습니다. +- **(Semgrep 오탐 대응, naruon#1486) `backend/alembic/versions/0011_email_read_state.py`의 + `op.execute(_UPGRADE_SQL)`/`op.execute(_DOWNGRADE_SQL)` 두 호출에 `nosemgrep` 억제 주석을 + 추가했습니다.** Semgrep OSS의 `sqlalchemy-execute-raw-query`/`formatted-sql-query` 규칙이 + 두 호출을 모두 오류로 표시했지만, 실제로는 오탐입니다 — 두 SQL 상수는 고정된 모듈 수준 + 리터럴 `_IS_READ_PROVENANCE_MARKER`만 보간하며, 외부 입력이나 식별자를 전혀 보간하지 + 않습니다(이미 모듈 docstring과 기존 `# nosec B608` 주석에 이 근거가 상세히 기록돼 + 있음). Semgrep 규칙은 "f-string을 인자로 받는 `op.execute()` 호출" 패턴만 매칭하고 보간되는 + 값이 상수인지는 판별하지 못해 오탐이 발생했습니다. 코드 자체는 변경하지 않았습니다(실제 + SQL 인젝션 위험이 없으므로). +- **(CodeRabbit 리뷰 대응, naruon#1501) 첨부파일 reparse content-graph 색인 후속(바로 아래 항목)의 + 전체 리뷰에서 실제 결함 2건이 나와 모두 고쳤습니다.** (1) reparse 임베딩 재생성이 + resolved parse 소스 텍스트 대신 `attachment.content`에서 값을 읽고 있었습니다. + `apply_reparsed_result`는 `result.content`(마크업을 걷어낸 *display* 문자열)가 비어있지 + 않을 때만 `attachment.content`를 덮어쓰는데, `"parsed"` 결과의 display 텍스트는 빈 문자열로 + 스트립되지만 raw `result.parse_content`는 그렇지 않은 경우(예: 보이는 텍스트 노드 없이 + 마크업만 있는 첨부파일) `attachment.content`가 base64로 인코딩된 채 그대로 남아있어, + 임베딩이 실제 재파싱된 텍스트가 아니라 base64 노이즈로부터 생성됐습니다 — content graph는 + 올바른 텍스트로 색인됐는데(`_append_reparsed_attachment_content_graph`가 이미 + `result.parse_content or result.content`를 직접 resolve했으므로, import 시점의 + `email_import_service._extract_and_generate_embeddings`와 동일한 resolve 방식), 임베딩만 + 어긋난 것입니다. `process_reparse_pending_attachment`는 이제 단순 상태 문자열 대신 + `ReparseOutcome(parse_status, embedding_source_text)`를 반환해, 그 동일한 resolved 텍스트를 + attachment 행에서 다시(불안정하게) 유도하지 않고 임베딩 재생성으로 명시적으로 전달합니다. + 신규 테스트: + `test_reparse_that_lands_on_parsed_with_markup_only_content_still_embeds_parse_content`. + (2) `0011_email_read_state.py`의 `downgrade()`가 legacy `emails` 테이블과 `is_read` + 컬럼이 둘 다 있으면 무조건 컬럼을 drop했습니다 — 이 리비전보다 먼저 존재했던(그래서 이 + 리비전의 `NOT EXISTS` 가드가 건드리지 않은) 동명의 `is_read` 컬럼까지 데이터째 파괴할 수 + 있었습니다. `upgrade()`가 이제 자신이 만든 컬럼에 `COMMENT ON COLUMN` provenance 마커 + (`_IS_READ_PROVENANCE_MARKER = "0011_email_read_state:added"`)를 남기고, `downgrade()`는 + `col_description`으로 그 마커가 정확히 있을 때만 drop합니다 — 이 리비전이 추가한 것만 + drop하고 그 외에는 손대지 않습니다. 신규 real-Postgres 테스트: + `test_legacy_email_read_state_downgrade_preserves_a_preexisting_column`(legacy + `emails.is_read` 컬럼에 데이터를 미리 심어두고 upgrade→downgrade를 실행해 컬럼과 데이터가 + 모두 살아남는지 확인). 추가로 `email_import_service._generate_source_embedding`을 공개 + `generate_source_embedding`으로 개명(CodeRabbit nitpick): `content_graph_source_record_uid`, + `append_knowledge_graph_edges`에 이어 `attachment_reparse_worker.py`가 가져다 쓰는 세 + 번째 cross-module 헬퍼이므로, 모든 cross-module 헬퍼가 public일 때 모듈 경계가 일관됩니다. + 검증: 전체 백엔드 스위트 1911 passed/43 skipped(`DATABASE_URL` 미설정, CI와 동일), 이번 + 수정이 건드린 테스트는 전부 실제 PostgreSQL 16 + pgvector에 대해 단독 실행 시 통과 — 같은 + 실제 DB에 대해 스위트 전체를 한 프로세스로 돌리면 이 PR에서 이미 보고된 기존 cross-file + test-ordering 실패 1건(`test_0001_initial_upgrade_succeeds_against_a_fresh_database`가 + 스위트 중간에 `email_records`를 drop·재생성)이 재현되지만, 이번 수정과는 무관합니다. ruff + clean. +- **(Devin 리뷰 대응, naruon#1486 후속) 첨부파일 reparse가 성공적으로 재인식된 콘텐츠를 + 초기 import 경로와 달리 content graph에 색인하지 않던 gap을 고쳤습니다.** + `services/email_import_service.py::_append_email_content_graph`는 첨부파일이 첫 + import에서 정상 파싱되면 `ContentNodeRecord`/`ContentSegmentRecord` 그래프를 + 만들지만, `attachment_reparse_worker.py::apply_reparsed_result`는 `Attachment` + 행 자체 컬럼만 갱신했습니다 — 격리(quarantine)됐던 첨부파일이 나중에 reparse로 + `"parsed"`가 되어도 content-graph 기반 검색/AI-hub 기능에는 계속 보이지 + 않았습니다(`AttachmentParseResult`가 import 경로와 동일한 `parse_content` 필드를 + 이미 들고 있었음에도). `apply_reparsed_result`가 결과 `parse_status`가 + `"parsed"`일 때 새 `_append_reparsed_attachment_content_graph`를 호출하도록 + 추가했습니다 — import 경로가 이미 쓰는 `services.content_graph.parse_content`와, + 새로 공개 API로 옮긴 `content_graph_source_record_uid`(원래 + `email_import_service.py`의 private 함수였던 것을 + `services/content_graph/parser.py`로 옮겨 두 호출부가 공유)를 그대로 재사용해 + 색인 경로를 두 개로 만들지 않았습니다. 영속화된 attachment가 자신이 속한 + 이메일의 첨부파일 목록에서 원래 몇 번째였는지는 신뢰성 있게 재현할 수 없으므로, + reparse 경로의 `source_record_uid`는 import 경로의 message-id + 목록 위치 + 조합 대신 attachment의 영구 `attachment_uid` 하나로만 구성하고, 새 레코드의 + `email_id`는 (import 경로처럼 아직 저장되지 않은 `Email`을 통한 관계 append로 + 간접 설정하는 대신) 이미 영속화된 attachment 행의 `email_id` 컬럼에서 직접 + 가져옵니다. 빈 문자열로만 파싱되는 `"parsed"` 결과(공백만 있는 첨부파일 등)는 + 기존 import 경로와 동일하게 색인을 건너뜁니다. 신규 테스트 3개 + (`test_reparse_that_lands_on_parsed_indexes_the_content_graph`, + blank-content 스킵, non-parsed 스킵). 검증: 전체 백엔드 스위트 1908 + passed/40 skipped, ruff clean. +- **(Devin 리뷰 대응, 🔴 실제 결함, `backend/services/attachment_reparse_worker.py:346-352`) + 커서보다 낮은 id(또는 document의 경우 더 이른 `(created_at, document_id)`)를 가진 + 행이 나중에 외부에서 다시 pending 상태로 되돌려지면, 커서+재시도-집합 설계로는 + 이를 절대 재발견할 수 없었습니다 — 재시도 집합은 이 워커 자신이 이미 보고 + 미해결로 판단한 행만 추적하기 때문입니다.** 실제로 이런 외부 되돌림 경로가 + 존재함을 코드에서 직접 확인: `POST /attachments/{uid}/reparse-intent`와 + `POST /documents/{id}/pdf-dom-recognition-intent`(둘 다 `backend/api/data.py`)가 + 임의의 기존 행을 다시 pending으로 표시할 수 있습니다(newsdom 첨부 sweep에 + 대해서는 이런 되돌림을 유발하는 살아있는 트리거를 찾지 못했지만, 동일한 + 설계 결함이므로 이 PR에서 이미 다뤄온 대칭적 수정 패턴에 따라 세 sweep + 메서드 모두에 동일하게 적용했습니다). 수정: 매 `FULL_RESCAN_EVERY_N_SWEEPS`(20)번째 + sweep마다 커서를 `None`으로 강제 리셋해 전체 재스캔을 수행합니다 — `parse_status`/ + `document_status` 필터가 이미 실제로 해결된 행을 모두 제외하므로, 어떤 주기로 + 실행하든 항상 안전합니다(결과 집합이 넓어질 뿐 틀려지지 않음). 이 수정 과정에서 + 발견한 잠재 버그: `_pending_attachment_statement`/`_pending_document_statement`가 + 커서가 `None`인데(첫 sweep 또는 이번 강제 재스캔) `retry_ids`가 비어있지 않으면 + 조건을 `retry_ids`로만 좁혀버려 나머지 pending 행을 모두 놓치는 경우가 있었습니다 + (지금까지는 최초 sweep에서 `retry_ids`가 항상 비어있어 발현되지 않았을 뿐) — + `retry_ids` 조건이 커서 조건 안에 중첩되어야만 의미가 있도록 고쳤습니다. 검증: + RED(id/커서를 이미 지난 행을 외부에서 pending으로 되돌려도 20 sweep 동안 + 재발견되지 않음을 실제 재현) → GREEN(정확히 20번째 sweep에서 재발견). + `PYTHONPATH=. python -m pytest tests/test_newsdom_worker.py + tests/test_attachment_reparse_worker.py -q` → 34 passed, 23 passed. +- **(Devin 리뷰 대응, 🟡 실제 결함) `backend/api/tenant_config.py`의 `TenantConfigCreate`/ + `TenantConfigResponse`가 `noema_orchestrator_base_url`/`noema_orchestrator_token`을 + 선언하지 않아, `services/orchestrator_gateway.py`가 요구하는 이 게이트웨이 + 자격증명을 지원되는 어떤 API 호출로도 설정할 수 없었습니다(기능이 사실상 + 도달 불가능).** 두 필드를 두 Pydantic 모델에 추가하고, `noema_orchestrator_token`을 + 다른 자격증명(`openai_api_key` 등)과 동일하게 `SECRET_FIELDS`에 등록해 조회 + 시 마스킹되도록 했습니다(생성/수정 로직은 이미 필드-무관 `setattr` 루프라 + 추가 배선이 필요 없었습니다). 동일한 선행 패턴인 + `batch_orchestrator_base_url`/`batch_orchestrator_token`에도 같은 배선 공백이 + 있음을 확인했으나, 이 PR이 추가한 필드만 범위로 좁혔습니다. 검증: RED(POST 후 + GET에서 `KeyError`로 필드 부재 확인) → GREEN. `PYTHONPATH=. python -m pytest + tests/test_tenant_config_api.py -q` → 29 passed, 1 skipped. +- **(Devin 리뷰 대응, 🔴 실제 결함, `backend/services/newsdom_worker.py:707-710`) + `Document.document_id`는 `db/models.py`에서 `f"doc_{uuid.uuid4().hex}"`로 + 무작위 생성되어 삽입 순서와 전혀 무관한데, 문서 sweep의 커서가 이 + `document_id`만으로 전진 여부를 판단하고 있었습니다 — 커서가 전진한 뒤에 + 삽입된 새 문서가 우연히 더 작게 정렬되는 id를 받으면, 한 번도 본 적 없어 + 재시도 집합에도 없고 "id > cursor" 조건도 만족하지 못해 영원히 pending으로 + 남을 수 있었습니다(바로 위 항목의 커서-고정 수정보다 더 심각 — 이미 본 행을 + 지연시키는 게 아니라 전혀 새로운 행을 완전히 놓칠 수 있음).** 커서를 + `document_id` 단일 값 대신 `(created_at, document_id)` 튜플로 바꿔 + `created_at`(삽입 시점에 Python 쪽에서 설정되는, 실제로 삽입 순서와 일치하는 + 타임스탬프)을 1차 정렬 키로, `document_id`는 동일 순간 충돌 시의 tie-breaker로만 + 사용하도록 수정했습니다. 수정 과정에서 발견한 관련 버그: 커서를 + bulk-loaded `rows` 목록에서 직접 계산하면(`document.created_at`) 이전 항목의 + rollback으로 이미 expire된 인스턴스의 속성을 읽을 위험이 있어(기존 + `_ExpiredDocument`류 회귀 테스트가 정확히 이를 검출), 매 반복에서 새로 + re-fetch한 인스턴스에서 즉시 캡처한 `(created_at, document_id)` 쌍만 사용하도록 + 고쳤습니다. 검증: RED(무작위 UUID가 낮게 정렬되지만 늦게 생성된 문서가 + 다음 sweep에서도 계속 pending으로 남음을 실제 재현) → GREEN(같은 시나리오가 + 이제 처리됨). `PYTHONPATH=. python -m pytest tests/test_newsdom_worker.py -q` + → 32 passed. +- **(Devin 리뷰 대응, 🔴 실제 결함, `backend/services/newsdom_worker.py:531-533`) 한 PDF의 + organization에 provider가 설정되어 있지 않으면 `_sweep_attachments`가 매 pass마다 그 + 행 앞에서 커서를 다시 고정해, 그 배치보다 뒤에 있는 PDF들이 무기한 pending으로 + 남을 수 있었습니다.** 먼저 실제로 재현해 검증: 300건 중 영구히 막힌 행 1건만 있는 + 경우는 (상태 필터가 이미 해결된 행을 자연히 제외하므로) 결국 수렴하지만, + batch_limit(50)보다 많은 60건이 연속으로 영구 pending 상태가 되면 매 sweep이 항상 + 같은 첫 50건만 재선택해 나머지 60건이 14 sweep이 지나도 전혀 처리되지 않음을 + 확인했습니다(한 organization이 provider를 설정하기 전에 PDF를 대량으로 import하는 + 경우 등으로 batch_limit을 넘는 연속 pending 행이 실제로 발생할 수 있음). 근본 + 수정: 커서를 "첫 미해결 행 직전"에 고정하는 대신, 커서(`_attachment_cursor`)는 + 본 적 있는 가장 큰 id로 단조 증가만 시키고, 아직 미해결인 행의 id는 별도의 + 영속적인 재시도 집합(`_attachment_retry_ids`)에 담아 커서와 무관하게 매 sweep + `id IN (...)` 조건으로 재시도합니다. 두 조건을 단순히 OR로 묶으면(정렬이 + id 오름차순이라) 재시도 집합이 batch_limit만큼 쌓였을 때 오히려 재시도 행들이 + 매번 전체 슬롯을 차지해 신규 전진(forward) 행을 다시 굶길 수 있어, 두 조건이 + 모두 있을 때는 CASE 버킷으로 forward 행을 항상 retry 행보다 먼저 정렬되도록 + 했습니다(신규 전진이 충분하지 않을 때만 재시도 행이 남은 슬롯을 채움 — 지속적으로 + 포화 상태인 forward 부하 아래서는 재시도 행이 더 오래 기다릴 수 있다는 트레이드오프를 + 의도적으로 받아들였는데, 이는 파이프라인 전체가 멈추는 이전 실패 모드보다 명백히 + 낫습니다). `_sweep_documents`(문자열 키 `document_id`, 사전식 최대값)에도 동일한 + 근본 원인이 있어 동일하게 수정했습니다. `services/attachment_reparse_worker.py`의 + `_sweep_attachments`도 (재사용 가능한) 동일한 커서-고정 패턴을 상속하고 있어(이전에 + CodeRabbit이 지적해 도입된 "첫 실패에서 고정" 수정 자체가 이 취약점의 원인이었음), + 동일한 원인·수정을 적용했습니다. 더 이상 쿼리가 0건을 반환할 때 커서를 `None`으로 + 되돌려 전체를 재스캔하는 방식(계속 새 행이 커서 뒤에 들어오는 한 절대 발동하지 + 않음)에 의존하지 않으므로, 두 워커의 `_load_pending_attachments`/ + `_load_reparse_pending_attachments`에서 wrap-to-None 분기를 제거했습니다. 검증: + RED로 두 워커 모두에서 정확히 이 실패(60건 연속 pending, batch_limit=50, 14 + sweep 후에도 미수렴)를 먼저 재현한 뒤, 수정 후 동일 테스트가 6~7 sweep 안에 + 수렴함을 확인. 기존 커서-고정 계약을 전제로 작성된 6개 테스트(`test_newsdom_worker.py` + 5개, `test_attachment_reparse_worker.py` 1개)를 새 계약(커서는 항상 전진, 재시도는 + 별도 집합)에 맞게 재작성했습니다. `PYTHONPATH=. python -m pytest + tests/test_newsdom_worker.py tests/test_attachment_reparse_worker.py -q` → 31 + + 22 passed; 전체 백엔드 스위트 `PYTHONWARNINGS=error DISABLE_BACKGROUND_WORKERS=1 + python -m pytest -q` → 1911 passed, 39 skipped, ruff clean. +- **(Devin 리뷰 대응, 🟡 실제 결함 2건) stacked-PR 트리거 수정(`a4e01191`)이 4개 워크플로의 + `pull_request:` 트리거에서 `branches:` 제한을 제거하면서, 그 값(`release/**`, `develop`)을 + 리터럴로 assert하던 기존 계약 테스트 2개(`test_app_ci_runs_backend_and_frontend_checks_without_duplicate_release_pushes`, + `test_docker_publish_validates_pr_images_and_publishes_semver_images_only_on_tags`)가 깨진 채 + 방치되어 있었다.** 실제로 재현: `backend/tests/test_release_governance.py`만 단독 실행하면 + 2 failed (owner 코멘트의 "workflow/Alembic contracts 29 passed"는 이 파일 전체를 포함하지 + 않았던 것으로 보임). 두 테스트를 새 의도(스택형 PR 지원을 위해 `pull_request:`가 베이스 + 브랜치를 제한하지 않아야 한다)에 맞게 갱신하고, 동일 계약을 app-ci/docker-publish 양쪽에 + `assert "branches:" not in pull_request_block`으로 통일. 추가로 Devin이 별도 지적한 CI 배선 + 누락도 같은 커밋에서 수정: `tests/test_stacked_pr_workflow_contract.py`는 repo-root + `tests/`에 있는데 `app-ci.yml`의 backend job은 `cd backend && pytest`만 실행해 이 계약 + 테스트를 전혀 collect하지 않았다 — `python -m pytest -q tests` 스텝을 추가하고, 이를 잠그는 + 회귀 테스트(`test_app_ci_collects_repository_root_governance_contract_tests`)를 추가해 진짜 + RED(스텝 부재) 확인 후 GREEN. 전체 백엔드 스위트 1906 passed / 40 skipped, ruff clean, + `scripts/ci/test_pr_governance_gate.sh: PASS`. +- **(Devin 리뷰 대응, 🟡 실제 결함 2건 추가) 위 CI 배선 수정 자체에 대한 Devin의 후속 지적 2건도 + 같은 커밋에서 반영.** (1) 새 회귀 테스트가 `"pytest -q tests" in workflow` 원문 substring + 검사였는데, 이후 누군가 실제 스텝을 지우고 주석에만 같은 문자열을 남기면(`# python -m pytest -q + tests`) 통과해 버려 계약이 무력화될 수 있었다 — `yaml.safe_load`로 워크플로를 파싱해 + `jobs.backend.steps[].run`에 실제로 존재하는 스텝만 인정하도록 재작성(주석은 YAML 파서 단계에서 + 이미 제거되므로 더 이상 매치되지 않음, 직접 확인). (2) 새로 추가한 repo-root `tests/` 스텝이 + 기존 backend 테스트 스텝과 달리 Timeout/Fatal/Warn/Denied 출력 스크리닝이 없어, 이 스텝만 + 금지된 출력을 내고도 CI를 통과할 수 있었다 — 동일한 `grep -qiE 'timeout|fatal|warn|denied'` + 가드를 추가하고, 이를 잠그는 assertion을 같은 회귀 테스트에 통합. 두 항목 모두 수정 전 상태로 + 되돌려 진짜 RED(주석 매치 무시 확인, 스크리닝 부재로 assert 실패) → GREEN 확인 후 복원. +- **(Devin 리뷰 대응, 🟡 minor → 실제로는 진짜 결함) NewsDOM 재인식 sweep의 커서가 + `RESULT_PENDING`(아직 provider 미설정) 행도 실패 없이 진행했다고 취급해 커서를 그 너머로 + 진행시켜, 계속 새 업로드가 들어오는 동안 해당 행이 무기한 굶주릴 수 있었습니다.** + `_sweep_attachments`/`_sweep_documents`는 이미 "예외가 발생한 행은 커서를 그 앞에서 멈춘다"는 + 불변식을 문서화하고 구현했지만, `RESULT_PENDING`(예외 없이 정상 반환되지만 + `pdf_dom_recognition_pending` 상태가 그대로인 경우)은 같은 취급을 받지 못했습니다 — + provider가 아직 설정되지 않은 organization의 첨부/문서가 배치 중간에 있으면, 이후 provider가 + 설정되어도 그 뒤로 새 행이 계속 쌓이는 한 그 특정 행은 `id > cursor` 필터에 걸려 영원히 + 재선택되지 못할 수 있었습니다. 두 sweep 모두 `RESULT_PENDING`을 예외와 동일하게(첫 미해결 + 행에서 커서를 멈추되, 같은 배치의 나머지 행은 계속 처리) 취급하도록 수정. 새 테스트 2개로 + 진짜 RED(커서가 last row까지 진행됨) 확인 후 GREEN. 기존 + `test_document_sweep_advances_and_wraps_without_starvation`은 이 버그가 고쳐지기 전 + 동작(모두 pending인 배치도 커서가 끝까지 진행)을 전제로 작성되어 있어, 수정된 계약(완전히 + resolve된 배치만 커서가 진행하고, wrap 이후에도 여전히 막힌 행은 커서를 None으로 유지)에 + 맞게 시나리오를 다시 작성. +- **(Devin 리뷰 대응, 🟨 실제 결함) 첨부파일 reparse-intent 엔드포인트가 락 없는 + read-then-write로 상태를 전이해, 동시 요청(또는 워커와의 경합)이 최신 결과를 덮어쓸 수 + 있었습니다.** `create_attachment_reparse_intent`가 `quarantined` 상태를 확인한 뒤 락 없이 + `reparse_pending`으로 갱신·커밋했는데, 오래된 읽기를 들고 있는 지연된 중복 요청이 그 사이 + 워커가 이미 처리를 마친 최신 상태를 `reparse_pending`으로 되돌려 덮어쓸 수 있는 TOCTOU + 경쟁이었습니다. `calendar_conflict_judgment_service.apply_correction`이 이미 쓰는 것과 같은 + `with_for_update()` 패턴을 `_get_scoped_attachment`에 `lock` 키워드 인자로 추가해 이 + 엔드포인트에서만 사용하도록 수정. 새 테스트로 컴파일된 쿼리에 `FOR UPDATE`가 포함됨을 + 확인(같은 파일의 다른 호출부는 계속 락 없이 조회), 실제 PostgreSQL에 대해 이 JOIN + + `FOR UPDATE OF` 조합이 유효한 SQL임을 별도로 확인. + 전체 백엔드 스위트: Postgres 기동 시 1942 passed / 3 skipped, 중지 시 1905 passed / 40 + skipped, ruff clean. +- **(🔴 critical, 현실성검증으로 발견: 신선한 DB에 대한 `alembic upgrade head`가 항상 실패) + `backend/.github/workflows/app-ci.yml`의 backend job에는 Postgres 서비스 컨테이너가 전혀 + 구성되어 있지 않다** — `@pytest.mark.postgres`로 표시된 모든 real-PostgreSQL 테스트는 CI에서 + 단 한 번도 실제로 실행된 적이 없고(연결 실패로 항상 조용히 skip), 이 세션에서 로컬 + PostgreSQL 16 + pgvector 확장을 직접 설치·기동해 처음으로 실행해 봄으로써 다음 두 클래스의 + 실재 결함이 드러났다. + 1. **`backend/alembic/versions/0001_initial_control_plane.py::upgrade()`가 + `execute_schema_backfill`의 guard를 우회해, 완전히 새 데이터베이스에 대한 + `alembic upgrade head`가 항상 실패했다.** `Base.metadata.create_all()`는 ORM에 없는 legacy + `emails` 테이블을 만들지 않는데, 0001이 `schema_backfill_sql()`을 직접 순회하며 실행해 + `CREATE INDEX IF NOT EXISTS ix_emails_owner_date ON emails (...)`가 + `relation "emails" does not exist`로 항상 실패했다(`CREATE INDEX IF NOT EXISTS`는 인덱스 + 이름만 보호하지 대상 테이블의 존재 여부는 보호하지 않는다). 완전히 새 데이터베이스에 대해 + 실제로 `alembic upgrade head`를 실행해 이 실패를 직접 재현한 뒤(진짜 RED), + `execute_schema_backfill(connection)`을 호출하도록 수정(진짜 GREEN, 같은 방식으로 재현). + 새 real-Postgres 테스트 `test_0001_initial_upgrade_succeeds_against_a_fresh_database` + (`tests/test_alembic_migrations.py`)와 + `test_schema_backfill_skips_legacy_emails_index_when_table_absent` + (`tests/test_bootstrap_db.py`) 추가. 관련 prose contract test + (`test_initial_alembic_revision_records_current_schema_path`)도 새 구현(`execute_schema_backfill` + 호출)에 맞게 갱신. + 2. **workspace_id NOT NULL 제약이 이 PR에서 추가된 이후, 이를 반영하지 못한 pre-existing + real-Postgres 테스트 19개가 하드 실패했다.** `test_project_graph_api.py`, + `test_project_graph_projection.py`, `test_search_postgres.py`, + `test_tasks_api.py`(각 파일이 이 PR에서 손대지 않은, 완전히 무관한 기존 파일들)의 공유 + `Email(...)` 시딩 헬퍼들이 `workspace_id`를 전혀 넘기지 않아 + `email_records.workspace_id`의 NOT NULL 위반으로 실패. `test_data_api.py`(이 PR이 수정한 + 파일)의 raw SQL INSERT 3건도 `workspace_id`뿐 아니라 `is_read`(ORM 쪽 Python-side + `default=True`, DB 서버측 default 없음)까지 빠뜨리고 있었고, 별도의 raw SQL + `email_attachments` INSERT도 `attachment_uid`(ORM 쪽 Python-side default, 서버측 default + 없음)를 빠뜨려 NOT NULL 위반이었다 — 둘 다 raw SQL이 ORM 레벨 Python 기본값을 우회하기 + 때문에 발생. 모든 위치에 `workspace_id`/`is_read`/`attachment_uid`를 명시적으로 채우도록 + 수정. + 로컬 PostgreSQL 16(+ pgvector)로 두 상태 모두 검증: Postgres 기동 시 1939 passed / 3 skipped, + 중지 시 1902 passed / 40 skipped(정상 skip), 양쪽 다 ruff clean. **CI에 Postgres 서비스가 + 없다는 사실 자체는 이번 커밋의 범위 밖으로 남겨둔다** — 별도 후속 작업으로 + `docs/product-technical-gap-baseline.md`(.github repo)에 기록. +- **(테스트 컨벤션 위반 수정) 병행 세션이 추가한 real-PostgreSQL 테스트 2개가 이 저장소의 + 표준 "Postgres 연결 불가 시 정상 skip" 패턴 없이 작성되어, Postgres가 없는 환경에서 + 스킵 대신 하드 실패하던 문제를 고쳤습니다.** 커밋 `96cd0c07`(`fix(db): skip absent + legacy email table during bootstrap`)가 추가한 + `test_schema_backfill_creates_legacy_emails_index_when_table_exists` + (`tests/test_bootstrap_db.py`)와 + `test_calendar_correction_rationale_real_postgres_smoke` + (`tests/test_alembic_migrations.py`)는 다른 기존 real-Postgres 테스트들과 달리 + `ConnectionRefusedError`/`OperationalError`/`asyncpg.CannotConnectNowError` 등을 잡아 + `pytest.skip(...)`하는 try/except 없이 바로 연결을 시도해, 이 환경(Postgres 미기동)에서 + 실제로 `ConnectionRefusedError`로 하드 실패함을 확인. 기존 real-Postgres 테스트들이 이미 + 쓰는 것과 동일한 except 절을 두 테스트에 추가. 로컬 PostgreSQL 16을 기동해 두 테스트가 + 실제로 통과함을 확인한 뒤, 다시 중지하고 정상적으로 skip됨을 확인 — 두 상태 모두 검증. + 전체 백엔드 스위트: Postgres 없이 1902 passed / 38 skipped, ruff clean. +- **(Devin 리뷰 대응, 🔍 analysis) hybrid 검색이 quarantine/deferred-recognition 상태의 + 첨부파일 base64 원본 payload를 정상 파싱된 콘텐츠처럼 검색 결과에 노출하던 문제를 + 고쳤습니다.** `content_type_mismatch_quarantined`(이 PR에서 새로 추가된 상태)와 + 기존 `pdf_dom_recognition_pending` 등 "parsed"가 아닌 모든 상태는 `Attachment.content`에 + 실제 파싱된 텍스트 대신 base64 인코딩된 원본 바이트 또는 빈 문자열을 저장하는데, + `build_lexical_attachment_statement`/`build_dense_attachment_statement`는 이를 필터링하지 + 않고 그대로 검색 대상에 포함시켰습니다(동일 파일의 `project_graph_object` 채널은 이미 + `_EXCLUDED_PROJECT_OBJECT_STATUS_CODES`로 유사한 필터링을 하고 있었음). 두 statement 모두 + `Attachment.parse_status == "parsed"` 조건을 추가. 새 테스트 + `test_lexical_attachment_statement_excludes_non_parsed_attachments`/ + `test_dense_attachment_statement_excludes_non_parsed_attachments`로 진짜 RED + 확인(`assert "email_attachments.parse_status" in sql`이 수정 전 실패) 후 GREEN. + 전체 백엔드 스위트 1902 passed/36 skipped, ruff clean. +- **(CodeRabbit 리뷰 대응, 🟠 major) 이메일 임포트가 project graph projection을 요청한 + workspace와 다른 workspace에 저장하던 문제를 고쳤습니다.** `_persist_project_graph_projection`이 + 호출자가 이미 해석한 `resolved_workspace_id`를 쓰지 않고 자기 자신이 다시 + `f"workspace-{organization_id}"`로 재계산했습니다 — 명시적으로 다른 workspace를 지정한 + 임포트에서는 Email 행은 요청된 workspace에 저장되지만, 거기서 파생된 project graph 객체는 + 기본 workspace로 잘못 들어갔습니다. `workspace_id`를 필수 인자로 받아 호출자가 넘긴 값을 + 그대로 사용하도록 수정(`services/email_import_service.py`). 새 테스트 + `test_persist_project_graph_projection_uses_the_resolved_workspace_id` — 수정 전 코드가 + `workspace_id` 키워드 인자 자체를 받지 않아 실제 `TypeError`로 RED 확인. 기존 + `tests/test_project_graph_import_wiring.py`의 5개 테스트도 새 계약(호출자가 workspace_id를 + 이미 해석해 넘김)에 맞춰 갱신. +- **(CodeRabbit 리뷰 대응, 🟡 minor) `import_fixtures.py`의 중복 확인 쿼리가 workspace로 + 스코프되지 않던 문제를 고쳤습니다.** email 고유 식별자가 이제 4열 + (`user_id`, `organization_id`, `workspace_id`, `message_id`)인데, 중복 검사 쿼리는 여전히 + 3열(`message_id`, `user_id`, `organization_id`)만 확인했습니다 — workspace B로의 임포트가 + workspace A의 행을 "이미 존재함"으로 잘못 판단해 정당한 재임포트를 건너뛸 수 있었습니다. + `Email.workspace_id == IMPORT_WORKSPACE_ID`를 쿼리에 추가. 기존 테스트 + `test_root_importer_duplicate_check_is_scoped_to_owner`에 WHERE절 전용 검증(단순 + `in query_text` 방식은 `select(Email)`이 workspace_id 컬럼을 SELECT 목록에 항상 포함하므로 + 실제로는 아무것도 증명하지 못함을 확인 후 WHERE절만 분리해 검사하도록 강화)을 추가해 실제 + RED를 먼저 확인. +- **(CodeRabbit 리뷰 대응, 🟡 minor) `bootstrap_db.py`가 Alembic ORM 메타데이터 쪽의 legacy + owner-only 식별자(`uq_emails_owner_message_id`)를 인식하지 못하던 문제를 고쳤습니다.** + 기존 코드는 bootstrap 자체가 만드는 이름(`uq_email_records_owner_message_id`)만 드롭했는데, + `0001_initial_control_plane.py`의 `Base.metadata.create_all()`로 workspace 스코핑 이전에 + 초기화된 DB는 ORM이 만든 다른 이름(`uq_emails_owner_message_id`, Alembic + `0020_email_workspace_scope.py`의 `_OLD_EMAIL_IDENTITY`와 동일)을 갖고 있어 영구히 3열 + 제약이 남을 수 있었습니다. 두 legacy 이름 모두(제약·인덱스 형태 포함) 드롭하도록 수정. + 기존 테스트에 이 두 번째 legacy 이름에 대한 동일한 검증을 추가해 실제 RED 확인. + 이 배치 전체 검증: 전체 백엔드 스위트 1900 passed/36 skipped, ruff clean. +- **(코드 품질 리뷰 대응) `test_calendar_correction_rationale_upgrade_renames_legacy_column`의 + 불필요한 lambda(`lambda: object()`)를 이름 있는 로컬 함수(`_fake_bind`)로 교체했습니다.** + 검증: 전체 백엔드 스위트 1899 passed/36 skipped, ruff clean. +- **(Devin 리뷰 대응, 🔍 analysis) `calendar_conflict_corrections.rationale` 컬럼명을 + 2단어 snake_case 컨벤션에 맞춰 `correction_rationale`로 변경했습니다.** (`db/models.py`, + `alembic/versions/0018_calendar_conflict_judgments.py`, + `services/calendar_conflict_judgment_service.py`.) 동일한 "correction" 테이블 형태를 쓰는 + 기존 `project_graph_object_corrections.rationale`(이 PR 이전부터 존재, 동일한 단어 사용 + 선례)는 이 PR의 범위 밖이라 그대로 두었습니다 — 두 테이블이 당장 일치하지 않게 되는 + 대가보다, 이 PR이 새로 만드는 컬럼이 문서화된 신규 컬럼 명명 규칙(2단어 이상 snake_case)을 + 지키는 쪽을 택했습니다. API 응답 필드명(`rationale`)과 서비스 함수 파라미터명은 변경하지 + 않았습니다 — 규칙은 테이블/컬럼명에 관한 것이지 API 필드명이 아니며, 이미 이 기능은 + 아직 배포되지 않은 신규 기능이라 마이그레이션은 안전하게 컬럼명을 바꿀 수 있었습니다. + 수정 전 실제 RED(`correction.rationale` → `correction.correction_rationale`로 테스트를 + 먼저 바꿔 `AttributeError` 확인) 후 고쳤습니다. 검증: 전체 백엔드 스위트 1897 passed/36 + skipped, ruff clean. +- **(Devin 리뷰 대응, 🟡) 소유자(owner)당 이메일 임포트 할당량이 workspace마다 곱절로 + 늘어나던 문제를 고쳤습니다.** `MAX_IMPORT_EMAILS_PER_OWNER`(1000)와 이를 보호하는 + advisory lock(`_acquire_owner_import_quota_lock`)은 둘 다 `(user_id, organization_id)` + 단위(owner 전체)로 스코프되어 있었는데, 실제 사용량을 세는 + `_owner_email_import_count`는 `Email.owner_filters()`를 그대로 재사용해 + `workspace_id`까지 필터링했습니다 — 같은 owner가 서로 다른 workspace로 임포트할 + 때마다 각 workspace가 독립적으로 새 1000건 한도를 받는 결과가 됩니다. 카운트 쿼리를 + `user_id`/`organization_id`만으로 스코프하도록 고쳐 lock의 스코프와 일치시켰습니다 + (조회/중복확인 등 다른 경로의 workspace 스코핑은 그대로 유지). 새 테스트 + `test_owner_import_quota_count_is_not_scoped_to_a_single_workspace`(`backend/tests/test_emails_api.py`) + — 수정 전 코드에서 카운트 쿼리 SQL 텍스트에 `workspace_id`가 실제로 포함됨을 먼저 + 확인했습니다(mock 세션은 실제 필터링을 하지 않으므로 SQL 텍스트 자체를 검증). 검증: + 전체 백엔드 스위트 1897 passed/35 skipped, ruff clean. +- **(Devin 리뷰 대응, 🔍 analysis) Calendar conflict judgment 영속화 경로에 실제 + PostgreSQL 스모크 커버리지가 없던 문제를 보강했습니다.** `test_calendar_conflict_judgment_api.py`의 + 기존 테스트는 전부 mock 세션(`_DummySession`, fake judgment/correction)만 사용해, + `apply_correction`의 `with_for_update()` row lock과 감사(audit) 스냅샷 영속화가 실제 + PostgreSQL 연결로 한 번도 검증된 적이 없었습니다. 새 테스트 + `test_calendar_conflict_judgment_lifecycle_real_postgres_smoke`(`pytest.mark.postgres`)는 + `create_judgment` → `apply_correction`(실제 row lock) → `list_judgments` 전체 흐름을 + 실제 PostgreSQL 커넥션으로 실행하고 영속/정렬/감사 스냅샷을 검증합니다. `Base.metadata.create_all()` + 대신 필요한 두 테이블(`calendar_conflict_judgments`, `calendar_conflict_corrections`)만 + 생성하도록 스코프했습니다 — pgvector가 설치되지 않은 환경에서 무관한 + `email_records`(vector 컬럼 포함) 생성까지 시도해 skip이 아니라 진짜 실패로 이어지는 + 것을 로컬 PostgreSQL 16으로 재현·회피했습니다. 검증: 로컬 PostgreSQL 기동 시 새 테스트 + 실제 통과 확인, 이후 중지 후 전체 스위트 1897 passed/36 skipped(기존 35+신규 1), ruff clean. +- **(Devin 리뷰 대응, 🟡) `import_fixtures.py`가 커스텀 `NARUON_IMPORT_WORKSPACE_ID`를 + 무시하던 문제를 고쳤습니다.** `import_eml_file`은 스레드 배정(`assign_thread_id`)에는 + `IMPORT_WORKSPACE_ID`(env var 반영)를 넘기면서도, 실제로 저장하는 `Email` 행의 + `workspace_id`는 이를 무시하고 `f"workspace-{IMPORT_ORGANIZATION_ID}"`(또는 + `IMPORT_USER_ID` 기반)를 그 자리에서 다시 계산해 사용했습니다 — env var를 기본값과 다르게 + 설정하면 스레드 배정과 저장이 서로 다른 workspace를 가리켜, 임포트된 대화가 분리되거나 + workspace 기준 조회에서 보이지 않을 수 있었습니다. `workspace_id=IMPORT_WORKSPACE_ID`로 + 단순화(기존 기본값 동작은 `IMPORT_WORKSPACE_ID`의 기본값 표현식 자체가 이미 동일하므로 + 변화 없음). 새 테스트 + `test_root_importer_stores_email_under_configured_workspace_id` — 수정 전 코드가 + `workspace-default` 대신 커스텀 값을 반환하지 못함을 먼저 확인했습니다. 검증: 전체 백엔드 + 스위트 1896 passed/35 skipped, ruff clean. +- **(Devin 리뷰 대응, 🔴 critical) POP3 동기화가 매 실행마다 조용히 메일을 0건 임포트하던 + 버그를 고쳤습니다.** `TenantConfig`에는 애초에 `workspace_id` 컬럼이 없는데, + `Pop3SyncWorker._import_messages`는 `getattr(config, "workspace_id", "")`로 이를 읽어 + 항상 빈 문자열을 얻었고, 곧바로 `if not workspace_id: return 0` 가드에 걸려 실제로 받아온 + POP3 메시지를 전부 버렸습니다(예외나 로그 없이 "0건 임포트"로만 보고). `ImapSyncWorker`가 + 이미 쓰고 있던, 소유자의 기존 임포트 메일이 속한 workspace를 역산하는 + `resolve_unambiguous_workspace_id()`(0건/모호하면 fail-closed)를 `imap_worker.py`에서 + 공용 헬퍼로 추출해 `pop3_worker.py`에서도 재사용하도록 고쳤습니다. `_sync()`가 세션 안에서 + 테넌트별로 workspace를 미리 해석해 `_sync_tenant()`/`_import_messages()`로 전달하며, 해석 + 불가능한 테넌트는 건너뜁니다. 새 테스트 `test_resolve_unambiguous_workspace_id_*`(imap_worker), + `test_pop3_sync_resolves_workspace_from_existing_mail`, + `test_pop3_sync_skips_tenant_with_no_unambiguous_workspace`(pop3_worker) — 수정 전 코드로 + 실제 RED(TypeError: `_sync_tenant`가 여전히 2-인자 시그니처)를 먼저 확인했습니다. 검증: 전체 + 백엔드 스위트 1895 passed/35 skipped, ruff clean. +- **Superseding workspace-scope correction:** historical bullets below that call + `Email.owner_filters()` workspace scoping deferred are no longer current. + The helper now requires `workspace_id`, and every production caller supplies + an authoritative workspace without a silent default. Background mailbox + processing fails closed when its owner-scoped account cannot be tied to an + unambiguous persisted workspace. +- **(Devin 리뷰 대응, 직전 수정 자체의 회귀) `0020_email_workspace_scope`가 legacy identity를 + CONSTRAINT로 잘못 `DROP INDEX`해 마이그레이션 전체를 중단시킬 수 있던 버그를 고쳤습니다.** + PostgreSQL은 UNIQUE CONSTRAINT를 내부적으로 동일 이름의 unique index로 구현하므로, + `inspector.get_indexes()`는 CONSTRAINT의 backing index도 함께 보고합니다. 직전 커밋은 + `existing_indexes` 확인을 `existing_constraints` 확인보다 먼저 실행했는데, legacy identity가 + 실제로는 CONSTRAINT인 경우 `op.drop_index()`가 먼저 시도되어 PostgreSQL이 + `cannot drop index ... because constraint ... requires it`로 거부 — 마이그레이션 전체가 + 중단됩니다(실제 로컬 PostgreSQL 16으로 재현·확인). CONSTRAINT 확인을 먼저 하도록(`elif`로 + 상호 배타화) 순서를 바꿨습니다. 새 실제-PostgreSQL 스모크 테스트 + `test_email_workspace_migration_real_postgres_smoke`(constraint/plain-index 두 형태 모두 + 파라미터화, `pytest.mark.postgres`)가 수정 전 CONSTRAINT 케이스에서 정확히 이 에러로 실패함을 + 먼저 확인한 뒤 고쳤습니다. 검증: 전체 백엔드 스위트 1891 passed/35 skipped(postgres 없이), + 로컬 PostgreSQL 16 기동 시 새 테스트 2건 모두 통과, ruff clean. +- **(Devin 리뷰 대응) Alembic 마이그레이션 `0020_email_workspace_scope`가 `bootstrap_db.py`가 + 만든 owner-only 고유 식별자를 인식하지 못하던 문제를 고쳤습니다.** 이 마이그레이션은 + `get_unique_constraints()`로 `uq_emails_owner_message_id`(Alembic 자체 명명)만 확인했는데, + `bootstrap_db.py`(이 PR 이전 코드)는 같은 개념을 다른 이름(`uq_email_records_owner_message_id`)의 + **plain index**로 만들었습니다 — 이름도 다르고 종류도 달라 마이그레이션이 절대 찾을 수 없는 + 상태였습니다. 과거에 `bootstrap_db.py`로 초기화된 뒤 Alembic으로 전환된 DB는 이 3열 고유 + 인덱스가 영구히 남아, workspace 간 동일 `message_id` 중복을 계속 차단합니다. 마이그레이션이 + 이제 `get_indexes()`로도 확인하고, constraint/index 두 형태 모두 대비해 제거합니다. 새 테스트 + `test_email_workspace_migration_also_drops_bootstrap_created_owner_only_index`. + 검증: 전체 백엔드 스위트 1891 passed/33 skipped, ruff clean. +- **(Devin 리뷰 대응, `b778fb69` 이후) 두 스크립트가 `uq_emails_workspace_message`(4열: + `user_id`, `organization_id`, `workspace_id`, `message_id`)로 교체된 email 고유성 + 계약을 따라가지 못하고 있던 문제를 고쳤습니다.** + - `backend/scripts/import_fixtures.py::process_zip_file`의 + `on_conflict_do_update`가 여전히 옛 3열 `uq_emails_owner_message_id` 대상을 + 가리키고 있었습니다 — 실제 PostgreSQL에서는 `ON CONFLICT` 대상이 기존 고유 + 제약과 정확히 일치해야 하므로, 이 상태로는 비어 있지 않은 ZIP을 임포트할 + 때마다 커밋이 거부됩니다(테스트가 기본으로 쓰는 SQLite는 이 불일치를 허용해 + 로컬에서는 발견되지 않았습니다). `index_elements`를 4열로 갱신했습니다. + 새 테스트 `test_process_zip_file_upsert_targets_workspace_scoped_identity`는 + PostgreSQL dialect로 직접 컴파일해 `ON CONFLICT` 절 자체를 검증합니다. + - `backend/scripts/bootstrap_db.py`(Alembic을 쓰지 않는 로컬/개발용 호환 경로)가 + `_get_validation_and_final_indexes_statements`에서 만든 옛 3열 고유 인덱스 + `uq_email_records_owner_message_id`를 한 번도 제거하지 않아, workspace_id를 + 백필한 뒤에도 같은 사용자/조직의 두 서로 다른 workspace가 동일 + `message_id`를 가질 수 없는 더 엄격한 제약이 남아 있었습니다 — Alembic이 + 관리하는 스키마와 조용히 어긋나는 상태였습니다. workspace_id를 NOT NULL로 + 만든 직후 옛 인덱스/제약을(둘 다 대비해) 제거하고, 동일한 4열 워크스페이스 + 스코프 고유 인덱스(`uq_email_records_workspace_message_id`)를 새로 만들도록 + 고쳤습니다. 새 테스트 + `test_schema_backfill_replaces_owner_only_email_uniqueness_with_workspace_scope`. + - 검증: 전체 백엔드 스위트 1890 passed/33 skipped, ruff clean. +- **Noema workspace/calendar identity hardening:** mail and content-graph tools now + include the independently signed `workspace_id` in SQL scope; signed workspace + identifiers are no longer derived from organization identifiers; email message + uniqueness includes workspace scope; and calendar conflict checks fail closed + until a scoped authoritative provider-calendar read seam exists. +- **(Devin 리뷰 대응) ZIP 아카이브 픽스처 임포트(`backend/scripts/import_fixtures.py::process_zip_file`)가 + `Email.workspace_id`(NOT NULL) 없이 벌크 INSERT를 구성해 비어 있지 않은 + 아카이브를 임포트할 때마다 커밋이 실패하던 문제를 고쳤습니다.** 같은 파일의 + 단일 EML 루트 임포터(`backend/import_fixtures.py`, 이전에 이미 수정함)와는 + 별도의 코드 경로였습니다. 동일한 `workspace-` 관례로 + `batch_values`와 `on_conflict_do_update`의 `set_`에 `workspace_id`를 + 추가했습니다. 새 테스트 + `test_process_zip_file_batch_insert_includes_workspace_id`(수정 전 실제 + RED 확인). +- **(Devin 리뷰 확인, 조치 없음) `backend/services/noema_agent.py`의 + `tool_search_mail`/`tool_read_mail`/`tool_content_graph_query`가 + `Email.owner_filters()`를 통해 `workspace_id` 없이 스코프되는 문제**는 + 검증 결과 이 PR이 새로 만든 노출이 아니라 `b6cb4e6f`(2026-07-13, 이 PR보다 + 한 달 이상 이전)부터 존재한, ADR-0005에 이미 별도 후속 작업으로 기록된 + `Email.owner_filters()`의 동일한 사전 존재 격차였습니다. 세션 초반의 + 명시적 결정("이 PR의 신규 노출만 좁게 수정")에 따라 이번 PR에서 확장 수정하지 + 않았습니다. +- **(Devin 리뷰 대응) `backend/scripts/bootstrap_db.py`에 이번 PR의 신규 컬럼 두 개가 + 누락되어 있던 문제를 고쳤습니다.** `email_records.workspace_id` + (`0020_email_workspace_scope`)와 `email_attachments.attachment_uid` + (`0019_attachment_uid`)는 Alembic 마이그레이션에만 반영돼 있었고, Alembic + 대신 `bootstrap_db.py`(로컬/개발용 `create_all` + 멱등 백필 호환 경로)로 + 기존 데이터베이스를 부트스트랩하면 두 컬럼이 그대로 빠진 채 남아 이후 모든 + 이메일/첨부파일 쿼리가 깨졌습니다. 기존 `webdav_accounts.workspace_id`/ + `project_folders.folder_uid` 백필과 동일한 관례(컬럼 추가 → 백필 → NOT + NULL → 인덱스 생성)로 두 컬럼을 추가했습니다. `workspace_id` 백필은 + `organization_id`가 이미 NOT NULL로 검증된 뒤(`_get_validation_and_final_indexes_statements` + 이후)에 실행되도록 순서를 맞췄습니다. 새 테스트 + `test_schema_backfill_adds_email_workspace_column_and_index`, + `test_schema_backfill_adds_attachment_uid_column_and_index`. +- **(Devin 리뷰 대응, 보안) 재파싱이 격리 보관 중이던 원본 바이트를 삭제하던 + 문제를 고쳤습니다.** `apply_reparsed_result`가 재분류 결과를 무조건 + `attachment.content`에 덮어썼는데, `parse_email_attachment`는 + `unsupported_content_type`/`parse_size_limit_exceeded`처럼 표시할 내용이 + 없는 상태에서 `content=""`을 반환합니다 — 격리(quarantine)된 첨부파일이 + 재파싱을 거쳐 "정상 파일이지만 아직 지원하지 않는 타입"으로 판정되면, 유일하게 + 보관돼 있던 원본 바이트가 빈 문자열로 영구히 사라졌습니다. 이제 결과의 + `content`가 비어 있지 않을 때만 덮어씁니다. 새 테스트 + `test_reparse_to_unsupported_content_type_preserves_retained_bytes`. +- **(보안 수정, 정정) 서명된 세션의 `workspace` 클레임이 `org` 클레임과 + 실제로 일치하는지 서버가 검증하지 않던 문제를 `api/auth.py`에서 + 고쳤습니다.** 이전 커밋의 CodeRabbit/Devin 리뷰 검증 항목(바로 아래)은 + "HMAC 경로는 이 저장소의 코드가 `workspace-` 외의 값을 + 절대 쓰지 않으므로 안전하다"고 결론 내렸으나, 이는 틀린 추론이었습니다 — + HMAC 세션은 이 저장소에 코드가 없는 외부 control-plane 토큰 발급자 + (`iss=naruon-control-plane`)가 발급하므로, 이 저장소의 데이터 기록 + 경로만 봐서는 발급되는 `workspace` 클레임 값을 전혀 증명할 수 없습니다. + CodeRabbit이 `_auth_context_from_session_payload`를 직접 추적해 + `org`·`workspace` 두 클레임이 각각 존재하는지만 검사할 뿐 둘의 관계는 + 전혀 검증하지 않음을 정확히 지적했습니다. 이제 + `_auth_context_from_session_payload`가 `workspace`가 정확히 + `workspace-`가 아니면 세션을 거부(401)합니다 — HMAC과 + OIDC 두 경로 모두 이 함수를 거치므로 한 곳에서 근본적으로 닫힙니다. + `Email` 뿐 아니라 `workspace_id`로 스코프되는 모든 테이블(`Document`, + `WebdavAccount`, `ProjectFolder`, `CalendarConflictJudgment`, + `CarddavAccount`)의 경계가 이제 실제로 서버가 강제하는 불변식이 + 됩니다. 새 테스트: + `backend/tests/test_auth_real.py::test_build_auth_context_rejects_workspace_claim_not_derived_from_org`. + 기존 테스트 2건(`test_security_api.py`의 HMAC 비인가 검사, + `test_data_api.py`의 데이터 품질 쿼리 스코프 검사)이 자신의 `org` + 클레임과 불일치하는 `workspace` 값을 우연히 쓰고 있어 이번 변경으로 + 의도치 않게 401로 막혔기에, 각 테스트가 실제로 검증하려던 동작만 + 격리되도록 org와 일치하는 workspace 값으로 수정했습니다. ADR-0005에 + 정정 경위를 기록했습니다. +- (CodeRabbit/Devin 리뷰 검증, 최초 결론 — 위 항목에서 정정됨) + `workspace-` 백필 관례의 신뢰 경계를 직접 추적해 + 확인했습니다 — HMAC 세션 경로에서는 `AuthContext.organization_id`가 + 항상 non-null이고 이 저장소 어디에도 조직 하나가 workspace를 두 개 + 이상 갖거나 커스텀 workspace 이름을 가질 수 있는 코드 경로가 없어 + (`WorkspaceRunnerConfig`가 두 컬럼 모두 `unique=True`), 파생값과 실제 + 서명된 값이 항상 일치함을 확인했습니다. 실제 노출 지점은 이 PR보다 + 오래된, 더 넓은 범위의 것이었습니다 — 엔터프라이즈 OIDC 경로 + (`api/auth.py`의 `_decode_cached_oidc_session_payload`)가 외부 IdP의 + `workspace` 클레임을 정규화 없이 그대로 신뢰하는데, 이는 + `docs/operations/auth-key-management.md`에 아직 "가설(Hypothesis)" + 단계로 명시된, 프로덕션에 배포되지 않은 경로이고, 이미 배포된 다른 모든 + `workspace_id` 스코프 테이블(`Document`, `WebdavAccount`, + `ProjectFolder`, `CalendarConflictJudgment`, `CarddavAccount`)에도 + 동일하게 적용되는 문제라 `Email` 마이그레이션 하나만 고쳐서 닫을 수 있는 + 범위가 아닙니다. ADR-0005에 별도의 후속 작업으로 기록했습니다. Devin이 + 지적한 `email_import_service.py`의 workspace_id 재파생(서명된 + `auth_context.workspace_id`를 쓰지 않고 organization_id로부터 다시 + 계산)도 같은 근거로 검증했습니다 — 아키텍처 관찰로는 정확하지만, 위 + 추적 결과 파생값과 실제 서명된 값이 오늘 기준 항상 일치하므로 현재 + 악용 가능한 버그는 아닙니다(다중 호출부 배관 변경이 필요해 이 PR의 + 범위를 벗어나는 별도 개선으로 기록). +- (보안, IDOR 근본 수정) `Email`이 `workspace_id`를 전혀 갖고 있지 않아 + `_email_scope_filter`(및 이를 쓰는 `_get_scoped_attachment`/모든 + quality-surface 통계 쿼리)가 `user_id`/`organization_id`로만 스코프돼, + 동일 사용자·동일 조직이지만 `workspace_id`가 다른 세션이 다른 workspace의 + 이메일/첨부파일을 읽거나(기존) 이번 PR이 추가한 + `POST /attachments/{attachment_uid}/reparse-intent`로 변경할(신규) 수 + 있던 문제를 고쳤습니다. `Email`에 `workspace_id` 컬럼을 추가하고(Alembic + `0020_email_workspace_scope`, 기존 행은 `workspace-`로 + 백필 — `organization_id`가 NOT NULL이고 `Email`이 실제 workspace_id를 + 가진 어떤 테이블과도 FK로 연결돼 있지 않아 조인 백필이 불가능함을 확인한 + 뒤, `services/email_import_service.py` 등에서 이미 쓰이던 동일 관례를 + 그대로 적용), `_email_scope_filter`가 `Document`/`WebdavAccount`/ + `ProjectFolder`에 이미 쓰이던 `_owner_scope_statement`의 패턴과 동일하게 + workspace 조건을 무조건 적용하도록 했습니다 — 호출부 14곳이 전부 + `*email_scope`로 언패킹하므로 코드 변경 없이 자동으로 반영됩니다. 새 + 이메일을 만드는 프로덕션 경로 3곳(`email_import_service.py`, + `imap_worker.py`, `import_fixtures.py`)도 동일 관례로 workspace_id를 + 채우도록 갱신했습니다. 신규 테스트: 동일 사용자·동일 조직·다른 workspace + 거부 케이스. **범위를 의도적으로 좁혔습니다**: 메일 목록/검색/온톨로지/ + 스레딩/Noema 에이전트가 쓰는 별도의 `Email.owner_filters()` classmethod도 + 동일한 결함을 갖고 있으나, 이를 고치려면 7개 이상 파일에 걸친 앱 전체 + 읽기 경로 변경이 필요해 이번 PR(캘린더 충돌 도구 추가)의 범위를 크게 + 벗어납니다 — ADR-0005 Consequences에 별도 후속 PR로 명시적으로 기록하고 + 이번에는 손대지 않았습니다. 검증: 신규/수정 테스트, 전체 백엔드 스위트 + 1880 passed/33 skipped, ruff clean. +- (CodeRabbit review 반영) `AttachmentReparseWorker`/`NewsdomRecognitionWorker` + 둘 다 advisory lease를 잡는 전용 `AsyncConnection`에 `AUTOCOMMIT` + isolation level을 설정하지 않고 있었습니다 — lock 획득 `SELECT`가 암묵적 + 트랜잭션을 열고, 그 트랜잭션이 스윕 전체 동안(실제 항목 처리는 별도의 + 세션에서 일어나는데도) 커밋되지 않은 채 idle 상태로 남아 있었습니다. + PostgreSQL의 `idle_in_transaction_session_timeout`이 설정된 환경에서는 + 이 커넥션이 스윕 도중 강제 종료될 수 있고, 그러면 lease가 조용히 + 풀려 다른 replica가 중복 스윕을 시작할 수 있었습니다. 두 워커의 + `_try_acquire_sweep_lease` 모두 lock 획득 전에 + `await connection.execution_options(isolation_level="AUTOCOMMIT")`를 + 호출하도록 고쳤습니다(advisory lock 자체는 세션 스코프라 AUTOCOMMIT과 + 무관하게 계속 유지됨). 검증: 신규 테스트 2개(두 워커 각각), 전체 + 백엔드 스위트 1879 passed/33 skipped, ruff clean. +- (G-15 follow-up, 근본 수정) `services/newsdom_worker.py`가 + `AttachmentReparseWorker`와 똑같은 두 결함을 그대로 갖고 있던 것을 고쳤습니다 — + ADR-0005 Revisions와 gap-baseline에 추적 기록해 둔 바로 그 후속 후보입니다. (1) 🔴 + PostgreSQL advisory lease를 매 항목 `commit()`/`rollback()`이 커넥션을 풀로 + 반환하는 동일한 `AsyncSession`으로 획득·해제해, 해제가 lock을 잡았던 것과 다른 + 물리 커넥션에서 실행되어 lease가 영구히 묶일 수 있던 문제 — 스윕 전체 동안 여는 + 전용 `AsyncConnection` 하나로만 획득·해제하도록 재설계했습니다 + (`_engine_uses_postgresql()`/`_try_acquire_sweep_lease`/`_release_sweep_lease`가 + 이제 세션이 아니라 커넥션을 받습니다). (2) 🔴 첨부파일/문서 두 스윕 모두 배치의 + 마지막 행으로 커서를 처리 *전에* 미리 전진시켜, 처리 중 예외로 pending 상태 그대로 + 남은 행이 커서 아래로 떨어져 전방 큐가 완전히 비워질 때까지 다시 선택되지 못하던 + 동일한 starvation 버그 — 첫 실패 행 바로 앞까지만 전진하도록 고쳤습니다. 문서 + 커서는 `Document.document_id`가 정수가 아닌 문자열 기본키라 "실패 id - 1" 같은 + 산술이 불가능해서, 실패 이전에 실제로 커밋된 마지막 행의 id를 추적하는 방식으로 + 구현했습니다(연속된 정수 키에서는 기존 방식과 동일한 결과, 비연속/문자열 키에서도 + 올바름). 두 스윕 모두 처리 전 각 행을 id로 다시 가져오도록도 바꿨습니다(기존 + bulk-loaded 인스턴스를 재사용하지 않음) — `AsyncSession.rollback()`이 이전 항목의 + 실패 이후 세션에 이미 로드된 모든 객체를 expire시키므로, 이전에 로드된 인스턴스의 + 속성을 읽으면 그 실패를 격리하는 대신 새로운 에러가 나기 때문입니다(같은 코드베이스의 + `AttachmentReparseWorker`가 이미 검증·적용한 패턴을 그대로 따랐습니다). 검증: 신규 + 테스트 6개(lease 커넥션 전환 1 + 커서 캡 2 + stale-instance 재현 방지 2 + 논-postgres + 엔진 분기 1), 전체 백엔드 스위트 1879 passed/33 skipped(기존 1875), ruff clean. +- (Devin/코드품질 봇 review 반영, G-15/캘린더 충돌) naruon#1486에 도착한 3건을 추가로 고쳤습니다: + (1) 🟡 Alembic `0019_attachment_uid`의 `downgrade()`가 항상 `op.drop_index`만 호출했는데, + `Base.metadata.create_all()`로 새로 부트스트랩된 DB(로컬/개발 전용 경로)에서는 + `attachment_uid`의 유일성이 (동일한 이름 `uq_email_attachments_uid`의) 테이블 수준 + `UniqueConstraint`로 만들어져 있어 PostgreSQL이 `DROP INDEX`를 거부하는 문제 — + 이제 `inspector.get_unique_constraints()`로 두 형태를 구분해 제약 형태면 + `op.drop_constraint(..., type_="unique")`를, 인덱스 형태면 기존 `op.drop_index`를 + 사용합니다(이 저장소에 Postgres 기반 마이그레이션 테스트 하네스가 없어 실행 검증은 + 불가 — "PostgreSQL persistence remains unverified" 스레드와 동일한 기존 repo 전역 + 한계). (2) 🔍 `list_judgments`가 `created_at`만으로 정렬해, 동일 타임스탬프를 가진 + 두 judgment가 200행 경계 근처에서 호출마다 순서가 바뀔 수 있던 문제 — 단조 증가 + primary key `calendar_conflict_judgment_id`를 2차 정렬 키로 추가해 결정적으로 + 만들었습니다. (3) 📝 코드 품질 지적 — 테스트 전용 스텁 `_ExpiredAttachment.__getattr__`가 + `AssertionError`를 raise하던 것을 던 던더 메서드 관례에 맞게 `AttributeError`로 + 교체했습니다(테스트 동작은 동일). `correction_action`이 고정 vocabulary 없이 자유 + 텍스트라는 지적은 회신만 남겼습니다 — `services/project_graph`의 동일 컬럼이 이미 + 같은 패턴(자유 텍스트, `String(64)`)을 쓰고 있어 이 PR이 새로 도입한 설계가 아니라 + 기존 컨벤션을 그대로 따른 것이며, vocabulary를 강제하려면 두 기능을 함께 바꿔야 하는 + 더 큰 범위의 결정이라 이 PR 단독으로 다루지 않았습니다. 검증: 신규/수정 테스트 2개, + 전체 백엔드 스위트 1875 passed/33 skipped, ruff clean. +- (CodeRabbit review 반영, G-15/AttachmentReparseWorker) naruon#1486에 도착한 2건의 실제 정합성 + 결함을 고쳤습니다: (1) 🔴 `_sweep_attachments`가 배치의 마지막 행 id로 커서를 배치 처리 *전에* + 미리 전진시켜, 처리 중 예외가 발생해 `reparse_pending` 상태 그대로 남은 행이 커서 아래로 + 떨어져 — `id > cursor` 필터 때문에 — 전방 큐가 완전히 비워질 때까지(지속적인 reparse-intent + 트래픽 하에서는 무한정) 다시 선택되지 못하고 굶주리던 문제. 커서를 이제 배치 내 "첫 실패 + 행 바로 앞"까지만 전진시켜(실패 이후 행들은 이미 처리됐어도 `parse_status` 필터가 걸러주므로 + 무해) 실패한 행이 다음 스윕에서 반드시 재선택되도록 했습니다. (2) 🔴 PostgreSQL advisory + lease를 매 항목 `commit()`/`rollback()`을 호출하는 동일한 `AsyncSession`으로 획득·해제하던 + 문제 — `AsyncSession.commit()`은 매 호출마다 커넥션을 풀로 반환하므로(SQLAlchemy의 통상 + "connectionless execution" 동작), lease 해제가 실제로 lock을 잡았던 것과 *다른* 물리 + 커넥션에서 실행될 수 있었습니다. PostgreSQL advisory lock은 획득한 backend 세션에 묶이므로, + 불일치하는 unlock은 조용한 no-op이 되어 그 커넥션이 나중에 재활용/종료될 때까지 lease가 + 묶인 채로 남아 모든 replica의 스윕을 조용히 멈추게 할 수 있었습니다 — 이제 스윕 전체 동안 + 열어두는 전용 커넥션 하나에서만 획득·해제합니다. `services/newsdom_worker.py`도 동일한 + 구조를 공유해 같은 잠재 결함을 가진 것으로 추정되나, 이 PR이 건드리지 않은 기존 코드라 이번 + 수정 범위 밖입니다 — ADR-0005 Revisions 및 gap-baseline에 추적 기록. 검증: 신규 테스트 + 1개(커서 캡 회귀) + 기존 lease 테스트 재구성, 전체 백엔드 스위트 1875 passed/33 skipped + (기존 1874), ruff clean. +- (Devin review 반영, G-15/AttachmentReparseWorker) naruon#1486에 도착한 3건을 실제로 고쳤습니다: + (1) 🟡 generic content_type(`application/octet-stream` 등, 확장자로도 해석 안 되는 경우)로 + 선언된 첨부파일이 알려진 매직 바이트로 sniff되기만 하면 영원히 quarantine되던 문제 — + `_is_genuine_content_type_mismatch`가 이제 `parse_content_type`이 여전히 generic 상태면 + (확장자로 구체적인 타입으로 해석된 경우는 제외) 불일치로 취급하지 않습니다. sender가 아무 것도 + 구체적으로 주장하지 않은 첨부파일은 애초에 반박할 "선언"이 없었으므로 quarantine 대상이 아닙니다. + (2) 🟡 `AttachmentReparseWorker._sweep_attachments`에서 한 첨부파일 처리 실패 시 `rollback()`이 + 같은 세션에 이미 로드된 나머지 행들을 전부 expire시켜, 후속 항목의 동기 속성 읽기가 실패하며 + 배치 전체가 굶주리던 문제 — 매 항목을 벌크 로드된 객체 대신 `session.get()`으로 매번 새로 + 가져오도록 변경해 이 클래스의 버그 전체를 근본적으로 회피합니다. (3) 🔍 `calendar_conflicts.py`의 + `_request_validation_error_response`가 correction error_code를 영어 메시지 부분 문자열로 + 선택하던 문제(저장소 관례 위반: "routes must not derive... from message substrings") — + `CalendarConflictCorrectionIncoherentError`에 `CalendarConflictUnsupportedValueError`와 + 동일한 패턴의 안정적 `error_code` 속성을 추가하고, 두 Pydantic model_validator 모두 일반 + `ValueError` 대신 `PydanticCustomError(error_code, message)`를 raise하도록 변경해 + `RequestValidationError.errors()[i]["type"]`이 메시지 문구와 무관한 안정적 식별자를 갖게 + 했습니다 — 매퍼는 이제 `type`으로만 분기합니다("must reject before calling apply_correction" + 동작은 그대로 유지). 검증: 신규 테스트 10개, 전체 백엔드 스위트 1874 passed/33 skipped + (기존 1864), ruff clean. +- **(G-15 두 번째 슬라이스) `reparse_pending`을 실제로 소비하는 `AttachmentReparseWorker`**를 + 추가했습니다(`services/attachment_reparse_worker.py`, `NewsdomRecognitionWorker`와 동일한 + jittered-loop + PostgreSQL advisory-lock lease + starvation-free cursor 구조로 `main.py` + lifespan에 배선). 매 스윕마다 `reparse_pending` 첨부파일을 보존된 원본 바이트 + 원래 선언된 + `content_type`으로 `parse_email_attachment`를 다시 호출해 재평가합니다 — sniff된 타입을 + 신뢰하는 별도 로직을 두지 않고 동일한 분류 파이프라인에 같은 질문을 다시 던지는 방식이라, + 향후 그 파이프라인에 생기는 어떤 수정(예: 이미 반영된 OOXML 오탐 수정)도 자동으로 적용됩니다. + 재평가 결과 더 이상 불일치가 아니면 정상 분류로, 여전히 실재하는 불일치면 다시 quarantine + 상태로 돌아갑니다. 보존된 payload가 유효한 base64가 아닌 경우(재시도해도 고쳐지지 않는 문제)만 + 새 terminal 상태 `reparse_payload_invalid`로 분류합니다. `services/attachment_parser.py`에 + PDF 전용이 아닌 범용 base64 디코더 `decode_quarantined_attachment_payload`를 추가했습니다. + ADR-0005의 "no consumer yet" 서술을 갱신했습니다. 검증: 신규 테스트 19개(worker 15개 + parser + 디코더 4개) 추가, 전체 백엔드 스위트 1864 passed/33 skipped, ruff clean. +- **(G-15 첫 슬라이스) 첨부파일 content-type 불일치 quarantine + `attachment_uid` + reparse-intent + API**를 추가했습니다. `services/attachment_parser.py`가 이제 첨부파일의 실제 바이트를 알려진 + 매직 바이트 시그니처(PDF/PNG/JPEG/GIF/ZIP)로 스니핑하고, sniff된 타입이 선언된(또는 확장자로 + 추론된) content_type과 다르면 파싱/보류/unsupported 분류 대신 `parse_status = + parse_error_code = "content_type_mismatch_quarantined"`으로 격리합니다 — 선언된 타입은 + `content_type`에, 실제 타입은 기존 `parse_content_type` 컬럼에 남겨 새 컬럼 없이 두 값을 + 비교하는 것만으로 불일치를 알 수 있게 했고, 원본 바이트는 base64로 보존합니다(기존 deferred-PDF와 + 동일한 `MAX_ATTACHMENT_PARSE_SOURCE_BYTES` 상한 적용). `Attachment`에 다른 신규 엔티티들과 + 동일한 컨벤션의 `attachment_uid` 오파크 id를 추가했습니다(Alembic `0019_attachment_uid`, + 기존 행 백필 포함). `POST /api/data/attachments/{attachment_uid}/reparse-intent`가 + quarantine된 첨부파일을 `reparse_pending`으로 전환하는 intent를 기록합니다(다른 `-intent` + 엔드포인트와 동일하게 실제 재파싱은 아직 없는 별도 워커 슬라이스로 미룸). 설계 근거는 + `docs/adr/0005-attachment-content-type-quarantine.md` 참고. 검증: 신규 테스트 8개 + (파서 5개 + API 3개) 추가, 전체 백엔드 스위트 1842 passed/33 skipped, ruff clean, + `alembic heads`가 `0019_attachment_uid` 단일 head로 수렴. +- (Devin/CodeRabbit review 반영, G-15) naruon#1486에 도착한 세 건을 실제로 고쳤습니다: (1) + 🟡 DOCX/XLSX/PPTX 등 ZIP 기반 컨테이너 형식이 ZIP 매직 바이트와 일치한다는 이유만으로 + content_type_mismatch_quarantined 오탐이 발생하던 문제 — `_is_genuine_content_type_mismatch`가 + 이제 sniff된 타입이 ZIP이고 선언된 타입이 알려진 ZIP 컨테이너 계열(OOXML/ODF/EPUB/JAR, MIME + 타입 부분 문자열로 판정해 개별 나열 불필요)이면 불일치로 취급하지 않습니다 — 다른 타입으로 + 선언된 ZIP은 여전히 격리됩니다. (2) 🟡 상한을 초과해 원본 바이트를 보존하지 못한 mismatch가 + 여전히 content_type_mismatch_quarantined 상태를 받아 reparse-intent API가 이를 그대로 + 수락해버리는 문제 — 이제 다른 초과-크기 첨부파일과 동일하게 parse_size_limit_exceeded(재시도 + 불가 terminal 상태)를 받아 reparse-intent가 애초에 받아들이지 않습니다. (3) CodeRabbit + 코드 컨벤션 지적 — `apply_correction`의 status_code/decision_code 검증이 텍스트 전용 + `ValueError`였던 것을, 저장소 관례(`CalendarPolicyValidationError`와 동일한 패턴)를 따라 + `error_code` 속성을 가진 `CalendarConflictUnsupportedValueError`로 교체하고 API 라우트에서 + 타입 기반으로 매핑하도록 했습니다(현재 REST 경로는 Literal 타입으로 이미 막혀 있어 도달 + 불가능하지만, 향후 비-HTTP 호출자를 위한 방어적 일관성 확보). 별도로, `_get_scoped_attachment`가 + workspace_id를 검증하지 않는 🟥 보안 지적은 실재하지만 이 PR이 만든 문제가 아니라 `Email` + 모델 자체가 애초에 workspace_id 컬럼을 가진 적이 없다는 저장소 전반의 기존 gap임을 확인했다 + (`docs/adr/0005-attachment-content-type-quarantine.md`의 Consequences에 상세 기록) — 제대로 + 고치려면 `Email` 마이그레이션 + 기존 모든 email/attachment 쿼리 갱신이 필요해 이 PR 범위를 + 벗어나므로, 조용히 임시방편을 넣는 대신 별도 후속 작업으로 명시했다. 검증: 신규/수정 테스트 + 다수 추가, 전체 백엔드 스위트 1845 passed/33 skipped, ruff clean. +- (Devin review 반영, 3차) override가 judgment의 **현재** decision_code와 동일한 값을 다시 + 제출하는 경우, `apply_correction`이 실제로 값이 바뀔 때만 `reason_code`/`recommended_action`을 + 교체하도록 고쳤습니다 — 이전에는 "override"라는 이유만으로 실제 변경이 없어도 원래 + 정확했던 이유/안내 문구를 불필요하게 지워버렸습니다. 또한 `docs/doctoring/status-weighted-calendar-conflicts.md`가 + "No database objects or migrations are introduced"라고 여전히 stateless하다고 서술하던 + 부분을 갱신했습니다 — judgment/correction 영속화 슬라이스가 실제로 Alembic + `0018_calendar_conflict_judgments`로 테이블 2개를 도입했으므로, shipped boundary·rollback + 순서(judgments/corrections는 실제 고객 데이터가 쌓이면 downgrade가 파괴적임)·verification + evidence(workspace 격리, row lock, coherence 검증, `default_recommended_action` 단일 소스)를 + 반영했습니다. `/evaluate` 자체는 여전히 완전히 무상태입니다. 검증: 신규 테스트 1개 추가, + 전체 백엔드 스위트 1836 passed/32 skipped, ruff clean. +- (Devin review 반영, 2차) `calendar_conflict_judgment_service.py`/API에 대한 6건의 추가 + 지적을 반영했습니다: (1) **[보안, 최우선]** judgment/correction 테이블과 조회·정정 쿼리에 + `workspace_id`를 추가했습니다 — 이전에는 `user_id`+`organization_id`만으로 범위를 제한했는데, + `AuthContext.workspace_id`는 세션 토큰의 독립 claim이라(테스트 스텁만 편의상 user_id/org에서 + 파생) 동일 user_id/organization_id가 서로 다른 workspace를 오갈 수 있어 워크스페이스 경계를 + 넘어 판단을 열람·정정할 수 있었습니다. 신설 `project_graph` 모듈의 기존 workspace_id 스코핑 + 관례를 그대로 따랐습니다(Alembic `0018`은 아직 어떤 DB에도 적용되지 않은 이번 PR 자체 + 마이그레이션이라 새 마이그레이션 대신 직접 수정). (2) `list_judgments`의 200건 상한 이후로 + 접근 불가능해지는 문제를, 전체 페이지네이션 대신 `GET + /api/calendar/conflicts/judgments/{judgment_uid}` 단건 조회 엔드포인트로 해소했습니다(judgment_uid를 + 아는 호출자는 언제나 개별 조회 가능). (3) `decision_code`를 바꾸는 정정이 그 rationale을 + `recommended_action`으로 그대로 저장해 사람이 남긴 "왜 바꿨는지" 설명이 향후 스케줄링 안내처럼 + 보이던 문제를, `calendar_conflict_policy.py`에 새로 추가한 `default_recommended_action()`(정책 + 자체의 decision_code→recommended_action 단일 소스, `evaluate_calendar_conflicts`도 이제 이걸 + 재사용)로 대체해 고쳤습니다 — rationale은 correction 감사 기록에만 남습니다. (4) + `status_code`(confirm/override/dismiss)와 `decision_code`가 서로 모순되는 조합(override인데 + 새 decision 없음, confirm/dismiss인데 decision을 바꾸려 함)을 API 요청 모델의 + model_validator와 `apply_correction` 양쪽에서(비-HTTP 호출자 대비) 거부하도록 + `validate_correction_coherence()`를 추가했습니다. (5) `calendar_conflict_ics.py`의 ICS + 파서가 별도로 하드코딩했던 500건 상한을 `MAX_EXISTING_COMMITMENTS` 공유 상수로 통합했습니다. + (6) Noema 도구(`check_calendar_conflict`)가 malformed 행을 건너뛴 개수를 + `skipped_existing_count`로 응답에 포함해, "정상적으로 available"과 "증거를 일부 버리고 + available"을 구분할 수 있게 했습니다. 검증: 신규 테스트 다수 추가, 전체 백엔드 스위트 + 1835 passed/32 skipped(무관한 process-group 타이밍 테스트 1건이 전체 스위트 동시 실행에서만 + 간헐적으로 실패했으나 단독 실행 시 통과 확인, 이번 변경과 무관), ruff clean, `alembic heads` + 단일 head 유지. +- Noema general agent(`services/noema_agent.py`)에 `check_calendar_conflict` 도구를 + 추가했습니다. `/api/calendar/conflicts/evaluate`와 동일한 상태 가중 결정론적 + 정책(`evaluate_calendar_conflicts`)을 그대로 재사용해 Noema의 일정 충돌 판단이 + 고객용 API와 절대 어긋나지 않습니다. 회의 제안/변경 메일을 다룰 때 이미 알고 + 있는 commitment(메일·태스크에서 파악한 일정)와 대조해 `available` / + `review_required` / `blocked`을 판단하도록 시스템 프롬프트에도 반영했습니다. + Naruon은 공급자 캘린더 이벤트를 서버에 저장하지 않으므로 이 도구는 provider를 + 직접 조회하지 않고, 호출자가 제시한 commitment만 평가합니다. 잘못된 형식의 + 기존 commitment 행은 건너뛰고 전체 판단을 막지 않습니다. `.github`의 + 중앙 리뷰 에이전트(Noema OIDC 브로커)와 이름은 같지만 서로 다른 개별 + 에이전트임을 `registered_agents.json`/`task_agent_mapping.json`에 명확히 + 했습니다: Noema는 `.github`에서는 CI 리뷰 에이전트, naruon에서는 워크스페이스 + 전반(메일·콘텐츠 그래프·태스크·일정)을 다루는 범용 어시스턴트입니다. + 검증: `PYTHONPATH=. python -m pytest backend/tests/test_noema_agent.py -q` + (21 passed), 전체 백엔드 스위트 `python -m pytest -q` (1808 passed, 32 skipped). + **(2026-09-02 정정)** 이 항목이 처음 쓰인 시점에는 naruon Noema가 "테넌트가 + 구성한 자체 LLM provider"로 직접 동작한다고 기술했으나, 이는 잘못된 경계 + 설정이었습니다 — 아래 "owner 아키텍처 지적 반영" 항목에서 이를 + contextual-orchestrator 게이트웨이 경유로 교정했습니다. naruon이 두 번째 + provider 라우팅 권한이 되는 것은 아닙니다. + **(2026-09-02 두 번째 정정, owner 코멘트 반영)** 위 "서로 다른 개별 + 에이전트임을... 명확히 했다"는 서술과, 아래 "owner 아키텍처 지적 반영" + 항목의 "naruon Noema 도메인 로직 vs `.github` 리뷰 Noema 도메인 로직"이라는 + 분리 서술 모두 owner가 직접 정정했습니다: `ContextualWisdomLab/.github`의 + `docs/CWL-MASTER-CONTEXT.md`는 Noema를 naruon·`.github` CI 리뷰 + 에이전트·wardnet AI SOC 격리 샌드박스가 함께 소비하는 **단일 공유 에이전트 + 런타임**(Pydantic-AI/Codex-Python)으로 명시적으로 정의하고 있으며, 이는 + 이름만 같은 우연이 아니라 처음부터 의도된 설계였다고 owner가 직접 + 확인했습니다. naruon#1527이 자신의 `docs/adr/0006-noema-bounded-context-separation.md` + (해당 PR 자신의 브랜치에만 존재하며, 병합 전인 이 브랜치에서는 아직 로컬 + 경로로 열람할 수 없음)에서 "영구적으로 분리 유지" 결론을 + superseded-on-arrival로 갱신했습니다 — 코드 수준 조사(각 저장소가 현재 + 실제로 무엇을 하는지) 자체는 유효하지만, "그러므로 영구히 분리한다"는 + 결론은 철회되었습니다. 이 파일의 + `noema_orchestrator_base_url`/`noema_orchestrator_token`을 `.github`의 + CI 리뷰 자격증명과 별도로 두는 것은 여전히 유효한 이 모듈의 보안 스코핑 + 선택이지만, 두 배포가 영구히 분리되어 있거나 이름 외에 아무것도 공유하지 + 않는다는 주장은 아닙니다. 실제 공유 런타임 설계는 이 ADR로 확정되지 않은 + 별도의 후속 과제로 남습니다. +- (Devin review 반영, G-06 증분) `calendar_conflict_judgment_service.py`에 대한 5건의 지적을 + 반영했습니다: (1) `apply_correction`이 대상 judgment 행을 `SELECT ... FOR UPDATE`로 잠가 + 동시 정정 요청이 같은 이전 상태를 읽고 감사 기록이 서로를 덮어쓰는 경쟁을 막습니다. (2) + 정정이 `decision_code`를 바꿀 때 `reason_code`/`recommended_action`도 함께 + `corrected_by_human_review`/정정 rationale로 교체해, "available인데 재조정을 + 안내"하는 것처럼 서로 다른 결정의 필드가 섞인 응답이 나오지 않게 했습니다(원본 값은 + before_json 감사 흔적에 그대로 남습니다). (3) `list_judgments`에 200건 상한을 추가해 + 장기 계정의 무제한 조회를 막았습니다. (4) `noema_agent.py`의 + `_MAX_EXISTING_COMMITMENTS`(500)를 `api/calendar_conflicts.py`의 동일 상수와 별도로 + 들고 있어 서로 어긋날 수 있었던 문제를, 두 곳 모두 + `services/calendar_conflict_policy.py`의 공유 상수 + `MAX_EXISTING_COMMITMENTS`를 참조하도록 고쳐 근본적으로 막았습니다. (5) + `tests/test_calendar_conflict_judgment_api.py`의 `api.calendar_conflicts` 이중 import + 스타일(`import` + `from ... import`)을 단일 `import ... as` 형태로 정리했습니다. + PostgreSQL 트랜잭션을 직접 구동하는 실 DB 동시성 테스트는 이 세션에 PostgreSQL 접근이 + 없어 추가하지 못했습니다 — `test_project_graph_api.py`의 기존 Postgres-스킵 스모크 + 테스트와 동일한 한계이며, PR 코멘트로 남겼습니다. 검증: 신규/변경 테스트 4개 추가(총 + 12 passed), 전체 백엔드 스위트 1825 passed/32 skipped, ruff clean. +- G-06(킬러 워크플로: thread/sender ontology → temporal commitment/conflict → + human correction) 증분: `evaluate_calendar_conflicts` 결정을 + `calendar_conflict_judgments` 테이블에 판단(judgment)으로 저장하고, 사람이 + 그 판단을 정정(correction)할 수 있는 API를 추가했습니다. `POST + /api/calendar/conflicts/judgments`는 기존 `/evaluate`와 동일한 정책을 + 평가한 뒤 `judgment_uid`로 결과를 영속화하고(발신 스레드/메시지 id를 + 선택적으로 함께 기록), `GET /api/calendar/conflicts/judgments`는 + 스레드별로 목록을 조회하며, `POST + /api/calendar/conflicts/judgments/{judgment_uid}/corrections`는 + `project_graph_corrections`와 동일한 before/after JSON 감사 흔적 패턴으로 + 사람의 override/confirm/dismiss를 기록합니다(`status_code`: + proposed→confirmed/overridden/dismissed). `evaluate_calendar_conflicts` + 자체의 순수 계산 계약은 바뀌지 않았고, `/evaluate`는 여전히 상태를 저장하지 + 않습니다. 새 테이블은 Alembic + `0018_calendar_conflict_judgments`에서 구조화 op으로 추가했습니다. 검증: + `python -m pytest backend/tests/test_calendar_conflict_judgment_service.py + backend/tests/test_calendar_conflict_judgment_api.py -q` (8 passed), 전체 + 백엔드 스위트 `python -m pytest -q` (1821 passed, 32 skipped). +- (Devin review 반영) `check_calendar_conflict`가 `existing`이 500건 상한을 넘으면 조용히 + 잘라내지 않고 `calendar_existing_batch_exceeded` 오류로 fail closed하도록 수정했습니다. + 상한 이후에 실존하는 충돌이 잘려나가 `available`로 오판되는 것을 막습니다(REST + 엔드포인트의 동일 상한 처리와 일치). 검증: `test_noema_agent.py` 22 passed. +- **(owner 아키텍처 지적 반영, 🔴 실제 경계 위반) `noema_agent.py`가 여전히 + `resolve_runtime_llm_provider`를 import해 테넌트의 직접 LLM provider + `base_url`/`api_key`로 `AsyncOpenAI`를 구성하고 있었습니다 — 이는 naruon의 + "provider 라우팅 권한이 아니라 Noema 도구/인가/컨텍스트만 소유한다"는 현재 + 경계와 충돌합니다.** 프로덕션 LLM 라우팅은 전적으로 + `ContextualWisdomLab/contextual-orchestrator`가 소유합니다. 오래된 Cursor PR + #1384가 의도된 계약(테넌트 스코프 orchestrator 자격증명, orchestrator 전용 + 모델 별칭, 업스트림 provider로의 fallback 없음)을 담고 있었으나 head/base가 + stale해 그대로 이식하지 않고, 계약만 현재 소스에 재구현했습니다. 먼저 RED로 + `run_noema_agent`가 `resolve_runtime_llm_provider`/직접 provider 구성에 도달할 + 수 없음을 증명(수정 전 코드에서 실제 실패 확인)한 뒤: 새 + `services/orchestrator_gateway.py`(`resolve_orchestrator_gateway` + + `OrchestratorGateway` — `tenant_configs.noema_orchestrator_base_url`/ + `noema_orchestrator_token`을 SSRF 허용목록 검증까지 마쳐 해석, 모델 별칭은 + 항상 고정된 `ORCHESTRATOR_MODEL_ALIAS`)을 추가하고, Alembic + `0022_noema_orchestrator_gateway`로 두 컬럼을 `tenant_configs`에 추가했습니다 + (기존 `batch_orchestrator_base_url`/`batch_orchestrator_token` + 패턴(migration `0012`)을 그대로 미러링). `noema_agent.py`는 이제 게이트웨이가 + 없거나 무효면 업스트림 provider로 폴백하지 않고 구조화된 + `status="unavailable"`/`error_code="orchestrator_gateway_unavailable"`로 + 즉시 abstain합니다. naruon의 이 게이트웨이 자격증명은 `.github`의 CI 리뷰 + 자격증명 경로와 별도로 스코핑되어 있어 이 경로로 워크스페이스 데이터가 + 흘러가지 않습니다 — 다만 이것이 naruon Noema와 `.github` 리뷰 Noema가 + 영구히 분리된 별개 에이전트라는 뜻은 아닙니다. **(2026-09-02 owner 코멘트로 + 정정)** `docs/CWL-MASTER-CONTEXT.md`에 따르면 Noema는 naruon·`.github` CI + 리뷰 에이전트·wardnet AI SOC 격리 샌드박스가 공유하는 단일 에이전트 + 런타임이며, 실제 공유 런타임 설계는 naruon#1527(자신의 + `docs/adr/0006-noema-bounded-context-separation.md`에서, 병합 전에는 그 + PR 자신의 브랜치에만 존재)이 확정하지 않은 별도 후속 과제입니다. + 검증: RED(패치 전 `resolve_runtime_llm_provider` 참조가 실제로 존재함을 + spy로 확인) → GREEN(픽스 후 같은 스파이는 `AttributeError`로 실패 — + `resolve_runtime_llm_provider`가 모듈에서 완전히 사라졌다는 가장 강한 + 증거이므로, 테스트를 `assert not hasattr(noema_agent, + "resolve_runtime_llm_provider")`로 재작성); 로컬에 pydantic-ai-slim[openai] + 2.9.0(`backend/requirements-agent.txt` 고정 버전)을 실제로 설치해 실 `AsyncOpenAI` + 구성 경로까지 스킵 없이 검증. `PYTHONPATH=. python -m pytest + backend/tests/test_noema_agent.py -q` → 22 passed, 0 skipped; 전체 백엔드 + 스위트 `PYTHONWARNINGS=error DISABLE_BACKGROUND_WORKERS=1 python -m pytest -q` + → 1908 passed, 39 skipped, ruff clean. - 긴 이메일·첨부 본문을 의미 단위 청크로 임베딩한 뒤 기존 email/attachment 벡터 계약으로 평균화하고, 청크 요청·벡터 누적을 제한된 창으로 처리합니다. OpenAI `text-embedding-3-*`에는 저장 차원(`1536`)을 직접 요청하도록 보강했습니다. 합성 메일 fixture 5건(70청크)과 provider 요청 계약으로 1,536차원 벡터 경로를 검증했으며, 실행 시 선택한 임베딩 제공자에 본문·파싱된 첨부 텍스트를 전송할 수 있습니다. 회사 기밀 데이터는 fixture·commit·PR·log에 포함하지 않습니다. - EmailDetail 테스트가 지원하지 않는 스레드 병합/분리 버튼을 `textContent`뿐 아니라 `aria-label`과 `title` 접근 가능 이름으로도 검출하도록 바꿔, 아이콘 전용 버튼 회귀를 놓치지 않습니다. diff --git a/backend/alembic/versions/0001_initial_control_plane.py b/backend/alembic/versions/0001_initial_control_plane.py index cc14ce39b..31b187f55 100644 --- a/backend/alembic/versions/0001_initial_control_plane.py +++ b/backend/alembic/versions/0001_initial_control_plane.py @@ -9,7 +9,7 @@ from sqlalchemy import text from db.models import Base -from scripts.bootstrap_db import schema_backfill_sql +from scripts.bootstrap_db import execute_schema_backfill revision = "0001_initial_control_plane" down_revision = None @@ -19,8 +19,7 @@ def upgrade() -> None: connection = op.get_bind() connection.execute(text("CREATE EXTENSION IF NOT EXISTS vector")) Base.metadata.create_all(connection) - for statement in schema_backfill_sql(): - connection.execute(statement) + execute_schema_backfill(connection) def downgrade() -> None: diff --git a/backend/alembic/versions/0011_email_read_state.py b/backend/alembic/versions/0011_email_read_state.py index 716590cd1..056b6eb73 100644 --- a/backend/alembic/versions/0011_email_read_state.py +++ b/backend/alembic/versions/0011_email_read_state.py @@ -1,10 +1,26 @@ """Add is_read to emails (IMAP \\Seen read state). Existing rows default to read so historical/file imports do not surface as unread. + +Deliberate exception to this repo's "Alembic migrations use structured +operations (``op.create_index``, ...), never ``sa.text(f"...")`` DDL" rule +(``AGENTS.md``/``CLAUDE.md``): ``upgrade()``/``downgrade()`` below use +``op.execute()`` with the module-level ``_UPGRADE_SQL``/``_DOWNGRADE_SQL`` +constants instead of a structured ``op.*`` call. That rule's actual target is +DDL built from interpolated identifier strings (an injection-safety concern); +these constants interpolate only ``_IS_READ_PROVENANCE_MARKER``, a fixed +module-level literal, never an identifier or a value built from a variable, +external input, or runtime state -- the same safety property a structured +call would have. The reason a structured call isn't used is different: this +migration's behavior must be conditional on whether the legacy ``emails`` +table exists, evaluated at apply time (see the comment on ``_UPGRADE_SQL`` +below for why that check cannot live in Python), and no structured Alembic +operation expresses "run this DDL only if a runtime condition holds" -- a +``DO $$ ... $$`` block is the correct primitive for that, not a workaround +for one. """ from alembic import op -import sqlalchemy as sa # revision identifiers, used by Alembic. revision = "0011_email_read_state" @@ -12,18 +28,94 @@ branch_labels = None depends_on = None +# Fresh installations materialize the current ``email_records`` model in the +# 0001 baseline, including ``is_read``. This historical side branch only +# applies to databases that still carry its legacy ``emails`` table. +# +# The condition has to be evaluated in SQL, not Python: offline SQL +# generation (``alembic upgrade --sql``, a real flag ``scripts/migrate_db.py`` +# exposes) has no live connection to introspect with and no specific target +# database to ask "does this legacy table exist" at generation time either -- +# the same static script is meant to later be applied by a DBA against +# whichever database they choose, fresh-install or legacy. A Python-side +# check (``sa.inspect(op.get_bind())``) can only ever answer that question +# for one hypothetical target chosen at generation time, so it is wrong for +# the other: skip unconditionally and the column silently never gets added +# for a legacy database that applies the generated script (while +# ``alembic_version`` still advances, permanently hiding the gap); inspect +# online and bake in one fixed answer and the same script fails outright +# against the other kind of target. A ``DO $$ ... $$`` block defers the +# check to apply time instead, so the one generated script is correct +# against either kind of target, online or offline-then-applied-later alike. +# +# ``to_regclass('emails')`` (not ``information_schema.tables`` by bare +# ``table_name``) deliberately: the unqualified ``ALTER TABLE emails`` below +# resolves through the connection's ``search_path``, and ``to_regclass`` +# resolves an unqualified name exactly the same way, returning NULL if it +# doesn't. ``information_schema.tables`` filtered only by ``table_name`` +# ignores ``search_path`` entirely and matches a same-named table in *any* +# schema the connecting role can see -- on a deployment with more than one +# accessible schema, that could find an unrelated ``emails`` table outside +# the search path while the unqualified ``ALTER TABLE emails`` targets a +# different (or no) table, passing the guard for the wrong relation or +# aborting the migration outright. Resolving both the check and the DDL +# through the same name lookup makes that mismatch structurally impossible. +# +# ``COMMENT ON COLUMN emails.is_read`` tags the column with a provenance +# marker (``_IS_READ_PROVENANCE_MARKER``) the moment upgrade() actually adds +# it. downgrade() only drops the column when that exact marker is present +# (CodeRabbit, naruon#1501): an ``emails.is_read`` column that already +# existed before this revision ran -- from some other, unrelated origin -- +# would upgrade()'s ``NOT EXISTS`` guard correctly leave alone, but an +# unconditional ``DROP COLUMN IF EXISTS`` on downgrade would still destroy it +# and its data, since a downgrade has no other way to tell "I added this" +# apart from "this happens to be present". Checking the marker via +# ``col_description`` makes downgrade drop only what this exact revision's +# upgrade created. +_IS_READ_PROVENANCE_MARKER = "0011_email_read_state:added" +_UPGRADE_SQL = f""" +DO $$ +BEGIN + IF to_regclass('emails') IS NOT NULL AND NOT EXISTS ( + SELECT 1 FROM pg_attribute + WHERE attrelid = to_regclass('emails') + AND attname = 'is_read' + AND NOT attisdropped + ) THEN + ALTER TABLE emails ADD COLUMN is_read boolean NOT NULL DEFAULT true; + COMMENT ON COLUMN emails.is_read IS '{_IS_READ_PROVENANCE_MARKER}'; + END IF; +END $$; +""" # nosec B608 + +_DOWNGRADE_SQL = f""" +DO $$ +BEGIN + IF to_regclass('emails') IS NOT NULL AND EXISTS ( + SELECT 1 FROM pg_attribute + WHERE attrelid = to_regclass('emails') + AND attname = 'is_read' + AND NOT attisdropped + ) AND col_description(to_regclass('emails'), ( + SELECT attnum FROM pg_attribute + WHERE attrelid = to_regclass('emails') + AND attname = 'is_read' + AND NOT attisdropped + )) = '{_IS_READ_PROVENANCE_MARKER}' THEN + ALTER TABLE emails DROP COLUMN IF EXISTS is_read; + END IF; +END $$; +""" # nosec B608 + +# False positive on both calls below: _UPGRADE_SQL/_DOWNGRADE_SQL interpolate only the +# fixed module-level literal _IS_READ_PROVENANCE_MARKER (see the module docstring above), +# never external input or an identifier -- the same safety property a parameterized query +# would have. Semgrep's raw-query/formatted-sql-query rules pattern-match on +# "op.execute(f-string)" and cannot see that the interpolated value is a constant. def upgrade() -> None: - op.add_column( - "emails", - sa.Column( - "is_read", - sa.Boolean(), - nullable=False, - server_default=sa.text("true"), - ), - ) + op.execute(_UPGRADE_SQL) # nosemgrep: python.sqlalchemy.security.sqlalchemy-execute-raw-query.sqlalchemy-execute-raw-query,python.lang.security.audit.formatted-sql-query.formatted-sql-query def downgrade() -> None: - op.drop_column("emails", "is_read") + op.execute(_DOWNGRADE_SQL) # nosemgrep: python.sqlalchemy.security.sqlalchemy-execute-raw-query.sqlalchemy-execute-raw-query,python.lang.security.audit.formatted-sql-query.formatted-sql-query diff --git a/backend/alembic/versions/0018_calendar_conflict_judgments.py b/backend/alembic/versions/0018_calendar_conflict_judgments.py new file mode 100644 index 000000000..a93654843 --- /dev/null +++ b/backend/alembic/versions/0018_calendar_conflict_judgments.py @@ -0,0 +1,127 @@ +"""add calendar conflict judgments and corrections + +Revision ID: 0018_calendar_conflict_judgments +Revises: 0017_merge_newsdom_carddav_heads +Create Date: 2026-08-30 00:00:00.000000 +""" + +from alembic import context, op +import sqlalchemy as sa + +revision = "0018_calendar_conflict_judgments" +down_revision = "0017_merge_newsdom_carddav_heads" + +_JUDGMENT_TABLE = "calendar_conflict_judgments" +_CORRECTION_TABLE = "calendar_conflict_corrections" + + +def _is_offline_mode() -> bool: + """Detect SQL-only execution without requiring an EnvironmentContext proxy.""" + try: + return bool(context.is_offline_mode()) + except (AttributeError, NameError): + # Focused migration tests invoke upgrade/downgrade with an Operations + # proxy but no Alembic EnvironmentContext. That path is online and + # supplies either a real or deliberately mocked bind. + return False + + +def _online_inspector(): + """Return a live schema inspector only when Alembic has a database bind.""" + if _is_offline_mode(): + return None + return sa.inspect(op.get_bind()) + + +def upgrade() -> None: + inspector = _online_inspector() + + if inspector is None or not inspector.has_table(_JUDGMENT_TABLE): + op.create_table( + _JUDGMENT_TABLE, + sa.Column("calendar_conflict_judgment_id", sa.Integer(), nullable=False), + sa.Column("judgment_uid", sa.String(length=96), nullable=False), + sa.Column("user_id", sa.String(), nullable=False), + sa.Column("organization_id", sa.String(), nullable=True), + sa.Column("workspace_id", sa.String(), nullable=False), + sa.Column("proposed_commitment_id", sa.String(length=256), nullable=False), + sa.Column("source_thread_id", sa.String(), nullable=True), + sa.Column("source_message_id", sa.String(), nullable=True), + sa.Column("decision_code", sa.String(length=32), nullable=False), + sa.Column("reason_code", sa.String(length=64), nullable=False), + sa.Column("recommended_action", sa.Text(), nullable=False), + sa.Column("policy_version", sa.String(length=32), nullable=False), + sa.Column("conflicts_json", sa.JSON(), nullable=False), + sa.Column("status_code", sa.String(length=32), nullable=False), + sa.Column("created_at", sa.DateTime(timezone=True), nullable=False), + sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False), + sa.PrimaryKeyConstraint("calendar_conflict_judgment_id"), + sa.UniqueConstraint( + "judgment_uid", name="uq_calendar_conflict_judgments_uid" + ), + ) + + if inspector is None or not inspector.has_table(_CORRECTION_TABLE): + op.create_table( + _CORRECTION_TABLE, + sa.Column("calendar_conflict_correction_id", sa.Integer(), nullable=False), + sa.Column("correction_uid", sa.String(length=96), nullable=False), + sa.Column("calendar_conflict_judgment_id", sa.Integer(), nullable=False), + sa.Column("user_id", sa.String(), nullable=False), + sa.Column("organization_id", sa.String(), nullable=True), + sa.Column("workspace_id", sa.String(), nullable=False), + sa.Column("actor_user_id", sa.String(), nullable=False), + sa.Column("correction_action", sa.String(length=64), nullable=False), + sa.Column("before_json", sa.JSON(), nullable=False), + sa.Column("after_json", sa.JSON(), nullable=False), + # 0018 intentionally records the historical column name; 0021 + # performs the released schema transition to correction_rationale. + sa.Column("rationale", sa.Text(), nullable=True), + sa.Column("created_at", sa.DateTime(timezone=True), nullable=False), + sa.ForeignKeyConstraint( + ["calendar_conflict_judgment_id"], + ["calendar_conflict_judgments.calendar_conflict_judgment_id"], + ), + sa.PrimaryKeyConstraint("calendar_conflict_correction_id"), + sa.UniqueConstraint( + "correction_uid", name="uq_calendar_conflict_corrections_uid" + ), + ) + + for table_name, indexes in _calendar_conflict_indexes().items(): + for index_name, column_names in indexes: + op.create_index( + index_name, + table_name, + column_names, + if_not_exists=True, + ) + + +def downgrade() -> None: + inspector = _online_inspector() + + for table_name in (_CORRECTION_TABLE, _JUDGMENT_TABLE): + if inspector is None or inspector.has_table(table_name): + for index_name, _column_names in reversed( + _calendar_conflict_indexes()[table_name] + ): + op.drop_index(index_name, table_name=table_name, if_exists=True) + op.drop_table(table_name) + + +def _calendar_conflict_indexes() -> dict[str, list[tuple[str, list[str]]]]: + return { + _JUDGMENT_TABLE: [ + ( + "ix_calendar_conflict_judgments_scope_thread", + ["user_id", "organization_id", "workspace_id", "source_thread_id"], + ), + ], + _CORRECTION_TABLE: [ + ( + "ix_calendar_conflict_corrections_judgment", + ["calendar_conflict_judgment_id"], + ), + ], + } diff --git a/backend/alembic/versions/0019_attachment_uid.py b/backend/alembic/versions/0019_attachment_uid.py new file mode 100644 index 000000000..a0ba9b982 --- /dev/null +++ b/backend/alembic/versions/0019_attachment_uid.py @@ -0,0 +1,97 @@ +"""add attachment_uid opaque id to email_attachments + +Revision ID: 0019_attachment_uid +Revises: 0018_calendar_conflict_judgments +Create Date: 2026-08-30 00:00:00.000000 +""" + +import uuid + +from alembic import op +import sqlalchemy as sa + +revision = "0019_attachment_uid" +down_revision = "0018_calendar_conflict_judgments" + +_ATTACHMENT_TABLE = "email_attachments" +_ATTACHMENT_UID_INDEX = "uq_email_attachments_uid" + + +def _attachment_table_stub() -> sa.TableClause: + return sa.table( + _ATTACHMENT_TABLE, + sa.column("id", sa.Integer()), + sa.column("attachment_uid", sa.String()), + ) + + +def upgrade() -> None: + connection = op.get_bind() + inspector = sa.inspect(connection) + if not inspector.has_table(_ATTACHMENT_TABLE): + return + + existing_columns = { + column["name"] for column in inspector.get_columns(_ATTACHMENT_TABLE) + } + if "attachment_uid" not in existing_columns: + op.add_column( + _ATTACHMENT_TABLE, + sa.Column("attachment_uid", sa.String(length=96), nullable=True), + ) + attachments = _attachment_table_stub() + rows = connection.execute( + sa.select(attachments.c.id).where(attachments.c.attachment_uid.is_(None)) + ).fetchall() + for (attachment_id,) in rows: + connection.execute( + sa.update(attachments) + .where(attachments.c.id == attachment_id) + .values(attachment_uid=f"attachment_{uuid.uuid4().hex}") + ) + op.alter_column(_ATTACHMENT_TABLE, "attachment_uid", nullable=False) + + existing_indexes = { + index["name"] for index in inspector.get_indexes(_ATTACHMENT_TABLE) + } + if _ATTACHMENT_UID_INDEX not in existing_indexes: + op.create_index( + _ATTACHMENT_UID_INDEX, + _ATTACHMENT_TABLE, + ["attachment_uid"], + unique=True, + if_not_exists=True, + ) + + +def downgrade() -> None: + connection = op.get_bind() + inspector = sa.inspect(connection) + if not inspector.has_table(_ATTACHMENT_TABLE): + return + + # A database built by alembic upgrade carries _ATTACHMENT_UID_INDEX as a + # plain unique index (this file's own upgrade() uses op.create_index), but + # one bootstrapped fresh via Base.metadata.create_all() (db/models.py's + # Attachment declares the same name as a table-level UniqueConstraint) + # carries a constraint-owned index of the identical name instead -- + # PostgreSQL rejects a bare DROP INDEX on that shape ("cannot drop index + # ... because constraint ... requires it"), so the two shapes need + # different drop statements rather than one op.drop_index() for both. + unique_constraint_names = { + constraint["name"] + for constraint in inspector.get_unique_constraints(_ATTACHMENT_TABLE) + } + if _ATTACHMENT_UID_INDEX in unique_constraint_names: + op.drop_constraint( + _ATTACHMENT_UID_INDEX, _ATTACHMENT_TABLE, type_="unique" + ) + else: + op.drop_index( + _ATTACHMENT_UID_INDEX, table_name=_ATTACHMENT_TABLE, if_exists=True + ) + existing_columns = { + column["name"] for column in inspector.get_columns(_ATTACHMENT_TABLE) + } + if "attachment_uid" in existing_columns: + op.drop_column(_ATTACHMENT_TABLE, "attachment_uid") diff --git a/backend/alembic/versions/0020_email_workspace_scope.py b/backend/alembic/versions/0020_email_workspace_scope.py new file mode 100644 index 000000000..9cfccd6f2 --- /dev/null +++ b/backend/alembic/versions/0020_email_workspace_scope.py @@ -0,0 +1,166 @@ +"""add workspace_id scope column to email_records + +Revision ID: 0020_email_workspace_scope +Revises: 0019_attachment_uid +Create Date: 2026-08-31 00:00:00.000000 +""" + +from alembic import op +import sqlalchemy as sa + +revision = "0020_email_workspace_scope" +down_revision = "0019_attachment_uid" + +_EMAIL_TABLE = "email_records" +_EMAIL_WORKSPACE_INDEX = "ix_email_records_workspace_id" +_OLD_EMAIL_IDENTITY = "uq_emails_owner_message_id" +# backend/scripts/bootstrap_db.py's dev-compat path predates this migration +# and creates the owner-only identity under a different name and as a plain +# index rather than a named unique constraint; a database bootstrapped before +# that script's own fix landed and later migrated via Alembic needs this +# shape recognized too, or it keeps the stricter, non-workspace-scoped +# identity forever. +_BOOTSTRAP_OLD_EMAIL_IDENTITY = "uq_email_records_owner_message_id" +_EMAIL_WORKSPACE_IDENTITY = "uq_emails_workspace_message" + + +def _email_table_stub() -> sa.TableClause: + return sa.table( + _EMAIL_TABLE, + sa.column("id", sa.Integer()), + sa.column("user_id", sa.String()), + sa.column("organization_id", sa.String()), + sa.column("workspace_id", sa.String()), + sa.column("message_id", sa.String()), + ) + + +def upgrade() -> None: + connection = op.get_bind() + inspector = sa.inspect(connection) + if not inspector.has_table(_EMAIL_TABLE): + return + + existing_columns = { + column["name"] for column in inspector.get_columns(_EMAIL_TABLE) + } + if "workspace_id" not in existing_columns: + op.add_column( + _EMAIL_TABLE, sa.Column("workspace_id", sa.String(), nullable=True) + ) + emails = _email_table_stub() + # email_records.organization_id is NOT NULL, so every existing row has + # a deterministic workspace under this codebase's established + # convention (services/email_import_service.py, project_graph): + # workspace-. There is no independent workspace + # registry to join against -- Email carries no FK to any + # account/mailbox table that itself has workspace_id (organization + # config lives in FK-less tenant_configs/caldav_accounts/webdav_accounts). + connection.execute( + sa.update(emails) + .where(emails.c.workspace_id.is_(None)) + .values(workspace_id=sa.func.concat("workspace-", emails.c.organization_id)) + ) + op.alter_column(_EMAIL_TABLE, "workspace_id", nullable=False) + + existing_indexes = {index["name"] for index in inspector.get_indexes(_EMAIL_TABLE)} + if _EMAIL_WORKSPACE_INDEX not in existing_indexes: + op.create_index( + _EMAIL_WORKSPACE_INDEX, + _EMAIL_TABLE, + ["workspace_id"], + if_not_exists=True, + ) + + existing_constraints = { + constraint["name"] + for constraint in inspector.get_unique_constraints(_EMAIL_TABLE) + } + unique_indexes = { + index["name"] + for index in inspector.get_indexes(_EMAIL_TABLE) + if index.get("unique") + } + # get_indexes() also reports the backing index of a unique constraint + # under the same name (PostgreSQL implements a unique constraint via a + # unique index), so each identity's constraint case must be checked -- + # and handled -- before its index case: DROP INDEX on a constraint's own + # backing index is rejected by PostgreSQL ("cannot drop index ... + # because constraint ... requires it"), which would abort this + # migration outright. + if _OLD_EMAIL_IDENTITY in existing_constraints: + op.drop_constraint(_OLD_EMAIL_IDENTITY, _EMAIL_TABLE, type_="unique") + elif _OLD_EMAIL_IDENTITY in unique_indexes: + op.drop_index(_OLD_EMAIL_IDENTITY, table_name=_EMAIL_TABLE) + + if _BOOTSTRAP_OLD_EMAIL_IDENTITY in existing_constraints: + op.drop_constraint( + _BOOTSTRAP_OLD_EMAIL_IDENTITY, _EMAIL_TABLE, type_="unique" + ) + elif _BOOTSTRAP_OLD_EMAIL_IDENTITY in unique_indexes: + op.drop_index(_BOOTSTRAP_OLD_EMAIL_IDENTITY, table_name=_EMAIL_TABLE) + + if _EMAIL_WORKSPACE_IDENTITY not in existing_constraints | unique_indexes: + op.create_unique_constraint( + _EMAIL_WORKSPACE_IDENTITY, + _EMAIL_TABLE, + ["user_id", "organization_id", "workspace_id", "message_id"], + ) + + +def downgrade() -> None: + connection = op.get_bind() + inspector = sa.inspect(connection) + if not inspector.has_table(_EMAIL_TABLE): + return + + emails = _email_table_stub() + duplicate_identity = connection.execute( + sa.select( + emails.c.user_id, + emails.c.organization_id, + emails.c.message_id, + ) + .group_by( + emails.c.user_id, + emails.c.organization_id, + emails.c.message_id, + ) + .having(sa.func.count() > 1) + .limit(1) + ).first() + if duplicate_identity is not None: + raise RuntimeError( + "Cannot downgrade email workspace identity while duplicate owner/message " + "rows exist across workspaces" + ) + + existing_indexes = {index["name"] for index in inspector.get_indexes(_EMAIL_TABLE)} + if _EMAIL_WORKSPACE_INDEX in existing_indexes: + op.drop_index(_EMAIL_WORKSPACE_INDEX, table_name=_EMAIL_TABLE, if_exists=True) + + existing_constraints = { + constraint["name"] + for constraint in inspector.get_unique_constraints(_EMAIL_TABLE) + } + unique_indexes = { + index["name"] + for index in inspector.get_indexes(_EMAIL_TABLE) + if index.get("unique") + } + if _EMAIL_WORKSPACE_IDENTITY in existing_constraints: + op.drop_constraint(_EMAIL_WORKSPACE_IDENTITY, _EMAIL_TABLE, type_="unique") + elif _EMAIL_WORKSPACE_IDENTITY in unique_indexes: + op.drop_index(_EMAIL_WORKSPACE_IDENTITY, table_name=_EMAIL_TABLE) + if _OLD_EMAIL_IDENTITY not in existing_constraints | unique_indexes: + op.create_unique_constraint( + _OLD_EMAIL_IDENTITY, + _EMAIL_TABLE, + ["user_id", "organization_id", "message_id"], + ) + + existing_columns = { + column["name"] for column in inspector.get_columns(_EMAIL_TABLE) + } + if "workspace_id" in existing_columns: + op.drop_column(_EMAIL_TABLE, "workspace_id") diff --git a/backend/alembic/versions/0021_calendar_correction_rationale.py b/backend/alembic/versions/0021_calendar_correction_rationale.py new file mode 100644 index 000000000..3d74e499f --- /dev/null +++ b/backend/alembic/versions/0021_calendar_correction_rationale.py @@ -0,0 +1,69 @@ +"""rename the calendar correction rationale column + +Revision ID: 0021_calendar_rationale +Revises: 0020_email_workspace_scope +Create Date: 2026-09-01 00:00:00.000000 +""" + +from alembic import context, op +import sqlalchemy as sa + +revision = "0021_calendar_rationale" +down_revision = "0020_email_workspace_scope" + +_CORRECTION_TABLE = "calendar_conflict_corrections" + + +def _is_offline_mode() -> bool: + """Detect SQL-only execution without requiring an EnvironmentContext proxy.""" + try: + return bool(context.is_offline_mode()) + except (AttributeError, NameError): + # Focused migration tests invoke upgrade/downgrade with an Operations + # proxy but no Alembic EnvironmentContext. That path is online and + # supplies either a real or deliberately mocked bind. + return False + + +def upgrade() -> None: + if _is_offline_mode(): + op.alter_column( + _CORRECTION_TABLE, + "rationale", + new_column_name="correction_rationale", + ) + return + + inspector = sa.inspect(op.get_bind()) + if not inspector.has_table(_CORRECTION_TABLE): + return + + columns = {column["name"] for column in inspector.get_columns(_CORRECTION_TABLE)} + if "rationale" in columns and "correction_rationale" not in columns: + op.alter_column( + _CORRECTION_TABLE, + "rationale", + new_column_name="correction_rationale", + ) + + +def downgrade() -> None: + if _is_offline_mode(): + op.alter_column( + _CORRECTION_TABLE, + "correction_rationale", + new_column_name="rationale", + ) + return + + inspector = sa.inspect(op.get_bind()) + if not inspector.has_table(_CORRECTION_TABLE): + return + + columns = {column["name"] for column in inspector.get_columns(_CORRECTION_TABLE)} + if "correction_rationale" in columns and "rationale" not in columns: + op.alter_column( + _CORRECTION_TABLE, + "correction_rationale", + new_column_name="rationale", + ) diff --git a/backend/alembic/versions/0022_noema_orchestrator_gateway.py b/backend/alembic/versions/0022_noema_orchestrator_gateway.py new file mode 100644 index 000000000..e84aaf111 --- /dev/null +++ b/backend/alembic/versions/0022_noema_orchestrator_gateway.py @@ -0,0 +1,58 @@ +"""add noema contextual-orchestrator gateway columns to tenant_configs + +Revision ID: 0022_noema_orchestrator_gateway +Revises: 0021_calendar_rationale +Create Date: 2026-09-02 00:00:00.000000 + +The general-purpose Noema workspace agent (``services/noema_agent.py``) must +route every chat-completion call through ``contextual-orchestrator`` -- the +org's routing/cost hub -- rather than a tenant's own direct LLM-provider key. +This migration adds the per-tenant gateway credential columns on +``tenant_configs``, mirroring the ``batch_orchestrator_*`` columns +(0012_llm_batch_orchestrator) already used for batch embedding routing +through the same orchestrator. All new config is resolved from the Fernet DB +at runtime, never from ``os.getenv``. +""" + +from alembic import op +import sqlalchemy as sa + +revision = "0022_noema_orchestrator_gateway" +down_revision = "0021_calendar_rationale" +branch_labels = None +depends_on = None + +_TENANT_TABLE = "tenant_configs" + + +def upgrade() -> None: + connection = op.get_bind() + inspector = sa.inspect(connection) + + if inspector.has_table(_TENANT_TABLE): + for column in _tenant_noema_gateway_columns(): + if not _has_column(inspector, _TENANT_TABLE, column.name): + op.add_column(_TENANT_TABLE, column) + + +def downgrade() -> None: + connection = op.get_bind() + inspector = sa.inspect(connection) + + if inspector.has_table(_TENANT_TABLE): + for column in reversed(_tenant_noema_gateway_columns()): + if _has_column(inspector, _TENANT_TABLE, column.name): + op.drop_column(_TENANT_TABLE, column.name) + + +def _tenant_noema_gateway_columns() -> list["sa.Column"]: + return [ + sa.Column("noema_orchestrator_base_url", sa.String(), nullable=True), + sa.Column("noema_orchestrator_token", sa.String(), nullable=True), + ] + + +def _has_column(inspector, table_name: str, column_name: str) -> bool: + return any( + column["name"] == column_name for column in inspector.get_columns(table_name) + ) diff --git a/backend/api/auth.py b/backend/api/auth.py index bd188351c..9eba038f9 100644 --- a/backend/api/auth.py +++ b/backend/api/auth.py @@ -517,12 +517,16 @@ def _auth_context_from_session_payload( organization_id = _optional_string_claim(payload, "org") if organization_id is None: raise _authentication_error() + # Workspace membership is the independently signed claim produced by the + # verified session authority. Workspace identifiers are opaque and are not + # derived from organization display/identity values. + workspace_id = _required_string_claim(payload, "workspace") return AuthContext( user_id=_required_string_claim(payload, "sub"), role=role, organization_id=organization_id, group_ids=_tuple_string_claim(payload, "groups"), - workspace_id=_required_string_claim(payload, "workspace"), + workspace_id=workspace_id, session_verifier=cast(SessionVerifier, session_verifier), ) diff --git a/backend/api/calendar_conflicts.py b/backend/api/calendar_conflicts.py index d61293f82..e3b5770ba 100644 --- a/backend/api/calendar_conflicts.py +++ b/backend/api/calendar_conflicts.py @@ -2,21 +2,46 @@ from __future__ import annotations +import datetime from typing import Literal, Self -from fastapi import APIRouter +from fastapi import APIRouter, Depends, HTTPException, Query from fastapi.exceptions import RequestValidationError from fastapi.responses import JSONResponse from fastapi.routing import APIRoute -from pydantic import AwareDatetime, BaseModel, ConfigDict, Field, model_validator +from pydantic import ( + AwareDatetime, + BaseModel, + ConfigDict, + Field, + ValidationError, + model_validator, +) +from pydantic_core import PydanticCustomError +from sqlalchemy.ext.asyncio import AsyncSession from starlette.requests import Request from starlette.responses import Response +from api.auth import AuthContext, get_auth_context +from db.models import CalendarConflictJudgment +from db.session import get_db from services.calendar_conflict_ics import ( parse_existing_calendar_commitments_from_ics, parse_proposed_calendar_commitment_from_ics, ) +from services.calendar_conflict_judgment_service import ( + CORRECTION_INCOHERENT_ERROR_CODE, + CalendarConflictCorrectionIncoherentError, + CalendarConflictJudgmentNotFoundError, + CalendarConflictUnsupportedValueError, + apply_correction, + create_judgment, + get_judgment, + list_judgments, + validate_correction_coherence, +) from services.calendar_conflict_policy import ( + MAX_EXISTING_COMMITMENTS, CalendarCommitment, CalendarConflictDecision, CalendarPolicyValidationError, @@ -24,19 +49,34 @@ evaluate_calendar_conflicts, ) -MAX_EXISTING_COMMITMENTS = 500 MAX_PROPOSED_ICS_CHARS = 65_536 MAX_EXISTING_ICS_CHARS = 262_144 POLICY_VALIDATION_HTTP_STATUS = 422 +STORED_EVIDENCE_HTTP_STATUS = 500 +# Matches api.ontology's SOURCE_IDENTIFIER_PATTERN: these ids are opaque +# RFC 5322-ish message/thread identifiers, not free text. +SOURCE_IDENTIFIER_PATTERN = r"^[\w\.\-\+@_<>]+$" REQUEST_INVALID_ERROR_CODE = "calendar_request_invalid" +PROPOSED_SOURCE_MISSING_ERROR_CODE = "calendar_proposed_source_missing" PROPOSED_SOURCE_REQUIRED_DETAIL = "Provide exactly one of proposed or proposed_ics" +STORED_EVIDENCE_ERROR_CODE = "calendar_conflict_stored_evidence_corrupt" +STORED_EVIDENCE_ERROR_DETAIL = "Stored calendar conflict evidence is corrupt" + + +class CalendarConflictStoredEvidenceError(RuntimeError): + """Signal corrupt persisted conflict evidence without leaking parser internals.""" + + error_code = STORED_EVIDENCE_ERROR_CODE + + def __init__(self) -> None: + super().__init__(STORED_EVIDENCE_ERROR_DETAIL) class CalendarConflictAPIRoute(APIRoute): - """Keep request-model failures on the stable calendar conflict error envelope.""" + """Keep request/model failures on the stable calendar conflict error envelope.""" def get_route_handler(self): - """Wrap the FastAPI handler so validation uses CalendarConflictErrorResponse.""" + """Wrap the FastAPI handler so boundary failures use stable error codes.""" original_route_handler = super().get_route_handler() async def calendar_conflict_route_handler(request: Request) -> Response: @@ -44,6 +84,15 @@ async def calendar_conflict_route_handler(request: Request) -> Response: return await original_route_handler(request) except RequestValidationError as exc: return _request_validation_error_response(exc) + except CalendarConflictStoredEvidenceError as exc: + error = CalendarConflictErrorResponse( + error_code=exc.error_code, + detail=str(exc), + ) + return JSONResponse( + status_code=STORED_EVIDENCE_HTTP_STATUS, + content=error.model_dump(), + ) return calendar_conflict_route_handler @@ -85,7 +134,9 @@ def require_exactly_one_proposed_source(self) -> Self: has_proposed = self.proposed is not None has_proposed_ics = self.proposed_ics is not None if has_proposed == has_proposed_ics: - raise ValueError(PROPOSED_SOURCE_REQUIRED_DETAIL) + raise PydanticCustomError( + PROPOSED_SOURCE_MISSING_ERROR_CODE, PROPOSED_SOURCE_REQUIRED_DETAIL + ) return self @@ -109,7 +160,7 @@ class CalendarConflictResponse(BaseModel): class CalendarConflictErrorResponse(BaseModel): - """Stable machine code plus safe explanation for policy validation failures.""" + """Stable machine code plus safe explanation for calendar boundary failures.""" error_code: str detail: str @@ -117,12 +168,17 @@ class CalendarConflictErrorResponse(BaseModel): def _request_validation_error_response(exc: RequestValidationError) -> JSONResponse: """Map FastAPI request validation onto the existing error_code envelope.""" - messages = [str(error.get("msg", "")) for error in exc.errors()] - if any(PROPOSED_SOURCE_REQUIRED_DETAIL in message for message in messages): + error_types = {str(error.get("type", "")) for error in exc.errors()} + if PROPOSED_SOURCE_MISSING_ERROR_CODE in error_types: error = CalendarConflictErrorResponse( - error_code="calendar_proposed_source_missing", + error_code=PROPOSED_SOURCE_MISSING_ERROR_CODE, detail=PROPOSED_SOURCE_REQUIRED_DETAIL, ) + elif CORRECTION_INCOHERENT_ERROR_CODE in error_types: + error = CalendarConflictErrorResponse( + error_code=CORRECTION_INCOHERENT_ERROR_CODE, + detail="status_code and decision_code disagree about whether the decision changed", + ) else: error = CalendarConflictErrorResponse( error_code=REQUEST_INVALID_ERROR_CODE, @@ -163,6 +219,31 @@ def _to_response(decision: CalendarConflictDecision) -> CalendarConflictResponse ) +def _resolve_commitments( + request: CalendarConflictRequest, +) -> tuple[CalendarCommitment, list[CalendarCommitment]]: + """Parse the proposed/existing commitments, raising on any policy violation.""" + proposed_payload = request.proposed + if request.proposed_ics is not None: + proposed = parse_proposed_calendar_commitment_from_ics(request.proposed_ics) + elif proposed_payload is not None: + proposed = _to_commitment(proposed_payload) + else: + raise CalendarPolicyValidationError( + "calendar_proposed_source_missing", + PROPOSED_SOURCE_REQUIRED_DETAIL, + ) + existing = [_to_commitment(item) for item in request.existing] + if request.existing_ics is not None: + existing.extend(parse_existing_calendar_commitments_from_ics(request.existing_ics)) + if len(existing) > MAX_EXISTING_COMMITMENTS: + raise CalendarPolicyValidationError( + "calendar_existing_batch_exceeded", + "existing evidence exceeds the bounded commitment batch", + ) + return proposed, existing + + @router.post( "/evaluate", response_model=CalendarConflictResponse, @@ -173,24 +254,7 @@ def evaluate_calendar_conflict_request( ) -> CalendarConflictResponse | JSONResponse: """Evaluate double-booking risk without mutating any provider calendar.""" try: - proposed_payload = request.proposed - if request.proposed_ics is not None: - proposed = parse_proposed_calendar_commitment_from_ics(request.proposed_ics) - elif proposed_payload is not None: - proposed = _to_commitment(proposed_payload) - else: - raise CalendarPolicyValidationError( - "calendar_proposed_source_missing", - PROPOSED_SOURCE_REQUIRED_DETAIL, - ) - existing = [_to_commitment(item) for item in request.existing] - if request.existing_ics is not None: - existing.extend(parse_existing_calendar_commitments_from_ics(request.existing_ics)) - if len(existing) > MAX_EXISTING_COMMITMENTS: - raise CalendarPolicyValidationError( - "calendar_existing_batch_exceeded", - "existing evidence exceeds the bounded commitment batch", - ) + proposed, existing = _resolve_commitments(request) except CalendarPolicyValidationError as exc: error = CalendarConflictErrorResponse( error_code=exc.error_code, @@ -202,3 +266,230 @@ def evaluate_calendar_conflict_request( ) return _to_response(evaluate_calendar_conflicts(proposed, existing)) + + +class CalendarConflictJudgeRequest(CalendarConflictRequest): + """An `/evaluate` request whose decision should be persisted as a judgment.""" + + source_thread_id: str | None = Field( + default=None, max_length=512, pattern=SOURCE_IDENTIFIER_PATTERN + ) + source_message_id: str | None = Field( + default=None, max_length=512, pattern=SOURCE_IDENTIFIER_PATTERN + ) + + +class CalendarConflictJudgmentResponse(BaseModel): + """A persisted conflict decision, correctable by a human reviewer.""" + + judgment_uid: str + proposed_commitment_id: str + source_thread_id: str | None + source_message_id: str | None + decision_code: Literal["available", "blocked", "review_required"] + reason_code: str + conflicts: list[CalendarConflictEvidence] + recommended_action: str + policy_version: str + status_code: Literal["proposed", "confirmed", "overridden", "dismissed"] + created_at: datetime.datetime + updated_at: datetime.datetime + + +class CalendarConflictCorrectionRequest(BaseModel): + """A human correction/confirmation applied to one persisted judgment.""" + + model_config = ConfigDict(extra="forbid") + + correction_action: str = Field(min_length=1, max_length=64) + decision_code: Literal["available", "blocked", "review_required"] | None = None + status_code: Literal["confirmed", "overridden", "dismissed"] + rationale: str | None = Field(default=None, max_length=2000) + + @model_validator(mode="after") + def require_coherent_status_and_decision(self) -> Self: + """Reject a status_code/decision_code pair the service would also reject.""" + try: + validate_correction_coherence( + status_code=self.status_code, decision_code=self.decision_code + ) + except CalendarConflictCorrectionIncoherentError as exc: + raise PydanticCustomError(exc.error_code, str(exc)) from exc + return self + + +class CalendarConflictCorrectionResponse(BaseModel): + """The recorded before/after audit trail for one correction.""" + + correction_uid: str + judgment_uid: str + correction_action: str + before_json: dict[str, object] + after_json: dict[str, object] + rationale: str | None + actor_user_id: str + created_at: datetime.datetime + + +def _stored_conflict_evidence(raw_conflicts: object) -> list[CalendarConflictEvidence]: + """Validate persisted JSON before exposing it through the public API boundary.""" + if not isinstance(raw_conflicts, list): + raise CalendarConflictStoredEvidenceError() + try: + return [CalendarConflictEvidence.model_validate(item) for item in raw_conflicts] + except (ValidationError, TypeError, ValueError) as exc: + raise CalendarConflictStoredEvidenceError() from exc + + +def _judgment_response( + judgment: CalendarConflictJudgment, +) -> CalendarConflictJudgmentResponse: + """Serialize one persisted judgment after validating stored evidence.""" + return CalendarConflictJudgmentResponse( + judgment_uid=judgment.judgment_uid, + proposed_commitment_id=judgment.proposed_commitment_id, + source_thread_id=judgment.source_thread_id, + source_message_id=judgment.source_message_id, + decision_code=judgment.decision_code, + reason_code=judgment.reason_code, + conflicts=_stored_conflict_evidence(judgment.conflicts_json), + recommended_action=judgment.recommended_action, + policy_version=judgment.policy_version, + status_code=judgment.status_code, + created_at=judgment.created_at, + updated_at=judgment.updated_at, + ) + + +@router.post( + "/judgments", + response_model=CalendarConflictJudgmentResponse, + responses={POLICY_VALIDATION_HTTP_STATUS: {"model": CalendarConflictErrorResponse}}, +) +async def create_calendar_conflict_judgment( + request: CalendarConflictJudgeRequest, + auth_ctx: AuthContext = Depends(get_auth_context), + db: AsyncSession = Depends(get_db), +) -> CalendarConflictJudgmentResponse | JSONResponse: + """Evaluate a conflict decision and persist it as a correctable judgment.""" + try: + proposed, existing = _resolve_commitments(request) + except CalendarPolicyValidationError as exc: + error = CalendarConflictErrorResponse( + error_code=exc.error_code, + detail=str(exc), + ) + return JSONResponse( + status_code=POLICY_VALIDATION_HTTP_STATUS, + content=error.model_dump(), + ) + + decision = evaluate_calendar_conflicts(proposed, existing) + judgment = await create_judgment( + db, + user_id=auth_ctx.user_id, + organization_id=auth_ctx.organization_id, + workspace_id=auth_ctx.workspace_id, + proposed_commitment_id=proposed.commitment_id, + source_thread_id=request.source_thread_id, + source_message_id=request.source_message_id, + decision=decision, + ) + await db.commit() + return _judgment_response(judgment) + + +@router.get( + "/judgments", + response_model=list[CalendarConflictJudgmentResponse], +) +async def list_calendar_conflict_judgments( + source_thread_id: str | None = Query( + default=None, max_length=512, pattern=SOURCE_IDENTIFIER_PATTERN + ), + auth_ctx: AuthContext = Depends(get_auth_context), + db: AsyncSession = Depends(get_db), +) -> list[CalendarConflictJudgmentResponse]: + """List persisted conflict judgments in the caller's own scope.""" + judgments = await list_judgments( + db, + user_id=auth_ctx.user_id, + organization_id=auth_ctx.organization_id, + workspace_id=auth_ctx.workspace_id, + source_thread_id=source_thread_id, + ) + return [_judgment_response(judgment) for judgment in judgments] + + +@router.get( + "/judgments/{judgment_uid}", + response_model=CalendarConflictJudgmentResponse, +) +async def get_calendar_conflict_judgment( + judgment_uid: str, + auth_ctx: AuthContext = Depends(get_auth_context), + db: AsyncSession = Depends(get_db), +) -> CalendarConflictJudgmentResponse: + """Fetch one judgment by uid, even if it has fallen out of the list bound.""" + try: + judgment = await get_judgment( + db, + judgment_uid=judgment_uid, + user_id=auth_ctx.user_id, + organization_id=auth_ctx.organization_id, + workspace_id=auth_ctx.workspace_id, + ) + except CalendarConflictJudgmentNotFoundError as exc: + raise HTTPException(status_code=404, detail=str(exc)) from exc + return _judgment_response(judgment) + + +@router.post( + "/judgments/{judgment_uid}/corrections", + response_model=CalendarConflictCorrectionResponse, +) +async def correct_calendar_conflict_judgment( + judgment_uid: str, + request: CalendarConflictCorrectionRequest, + auth_ctx: AuthContext = Depends(get_auth_context), + db: AsyncSession = Depends(get_db), +) -> CalendarConflictCorrectionResponse | JSONResponse: + """Record a human override/confirmation of one persisted judgment.""" + try: + correction = await apply_correction( + db, + judgment_uid=judgment_uid, + user_id=auth_ctx.user_id, + organization_id=auth_ctx.organization_id, + workspace_id=auth_ctx.workspace_id, + actor_user_id=auth_ctx.user_id, + correction_action=request.correction_action, + decision_code=request.decision_code, + status_code=request.status_code, + rationale=request.rationale, + ) + await db.commit() + except CalendarConflictJudgmentNotFoundError as exc: + raise HTTPException(status_code=404, detail=str(exc)) from exc + except CalendarConflictUnsupportedValueError as exc: + error = CalendarConflictErrorResponse(error_code=exc.error_code, detail=str(exc)) + return JSONResponse( + status_code=POLICY_VALIDATION_HTTP_STATUS, + content=error.model_dump(), + ) + except CalendarConflictCorrectionIncoherentError as exc: + error = CalendarConflictErrorResponse(error_code=exc.error_code, detail=str(exc)) + return JSONResponse( + status_code=POLICY_VALIDATION_HTTP_STATUS, + content=error.model_dump(), + ) + return CalendarConflictCorrectionResponse( + correction_uid=correction.correction_uid, + judgment_uid=judgment_uid, + correction_action=correction.correction_action, + before_json=correction.before_json, + after_json=correction.after_json, + rationale=correction.correction_rationale, + actor_user_id=correction.actor_user_id, + created_at=correction.created_at, + ) diff --git a/backend/api/data.py b/backend/api/data.py index dccd85890..bb641b8a3 100644 --- a/backend/api/data.py +++ b/backend/api/data.py @@ -30,7 +30,10 @@ WebdavAccount, ) from db.session import get_db -from services.attachment_parser import get_attachment_parser_manifest +from services.attachment_parser import ( + CONTENT_TYPE_MISMATCH_QUARANTINED_STATUS, + get_attachment_parser_manifest, +) from services.newsdom_pdf_recognition import ( PDF_DOM_RECOGNITION_PENDING_STATUS, ) @@ -45,6 +48,10 @@ # would let a caller stash a pending document the configured sidecar will always # reject while the base64 copy inflates the database. _MAX_PDF_DOM_UPLOAD_BYTES = 20 * 1024 * 1024 +# The only status a reparse-intent may act on -- an attachment that parsed +# cleanly, is still pending, or already failed for an unrelated reason has +# nothing for a reparse pass to usefully re-evaluate. +ATTACHMENT_REPARSE_PENDING_STATUS = "reparse_pending" ATTACHMENT_PARSE_BREAKDOWN_EVIDENCE_SOURCE = ( "email_attachments.content_type, " "email_attachments.parse_content_type, " @@ -116,7 +123,7 @@ "attachment_repository", "document_repository", ] -EmailScopeFilter = tuple[ColumnElement[bool], ColumnElement[bool]] +EmailScopeFilter = tuple[ColumnElement[bool], ColumnElement[bool], ColumnElement[bool]] AttachmentAssetRow = Row[tuple[Attachment, Email]] @@ -289,6 +296,19 @@ class DataAttachmentParseBreakdown(BaseModel): provider_write_executed: bool +class DataAttachmentActionResponse(BaseModel): + attachment_uid: str + filename: str + content_type: str + parse_content_type: str + parse_status: str + parse_error_code: str | None + provider_write_executed: bool + provenance: Literal["server-authoritative"] + audit_event: str + message: str + + class DataContentGraphBreakdown(BaseModel): source_kind: str segment_kind: str @@ -2463,15 +2483,20 @@ def _owner_scope_statement(model, auth_context: AuthContext): def _email_scope_filter(auth_context: AuthContext) -> EmailScopeFilter: + # workspace_id is the finest-grained scope and is applied unconditionally, + # mirroring _owner_scope_statement's pattern for every other + # workspace_id-bearing model: even an organization admin is scoped to + # their own workspace, not every workspace in the organization. + workspace_filter = Email.workspace_id == auth_context.workspace_id if _can_read_org_scope(auth_context): organization_filter = Email.organization_id == auth_context.organization_id - return (organization_filter, organization_filter) + return (workspace_filter, organization_filter, organization_filter) organization_filter = ( Email.organization_id == auth_context.organization_id if auth_context.organization_id is not None else Email.organization_id.is_(None) ) - return (Email.user_id == auth_context.user_id, organization_filter) + return (workspace_filter, Email.user_id == auth_context.user_id, organization_filter) async def _scoped_rows(db: AsyncSession, statement): @@ -2501,6 +2526,57 @@ async def _get_workspace_document( return document +async def _get_scoped_attachment( + db: AsyncSession, + auth_context: AuthContext, + attachment_uid: str, + *, + lock: bool = False, +) -> Attachment: + # Attachment carries no workspace_id/user_id/organization_id of its own -- + # it is scoped through its parent Email, same as every other + # attachment-facing query in this file. + email_scope = _email_scope_filter(auth_context) + statement = ( + select(Attachment) + .join(Email) + .where(Attachment.attachment_uid == attachment_uid, *email_scope) + ) + if lock: + # Concurrent read-then-write status transitions (e.g. reparse-intent) + # must not race unlocked: without this, a stale request that already + # read an earlier status could commit after a concurrent request (or + # the reparse worker) has moved the row further along, silently + # clobbering that newer result. Scoped to Attachment only so this + # doesn't also lock the joined Email row. + statement = statement.with_for_update(of=Attachment) + result = await db.execute(statement) + attachment = result.scalar_one_or_none() + if attachment is None: + raise HTTPException(status_code=404, detail="Attachment not found") + return attachment + + +def _attachment_response( + attachment: Attachment, + *, + audit_event: str, + message: str, +) -> DataAttachmentActionResponse: + return DataAttachmentActionResponse( + attachment_uid=attachment.attachment_uid, + filename=attachment.filename, + content_type=attachment.content_type, + parse_content_type=attachment.parse_content_type, + parse_status=attachment.parse_status, + parse_error_code=attachment.parse_error_code, + provider_write_executed=False, + provenance="server-authoritative", + audit_event=audit_event, + message=message, + ) + + def _status_from_ratio(total_count: int, ready_count: int) -> SurfaceStatus: if total_count <= 0: return "pending" @@ -3279,6 +3355,37 @@ async def create_document_pdf_dom_recognition_intent( ) +@router.post( + "/attachments/{attachment_uid}/reparse-intent", + response_model=DataAttachmentActionResponse, +) +async def create_attachment_reparse_intent( + attachment_uid: str, + auth_context: AuthContext = Depends(get_auth_context), + db: AsyncSession = Depends(get_db), +) -> DataAttachmentActionResponse: + attachment = await _get_scoped_attachment( + db, auth_context, attachment_uid, lock=True + ) + if attachment.parse_status != CONTENT_TYPE_MISMATCH_QUARANTINED_STATUS: + raise HTTPException( + status_code=422, + detail="Reparse intent is only available for quarantined attachments.", + ) + attachment.parse_status = ATTACHMENT_REPARSE_PENDING_STATUS + attachment.parse_error_code = None + await db.commit() + await db.refresh(attachment) + return _attachment_response( + attachment, + audit_event="data.attachment.reparse_intent", + message=( + "Reparse intent recorded; a future worker pass re-evaluates this " + "attachment. No synchronous re-parse executed." + ), + ) + + @router.post( "/documents/pdf-dom-recognition", response_model=DataDocumentActionResponse, diff --git a/backend/api/emails.py b/backend/api/emails.py index 2b0a9dbd6..74df632d0 100644 --- a/backend/api/emails.py +++ b/backend/api/emails.py @@ -280,7 +280,9 @@ async def get_emails( user_addresses = configured_email_addresses(tenant_config) is_sent_folder = folder == "sent" owner_filters = Email.owner_filters( - auth_context.user_id, auth_context.organization_id + auth_context.user_id, + auth_context.organization_id, + auth_context.workspace_id, ) # Thread winnowing happens in SQL: one newest head row per thread via a @@ -384,7 +386,10 @@ async def get_pending_replies( ): # Ensure auth context validates the request payload and scopes access pending_emails = await check_missing_replies( - db, auth_context.user_id, auth_context.organization_id + db, + auth_context.user_id, + auth_context.organization_id, + auth_context.workspace_id, ) items = [] for email in pending_emails[:limit]: @@ -446,7 +451,11 @@ async def _fetch_existing_emails_for_candidates( result = await db.execute( select(Email).where( - *Email.owner_filters(auth_context.user_id, auth_context.organization_id), + *Email.owner_filters( + auth_context.user_id, + auth_context.organization_id, + auth_context.workspace_id, + ), or_(*predicates), ) ) @@ -610,6 +619,7 @@ async def import_email_files( uploads=uploads, user_id=auth_context.user_id, organization_id=auth_context.organization_id, + workspace_id=auth_context.workspace_id, embedding_provider=embedding_provider, ) except EmailImportQuotaExceeded as exc: @@ -647,7 +657,11 @@ async def get_email( result = await db.execute( select(Email).where( Email.id == email_id, - *Email.owner_filters(auth_context.user_id, auth_context.organization_id), + *Email.owner_filters( + auth_context.user_id, + auth_context.organization_id, + auth_context.workspace_id, + ), ) ) email = result.scalar_one_or_none() @@ -669,7 +683,11 @@ async def get_email_thread( result = await db.execute( select(Email) .where( - *Email.owner_filters(auth_context.user_id, auth_context.organization_id), + *Email.owner_filters( + auth_context.user_id, + auth_context.organization_id, + auth_context.workspace_id, + ), or_( Email.thread_id.in_(lookup_values), Email.message_id.in_(lookup_values) ), diff --git a/backend/api/network.py b/backend/api/network.py index 07d389e9c..e5fb3b6f0 100644 --- a/backend/api/network.py +++ b/backend/api/network.py @@ -46,15 +46,15 @@ async def get_network_graph( if user_id and user_id != current_user: raise HTTPException(status_code=403, detail="Not authorized") target_user_id = user_id or current_user - organization_filter = ( - Email.organization_id == auth_context.organization_id - if auth_context.organization_id is not None - else Email.organization_id.is_(None) - ) - result = await db.execute( select(Email.sender, Email.recipients) - .where(Email.user_id == target_user_id, organization_filter) + .where( + *Email.owner_filters( + target_user_id, + auth_context.organization_id, + auth_context.workspace_id, + ) + ) .limit(limit) ) rows = result.fetchall() diff --git a/backend/api/ontology.py b/backend/api/ontology.py index 738679619..314df0b55 100644 --- a/backend/api/ontology.py +++ b/backend/api/ontology.py @@ -179,7 +179,9 @@ async def capture_relationship_from_source( ): result = await db.execute( select(Email).where( - *Email.owner_filters(auth_ctx.user_id, auth_ctx.organization_id), + *Email.owner_filters( + auth_ctx.user_id, auth_ctx.organization_id, auth_ctx.workspace_id + ), Email.message_id == req.source_message_id, ) ) diff --git a/backend/api/search.py b/backend/api/search.py index 5a81659f2..2044dd6ad 100644 --- a/backend/api/search.py +++ b/backend/api/search.py @@ -112,7 +112,10 @@ def thread_group_key(): def build_reply_counts_stmt( - thread_keys: list[str], user_id: str, organization_id: str | None + thread_keys: list[str], + user_id: str, + organization_id: str | None, + workspace_id: str, ) -> Select: group_key = thread_group_key() return ( @@ -121,7 +124,7 @@ def build_reply_counts_stmt( func.count(Email.id).label("reply_count"), ) .select_from(Email) - .where(*Email.owner_filters(user_id, organization_id)) + .where(*Email.owner_filters(user_id, organization_id, workspace_id)) .where(group_key.in_(thread_keys)) .group_by(group_key) ) @@ -229,9 +232,7 @@ def merge_candidate_rows( one_based_rank=zero_based_position + 1, result_kind=row.result_kind, matched_text=row.matched_text, - word_similarity_score=getattr( - row, "word_similarity_score", None - ), + word_similarity_score=getattr(row, "word_similarity_score", None), cosine_distance=getattr(row, "cosine_distance", None), fusion_settings=fusion_settings, ) @@ -296,9 +297,7 @@ async def _resolve_query_embedding( organization_id=organization_id, ) if runtime_provider is None: - logger.info( - "No LLM provider configured; hybrid search running lexical-only" - ) + logger.info("No LLM provider configured; hybrid search running lexical-only") return None try: embeddings = await generate_embeddings( @@ -430,7 +429,7 @@ async def hybrid_search( ) owner_filters = Email.owner_filters( - target_user_id, auth_context.organization_id + target_user_id, auth_context.organization_id, auth_context.workspace_id ) channel_statements = _build_channel_statements( normalized_query, query_embedding, owner_filters @@ -453,11 +452,11 @@ async def hybrid_search( candidate_thread_keys, target_user_id, auth_context.organization_id, + auth_context.workspace_id, ) ) reply_counts_by_thread_key = { - row.thread_key: row.reply_count - for row in reply_counts_result.all() + row.thread_key: row.reply_count for row in reply_counts_result.all() } search_results = build_search_result_items( @@ -513,7 +512,9 @@ async def grounded_answer( fusion_settings = resolve_fusion_settings() owner_filters = Email.owner_filters( - auth_context.user_id, auth_context.organization_id + auth_context.user_id, + auth_context.organization_id, + auth_context.workspace_id, ) channel_statements = _build_channel_statements( normalized_query, query_embedding, owner_filters diff --git a/backend/api/tasks.py b/backend/api/tasks.py index ad36abeaa..7ef491da2 100644 --- a/backend/api/tasks.py +++ b/backend/api/tasks.py @@ -3,7 +3,7 @@ from fastapi import APIRouter, Depends, HTTPException from pydantic import BaseModel, ConfigDict, Field -from sqlalchemy import and_, select +from sqlalchemy import and_, or_, select from sqlalchemy.ext.asyncio import AsyncSession from api.auth import AuthContext, get_auth_context @@ -154,16 +154,17 @@ async def create_reply_sla_escalations( db, user_id=auth_context.user_id, organization_id=auth_context.organization_id, + workspace_id=auth_context.workspace_id, overdue_hours=request.overdue_hours, limit=request.limit, ) - except ReplySlaTaskConflict: + except ReplySlaTaskConflict as conflict_error: raise HTTPException( status_code=409, detail={ - "error_code": "reply_sla_task_conflict", + "error_code": conflict_error.error_code, "message": "Overdue reply follow-up task conflict", - } + }, ) from None return _reply_sla_response(escalation_result) @@ -177,14 +178,20 @@ def _build_task_query(auth_context: AuthContext): TicketTask.related_email_id == Email.id, Email.user_id == auth_context.user_id, Email.organization_id == auth_context.organization_id, + Email.workspace_id == auth_context.workspace_id, ), ) .where( TicketTask.user_id == auth_context.user_id, TicketTask.organization_id == auth_context.organization_id, + or_( + TicketTask.related_email_id.is_(None), + Email.id.is_not(None), + ), ) ) + @router.get("", response_model=list[TicketTaskResponse]) async def list_ticket_tasks( db: AsyncSession = Depends(get_db), @@ -248,6 +255,7 @@ async def _fetch_source_email( Email.message_id == request.source_email_id, Email.user_id == auth_context.user_id, Email.organization_id == auth_context.organization_id, + Email.workspace_id == auth_context.workspace_id, ) ) email = email_result.scalar_one_or_none() diff --git a/backend/api/tenant_config.py b/backend/api/tenant_config.py index 65b18102f..790fbfd4c 100644 --- a/backend/api/tenant_config.py +++ b/backend/api/tenant_config.py @@ -66,6 +66,8 @@ class TenantConfigCreate(BaseModel): openai_api_key: Optional[str] = None google_client_id: Optional[str] = None google_client_secret: Optional[str] = None + noema_orchestrator_base_url: Optional[str] = None + noema_orchestrator_token: Optional[str] = None class TenantConfigResponse(BaseModel): @@ -88,6 +90,8 @@ class TenantConfigResponse(BaseModel): openai_api_key: Optional[str] = None google_client_id: Optional[str] = None google_client_secret: Optional[str] = None + noema_orchestrator_base_url: Optional[str] = None + noema_orchestrator_token: Optional[str] = None model_config = ConfigDict(from_attributes=True) @@ -99,6 +103,7 @@ class TenantConfigResponse(BaseModel): "oauth_client_secret", "openai_api_key", "google_client_secret", + "noema_orchestrator_token", } MAILBOX_MANAGE_FORBIDDEN = ( diff --git a/backend/db/models.py b/backend/db/models.py index 98e17eef2..d3466263f 100644 --- a/backend/db/models.py +++ b/backend/db/models.py @@ -137,7 +137,9 @@ class SecurityAuditEvent(Base): ) actor_user_id: Mapped[str] = mapped_column(String, index=True, nullable=False) actor_role: Mapped[str] = mapped_column(String, index=True, nullable=False) - organization_id: Mapped[str | None] = mapped_column(String, index=True, nullable=True) + organization_id: Mapped[str | None] = mapped_column( + String, index=True, nullable=True + ) workspace_id: Mapped[str] = mapped_column(String, index=True, nullable=False) event_action: Mapped[str] = mapped_column(String, index=True, nullable=False) resource_type: Mapped[str] = mapped_column(String, index=True, nullable=False) @@ -164,6 +166,7 @@ class SecurityAuditEvent(Base): ), ) + class LLMProvider(Base): __tablename__ = "llm_providers" __table_args__ = ( @@ -266,9 +269,7 @@ class ScopeweavePromotionLink(Base): object_uid: Mapped[str] = mapped_column(String, index=True, nullable=False) object_type: Mapped[str] = mapped_column(String, nullable=False) scopeweave_work_item_id: Mapped[str] = mapped_column(String, nullable=False) - scopeweave_work_item_url: Mapped[str | None] = mapped_column( - String, nullable=True - ) + scopeweave_work_item_url: Mapped[str | None] = mapped_column(String, nullable=True) promoted_confidence: Mapped[float] = mapped_column(Float, nullable=False) citation_count: Mapped[int] = mapped_column(Integer, nullable=False, default=0) promoted_by_user_id: Mapped[str] = mapped_column(String, nullable=False) @@ -620,7 +621,6 @@ class PromptTemplate(Base): ), ) - id: Mapped[int] = mapped_column(primary_key=True) prompt_uid: Mapped[str] = mapped_column( String, @@ -681,9 +681,7 @@ class WorkflowDefinition(Base): steps_json: Mapped[list[dict[str, object]]] = mapped_column( JSON, default=list, nullable=False ) - state_code: Mapped[str] = mapped_column( - String, default="draft", nullable=False - ) + state_code: Mapped[str] = mapped_column(String, default="draft", nullable=False) created_at: Mapped[datetime.datetime] = mapped_column( DateTime(timezone=True), default=lambda: datetime.datetime.now(datetime.timezone.utc), @@ -737,9 +735,7 @@ class AgentRunRecord(Base): organization_id: Mapped[str] = mapped_column(String, nullable=False) workspace_id: Mapped[str] = mapped_column(String, nullable=False) user_id: Mapped[str] = mapped_column(String, nullable=False) - status_code: Mapped[str] = mapped_column( - String, default="pending", nullable=False - ) + status_code: Mapped[str] = mapped_column(String, default="pending", nullable=False) started_at: Mapped[datetime.datetime] = mapped_column( DateTime(timezone=True), default=lambda: datetime.datetime.now(datetime.timezone.utc), @@ -762,8 +758,9 @@ class Email(Base): UniqueConstraint( "user_id", "organization_id", + "workspace_id", "message_id", - name="uq_emails_owner_message_id", + name="uq_emails_workspace_message", ), Index( "ix_email_records_owner_date", @@ -774,26 +771,35 @@ class Email(Base): ) @classmethod - def owner_filters(cls, user_id: str, organization_id: str | None): + def owner_filters( + cls, user_id: str, organization_id: str | None, workspace_id: str + ): organization_filter = ( cls.organization_id == organization_id if organization_id is not None else cls.organization_id.is_(None) ) - return (cls.user_id == user_id, organization_filter) + return ( + cls.user_id == user_id, + organization_filter, + cls.workspace_id == workspace_id, + ) id: Mapped[int] = mapped_column(primary_key=True) # Owner scope columns are intentionally plaintext and indexed: signed-session # ABAC filters must be enforced by SQL before email rows are returned. user_id: Mapped[str] = mapped_column(String, index=True, nullable=False) organization_id: Mapped[str] = mapped_column(String, index=True, nullable=False) + # Finest-grained scope: api/data.py's _email_scope_filter enforces this + # unconditionally (mirroring _owner_scope_statement's pattern for every + # other workspace_id-bearing model), so a same-user/same-organization + # session in a different workspace cannot read or mutate these rows. + workspace_id: Mapped[str] = mapped_column(String, index=True, nullable=False) message_id: Mapped[str] = mapped_column(String, index=True) thread_id: Mapped[str | None] = mapped_column( String, index=True, nullable=True ) # O3: email threading support - fingerprint: Mapped[str | None] = mapped_column( - String, index=True, nullable=True - ) + fingerprint: Mapped[str | None] = mapped_column(String, index=True, nullable=True) sender: Mapped[str] = mapped_column(String) reply_to: Mapped[str | None] = mapped_column(String, nullable=True) recipients: Mapped[str | None] = mapped_column(String, nullable=True) @@ -877,8 +883,16 @@ class TicketTask(Base): class Attachment(Base): __tablename__ = "email_attachments" + __table_args__ = ( + UniqueConstraint("attachment_uid", name="uq_email_attachments_uid"), + ) id: Mapped[int] = mapped_column(primary_key=True) + attachment_uid: Mapped[str] = mapped_column( + String(96), + default=lambda: f"attachment_{uuid.uuid4().hex}", + nullable=False, + ) email_id: Mapped[int] = mapped_column(ForeignKey("email_records.id")) filename: Mapped[str] = mapped_column(String) content: Mapped[str] = mapped_column(Text) @@ -894,9 +908,7 @@ class Attachment(Base): parser_key: Mapped[str] = mapped_column( String(64), default="plain_text", nullable=False ) - parse_error_code: Mapped[str | None] = mapped_column( - String(120), nullable=True - ) + parse_error_code: Mapped[str | None] = mapped_column(String(120), nullable=True) # Defer large pgvector payloads on default entity loads. embedding = mapped_column(Vector(1536), deferred=True) @@ -1116,13 +1128,17 @@ class ProjectGraphObjectRecord(Base): ), Index("ix_project_graph_objects_email", "email_id"), Index("ix_project_graph_objects_primary_segment", "primary_content_segment_id"), - Index("ix_project_graph_objects_extractor", "extractor_name", "extractor_version"), + Index( + "ix_project_graph_objects_extractor", "extractor_name", "extractor_version" + ), ) project_graph_object_id: Mapped[int] = mapped_column(primary_key=True) object_uid: Mapped[str] = mapped_column(String(96), nullable=False) user_id: Mapped[str] = mapped_column(String, index=True, nullable=False) - organization_id: Mapped[str | None] = mapped_column(String, index=True, nullable=True) + organization_id: Mapped[str | None] = mapped_column( + String, index=True, nullable=True + ) workspace_id: Mapped[str] = mapped_column(String, index=True, nullable=False) email_id: Mapped[int] = mapped_column( ForeignKey("email_records.id"), nullable=False @@ -1201,7 +1217,9 @@ class ProjectGraphEdgeRecord(Base): project_graph_edge_id: Mapped[int] = mapped_column(primary_key=True) edge_uid: Mapped[str] = mapped_column(String(96), nullable=False) user_id: Mapped[str] = mapped_column(String, index=True, nullable=False) - organization_id: Mapped[str | None] = mapped_column(String, index=True, nullable=True) + organization_id: Mapped[str | None] = mapped_column( + String, index=True, nullable=True + ) workspace_id: Mapped[str] = mapped_column(String, index=True, nullable=False) source_uid: Mapped[str] = mapped_column(String(160), nullable=False) target_uid: Mapped[str] = mapped_column(String(160), nullable=False) @@ -1267,7 +1285,9 @@ class ProjectGraphCorrectionRecord(Base): nullable=False, ) user_id: Mapped[str] = mapped_column(String, index=True, nullable=False) - organization_id: Mapped[str | None] = mapped_column(String, index=True, nullable=True) + organization_id: Mapped[str | None] = mapped_column( + String, index=True, nullable=True + ) workspace_id: Mapped[str] = mapped_column(String, index=True, nullable=False) actor_user_id: Mapped[str] = mapped_column(String, nullable=False) correction_action: Mapped[str] = mapped_column(String(64), nullable=False) @@ -1362,6 +1382,22 @@ class TenantConfig(Base): batch_attribution_group: Mapped[str | None] = mapped_column(String, nullable=True) batch_attribution_company: Mapped[str | None] = mapped_column(String, nullable=True) + # Noema general-agent routing via contextual-orchestrator. All config here + # lives in the Fernet DB, never in os.getenv. The general-purpose Noema + # workspace agent (services/noema_agent.py) sends every chat-completion + # call through this gateway using this per-tenant credential -- never a + # direct tenant LLM-provider key, and never the separate credential + # ContextualWisdomLab/.github's central review-pipeline Noema uses. The + # bearer token is a secret so it is stored EncryptedString (Fernet at + # rest); the base URL is not a secret but is SSRF-guarded + allowlisted at + # call time, same as batch_orchestrator_base_url above. + noema_orchestrator_base_url: Mapped[str | None] = mapped_column( + String, nullable=True + ) + noema_orchestrator_token: Mapped[str | None] = mapped_column( + EncryptedString, nullable=True + ) + def __repr__(self) -> str: return ( f" bool: Email.message_id == parsed["message_id"], Email.user_id == IMPORT_USER_ID, Email.organization_id == IMPORT_ORGANIZATION_ID, + Email.workspace_id == IMPORT_WORKSPACE_ID, ) ) if existing.scalar_one_or_none(): @@ -64,11 +68,13 @@ async def import_eml_file(session, eml_file: Path) -> bool: parsed, user_id=IMPORT_USER_ID, organization_id=IMPORT_ORGANIZATION_ID, + workspace_id=IMPORT_WORKSPACE_ID, ) email_obj = Email( user_id=IMPORT_USER_ID, organization_id=IMPORT_ORGANIZATION_ID, + workspace_id=IMPORT_WORKSPACE_ID, message_id=parsed["message_id"], sender=parsed["sender"], reply_to=parsed.get("reply_to"), @@ -94,7 +100,9 @@ async def import_eml_file(session, eml_file: Path) -> bool: ) ) except Exception as e: - logger.error(f"Failed to generate embedding for attachment {att['filename']}: {e}") + logger.error( + f"Failed to generate embedding for attachment {att['filename']}: {e}" + ) session.add(email_obj) try: diff --git a/backend/main.py b/backend/main.py index 51b054dbf..ea86863b5 100644 --- a/backend/main.py +++ b/backend/main.py @@ -35,6 +35,7 @@ from core.config import canonical_origin, settings from core.telemetry import setup_telemetry from core.version import get_release_version +from services.attachment_reparse_worker import AttachmentReparseWorker from services.imap_worker import ImapSyncWorker from services.newsdom_worker import NewsdomRecognitionWorker from services.pop3_worker import Pop3SyncWorker @@ -46,6 +47,7 @@ pop3_worker = Pop3SyncWorker() reply_sla_scheduler = ReplySlaScheduler() newsdom_recognition_worker = NewsdomRecognitionWorker() +attachment_reparse_worker = AttachmentReparseWorker() provider_writeback_retry_worker = ProviderWritebackRetryWorker( runner_manager.dispatch_command, ) @@ -63,10 +65,12 @@ async def lifespan(app: FastAPI): await pop3_worker.start() await reply_sla_scheduler.start() await newsdom_recognition_worker.start() + await attachment_reparse_worker.start() await provider_writeback_retry_worker.start() yield if not DISABLE_WORKERS: await provider_writeback_retry_worker.stop() + await attachment_reparse_worker.stop() await newsdom_recognition_worker.stop() await reply_sla_scheduler.stop() await pop3_worker.stop() diff --git a/backend/pyproject.toml b/backend/pyproject.toml index c156d9776..046829400 100644 --- a/backend/pyproject.toml +++ b/backend/pyproject.toml @@ -43,6 +43,7 @@ dependencies = [ [dependency-groups] dev = [ "coverage==7.15.1", + "httpx2==2.5.0", "pytest==9.1.1", "pytest-asyncio==1.4.0", "ruff==0.15.21", diff --git a/backend/pytest.ini b/backend/pytest.ini index 4e58599ac..c57a00260 100644 --- a/backend/pytest.ini +++ b/backend/pytest.ini @@ -1,7 +1,6 @@ [pytest] asyncio_default_fixture_loop_scope = function filterwarnings = - ignore:Using `httpx` with `starlette.testclient` is deprecated.*:starlette.exceptions.StarletteDeprecationWarning ignore:You are using a Python version.*which Google will stop supporting.*:FutureWarning ignore:Unclosed Executable: @@ -37,6 +41,11 @@ def _get_add_columns_statements() -> list[Executable]: text("ALTER TABLE email_records ADD COLUMN IF NOT EXISTS in_reply_to varchar"), text('ALTER TABLE email_records ADD COLUMN IF NOT EXISTS "references" varchar'), text("ALTER TABLE email_records ADD COLUMN IF NOT EXISTS reply_to varchar"), + text("ALTER TABLE email_records ADD COLUMN IF NOT EXISTS workspace_id varchar"), + text( + "ALTER TABLE email_attachments " + "ADD COLUMN IF NOT EXISTS attachment_uid varchar" + ), text("ALTER TABLE llm_providers ADD COLUMN IF NOT EXISTS user_id varchar"), text( "ALTER TABLE llm_providers ADD COLUMN IF NOT EXISTS organization_id varchar" @@ -93,8 +102,7 @@ def _get_add_columns_statements() -> list[Executable]: "ADD COLUMN IF NOT EXISTS organization_id varchar" ), _static_bootstrap_sql( - "ALTER TABLE prompt_templates " - "ADD COLUMN IF NOT EXISTS workspace_id varchar" + "ALTER TABLE prompt_templates ADD COLUMN IF NOT EXISTS workspace_id varchar" ), ] @@ -186,10 +194,7 @@ def _get_create_indexes_statements() -> list[Executable]: "CREATE INDEX IF NOT EXISTS ix_email_records_owner_date " "ON email_records (user_id, organization_id, date)" ), - text( - "CREATE INDEX IF NOT EXISTS ix_emails_owner_date " - "ON emails (user_id, organization_id, date)" - ), + LEGACY_EMAILS_INDEX, text( "CREATE INDEX IF NOT EXISTS ix_sender_relationships_owner_source " "ON sender_relationships " @@ -317,6 +322,73 @@ def _get_update_project_folders_statements() -> list[Executable]: ] +def _get_update_email_workspace_statements() -> list[Executable]: + return [ + text( + "UPDATE email_records " + "SET workspace_id = 'workspace-' || organization_id " + "WHERE workspace_id IS NULL OR workspace_id = ''" + ), + text("ALTER TABLE email_records ALTER COLUMN workspace_id SET NOT NULL"), + text( + "CREATE INDEX IF NOT EXISTS ix_email_records_workspace_id " + "ON email_records (workspace_id)" + ), + # uq_email_records_owner_message_id (created earlier by validation, + # before workspace_id is guaranteed populated) is stricter than + # Alembic 0020's uq_emails_workspace_message: it forbids the same + # message_id from ever existing in two different workspaces of the + # same owner. Drop it now that workspace_id is backfilled and + # non-null, and replace it with the same workspace-scoped identity so + # a bootstrap-provisioned database doesn't silently diverge from the + # Alembic-managed schema it exists to mirror. Historical bootstrap + # runs always created this as a plain index (see + # _get_validation_and_final_indexes_statements), but a constraint + # drop is included too in case an even older schema shape used one. + text( + "ALTER TABLE email_records " + "DROP CONSTRAINT IF EXISTS uq_email_records_owner_message_id" + ), + text("DROP INDEX IF EXISTS uq_email_records_owner_message_id"), + # uq_emails_owner_message_id is the DIFFERENT owner-only identity name + # Alembic's own ORM metadata (and 0020_email_workspace_scope.py, as + # _OLD_EMAIL_IDENTITY) used before workspace scoping. A database + # provisioned via Base.metadata.create_all() + # (0001_initial_control_plane.py) before the workspace-scoped model + # landed carries this name instead of the bootstrap-specific one + # above -- drop it too, or such a database keeps the stricter + # 3-column identity forever. + text( + "ALTER TABLE email_records " + "DROP CONSTRAINT IF EXISTS uq_emails_owner_message_id" + ), + text("DROP INDEX IF EXISTS uq_emails_owner_message_id"), + text( + "CREATE UNIQUE INDEX IF NOT EXISTS " + "uq_emails_workspace_message " + "ON email_records (user_id, organization_id, workspace_id, message_id)" + ), + ] + + +def _get_update_email_attachment_statements() -> list[Executable]: + return [ + text( + "UPDATE email_attachments " + "SET attachment_uid = 'attachment_' || encode(sha256((" + "random()::text || ':' || clock_timestamp()::text || ':' || " + "id::text" + ")::bytea), 'hex') " + "WHERE attachment_uid IS NULL OR attachment_uid = ''" + ), + text("ALTER TABLE email_attachments ALTER COLUMN attachment_uid SET NOT NULL"), + text( + "CREATE UNIQUE INDEX IF NOT EXISTS uq_email_attachments_uid " + "ON email_attachments (attachment_uid)" + ), + ] + + def _get_update_prompt_template_statements() -> list[Executable]: return [ _static_bootstrap_sql( @@ -455,10 +527,6 @@ def _get_validation_and_final_indexes_statements() -> list[Executable]: ), text("ALTER TABLE llm_providers ALTER COLUMN user_id SET NOT NULL"), text("ALTER TABLE llm_providers ALTER COLUMN organization_id SET NOT NULL"), - text( - "CREATE UNIQUE INDEX IF NOT EXISTS uq_email_records_owner_message_id " - "ON email_records (user_id, organization_id, message_id)" - ), text( "CREATE UNIQUE INDEX IF NOT EXISTS uq_llm_providers_org_name " "ON llm_providers (organization_id, name)" @@ -512,6 +580,7 @@ def schema_backfill_sql() -> list[Executable]: statements.extend(_get_update_webdav_accounts_statements()) statements.extend(_get_update_project_folders_statements()) statements.extend(_get_update_prompt_template_statements()) + statements.extend(_get_update_email_attachment_statements()) statements.extend(_get_drop_constraints_and_indexes_statements()) statements.extend(_get_create_new_indexes_statements()) @@ -524,11 +593,23 @@ def schema_backfill_sql() -> list[Executable]: statements.extend(_get_validation_and_final_indexes_statements()) + # email_records.organization_id is only guaranteed NOT NULL once the + # validation above passes, so the workspace_id backfill (derived from + # organization_id) must run after it, not alongside the other + # independent per-table backfills above. + statements.extend(_get_update_email_workspace_statements()) + return statements -def _execute_statements(conn: Connection, statements: Sequence[Executable]) -> None: +def execute_schema_backfill( + conn: Connection, statements: Sequence[Executable] | None = None +) -> None: + statements = schema_backfill_sql() if statements is None else statements + legacy_emails_exists = inspect(conn).has_table("emails") for statement in statements: + if statement is LEGACY_EMAILS_INDEX and not legacy_emails_exists: + continue conn.execute(statement) @@ -536,7 +617,7 @@ async def bootstrap_db() -> None: async with engine.begin() as conn: await conn.execute(text("CREATE EXTENSION IF NOT EXISTS vector")) await conn.run_sync(Base.metadata.create_all) - await conn.run_sync(_execute_statements, schema_backfill_sql()) + await conn.run_sync(execute_schema_backfill) if __name__ == "__main__": diff --git a/backend/scripts/import_fixtures.py b/backend/scripts/import_fixtures.py index 51ea83b30..af0605731 100644 --- a/backend/scripts/import_fixtures.py +++ b/backend/scripts/import_fixtures.py @@ -28,6 +28,9 @@ logger = logging.getLogger(__name__) IMPORT_USER_ID = os.environ.get("NARUON_IMPORT_USER_ID", "default") IMPORT_ORGANIZATION_ID = os.environ.get("NARUON_IMPORT_ORGANIZATION_ID", "default") +IMPORT_WORKSPACE_ID = os.environ.get( + "NARUON_IMPORT_WORKSPACE_ID", f"workspace-{IMPORT_ORGANIZATION_ID}" +) async def process_zip_file(zip_path: str | Path, session: AsyncSession): @@ -35,7 +38,6 @@ async def process_zip_file(zip_path: str | Path, session: AsyncSession): logger.info(f"Extracting {zip_path}...") extracted_files = await extract_backup_async(zip_path, temp_dir) - batch_values = [] for file_path in extracted_files: if not str(file_path).endswith(".eml"): @@ -81,28 +83,42 @@ async def process_zip_file(zip_path: str | Path, session: AsyncSession): email_data, user_id=IMPORT_USER_ID, organization_id=IMPORT_ORGANIZATION_ID, + workspace_id=IMPORT_WORKSPACE_ID, ) - batch_values.append(dict( - user_id=IMPORT_USER_ID, - organization_id=IMPORT_ORGANIZATION_ID, - message_id=email_data["message_id"], - sender=email_data["sender"], - reply_to=email_data.get("reply_to"), - recipients=email_data["recipients"], - subject=email_data["subject"], - in_reply_to=email_data.get("in_reply_to"), - references=email_data.get("references"), - thread_id=thread_id, - date=email_data["date"], - body=email_data["body"], - embedding=embedding, - )) + batch_values.append( + dict( + user_id=IMPORT_USER_ID, + organization_id=IMPORT_ORGANIZATION_ID, + workspace_id=IMPORT_WORKSPACE_ID, + message_id=email_data["message_id"], + sender=email_data["sender"], + reply_to=email_data.get("reply_to"), + recipients=email_data["recipients"], + subject=email_data["subject"], + in_reply_to=email_data.get("in_reply_to"), + references=email_data.get("references"), + thread_id=thread_id, + date=email_data["date"], + body=email_data["body"], + embedding=embedding, + ) + ) if batch_values: stmt = insert(Email) stmt = stmt.on_conflict_do_update( - index_elements=["user_id", "organization_id", "message_id"], + # Alembic 0020_email_workspace_scope replaced the 3-column + # uq_emails_owner_message_id constraint with the 4-column + # uq_emails_workspace_message; an ON CONFLICT target that still + # names only the old 3-column shape matches no constraint on a + # real PostgreSQL database and the insert is rejected outright. + index_elements=[ + "user_id", + "organization_id", + "workspace_id", + "message_id", + ], set_=dict( sender=stmt.excluded.sender, reply_to=stmt.excluded.reply_to, @@ -113,6 +129,7 @@ async def process_zip_file(zip_path: str | Path, session: AsyncSession): thread_id=stmt.excluded.thread_id, user_id=stmt.excluded.user_id, organization_id=stmt.excluded.organization_id, + workspace_id=stmt.excluded.workspace_id, date=stmt.excluded.date, body=stmt.excluded.body, embedding=stmt.excluded.embedding, diff --git a/backend/services/attachment_parser.py b/backend/services/attachment_parser.py index 7359d6b2f..181c8e1ee 100644 --- a/backend/services/attachment_parser.py +++ b/backend/services/attachment_parser.py @@ -18,6 +18,30 @@ MAX_ATTACHMENT_PARSE_SOURCE_CHARS = 1_000_000 MAX_ATTACHMENT_PARSE_SOURCE_BYTES = 20 * 1024 * 1024 MAX_ATTACHMENT_FILENAME_DECODE_ROUNDS = 3 +CONTENT_TYPE_MISMATCH_QUARANTINED_STATUS = "content_type_mismatch_quarantined" +# Magic-byte signatures for content whose real type is cheaply verifiable from +# its first bytes, independent of whatever content_type/filename the sender +# claimed. Order matters: checked in sequence, first match wins. +_MAGIC_BYTE_SIGNATURES: tuple[tuple[bytes, str], ...] = ( + (b"%PDF-", "application/pdf"), + (b"\x89PNG\r\n\x1a\n", "image/png"), + (b"\xff\xd8\xff", "image/jpeg"), + (b"GIF87a", "image/gif"), + (b"GIF89a", "image/gif"), + (b"PK\x03\x04", "application/zip"), + (b"PK\x05\x06", "application/zip"), +) +# Substrings of MIME types whose files are legitimately ZIP containers under +# the hood (OOXML Office documents, OpenDocument formats, EPUB, JAR). Sniffing +# "application/zip" from magic bytes must not quarantine a declared type in +# this family -- only a declared type outside it that still sniffs as ZIP is +# a genuine disguise. +_ZIP_CONTAINER_CONTENT_TYPE_MARKERS = ( + "openxmlformats-officedocument", + "vnd.oasis.opendocument", + "application/epub+zip", + "application/java-archive", +) @dataclass(frozen=True) @@ -140,6 +164,97 @@ def get_attachment_parser_manifest() -> list[AttachmentParserDescriptor]: return list(_PARSER_MANIFEST) +def _sniff_content_type(raw_content: Any) -> str | None: + """Return the MIME type implied by known magic bytes, or None if unrecognized. + + Only content families with a cheap, reliable byte signature are covered + (see ``_MAGIC_BYTE_SIGNATURES``) -- text formats have no such signature and + are intentionally left unsniffed. + """ + payload = _coerce_deferred_payload_bytes(raw_content) + for signature, sniffed_content_type in _MAGIC_BYTE_SIGNATURES: + if payload.startswith(signature): + return sniffed_content_type + return None + + +def _is_zip_container_content_type(content_type: str) -> bool: + """Return True for a declared MIME type whose files are legitimately ZIPs.""" + return content_type == "application/zip" or any( + marker in content_type for marker in _ZIP_CONTAINER_CONTENT_TYPE_MARKERS + ) + + +def _is_genuine_content_type_mismatch( + *, sniffed_content_type: str | None, parse_content_type: str +) -> bool: + """Return True only for a sniff/declared disagreement worth quarantining. + + A ZIP-sniffed payload declared as an OOXML/ODF/EPUB/JAR type is not a + mismatch -- those formats are ZIP containers by specification, so their + magic bytes are supposed to match ZIP's. Only a ZIP-sniffed payload + declared as something outside that family (or any other sniff/declared + disagreement) is a genuine disguise. + + A ``parse_content_type`` still equal to one of ``_GENERIC_CONTENT_TYPES`` + after ``_parse_content_type_for`` ran is not a mismatch either: that + family is MIME's own "no more specific type available" placeholder, not + a positive claim, so there is nothing for the sniffed bytes to disagree + with -- an ordinary PNG/PDF/ZIP sent this way was never disguised, only + undeclared. This does not apply once a generic declaration resolves to + something specific via a recognized filename extension (that *is* a + real claim by the time it reaches here). + """ + if sniffed_content_type is None or sniffed_content_type == parse_content_type: + return False + if parse_content_type in _GENERIC_CONTENT_TYPES: + return False + if sniffed_content_type == "application/zip" and _is_zip_container_content_type( + parse_content_type + ): + return False + return True + + +def _quarantine_result( + *, + safe_filename: str, + normalized_content_type: str, + sniffed_content_type: str, + raw_content: Any, +) -> AttachmentParseResult: + quarantined_payload = _coerce_deferred_payload_bytes(raw_content) + if len(quarantined_payload) > MAX_ATTACHMENT_PARSE_SOURCE_BYTES: + # An oversized mismatched payload retains no bytes, so it can never + # be usefully reparsed -- classify it the same way every other + # oversized attachment in this file already is (a non-retryable + # terminal status), rather than as a quarantine the reparse-intent + # API would otherwise accept for a row it can do nothing with. + return AttachmentParseResult( + filename=safe_filename, + content="", + content_type=normalized_content_type, + parse_content="", + parse_content_type=sniffed_content_type, + parser_key=_parser_key_for(sniffed_content_type, "parsed"), + parse_status="parse_size_limit_exceeded", + parse_error_code="parse_size_limit_exceeded", + ) + return AttachmentParseResult( + filename=safe_filename, + content=_encode_deferred_payload(quarantined_payload), + content_type=normalized_content_type, + parse_content="", + # parse_content_type carries what the bytes actually are here (not + # what was declared/resolved) so a caller can compare content_type + # (declared) against parse_content_type (sniffed) to see the mismatch. + parse_content_type=sniffed_content_type, + parser_key=_parser_key_for(sniffed_content_type, "parsed"), + parse_status=CONTENT_TYPE_MISMATCH_QUARANTINED_STATUS, + parse_error_code=CONTENT_TYPE_MISMATCH_QUARANTINED_STATUS, + ) + + def parse_email_attachment( *, filename: str | None, @@ -154,6 +269,22 @@ def parse_email_attachment( normalized_content_type, ) + # A recognized signature that disagrees with the declared/resolved type + # is a stronger, independent signal than anything below (size limits, + # PDF-specific magic-byte validation, supported-type lookup) -- a mislabeled + # or disguised attachment must never reach those paths and get silently + # parsed or classified under the wrong type. + sniffed_content_type = _sniff_content_type(raw_content) + if _is_genuine_content_type_mismatch( + sniffed_content_type=sniffed_content_type, parse_content_type=parse_content_type + ): + return _quarantine_result( + safe_filename=safe_filename, + normalized_content_type=normalized_content_type, + sniffed_content_type=sniffed_content_type, + raw_content=raw_content, + ) + deferred_descriptor = _DEFERRED_DESCRIPTORS_BY_CONTENT_TYPE.get(parse_content_type) if deferred_descriptor is not None: # Heavy recognition (OCR/MinerU via the NewsDOM sidecar) must not run @@ -319,6 +450,25 @@ def decode_deferred_attachment_payload(content: str | None) -> bytes: return payload +def decode_quarantined_attachment_payload(content: str | None) -> bytes: + """Decode the base64 payload retained on a reparse-pending attachment. + + Raises ``ValueError`` when the stored payload is not valid base64, so the + reparse worker can record a terminal failure instead of crashing. Unlike + ``decode_deferred_attachment_payload`` (PDF-only, used by the NewsDOM + worker), a quarantined attachment's sniffed type can be any of the + magic-byte families this module recognizes, so no single-format check + narrows this one. + """ + try: + payload = base64.b64decode((content or "").encode("ascii"), validate=True) + except (binascii.Error, UnicodeEncodeError, ValueError) as exc: + raise ValueError("Quarantined attachment payload is not valid base64") from exc + if len(payload) > MAX_ATTACHMENT_PARSE_SOURCE_BYTES: + raise ValueError("Quarantined attachment payload exceeds the parse size limit") + return payload + + def _coerce_text(raw_content: Any) -> str: """Coerce arbitrary attachment content to NUL-free text.""" if raw_content is None: diff --git a/backend/services/attachment_reparse_worker.py b/backend/services/attachment_reparse_worker.py new file mode 100644 index 000000000..c3edaf249 --- /dev/null +++ b/backend/services/attachment_reparse_worker.py @@ -0,0 +1,571 @@ +"""Background worker glue for re-evaluating quarantined attachments. + +``POST /api/data/attachments/{attachment_uid}/reparse-intent`` moves a +``content_type_mismatch_quarantined`` attachment to ``reparse_pending``, +retaining its raw bytes (base64) but performing no synchronous re-parse (see +``docs/adr/0005-attachment-content-type-quarantine.md``). This worker sweeps +``reparse_pending`` rows and replays the exact same classification pipeline +(:func:`services.attachment_parser.parse_email_attachment`) against those +retained bytes and the attachment's original declared ``content_type`` -- so +a fix to that pipeline (e.g. the OOXML/ODF/EPUB/JAR false-positive +carve-out) is automatically picked up on the next reparse pass with no +bespoke logic here. An attachment whose disagreement was genuine simply +lands back in the quarantine status; nothing here decides that on its own. +""" + +from __future__ import annotations + +import asyncio +import logging +import random +from dataclasses import dataclass + +from sqlalchemy import bindparam, case, func, or_, select +from sqlalchemy.ext.asyncio import AsyncConnection, AsyncSession + +from db.models import Attachment, ContentNodeRecord, ContentSegmentRecord +from db.session import AsyncSessionLocal, engine +from services.attachment_parser import ( + AttachmentParseResult, + decode_quarantined_attachment_payload, + parse_email_attachment, +) +from services.content_graph import content_graph_source_record_uid, parse_content +from services.email_import_service import ( + EmailImportEmbeddingProvider, + append_knowledge_graph_edges, + generate_source_embedding, +) +from services.llm_provider_selection import resolve_runtime_llm_provider + +logger = logging.getLogger(__name__) +_sysrand = random.SystemRandom() + +DEFAULT_ATTACHMENT_REPARSE_INTERVAL_SECONDS = 60 +DEFAULT_ATTACHMENT_REPARSE_BATCH_LIMIT = 10 +ATTACHMENT_REPARSE_SWEEP_LOCK_NAMESPACE = "naruon-attachment-reparse-sweep" +MAX_STARTUP_JITTER_SECONDS = 30 +# POST .../reparse-intent can re-mark ANY existing attachment reparse_pending, +# including one whose id is already behind the forward cursor -- the +# cursor+retry-id design alone can never discover that (retry_ids only +# tracks rows this worker itself already saw and found unresolved). Forcing +# a full rescan (cursor -> None) on this cadence bounds how stale such a row +# can get; always safe regardless of cadence, since the parse_status filter +# already excludes every row that's actually resolved. See _sweep_attachments +# (mirrors services.newsdom_worker.NewsdomRecognitionWorker's identical fix). +FULL_RESCAN_EVERY_N_SWEEPS = 20 + +# Mirrors api.data.ATTACHMENT_REPARSE_PENDING_STATUS. Duplicated as a literal +# (not imported) to avoid a services -> api import: api already imports from +# services, and this worker has no other reason to depend on the router module. +ATTACHMENT_REPARSE_PENDING_STATUS = "reparse_pending" +# A retained payload that is not valid base64 cannot become valid on a later +# sweep -- a terminal, non-retryable data problem, not a transient one. +ATTACHMENT_REPARSE_PAYLOAD_INVALID_STATUS = "reparse_payload_invalid" + +# Per-item processing outcome for the one case this worker can never turn +# into a fresh AttachmentParseResult. Every other outcome is reported as the +# resulting parse_status itself (e.g. "parsed", the quarantine status again, +# "unsupported_content_type") so a caller sees exactly what classification +# decided, not a coarse success/failure flag. +RESULT_DECODE_FAILED = "decode_failed" + +_SWEEP_LOCK_PARAMS = { + "namespace_key": ATTACHMENT_REPARSE_SWEEP_LOCK_NAMESPACE, + "sweep_key": "sweep", +} + + +def apply_reparsed_result(*, attachment: Attachment, result: AttachmentParseResult) -> None: + """Land a fresh classification result onto an existing attachment row. + + ``filename`` and ``content_type`` are not overwritten: they are the + attachment's identity and the sender's original declaration, neither of + which a reparse pass should silently rewrite (``parse_email_attachment`` + only ever normalizes them for a *new* row, and the retained + ``attachment.content_type`` is what is fed back in as its input here). + + ``content`` is only overwritten when the result actually has content to + store. A reparse that lands on a status with no displayable content + (``unsupported_content_type``, ``parse_size_limit_exceeded``) returns + ``content=""`` by design -- storing that would destroy the only retained + copy of the original quarantined bytes, permanently losing a file that + later parser support could otherwise still recover. + + A reparse that lands on ``"parsed"`` also indexes the recognized content + into the content graph, mirroring what the initial import path already + does for an attachment that parses cleanly on first import + (``email_import_service._append_email_content_graph``) -- without this, a + previously-quarantined attachment stayed invisible to content-graph-backed + search/AI-hub features even after successful reparse recognition. + """ + if result.content: + attachment.content = result.content + attachment.parse_content_type = result.parse_content_type + attachment.parser_key = result.parser_key + attachment.parse_status = result.parse_status + attachment.parse_error_code = result.parse_error_code + if result.parse_status == "parsed": + _append_reparsed_attachment_content_graph(attachment=attachment, result=result) + + +def _append_reparsed_attachment_content_graph( + *, attachment: Attachment, result: AttachmentParseResult +) -> None: + """Build content graph records for a successfully reparsed attachment. + + Reuses the same ``parse_content`` helper and ``source_record_uid`` + identity convention (``content_graph_source_record_uid``) the import path + uses in ``email_import_service._append_email_content_graph`` -- this is + not a second indexing path, just a second call site for the same one. + + It differs only in how the new records attach to their parents. The + import path appends to a transient ``Email``/``Attachment`` pair (neither + has a real id yet) and lets SQLAlchemy's relationship cascade resolve + ``email_id``/``attachment_id`` at flush time. Here ``attachment`` is + already a persisted row with a stable, permanent ``attachment_uid``, so + ``source_record_uid`` is keyed on that uid alone (not the message-id + + list-position convention the import path uses, since a persisted + attachment's position among its email's siblings is not reliably + reproducible) and ``email_id`` is taken directly from the attachment's + already-loaded ``email_id`` column instead of an ``Email`` relationship + append. + """ + parse_source_content = result.parse_content or result.content + if not parse_source_content.strip(): + return + + parse_result = parse_content( + source_kind="attachment", + source_record_uid=content_graph_source_record_uid( + "attachment", attachment.attachment_uid + ), + content=parse_source_content, + content_type=result.parse_content_type or result.content_type or "text/plain", + display_name=attachment.filename, + ) + + node_records_by_uid: dict[str, ContentNodeRecord] = {} + for parsed_node in parse_result.nodes: + node_record = ContentNodeRecord( + email_id=attachment.email_id, + content_node_uid=parsed_node.content_node_uid, + source_kind=parsed_node.source_kind, + source_record_uid=parsed_node.source_record_uid, + parent_node_uid=parsed_node.parent_node_uid, + node_kind=parsed_node.node_kind, + node_path=parsed_node.node_path, + ordinal_index=parsed_node.ordinal_index, + display_label=parsed_node.display_label, + safe_text_content=parsed_node.safe_text_content, + content_hash=parsed_node.content_hash, + ) + attachment.content_nodes.append(node_record) + node_records_by_uid[parsed_node.content_node_uid] = node_record + + segment_records: list[ContentSegmentRecord] = [] + for parsed_segment in parse_result.segments: + segment_record = ContentSegmentRecord( + email_id=attachment.email_id, + content_segment_uid=parsed_segment.content_segment_uid, + source_kind=parsed_segment.source_kind, + source_record_uid=parsed_segment.source_record_uid, + segment_kind=parsed_segment.segment_kind, + segment_path=parsed_segment.segment_path, + ordinal_index=parsed_segment.ordinal_index, + heading_path=parsed_segment.heading_path, + safe_text_content=parsed_segment.safe_text_content, + content_hash=parsed_segment.content_hash, + word_count=parsed_segment.word_count, + ) + node_records_by_uid[parsed_segment.content_node_uid].segments.append( + segment_record + ) + attachment.content_segments.append(segment_record) + segment_records.append(segment_record) + + append_knowledge_graph_edges( + nodes=list(node_records_by_uid.values()), + segments=segment_records, + attachment_obj=attachment, + ) + + +async def _refresh_reparsed_attachment_embedding( + session: AsyncSession, attachment: Attachment, *, source_text: str +) -> None: + """Regenerate the attachment vector through its tenant's active provider. + + ``source_text`` must be the caller's already-resolved embedding source + (see ``ReparseOutcome.embedding_source_text``), not read from + ``attachment.content``: ``apply_reparsed_result`` only overwrites that + column when ``result.content`` is non-empty (see its docstring), so a + "parsed" result whose *display* text strips to empty while its *parse* + text does not (e.g. markup-only content) would leave ``attachment.content`` + stale and this would otherwise embed unrelated, already-superseded bytes. + """ + provider = await resolve_runtime_llm_provider( + session, + user_id=attachment.email.user_id, + organization_id=attachment.email.organization_id, + ) + embedding_provider = ( + EmailImportEmbeddingProvider( + api_key=provider.api_key, + base_url=provider.base_url, + embedding_model=provider.embedding_model, + ) + if provider is not None + else None + ) + attachment.embedding = await generate_source_embedding( + source_text, + embedding_provider=embedding_provider, + ) + + +@dataclass(frozen=True, slots=True) +class ReparseOutcome: + """Result of one ``process_reparse_pending_attachment`` call. + + ``embedding_source_text`` mirrors the exact source-text resolution + ``_append_reparsed_attachment_content_graph`` uses (``parse_content`` + preferred over ``content``) and the import path's + ``email_import_service._extract_and_generate_embeddings`` already uses + for the same reason -- it is meaningful only when ``parse_status == + "parsed"``, but is always populated for a uniform return shape. + """ + + parse_status: str + embedding_source_text: str + + +def process_reparse_pending_attachment(*, attachment: Attachment) -> ReparseOutcome: + """Re-evaluate one ``reparse_pending`` attachment in place. + + Returns a :class:`ReparseOutcome` carrying the resulting ``parse_status`` + on a successful re-evaluation (``"parsed"``, the quarantine status again, + or any other terminal status ``parse_email_attachment`` can return), or + ``RESULT_DECODE_FAILED`` when the retained payload itself is not valid + base64 -- moved to a dedicated failure status so the sweep does not retry + it forever. + """ + try: + raw_content = decode_quarantined_attachment_payload(attachment.content) + except ValueError as exc: + attachment.parse_status = ATTACHMENT_REPARSE_PAYLOAD_INVALID_STATUS + attachment.parse_error_code = ATTACHMENT_REPARSE_PAYLOAD_INVALID_STATUS + logger.warning( + "Attachment %s reparse rejected: %s", + getattr(attachment, "id", "?"), + exc, + ) + return ReparseOutcome( + parse_status=RESULT_DECODE_FAILED, embedding_source_text="" + ) + + result = parse_email_attachment( + filename=attachment.filename, + content_type=attachment.content_type, + raw_content=raw_content, + ) + apply_reparsed_result(attachment=attachment, result=result) + return ReparseOutcome( + parse_status=result.parse_status, + embedding_source_text=result.parse_content or result.content, + ) + + +def _engine_uses_postgresql() -> bool: + """Return whether advisory-lock SQL is supported by the configured engine.""" + return engine.dialect.name == "postgresql" + + +async def _try_acquire_sweep_lease(connection: AsyncConnection) -> bool: + """Become the sweep leader for this cycle on this dedicated connection. + + Takes an ``AsyncConnection`` rather than the per-item ``AsyncSession`` + deliberately: ``AsyncSession.commit()``/``rollback()`` release their + underlying connection back to the pool on every call (SQLAlchemy's normal + "connectionless execution" behavior), so acquiring the lock through that + session risks releasing it -- or never releasing it -- from a *different* + physical backend connection than the one PostgreSQL actually granted the + advisory lock to. Advisory locks are scoped to the acquiring backend + session, so a mismatched unlock is a silent no-op and the lock stays held + (blocking every replica's sweep) until the stray connection is eventually + recycled or closed. + + Sets AUTOCOMMIT on this connection first: without it, the advisory-lock + SELECT below opens an implicit transaction that would otherwise stay + open and idle on this connection for the whole sweep (every other + statement runs through the separate per-item session), risking + PostgreSQL's ``idle_in_transaction_session_timeout`` killing this + connection mid-sweep and silently dropping the lease. + """ + await connection.execution_options(isolation_level="AUTOCOMMIT") + acquired = await connection.scalar( + select( + func.pg_try_advisory_lock( + func.hashtext(bindparam("namespace_key")), + func.hashtext(bindparam("sweep_key")), + ) + ), + _SWEEP_LOCK_PARAMS, + ) + return bool(acquired) + + +async def _release_sweep_lease(connection: AsyncConnection) -> None: + """Release the PostgreSQL advisory lock on the connection that holds it.""" + await connection.scalar( + select( + func.pg_advisory_unlock( + func.hashtext(bindparam("namespace_key")), + func.hashtext(bindparam("sweep_key")), + ) + ), + _SWEEP_LOCK_PARAMS, + ) + + +class AttachmentReparseWorker: + """Periodically re-evaluate ``reparse_pending`` attachments. + + Mirrors :class:`services.newsdom_worker.NewsdomRecognitionWorker`: a + jittered periodic loop, a PostgreSQL advisory-lock lease so only one + replica sweeps per cycle, per-item error isolation, and a starvation-free + cursor. Unlike that worker, re-evaluation never depends on an external + provider being configured, so there is no "left pending, retry later" + outcome here -- every row is fully resolved (to a parsed status, back to + quarantine, or to the dedicated payload-invalid failure) on its very + first sweep. + """ + + def __init__( + self, + *, + interval_seconds: int = DEFAULT_ATTACHMENT_REPARSE_INTERVAL_SECONDS, + batch_limit: int = DEFAULT_ATTACHMENT_REPARSE_BATCH_LIMIT, + ): + """Configure the sweep cadence and batch size.""" + self.interval_seconds = interval_seconds + self.batch_limit = batch_limit + self._task: asyncio.Task | None = None + self._is_running = False + self._attachment_cursor: int | None = None + # Rows a past sweep saw raise. Retried every sweep via an explicit + # id filter, independent of the forward cursor -- see + # _sweep_attachments. + self._attachment_retry_ids: set[int] = set() + self._attachment_sweep_count = 0 + + async def start(self) -> None: + """Start the reparse loop once.""" + if self._is_running: + logger.warning("AttachmentReparseWorker is already running.") + return + self._is_running = True + self._task = asyncio.create_task(self._run_loop()) + logger.info("AttachmentReparseWorker started.") + + async def stop(self) -> None: + """Cancel and await the active reparse loop.""" + if not self._is_running: + return + self._is_running = False + if self._task: + self._task.cancel() + try: + await self._task + except asyncio.CancelledError: + logger.debug("AttachmentReparseWorker cancellation acknowledged.") + logger.info("AttachmentReparseWorker stopped.") + + async def _run_loop(self) -> None: + """Run jittered reparse sweeps until stopped.""" + try: + await asyncio.sleep( + _sysrand.uniform( + 0, min(self.interval_seconds / 10, MAX_STARTUP_JITTER_SECONDS) + ) + ) + except asyncio.CancelledError: + return + + while self._is_running: + try: + await self._sweep() + except asyncio.CancelledError: + break + except Exception: + logger.error("Error in AttachmentReparseWorker loop.", exc_info=True) + if self._is_running: + try: + await asyncio.sleep(self.interval_seconds) + except asyncio.CancelledError: + break + + async def _sweep(self) -> None: + """Process one leased reparse sweep. + + The advisory lock (when the engine supports it) is acquired and + released on one dedicated connection held open for the whole sweep -- + never through the per-item ``AsyncSession`` used by + ``_sweep_attachments``. See ``_try_acquire_sweep_lease`` for why that + distinction matters. + """ + if not _engine_uses_postgresql(): + async with AsyncSessionLocal() as session: + await self._sweep_attachments(session) + return + async with engine.connect() as lock_connection: + if not await _try_acquire_sweep_lease(lock_connection): + logger.debug( + "Attachment reparse sweep skipped: another replica holds " + "the lease." + ) + return + try: + async with AsyncSessionLocal() as session: + await self._sweep_attachments(session) + finally: + await _release_sweep_lease(lock_connection) + + async def _sweep_attachments(self, session: AsyncSession) -> None: + """Process a bounded, starvation-free batch of reparse-pending rows. + + ``_attachment_cursor`` is a forward-scan position that only ever + advances (to the highest id seen in a batch), and + ``_attachment_retry_ids`` is the set of ids a past sweep saw raise, + retried every sweep via an explicit ``id IN (...)`` filter + independent of the cursor -- the same design as + ``services.newsdom_worker.NewsdomRecognitionWorker._sweep_attachments`` + (see its docstring for the full rationale). An earlier version + instead capped the cursor itself at the first failure: that kept one + failing row selectable, but pinned the whole batch window behind it + once more than ``batch_limit`` consecutive rows failed at once + (e.g. a systematic classification bug affecting a burst of + simultaneous reparse-intent requests) -- nothing past them would + ever be reached. Decoupling retry tracking from the forward cursor + fixes that. + + Neither piece of state can discover a row explicitly re-marked + ``reparse_pending`` after the cursor already passed it -- + ``POST .../reparse-intent`` can retarget any existing attachment, + including an old one this worker already resolved. Every + ``FULL_RESCAN_EVERY_N_SWEEPS``-th sweep forces a full rescan + (cursor reset to ``None``) to bound how stale such a row can get; + see ``services.newsdom_worker`` for why this is always safe. + """ + self._attachment_sweep_count += 1 + if self._attachment_sweep_count % FULL_RESCAN_EVERY_N_SWEEPS == 0: + self._attachment_cursor = None + rows = await self._load_reparse_pending_attachments(session) + processed_ids: set[int] = set() + unresolved_ids: set[int] = set() + for attachment_id in [attachment.id for attachment in rows]: + processed_ids.add(attachment_id) + try: + # Re-fetch fresh rather than reusing the bulk-loaded instance: + # AsyncSession.rollback() expires every object already loaded + # in this session, and process_reparse_pending_attachment + # reads attachment attributes synchronously -- a stale, + # expired instance from an earlier item's failure would raise + # on that read instead of just isolating the one failure. + # Fetching fresh here sidesteps that whole class of bug + # regardless of exactly when SQLAlchemy decides to expire. + attachment = await session.get(Attachment, attachment_id) + if attachment is None: + continue + # ``apply_reparsed_result`` appends graph rows through these + # relationships. Load them explicitly at the async boundary; + # implicit lazy IO from the synchronous parser path raises + # MissingGreenlet for persisted attachments. + await session.refresh( + attachment, + attribute_names=[ + "email", + "content_nodes", + "content_segments", + "knowledge_graph_edges", + ], + ) + outcome = process_reparse_pending_attachment(attachment=attachment) + if outcome.parse_status == "parsed": + await _refresh_reparsed_attachment_embedding( + session, + attachment, + source_text=outcome.embedding_source_text, + ) + await session.commit() + logger.info( + "Attachment %s reparse result: %s", + attachment_id, + outcome.parse_status, + ) + except Exception: + await session.rollback() + logger.error( + "Attachment %s reparse raised.", + attachment_id, + exc_info=True, + ) + unresolved_ids.add(attachment_id) + if rows: + highest_seen = max(attachment.id for attachment in rows) + self._attachment_cursor = ( + highest_seen + if self._attachment_cursor is None + else max(self._attachment_cursor, highest_seen) + ) + self._attachment_retry_ids = ( + self._attachment_retry_ids - processed_ids + ) | unresolved_ids + + def _reparse_pending_statement( + self, after_id: int | None, retry_ids: set[int] + ): + """Build the next deterministic reparse-pending batch query. + + Selects rows past the forward cursor OR still tracked in + ``retry_ids``, forward rows ordered ahead of retry rows when both + are present -- identical shape to + ``services.newsdom_worker.NewsdomRecognitionWorker._pending_attachment_statement`` + (see its docstring for why the priority must run that way). + """ + statement = select(Attachment).where( + Attachment.parse_status == ATTACHMENT_REPARSE_PENDING_STATUS + ) + # retry_ids only narrows the result when there's a cursor to narrow + # *against* -- with no cursor yet (after_id is None, including a + # forced full rescan), every pending row already qualifies, and + # restricting to just retry_ids here would incorrectly hide + # everything else that's pending. + if after_id is not None: + conditions = [Attachment.id > after_id] + if retry_ids: + conditions.append(Attachment.id.in_(retry_ids)) + statement = statement.where( + conditions[0] if len(conditions) == 1 else or_(*conditions) + ) + order_columns = [] + if after_id is not None and retry_ids: + order_columns.append(case((Attachment.id > after_id, 0), else_=1)) + order_columns.append(Attachment.id) + return statement.order_by(*order_columns).limit(self.batch_limit) + + async def _load_reparse_pending_attachments( + self, session: AsyncSession + ) -> list[Attachment]: + """Load the next batch: past the cursor, plus any known-stuck rows.""" + return ( + ( + await session.execute( + self._reparse_pending_statement( + self._attachment_cursor, self._attachment_retry_ids + ) + ) + ) + .scalars() + .all() + ) diff --git a/backend/services/calendar_conflict_ics.py b/backend/services/calendar_conflict_ics.py index 4f21a5cc3..0814f13db 100644 --- a/backend/services/calendar_conflict_ics.py +++ b/backend/services/calendar_conflict_ics.py @@ -8,6 +8,7 @@ from icalendar import Calendar from services.calendar_conflict_policy import ( + MAX_EXISTING_COMMITMENTS, CalendarCommitment, CalendarConflictDecision, CalendarPolicyValidationError, @@ -21,8 +22,7 @@ "TENTATIVE": "tentative", "CANCELLED": "cancelled", } -_MAX_EXISTING_ICS_COMMITMENTS = 500 -_MAX_CONVERTED_VEVENTS = _MAX_EXISTING_ICS_COMMITMENTS + 1 +_MAX_CONVERTED_VEVENTS = MAX_EXISTING_COMMITMENTS + 1 _MAX_ICS_DOCUMENT_BYTES = 262_144 _RECURRENCE_PROPERTY_NAMES = ("RRULE", "RDATE", "EXDATE") @@ -69,7 +69,7 @@ def evaluate_calendar_conflicts_from_ics( """Evaluate one proposed VEVENT against existing VEVENT evidence.""" proposed_commitment = parse_proposed_calendar_commitment_from_ics(proposed_ics) existing_commitments = parse_existing_calendar_commitments_from_ics(existing_ics) - if len(existing_commitments) > _MAX_EXISTING_ICS_COMMITMENTS: + if len(existing_commitments) > MAX_EXISTING_COMMITMENTS: raise CalendarPolicyValidationError( "calendar_existing_batch_exceeded", "existing iCalendar evidence exceeds the bounded commitment batch", diff --git a/backend/services/calendar_conflict_judgment_service.py b/backend/services/calendar_conflict_judgment_service.py new file mode 100644 index 000000000..6cfcce375 --- /dev/null +++ b/backend/services/calendar_conflict_judgment_service.py @@ -0,0 +1,302 @@ +"""Persist `evaluate_calendar_conflicts` decisions and human corrections to them.""" + +from __future__ import annotations + +import datetime +from typing import Any + +from sqlalchemy import select +from sqlalchemy.ext.asyncio import AsyncSession + +from db.models import CalendarConflictCorrection, CalendarConflictJudgment +from services.calendar_conflict_policy import ( + CalendarConflictDecision, + default_recommended_action, +) + +ALLOWED_DECISION_CODES = frozenset({"available", "blocked", "review_required"}) +ALLOWED_STATUS_CODES = frozenset({"proposed", "confirmed", "overridden", "dismissed"}) +# Coherence contract between status_code and decision_code: an override must +# replace the decision (there is nothing else it could mean), while every +# other status must leave the existing decision untouched -- a "confirmed" or +# "dismissed" judgment whose decision silently changed would let the audit +# history and the current decision disagree. +_STATUS_CODES_REQUIRING_DECISION_CHANGE = frozenset({"overridden"}) +_STATUS_CODES_FORBIDDING_DECISION_CHANGE = frozenset({"proposed", "confirmed", "dismissed"}) +# A human correction that changes decision_code always replaces reason_code +# and recommended_action together, so a caller can never observe a decision +# paired with a reason/action that describes a different decision. +CORRECTED_DECISION_REASON_CODE = "corrected_by_human_review" +_MAX_JUDGMENTS_PER_LIST = 200 + + +class CalendarConflictJudgmentNotFoundError(ValueError): + """Raised when a judgment_uid does not resolve inside the caller's own scope.""" + + +CORRECTION_INCOHERENT_ERROR_CODE = "calendar_correction_incoherent" + + +class CalendarConflictCorrectionIncoherentError(ValueError): + """Raised when a correction's status_code and decision_code disagree. + + Carries a stable ``error_code`` (mirroring + ``CalendarConflictUnsupportedValueError``'s pattern below) so a caller + can map this to a deterministic response without parsing ``str(exc)``. + """ + + def __init__(self, message: str) -> None: + """Create a coherence failure with the shared, stable error_code.""" + super().__init__(message) + self.error_code = CORRECTION_INCOHERENT_ERROR_CODE + + +class CalendarConflictUnsupportedValueError(ValueError): + """Stable typed failure for an unsupported status_code/decision_code value. + + Mirrors ``CalendarPolicyValidationError``'s pattern (a message-independent + ``error_code`` attribute) so a route can map this to a deterministic + response without parsing ``str(exc)``. + """ + + def __init__(self, error_code: str, message: str) -> None: + """Create a validation failure with a stable public-facing code.""" + super().__init__(message) + self.error_code = error_code + + +UNSUPPORTED_STATUS_CODE_ERROR_CODE = "calendar_correction_status_code_unsupported" +UNSUPPORTED_DECISION_CODE_ERROR_CODE = "calendar_correction_decision_code_unsupported" + + +def _utcnow() -> datetime.datetime: + return datetime.datetime.now(datetime.timezone.utc) + + +def _conflicts_to_json(decision: CalendarConflictDecision) -> list[dict[str, Any]]: + return [ + { + "commitment_id": conflict.commitment_id, + "start_at": conflict.start_at.isoformat(), + "end_at": conflict.end_at.isoformat(), + "status": conflict.status, + } + for conflict in decision.conflicts + ] + + +def _organization_filter(organization_id: str | None): + if organization_id is not None: + return CalendarConflictJudgment.organization_id == organization_id + return CalendarConflictJudgment.organization_id.is_(None) + + +def validate_correction_coherence(*, status_code: str, decision_code: str | None) -> None: + """Reject a status_code/decision_code combination that would leave the + judgment's decision and its correction status describing different things. + + Shared by the API request model and apply_correction (for non-HTTP + callers), so neither entry point can bypass the other's check. + """ + if status_code in _STATUS_CODES_REQUIRING_DECISION_CHANGE and decision_code is None: + raise CalendarConflictCorrectionIncoherentError( + f"status_code={status_code!r} requires a replacement decision_code" + ) + if status_code in _STATUS_CODES_FORBIDDING_DECISION_CHANGE and decision_code is not None: + raise CalendarConflictCorrectionIncoherentError( + f"status_code={status_code!r} must not change decision_code" + ) + + +async def create_judgment( + db: AsyncSession, + *, + user_id: str, + organization_id: str | None, + workspace_id: str, + proposed_commitment_id: str, + source_thread_id: str | None, + source_message_id: str | None, + decision: CalendarConflictDecision, +) -> CalendarConflictJudgment: + """Persist one deterministic conflict decision as a correctable judgment record.""" + judgment = CalendarConflictJudgment( + user_id=user_id, + organization_id=organization_id, + workspace_id=workspace_id, + proposed_commitment_id=proposed_commitment_id, + source_thread_id=source_thread_id, + source_message_id=source_message_id, + decision_code=decision.decision_code, + reason_code=decision.reason_code, + recommended_action=decision.recommended_action, + policy_version=decision.policy_version, + conflicts_json=_conflicts_to_json(decision), + ) + db.add(judgment) + await db.flush() + return judgment + + +async def list_judgments( + db: AsyncSession, + *, + user_id: str, + organization_id: str | None, + workspace_id: str, + source_thread_id: str | None = None, +) -> list[CalendarConflictJudgment]: + """List persisted judgments in the caller's own scope, newest first.""" + filters = [ + CalendarConflictJudgment.user_id == user_id, + _organization_filter(organization_id), + CalendarConflictJudgment.workspace_id == workspace_id, + ] + if source_thread_id is not None: + filters.append(CalendarConflictJudgment.source_thread_id == source_thread_id) + stmt = ( + select(CalendarConflictJudgment) + .where(*filters) + .order_by( + CalendarConflictJudgment.created_at.desc(), + CalendarConflictJudgment.calendar_conflict_judgment_id.desc(), + ) + .limit(_MAX_JUDGMENTS_PER_LIST) + ) + result = await db.execute(stmt) + return list(result.scalars().all()) + + +async def get_judgment( + db: AsyncSession, + *, + judgment_uid: str, + user_id: str, + organization_id: str | None, + workspace_id: str, +) -> CalendarConflictJudgment: + """Fetch one judgment by its opaque uid, regardless of list_judgments' bound. + + A judgment older than the most recent _MAX_JUDGMENTS_PER_LIST rows falls + out of list_judgments' window, but it is never unreachable: the caller + that received its judgment_uid (from the original create_judgment + response, or from a correction) can always look it up directly here. + """ + return await _get_scoped_judgment( + db, + judgment_uid=judgment_uid, + user_id=user_id, + organization_id=organization_id, + workspace_id=workspace_id, + ) + + +async def _get_scoped_judgment( + db: AsyncSession, + *, + judgment_uid: str, + user_id: str, + organization_id: str | None, + workspace_id: str, + for_update: bool = False, +) -> CalendarConflictJudgment: + stmt = select(CalendarConflictJudgment).where( + CalendarConflictJudgment.judgment_uid == judgment_uid, + CalendarConflictJudgment.user_id == user_id, + _organization_filter(organization_id), + CalendarConflictJudgment.workspace_id == workspace_id, + ) + if for_update: + # Serialize concurrent corrections to the same judgment: without this, + # two concurrent apply_correction calls can both read the same prior + # state, both record a "before" snapshot that matches, and race on + # which correction's decision/status the row ends up with. + stmt = stmt.with_for_update() + result = await db.execute(stmt) + judgment = result.scalar_one_or_none() + if judgment is None: + raise CalendarConflictJudgmentNotFoundError( + "Calendar conflict judgment is outside the requested scope" + ) + return judgment + + +def _judgment_snapshot(judgment: CalendarConflictJudgment) -> dict[str, Any]: + return { + "decision_code": judgment.decision_code, + "reason_code": judgment.reason_code, + "recommended_action": judgment.recommended_action, + "status_code": judgment.status_code, + } + + +async def apply_correction( + db: AsyncSession, + *, + judgment_uid: str, + user_id: str, + organization_id: str | None, + workspace_id: str, + actor_user_id: str, + correction_action: str, + decision_code: str | None, + status_code: str, + rationale: str | None, +) -> CalendarConflictCorrection: + """Record a human override/confirmation of a persisted conflict judgment.""" + if status_code not in ALLOWED_STATUS_CODES: + raise CalendarConflictUnsupportedValueError( + UNSUPPORTED_STATUS_CODE_ERROR_CODE, + f"Unsupported calendar conflict status_code: {status_code!r}", + ) + if decision_code is not None and decision_code not in ALLOWED_DECISION_CODES: + raise CalendarConflictUnsupportedValueError( + UNSUPPORTED_DECISION_CODE_ERROR_CODE, + f"Unsupported calendar conflict decision_code: {decision_code!r}", + ) + validate_correction_coherence(status_code=status_code, decision_code=decision_code) + + judgment = await _get_scoped_judgment( + db, + judgment_uid=judgment_uid, + user_id=user_id, + organization_id=organization_id, + workspace_id=workspace_id, + for_update=True, + ) + before_json = _judgment_snapshot(judgment) + if decision_code is not None and decision_code != judgment.decision_code: + # Replace reason_code/recommended_action together with decision_code + # so a later read can never pair a corrected decision with the + # original decision's now-stale reason and instruction. The original + # values are never lost -- they are exactly what before_json above + # already captured. recommended_action is restated from the policy's + # own canonical mapping, never from rationale: rationale explains why + # a human overrode the decision, it is not forward-looking scheduling + # guidance, and the two must never be conflated. + # + # Gated on an actual change (not just decision_code is not None): an + # "override" that repeats the judgment's current decision is a no-op + # on the decision itself, and must not wipe out the original, + # still-accurate reason_code/recommended_action for no real reason. + judgment.decision_code = decision_code + judgment.reason_code = CORRECTED_DECISION_REASON_CODE + judgment.recommended_action = default_recommended_action(decision_code) + judgment.status_code = status_code + judgment.updated_at = _utcnow() + after_json = _judgment_snapshot(judgment) + + correction = CalendarConflictCorrection( + judgment=judgment, + user_id=user_id, + organization_id=organization_id, + workspace_id=workspace_id, + actor_user_id=actor_user_id, + correction_action=correction_action, + before_json=before_json, + after_json=after_json, + correction_rationale=rationale, + ) + db.add(correction) + await db.flush() + return correction diff --git a/backend/services/calendar_conflict_policy.py b/backend/services/calendar_conflict_policy.py index 6e1a58546..287a5ce49 100644 --- a/backend/services/calendar_conflict_policy.py +++ b/backend/services/calendar_conflict_policy.py @@ -34,6 +34,12 @@ "calendar_proposed_source_missing", ] +# The bounded existing-commitment batch size every caller enforces: the REST +# endpoint (api/calendar_conflicts.py), the Noema agent tool +# (services/noema_agent.py), and any future caller. A single shared constant +# so the two enforcement points cannot silently drift apart. +MAX_EXISTING_COMMITMENTS = 500 + _STATUS_PRIORITY: dict[str, int] = { "desired": 1, "tentative": 2, @@ -106,6 +112,31 @@ class CalendarConflictDecision: policy_version: str = "status-weighted-v1" +_DEFAULT_RECOMMENDED_ACTIONS: dict[DecisionCode, str] = { + "available": "Proceed with scheduling.", + "blocked": ( + "Choose another time or explicitly resolve the equal/higher-priority " + "conflict first." + ), + "review_required": ( + "Review and explicitly reschedule or accept the lower-priority conflict " + "before proceeding." + ), +} + + +def default_recommended_action(decision_code: DecisionCode) -> str: + """Return the canonical next-action text this policy pairs with a decision. + + The single source of truth for decision_code -> recommended_action, so a + human-corrected judgment that changes decision_code (see + services/calendar_conflict_judgment_service.py) can restate a coherent + recommended_action instead of drifting from what this policy itself would + have said for that decision. + """ + return _DEFAULT_RECOMMENDED_ACTIONS[decision_code] + + def _require_timezone_aware(value: datetime.datetime) -> None: """Reject local/naive timestamps whose absolute instant is ambiguous.""" if value.tzinfo is None or value.utcoffset() is None: @@ -174,7 +205,7 @@ def evaluate_calendar_conflicts( decision_code="available", reason_code="no_overlapping_commitment", conflicts=(), - recommended_action="Proceed with scheduling.", + recommended_action=default_recommended_action("available"), ) conflicts = tuple( @@ -194,7 +225,7 @@ def evaluate_calendar_conflicts( decision_code="available", reason_code="no_overlapping_commitment", conflicts=(), - recommended_action="Proceed with scheduling.", + recommended_action=default_recommended_action("available"), ) proposed_priority = _STATUS_PRIORITY[proposed.status] @@ -206,18 +237,12 @@ def evaluate_calendar_conflicts( decision_code="blocked", reason_code="equal_or_higher_priority_conflict", conflicts=conflicts, - recommended_action=( - "Choose another time or explicitly resolve the equal/higher-priority " - "conflict first." - ), + recommended_action=default_recommended_action("blocked"), ) return CalendarConflictDecision( decision_code="review_required", reason_code="lower_priority_conflict_requires_explicit_resolution", conflicts=conflicts, - recommended_action=( - "Review and explicitly reschedule or accept the lower-priority conflict " - "before proceeding." - ), + recommended_action=default_recommended_action("review_required"), ) diff --git a/backend/services/content_graph/__init__.py b/backend/services/content_graph/__init__.py index 917e90a27..0c2d23c13 100644 --- a/backend/services/content_graph/__init__.py +++ b/backend/services/content_graph/__init__.py @@ -1,11 +1,12 @@ from .models import ContentNode, ContentSegment, ParseResult, PdfDomSection -from .parser import parse_content, parse_pdf_dom +from .parser import content_graph_source_record_uid, parse_content, parse_pdf_dom __all__ = [ "ContentNode", "ContentSegment", "ParseResult", "PdfDomSection", + "content_graph_source_record_uid", "parse_content", "parse_pdf_dom", ] diff --git a/backend/services/content_graph/parser.py b/backend/services/content_graph/parser.py index a134eee34..26a25b140 100644 --- a/backend/services/content_graph/parser.py +++ b/backend/services/content_graph/parser.py @@ -297,6 +297,19 @@ def _emit_node_and_segment(self, pending: _PendingHtmlNode) -> None: ) +def content_graph_source_record_uid(prefix: str, *parts: str) -> str: + """Build the one canonical ``source_record_uid`` for a content-graph source. + + Every pipeline stage that indexes content into the content graph (initial + email import, attachment reparse) must call this instead of hashing its + own identity string, so the same logical source always resolves to the + same ``source_record_uid`` no matter which stage indexed it. + """ + payload = "\x00".join(str(part) for part in parts) + digest = hashlib.sha256(payload.encode("utf-8", errors="surrogatepass")).hexdigest() + return f"{prefix}:{digest[:32]}" + + def parse_content( *, source_kind: str, diff --git a/backend/services/email_import_service.py b/backend/services/email_import_service.py index ddfa350fd..e68d2ec4b 100644 --- a/backend/services/email_import_service.py +++ b/backend/services/email_import_service.py @@ -30,7 +30,11 @@ BatchEmbeddingPartial, try_batch_import_embeddings, ) -from services.content_graph import ParseResult, parse_content +from services.content_graph import ( + ParseResult, + content_graph_source_record_uid, + parse_content, +) from services.email_dedupe_service import strong_email_fingerprint from services.email_parser import EmailData, parse_eml_bytes from services.embedding import ( @@ -216,13 +220,14 @@ async def _find_existing_email( *, user_id: str, organization_id: str, + workspace_id: str, message_id: str, fingerprint: str, ) -> Email | None: message_lookup_values = {message_id, f"<{message_id}>"} result = await session.execute( select(Email).where( - *Email.owner_filters(user_id, organization_id), + *Email.owner_filters(user_id, organization_id, workspace_id), or_( Email.message_id.in_(message_lookup_values), Email.fingerprint == fingerprint, @@ -233,11 +238,25 @@ async def _find_existing_email( async def _owner_email_import_count( - session: AsyncSession, *, user_id: str, organization_id: str + session: AsyncSession, + *, + user_id: str, + organization_id: str, ) -> int: + # Owner-wide on purpose: MAX_IMPORT_EMAILS_PER_OWNER and the advisory + # quota lock this feeds are both scoped to (user_id, organization_id) + # only, not to a single workspace -- Email.owner_filters() additionally + # requires workspace_id, which would let each workspace the same owner + # imports through grant another full allowance. + organization_filter = ( + Email.organization_id == organization_id + if organization_id is not None + else Email.organization_id.is_(None) + ) count = await session.scalar( select(func.count(Email.id)).where( - *Email.owner_filters(user_id, organization_id) + Email.user_id == user_id, + organization_filter, ) ) return int(count or 0) @@ -318,33 +337,54 @@ async def _extract_and_generate_embeddings( ) fitted_embeddings: list[list[float]] = [] for source_text in source_texts: - source_chunks = chunk_text(source_text) - if not source_chunks: - fitted_embeddings.append(_zero_embedding()) - continue - - vector_sum: list[float] | None = None - vector_count = 0 - for start in range(0, len(source_chunks), MAX_EMBEDDING_CHUNKS_PER_WINDOW): - chunk_embeddings = await _generate_import_embeddings( - source_chunks[start : start + MAX_EMBEDDING_CHUNKS_PER_WINDOW], + fitted_embeddings.append( + await generate_source_embedding( + source_text, embedding_provider=embedding_provider, batch_context=batch_context, ) - for embedding in chunk_embeddings: - if vector_sum is None: - vector_sum = [0.0] * len(embedding) - for index, value in enumerate(embedding): - vector_sum[index] += value - vector_count += 1 - fitted_embeddings.append( - [value / vector_count for value in vector_sum] - if vector_sum and vector_count - else _zero_embedding() ) return attachment_payloads, fitted_embeddings +async def generate_source_embedding( + source_text: str, + *, + embedding_provider: EmailImportEmbeddingProvider | None, + batch_context: "EmailImportBatchContext | None" = None, +) -> list[float]: + """Chunk, embed in bounded windows, and average one source vector. + + Public (not ``_``-prefixed): ``attachment_reparse_worker.py`` imports + this cross-module, alongside ``content_graph_source_record_uid`` and + ``append_knowledge_graph_edges`` -- the module boundary stays consistent + when every cross-module helper is public (CodeRabbit, naruon#1501). + """ + source_chunks = chunk_text(source_text) + if not source_chunks: + return _zero_embedding() + + vector_sum: list[float] | None = None + vector_count = 0 + for start in range(0, len(source_chunks), MAX_EMBEDDING_CHUNKS_PER_WINDOW): + chunk_embeddings = await _generate_import_embeddings( + source_chunks[start : start + MAX_EMBEDDING_CHUNKS_PER_WINDOW], + embedding_provider=embedding_provider, + batch_context=batch_context, + ) + for embedding in chunk_embeddings: + if vector_sum is None: + vector_sum = [0.0] * len(embedding) + for index, value in enumerate(embedding): + vector_sum[index] += value + vector_count += 1 + return ( + [value / vector_count for value in vector_sum] + if vector_sum and vector_count + else _zero_embedding() + ) + + def _build_email_object( *, parsed: EmailData, @@ -356,10 +396,13 @@ def _build_email_object( persisted_date: datetime.datetime, attachment_payloads: list[dict], fitted_embeddings: list[list[float]], + workspace_id: str | None = None, ) -> tuple[Email, int]: + resolved_workspace_id = workspace_id or f"workspace-{organization_id}" email_obj = Email( user_id=user_id, organization_id=organization_id, + workspace_id=resolved_workspace_id, message_id=message_id, thread_id=thread_id, fingerprint=fingerprint, @@ -418,7 +461,11 @@ def _build_email_object( message_id=message_id, attachment_payloads=attachment_payloads, ) - _append_knowledge_graph_edges(email_obj) + append_knowledge_graph_edges( + nodes=email_obj.content_nodes, + segments=email_obj.content_segments, + email_obj=email_obj, + ) return email_obj, attachment_count @@ -459,7 +506,7 @@ def _append_email_content_graph( ) -> None: body_parse_result = parse_content( source_kind="email_body", - source_record_uid=_content_graph_source_record_uid("email", message_id), + source_record_uid=content_graph_source_record_uid("email", message_id), content=str(parsed.get("body_parse_content") or parsed.get("body") or ""), content_type=str(parsed.get("body_content_type") or "text/plain"), display_name="Email body", @@ -485,7 +532,7 @@ def _append_email_content_graph( continue attachment_parse_result = parse_content( source_kind="attachment", - source_record_uid=_content_graph_source_record_uid( + source_record_uid=content_graph_source_record_uid( "attachment", message_id, str(attachment_index), @@ -552,11 +599,20 @@ def _append_parse_result_records( attachment_obj.content_segments.append(segment_record) -def _append_knowledge_graph_edges(email_obj: Email) -> None: +def append_knowledge_graph_edges( + *, + nodes: list[ContentNodeRecord], + segments: list[ContentSegmentRecord], + email_obj: Email | None = None, + attachment_obj: Attachment | None = None, +) -> None: + """Append the canonical content-graph topology for one indexed source.""" + if email_obj is None and attachment_obj is None: + raise ValueError("email_obj or attachment_obj is required") nodes_by_uid = { node.content_node_uid: node for node in sorted( - email_obj.content_nodes, + nodes, key=lambda item: ( item.source_kind, item.source_record_uid, @@ -580,6 +636,7 @@ def add_edge( ) -> None: nonlocal ordinal_index edge = KnowledgeGraphEdgeRecord( + email_id=attachment_obj.email_id if attachment_obj is not None else None, edge_uid=_knowledge_graph_edge_uid( edge_kind, _edge_endpoint_uid(source_node, source_segment), @@ -596,12 +653,11 @@ def add_edge( source_segment=source_segment, target_segment=target_segment, ) - email_obj.knowledge_graph_edges.append(edge) - attachment = _edge_attachment( - source_node=source_node, - target_node=target_node, - source_segment=source_segment, - target_segment=target_segment, + if email_obj is not None: + email_obj.knowledge_graph_edges.append(edge) + attachment = attachment_obj or _edge_attachment( + source_node=source_node, target_node=target_node, + source_segment=source_segment, target_segment=target_segment, ) if attachment is not None: attachment.knowledge_graph_edges.append(edge) @@ -627,7 +683,7 @@ def add_edge( list[ContentSegmentRecord], ] = defaultdict(list) for segment in sorted( - email_obj.content_segments, + segments, key=lambda item: ( item.source_kind, item.source_record_uid, @@ -736,12 +792,6 @@ def _knowledge_graph_edge_uid( return f"kgedge_{digest[:32]}" -def _content_graph_source_record_uid(prefix: str, *parts: str) -> str: - payload = "\x00".join(str(part) for part in parts) - digest = hashlib.sha256(payload.encode("utf-8", errors="surrogatepass")).hexdigest() - return f"{prefix}:{digest[:32]}" - - def _project_source_segments(email_obj: Email) -> list[ProjectSourceSegment]: """Snapshot the imported email's content segments as project source segments. @@ -797,14 +847,17 @@ async def _persist_project_graph_projection( *, user_id: str, organization_id: str, + workspace_id: str, embedding_provider: EmailImportEmbeddingProvider | None = None, ) -> None: """Best-effort projection of imported content segments into the project graph. Runs after the email is already committed. Flag-gated and defensive: any - failure is logged and rolled back so it never fails the email import. The - workspace scope mirrors the convention enforced by the project graph - repository (``workspace-``). + failure is logged and rolled back so it never fails the email import. + ``workspace_id`` must be the same resolved workspace the imported email + itself was stored under -- recomputing a default here would put the + email and its derived project-graph objects in different workspaces + whenever the caller resolved a non-default one. """ if not source_segments: return @@ -814,11 +867,6 @@ async def _persist_project_graph_projection( ) if not extraction.objects: return - workspace_id = ( - f"workspace-{organization_id}" - if organization_id - else f"workspace-{user_id}" - ) await persist_project_graph_projection( session, extraction=extraction, @@ -842,9 +890,11 @@ async def _import_single_eml( display_filename: str, user_id: str, organization_id: str, + workspace_id: str | None = None, embedding_provider: EmailImportEmbeddingProvider | None = None, batch_context: "EmailImportBatchContext | None" = None, ) -> EmailImportItemResult: + resolved_workspace_id = workspace_id or f"workspace-{organization_id}" try: content, parsed = await asyncio.to_thread(_read_and_parse_eml, eml_path) except EmailParseError as exc: @@ -868,6 +918,7 @@ async def _import_single_eml( session, user_id=user_id, organization_id=organization_id, + workspace_id=resolved_workspace_id, message_id=message_id, fingerprint=fingerprint, ) @@ -883,6 +934,7 @@ async def _import_single_eml( parsed, user_id=user_id, organization_id=organization_id, + workspace_id=resolved_workspace_id, ) attachment_payloads, fitted_embeddings = await _extract_and_generate_embeddings( @@ -893,6 +945,7 @@ async def _import_single_eml( parsed=parsed, user_id=user_id, organization_id=organization_id, + workspace_id=resolved_workspace_id, message_id=message_id, thread_id=thread_id, fingerprint=fingerprint, @@ -927,6 +980,7 @@ async def _import_single_eml( project_source_segments, user_id=user_id, organization_id=organization_id, + workspace_id=resolved_workspace_id, embedding_provider=embedding_provider, ) @@ -1160,8 +1214,10 @@ async def import_email_uploads( uploads: list[EmailImportUpload], user_id: str, organization_id: str, + workspace_id: str | None = None, embedding_provider: EmailImportEmbeddingProvider | None = None, ) -> EmailImportResult: + resolved_workspace_id = workspace_id or f"workspace-{organization_id}" lock_acquired = await _acquire_owner_import_quota_lock( session, user_id=user_id, organization_id=organization_id ) @@ -1173,7 +1229,9 @@ async def import_email_uploads( try: result = EmailImportResult() existing_email_count = await _owner_email_import_count( - session, user_id=user_id, organization_id=organization_id + session, + user_id=user_id, + organization_id=organization_id, ) remaining_quota = MAX_IMPORT_EMAILS_PER_OWNER - existing_email_count if remaining_quota <= 0: @@ -1224,6 +1282,7 @@ async def import_email_uploads( display_filename=display_filename, user_id=user_id, organization_id=organization_id, + workspace_id=resolved_workspace_id, embedding_provider=embedding_provider, batch_context=batch_context, ) diff --git a/backend/services/hybrid_retrieval/retrieval_channels.py b/backend/services/hybrid_retrieval/retrieval_channels.py index 27b01b375..92162070c 100644 --- a/backend/services/hybrid_retrieval/retrieval_channels.py +++ b/backend/services/hybrid_retrieval/retrieval_channels.py @@ -45,6 +45,15 @@ # {candidate, confirmed}; the exclusion list is defensive. _EXCLUDED_PROJECT_OBJECT_STATUS_CODES = ("dismissed", "rejected") +# Only a "parsed" attachment's `content` is genuine parsed text. Every other +# status (quarantined content-type mismatch, pdf_dom_recognition_pending +# awaiting the NewsDOM sidecar, reparse_pending/reparse_payload_invalid, +# unsupported_content_type, parse_size_limit_exceeded, ...) stores either a +# base64-encoded raw payload or an empty string -- neither is a legitimate +# search match, and the base64 payload in particular must never surface as +# if it were the attachment's real content. +_SEARCHABLE_ATTACHMENT_PARSE_STATUS = "parsed" + def _thread_key_expression(): normalized_thread_id = func.nullif( @@ -145,7 +154,9 @@ def build_lexical_attachment_statement( owner_filters=owner_filters, candidate_limit=candidate_limit, ) - return statement.join(Email, Attachment.email_id == Email.id) + return statement.join(Email, Attachment.email_id == Email.id).where( + Attachment.parse_status == _SEARCHABLE_ATTACHMENT_PARSE_STATUS + ) def build_lexical_content_segment_statement( @@ -228,4 +239,6 @@ def build_dense_attachment_statement( owner_filters=owner_filters, candidate_limit=candidate_limit, ) - return statement.join(Email, Attachment.email_id == Email.id) + return statement.join(Email, Attachment.email_id == Email.id).where( + Attachment.parse_status == _SEARCHABLE_ATTACHMENT_PARSE_STATUS + ) diff --git a/backend/services/imap_worker.py b/backend/services/imap_worker.py index d618f1d3c..73cbeecec 100644 --- a/backend/services/imap_worker.py +++ b/backend/services/imap_worker.py @@ -25,6 +25,7 @@ async def process_fetched_email( email_data: EmailData, user_id: str, organization_id: str | None, + workspace_id: str, owner_addresses: Iterable[str] | None = None, is_read: bool = True, ): @@ -59,8 +60,7 @@ async def process_fetched_email( # Check if duplicate stmt = select(Email).where( - Email.user_id == user_id, - Email.organization_id == (organization_id if organization_id else None), + *Email.owner_filters(user_id, organization_id, workspace_id), Email.fingerprint == fingerprint, ) result = await session.execute(stmt) @@ -74,12 +74,21 @@ async def process_fetched_email( return existing_email thread_id = await assign_thread_id( - session, email_data, user_id=user_id, organization_id=organization_id + session, + email_data, + user_id=user_id, + organization_id=organization_id, + workspace_id=workspace_id, ) new_email = Email( user_id=user_id, organization_id=organization_id or None, + # Mirrors the workspace_id convention used across this codebase + # (services/email_import_service.py, services/project_graph/): + # workspace-, falling back to workspace- + # for an org-less/personal scope. + workspace_id=workspace_id, message_id=email_data.get("message_id", ""), thread_id=thread_id, fingerprint=fingerprint, @@ -98,10 +107,39 @@ async def process_fetched_email( await extract_knowledge_from_self_sent(session, new_email, owner_addresses) return new_email + logger = logging.getLogger(__name__) MAX_IMAP_FETCH_MESSAGES = 10 +async def resolve_unambiguous_workspace_id( + session, user_id: str, organization_id: str | None +) -> str | None: + """Resolve the single workspace an owner's already-imported mail belongs to. + + Background mail sync (IMAP, POP3) has no signed session and therefore no + independently authoritative workspace claim to thread through -- the only + evidence available is whatever workspace this owner's mail has already + been imported under. Returns ``None`` (fail closed; the caller must skip + the sync) when zero or more than one distinct workspace is found, so a + first-run or genuinely multi-workspace mailbox is never guessed at. + """ + workspace_ids = list( + await session.scalars( + select(Email.workspace_id) + .where( + Email.user_id == user_id, + Email.organization_id == organization_id, + ) + .distinct() + .limit(2) + ) + ) + if len(workspace_ids) != 1: + return None + return workspace_ids[0] + + def flags_indicate_seen(fetch_data) -> bool: """True when an IMAP FETCH response's FLAGS envelope contains ``\\Seen``. @@ -112,7 +150,11 @@ def flags_indicate_seen(fetch_data) -> bool: for item in fetch_data or []: parts = item if isinstance(item, (tuple, list)) else (item,) for part in parts: - raw = part if isinstance(part, bytes) else str(part).encode("utf-8", "replace") + raw = ( + part + if isinstance(part, bytes) + else str(part).encode("utf-8", "replace") + ) upper = raw.upper() if b"FLAGS" in upper and b"\\SEEN" in upper: return True @@ -123,6 +165,7 @@ def flags_indicate_seen(fetch_data) -> bool: class ImapSyncConfig: user_id: str organization_id: str | None + workspace_id: str imap_server: str imap_port: int imap_username: str | None @@ -187,10 +230,11 @@ async def _sync(self): TenantConfig.imap_port.isnot(None), ) ) - configs = [ + tenant_configs = [ ImapSyncConfig( user_id=row.user_id, organization_id=row.organization_id, + workspace_id="", imap_server=str(row.imap_server), imap_port=int(row.imap_port), imap_username=row.imap_username, @@ -198,6 +242,27 @@ async def _sync(self): ) for row in result ] + configs = [] + for config in tenant_configs: + workspace_id = await resolve_unambiguous_workspace_id( + session, config.user_id, config.organization_id + ) + if workspace_id is None: + logger.info( + "Skipping IMAP sync because an unambiguous workspace is unavailable" + ) + continue + configs.append( + ImapSyncConfig( + user_id=config.user_id, + organization_id=config.organization_id, + workspace_id=workspace_id, + imap_server=config.imap_server, + imap_port=config.imap_port, + imap_username=config.imap_username, + imap_password=config.imap_password, + ) + ) tasks = [] for config in configs: @@ -217,7 +282,7 @@ async def _sync_tenant(self, config: TenantConfig | ImapSyncConfig): config.user_id, ) return 0 - + logger.info( "Connecting to IMAP server %s:%s for user %s", imap_server, @@ -252,6 +317,7 @@ async def _fetch_messages( if imap_server is None or imap_port is None: imap_server, imap_port = self._validated_destination(config) import ssl + ssl_context = ssl.create_default_context() imap_client = aioimaplib.IMAP4_SSL( imap_server, imap_port, ssl_context=ssl_context @@ -316,6 +382,12 @@ async def _import_messages( imported_count = 0 owner_addresses = [config.imap_username] if config.imap_username else None + workspace_id = getattr(config, "workspace_id", "") + if not workspace_id: + logger.info( + "Skipping IMAP sync because an authoritative workspace is unavailable" + ) + return 0 async with AsyncSessionLocal() as session: try: for raw_message, is_read in messages: @@ -332,6 +404,7 @@ async def _import_messages( email_data, config.user_id, config.organization_id, + workspace_id, owner_addresses=owner_addresses, is_read=is_read, ) @@ -388,6 +461,4 @@ def _looks_like_rfc822_message(self, value: bytes) -> bool: header_block = value.split(b"\r\n\r\n", maxsplit=1)[0] if header_block == value: header_block = value.split(b"\n\n", maxsplit=1)[0] - return b":" in header_block and ( - b"\r\n\r\n" in value or b"\n\n" in value - ) + return b":" in header_block and (b"\r\n\r\n" in value or b"\n\n" in value) diff --git a/backend/services/newsdom_worker.py b/backend/services/newsdom_worker.py index 26e4cffbb..cb5e85605 100644 --- a/backend/services/newsdom_worker.py +++ b/backend/services/newsdom_worker.py @@ -13,12 +13,13 @@ from __future__ import annotations import asyncio +import datetime import logging import random from collections.abc import Awaitable, Callable -from sqlalchemy import bindparam, func, select -from sqlalchemy.ext.asyncio import AsyncSession +from sqlalchemy import and_, bindparam, case, func, or_, select +from sqlalchemy.ext.asyncio import AsyncConnection, AsyncSession from sqlalchemy.orm import selectinload from db.models import ( @@ -28,7 +29,7 @@ Document, Email, ) -from db.session import AsyncSessionLocal +from db.session import AsyncSessionLocal, engine from services.attachment_parser import decode_deferred_attachment_payload from services.content_graph import ParseResult from services.newsdom_client import ( @@ -186,6 +187,16 @@ async def recognize_document_pdf( DEFAULT_NEWSDOM_BATCH_LIMIT = 10 NEWSDOM_SWEEP_LOCK_NAMESPACE = "naruon-newsdom-recognition-sweep" MAX_STARTUP_JITTER_SECONDS = 30 +# A row already past the forward cursor can still be explicitly re-marked +# pending later (e.g. POST .../pdf-dom-recognition-intent re-triggering an +# already-resolved document) -- the cursor+retry-id design alone can never +# discover that, since retry_ids only tracks rows this worker itself has +# already seen and found unresolved. Forcing a full rescan (cursor -> None) +# on this cadence bounds how stale such a row can get; it's always safe +# regardless of cadence, since the parse_status/document_status filter +# already excludes every row that's actually resolved, so a full rescan +# only ever costs a wider (not wrong) result set. See _sweep_attachments. +FULL_RESCAN_EVERY_N_SWEEPS = 20 # Per-item processing outcomes. RESULT_RECOGNIZED = "recognized" @@ -326,20 +337,35 @@ async def process_pending_document( return RESULT_RECOGNIZED -def _session_uses_postgresql(session: AsyncSession) -> bool: - """Return whether advisory-lock SQL is supported by the session bind.""" - try: - bind = session.get_bind() - except Exception: - return False - return getattr(getattr(bind, "dialect", None), "name", None) == "postgresql" - - -async def _try_acquire_sweep_lease(session: AsyncSession) -> bool | None: - """Become the sweep leader for this cycle (None when not PostgreSQL).""" - if not _session_uses_postgresql(session): - return None - acquired = await session.scalar( +def _engine_uses_postgresql() -> bool: + """Return whether advisory-lock SQL is supported by the configured engine.""" + return engine.dialect.name == "postgresql" + + +async def _try_acquire_sweep_lease(connection: AsyncConnection) -> bool: + """Become the sweep leader for this cycle on this dedicated connection. + + Takes an ``AsyncConnection`` rather than the per-item ``AsyncSession`` + deliberately: ``AsyncSession.commit()``/``rollback()`` release their + underlying connection back to the pool on every call (SQLAlchemy's normal + "connectionless execution" behavior), so acquiring the lock through that + session risks releasing it -- or never releasing it -- from a *different* + physical backend connection than the one PostgreSQL actually granted the + advisory lock to. Advisory locks are scoped to the acquiring backend + session, so a mismatched unlock is a silent no-op and the lock stays held + (blocking every replica's sweep) until the stray connection is eventually + recycled or closed. Mirrors + ``services.attachment_reparse_worker._try_acquire_sweep_lease``. + + Sets AUTOCOMMIT on this connection first: without it, the advisory-lock + SELECT below opens an implicit transaction that would otherwise stay + open and idle on this connection for the whole sweep (every other + statement runs through the separate per-item session), risking + PostgreSQL's ``idle_in_transaction_session_timeout`` killing this + connection mid-sweep and silently dropping the lease. + """ + await connection.execution_options(isolation_level="AUTOCOMMIT") + acquired = await connection.scalar( select( func.pg_try_advisory_lock( func.hashtext(bindparam("namespace_key")), @@ -351,9 +377,9 @@ async def _try_acquire_sweep_lease(session: AsyncSession) -> bool | None: return bool(acquired) -async def _release_sweep_lease(session: AsyncSession) -> None: - """Release the PostgreSQL advisory lock for a recognition sweep.""" - await session.scalar( +async def _release_sweep_lease(connection: AsyncConnection) -> None: + """Release the PostgreSQL advisory lock on the connection that holds it.""" + await connection.scalar( select( func.pg_advisory_unlock( func.hashtext(bindparam("namespace_key")), @@ -371,7 +397,12 @@ class NewsdomRecognitionWorker: advisory-lock lease so only one replica sweeps per cycle, and per-item error isolation. Items whose organization has no active NewsDOM provider are left pending (they recognize once a provider is configured); unusable - payloads/responses are marked failed rather than parsed. + payloads/responses are marked failed rather than parsed. The lease is + acquired and released on one dedicated connection held open for the + whole sweep, and each row is re-fetched fresh by id before processing -- + the same two corrections applied to + :class:`services.attachment_reparse_worker.AttachmentReparseWorker`, + which shares this same lease/cursor/per-item-isolation design. """ def __init__( @@ -390,7 +421,14 @@ def __init__( self._task: asyncio.Task | None = None self._is_running = False self._attachment_cursor: int | None = None - self._document_cursor: str | None = None + self._document_cursor: tuple[datetime.datetime, str] | None = None + # Rows a past sweep saw as still-pending (RESULT_PENDING or a raised + # exception). Retried every sweep via an explicit id filter, + # independent of the forward cursor below -- see _sweep_attachments. + self._attachment_retry_ids: set[int] = set() + self._document_retry_ids: set[str] = set() + self._attachment_sweep_count = 0 + self._document_sweep_count = 0 async def start(self) -> None: """Start the recognition loop once.""" @@ -439,29 +477,97 @@ async def _run_loop(self) -> None: break async def _sweep(self) -> None: - """Process one leased attachment and document sweep.""" - async with AsyncSessionLocal() as session: - lease = await _try_acquire_sweep_lease(session) - if lease is False: + """Process one leased attachment and document sweep. + + The advisory lock (when the engine supports it) is acquired and + released on one dedicated connection held open for the whole sweep -- + never through the per-item ``AsyncSession`` used by + ``_sweep_attachments``/``_sweep_documents``. See + ``_try_acquire_sweep_lease`` for why that distinction matters. + """ + if not _engine_uses_postgresql(): + async with AsyncSessionLocal() as session: + await self._sweep_attachments(session) + await self._sweep_documents(session) + return + async with engine.connect() as lock_connection: + if not await _try_acquire_sweep_lease(lock_connection): logger.debug( "NewsDOM recognition sweep skipped: another replica holds " "the lease." ) return try: - await self._sweep_attachments(session) - await self._sweep_documents(session) + async with AsyncSessionLocal() as session: + await self._sweep_attachments(session) + await self._sweep_documents(session) finally: - if lease is True: - await _release_sweep_lease(session) + await _release_sweep_lease(lock_connection) async def _sweep_attachments(self, session: AsyncSession) -> None: - """Process a bounded, starvation-free batch of pending attachments.""" + """Process a bounded, starvation-free batch of pending attachments. + + Two independent pieces of state make progress unconditional: + + * ``_attachment_cursor`` is a forward-scan position that only ever + advances (to the highest id seen in a batch) -- so rows the + worker has never looked at are always reached eventually, no + matter how many earlier rows stay stuck. + * ``_attachment_retry_ids`` is the set of ids a past sweep saw as + still-pending (raised, or returned ``RESULT_PENDING`` -- e.g. no + active provider configured for its organization yet). These are + retried every sweep via an explicit ``id IN (...)`` filter, + independent of the cursor, and dropped from the set once they + resolve. + + An earlier version capped the cursor itself at the first unresolved + row instead of tracking a separate retry set. That avoided losing + track of one stuck row, but pinned the *entire batch window* behind + it: once more than ``batch_limit`` consecutive rows were + permanently stuck (e.g. one organization bulk-imports a burst of + PDFs before ever configuring a provider), the same stuck rows filled + every batch forever and nothing past them was ever reached -- + confirmed by reproduction, not merely suspected (300 rows, one + permanently-stuck row: converges; 60 consecutive permanently-stuck + rows with batch_limit=50: never converges). Decoupling "retry this + stuck row" from "where the forward scan has reached" fixes both + shapes of starvation at once. Each row is also re-fetched fresh by + id rather than reusing the bulk-loaded instance: + ``AsyncSession.rollback()`` expires every object already loaded in + this session, and ``process_pending_attachment`` reads + attachment/email attributes synchronously, so a stale, expired + instance from an earlier item's failure would raise on that read + instead of just isolating the one failure. Mirrors + ``services.attachment_reparse_worker.AttachmentReparseWorker._sweep_attachments``. + + Neither piece of state can discover a row that gets explicitly + re-marked pending *after* the cursor already passed it -- e.g. an + already-recognized attachment whose organization requests it be + looked at again. It was never seen as unresolved by this worker (so + it's not in the retry set) and its id no longer satisfies + ``id > cursor``. Every ``FULL_RESCAN_EVERY_N_SWEEPS``-th sweep forces + a full rescan (cursor reset to ``None``) to bound how stale such a + row can get; this is always safe regardless of cadence, since the + ``parse_status`` filter already excludes every row that's actually + resolved, so a full rescan only ever widens the candidate set. + """ + self._attachment_sweep_count += 1 + if self._attachment_sweep_count % FULL_RESCAN_EVERY_N_SWEEPS == 0: + self._attachment_cursor = None rows = await self._load_pending_attachments(session) - if rows: - self._attachment_cursor = rows[-1].id - for attachment in rows: + batch_ids = [attachment.id for attachment in rows] + processed_ids: set[int] = set() + unresolved_ids: set[int] = set() + for attachment_id in batch_ids: + processed_ids.add(attachment_id) try: + attachment = await session.get( + Attachment, + attachment_id, + options=[selectinload(Attachment.email)], + ) + if attachment is None: + continue result = await process_pending_attachment( session=session, attachment=attachment, @@ -469,29 +575,73 @@ async def _sweep_attachments(self, session: AsyncSession) -> None: request_fn=self._request_fn, ) await session.commit() - if result != RESULT_PENDING: + if result == RESULT_PENDING: + unresolved_ids.add(attachment_id) + else: logger.info( "NewsDOM attachment %s recognition result: %s", - attachment.id, + attachment_id, result, ) except Exception: await session.rollback() logger.error( "NewsDOM attachment %s recognition raised.", - getattr(attachment, "id", "?"), + attachment_id, exc_info=True, ) + unresolved_ids.add(attachment_id) + if batch_ids: + highest_seen = max(batch_ids) + self._attachment_cursor = ( + highest_seen + if self._attachment_cursor is None + else max(self._attachment_cursor, highest_seen) + ) + self._attachment_retry_ids = ( + self._attachment_retry_ids - processed_ids + ) | unresolved_ids - def _pending_attachment_statement(self, after_id: int | None): - """Build the next deterministic attachment batch query.""" + def _pending_attachment_statement( + self, after_id: int | None, retry_ids: set[int] + ): + """Build the next deterministic attachment batch query. + + Selects rows past the forward cursor OR still tracked in + ``retry_ids``, so a persistently-stuck row keeps getting retried + without pinning the forward scan behind it. When both are present, + never-yet-seen forward rows sort before already-tracked retry rows + (a ``CASE`` bucket ahead of the plain id order) -- otherwise, once + ``retry_ids`` alone reached ``batch_limit`` in size, ``ORDER BY id`` + would let those (necessarily smaller, already-seen) ids win every + slot in the ``LIMIT``, starving forward progress right back into the + bug this is fixing. This does mean retry rows can wait longer under + sustained, saturating forward load -- a deliberate trade-off, since + that only delays retrying a handful of already-known-stuck rows, + never blocks the rest of the pipeline the way the reverse priority + did. + """ statement = select(Attachment).where( Attachment.parse_status == PDF_DOM_RECOGNITION_PENDING_STATUS ) + # retry_ids only narrows the result when there's a cursor to narrow + # *against* -- with no cursor yet (after_id is None, including a + # forced full rescan), every pending row already qualifies, and + # restricting to just retry_ids here would incorrectly hide + # everything else that's pending. if after_id is not None: - statement = statement.where(Attachment.id > after_id) + conditions = [Attachment.id > after_id] + if retry_ids: + conditions.append(Attachment.id.in_(retry_ids)) + statement = statement.where( + conditions[0] if len(conditions) == 1 else or_(*conditions) + ) + order_columns = [] + if after_id is not None and retry_ids: + order_columns.append(case((Attachment.id > after_id, 0), else_=1)) + order_columns.append(Attachment.id) return ( - statement.order_by(Attachment.id) + statement.order_by(*order_columns) .options(selectinload(Attachment.email)) .limit(self.batch_limit) ) @@ -499,36 +649,68 @@ def _pending_attachment_statement(self, after_id: int | None): async def _load_pending_attachments( self, session: AsyncSession ) -> list[Attachment]: - """Load after the last attempted row, wrapping at the table tail. - - Advancing over rows that remain pending prevents an unconfigured - organization's first batch from permanently starving configured rows. - """ - rows = ( + """Load the next batch: past the cursor, plus any known-stuck rows.""" + return ( ( await session.execute( - self._pending_attachment_statement(self._attachment_cursor) + self._pending_attachment_statement( + self._attachment_cursor, self._attachment_retry_ids + ) ) ) .scalars() .all() ) - if not rows and self._attachment_cursor is not None: - self._attachment_cursor = None - rows = ( - (await session.execute(self._pending_attachment_statement(None))) - .scalars() - .all() - ) - return rows async def _sweep_documents(self, session: AsyncSession) -> None: - """Process a bounded, starvation-free batch of pending documents.""" + """Process a bounded, starvation-free batch of pending documents. + + Same design as ``_sweep_attachments`` -- a monotonically-advancing + forward cursor plus a persistent ``_document_retry_ids`` set for + rows that raised or came back ``RESULT_PENDING`` (no active provider + configured for its organization yet), retried every sweep + independent of the cursor. See ``_sweep_attachments`` for why a + cursor that instead capped itself at the first unresolved row let + more than ``batch_limit`` consecutive stuck rows starve everything + past them -- the same fix applies here. + + Unlike ``Attachment.id`` (an auto-incrementing integer, monotonic + with insertion order), ``Document.document_id`` defaults to a random + UUID (``f"doc_{uuid.uuid4().hex}"``) -- not monotonic at all. A + document inserted *after* the cursor already advanced can still sort + lexicographically *below* it; since it was never seen before, it is + not in the retry set either, so it would never be selected again -- + confirmed by Devin Review and by direct inspection of the model. + ``_document_cursor`` is therefore a ``(created_at, document_id)`` + tuple: ``created_at`` (a Python-side timestamp set at insert time) + is genuinely monotonic with insertion order, and ``document_id`` + only breaks ties between rows inserted in the same instant. + + Neither the cursor nor the retry set can discover a document + explicitly re-marked pending after the cursor already passed it -- + e.g. ``POST .../pdf-dom-recognition-intent`` re-triggering + recognition on an already-resolved document. Every + ``FULL_RESCAN_EVERY_N_SWEEPS``-th sweep forces a full rescan + (cursor reset to ``None``) to bound how stale such a row can get; + see ``_sweep_attachments`` for why this is always safe. + """ + self._document_sweep_count += 1 + if self._document_sweep_count % FULL_RESCAN_EVERY_N_SWEEPS == 0: + self._document_cursor = None rows = await self._load_pending_documents(session) - if rows: - self._document_cursor = rows[-1].document_id - for document in rows: + processed_ids: set[str] = set() + unresolved_ids: set[str] = set() + # (created_at, document_id) pairs captured from the freshly re-fetched + # instance only -- never from the bulk-loaded `rows` list, which may + # hold an instance a *prior* item's rollback already expired. + seen_pairs: list[tuple[datetime.datetime, str]] = [] + for document_id in [document.document_id for document in rows]: + processed_ids.add(document_id) try: + document = await session.get(Document, document_id) + if document is None: + continue + seen_pairs.append((document.created_at, document.document_id)) result = await process_pending_document( session=session, document=document, @@ -536,45 +718,88 @@ async def _sweep_documents(self, session: AsyncSession) -> None: request_fn=self._request_fn, ) await session.commit() - if result != RESULT_PENDING: + if result == RESULT_PENDING: + unresolved_ids.add(document_id) + else: logger.info( "NewsDOM document %s recognition result: %s", - document.document_id, + document_id, result, ) except Exception: await session.rollback() logger.error( "NewsDOM document %s recognition raised.", - getattr(document, "document_id", "?"), + document_id, exc_info=True, ) + unresolved_ids.add(document_id) + if seen_pairs: + highest_seen = max(seen_pairs) + self._document_cursor = ( + highest_seen + if self._document_cursor is None + else max(self._document_cursor, highest_seen) + ) + self._document_retry_ids = ( + self._document_retry_ids - processed_ids + ) | unresolved_ids - def _pending_document_statement(self, after_id: str | None): - """Build the next deterministic workspace-document batch query.""" + def _pending_document_statement( + self, + after_cursor: tuple[datetime.datetime, str] | None, + retry_ids: set[str], + ): + """Build the next deterministic workspace-document batch query. + + Selects rows past the forward ``(created_at, document_id)`` cursor + OR still tracked in ``retry_ids``, and orders forward rows ahead of + retry rows when both are present, exactly like + ``_pending_attachment_statement`` (see its docstring for why the + priority must run that way). The forward comparison is a row-value + (tuple) comparison -- ``created_at`` first, ``document_id`` only to + break same-instant ties -- so it stays correct however + ``document_id`` sorts. + """ statement = select(Document).where( Document.document_status == PDF_DOM_RECOGNITION_PENDING_STATUS ) - if after_id is not None: - statement = statement.where(Document.document_id > after_id) - return statement.order_by(Document.document_id).limit(self.batch_limit) + forward_condition = None + if after_cursor is not None: + after_created_at, after_document_id = after_cursor + forward_condition = or_( + Document.created_at > after_created_at, + and_( + Document.created_at == after_created_at, + Document.document_id > after_document_id, + ), + ) + # retry_ids only narrows the result when there's a cursor to narrow + # *against* -- see _pending_attachment_statement for why (identical + # reasoning, shared between both sweeps). + if forward_condition is not None: + conditions = [forward_condition] + if retry_ids: + conditions.append(Document.document_id.in_(retry_ids)) + statement = statement.where( + conditions[0] if len(conditions) == 1 else or_(*conditions) + ) + order_columns = [] + if forward_condition is not None and retry_ids: + order_columns.append(case((forward_condition, 0), else_=1)) + order_columns.extend([Document.created_at, Document.document_id]) + return statement.order_by(*order_columns).limit(self.batch_limit) async def _load_pending_documents(self, session: AsyncSession) -> list[Document]: - """Load after the last attempted document and wrap at the tail.""" - rows = ( + """Load the next batch: past the cursor, plus any known-stuck rows.""" + return ( ( await session.execute( - self._pending_document_statement(self._document_cursor) + self._pending_document_statement( + self._document_cursor, self._document_retry_ids + ) ) ) .scalars() .all() ) - if not rows and self._document_cursor is not None: - self._document_cursor = None - rows = ( - (await session.execute(self._pending_document_statement(None))) - .scalars() - .all() - ) - return rows diff --git a/backend/services/noema_agent.py b/backend/services/noema_agent.py index 99ab683df..ac69e4ba5 100644 --- a/backend/services/noema_agent.py +++ b/backend/services/noema_agent.py @@ -1,13 +1,31 @@ """Noema general agent. A general-purpose `Pydantic-AI `_ (MIT) agent that -reasons over the naruon workspace. It runs on the tenant's configured LLM -provider — resolved through :func:`resolve_runtime_llm_provider` from the -Fernet-encrypted provider records, never from ``os.getenv`` — and it is given a -small set of tools that plug into the existing service and runner seams: - -* **read/search mail** and **content-graph queries** are owner-scoped SQL reads. +reasons over the naruon workspace. Every chat-completion call is routed +through ``ContextualWisdomLab/contextual-orchestrator`` — resolved through +:func:`services.orchestrator_gateway.resolve_orchestrator_gateway` from the +tenant's own Fernet-encrypted gateway credential, never ``os.getenv`` and +never a direct tenant LLM-provider key. Production LLM routing belongs to +contextual-orchestrator; naruon owns the Noema tools/authorization/context, +not a second provider-routing authority. This is a distinct, tenant-scoped +credential from the one ``ContextualWisdomLab/.github``'s central +review-pipeline Noema uses (``scripts/ci/noema_review_gate.py``); naruon's +workspace data is never sent through that CI credential path. Per +``docs/CWL-MASTER-CONTEXT.md`` (``ContextualWisdomLab/.github``), Noema is +actually one shared agent runtime (Pydantic-AI/Codex-Python) consumed by +naruon, the ``.github`` CI review agent, and wardnet's AI SOC quarantine +sandbox — the per-deployment credential scoping here is a security choice +for this module, not a claim that the deployments are permanently separate +or share nothing beyond a name; see ``ContextualWisdomLab/naruon#1527`` for +the corrected reasoning (its ``docs/adr/0006-noema-bounded-context-separation.md`` +lives on that PR's own branch, not this one — this docstring cannot resolve +it as a local path until that PR merges). It is given a small set of tools +that plug into the existing service and runner seams: + +* **read/search mail** and **content-graph queries** are workspace-scoped SQL reads. * **task actions** update ``TicketTask`` rows and are audit-logged. +* **calendar conflict check** validates the proposal but fails closed until a + scoped authoritative provider-calendar read seam exists. * **writeback** is dispatched to the self-hosted runner (the ``write_caldav`` / ``write_webdav`` actions handled by :class:`SelfHostedConnector`), preserving naruon's opt-in-writeback and audit-logged contract. @@ -35,11 +53,15 @@ KnowledgeGraphEdgeRecord, TicketTask, ) -from services.llm_provider_selection import ( - RuntimeLLMProvider, - resolve_runtime_llm_provider, +from services.calendar_conflict_policy import ( + CalendarCommitment, + CalendarPolicyValidationError, ) from services.llm_provider_urls import build_llm_provider_http_client +from services.orchestrator_gateway import ( + OrchestratorGateway, + resolve_orchestrator_gateway, +) if TYPE_CHECKING: # pragma: no cover - typing only from pydantic_ai import Agent @@ -62,7 +84,9 @@ # Task statuses the agent is allowed to set. Anything else is refused so the LLM # cannot write arbitrary status codes into the tenant's tickets. -ALLOWED_TASK_STATUSES = frozenset({"open", "in_progress", "blocked", "done", "cancelled"}) +ALLOWED_TASK_STATUSES = frozenset( + {"open", "in_progress", "blocked", "done", "cancelled"} +) # Runner actions the agent may dispatch. These are exactly the writeback actions # handled by SelfHostedConnector. @@ -101,6 +125,7 @@ class NoemaAgentResult: notice: str | None = None provider_name: str | None = None tool_calls: tuple[str, ...] = () + error_code: str | None = None @property def ok(self) -> bool: @@ -153,7 +178,9 @@ async def tool_search_mail( deps.tool_calls.append("search_mail") query = (query or "").strip() bounded = max(1, min(int(limit or 1), _MAX_MAIL_RESULTS)) - statement = select(Email).where(*Email.owner_filters(deps.user_id, deps.organization_id)) + statement = select(Email).where( + *Email.owner_filters(deps.user_id, deps.organization_id, deps.workspace_id), + ) if query: pattern = f"%{query}%" statement = statement.where( @@ -188,7 +215,9 @@ async def tool_read_mail(deps: NoemaAgentDeps, message_id: str) -> dict[str, Any return {"status": "error", "reason": "message_id is required"} statement = ( select(Email) - .where(*Email.owner_filters(deps.user_id, deps.organization_id)) + .where( + *Email.owner_filters(deps.user_id, deps.organization_id, deps.workspace_id) + ) .where(Email.message_id == message_id) .limit(1) ) @@ -220,7 +249,9 @@ async def tool_content_graph_query( email_result = await deps.session.execute( select(Email.id) - .where(*Email.owner_filters(deps.user_id, deps.organization_id)) + .where( + *Email.owner_filters(deps.user_id, deps.organization_id, deps.workspace_id) + ) .where(Email.message_id == message_id) .limit(1) ) @@ -280,7 +311,9 @@ async def tool_list_tasks( status = (status or "").strip() if status: statement = statement.where(TicketTask.status == status) - statement = statement.order_by(TicketTask.updated_at.desc()).limit(_MAX_MAIL_RESULTS) + statement = statement.order_by(TicketTask.updated_at.desc()).limit( + _MAX_MAIL_RESULTS + ) result = await deps.session.execute(statement) tasks = result.scalars().all() return [ @@ -376,16 +409,16 @@ async def tool_dispatch_writeback( "target_path": target_path, "content": content or "", } - dispatch_result = await dispatcher( - deps.organization_id, deps.workspace_id, command - ) + dispatch_result = await dispatcher(deps.organization_id, deps.workspace_id, command) provider_write_executed = bool( isinstance(dispatch_result, dict) and dispatch_result.get("provider_write_executed", False) ) await _record_audit( deps, - action="writeback_executed" if provider_write_executed else "writeback_dispatched", + action="writeback_executed" + if provider_write_executed + else "writeback_dispatched", resource_type="runner_writeback", resource_id=target_path, details=f"noema-agent {action} provider_write_executed={provider_write_executed}", @@ -404,11 +437,65 @@ async def _default_dispatcher( """Resolve the live runner connection manager lazily to avoid import cycles.""" from api.runner_ws import manager as runner_manager - return await runner_manager.dispatch_command( - organization_id, workspace_id, command + return await runner_manager.dispatch_command(organization_id, workspace_id, command) + + +def _parse_commitment( + commitment_id: str, start_at: str, end_at: str, status: str +) -> CalendarCommitment: + """Parse ISO 8601 timestamps into a validated :class:`CalendarCommitment`. + + Raises :class:`CalendarPolicyValidationError` — the same typed failure + :mod:`services.calendar_conflict_policy` itself raises — on a malformed + timestamp, so callers only need to catch one exception type. + """ + try: + parsed_start = datetime.datetime.fromisoformat((start_at or "").strip()) + parsed_end = datetime.datetime.fromisoformat((end_at or "").strip()) + except (TypeError, ValueError) as exc: + raise CalendarPolicyValidationError( + "calendar_timestamp_timezone_required", + "start_at/end_at must be ISO 8601 timestamps with a UTC offset", + ) from exc + return CalendarCommitment( + commitment_id=commitment_id, + start_at=parsed_start, + end_at=parsed_end, + status=status, # type: ignore[arg-type] # validated by __post_init__ ) +async def tool_check_calendar_conflict( + deps: NoemaAgentDeps, + proposed_commitment_id: str, + proposed_start_at: str, + proposed_end_at: str, + proposed_status: str, + existing: list[dict[str, str]], +) -> dict[str, Any]: + """Refuse to assert availability without authoritative calendar evidence. + + Proposed timestamps still use the deterministic policy's validation + contract. Naruon currently exposes only an outbound CalDAV write seam; it has no + scoped inbound provider-calendar reader. ``existing`` is therefore + untrusted conversational evidence and cannot establish availability. + """ + deps.tool_calls.append("check_calendar_conflict") + try: + _parse_commitment( + proposed_commitment_id, proposed_start_at, proposed_end_at, proposed_status + ) + except CalendarPolicyValidationError as exc: + return {"status": "error", "error_code": exc.error_code, "reason": str(exc)} + + return { + "status": "error", + "error_code": "calendar_authoritative_evidence_unavailable", + "decision_code": "review_required", + "reason": "Authoritative scoped provider calendar evidence is unavailable", + } + + # Introspectable catalog of the tools the agent exposes. Used for wiring tests # and for documenting the agent's surface without importing pydantic-ai. NOEMA_TOOL_SPECS: tuple[dict[str, Any], ...] = ( @@ -425,6 +512,11 @@ async def _default_dispatcher( "impl": tool_update_task_status, "capability": "tasks.update", }, + { + "name": "check_calendar_conflict", + "impl": tool_check_calendar_conflict, + "capability": "calendar.conflict_check", + }, { "name": "dispatch_writeback", "impl": tool_dispatch_writeback, @@ -435,10 +527,13 @@ async def _default_dispatcher( SYSTEM_PROMPT = ( "You are Noema, the general assistant for a naruon email workspace. " "Use the provided tools to read and search the owner's mail, inspect the " - "content graph of an email, and manage tasks. Only change task status or " - "dispatch a writeback when the user clearly asks for it. Writebacks target " - "the customer's own systems and require opt-in; if a writeback is skipped, " - "explain that it must be enabled. Be concise and cite message ids you used." + "content graph of an email, and manage tasks. When a message proposes or " + "moves a meeting, use check_calendar_conflict, but never claim a time is " + "available unless that tool has authoritative provider evidence. Only " + "change task status or dispatch a writeback when the user clearly asks " + "for it. Writebacks target the customer's own systems and require opt-in; " + "if a writeback is skipped, explain that it must be enabled. Be concise " + "and cite message ids you used." ) @@ -447,6 +542,7 @@ def _load_pydantic_ai() -> Any | None: try: import pydantic_ai # noqa: F401 from pydantic_ai import Agent, RunContext + # pydantic-ai 2.x renamed ``OpenAIModel`` to ``OpenAIChatModel``. Import # the current name; the old alias no longer exists on 2.x. from pydantic_ai.models.openai import OpenAIChatModel @@ -463,12 +559,15 @@ def _load_pydantic_ai() -> Any | None: async def build_noema_agent( - provider: RuntimeLLMProvider, + gateway: OrchestratorGateway, ) -> tuple["Agent | None", Callable[[], Awaitable[None]]]: - """Build the pydantic-ai agent for a resolved provider. + """Build the pydantic-ai agent for a resolved contextual-orchestrator gateway. Returns ``(agent, closer)``. ``agent`` is ``None`` when pydantic-ai is not - installed; ``closer`` always closes any opened HTTP client. + installed, or when the gateway's ``base_url`` fails SSRF/allowlist + validation (a stored credential can still be malformed); ``closer`` + always closes any opened HTTP client. Never falls back to a direct + tenant LLM-provider client on either condition. """ from openai import AsyncOpenAI @@ -480,10 +579,14 @@ async def _noop_closer() -> None: return None, _noop_closer validated_base_url, http_client = await build_llm_provider_http_client( - provider.base_url + gateway.base_url ) + if validated_base_url is None: + await http_client.aclose() + return None, _noop_closer + openai_client = AsyncOpenAI( - api_key=provider.api_key, + api_key=gateway.inference_token, base_url=validated_base_url, http_client=http_client, ) @@ -492,7 +595,7 @@ async def _closer() -> None: await openai_client.close() model = modules["OpenAIChatModel"]( - provider.chat_model, + gateway.model_alias, provider=modules["OpenAIProvider"](openai_client=openai_client), ) agent = modules["Agent"]( @@ -509,7 +612,9 @@ async def search_mail( # type: ignore[unused-ignore] return await tool_search_mail(ctx.deps, query, limit) @agent.tool - async def read_mail(ctx: RunContext[NoemaAgentDeps], message_id: str) -> dict[str, Any]: + async def read_mail( + ctx: RunContext[NoemaAgentDeps], message_id: str + ) -> dict[str, Any]: """Read the full body of a single owned email by message id.""" return await tool_read_mail(ctx.deps, message_id) @@ -534,6 +639,32 @@ async def update_task_status( """Update the status of an owned task (audit-logged).""" return await tool_update_task_status(ctx.deps, task_uid, status) + @agent.tool + async def check_calendar_conflict( + ctx: RunContext[NoemaAgentDeps], + proposed_commitment_id: str, + proposed_start_at: str, + proposed_end_at: str, + proposed_status: str, + existing: list[dict[str, str]], + ) -> dict[str, Any]: + """Check a proposed meeting time against known commitments for a conflict. + + Timestamps are ISO 8601 with a UTC offset; ``proposed_status`` and each + row's ``status`` in ``existing`` are one of confirmed/tentative/desired/ + cancelled. ``existing`` rows come from commitments already surfaced in + this conversation (e.g. via search_mail/read_mail), not a live provider + fetch. + """ + return await tool_check_calendar_conflict( + ctx.deps, + proposed_commitment_id, + proposed_start_at, + proposed_end_at, + proposed_status, + existing, + ) + @agent.tool async def dispatch_writeback( ctx: RunContext[NoemaAgentDeps], @@ -564,21 +695,25 @@ async def run_noema_agent( This is the entrypoint referenced by ``registered_agents.json``. """ - provider = await resolve_runtime_llm_provider( + gateway = await resolve_orchestrator_gateway( session, user_id=user_id, organization_id=organization_id ) - if provider is None: + if gateway is None: return NoemaAgentResult( status="unavailable", - notice="No LLM provider is configured for this workspace.", + notice=( + "The contextual-orchestrator gateway is not configured for " + "this workspace." + ), + error_code="orchestrator_gateway_unavailable", ) - agent, closer = await build_noema_agent(provider) + agent, closer = await build_noema_agent(gateway) if agent is None: return NoemaAgentResult( status="unavailable", notice="The pydantic-ai runtime is not installed; agent is disabled.", - provider_name=provider.provider_name, + provider_name=gateway.model_alias, ) deps = NoemaAgentDeps( @@ -594,7 +729,7 @@ async def run_noema_agent( return NoemaAgentResult( status="ok", output=str(getattr(result, "output", "")), - provider_name=provider.provider_name, + provider_name=gateway.model_alias, tool_calls=tuple(deps.tool_calls), ) except Exception as exc: # noqa: BLE001 - degrade gracefully, never propagate @@ -602,7 +737,7 @@ async def run_noema_agent( return NoemaAgentResult( status="error", notice="The agent run could not be completed.", - provider_name=provider.provider_name, + provider_name=gateway.model_alias, tool_calls=tuple(deps.tool_calls), ) finally: diff --git a/backend/services/orchestrator_gateway.py b/backend/services/orchestrator_gateway.py new file mode 100644 index 000000000..4df00c81a --- /dev/null +++ b/backend/services/orchestrator_gateway.py @@ -0,0 +1,91 @@ +"""contextual-orchestrator inference gateway for naruon's in-process Noema agent. + +naruon is a consumer of ``ContextualWisdomLab/contextual-orchestrator``, not a +second LLM-provider-routing authority. The general-purpose Noema workspace +agent (:mod:`services.noema_agent`) sends every chat-completion call through +this gateway using a per-tenant Fernet-encrypted credential +(``tenant_configs.noema_orchestrator_base_url`` / +``noema_orchestrator_token``), never a direct tenant LLM-provider key. The +orchestrator owns model selection, upstream failover, and cost -- naruon does +not pick or fail over across models here; the model alias sent on every call +is always :data:`ORCHESTRATOR_MODEL_ALIAS`. + +This keeps two separately-authorized scopes distinct: naruon's tenant-scoped +Noema gateway credential resolved here, versus ``ContextualWisdomLab/.github``'s +central review-pipeline credential used by the org's CI Noema reviewer +(``scripts/ci/noema_review_gate.py``). This module never reads or writes +anything under that CI credential, and nothing here ever sends workspace data +through it. + +Per ``docs/CWL-MASTER-CONTEXT.md`` (``ContextualWisdomLab/.github``), Noema is +actually one shared agent runtime (Pydantic-AI/Codex-Python) consumed by +naruon, the ``.github`` CI review agent, and wardnet's AI SOC quarantine +sandbox -- the credential scoping above is a security choice for this module, +not a claim that the deployments are permanently separate or share nothing +beyond a name; see ``ContextualWisdomLab/naruon#1527`` for the corrected +reasoning (its ``docs/adr/0006-noema-bounded-context-separation.md`` lives +on that PR's own branch, not this one, until it merges). +""" + +from __future__ import annotations + +from dataclasses import dataclass + +from sqlalchemy.ext.asyncio import AsyncSession + +from services.llm_provider_urls import validate_llm_provider_base_url_async +from services.tenant_config_scope import get_scoped_tenant_config + +ORCHESTRATOR_MODEL_ALIAS = "contextual-orchestrator" + + +@dataclass(frozen=True) +class OrchestratorGateway: + """A tenant's resolved contextual-orchestrator gateway credential. + + ``base_url`` and ``inference_token`` come from the per-tenant + Fernet-encrypted ``tenant_configs`` row (never ``os.getenv``). + ``model_alias`` is always :data:`ORCHESTRATOR_MODEL_ALIAS`: naruon asks + the orchestrator to select the underlying model, it never chooses one. + """ + + base_url: str + inference_token: str + model_alias: str = ORCHESTRATOR_MODEL_ALIAS + + +def _clean(value: str | None) -> str | None: + if value is None: + return None + stripped = value.strip() + return stripped or None + + +async def resolve_orchestrator_gateway( + session: AsyncSession, + *, + user_id: str, + organization_id: str | None, +) -> OrchestratorGateway | None: + """Resolve the tenant's contextual-orchestrator gateway config. + + Returns ``None`` when the tenant has not configured the gateway (missing + base URL or token) or when the stored base URL fails SSRF/allowlist + validation, so the caller can degrade to a single, structured + "unavailable" result for every one of those reasons -- this never falls + back to constructing a direct provider client. Validating here (rather + than deferring to :func:`services.llm_provider_urls.build_llm_provider_http_client` + at agent-build time) keeps that later call's own ``None`` result reserved + for one thing only: the pydantic-ai runtime being absent. + """ + tenant_config = await get_scoped_tenant_config(session, user_id, organization_id) + if tenant_config is None: + return None + base_url = _clean(getattr(tenant_config, "noema_orchestrator_base_url", None)) + token = _clean(getattr(tenant_config, "noema_orchestrator_token", None)) + if not base_url or not token: + return None + validated_base_url = await validate_llm_provider_base_url_async(base_url) + if not validated_base_url: + return None + return OrchestratorGateway(base_url=validated_base_url, inference_token=token) diff --git a/backend/services/pop3_worker.py b/backend/services/pop3_worker.py index 601180027..5b7563f33 100644 --- a/backend/services/pop3_worker.py +++ b/backend/services/pop3_worker.py @@ -7,7 +7,7 @@ from services.email_client import validate_pop3_destination from services.email_parser import parse_eml_bytes from services.exceptions import EmailParseError -from services.imap_worker import process_fetched_email +from services.imap_worker import process_fetched_email, resolve_unambiguous_workspace_id logger = logging.getLogger(__name__) MAX_POP3_FETCH_MESSAGES = 10 @@ -56,21 +56,42 @@ async def _run_loop(self): break async def _sync(self): + resolved = [] async with AsyncSessionLocal() as session: - result = await session.execute(select(TenantConfig).where(TenantConfig.pop3_server.isnot(None))) + result = await session.execute( + select(TenantConfig).where(TenantConfig.pop3_server.isnot(None)) + ) configs = result.scalars().all() - + + for config in configs: + if not config.pop3_server or not config.pop3_port: + continue + # TenantConfig has no workspace_id column; POP3 sync has no + # signed session either, so the only evidence of this owner's + # workspace is whatever workspace their already-imported mail + # belongs to (same fail-closed resolution as ImapSyncWorker). + workspace_id = await resolve_unambiguous_workspace_id( + session, config.user_id, config.organization_id + ) + if workspace_id is None: + logger.info( + "Skipping POP3 sync because an unambiguous workspace is unavailable" + ) + continue + resolved.append((config, workspace_id)) + semaphore = asyncio.Semaphore(10) - tasks = [] - for config in configs: - if not config.pop3_server or not config.pop3_port: - continue - tasks.append(self._sync_tenant(config, semaphore)) - + tasks = [ + self._sync_tenant(config, workspace_id, semaphore) + for config, workspace_id in resolved + ] + if tasks: await asyncio.gather(*tasks, return_exceptions=True) - async def _sync_tenant(self, config: TenantConfig, semaphore: asyncio.Semaphore): + async def _sync_tenant( + self, config: TenantConfig, workspace_id: str, semaphore: asyncio.Semaphore + ): async with semaphore: try: pop3_server, pop3_port = self._validated_destination(config) @@ -88,7 +109,9 @@ async def _sync_tenant(self, config: TenantConfig, semaphore: asyncio.Semaphore) messages = await asyncio.to_thread( self._do_pop3_sync, config, pop3_server, pop3_port ) - imported_count = await self._import_messages(config, messages) + imported_count = await self._import_messages( + config, workspace_id, messages + ) logger.info( "Successfully synced POP3 server for user %s with %s imported messages.", config.user_id, @@ -102,7 +125,7 @@ async def _sync_tenant(self, config: TenantConfig, semaphore: asyncio.Semaphore) ) async def _import_messages( - self, config: TenantConfig, messages: list[bytes] + self, config: TenantConfig, workspace_id: str, messages: list[bytes] ) -> int: if not messages: return 0 @@ -125,6 +148,7 @@ async def _import_messages( email_data, config.user_id, config.organization_id, + workspace_id=workspace_id, owner_addresses=owner_addresses, ) imported_count += 1 @@ -195,7 +219,5 @@ def _message_number_from_listing(self, listing: bytes | str) -> int | None: def _bytes_line(self, line: bytes | str) -> bytes: return ( - line - if isinstance(line, bytes) - else line.encode("utf-8", errors="replace") + line if isinstance(line, bytes) else line.encode("utf-8", errors="replace") ) diff --git a/backend/services/reply_sla_escalation_service.py b/backend/services/reply_sla_escalation_service.py index 7800f121c..db63ee099 100644 --- a/backend/services/reply_sla_escalation_service.py +++ b/backend/services/reply_sla_escalation_service.py @@ -2,6 +2,7 @@ from contextlib import nullcontext from dataclasses import dataclass +from sqlalchemy import inspect as sqlalchemy_inspect from sqlalchemy import select from sqlalchemy.exc import IntegrityError from sqlalchemy.ext.asyncio import AsyncSession @@ -12,20 +13,30 @@ from services.threading_service import normalize_message_id REPLY_SLA_SOURCE_TYPE = "reply_sla" +REPLY_SLA_MAX_BATCH_ATTEMPTS = 3 class ReplySlaTaskConflict(Exception): - pass + """Report a concurrent source-task conflict that cannot be reconciled.""" + + def __init__(self, error_code: str, message: str) -> None: + """Create a conflict with stable classification independent of prose.""" + super().__init__(message) + self.error_code = error_code @dataclass(frozen=True) class ReplySlaEscalatedTask: + """Pair a persisted task with its original external email reference.""" + task: TicketTask source_email_id: str | None @dataclass(frozen=True) class ReplySlaEscalationResult: + """Summarize evaluated mail and created or updated follow-up tasks.""" + evaluated: int created: int overdue_hours: int @@ -33,6 +44,7 @@ class ReplySlaEscalationResult: def canonical_reply_sla_thread_key(email: Email) -> str: + """Prefer the normalized thread reference, then the message reference.""" return ( normalize_message_id(email.thread_id) or normalize_message_id(email.message_id) @@ -41,6 +53,7 @@ def canonical_reply_sla_thread_key(email: Email) -> str: def _safe_email_subject(subject: str | None) -> str: + """Keep a bounded plain-text title without active email markup.""" trimmed = (subject or "제목 없음").replace("\x00", " ").strip() if not trimmed or contains_html_markup(trimmed): return "제목 정리 필요" @@ -48,10 +61,12 @@ def _safe_email_subject(subject: str | None) -> str: def _reply_sla_task_title(email: Email) -> str: + """Build the existing follow-up label from a sanitized mail subject.""" return f"미답변 팔로업: {_safe_email_subject(email.subject)}" def _email_date_utc(email: Email) -> datetime.datetime: + """Interpret legacy naive timestamps as UTC for deadline comparisons.""" message_date = email.date if message_date.tzinfo is None: return message_date.replace(tzinfo=datetime.timezone.utc) @@ -61,6 +76,7 @@ def _email_date_utc(email: Email) -> datetime.datetime: async def _fetch_existing_tasks_by_email( db: AsyncSession, user_id: str, organization_id: str | None, email_ids: list[int] ) -> dict[int, TicketTask]: + """Select the most recently updated owner-scoped task per source email.""" result = await db.execute( select(TicketTask) .where( @@ -81,6 +97,7 @@ async def _fetch_existing_tasks_by_email( def _update_task_for_escalation( task: TicketTask, email: Email, now: datetime.datetime ) -> None: + """Escalate pending work without reopening a completed task.""" if task.status != "done": task.title = _reply_sla_task_title(email) task.status = "blocked" @@ -92,6 +109,7 @@ def _update_task_for_escalation( def _create_task_for_escalation( user_id: str, organization_id: str | None, email: Email ) -> TicketTask: + """Create a source-linked urgent task using the established identity contract.""" return TicketTask( user_id=user_id, organization_id=organization_id, @@ -111,6 +129,7 @@ async def _refresh_escalated_tasks( email_ids: list[int], escalated_tasks: list[tuple[TicketTask, str | None]], ) -> None: + """Replace task references with persisted owner-scoped rows when present.""" refreshed_tasks_by_email = await _fetch_existing_tasks_by_email( db, user_id, organization_id, email_ids ) @@ -127,6 +146,7 @@ async def _process_bulk_escalation( overdue_replies: list[Email], now: datetime.datetime, ) -> tuple[int, list[tuple[TicketTask, str | None]]]: + """Create or update overdue follow-ups in the ordinary single commit path.""" email_ids = [email.id for email in overdue_replies] existing_tasks_by_email = await _fetch_existing_tasks_by_email( db, user_id, organization_id, email_ids @@ -159,6 +179,24 @@ async def _process_bulk_escalation( return created_count, escalated_tasks +def _is_non_unique_constraint_failure(error: IntegrityError) -> bool: + """Classify known driver constraint codes without parsing localized prose.""" + sqlstate = getattr(error.orig, "sqlstate", None) or getattr( + error.orig, "pgcode", None + ) + if sqlstate is not None: + return sqlstate != "23505" + sqlite_errorname = getattr(error.orig, "sqlite_errorname", None) + if sqlite_errorname is not None: + return sqlite_errorname not in { + "SQLITE_CONSTRAINT_UNIQUE", + "SQLITE_CONSTRAINT_PRIMARYKEY", + } + # Untyped IntegrityError retains the established conflict contract. Do not + # infer a constraint category from provider- or locale-specific text. + return False + + async def _process_fallback_escalation( db: AsyncSession, user_id: str, @@ -166,103 +204,92 @@ async def _process_fallback_escalation( overdue_replies: list[Email], now: datetime.datetime, ) -> tuple[int, list[tuple[TicketTask, str | None]]]: + """Bound contention recovery without per-row SAVEPOINT retries.""" email_ids = [email.id for email in overdue_replies] existing_tasks_by_email = await _fetch_existing_tasks_by_email( db, user_id, organization_id, email_ids ) - - created_count = 0 - escalated_tasks: list[tuple[TicketTask, str | None]] = [] - - fallback_entries: list[tuple[Email, TicketTask | None]] = [] - conflicted_email_ids: list[int] = [] - new_tasks: list[tuple[int, Email, TicketTask]] = [] + entries: list[tuple[Email, TicketTask]] = [] + pending: list[tuple[int, Email, TicketTask]] = [] for email in overdue_replies: - if email.id in existing_tasks_by_email: - task = existing_tasks_by_email[email.id] - _update_task_for_escalation(task, email, now) - else: + task = existing_tasks_by_email.get(email.id) + if task is None: task = _create_task_for_escalation(user_id, organization_id, email) - new_tasks.append((len(fallback_entries), email, task)) - fallback_entries.append((email, task)) + pending.append((len(entries), email, task)) + else: + _update_task_for_escalation(task, email, now) + entries.append((email, task)) - if new_tasks: + created_count = 0 + for _ in range(REPLY_SLA_MAX_BATCH_ATTEMPTS): + if not pending: + break + savepoint_started = False + flush_started = False try: async with db.begin_nested(): - for _, _, task in new_tasks: + savepoint_started = True + for _, _, task in pending: db.add(task) + flush_started = True await db.flush() - created_count += len(new_tasks) - except IntegrityError: - # OPTIMIZATION: A concurrent process likely created some of these tasks. - # We fetch all currently existing tasks for our `new_tasks` list, - # filter them out, and bulk insert the truly new ones. - current_email_ids = [email.id for _, email, _ in new_tasks] + except IntegrityError as error: + if not savepoint_started: + # AsyncSession.begin_nested() flushes dirty state before the + # SAVEPOINT exists. A pre-savepoint failure poisons the outer + # transaction and must not be treated as a uniqueness race. + await db.rollback() + raise + if _is_non_unique_constraint_failure(error): + await db.rollback() + raise + + # SAVEPOINT rollback already detaches failed inserts. Read visible + # winners in one owner-scoped query and retry only the remainder. with getattr(db, "no_autoflush", nullcontext()): - currently_existing = await _fetch_existing_tasks_by_email( - db, user_id, organization_id, current_email_ids + winners = await _fetch_existing_tasks_by_email( + db, + user_id, + organization_id, + [email.id for _, email, _ in pending], ) - - remaining_tasks = [] - for index, email, task in new_tasks: - if email.id in currently_existing: - # We know this one conflicted. - conflicted_email_ids.append(email.id) - fallback_entries[index] = (email, None) - if hasattr(db, "expunge"): - db.expunge(task) - else: - remaining_tasks.append((index, email, task)) - - if remaining_tasks: - # Highly likely the remaining tasks can now be bulk inserted. - try: - async with db.begin_nested(): - for _, _, task in remaining_tasks: - db.add(task) - await db.flush() - created_count += len(remaining_tasks) - except IntegrityError: - # Rare extreme concurrency: fallback to individual inserts. - for index, email, task in remaining_tasks: - task_or_none: TicketTask | None = task - try: - async with db.begin_nested(): - db.add(task) - await db.flush() - created_count += 1 - except IntegrityError: - conflicted_email_ids.append(email.id) - task_or_none = None - if hasattr(db, "expunge"): - db.expunge(task) - fallback_entries[index] = (email, task_or_none) - - if conflicted_email_ids: - conflicted_tasks_by_email = await _fetch_existing_tasks_by_email( - db, user_id, organization_id, conflicted_email_ids - ) - - for index, (email, task) in enumerate(fallback_entries): - if task is not None or email.id not in conflicted_email_ids: - continue - task = conflicted_tasks_by_email.get(email.id) - if task is None: + remaining: list[tuple[int, Email, TicketTask]] = [] + for index, email, task in pending: + winner = winners.get(email.id) + if winner is None: + remaining.append((index, email, task)) + continue + _update_task_for_escalation(winner, email, now) + entries[index] = (email, winner) + + # Some scripted compatibility sessions surface a simulated unique + # race from add() rather than flush(). A real AsyncSession does not, + # but preserving that harness is useful: reconcile a visible winner + # once, and fail closed if the synthetic conflict has no winner. + if not flush_started and len(remaining) == len(pending): + await db.rollback() raise ReplySlaTaskConflict( - "reply_sla_task_conflict: " - f"user_id={user_id!r} organization_id={organization_id!r} " - f"scoped_email_key={email.id!r}" - ) from None - - _update_task_for_escalation(task, email, now) - fallback_entries[index] = (email, task) + "reply_sla_task_conflict", + "no visible duplicate winner", + ) from error + pending = remaining + else: + created_count = len(pending) + pending = [] + break - escalated_tasks.extend( - (task, email.message_id) for email, task in fallback_entries if task is not None - ) + if pending: + # Sustained contention is bounded by batch attempts. Rolling back here + # also reverts updates to pre-existing tasks, so no partial write leaks. + await db.rollback() + raise ReplySlaTaskConflict( + "reply_sla_batch_retry_exhausted", + "batch retry budget exhausted", + ) - if created_count > 0 or any(t.status != "done" for t, _ in escalated_tasks): + escalated_tasks = [(task, email.message_id) for email, task in entries] + if created_count > 0 or any(task.status != "done" for task, _ in escalated_tasks): await db.commit() await _refresh_escalated_tasks( db, user_id, organization_id, email_ids, escalated_tasks @@ -271,17 +298,59 @@ async def _process_fallback_escalation( return created_count, escalated_tasks +async def _reload_overdue_replies( + db: AsyncSession, + user_id: str, + organization_id: str | None, + workspace_id: str, + email_ids: list[int], +) -> list[Email]: + """Reload rollback-expired source mail in one workspace-scoped query.""" + result = await db.execute( + select(Email) + .where( + Email.user_id == user_id, + Email.organization_id == organization_id, + Email.workspace_id == workspace_id, + Email.id.in_(email_ids), + ) + .execution_options(populate_existing=True) + ) + by_email_id = {email.id: email for email in result.scalars().all()} + if any(email_id not in by_email_id for email_id in email_ids): + await db.rollback() + raise ReplySlaTaskConflict( + "reply_sla_source_email_unavailable", + "source email no longer available in the authorized workspace", + ) + return [by_email_id[email_id] for email_id in email_ids] + + +def _rollback_expired_any_source(overdue_replies: list[Email]) -> bool: + """Detect whether SQLAlchemy rollback made a source unsafe to read directly.""" + return any( + sqlalchemy_inspect(email).expired or sqlalchemy_inspect(email).detached + for email in overdue_replies + ) + + async def create_reply_sla_escalation_tasks( db: AsyncSession, *, user_id: str, organization_id: str | None, + workspace_id: str, overdue_hours: int, limit: int, tenant_config: TenantConfig | None = None, ) -> ReplySlaEscalationResult: + """Persist bounded follow-ups selected by authoritative scoped reply tracking.""" pending_replies = await check_missing_replies( - db, user_id, organization_id, tenant_config=tenant_config + db, + user_id, + organization_id, + workspace_id, + tenant_config=tenant_config, ) now = datetime.datetime.now(datetime.timezone.utc) overdue_cutoff = now - datetime.timedelta(hours=overdue_hours) @@ -302,12 +371,27 @@ async def create_reply_sla_escalation_tasks( tasks=[], ) + # Keep primitive IDs before the transaction can expire mapped source rows. + email_ids = [email.id for email in overdue_replies] try: created_count, escalated_tasks = await _process_bulk_escalation( db, user_id, organization_id, overdue_replies, now ) - except IntegrityError: + except IntegrityError as error: await db.rollback() + if _is_non_unique_constraint_failure(error): + raise + # Real SQLAlchemy rollback expires mapped source rows. Scripted unit + # sessions that do not model expiration retain their in-memory fixtures; + # production takes the workspace-scoped one-query reload path. + if _rollback_expired_any_source(overdue_replies): + overdue_replies = await _reload_overdue_replies( + db, + user_id, + organization_id, + workspace_id, + email_ids, + ) created_count, escalated_tasks = await _process_fallback_escalation( db, user_id, organization_id, overdue_replies, now ) diff --git a/backend/services/reply_sla_scheduler.py b/backend/services/reply_sla_scheduler.py index ae3250338..7203b17f3 100644 --- a/backend/services/reply_sla_scheduler.py +++ b/backend/services/reply_sla_scheduler.py @@ -3,9 +3,10 @@ import random from sqlalchemy import bindparam, func, or_, select +from sqlalchemy.exc import DBAPIError -from db.models import TenantConfig -from db.session import AsyncSessionLocal +from db.models import Email, TenantConfig +from db.session import AsyncSessionLocal, engine from services.reply_sla_escalation_service import create_reply_sla_escalation_tasks logger = logging.getLogger(__name__) @@ -18,6 +19,7 @@ def _session_uses_postgresql(session) -> bool: + """Identify PostgreSQL coordination without assuming a development bind.""" try: bind = session.get_bind() except Exception: @@ -54,9 +56,8 @@ async def _try_acquire_sweep_lease(session) -> bool | None: async def _release_sweep_lease(session) -> None: - # Session-level advisory locks outlive pooled connections; always release - # explicitly so a returned connection cannot keep the lease forever. - await session.scalar( + """Require confirmed release on the connection that acquired the lease.""" + released = await session.scalar( select( func.pg_advisory_unlock( func.hashtext(bindparam("namespace_key")), @@ -65,9 +66,13 @@ async def _release_sweep_lease(session) -> None: ), _SWEEP_LOCK_PARAMS, ) + if released is not True: + raise RuntimeError("Reply SLA sweep lease release was not confirmed.") class ReplySlaScheduler: + """Schedule owner-scoped overdue reply tasks under a database sweep lease.""" + def __init__( self, *, @@ -75,6 +80,7 @@ def __init__( overdue_hours: int = DEFAULT_REPLY_SLA_OVERDUE_HOURS, limit: int = DEFAULT_REPLY_SLA_LIMIT, ): + """Set the cycle interval and existing escalation policy limits.""" self.interval_seconds = interval_seconds self.overdue_hours = overdue_hours self.limit = limit @@ -82,6 +88,7 @@ def __init__( self._is_running = False async def start(self): + """Start at most one local scheduling task.""" if self._is_running: logger.warning("ReplySlaScheduler is already running.") return @@ -91,6 +98,7 @@ async def start(self): logger.info("ReplySlaScheduler started.") async def stop(self): + """Cancel the scheduling task and await its cleanup.""" if not self._is_running: return @@ -106,6 +114,7 @@ async def stop(self): logger.info("ReplySlaScheduler stopped.") async def _run_loop(self): + """Jitter replica startup and retry failed cycles on the normal interval.""" # Startup jitter de-synchronizes replicas started by the same deploy # so they do not contend for the sweep lease at the same instant. try: @@ -132,43 +141,76 @@ async def _run_loop(self): break async def _sync(self): - async with AsyncSessionLocal() as session: - lease = await _try_acquire_sweep_lease(session) - if lease is False: - logger.debug( - "Reply SLA sweep skipped: another replica holds the lease." - ) - return + """Keep one physical lease connection across escalation transactions.""" + async with ( + engine.connect() as connection, + AsyncSessionLocal(bind=connection) as session, + ): try: + lease = await _try_acquire_sweep_lease(session) + if lease is False: + logger.debug( + "Reply SLA sweep skipped: another replica holds the lease." + ) + return await self._sweep_configured_owners(session) - finally: if lease is True: + await session.rollback() await _release_sweep_lease(session) + except BaseException: + # Invalidate before AsyncSession's shielded close/rollback can wait. + await connection.invalidate() + raise async def _sweep_configured_owners(self, session): + """Reload owner records after rollback without replacing the lease backend.""" result = await session.execute( - select(TenantConfig).where( + select(TenantConfig.id).where( or_( TenantConfig.smtp_username.isnot(None), TenantConfig.imap_username.isnot(None), ) ) ) - configs = result.scalars().all() + config_ids = result.scalars().all() - for config in configs: + for config_id in config_ids: try: - await create_reply_sla_escalation_tasks( - session, - user_id=config.user_id, - organization_id=config.organization_id, - overdue_hours=self.overdue_hours, - limit=self.limit, - tenant_config=config, + config = await session.get( + TenantConfig, config_id, populate_existing=True ) - except Exception: + if config is None: + continue + workspace_ids = await session.scalars( + select(Email.workspace_id) + .where( + Email.user_id == config.user_id, + Email.organization_id == config.organization_id, + ) + .distinct() + ) + for workspace_id in workspace_ids: + # Bypass cached state after commit, and expired state after + # conflict rollback, before authorizing another workspace. + config = await session.get( + TenantConfig, config_id, populate_existing=True + ) + if config is None: + break + await create_reply_sla_escalation_tasks( + session, + user_id=config.user_id, + organization_id=config.organization_id, + workspace_id=workspace_id, + overdue_hours=self.overdue_hours, + limit=self.limit, + tenant_config=config, + ) + except Exception as exc: + if isinstance(exc, DBAPIError) and exc.connection_invalidated: + raise + await session.rollback() logger.error( - "Overdue reply follow-up failed for configured owner %s.", - config.user_id, - exc_info=True, + "Overdue reply follow-up failed for a configured owner (%s).", + type(exc).__name__, ) diff --git a/backend/services/reply_tracking_service.py b/backend/services/reply_tracking_service.py index a8933848c..9fa3124c6 100644 --- a/backend/services/reply_tracking_service.py +++ b/backend/services/reply_tracking_service.py @@ -123,6 +123,7 @@ async def check_missing_replies( session: AsyncSession, user_id: str, organization_id: str | None, + workspace_id: str, tenant_config: TenantConfig | None = None, ) -> list[Email]: """ @@ -145,16 +146,10 @@ async def check_missing_replies( recent_limit = datetime.datetime.now(datetime.timezone.utc) - datetime.timedelta( days=7 ) - organization_filter = ( - Email.organization_id == organization_id - if organization_id is not None - else Email.organization_id.is_(None) - ) stmt = ( select(Email) .where( - Email.user_id == user_id, - organization_filter, + *Email.owner_filters(user_id, organization_id, workspace_id), Email.date > recent_limit, ) .order_by(Email.date.asc(), Email.id.asc()) diff --git a/backend/services/threading_service.py b/backend/services/threading_service.py index c500e4d35..1c626ac48 100644 --- a/backend/services/threading_service.py +++ b/backend/services/threading_service.py @@ -13,6 +13,7 @@ # email header processing, yielding a measurable speedup when handling long reference lists. REFERENCE_PATTERN = re.compile(r"<([^>]+)>") + def generate_email_fingerprint( subject: str | None, date_str: str | None, @@ -81,6 +82,7 @@ async def _find_existing_thread_ids( *, user_id: str, organization_id: str | None, + workspace_id: str, ) -> dict[str, str]: if not message_ids: return {} @@ -95,7 +97,7 @@ async def _find_existing_thread_ids( result = await session.execute( select(Email.message_id, Email.thread_id).where( - *Email.owner_filters(user_id, organization_id), + *Email.owner_filters(user_id, organization_id, workspace_id), Email.message_id.in_(target_ids), ) ) @@ -118,6 +120,7 @@ async def assign_thread_id( *, user_id: str, organization_id: str | None, + workspace_id: str, ) -> str: """ Determine the thread_id for a new email based on in_reply_to and references. @@ -145,6 +148,7 @@ async def assign_thread_id( existing_candidates, user_id=user_id, organization_id=organization_id, + workspace_id=workspace_id, ) for candidate in existing_candidates: thread_id = thread_ids_by_message_id.get(candidate) diff --git a/backend/services/webdav_service.py b/backend/services/webdav_service.py index 284bd1696..dbbab2bb7 100644 --- a/backend/services/webdav_service.py +++ b/backend/services/webdav_service.py @@ -216,7 +216,8 @@ async def determine_knowledge_materialization_intent_from_db( Email, (TicketTask.related_email_id == Email.id) & (Email.user_id == user_id) - & (Email.organization_id == organization_id), + & (Email.organization_id == organization_id) + & (Email.workspace_id == workspace_id), ) .where( TicketTask.task_uid == source_task_id, diff --git a/backend/tests/fixtures/reply_sla_observed_message.eml b/backend/tests/fixtures/reply_sla_observed_message.eml new file mode 100644 index 000000000..066c97d4f --- /dev/null +++ b/backend/tests/fixtures/reply_sla_observed_message.eml @@ -0,0 +1,7 @@ +Message-ID: +From: Archive Author +To: Archive List +Subject: Queue lease question +Date: Sun, 27 Apr 2014 19:31:42 +0000 + +Is there a problem with 100 open sessions (behind a connection pooler?) diff --git a/backend/tests/live/seed_live_data.py b/backend/tests/live/seed_live_data.py index 5ceff9e7e..e6e8871e2 100644 --- a/backend/tests/live/seed_live_data.py +++ b/backend/tests/live/seed_live_data.py @@ -99,6 +99,7 @@ def _seed_emails(session: AsyncSession) -> None: Email( user_id=LIVE_E2E_USER_ID, organization_id=LIVE_E2E_ORGANIZATION_ID, + workspace_id=LIVE_E2E_WORKSPACE_ID, message_id=MESSAGE_IDS[0], thread_id=THREAD_ID, fingerprint="sha256:live-e2e-root", @@ -112,6 +113,7 @@ def _seed_emails(session: AsyncSession) -> None: Email( user_id=LIVE_E2E_USER_ID, organization_id=LIVE_E2E_ORGANIZATION_ID, + workspace_id=LIVE_E2E_WORKSPACE_ID, message_id=MESSAGE_IDS[1], thread_id=THREAD_ID, fingerprint="sha256:live-e2e-reply", diff --git a/backend/tests/test_alembic_migrations.py b/backend/tests/test_alembic_migrations.py index f8f3ffeae..53502018c 100644 --- a/backend/tests/test_alembic_migrations.py +++ b/backend/tests/test_alembic_migrations.py @@ -1,9 +1,86 @@ +import importlib.util from pathlib import Path +import asyncpg +import pytest +from sqlalchemy import inspect, text +from sqlalchemy.exc import OperationalError +from sqlalchemy.ext.asyncio import create_async_engine + +from core.config import settings + BACKEND_ROOT = Path(__file__).resolve().parents[1] +def _load_revision_module(revision_filename: str): + # Revision filenames start with a digit and aren't valid module names, + # so they can't be imported with a normal `import` statement. + path = BACKEND_ROOT / "alembic" / "versions" / revision_filename + spec = importlib.util.spec_from_file_location(path.stem, path) + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +def _setup_pre_0020_email_records( + sync_conn, *, legacy_identity_as_constraint: bool +) -> None: + sync_conn.execute(text("DROP TABLE IF EXISTS email_records CASCADE")) + sync_conn.execute( + text( + "CREATE TABLE email_records (" + "id serial primary key, user_id varchar, " + "organization_id varchar, message_id varchar)" + ) + ) + if legacy_identity_as_constraint: + sync_conn.execute( + text( + "ALTER TABLE email_records ADD CONSTRAINT " + "uq_email_records_owner_message_id " + "UNIQUE (user_id, organization_id, message_id)" + ) + ) + else: + sync_conn.execute( + text( + "CREATE UNIQUE INDEX uq_email_records_owner_message_id " + "ON email_records (user_id, organization_id, message_id)" + ) + ) + + +def _run_0020_upgrade(sync_conn) -> None: + from alembic.operations import Operations + from alembic.runtime.migration import MigrationContext + + module = _load_revision_module("0020_email_workspace_scope.py") + context = MigrationContext.configure(sync_conn, opts={"target_metadata": None}) + with Operations.context(context): + module.upgrade() + + +def _run_0021_upgrade(sync_conn) -> None: + from alembic.operations import Operations + from alembic.runtime.migration import MigrationContext + + module = _load_revision_module("0021_calendar_correction_rationale.py") + context = MigrationContext.configure(sync_conn, opts={"target_metadata": None}) + with Operations.context(context): + module.upgrade() + + +def _run_0001_upgrade(sync_conn) -> None: + from alembic.operations import Operations + from alembic.runtime.migration import MigrationContext + + module = _load_revision_module("0001_initial_control_plane.py") + context = MigrationContext.configure(sync_conn, opts={"target_metadata": None}) + with Operations.context(context): + module.upgrade() + + def test_alembic_scaffold_exists_with_model_metadata_target(): alembic_ini = BACKEND_ROOT / "alembic.ini" env_py = BACKEND_ROOT / "alembic" / "env.py" @@ -32,7 +109,268 @@ def test_initial_alembic_revision_records_current_schema_path(): assert "down_revision = None" in revision_text assert "CREATE EXTENSION IF NOT EXISTS vector" in revision_text assert "Base.metadata.create_all" in revision_text - assert "schema_backfill_sql" in revision_text + # Must delegate to the guarded execute_schema_backfill (which skips + # legacy-table-only statements when the table doesn't exist yet on a + # fresh database) rather than iterating schema_backfill_sql() directly. + assert "execute_schema_backfill" in revision_text + + +def test_email_workspace_migration_replaces_owner_only_identity_constraint(): + revision_text = ( + BACKEND_ROOT / "alembic" / "versions" / "0020_email_workspace_scope.py" + ).read_text() + + assert 'down_revision = "0019_attachment_uid"' in revision_text + assert '"uq_emails_owner_message_id"' in revision_text + assert '"uq_emails_workspace_message"' in revision_text + assert "op.drop_constraint(" in revision_text + assert "op.create_unique_constraint(" in revision_text + assert ( + '["user_id", "organization_id", "workspace_id", "message_id"]' in revision_text + ) + assert "sa.text(" not in revision_text + + +def test_calendar_correction_rationale_uses_append_only_rename_migration(): + original_revision = ( + BACKEND_ROOT / "alembic" / "versions" / "0018_calendar_conflict_judgments.py" + ).read_text() + rename_revision = ( + BACKEND_ROOT + / "alembic" + / "versions" + / "0021_calendar_correction_rationale.py" + ).read_text() + + assert 'sa.Column("rationale"' in original_revision + assert 'down_revision = "0020_email_workspace_scope"' in rename_revision + assert 'new_column_name="correction_rationale"' in rename_revision + assert "op.alter_column(" in rename_revision + assert "sa.text(" not in rename_revision + + +def test_calendar_correction_rationale_upgrade_renames_legacy_column(monkeypatch): + module = _load_revision_module("0021_calendar_correction_rationale.py") + calls = [] + + class Inspector: + @staticmethod + def has_table(table_name): + return table_name == "calendar_conflict_corrections" + + @staticmethod + def get_columns(_table_name): + return [{"name": "rationale"}] + + def _fake_bind(): + return object() + + monkeypatch.setattr(module.op, "get_bind", _fake_bind) + monkeypatch.setattr(module.sa, "inspect", lambda _connection: Inspector()) + monkeypatch.setattr( + module.op, + "alter_column", + lambda *args, **kwargs: calls.append((args, kwargs)), + ) + + module.upgrade() + + assert calls == [ + ( + ("calendar_conflict_corrections", "rationale"), + {"new_column_name": "correction_rationale"}, + ) + ] + + +@pytest.mark.asyncio +@pytest.mark.postgres +async def test_calendar_correction_rationale_real_postgres_smoke(): + engine = create_async_engine(settings.DATABASE_URL) + try: + async with engine.begin() as conn: + await conn.execute( + text( + "CREATE TEMP TABLE calendar_conflict_corrections (" + "rationale text) ON COMMIT DROP" + ) + ) + await conn.run_sync(_run_0021_upgrade) + + def _column_names(sync_conn): + return { + column["name"] + for column in inspect(sync_conn).get_columns( + "calendar_conflict_corrections" + ) + } + + column_names = await conn.run_sync(_column_names) + assert "correction_rationale" in column_names + assert "rationale" not in column_names + except ( + ConnectionRefusedError, + OSError, + OperationalError, + asyncpg.CannotConnectNowError, + asyncpg.InvalidAuthorizationSpecificationError, + asyncpg.InvalidCatalogNameError, + asyncpg.InvalidPasswordError, + ): + await engine.dispose() + pytest.skip("PostgreSQL smoke path unavailable") + except Exception: + await engine.dispose() + raise + finally: + await engine.dispose() + + +@pytest.mark.asyncio +@pytest.mark.postgres +async def test_0001_initial_upgrade_succeeds_against_a_fresh_database(): + # 0001_initial_control_plane.py::upgrade() is what a genuinely fresh + # `alembic upgrade head` runs first. Base.metadata.create_all() never + # creates a table named "emails" (only "email_records" is ORM-modeled), + # so if this migration bypasses execute_schema_backfill's guard and + # blindly executes every schema_backfill_sql() statement itself, the + # legacy "ix_emails_owner_date" index statement raises + # 'relation "emails" does not exist' and a fresh install can never + # migrate at all. + engine = create_async_engine(settings.DATABASE_URL) + try: + async with engine.begin() as conn: + await conn.run_sync(_run_0001_upgrade) + result = await conn.execute( + text( + "SELECT indexname FROM pg_indexes " + "WHERE tablename = 'email_records' " + "AND indexname = 'ix_email_records_owner_date'" + ) + ) + assert result.scalar_one() == "ix_email_records_owner_date" + except ( + ConnectionRefusedError, + OSError, + OperationalError, + asyncpg.CannotConnectNowError, + asyncpg.InvalidAuthorizationSpecificationError, + asyncpg.InvalidCatalogNameError, + asyncpg.InvalidPasswordError, + ): + await engine.dispose() + pytest.skip("PostgreSQL smoke path unavailable") + except Exception: + await engine.dispose() + raise + finally: + await engine.dispose() + + +def test_email_workspace_migration_also_drops_bootstrap_created_owner_only_index(): + """backend/scripts/bootstrap_db.py's owner-only identity predates this + migration's own uq_emails_owner_message_id and uses a different name + (uq_email_records_owner_message_id) and a different catalog shape (a + plain index, not a named unique constraint). A database that was + bootstrap-initialized before bootstrap_db.py's own fix landed and is + later migrated via Alembic would keep that stricter 3-column identity + forever -- this migration's own get_unique_constraints()-only check can + never see it (wrong name, and get_unique_constraints never returns plain + indexes at all).""" + revision_text = ( + BACKEND_ROOT / "alembic" / "versions" / "0020_email_workspace_scope.py" + ).read_text() + + assert '"uq_email_records_owner_message_id"' in revision_text + assert "get_indexes(" in revision_text + assert "op.drop_index(" in revision_text + + +@pytest.mark.asyncio +@pytest.mark.postgres +@pytest.mark.parametrize("legacy_identity_as_constraint", [True, False]) +async def test_email_workspace_migration_real_postgres_smoke( + legacy_identity_as_constraint, +): + """inspector.get_indexes() also reports the backing index of a unique + constraint under the same name (PostgreSQL implements a unique + constraint via a unique index), so a check that only looks at + get_indexes() before get_unique_constraints() would try `DROP INDEX` on + a constraint's own backing index -- PostgreSQL rejects that outright + ("cannot drop index ... because constraint ... requires it"), aborting + the whole migration. bootstrap_db.py has only ever produced the legacy + identity as a plain index, but this proves the migration itself handles + either catalog shape without relying on that assumption.""" + engine = create_async_engine(settings.DATABASE_URL) + try: + async with engine.connect() as probe_conn: + original_email_records_oid = ( + await probe_conn.execute( + text( + "SELECT oid FROM pg_class " + "WHERE oid = to_regclass('email_records')" + ) + ) + ).scalar_one_or_none() + assert original_email_records_oid is not None + + async with engine.connect() as conn: + transaction = await conn.begin() + try: + + def _setup(sync_conn): + _setup_pre_0020_email_records( + sync_conn, + legacy_identity_as_constraint=legacy_identity_as_constraint, + ) + + await conn.run_sync(_setup) + await conn.run_sync(_run_0020_upgrade) + + def _inspect(sync_conn): + insp = inspect(sync_conn) + return ( + {i["name"] for i in insp.get_indexes("email_records")}, + { + c["name"] + for c in insp.get_unique_constraints("email_records") + }, + ) + + index_names, constraint_names = await conn.run_sync(_inspect) + finally: + await transaction.rollback() + + async with engine.connect() as probe_conn: + restored_email_records_oid = ( + await probe_conn.execute( + text( + "SELECT oid FROM pg_class " + "WHERE oid = to_regclass('email_records')" + ) + ) + ).scalar_one_or_none() + assert restored_email_records_oid == original_email_records_oid + except ( + ConnectionRefusedError, + OSError, + OperationalError, + asyncpg.CannotConnectNowError, + asyncpg.InvalidAuthorizationSpecificationError, + asyncpg.InvalidCatalogNameError, + asyncpg.InvalidPasswordError, + ): + await engine.dispose() + pytest.skip("PostgreSQL smoke path unavailable") + except Exception: + await engine.dispose() + raise + finally: + await engine.dispose() + + assert "uq_email_records_owner_message_id" not in index_names + assert "uq_email_records_owner_message_id" not in constraint_names + assert "uq_emails_workspace_message" in constraint_names def test_provider_writeback_retry_queue_has_incremental_revision(): @@ -422,6 +760,195 @@ def test_merge_revision_reconciles_email_read_state_branch(): assert "op.drop_column(" not in revision_text +def test_legacy_email_read_state_branch_defers_check_to_sql(monkeypatch): + revision_path = ( + BACKEND_ROOT / "alembic" / "versions" / "0011_email_read_state.py" + ) + revision_text = revision_path.read_text() + + # The legacy-table check must be evaluated in SQL (at apply time), not in + # Python at generation time: offline SQL generation (`alembic upgrade + # --sql`, a real flag `scripts/migrate_db.py` exposes) has no live + # connection to introspect with, and the one generated script is meant to + # later be applied against whichever database a DBA chooses -- a + # Python-side sa.inspect(op.get_bind()) check can only ever bake in one + # fixed answer, which is wrong for whichever kind of target it didn't + # assume (silently skips the real column on a legacy target while + # `alembic_version` still advances, or crashes outright against a fresh + # one). upgrade()/downgrade() themselves must contain no such check -- + # only op.execute() calls -- so this can't regress into + # either failure mode. + assert "def upgrade" in revision_text + upgrade_and_after = revision_text.split("def upgrade", 1)[1] + assert "sa.inspect(op.get_bind())" not in upgrade_and_after + assert "context.is_offline_mode()" not in upgrade_and_after + + module = _load_revision_module("0011_email_read_state.py") + # to_regclass('emails'), not information_schema.tables by bare + # table_name: the latter ignores search_path and can match an unrelated + # same-named table in a different accessible schema than the one the + # unqualified ALTER TABLE below actually resolves to. Checked on the + # loaded module's own SQL constants, not the raw file text, so this + # can't be fooled by a comment mentioning either string for context. + assert "DO $$" in module._UPGRADE_SQL + assert "DO $$" in module._DOWNGRADE_SQL + assert "to_regclass('emails')" in module._UPGRADE_SQL + assert "to_regclass('emails')" in module._DOWNGRADE_SQL + assert "information_schema.tables" not in module._UPGRADE_SQL + assert "information_schema.tables" not in module._DOWNGRADE_SQL + assert "ALTER TABLE emails ADD COLUMN is_read" in module._UPGRADE_SQL + # CodeRabbit (naruon#1501): downgrade must drop emails.is_read only when + # this revision's own upgrade created it, not whenever the column merely + # happens to be present -- an unconditional DROP would also destroy a + # pre-existing, unrelated is_read column and its data. upgrade() tags the + # column it creates with a provenance marker comment; downgrade() checks + # that exact marker via col_description before dropping. + assert "COMMENT ON COLUMN emails.is_read" in module._UPGRADE_SQL + assert module._IS_READ_PROVENANCE_MARKER in module._UPGRADE_SQL + assert "col_description" in module._DOWNGRADE_SQL + assert module._IS_READ_PROVENANCE_MARKER in module._DOWNGRADE_SQL + + calls = [] + monkeypatch.setattr(module.op, "execute", lambda sql: calls.append(sql)) + module.upgrade() + module.downgrade() + assert len(calls) == 2 + assert "ADD COLUMN is_read" in calls[0] + assert "DROP COLUMN IF EXISTS is_read" in calls[1] + + +def _run_0011_upgrade(sync_conn) -> None: + from alembic.operations import Operations + from alembic.runtime.migration import MigrationContext + + module = _load_revision_module("0011_email_read_state.py") + context = MigrationContext.configure(sync_conn, opts={"target_metadata": None}) + with Operations.context(context): + module.upgrade() + + +def _run_0011_downgrade(sync_conn) -> None: + from alembic.operations import Operations + from alembic.runtime.migration import MigrationContext + + module = _load_revision_module("0011_email_read_state.py") + context = MigrationContext.configure(sync_conn, opts={"target_metadata": None}) + with Operations.context(context): + module.downgrade() + + +@pytest.mark.asyncio +@pytest.mark.postgres +async def test_legacy_email_read_state_real_postgres_smoke(): + """Both directions this migration must get right against a real database: + a legacy target (still has the ``emails`` table) gets the column added and + later removed; a fresh-baseline target (no ``emails`` table at all, the + now-common case) is left untouched rather than erroring. + """ + engine = create_async_engine(settings.DATABASE_URL) + try: + try: + async with engine.connect() as probe_conn: + await probe_conn.execute(text("SELECT 1")) + except ( + ConnectionRefusedError, + OSError, + OperationalError, + asyncpg.CannotConnectNowError, + asyncpg.InvalidAuthorizationSpecificationError, + asyncpg.InvalidCatalogNameError, + asyncpg.InvalidPasswordError, + ): + pytest.skip("PostgreSQL smoke path unavailable") + + async with engine.begin() as conn: + # Fresh-baseline case first, on a connection with no "emails" + # table anywhere in scope: must no-op, not raise. + await conn.run_sync(_run_0011_upgrade) + + await conn.execute( + text("CREATE TEMP TABLE emails (id serial primary key) ON COMMIT DROP") + ) + + def _has_is_read(sync_conn): + return any( + column["name"] == "is_read" + for column in inspect(sync_conn).get_columns("emails") + ) + + assert not await conn.run_sync(_has_is_read) + await conn.run_sync(_run_0011_upgrade) + assert await conn.run_sync(_has_is_read) + + await conn.run_sync(_run_0011_downgrade) + assert not await conn.run_sync(_has_is_read) + finally: + await engine.dispose() + + +@pytest.mark.asyncio +@pytest.mark.postgres +async def test_legacy_email_read_state_downgrade_preserves_a_preexisting_column(): + """downgrade() must not drop an ``emails.is_read`` column (or its data) + that predates this revision -- CodeRabbit flagged the earlier + unconditional ``DROP COLUMN IF EXISTS`` on naruon#1501: since upgrade()'s + ``NOT EXISTS`` guard already leaves a pre-existing column untouched + (never adding its own provenance marker to it), downgrade() must + symmetrically leave it alone too, distinguishing "this revision added it" + from "it merely happens to be present" via the marker set on the + ``COMMENT ON COLUMN`` this revision's own upgrade() applies. + """ + engine = create_async_engine(settings.DATABASE_URL) + try: + try: + async with engine.connect() as probe_conn: + await probe_conn.execute(text("SELECT 1")) + except ( + ConnectionRefusedError, + OSError, + OperationalError, + asyncpg.CannotConnectNowError, + asyncpg.InvalidAuthorizationSpecificationError, + asyncpg.InvalidCatalogNameError, + asyncpg.InvalidPasswordError, + ): + pytest.skip("PostgreSQL smoke path unavailable") + + async with engine.begin() as conn: + await conn.execute( + text( + "CREATE TEMP TABLE emails (id serial primary key, " + "is_read boolean NOT NULL DEFAULT false) ON COMMIT DROP" + ) + ) + await conn.execute(text("INSERT INTO emails (is_read) VALUES (false)")) + + def _has_is_read(sync_conn): + return any( + column["name"] == "is_read" + for column in inspect(sync_conn).get_columns("emails") + ) + + # upgrade() must be a no-op here: the column already exists, so + # its NOT EXISTS guard skips both the ADD COLUMN and the marker + # COMMENT -- this pre-existing column is never tagged as "added + # by this revision". + assert await conn.run_sync(_has_is_read) + await conn.run_sync(_run_0011_upgrade) + assert await conn.run_sync(_has_is_read) + + # downgrade() must leave the untagged, pre-existing column (and + # its data) alone rather than dropping it. + await conn.run_sync(_run_0011_downgrade) + assert await conn.run_sync(_has_is_read) + preserved_value = ( + await conn.execute(text("SELECT is_read FROM emails")) + ).scalar_one() + assert preserved_value is False + finally: + await engine.dispose() + + def test_merge_revision_reconciles_newsdom_provider_branch(): revision_path = ( BACKEND_ROOT / "alembic" / "versions" / "0015_merge_newsdom_email_heads.py" diff --git a/backend/tests/test_attachment_parser.py b/backend/tests/test_attachment_parser.py index 4eeb27228..7ae541886 100644 --- a/backend/tests/test_attachment_parser.py +++ b/backend/tests/test_attachment_parser.py @@ -7,6 +7,7 @@ MAX_ATTACHMENT_PARSE_SOURCE_BYTES, MAX_ATTACHMENT_PARSE_SOURCE_CHARS, decode_deferred_attachment_payload, + decode_quarantined_attachment_payload, get_attachment_parser_manifest, parse_email_attachment, ) @@ -258,6 +259,190 @@ def test_deferred_pdf_decoder_rejects_non_pdf_and_oversized_payloads(monkeypatch decode_deferred_attachment_payload(oversized) +def test_quarantined_payload_decoder_round_trips_any_sniffed_type(): + raw = b"\x89PNG\r\n\x1a\n" + b"real png bytes, not a pdf" + encoded = base64.b64encode(raw).decode("ascii") + + assert decode_quarantined_attachment_payload(encoded) == raw + + +def test_quarantined_payload_decoder_rejects_bad_base64_and_oversized_payloads( + monkeypatch, +): + with pytest.raises(ValueError, match="not valid base64"): + decode_quarantined_attachment_payload("not@@base64!!") + + monkeypatch.setattr( + "services.attachment_parser.MAX_ATTACHMENT_PARSE_SOURCE_BYTES", 5 + ) + oversized = base64.b64encode(b"more than five bytes").decode("ascii") + with pytest.raises(ValueError, match="size limit"): + decode_quarantined_attachment_payload(oversized) + + +def test_declared_pdf_with_real_png_bytes_is_quarantined(): + raw = b"\x89PNG\r\n\x1a\n" + b"rest of a real png payload" + result = parse_email_attachment( + filename="invoice.pdf", + content_type="application/pdf", + raw_content=raw, + ) + + assert result.content_type == "application/pdf" + assert result.parse_content_type == "image/png" + assert result.parse_content == "" + assert result.parse_status == "content_type_mismatch_quarantined" + assert result.parse_error_code == "content_type_mismatch_quarantined" + assert base64.b64decode(result.content) == raw + + +def test_declared_text_plain_with_real_zip_bytes_is_quarantined(): + raw = b"PK\x03\x04" + b"fake zip body" + result = parse_email_attachment( + filename="notes.txt", + content_type="text/plain", + raw_content=raw, + ) + + assert result.content_type == "text/plain" + assert result.parse_content_type == "application/zip" + assert result.parser_key == "unsupported_binary" + assert result.parse_status == "content_type_mismatch_quarantined" + assert result.parse_error_code == "content_type_mismatch_quarantined" + assert base64.b64decode(result.content) == raw + + +def test_oversized_mismatched_payload_is_size_limited_not_quarantined(monkeypatch): + """An oversized mismatch must not enter quarantine with no retained bytes. + + A quarantined row with no bytes is indistinguishable, to the + reparse-intent API, from one that still has something to re-evaluate -- + it must get the same non-retryable status every other oversized + attachment already gets instead. + """ + monkeypatch.setattr( + "services.attachment_parser.MAX_ATTACHMENT_PARSE_SOURCE_BYTES", 10 + ) + raw = b"\x89PNG\r\n\x1a\n" + b"A" * 20 + + result = parse_email_attachment( + filename="picture.pdf", + content_type="application/pdf", + raw_content=raw, + ) + + assert result.parse_status == "parse_size_limit_exceeded" + assert result.parse_error_code == "parse_size_limit_exceeded" + assert result.content == "" + + +def test_declared_docx_with_real_zip_bytes_is_not_quarantined(): + """A real DOCX is a ZIP container by specification -- its own magic bytes.""" + raw = b"PK\x03\x04" + b"fake docx body" + result = parse_email_attachment( + filename="report.docx", + content_type=( + "application/vnd.openxmlformats-officedocument.wordprocessingml.document" + ), + raw_content=raw, + ) + + assert result.parse_status == "unsupported_content_type" + assert result.parse_error_code == "unsupported_content_type" + assert result.content_type == ( + "application/vnd.openxmlformats-officedocument.wordprocessingml.document" + ) + + +def test_declared_xlsx_and_pptx_with_real_zip_bytes_are_not_quarantined(): + raw = b"PK\x03\x04" + b"fake office body" + cases = [ + ( + "budget.xlsx", + "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet", + ), + ( + "deck.pptx", + "application/vnd.openxmlformats-officedocument.presentationml.presentation", + ), + ] + for filename, content_type in cases: + result = parse_email_attachment( + filename=filename, content_type=content_type, raw_content=raw + ) + assert result.parse_status != "content_type_mismatch_quarantined" + + +def test_declared_pdf_with_real_zip_bytes_is_still_quarantined(): + """A ZIP disguised as a PDF is a genuine mismatch, unlike a real OOXML file.""" + raw = b"PK\x03\x04" + b"fake body" + result = parse_email_attachment( + filename="not-a-pdf.pdf", + content_type="application/pdf", + raw_content=raw, + ) + + assert result.parse_status == "content_type_mismatch_quarantined" + + +def test_matching_declared_and_sniffed_binary_type_is_not_quarantined(): + """A real PDF declared as a PDF must go through the normal deferred path.""" + result = parse_email_attachment( + filename="contract.pdf", + content_type="application/pdf", + raw_content=b"%PDF-1.7 real payload", + ) + + assert result.parse_status == "pdf_dom_recognition_pending" + + +@pytest.mark.parametrize( + ("content_type", "filename"), + [ + ("application/octet-stream", "photo.bin"), + ("binary/octet-stream", "photo"), + ("application/x-binary", "photo.dat"), + (None, "photo"), + ("", "photo"), + ], +) +def test_generic_content_type_with_no_recognized_extension_is_not_quarantined( + content_type, filename +): + """A generic MIME type is not a claim, so recognized bytes cannot disagree with it. + + ``application/octet-stream`` (and its variants) is MIME's own "no more + specific type available" placeholder, never a positive assertion about + content -- a sender who sends an ordinary PNG/PDF/ZIP this way hasn't + disguised anything, and quarantining it forever (with no declared-type + fix ever able to clear it, since the sender already sent the only type + they were going to send) defeats attachments no one lied about. + """ + raw = b"\x89PNG\r\n\x1a\n" + b"real png bytes" + result = parse_email_attachment( + filename=filename, content_type=content_type, raw_content=raw + ) + + assert result.parse_status != "content_type_mismatch_quarantined" + + +def test_generic_content_type_resolved_via_extension_still_detects_mismatch(): + """A generic type that resolves to something specific via extension keeps quarantining. + + Once ``_parse_content_type_for`` resolves a generic declaration to a + real claim via a recognized extension (here ``.pdf``), that *is* a + specific assertion the sniffed bytes can genuinely disagree with -- the + generic-type carve-out must not swallow this case. + """ + raw = b"\x89PNG\r\n\x1a\n" + b"real png bytes" + result = parse_email_attachment( + filename="invoice.pdf", content_type="application/octet-stream", raw_content=raw + ) + + assert result.parse_status == "content_type_mismatch_quarantined" + assert result.parse_error_code == "content_type_mismatch_quarantined" + + def test_safe_filename_handles_windows_path_traversal(): assert _safe_filename("..\\..\\upload.txt") == "upload.txt" assert _safe_filename("C:\\mail\\report.pdf") == "report.pdf" diff --git a/backend/tests/test_attachment_parser_zip_magic_regression.py b/backend/tests/test_attachment_parser_zip_magic_regression.py new file mode 100644 index 000000000..9d61ecebf --- /dev/null +++ b/backend/tests/test_attachment_parser_zip_magic_regression.py @@ -0,0 +1,22 @@ +"""Regression coverage for ZIP signatures without local file headers.""" + +from services.attachment_parser import ( + CONTENT_TYPE_MISMATCH_QUARANTINED_STATUS, + parse_email_attachment, +) + + +def test_empty_zip_disguised_as_text_is_quarantined() -> None: + """An empty ZIP starts with EOCD, not the usual local-file-header magic.""" + empty_zip = b"PK\x05\x06" + (b"\x00" * 18) + + result = parse_email_attachment( + filename="meeting-notes.txt", + content_type="text/plain", + raw_content=empty_zip, + ) + + assert result.parse_status == CONTENT_TYPE_MISMATCH_QUARANTINED_STATUS + assert result.parse_error_code == CONTENT_TYPE_MISMATCH_QUARANTINED_STATUS + assert result.parse_content_type == "application/zip" + assert result.content_type == "text/plain" diff --git a/backend/tests/test_attachment_reparse_worker.py b/backend/tests/test_attachment_reparse_worker.py new file mode 100644 index 000000000..13e8cf6cb --- /dev/null +++ b/backend/tests/test_attachment_reparse_worker.py @@ -0,0 +1,1125 @@ +"""Tests for attachment reparse classification and persisted worker processing. + +Most tests use in-memory ``Attachment`` instances and a fake async session; +one PostgreSQL smoke test covers the real async persistence boundary. Covers +the fail-closed outcome (invalid retained +payload -> a dedicated terminal status) alongside the two "successful +re-evaluation" outcomes: a previously-quarantined attachment whose +disagreement is now recognized as legitimate (escapes quarantine), and one +whose disagreement is still genuine (stays quarantined). +""" + +import asyncio +import base64 +import datetime +from types import SimpleNamespace +import uuid + +import asyncpg +import pytest +from sqlalchemy import delete, select, text +from sqlalchemy.exc import OperationalError +from sqlalchemy.orm import selectinload, undefer + +from core.config import settings +from db.models import ( + Attachment, + ContentNodeRecord, + ContentSegmentRecord, + Email, + KnowledgeGraphEdgeRecord, +) +from db.session import AsyncSessionLocal +import services.attachment_reparse_worker as attachment_reparse_worker_module +import services.email_import_service as email_import_service_module +from services.content_graph import content_graph_source_record_uid + +AttachmentReparseWorker = attachment_reparse_worker_module.AttachmentReparseWorker +ATTACHMENT_REPARSE_PENDING_STATUS = ( + attachment_reparse_worker_module.ATTACHMENT_REPARSE_PENDING_STATUS +) +ATTACHMENT_REPARSE_PAYLOAD_INVALID_STATUS = ( + attachment_reparse_worker_module.ATTACHMENT_REPARSE_PAYLOAD_INVALID_STATUS +) +RESULT_DECODE_FAILED = attachment_reparse_worker_module.RESULT_DECODE_FAILED +process_reparse_pending_attachment = ( + attachment_reparse_worker_module.process_reparse_pending_attachment +) + +_QUARANTINED_STATUS = "content_type_mismatch_quarantined" + + +@pytest.mark.asyncio +@pytest.mark.postgres +async def test_persisted_reparse_commits_topology_and_provider_embedding(monkeypatch): + """Exercise the real AsyncSession relationship and pgvector persistence path.""" + if not settings.DATABASE_URL: + pytest.skip("PostgreSQL smoke path unavailable") + try: + async with AsyncSessionLocal() as probe_session: + await probe_session.execute(text("SELECT 1")) + except ( + ConnectionRefusedError, + OSError, + OperationalError, + asyncpg.CannotConnectNowError, + asyncpg.InvalidAuthorizationSpecificationError, + asyncpg.InvalidCatalogNameError, + asyncpg.InvalidPasswordError, + ): + pytest.skip("PostgreSQL smoke path unavailable") + + suffix = uuid.uuid4().hex + long_content = ("alpha " * 400) + "\n\n" + ("beta " * 400) + provider_batches: list[list[str]] = [] + + async def runtime_provider(*_args, **_kwargs): + return SimpleNamespace( + api_key="test-provider-key", + base_url="https://provider.example/v1", + embedding_model="embedding-test-model", + ) + + async def generated_embeddings(texts, *, embedding_provider, batch_context=None): + assert embedding_provider.base_url == "https://provider.example/v1" + assert embedding_provider.embedding_model == "embedding-test-model" + assert batch_context is None + start = sum(len(batch) for batch in provider_batches) + provider_batches.append(list(texts)) + return [ + [float(start + index + 1)] * 1536 for index in range(len(texts)) + ] + + monkeypatch.setattr( + attachment_reparse_worker_module, + "resolve_runtime_llm_provider", + runtime_provider, + ) + monkeypatch.setattr( + email_import_service_module, + "_generate_import_embeddings", + generated_embeddings, + ) + + async with AsyncSessionLocal() as session: + email = Email( + user_id=f"reparse-user-{suffix}", + organization_id=f"reparse-org-{suffix}", + workspace_id=f"reparse-workspace-{suffix}", + message_id=f"reparse-message-{suffix}", + sender="sender@example.com", + recipients="recipient@example.com", + subject="Reparse persistence smoke", + date=datetime.datetime.now(datetime.timezone.utc), + body="body", + embedding=[0.0] * 1536, + ) + attachment = _reparse_pending_attachment( + content_type="text/plain", + payload=long_content.encode(), + attachment_uid=f"attachment_{suffix}", + ) + email.attachments.append(attachment) + session.add(email) + await session.commit() + attachment_id = attachment.id + email_id = email.id + + worker = AttachmentReparseWorker(batch_limit=1) + worker._attachment_cursor = attachment.id - 1 + try: + await worker._sweep_attachments(session) + persisted = ( + await session.execute( + select(Attachment) + .where(Attachment.attachment_uid == attachment.attachment_uid) + .options( + selectinload(Attachment.content_nodes), + selectinload(Attachment.content_segments), + selectinload(Attachment.knowledge_graph_edges), + undefer(Attachment.embedding), + ) + .execution_options(populate_existing=True) + ) + ).scalar_one() + assert persisted.parse_status == "parsed" + assert len(persisted.content_nodes) == 3 + assert len(persisted.content_segments) == 2 + assert {edge.edge_kind for edge in persisted.knowledge_graph_edges} == { + "node_contains_node", + "node_has_segment", + "segment_next", + } + chunk_count = sum(len(batch) for batch in provider_batches) + assert chunk_count > 1 + assert all( + 0 < len(batch) <= email_import_service_module.MAX_EMBEDDING_CHUNKS_PER_WINDOW + for batch in provider_batches + ) + expected_value = sum(range(1, chunk_count + 1)) / chunk_count + expected_embedding = [expected_value] * 1536 + assert list(persisted.embedding) == expected_embedding + finally: + await session.rollback() + await session.execute( + delete(KnowledgeGraphEdgeRecord).where( + KnowledgeGraphEdgeRecord.attachment_id == attachment_id + ) + ) + await session.execute( + delete(ContentSegmentRecord).where( + ContentSegmentRecord.attachment_id == attachment_id + ) + ) + await session.execute( + delete(ContentNodeRecord).where( + ContentNodeRecord.attachment_id == attachment_id + ) + ) + await session.execute( + delete(Attachment).where(Attachment.id == attachment_id) + ) + await session.execute(delete(Email).where(Email.id == email_id)) + await session.commit() + + +def _reparse_pending_attachment( + *, + content_type: str, + payload: bytes, + filename: str = "attachment.bin", + attachment_id: int | None = None, + email_id: int | None = None, + attachment_uid: str = "attachment_test-uid", +) -> Attachment: + return Attachment( + id=attachment_id, + email_id=email_id, + attachment_uid=attachment_uid, + filename=filename, + content_type=content_type, + content=base64.b64encode(payload).decode("ascii"), + parse_content_type="application/zip", + parser_key="unsupported_binary", + parse_status=ATTACHMENT_REPARSE_PENDING_STATUS, + parse_error_code=None, + ) + + +def test_reparse_escapes_a_now_recognized_false_positive(): + # A .docx declared with its real OOXML content type but sniffed as a + # plain ZIP -- exactly the false-positive family the OOXML/ODF/EPUB/JAR + # carve-out fixed. Reparsing no longer flags it as a mismatch; it lands + # on the ordinary "unsupported_content_type" classification (this parser + # has no dedicated OOXML parser), never back in quarantine. + docx_content_type = ( + "application/vnd.openxmlformats-officedocument.wordprocessingml.document" + ) + attachment = _reparse_pending_attachment( + content_type=docx_content_type, + payload=b"PK\x03\x04" + b"real docx zip bytes", + filename="report.docx", + ) + + outcome = process_reparse_pending_attachment(attachment=attachment) + + assert outcome.parse_status == "unsupported_content_type" + assert attachment.parse_status == "unsupported_content_type" + assert attachment.parse_error_code == "unsupported_content_type" + assert attachment.parse_status != _QUARANTINED_STATUS + + +def test_reparse_to_unsupported_content_type_preserves_retained_bytes(): + # Same false-positive escape as above, but this asserts the one thing + # that test doesn't: parse_email_attachment returns content="" for + # unsupported_content_type (nothing to display), and apply_reparsed_result + # must not let that empty result overwrite the only retained copy of the + # original quarantined bytes -- there would be no way to ever recover or + # re-attempt parsing on this attachment again. + docx_content_type = ( + "application/vnd.openxmlformats-officedocument.wordprocessingml.document" + ) + payload = b"PK\x03\x04" + b"real docx zip bytes" + attachment = _reparse_pending_attachment( + content_type=docx_content_type, + payload=payload, + filename="report.docx", + ) + retained_content = attachment.content + + outcome = process_reparse_pending_attachment(attachment=attachment) + + assert outcome.parse_status == "unsupported_content_type" + assert attachment.content == retained_content + assert base64.b64decode(attachment.content) == payload + + +def test_reparse_of_a_genuine_mismatch_returns_to_quarantine(): + # PNG bytes declared as a PDF -- a real disguise, unrelated to any parser + # bug. Reparsing must reconfirm the same quarantine, not silently parse. + attachment = _reparse_pending_attachment( + content_type="application/pdf", + payload=b"\x89PNG\r\n\x1a\n" + b"real png bytes", + filename="invoice.pdf", + ) + + outcome = process_reparse_pending_attachment(attachment=attachment) + + assert outcome.parse_status == _QUARANTINED_STATUS + assert attachment.parse_status == _QUARANTINED_STATUS + assert attachment.parse_error_code == _QUARANTINED_STATUS + assert attachment.parse_content_type == "image/png" + + +def test_reparse_rejects_an_invalid_retained_payload(): + attachment = _reparse_pending_attachment( + content_type="application/pdf", payload=b"irrelevant" + ) + attachment.content = "not@@base64!!" + + outcome = process_reparse_pending_attachment(attachment=attachment) + + assert outcome.parse_status == RESULT_DECODE_FAILED + assert attachment.parse_status == ATTACHMENT_REPARSE_PAYLOAD_INVALID_STATUS + assert attachment.parse_error_code == ATTACHMENT_REPARSE_PAYLOAD_INVALID_STATUS + + +def test_reparse_preserves_filename_and_declared_content_type(): + attachment = _reparse_pending_attachment( + content_type="application/pdf", + payload=b"\x89PNG\r\n\x1a\n" + b"real png bytes", + filename="invoice.pdf", + ) + + process_reparse_pending_attachment(attachment=attachment) + + assert attachment.filename == "invoice.pdf" + assert attachment.content_type == "application/pdf" + + +def test_reparse_that_lands_on_parsed_indexes_the_content_graph(): + # Unlike the OOXML/PDF/PNG scenarios above, plain text is never + # magic-byte-sniffed (see attachment_parser._MAGIC_BYTE_SIGNATURES), so + # this reparse lands on the ordinary "parsed" status -- exactly the + # outcome email_import_service._append_email_content_graph already + # builds a content graph record for on a cleanly-first-parsed attachment. + # apply_reparsed_result must do the same on this path, or a reparsed + # attachment stays invisible to content-graph-backed search/AI-hub + # features even after successful recognition. + attachment = _reparse_pending_attachment( + content_type="text/plain", + payload=b"Meeting notes\n\nDiscuss the roadmap.", + filename="notes.txt", + email_id=42, + attachment_uid="attachment_notes-uid", + ) + + outcome = process_reparse_pending_attachment(attachment=attachment) + + assert outcome.parse_status == "parsed" + assert outcome.embedding_source_text == "Meeting notes\n\nDiscuss the roadmap." + assert attachment.parse_status == "parsed" + assert [node.node_kind for node in attachment.content_nodes] == [ + "document", + "paragraph", + "paragraph", + ] + assert [ + segment.safe_text_content for segment in attachment.content_segments + ] == ["Meeting notes", "Discuss the roadmap."] + assert {node.source_kind for node in attachment.content_nodes} == {"attachment"} + assert {segment.source_kind for segment in attachment.content_segments} == { + "attachment" + } + expected_source_record_uid = content_graph_source_record_uid( + "attachment", "attachment_notes-uid" + ) + assert {node.source_record_uid for node in attachment.content_nodes} == { + expected_source_record_uid + } + assert {node.email_id for node in attachment.content_nodes} == {42} + assert {segment.email_id for segment in attachment.content_segments} == {42} + # Every segment is linked back to its parent node's own segments list too + # (the same node<->segment wiring _append_parse_result_records builds). + assert sum(len(node.segments) for node in attachment.content_nodes) == 2 + assert {edge.edge_kind for edge in attachment.knowledge_graph_edges} == { + "node_contains_node", + "node_has_segment", + "segment_next", + } + + +def test_reparse_that_lands_on_parsed_with_blank_content_does_not_index_content_graph(): + # A reparse can land on "parsed" with nothing displayable (an empty or + # whitespace-only retained payload) -- parse_email_attachment does not + # special-case that. Indexing an empty content graph record for it would + # be pure noise, so this must be skipped exactly like + # _append_email_content_graph skips a blank attachment on import. + attachment = _reparse_pending_attachment( + content_type="text/plain", + payload=b" ", + filename="blank.txt", + email_id=42, + ) + + outcome = process_reparse_pending_attachment(attachment=attachment) + + assert outcome.parse_status == "parsed" + assert attachment.content_nodes == [] + assert attachment.content_segments == [] + + +def test_reparse_that_lands_on_parsed_with_markup_only_content_still_embeds_parse_content(): + # A "parsed" result whose *display* text (content) strips down to empty + # while its *parse* text (parse_content) does not -- e.g. an attachment + # that is only markup with no visible text nodes. apply_reparsed_result's + # `if result.content:` guard (see its docstring) then leaves + # attachment.content untouched, so the embedding source must come from + # ReparseOutcome.embedding_source_text (parse_content preferred over + # content, matching _append_reparsed_attachment_content_graph and + # email_import_service._extract_and_generate_embeddings), never from + # attachment.content directly -- CodeRabbit flagged this exact mismatch + # on naruon#1501. + attachment = _reparse_pending_attachment( + content_type="text/plain", + payload=b"
", + filename="markup-only.txt", + email_id=42, + ) + + outcome = process_reparse_pending_attachment(attachment=attachment) + + assert outcome.parse_status == "parsed" + # result.content ("") is falsy, so apply_reparsed_result's `if + # result.content:` guard leaves attachment.content at its retained, + # still-base64-encoded original value -- exactly why the embedding + # source cannot come from attachment.content. + assert attachment.content == base64.b64encode(b"
").decode("ascii") + assert outcome.embedding_source_text == "
" + + +def test_reparse_that_does_not_land_on_parsed_does_not_index_content_graph(): + attachment = _reparse_pending_attachment( + content_type="application/pdf", + payload=b"\x89PNG\r\n\x1a\n" + b"real png bytes", + filename="invoice.pdf", + email_id=42, + ) + + outcome = process_reparse_pending_attachment(attachment=attachment) + + assert outcome.parse_status == _QUARANTINED_STATUS + assert attachment.content_nodes == [] + assert attachment.content_segments == [] + + +class _RowsResult: + def __init__(self, rows): + self._rows = rows + + def scalars(self): + return self + + def all(self): + return self._rows + + +class _SequenceSession: + """A fake session whose ``get`` always returns a fresh, healthy instance. + + Mirrors the real ``AttachmentReparseWorker._sweep_attachments`` contract: + the bulk-loaded rows only supply ids and the cursor; every actual + processing target is re-fetched via ``get`` by id, exactly like a real + ``AsyncSession`` would after an earlier item's rollback expired the + bulk-loaded instances. ``by_id`` lets a test register a *different* + object than the bulk-loaded one to prove that re-fetch is what the sweep + actually uses. + """ + + def __init__(self, row_batches, *, by_id=None): + self._row_batches = list(row_batches) + self._by_id = dict(by_id or {}) + for batch in self._row_batches: + for attachment in batch: + self._by_id.setdefault(attachment.id, attachment) + self.statements = [] + self.commit_count = 0 + self.rollback_count = 0 + self.refresh_calls = [] + + async def execute(self, statement): + self.statements.append(statement) + return _RowsResult(self._row_batches.pop(0)) + + async def get(self, _model, attachment_id): + return self._by_id.get(attachment_id) + + async def refresh(self, attachment, *, attribute_names): + self.refresh_calls.append((attachment.id, tuple(attribute_names))) + + async def commit(self): + self.commit_count += 1 + + async def rollback(self): + self.rollback_count += 1 + + +class _AsyncSessionContext: + def __init__(self, session): + self.session = session + + async def __aenter__(self): + return self.session + + async def __aexit__(self, *_args): + return False + + +class _LeaseConnection: + def __init__(self, *, scalar_result=True): + self.scalar_result = scalar_result + self.scalar_calls = [] + self.execution_options_calls = [] + # Ordered log spanning both call types, so a test can prove + # AUTOCOMMIT was set BEFORE the advisory-lock query ran, not just + # that both happened at some point. + self.ordered_calls = [] + + async def execution_options(self, **options): + self.execution_options_calls.append(options) + self.ordered_calls.append(("execution_options", options)) + return self + + async def scalar(self, statement, params): + self.scalar_calls.append((statement, params)) + self.ordered_calls.append(("scalar", params)) + return self.scalar_result + + +class _FakeEngine: + def __init__(self, *, dialect_name="postgresql", connection=None): + self.dialect = SimpleNamespace(name=dialect_name) + self._connection = connection + + def connect(self): + return _LeaseConnectionContext(self._connection) + + +@pytest.mark.asyncio +async def test_sweep_advances_the_cursor_and_retries_the_failed_row( + monkeypatch, +): + # An earlier version (CodeRabbit-flagged) capped the cursor at the first + # failure instead of advancing it, to keep that one failing row + # selectable later. That pinned the whole batch window behind it once + # more than batch_limit rows failed at once -- nothing past them was + # ever reached (same class of bug fixed in + # services.newsdom_worker.NewsdomRecognitionWorker._sweep_attachments; + # see its docstring). The cursor now always advances to the batch's + # last row; the failed row is retried instead via the independent + # _attachment_retry_ids set. + first = _reparse_pending_attachment( + content_type="application/pdf", + payload=b"\x89PNG\r\n\x1a\n" + b"png", + attachment_id=1, + ) + second = _reparse_pending_attachment( + content_type="application/pdf", + payload=b"\x89PNG\r\n\x1a\n" + b"png", + attachment_id=2, + ) + third = _reparse_pending_attachment( + content_type="application/pdf", + payload=b"\x89PNG\r\n\x1a\n" + b"png", + attachment_id=3, + ) + session = _SequenceSession([[first, second, third], [second]]) + + real_process = attachment_reparse_worker_module.process_reparse_pending_attachment + failed_once = set() + + def fail_the_middle_item_once(*, attachment): + if attachment.id == 2 and attachment.id not in failed_once: + failed_once.add(attachment.id) + raise RuntimeError("classification blew up") + return real_process(attachment=attachment) + + monkeypatch.setattr( + attachment_reparse_worker_module, + "process_reparse_pending_attachment", + fail_the_middle_item_once, + ) + worker = AttachmentReparseWorker() + await worker._sweep_attachments(session) + + assert worker._attachment_cursor == 3 + assert worker._attachment_retry_ids == {2} + assert first.parse_status == _QUARANTINED_STATUS + assert second.parse_status == ATTACHMENT_REPARSE_PENDING_STATUS + assert third.parse_status == _QUARANTINED_STATUS + assert session.commit_count == 2 + assert session.rollback_count == 1 + + # The next sweep reselects row 2 via the retry set, not the cursor + # (which stays at 3, well past it). + await worker._sweep_attachments(session) + + assert worker._attachment_cursor == 3 + assert worker._attachment_retry_ids == set() + assert second.parse_status == _QUARANTINED_STATUS + assert session.commit_count == 3 + + +@pytest.mark.asyncio +async def test_sweep_advances_the_cursor_across_batches(): + first = _reparse_pending_attachment( + content_type="application/pdf", + payload=b"\x89PNG\r\n\x1a\n" + b"png", + attachment_id=1, + ) + second = _reparse_pending_attachment( + content_type="application/pdf", + payload=b"\x89PNG\r\n\x1a\n" + b"png", + attachment_id=2, + ) + session = _SequenceSession([[first], [second]]) + worker = AttachmentReparseWorker(batch_limit=1) + + await worker._sweep_attachments(session) + await worker._sweep_attachments(session) + + second_query = session.statements[1].compile() + assert "email_attachments.id >" in str(second_query) + assert 1 in second_query.params.values() + assert worker._attachment_cursor == 2 + assert first.parse_status == _QUARANTINED_STATUS + assert second.parse_status == _QUARANTINED_STATUS + assert session.commit_count == 2 + assert session.rollback_count == 0 + assert session.refresh_calls == [ + (1, ("email", "content_nodes", "content_segments", "knowledge_graph_edges")), + (2, ("email", "content_nodes", "content_segments", "knowledge_graph_edges")), + ] + + +@pytest.mark.asyncio +async def test_load_reparse_pending_attachments_queries_forward_cursor_and_retry_ids(): + # No more wraparound: a persistently-failing row is retried via an + # explicit "id IN retry_ids" filter, independent of the forward cursor, + # instead of relying on the query going empty to trigger a rescan (which + # never happens once new reparse-intent rows keep landing past the + # cursor). + row = _reparse_pending_attachment( + content_type="application/pdf", + payload=b"\x89PNG\r\n\x1a\n" + b"png", + attachment_id=5, + ) + session = _SequenceSession([[row]]) + worker = AttachmentReparseWorker(batch_limit=10) + worker._attachment_cursor = 3 + worker._attachment_retry_ids = {1} + + rows = await worker._load_reparse_pending_attachments(session) + + assert rows == [row] + compiled = str(session.statements[0].compile()) + assert "email_attachments.id >" in compiled + assert "IN" in compiled.upper() + + +@pytest.mark.asyncio +async def test_empty_sweep_leaves_cursor_unset(): + session = _SequenceSession([[]]) + worker = AttachmentReparseWorker() + + await worker._sweep_attachments(session) + + assert worker._attachment_cursor is None + assert session.commit_count == 0 + + +@pytest.mark.asyncio +async def test_sweep_rolls_back_one_item_failure_and_continues_isolation(monkeypatch): + attachment = _reparse_pending_attachment( + content_type="application/pdf", + payload=b"\x89PNG\r\n\x1a\n" + b"png", + attachment_id=1, + ) + session = _SequenceSession([[attachment]]) + + def broken_processor(*, attachment): + raise RuntimeError("classification blew up") + + monkeypatch.setattr( + attachment_reparse_worker_module, + "process_reparse_pending_attachment", + broken_processor, + ) + worker = AttachmentReparseWorker() + await worker._sweep_attachments(session) + + assert session.commit_count == 0 + assert session.rollback_count == 1 + + +class _ExpiredAttachment: + """Stands in for an ORM instance ``AsyncSession.rollback()`` expired. + + Any attribute read raises, exactly like touching an expired async-mapped + instance outside an active await/greenlet context would in real + SQLAlchemy -- proving the sweep never touches this bulk-loaded object + for its *own* processing once it has already failed and been rolled + back once. + """ + + id = 1 + + def __getattr__(self, _name): + raise AttributeError( + "must not read attributes off the stale bulk-loaded instance" + ) + + +@pytest.mark.asyncio +async def test_sweep_never_processes_the_bulk_loaded_instance_directly(): + poisoned_first = _ExpiredAttachment() + fresh_first = _reparse_pending_attachment( + content_type="application/pdf", + payload=b"\x89PNG\r\n\x1a\n" + b"png", + attachment_id=1, + ) + second = _reparse_pending_attachment( + content_type="application/pdf", + payload=b"\x89PNG\r\n\x1a\n" + b"png", + attachment_id=2, + ) + # The bulk query "sees" the poisoned stand-in for id 1 (as a real + # AsyncSession would after some earlier rollback expired it), but `get` + # returns the real, healthy row -- proving the sweep re-fetches instead + # of processing the bulk-loaded object directly. + session = _SequenceSession( + [[poisoned_first, second]], by_id={1: fresh_first, 2: second} + ) + + worker = AttachmentReparseWorker() + await worker._sweep_attachments(session) + + assert fresh_first.parse_status == _QUARANTINED_STATUS + assert second.parse_status == _QUARANTINED_STATUS + assert session.commit_count == 2 + assert session.rollback_count == 0 + + +class _LiveReparsePendingSession: + """Mirrors ``AttachmentReparseWorker._reparse_pending_statement`` for + real, across many sweeps -- see + ``tests.test_newsdom_worker._LivePendingAttachmentSession`` (identical + fake, same reason: proving a multi-sweep scheduling claim needs a fake + that reflects the worker's own (cursor, retry_ids) state, not a + pre-scripted batch sequence). + """ + + def __init__(self, worker, attachments): + self._worker = worker + self._table = {attachment.id: attachment for attachment in attachments} + self.commit_count = 0 + self.rollback_count = 0 + + async def execute(self, _statement): + cursor = self._worker._attachment_cursor + retry_ids = self._worker._attachment_retry_ids + pending = [ + row + for row in self._table.values() + if row.parse_status == ATTACHMENT_REPARSE_PENDING_STATUS + and (cursor is None or row.id > cursor or row.id in retry_ids) + ] + pending.sort(key=lambda row: (0 if (cursor is None or row.id > cursor) else 1, row.id)) + return _RowsResult(pending[: self._worker.batch_limit]) + + async def get(self, _model, attachment_id): + return self._table.get(attachment_id) + + async def refresh(self, _attachment, *, attribute_names): + # No-op: the fake table already holds the live, fully-populated + # instances (see _SequenceSession.refresh for the sibling fake that + # records calls instead -- this one has no need to, since nothing + # here asserts on refresh() itself, only on the resulting sweep + # behavior across many sweeps). + del attribute_names + + async def commit(self): + self.commit_count += 1 + + async def rollback(self): + self.rollback_count += 1 + + +@pytest.mark.asyncio +async def test_sweep_does_not_starve_rows_behind_many_failing_rows(monkeypatch): + # Same starvation class fixed on the NewsDOM worker (see + # test_newsdom_worker.test_attachment_sweep_does_not_starve_rows_behind_many_stuck_rows): + # a systematic classification bug affecting a burst of simultaneous + # reparse-intent requests could put more than batch_limit consecutive + # rows into a permanent-failure state at once. + failing = [ + _reparse_pending_attachment( + content_type="application/pdf", + payload=b"\x89PNG\r\n\x1a\n" + b"png", + attachment_id=index, + ) + for index in range(1, 61) + ] + healthy = [ + _reparse_pending_attachment( + content_type="application/pdf", + payload=b"\x89PNG\r\n\x1a\n" + b"png", + attachment_id=index, + ) + for index in range(61, 121) + ] + all_attachments = failing + healthy + failing_ids = {attachment.id for attachment in failing} + + real_process = attachment_reparse_worker_module.process_reparse_pending_attachment + + def fail_the_first_burst(*, attachment): + if attachment.id in failing_ids: + raise RuntimeError("classification blew up") + return real_process(attachment=attachment) + + monkeypatch.setattr( + attachment_reparse_worker_module, + "process_reparse_pending_attachment", + fail_the_first_burst, + ) + worker = AttachmentReparseWorker(batch_limit=50) + session = _LiveReparsePendingSession(worker, all_attachments) + + for _ in range(10): + await worker._sweep_attachments(session) + if all(a.parse_status == _QUARANTINED_STATUS for a in healthy): + break + + assert all(a.parse_status == _QUARANTINED_STATUS for a in healthy) + assert all( + a.parse_status == ATTACHMENT_REPARSE_PENDING_STATUS for a in failing + ) + assert worker._attachment_retry_ids == failing_ids + + +@pytest.mark.asyncio +async def test_sweep_rediscovers_a_row_reverted_to_pending_behind_the_cursor(): + # POST .../reparse-intent can re-mark ANY existing attachment + # reparse_pending, including one whose id is already behind the forward + # cursor. Devin Review flagged that the cursor+retry-id design alone can + # never discover that: retry_ids only tracks rows this worker itself + # already saw and found unresolved -- it has no way to learn about a + # brand-new external transition on an old, already-resolved row. A + # periodic full rescan (every FULL_RESCAN_EVERY_N_SWEEPS sweeps) bounds + # how long such a row can stay invisible. + reverted = _reparse_pending_attachment( + content_type="application/pdf", + payload=b"\x89PNG\r\n\x1a\n" + b"png", + attachment_id=1, + ) + worker = AttachmentReparseWorker() + session = _LiveReparsePendingSession(worker, [reverted]) + + await worker._sweep_attachments(session) + assert reverted.parse_status == _QUARANTINED_STATUS + assert worker._attachment_cursor == 1 + + # Simulate an operator re-triggering reparse via POST .../reparse-intent + # on this now-old attachment -- its id is already behind the cursor. + reverted.parse_status = ATTACHMENT_REPARSE_PENDING_STATUS + + rediscovered_at = None + for sweep_number in range( + 2, attachment_reparse_worker_module.FULL_RESCAN_EVERY_N_SWEEPS + 1 + ): + await worker._sweep_attachments(session) + if reverted.parse_status != ATTACHMENT_REPARSE_PENDING_STATUS: + rediscovered_at = sweep_number + break + + assert rediscovered_at == attachment_reparse_worker_module.FULL_RESCAN_EVERY_N_SWEEPS + assert reverted.parse_status == _QUARANTINED_STATUS + + +@pytest.mark.asyncio +async def test_sweep_isolates_one_items_failure_from_the_next_items_refetch( + monkeypatch, +): + first = _reparse_pending_attachment( + content_type="application/pdf", + payload=b"\x89PNG\r\n\x1a\n" + b"png", + attachment_id=1, + ) + second = _reparse_pending_attachment( + content_type="application/pdf", + payload=b"\x89PNG\r\n\x1a\n" + b"png", + attachment_id=2, + ) + session = _SequenceSession([[first, second]]) + + real_process = attachment_reparse_worker_module.process_reparse_pending_attachment + + def process_and_fail_only_the_first(*, attachment): + if attachment.id == 1: + raise RuntimeError("classification blew up") + return real_process(attachment=attachment) + + monkeypatch.setattr( + attachment_reparse_worker_module, + "process_reparse_pending_attachment", + process_and_fail_only_the_first, + ) + worker = AttachmentReparseWorker() + await worker._sweep_attachments(session) + + assert second.parse_status == _QUARANTINED_STATUS + assert session.commit_count == 1 + assert session.rollback_count == 1 + + +@pytest.mark.asyncio +async def test_postgresql_lease_helpers_and_non_postgresql_fallback(monkeypatch): + postgres_connection = _LeaseConnection(scalar_result=1) + + assert ( + await attachment_reparse_worker_module._try_acquire_sweep_lease( + postgres_connection + ) + is True + ) + # AUTOCOMMIT before the lock-acquire statement: a plain (non-autocommit) + # connection would otherwise leave this connection's implicit + # transaction open and idle for the whole sweep -- a PostgreSQL + # idle_in_transaction_session_timeout could then kill this connection + # mid-sweep, silently dropping the lease. + assert postgres_connection.execution_options_calls == [ + {"isolation_level": "AUTOCOMMIT"} + ] + assert ( + postgres_connection.scalar_calls[0][1] + == attachment_reparse_worker_module._SWEEP_LOCK_PARAMS + ) + # Ordering, not just occurrence: AUTOCOMMIT must be set before the + # advisory-lock query runs, or the query could still open an implicit + # transaction under the connection's prior isolation level. + assert postgres_connection.ordered_calls[0][0] == "execution_options" + assert postgres_connection.ordered_calls[1][0] == "scalar" + await attachment_reparse_worker_module._release_sweep_lease(postgres_connection) + assert len(postgres_connection.scalar_calls) == 2 + + monkeypatch.setattr( + attachment_reparse_worker_module, "engine", _FakeEngine(dialect_name="sqlite") + ) + assert attachment_reparse_worker_module._engine_uses_postgresql() is False + + monkeypatch.setattr( + attachment_reparse_worker_module, + "engine", + _FakeEngine(dialect_name="postgresql"), + ) + assert attachment_reparse_worker_module._engine_uses_postgresql() is True + + +class _LeaseConnectionContext: + def __init__(self, connection): + self.connection = connection + + async def __aenter__(self): + return self.connection + + async def __aexit__(self, *_args): + return False + + +@pytest.mark.asyncio +async def test_worker_sweep_skips_locking_when_engine_is_not_postgresql(monkeypatch): + session = object() + calls = [] + worker = AttachmentReparseWorker() + + monkeypatch.setattr( + attachment_reparse_worker_module, "engine", _FakeEngine(dialect_name="sqlite") + ) + monkeypatch.setattr( + attachment_reparse_worker_module, + "AsyncSessionLocal", + lambda: _AsyncSessionContext(session), + ) + + async def sweep_attachments(actual_session): + calls.append(actual_session) + + monkeypatch.setattr(worker, "_sweep_attachments", sweep_attachments) + + await worker._sweep() + + assert calls == [session] + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + ("lease", "expected_sweeps", "expected_releases"), + [(False, 0, 0), (True, 1, 1)], +) +async def test_worker_sweep_honors_lease_outcome( + monkeypatch, lease, expected_sweeps, expected_releases +): + session = object() + lock_connection = object() + calls = [] + releases = [] + worker = AttachmentReparseWorker() + + monkeypatch.setattr( + attachment_reparse_worker_module, + "engine", + _FakeEngine(dialect_name="postgresql", connection=lock_connection), + ) + monkeypatch.setattr( + attachment_reparse_worker_module, + "AsyncSessionLocal", + lambda: _AsyncSessionContext(session), + ) + + async def acquire(actual_connection): + assert actual_connection is lock_connection + return lease + + async def release(actual_connection): + releases.append(actual_connection) + + async def sweep_attachments(actual_session): + calls.append(("attachments", actual_session)) + + monkeypatch.setattr( + attachment_reparse_worker_module, "_try_acquire_sweep_lease", acquire + ) + monkeypatch.setattr( + attachment_reparse_worker_module, "_release_sweep_lease", release + ) + monkeypatch.setattr(worker, "_sweep_attachments", sweep_attachments) + + await worker._sweep() + + assert len(calls) == expected_sweeps + assert len(releases) == expected_releases + + +@pytest.mark.asyncio +async def test_worker_start_stop_are_idempotent(monkeypatch): + worker = AttachmentReparseWorker() + entered = asyncio.Event() + blocker = asyncio.Event() + + async def blocked_loop(): + entered.set() + await blocker.wait() + + monkeypatch.setattr(worker, "_run_loop", blocked_loop) + await worker.start() + await entered.wait() + task = worker._task + await worker.start() + await worker.stop() + await worker.stop() + + assert task is not None + assert task.cancelled() + + worker._is_running = True + worker._task = None + await worker.stop() + + +@pytest.mark.asyncio +async def test_worker_loop_reports_errors_and_honors_cancellation(monkeypatch): + worker = AttachmentReparseWorker(interval_seconds=1) + worker._is_running = True + sleep_calls = 0 + + async def sleep_then_cancel(_seconds): + nonlocal sleep_calls + sleep_calls += 1 + if sleep_calls > 1: + raise asyncio.CancelledError + + async def failing_sweep(): + raise RuntimeError("sweep failed") + + monkeypatch.setattr(attachment_reparse_worker_module.asyncio, "sleep", sleep_then_cancel) + monkeypatch.setattr(worker, "_sweep", failing_sweep) + await worker._run_loop() + assert sleep_calls == 2 + + async def cancel_immediately(_seconds): + raise asyncio.CancelledError + + monkeypatch.setattr(attachment_reparse_worker_module.asyncio, "sleep", cancel_immediately) + await worker._run_loop() + + +@pytest.mark.asyncio +async def test_worker_loop_stops_when_sweep_is_cancelled(monkeypatch): + worker = AttachmentReparseWorker(interval_seconds=1) + worker._is_running = True + + async def no_sleep(_seconds): + return None + + async def cancelled_sweep(): + raise asyncio.CancelledError + + monkeypatch.setattr(attachment_reparse_worker_module.asyncio, "sleep", no_sleep) + monkeypatch.setattr(worker, "_sweep", cancelled_sweep) + await worker._run_loop() + + +@pytest.mark.asyncio +async def test_worker_loop_returns_after_a_normal_interval(monkeypatch): + worker = AttachmentReparseWorker(interval_seconds=1) + worker._is_running = True + sleep_calls = 0 + sweep_calls = 0 + + async def stop_after_interval(_seconds): + nonlocal sleep_calls + sleep_calls += 1 + if sleep_calls == 2: + worker._is_running = False + + async def successful_sweep(): + nonlocal sweep_calls + sweep_calls += 1 + + monkeypatch.setattr(attachment_reparse_worker_module.asyncio, "sleep", stop_after_interval) + monkeypatch.setattr(worker, "_sweep", successful_sweep) + await worker._run_loop() + + assert sleep_calls == 2 + assert sweep_calls == 1 + + +@pytest.mark.asyncio +async def test_worker_loop_skips_interval_when_sweep_stops_worker(monkeypatch): + worker = AttachmentReparseWorker(interval_seconds=1) + worker._is_running = True + sleep_calls = 0 + + async def record_sleep(_seconds): + nonlocal sleep_calls + sleep_calls += 1 + + async def stopping_sweep(): + worker._is_running = False + + monkeypatch.setattr(attachment_reparse_worker_module.asyncio, "sleep", record_sleep) + monkeypatch.setattr(worker, "_sweep", stopping_sweep) + await worker._run_loop() + + assert sleep_calls == 1 diff --git a/backend/tests/test_auth_real.py b/backend/tests/test_auth_real.py index 11450683e..e5535b4b6 100644 --- a/backend/tests/test_auth_real.py +++ b/backend/tests/test_auth_real.py @@ -254,6 +254,19 @@ def test_build_auth_context_rejects_invalid_token(): assert exc.value.status_code == 401 +def test_build_auth_context_accepts_independently_signed_workspace_membership(): + # Workspace ids are opaque membership identifiers signed by the verified + # session authority; they are not names derived from organization ids. + settings.AUTH_SESSION_HMAC_SECRET = SecretStr(TEST_SESSION_HMAC_SECRET) + token = _signed_session_token( + _valid_session_payload(org="org-acme", workspace="workspace-project-blue") + ) + + context = build_auth_context(authorization=f"Bearer {token}") + assert context.organization_id == "org-acme" + assert context.workspace_id == "workspace-project-blue" + + @pytest.mark.asyncio async def test_get_auth_context_rejects_missing_auth(): # It should raise HTTP 401 when auth is absent instead of defaulting. diff --git a/backend/tests/test_bootstrap_db.py b/backend/tests/test_bootstrap_db.py index 5af0540f0..357a0126e 100644 --- a/backend/tests/test_bootstrap_db.py +++ b/backend/tests/test_bootstrap_db.py @@ -6,7 +6,11 @@ from core.config import settings from db.models import Base -from scripts.bootstrap_db import schema_backfill_sql +from scripts.bootstrap_db import ( + LEGACY_EMAILS_INDEX, + execute_schema_backfill, + schema_backfill_sql, +) from db.models import ( AgentRunRecord, CalendarWritebackSource, @@ -30,8 +34,7 @@ def _get_schema_statements(monkeypatch): def _execute_schema_backfill(sync_conn): - for statement in schema_backfill_sql(): - sync_conn.execute(statement) + execute_schema_backfill(sync_conn) def test_schema_backfill_adds_email_columns(monkeypatch): @@ -126,8 +129,11 @@ def test_schema_backfill_adds_email_indexes(monkeypatch): for statement in statements ) assert any( - "create unique index if not exists uq_email_records_owner_message_id" - in statement + "create unique index if not exists uq_emails_workspace_message" in statement + for statement in statements + ) + assert any( + "user_id, organization_id, workspace_id, message_id" in statement for statement in statements ) assert any( @@ -204,13 +210,11 @@ def test_schema_backfill_adds_prompt_template_scope_columns_and_indexes(monkeypa for statement in statements ) assert any( - "alter table prompt_templates alter column prompt_uid set not null" - in statement + "alter table prompt_templates alter column prompt_uid set not null" in statement for statement in statements ) assert any( - "create unique index if not exists uq_prompt_templates_prompt_uid" - in statement + "create unique index if not exists uq_prompt_templates_prompt_uid" in statement for statement in statements ) assert any( @@ -298,8 +302,7 @@ def test_schema_backfill_creates_ai_hub_workflow_tables(monkeypatch): "ix_agent_run_records_scope_time" in statement for statement in statements ) assert any( - "ix_agent_run_records_workflow_uid" in statement - and "workflow_uid" in statement + "ix_agent_run_records_workflow_uid" in statement and "workflow_uid" in statement for statement in statements ) assert any( @@ -412,6 +415,132 @@ def test_schema_backfill_adds_project_folder_columns_and_indexes(monkeypatch): ) +def test_schema_backfill_adds_email_workspace_column_and_index(monkeypatch): + """A pre-existing database bootstrapped via create_all/backfill (not Alembic) + must gain email_records.workspace_id too, or every workspace-scoped email + query fails after deployment (see Alembic 0020_email_workspace_scope).""" + statements = _get_schema_statements(monkeypatch) + assert any( + "alter table email_records add column if not exists workspace_id" in statement + for statement in statements + ) + assert any( + "update email_records set workspace_id" in statement + and "'workspace-' || organization_id" in statement + for statement in statements + ) + assert any( + "alter table email_records alter column workspace_id set not null" in statement + for statement in statements + ) + assert any( + "create index if not exists ix_email_records_workspace_id" in statement + and "email_records (workspace_id)" in statement + for statement in statements + ) + + +def test_schema_backfill_replaces_owner_only_email_uniqueness_with_workspace_scope( + monkeypatch, +): + """uq_email_records_owner_message_id (user_id, organization_id, message_id) + predates workspace scoping and is stricter than Alembic 0020's replacement + identity: it silently forbids the same message_id from ever existing in two + different workspaces of the same owner, which Alembic 0020 now explicitly + allows via uq_emails_workspace_message. A bootstrap-provisioned database + must drop the owner-only shape (as either a plain index or a named + constraint, since historical bootstrap runs may have produced either) and + replace it with the same workspace-scoped identity, or it silently diverges + from the Alembic-managed schema it exists to mirror.""" + statements = _get_schema_statements(monkeypatch) + + workspace_not_null = next( + i + for i, statement in enumerate(statements) + if "alter table email_records alter column workspace_id set not null" + in statement + ) + drop_constraint_index = next( + i + for i, statement in enumerate(statements) + if "alter table email_records drop constraint if exists " + "uq_email_records_owner_message_id" in statement + ) + drop_index_index = next( + i + for i, statement in enumerate(statements) + if statement + == "drop index if exists uq_email_records_owner_message_id" + ) + new_create_index = next( + i + for i, statement in enumerate(statements) + if "create unique index if not exists " + "uq_emails_workspace_message" in statement + and "user_id, organization_id, workspace_id, message_id" in statement + ) + + # The owner-only index must still exist earlier (validation runs before + # workspace_id is guaranteed populated) but must be dropped and replaced + # only after workspace_id is backfilled and non-null. + assert workspace_not_null < drop_constraint_index + assert drop_index_index != new_create_index + + # uq_emails_owner_message_id is the DIFFERENT owner-only identity name + # Alembic's own ORM metadata (and 0020_email_workspace_scope.py) used + # before workspace scoping -- a database provisioned via + # Base.metadata.create_all() (0001_initial_control_plane.py) before the + # workspace-scoped model landed carries this name, not the bootstrap + # script's own uq_email_records_owner_message_id. Bootstrap must drop + # both legacy names (constraint and index forms) or such a database + # keeps the stricter 3-column identity forever. + drop_alembic_constraint_index = next( + i + for i, statement in enumerate(statements) + if "alter table email_records drop constraint if exists " + "uq_emails_owner_message_id" in statement + ) + drop_alembic_index_index = next( + i + for i, statement in enumerate(statements) + if statement == "drop index if exists uq_emails_owner_message_id" + ) + assert workspace_not_null < drop_alembic_constraint_index + assert drop_alembic_index_index != new_create_index + assert workspace_not_null < new_create_index + + +def test_schema_backfill_adds_attachment_uid_column_and_index(monkeypatch): + """A pre-existing database bootstrapped via create_all/backfill (not Alembic) + must gain email_attachments.attachment_uid too, or every opaque-id + attachment lookup fails after deployment (see Alembic 0019_attachment_uid).""" + statements = _get_schema_statements(monkeypatch) + assert any( + "alter table email_attachments add column if not exists attachment_uid" + in statement + for statement in statements + ) + assert any( + "update email_attachments set attachment_uid" in statement + and "attachment_" in statement + and "encode(sha256" in statement + and "bytea" in statement + and "hex" in statement + and "random()::text" in statement + and "clock_timestamp()::text" in statement + for statement in statements + ) + assert any( + "alter table email_attachments alter column attachment_uid set not null" + in statement + for statement in statements + ) + assert any( + "create unique index if not exists uq_email_attachments_uid" in statement + for statement in statements + ) + + def test_schema_backfill_adds_tenant_config_columns_and_indexes(monkeypatch): statements = _get_schema_statements(monkeypatch) assert any( @@ -461,7 +590,20 @@ def test_schema_backfill_uses_only_explicit_non_default_owner_ids(monkeypatch): and "where user_id is null and organization_id is null" in statement for statement in statements ) - assert sum("update email_records" in statement for statement in statements) == 1 + # email_records also gets a separate, unrelated workspace_id backfill + # (see test_schema_backfill_adds_email_workspace_column_and_index), so + # this counts only the owner-backfill statement specifically, not every + # "update email_records" statement. + assert ( + sum( + "update email_records" in statement + and "set user_id" in statement + and "organization_id = :organization_id" in statement + and "where user_id is null and organization_id is null" in statement + for statement in statements + ) + == 1 + ) assert any( "update llm_providers" in statement and "set user_id" in statement @@ -742,6 +884,77 @@ def test_schema_backfill_creates_connector_signal_events(): ) +@pytest.mark.asyncio +@pytest.mark.postgres +async def test_schema_backfill_creates_legacy_emails_index_when_table_exists(): + engine = create_async_engine(settings.DATABASE_URL) + try: + async with engine.begin() as conn: + await conn.execute( + text( + "CREATE TEMP TABLE emails (" + "user_id varchar, organization_id varchar, date timestamptz" + ") ON COMMIT DROP" + ) + ) + await conn.run_sync(execute_schema_backfill, [LEGACY_EMAILS_INDEX]) + result = await conn.execute( + text( + "SELECT indexname FROM pg_indexes " + "WHERE tablename = 'emails' " + "AND indexname = 'ix_emails_owner_date'" + ) + ) + assert result.scalar_one() == "ix_emails_owner_date" + except ( + ConnectionRefusedError, + OSError, + OperationalError, + asyncpg.CannotConnectNowError, + asyncpg.InvalidAuthorizationSpecificationError, + asyncpg.InvalidCatalogNameError, + asyncpg.InvalidPasswordError, + ): + await engine.dispose() + pytest.skip("PostgreSQL smoke path unavailable") + except Exception: + await engine.dispose() + raise + finally: + await engine.dispose() + + +@pytest.mark.asyncio +@pytest.mark.postgres +async def test_schema_backfill_skips_legacy_emails_index_when_table_absent(): + # A genuinely fresh database (0001_initial_control_plane.py's own + # Base.metadata.create_all()) never creates a table named "emails" -- + # only "email_records" is ORM-modeled. execute_schema_backfill's guard + # must skip LEGACY_EMAILS_INDEX in exactly this case rather than raising + # "relation \"emails\" does not exist" (Postgres's CREATE INDEX IF NOT + # EXISTS only guards the index name, not the target table's existence). + engine = create_async_engine(settings.DATABASE_URL) + try: + async with engine.begin() as conn: + await conn.run_sync(execute_schema_backfill, [LEGACY_EMAILS_INDEX]) + except ( + ConnectionRefusedError, + OSError, + OperationalError, + asyncpg.CannotConnectNowError, + asyncpg.InvalidAuthorizationSpecificationError, + asyncpg.InvalidCatalogNameError, + asyncpg.InvalidPasswordError, + ): + await engine.dispose() + pytest.skip("PostgreSQL smoke path unavailable") + except Exception: + await engine.dispose() + raise + finally: + await engine.dispose() + + @pytest.mark.asyncio @pytest.mark.postgres async def test_connector_signal_events_real_postgres_bootstrap_smoke(): @@ -769,18 +982,19 @@ async def test_connector_signal_events_real_postgres_bootstrap_smoke(): email_result = await conn.execute( text(""" INSERT INTO email_records ( - user_id, organization_id, message_id, sender, recipients, - subject, "date", body + user_id, organization_id, workspace_id, message_id, + sender, recipients, subject, "date", body, is_read ) VALUES ( - :user_id, :organization_id, :message_id, :sender, - :recipients, :subject, now(), :body + :user_id, :organization_id, :workspace_id, :message_id, + :sender, :recipients, :subject, now(), :body, false ) RETURNING id """), { "user_id": smoke_user_id, "organization_id": smoke_organization_id, + "workspace_id": f"workspace-{smoke_organization_id}", "message_id": "", "sender": "smoke@example.com", "recipients": "owner@example.com", diff --git a/backend/tests/test_calendar_conflict_judgment_api.py b/backend/tests/test_calendar_conflict_judgment_api.py new file mode 100644 index 000000000..55b0e21f7 --- /dev/null +++ b/backend/tests/test_calendar_conflict_judgment_api.py @@ -0,0 +1,540 @@ +"""API contracts for persisted calendar-conflict judgments and corrections.""" + +from __future__ import annotations + +import datetime +import json + +import httpx +import pytest +from fastapi.exceptions import RequestValidationError + +import api.calendar_conflicts as calendar_conflicts_api +from db.session import get_db +from main import app + + +def _client(*, user_id: str, organization_id: str | None = None) -> httpx.AsyncClient: + headers = {"X-User-Id": user_id} + if organization_id is not None: + headers["X-Organization-Id"] = organization_id + return httpx.AsyncClient( + transport=httpx.ASGITransport(app=app), + base_url="http://testserver", + headers=headers, + ) + + +class _FakeJudgment: + def __init__(self, **overrides: object) -> None: + observed_at = datetime.datetime(2026, 8, 30, 0, 0, tzinfo=datetime.timezone.utc) + defaults = { + "judgment_uid": "conflict_judgment_test", + "proposed_commitment_id": "proposal-1", + "source_thread_id": "thread-1", + "source_message_id": "", + "decision_code": "review_required", + "reason_code": "lower_priority_conflict_requires_explicit_resolution", + "conflicts_json": [ + { + "commitment_id": "existing-1", + "start_at": "2026-08-17T10:30:00+00:00", + "end_at": "2026-08-17T11:30:00+00:00", + "status": "tentative", + } + ], + "recommended_action": "Ask the proposer to confirm or reschedule.", + "policy_version": "status-weighted-v1", + "status_code": "proposed", + "created_at": observed_at, + "updated_at": observed_at, + } + defaults.update(overrides) + self.__dict__.update(defaults) + + +class _FakeCorrection: + def __init__(self, **overrides: object) -> None: + observed_at = datetime.datetime(2026, 8, 30, 0, 0, tzinfo=datetime.timezone.utc) + defaults = { + "correction_uid": "conflict_correction_test", + "correction_action": "override_decision", + "before_json": {"decision_code": "review_required", "status_code": "proposed"}, + "after_json": {"decision_code": "available", "status_code": "overridden"}, + "correction_rationale": "Confirmed with the proposer directly.", + "actor_user_id": "reviewer", + "created_at": observed_at, + } + defaults.update(overrides) + self.__dict__.update(defaults) + + +def _proposed_vs_tentative_payload() -> dict[str, object]: + return { + "proposed": { + "commitment_id": "proposal-1", + "start_at": "2026-08-17T10:00:00+09:00", + "end_at": "2026-08-17T11:00:00+09:00", + "status": "confirmed", + }, + "existing": [ + { + "commitment_id": "existing-1", + "start_at": "2026-08-17T10:30:00+09:00", + "end_at": "2026-08-17T11:30:00+09:00", + "status": "tentative", + } + ], + "source_thread_id": "thread-1", + "source_message_id": "", + } + + +class _DummySession: + def __init__(self) -> None: + self.committed = False + + async def commit(self) -> None: + self.committed = True + + +@pytest.mark.asyncio +async def test_create_judgment_persists_decision_and_returns_it( + dev_auth_dependency_overrides, + monkeypatch, +): + """Creating a judgment evaluates the policy once and hands back the persisted row.""" + captured = {} + dummy_session = _DummySession() + + async def fake_create_judgment(session, **kwargs): + captured["session"] = session + captured.update(kwargs) + return _FakeJudgment() + + async def override_get_db(): + yield dummy_session + + monkeypatch.setattr(calendar_conflicts_api, "create_judgment", fake_create_judgment) + app.dependency_overrides[get_db] = override_get_db + try: + async with _client(user_id="reviewer", organization_id="org-acme") as client: + response = await client.post( + "/api/calendar/conflicts/judgments", + json=_proposed_vs_tentative_payload(), + ) + finally: + app.dependency_overrides.pop(get_db, None) + + assert response.status_code == 200 + assert dummy_session.committed is True + assert captured["session"] is dummy_session + assert captured["user_id"] == "reviewer" + assert captured["organization_id"] == "org-acme" + assert captured["workspace_id"] == "workspace-org-acme" + assert captured["proposed_commitment_id"] == "proposal-1" + assert captured["source_thread_id"] == "thread-1" + assert captured["source_message_id"] == "" + assert captured["decision"].decision_code == "review_required" + + body = response.json() + assert body["judgment_uid"] == "conflict_judgment_test" + assert body["status_code"] == "proposed" + assert body["conflicts"][0]["commitment_id"] == "existing-1" + + +@pytest.mark.asyncio +async def test_create_judgment_rejects_malformed_request_without_persisting( + dev_auth_dependency_overrides, + monkeypatch, +): + """A policy-invalid request must never reach the persistence call.""" + calls = [] + + async def fake_create_judgment(session, **kwargs): + calls.append(kwargs) + return _FakeJudgment() + + async def override_get_db(): + yield _DummySession() + + monkeypatch.setattr(calendar_conflicts_api, "create_judgment", fake_create_judgment) + app.dependency_overrides[get_db] = override_get_db + try: + async with _client(user_id="reviewer") as client: + response = await client.post( + "/api/calendar/conflicts/judgments", + json={"existing": []}, + ) + finally: + app.dependency_overrides.pop(get_db, None) + + assert response.status_code == 422 + assert response.json()["error_code"] == "calendar_proposed_source_missing" + assert calls == [] + + +@pytest.mark.asyncio +async def test_list_judgments_scopes_by_source_thread_id( + dev_auth_dependency_overrides, + monkeypatch, +): + """Listing must forward the caller's own scope and the thread filter untouched.""" + captured = {} + + async def fake_list_judgments(session, **kwargs): + captured.update(kwargs) + return [_FakeJudgment()] + + async def override_get_db(): + yield _DummySession() + + monkeypatch.setattr(calendar_conflicts_api, "list_judgments", fake_list_judgments) + app.dependency_overrides[get_db] = override_get_db + try: + async with _client(user_id="reviewer", organization_id="org-acme") as client: + response = await client.get( + "/api/calendar/conflicts/judgments", + params={"source_thread_id": "thread-1"}, + ) + finally: + app.dependency_overrides.pop(get_db, None) + + assert response.status_code == 200 + assert captured["user_id"] == "reviewer" + assert captured["organization_id"] == "org-acme" + assert captured["workspace_id"] == "workspace-org-acme" + assert captured["source_thread_id"] == "thread-1" + assert response.json()[0]["judgment_uid"] == "conflict_judgment_test" + + +@pytest.mark.asyncio +async def test_get_judgment_returns_it_by_uid( + dev_auth_dependency_overrides, + monkeypatch, +): + """A single judgment must be fetchable by uid regardless of list ordering.""" + captured = {} + + async def fake_get_judgment(session, **kwargs): + captured.update(kwargs) + return _FakeJudgment() + + async def override_get_db(): + yield _DummySession() + + monkeypatch.setattr(calendar_conflicts_api, "get_judgment", fake_get_judgment) + app.dependency_overrides[get_db] = override_get_db + try: + async with _client(user_id="reviewer", organization_id="org-acme") as client: + response = await client.get( + "/api/calendar/conflicts/judgments/conflict_judgment_test" + ) + finally: + app.dependency_overrides.pop(get_db, None) + + assert response.status_code == 200 + assert captured["judgment_uid"] == "conflict_judgment_test" + assert captured["user_id"] == "reviewer" + assert captured["organization_id"] == "org-acme" + assert captured["workspace_id"] == "workspace-org-acme" + assert response.json()["judgment_uid"] == "conflict_judgment_test" + + +@pytest.mark.asyncio +async def test_get_judgment_404s_when_outside_caller_scope( + dev_auth_dependency_overrides, + monkeypatch, +): + """A judgment_uid outside the caller's own scope must 404, not leak another scope's data.""" + + async def fake_get_judgment(session, **kwargs): + raise calendar_conflicts_api.CalendarConflictJudgmentNotFoundError( + "Calendar conflict judgment is outside the requested scope" + ) + + async def override_get_db(): + yield _DummySession() + + monkeypatch.setattr(calendar_conflicts_api, "get_judgment", fake_get_judgment) + app.dependency_overrides[get_db] = override_get_db + try: + async with _client(user_id="reviewer") as client: + response = await client.get("/api/calendar/conflicts/judgments/not-mine") + finally: + app.dependency_overrides.pop(get_db, None) + + assert response.status_code == 404 + + +@pytest.mark.asyncio +async def test_correct_judgment_returns_audit_trail_and_commits( + dev_auth_dependency_overrides, + monkeypatch, +): + """A correction call commits and returns the full before/after audit trail.""" + captured = {} + dummy_session = _DummySession() + + async def fake_apply_correction(session, **kwargs): + captured["session"] = session + captured.update(kwargs) + return _FakeCorrection() + + async def override_get_db(): + yield dummy_session + + monkeypatch.setattr(calendar_conflicts_api, "apply_correction", fake_apply_correction) + app.dependency_overrides[get_db] = override_get_db + try: + async with _client(user_id="reviewer", organization_id="org-acme") as client: + response = await client.post( + "/api/calendar/conflicts/judgments/conflict_judgment_test/corrections", + json={ + "correction_action": "override_decision", + "decision_code": "available", + "status_code": "overridden", + "rationale": "Confirmed with the proposer directly.", + }, + ) + finally: + app.dependency_overrides.pop(get_db, None) + + assert response.status_code == 200 + assert dummy_session.committed is True + assert captured["judgment_uid"] == "conflict_judgment_test" + assert captured["actor_user_id"] == "reviewer" + assert captured["organization_id"] == "org-acme" + assert captured["workspace_id"] == "workspace-org-acme" + assert captured["decision_code"] == "available" + assert captured["status_code"] == "overridden" + + body = response.json() + assert body["correction_uid"] == "conflict_correction_test" + assert body["after_json"]["status_code"] == "overridden" + + +@pytest.mark.asyncio +async def test_correct_judgment_rejects_incoherent_status_and_decision( + dev_auth_dependency_overrides, + monkeypatch, +): + """Confirming a judgment while also changing its decision must fail closed at the request layer.""" + calls = [] + + async def fake_apply_correction(session, **kwargs): + calls.append(kwargs) + return _FakeCorrection() + + async def override_get_db(): + yield _DummySession() + + monkeypatch.setattr(calendar_conflicts_api, "apply_correction", fake_apply_correction) + app.dependency_overrides[get_db] = override_get_db + try: + async with _client(user_id="reviewer") as client: + response = await client.post( + "/api/calendar/conflicts/judgments/conflict_judgment_test/corrections", + json={ + "correction_action": "confirm_decision", + "decision_code": "available", + "status_code": "confirmed", + }, + ) + finally: + app.dependency_overrides.pop(get_db, None) + + assert response.status_code == 422 + assert response.json()["error_code"] == "calendar_correction_incoherent" + assert calls == [] + + +def test_request_validation_error_response_dispatches_by_type_not_wording(): + """The error_code mapper must key off errors()[i]["type"], never ["msg"]. + + Constructs a RequestValidationError whose message text does not contain + any of the phrases the mapper used to substring-match on -- proving the + dispatch is now driven entirely by the stable ``type`` a + PydanticCustomError carries, independent of how either error is worded. + """ + exc = RequestValidationError( + [ + { + "type": "calendar_correction_incoherent", + "msg": "this message says nothing about decision codes at all", + "loc": ("body",), + } + ] + ) + + response = calendar_conflicts_api._request_validation_error_response(exc) + + assert response.status_code == 422 + assert json.loads(response.body)["error_code"] == "calendar_correction_incoherent" + + +def test_request_validation_error_response_falls_back_for_unrecognized_types(): + exc = RequestValidationError( + [{"type": "value_error", "msg": "some other failure", "loc": ("body",)}] + ) + + response = calendar_conflicts_api._request_validation_error_response(exc) + + assert json.loads(response.body)["error_code"] == "calendar_request_invalid" + + +@pytest.mark.asyncio +async def test_correct_judgment_404s_when_outside_caller_scope( + dev_auth_dependency_overrides, + monkeypatch, +): + """A judgment_uid outside the caller's own scope must never leak as a 500 or a silent no-op.""" + + async def fake_apply_correction(session, **kwargs): + raise calendar_conflicts_api.CalendarConflictJudgmentNotFoundError( + "Calendar conflict judgment is outside the requested scope" + ) + + async def override_get_db(): + yield _DummySession() + + monkeypatch.setattr(calendar_conflicts_api, "apply_correction", fake_apply_correction) + app.dependency_overrides[get_db] = override_get_db + try: + async with _client(user_id="reviewer") as client: + response = await client.post( + "/api/calendar/conflicts/judgments/not-mine/corrections", + json={ + "correction_action": "override_decision", + "decision_code": "available", + "status_code": "overridden", + }, + ) + finally: + app.dependency_overrides.pop(get_db, None) + + assert response.status_code == 404 + + +@pytest.mark.postgres +@pytest.mark.asyncio +async def test_calendar_conflict_judgment_lifecycle_real_postgres_smoke(): + """Every test above drives a fully mocked session or _DummySession -- + apply_correction's with_for_update() row lock and the audit-snapshot + persistence it protects have never round-tripped through a real + PostgreSQL connection. Prove create_judgment -> apply_correction -> + list_judgments actually persists and locks against a live database.""" + from asyncpg.exceptions import InvalidAuthorizationSpecificationError + from asyncpg.exceptions import InvalidPasswordError + from sqlalchemy import delete, text + from sqlalchemy.exc import OperationalError + from sqlalchemy.ext.asyncio import async_sessionmaker, create_async_engine + + from core.config import settings + from db.models import Base, CalendarConflictCorrection, CalendarConflictJudgment + from services.calendar_conflict_judgment_service import ( + apply_correction, + create_judgment, + list_judgments, + ) + from services.calendar_conflict_policy import CalendarConflictDecision + + smoke_tables = [ + CalendarConflictJudgment.__table__, + CalendarConflictCorrection.__table__, + ] + engine = create_async_engine(settings.DATABASE_URL) + try: + async with engine.begin() as conn: + await conn.execute(text("SELECT 1")) + # Scoped to just these two tables (neither has a vector column) -- + # Base.metadata.create_all() would also try to create + # email_records, which needs the pgvector extension and is + # unrelated to what this smoke test exercises. + await conn.run_sync( + lambda sync_conn: Base.metadata.create_all( + sync_conn, tables=smoke_tables + ) + ) + except ( + InvalidAuthorizationSpecificationError, + InvalidPasswordError, + OperationalError, + OSError, + ) as exc: + await engine.dispose() + pytest.skip(f"PostgreSQL smoke database unavailable: {exc}") + + Session = async_sessionmaker(engine, expire_on_commit=False) + user_id = "calendar-judgment-smoke-user" + organization_id = "calendar-judgment-smoke-org" + workspace_id = "workspace-calendar-judgment-smoke" + + async def cleanup_seed_rows(): + async with Session() as session: + await session.execute( + delete(CalendarConflictCorrection).where( + CalendarConflictCorrection.user_id == user_id + ) + ) + await session.execute( + delete(CalendarConflictJudgment).where( + CalendarConflictJudgment.user_id == user_id + ) + ) + await session.commit() + + await cleanup_seed_rows() + try: + decision = CalendarConflictDecision( + decision_code="review_required", + reason_code="lower_priority_conflict_requires_explicit_resolution", + conflicts=(), + recommended_action="Ask the proposer to confirm or reschedule.", + ) + async with Session() as session: + judgment = await create_judgment( + session, + user_id=user_id, + organization_id=organization_id, + workspace_id=workspace_id, + proposed_commitment_id="proposal-smoke-1", + source_thread_id="thread-smoke-1", + source_message_id="", + decision=decision, + ) + await session.commit() + judgment_uid = judgment.judgment_uid + + async with Session() as session: + correction = await apply_correction( + session, + judgment_uid=judgment_uid, + user_id=user_id, + organization_id=organization_id, + workspace_id=workspace_id, + actor_user_id="smoke-reviewer", + correction_action="override_decision", + decision_code="available", + status_code="overridden", + rationale="Confirmed directly with the proposer.", + ) + await session.commit() + + async with Session() as session: + judgments = await list_judgments( + session, + user_id=user_id, + organization_id=organization_id, + workspace_id=workspace_id, + ) + finally: + await cleanup_seed_rows() + await engine.dispose() + + assert len(judgments) == 1 + assert judgments[0].judgment_uid == judgment_uid + assert judgments[0].decision_code == "available" + assert judgments[0].status_code == "overridden" + assert correction.before_json["decision_code"] == "review_required" + assert correction.after_json["decision_code"] == "available" diff --git a/backend/tests/test_calendar_conflict_judgment_service.py b/backend/tests/test_calendar_conflict_judgment_service.py new file mode 100644 index 000000000..e38f6f642 --- /dev/null +++ b/backend/tests/test_calendar_conflict_judgment_service.py @@ -0,0 +1,348 @@ +"""Unit tests for the pure/validation logic in the judgment persistence service.""" + +from __future__ import annotations + +import datetime + +import pytest + +from db.models import CalendarConflictJudgment +from services.calendar_conflict_judgment_service import ( + CORRECTED_DECISION_REASON_CODE, + UNSUPPORTED_DECISION_CODE_ERROR_CODE, + UNSUPPORTED_STATUS_CODE_ERROR_CODE, + _MAX_JUDGMENTS_PER_LIST, + CalendarConflictCorrectionIncoherentError, + CalendarConflictUnsupportedValueError, + _conflicts_to_json, + apply_correction, + get_judgment, + list_judgments, + validate_correction_coherence, +) +from services.calendar_conflict_policy import ( + CalendarCommitment, + CalendarConflictDecision, + default_recommended_action, +) + + +class _FakeScalars: + """The `.scalars()` half of a fake result, for list-style queries.""" + + def __init__(self, rows): + self._rows = rows + + def all(self): + return self._rows + + +class _FakeResult: + """A scalar-result stand-in that never touches a real database.""" + + def __init__(self, row): + self._row = row + + def scalar_one_or_none(self): + return self._row + + def scalars(self): + rows = [] if self._row is None else [self._row] + return _FakeScalars(rows) + + +class _RecordingSession: + """Captures every statement passed to execute() instead of running it.""" + + def __init__(self, row=None): + self._row = row + self.captured_statements: list[object] = [] + + async def execute(self, stmt): + self.captured_statements.append(stmt) + return _FakeResult(self._row) + + async def flush(self) -> None: + """No-op: this fake never talks to a real database.""" + + def add(self, obj) -> None: + """No-op: this fake never talks to a real database.""" + + +def _judgment(**overrides) -> CalendarConflictJudgment: + defaults = { + "user_id": "user-1", + "organization_id": None, + "workspace_id": "workspace-1", + "proposed_commitment_id": "proposal-1", + "source_thread_id": "thread-1", + "source_message_id": None, + "decision_code": "review_required", + "reason_code": "lower_priority_conflict_requires_explicit_resolution", + "recommended_action": "Ask the proposer to confirm or reschedule.", + "policy_version": "status-weighted-v1", + "conflicts_json": [], + "status_code": "proposed", + } + defaults.update(overrides) + return CalendarConflictJudgment(**defaults) + + +def _decision() -> CalendarConflictDecision: + conflict = CalendarCommitment( + commitment_id="existing-1", + start_at=datetime.datetime(2026, 8, 17, 10, 30, tzinfo=datetime.timezone.utc), + end_at=datetime.datetime(2026, 8, 17, 11, 30, tzinfo=datetime.timezone.utc), + status="tentative", + ) + return CalendarConflictDecision( + decision_code="review_required", + reason_code="lower_priority_conflict_requires_explicit_resolution", + conflicts=(conflict,), + recommended_action="Ask the proposer to confirm or reschedule.", + ) + + +def test_conflicts_to_json_serializes_iso_timestamps_and_status() -> None: + """The persisted evidence blob must be plain JSON, not raw datetimes.""" + payload = _conflicts_to_json(_decision()) + + assert payload == [ + { + "commitment_id": "existing-1", + "start_at": "2026-08-17T10:30:00+00:00", + "end_at": "2026-08-17T11:30:00+00:00", + "status": "tentative", + } + ] + + +def test_validate_correction_coherence_requires_decision_for_override() -> None: + """An override with no replacement decision is meaningless.""" + with pytest.raises(CalendarConflictCorrectionIncoherentError, match="requires a replacement"): + validate_correction_coherence(status_code="overridden", decision_code=None) + + +@pytest.mark.parametrize("status_code", ["proposed", "confirmed", "dismissed"]) +def test_validate_correction_coherence_forbids_decision_change_without_override( + status_code: str, +) -> None: + """Confirming or dismissing must never silently swap the decision.""" + with pytest.raises(CalendarConflictCorrectionIncoherentError, match="must not change"): + validate_correction_coherence(status_code=status_code, decision_code="available") + + +def test_validate_correction_coherence_accepts_matching_pairs() -> None: + """The two coherent shapes (override+decision, confirm/dismiss with none) pass.""" + validate_correction_coherence(status_code="overridden", decision_code="available") + validate_correction_coherence(status_code="confirmed", decision_code=None) + validate_correction_coherence(status_code="dismissed", decision_code=None) + + +@pytest.mark.asyncio +async def test_apply_correction_rejects_unsupported_status_code() -> None: + """A bogus status_code must fail closed before any database lookup runs.""" + with pytest.raises( + CalendarConflictUnsupportedValueError, match="status_code" + ) as exc_info: + await apply_correction( + object(), + judgment_uid="conflict_judgment_test", + user_id="user-1", + organization_id=None, + workspace_id="workspace-1", + actor_user_id="user-1", + correction_action="override", + decision_code=None, + status_code="not_a_real_status", + rationale=None, + ) + assert exc_info.value.error_code == UNSUPPORTED_STATUS_CODE_ERROR_CODE + + +@pytest.mark.asyncio +async def test_apply_correction_rejects_unsupported_decision_code() -> None: + """A bogus decision_code must fail closed before any database lookup runs.""" + with pytest.raises( + CalendarConflictUnsupportedValueError, match="decision_code" + ) as exc_info: + await apply_correction( + object(), + judgment_uid="conflict_judgment_test", + user_id="user-1", + organization_id=None, + workspace_id="workspace-1", + actor_user_id="user-1", + correction_action="override", + decision_code="not_a_real_decision", + status_code="confirmed", + rationale=None, + ) + assert exc_info.value.error_code == UNSUPPORTED_DECISION_CODE_ERROR_CODE + + +@pytest.mark.asyncio +async def test_apply_correction_rejects_incoherent_status_and_decision() -> None: + """A confirm/dismiss that also tries to change the decision must fail closed.""" + with pytest.raises(CalendarConflictCorrectionIncoherentError): + await apply_correction( + object(), + judgment_uid="conflict_judgment_test", + user_id="user-1", + organization_id=None, + workspace_id="workspace-1", + actor_user_id="user-1", + correction_action="confirm_decision", + decision_code="available", + status_code="confirmed", + rationale=None, + ) + + +@pytest.mark.asyncio +async def test_apply_correction_locks_the_judgment_row() -> None: + """Concurrent corrections must never read the same unlocked row.""" + session = _RecordingSession(row=_judgment()) + + await apply_correction( + session, + judgment_uid="conflict_judgment_test", + user_id="user-1", + organization_id=None, + workspace_id="workspace-1", + actor_user_id="reviewer", + correction_action="override_decision", + decision_code="available", + status_code="overridden", + rationale="Confirmed with the proposer directly.", + ) + + assert len(session.captured_statements) == 1 + compiled = str(session.captured_statements[0]) + assert "FOR UPDATE" in compiled + assert "workspace_id" in compiled + + +@pytest.mark.asyncio +async def test_apply_correction_overriding_decision_keeps_reason_and_action_coherent() -> None: + """A corrected decision must never be paired with the old decision's reason/action.""" + judgment = _judgment() + session = _RecordingSession(row=judgment) + + correction = await apply_correction( + session, + judgment_uid="conflict_judgment_test", + user_id="user-1", + organization_id=None, + workspace_id="workspace-1", + actor_user_id="reviewer", + correction_action="override_decision", + decision_code="available", + status_code="overridden", + rationale="Confirmed with the proposer directly.", + ) + + assert judgment.decision_code == "available" + assert judgment.reason_code == CORRECTED_DECISION_REASON_CODE + # recommended_action is restated from the policy's own canonical mapping, + # never from rationale -- rationale is an explanation, not scheduling advice. + assert judgment.recommended_action == default_recommended_action("available") + assert judgment.recommended_action != "Confirmed with the proposer directly." + # The rationale itself is preserved, just not as recommended_action. + assert correction.correction_rationale == "Confirmed with the proposer directly." + # The original decision's reason/action are never lost -- they are in before_json. + assert correction.before_json["reason_code"] == "lower_priority_conflict_requires_explicit_resolution" + assert correction.after_json["reason_code"] == CORRECTED_DECISION_REASON_CODE + assert correction.after_json["recommended_action"] == default_recommended_action("available") + + +@pytest.mark.asyncio +async def test_apply_correction_confirming_without_decision_change_keeps_original_reason() -> None: + """Confirming a judgment as-is must not fabricate a new reason/action.""" + judgment = _judgment() + session = _RecordingSession(row=judgment) + + await apply_correction( + session, + judgment_uid="conflict_judgment_test", + user_id="user-1", + organization_id=None, + workspace_id="workspace-1", + actor_user_id="reviewer", + correction_action="confirm_decision", + decision_code=None, + status_code="confirmed", + rationale=None, + ) + + assert judgment.decision_code == "review_required" + assert judgment.reason_code == "lower_priority_conflict_requires_explicit_resolution" + assert judgment.recommended_action == "Ask the proposer to confirm or reschedule." + assert judgment.status_code == "confirmed" + + +@pytest.mark.asyncio +async def test_apply_correction_override_repeating_current_decision_keeps_original_reason() -> None: + """An override that repeats the current decision must not erase the original reason.""" + judgment = _judgment() + session = _RecordingSession(row=judgment) + + await apply_correction( + session, + judgment_uid="conflict_judgment_test", + user_id="user-1", + organization_id=None, + workspace_id="workspace-1", + actor_user_id="reviewer", + correction_action="override_decision", + decision_code="review_required", # same as the judgment's current decision + status_code="overridden", + rationale=None, + ) + + assert judgment.decision_code == "review_required" + assert judgment.reason_code == "lower_priority_conflict_requires_explicit_resolution" + assert judgment.recommended_action == "Ask the proposer to confirm or reschedule." + assert judgment.status_code == "overridden" + + +@pytest.mark.asyncio +async def test_list_judgments_bounds_the_result_set_and_scopes_by_workspace() -> None: + """An unbounded list query could grow without limit for a long-lived account.""" + session = _RecordingSession() + + await list_judgments( + session, user_id="user-1", organization_id=None, workspace_id="workspace-1" + ) + + assert len(session.captured_statements) == 1 + compiled = str( + session.captured_statements[0].compile(compile_kwargs={"literal_binds": True}) + ) + assert f"LIMIT {_MAX_JUDGMENTS_PER_LIST}" in compiled + assert "workspace_id" in compiled + # created_at alone is not a unique key: two judgments created in the same + # instant could otherwise reorder across the 200-row boundary between + # calls. calendar_conflict_judgment_id (a monotonic primary key) breaks + # the tie deterministically. + assert "ORDER BY calendar_conflict_judgments.created_at DESC, " in compiled + assert "calendar_conflict_judgments.calendar_conflict_judgment_id DESC" in compiled + + +@pytest.mark.asyncio +async def test_get_judgment_reaches_a_row_outside_the_list_bound() -> None: + """A judgment past the 200-row list window must still be individually reachable.""" + judgment = _judgment() + session = _RecordingSession(row=judgment) + + fetched = await get_judgment( + session, + judgment_uid="conflict_judgment_test", + user_id="user-1", + organization_id=None, + workspace_id="workspace-1", + ) + + assert fetched is judgment + assert len(session.captured_statements) == 1 + assert "workspace_id" in str(session.captured_statements[0]) diff --git a/backend/tests/test_calendar_conflict_migration_offline.py b/backend/tests/test_calendar_conflict_migration_offline.py new file mode 100644 index 000000000..b8002a5c0 --- /dev/null +++ b/backend/tests/test_calendar_conflict_migration_offline.py @@ -0,0 +1,71 @@ +"""Offline-mode contract tests for calendar conflict Alembic revisions.""" + +from __future__ import annotations + +import importlib.util +from pathlib import Path +from types import SimpleNamespace + +import pytest + + +_VERSIONS_DIR = Path(__file__).resolve().parents[1] / "alembic" / "versions" +_MIGRATIONS = ( + "0018_calendar_conflict_judgments.py", + "0021_calendar_correction_rationale.py", +) + + +def _load_migration(filename: str): + """Load one revision module without requiring a Python-safe filename.""" + module_path = _VERSIONS_DIR / filename + spec = importlib.util.spec_from_file_location( + f"offline_contract_{module_path.stem}", module_path + ) + assert spec is not None and spec.loader is not None + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +@pytest.mark.parametrize("filename", _MIGRATIONS) +@pytest.mark.parametrize("direction", ("upgrade", "downgrade")) +def test_calendar_conflict_migrations_do_not_introspect_in_offline_mode( + monkeypatch: pytest.MonkeyPatch, + filename: str, + direction: str, +) -> None: + """Offline SQL generation must never require a live database connection.""" + migration = _load_migration(filename) + monkeypatch.setattr( + migration, + "context", + SimpleNamespace(is_offline_mode=lambda: True), + raising=False, + ) + monkeypatch.setattr( + migration.op, + "get_bind", + lambda: pytest.fail("offline migration requested a live Alembic bind"), + ) + monkeypatch.setattr( + migration.sa, + "inspect", + lambda *_args, **_kwargs: pytest.fail( + "offline migration attempted live schema introspection" + ), + ) + for operation_name in ( + "alter_column", + "create_index", + "create_table", + "drop_index", + "drop_table", + ): + monkeypatch.setattr( + migration.op, + operation_name, + lambda *_args, **_kwargs: None, + ) + + getattr(migration, direction)() diff --git a/backend/tests/test_calendar_conflict_persisted_evidence_regression.py b/backend/tests/test_calendar_conflict_persisted_evidence_regression.py new file mode 100644 index 000000000..249ba9e76 --- /dev/null +++ b/backend/tests/test_calendar_conflict_persisted_evidence_regression.py @@ -0,0 +1,48 @@ +"""Regression coverage for malformed persisted calendar-conflict evidence.""" + +import datetime +from types import SimpleNamespace + +import pytest + +from api.calendar_conflicts import ( + CalendarConflictStoredEvidenceError, + _judgment_response, +) + + +def _persisted_judgment_with(conflicts_json): + now = datetime.datetime.now(datetime.UTC) + return SimpleNamespace( + judgment_uid="judgment_1", + proposed_commitment_id="proposal_1", + source_thread_id=None, + source_message_id=None, + decision_code="blocked", + reason_code="overlap", + conflicts_json=conflicts_json, + recommended_action="Choose another time.", + policy_version="v1", + status_code="proposed", + created_at=now, + updated_at=now, + ) + + +def test_malformed_persisted_conflict_evidence_fails_with_stable_integrity_error() -> None: + """Corrupt stored evidence must fail closed without leaking KeyError/Pydantic internals.""" + judgment = _persisted_judgment_with( + [{"commitment_id": "existing_1", "status": "confirmed"}] + ) + + with pytest.raises(CalendarConflictStoredEvidenceError) as exc_info: + _judgment_response(judgment) + + assert exc_info.value.error_code == "calendar_conflict_stored_evidence_corrupt" + + +@pytest.mark.parametrize("conflicts_json", [None, {}, "not-a-list"]) +def test_non_list_persisted_conflict_evidence_fails_closed(conflicts_json) -> None: + """The persisted JSON boundary requires a list of complete typed evidence rows.""" + with pytest.raises(CalendarConflictStoredEvidenceError): + _judgment_response(_persisted_judgment_with(conflicts_json)) diff --git a/backend/tests/test_data_api.py b/backend/tests/test_data_api.py index cd0b7bf37..a96c2b19b 100644 --- a/backend/tests/test_data_api.py +++ b/backend/tests/test_data_api.py @@ -59,6 +59,7 @@ class MockAsyncSession: def __init__(self, results): self.results = results self.documents: list[Document] = [] + self.attachments: list[Attachment] = [] self.queries = [] self.execute_calls = 0 @@ -66,6 +67,53 @@ async def execute(self, query): self.queries.append(query) rendered_query = str(query) rendered_query_lower = rendered_query.lower() + if "where email_attachments.attachment_uid = " in rendered_query_lower: + compiled = query.compile() + params = compiled.params + attachment_uid = next( + ( + value + for key, value in params.items() + if key.startswith("attachment_uid") + ), + None, + ) + user_id = next( + (value for key, value in params.items() if key.startswith("user_id")), + None, + ) + organization_id = next( + ( + value + for key, value in params.items() + if key.startswith("organization_id") + ), + None, + ) + workspace_id = next( + ( + value + for key, value in params.items() + if key.startswith("workspace_id") + ), + None, + ) + rows = [ + attachment + for attachment in self.attachments + if attachment.attachment_uid == attachment_uid + and (user_id is None or attachment.email.user_id == user_id) + and ( + attachment.email.organization_id == organization_id + if organization_id is not None + else attachment.email.organization_id is None + ) + and ( + workspace_id is None + or attachment.email.workspace_id == workspace_id + ) + ] + return MockResult(rows[0] if rows else None) if ( "webdav_accounts.source_uid" in rendered_query_lower and "webdav_accounts.account_id" not in rendered_query_lower @@ -189,10 +237,12 @@ def _email( *, thread_id: str | None, subject: str = "Data source package", + workspace_id: str = "workspace-org-acme", ) -> Email: return Email( user_id="owner", organization_id="org-acme", + workspace_id=workspace_id, message_id=message_id, thread_id=thread_id, fingerprint=f"sha256:{message_id}", @@ -2458,9 +2508,7 @@ async def override_get_db(): def test_member_data_quality_queries_are_owner_scoped(mock_db): token = _signed_session_token( - _valid_session_payload( - sub="member", role="member", workspace="workspace-member" - ) + _valid_session_payload(sub="member", role="member") ) client, previous_secret, original_overrides = _with_signed_auth(mock_db, token) try: @@ -2475,6 +2523,7 @@ def test_member_data_quality_queries_are_owner_scoped(mock_db): assert "webdav_accounts.workspace_id = :workspace_id_1" in rendered_queries assert "project_folders.user_id = :user_id_1" in rendered_queries assert "email_records.user_id = :user_id_1" in rendered_queries + assert "email_records.workspace_id = :workspace_id_1" in rendered_queries assert "sender_relationships.user_id = :user_id_1" in rendered_queries @@ -2640,6 +2689,156 @@ def test_data_document_actions_are_workspace_scoped_and_intent_only(mock_db): assert "doc_rival" not in rival_response.text +def test_data_attachment_reparse_intent_transitions_quarantined_to_pending(mock_db): + owned_email = _email("", thread_id="thread-owned") + attachment = _attachment("invoice.pdf", "") + attachment.attachment_uid = "attachment_owned" + attachment.content_type = "application/pdf" + attachment.parse_content_type = "image/png" + attachment.parse_status = "content_type_mismatch_quarantined" + attachment.parse_error_code = "content_type_mismatch_quarantined" + attachment.email = owned_email + mock_db.attachments.append(attachment) + + token = _signed_session_token(_valid_session_payload(sub="owner")) + client, previous_secret, original_overrides = _with_signed_auth(mock_db, token) + try: + response = client.post( + "/api/data/attachments/attachment_owned/reparse-intent" + ) + finally: + client.close() + _restore_overrides(previous_secret, original_overrides) + + assert response.status_code == 200, response.text + data = response.json() + assert data["attachment_uid"] == "attachment_owned" + assert data["parse_status"] == "reparse_pending" + assert data["parse_error_code"] is None + assert data["provider_write_executed"] is False + assert data["audit_event"] == "data.attachment.reparse_intent" + assert attachment.parse_status == "reparse_pending" + assert attachment.parse_error_code is None + + +def test_data_attachment_reparse_intent_locks_the_attachment_row(mock_db): + # Concurrent reparse-intent requests (or a request racing the reparse + # worker) must never read-then-blindly-overwrite the same unlocked row: + # a stale request holding an earlier "quarantined" read could otherwise + # clobber a result the worker already landed. Locking the row for the + # duration of this transaction serializes that race, exactly like + # calendar_conflict_judgment_service.apply_correction's FOR UPDATE. + owned_email = _email("", thread_id="thread-owned") + attachment = _attachment("invoice.pdf", "") + attachment.attachment_uid = "attachment_owned_lock" + attachment.content_type = "application/pdf" + attachment.parse_content_type = "image/png" + attachment.parse_status = "content_type_mismatch_quarantined" + attachment.parse_error_code = "content_type_mismatch_quarantined" + attachment.email = owned_email + mock_db.attachments.append(attachment) + + token = _signed_session_token(_valid_session_payload(sub="owner")) + client, previous_secret, original_overrides = _with_signed_auth(mock_db, token) + try: + response = client.post( + "/api/data/attachments/attachment_owned_lock/reparse-intent" + ) + finally: + client.close() + _restore_overrides(previous_secret, original_overrides) + + assert response.status_code == 200, response.text + attachment_query = next( + query + for query in mock_db.queries + if "email_attachments.attachment_uid = " in str(query).lower() + ) + assert "FOR UPDATE" in str(attachment_query) + + +def test_data_attachment_reparse_intent_rejects_non_quarantined_status(mock_db): + owned_email = _email("", thread_id="thread-owned") + attachment = _attachment("notes.txt", "already parsed content") + attachment.attachment_uid = "attachment_parsed" + attachment.parse_status = "parsed" + attachment.email = owned_email + mock_db.attachments.append(attachment) + + token = _signed_session_token(_valid_session_payload(sub="owner")) + client, previous_secret, original_overrides = _with_signed_auth(mock_db, token) + try: + response = client.post( + "/api/data/attachments/attachment_parsed/reparse-intent" + ) + finally: + client.close() + _restore_overrides(previous_secret, original_overrides) + + assert response.status_code == 422, response.text + assert attachment.parse_status == "parsed" + + +def test_data_attachment_reparse_intent_is_scoped_to_caller(mock_db): + rival_email = _email( + "", thread_id="thread-rival" + ) + rival_email.user_id = "rival-owner" + rival_attachment = _attachment("payload.pdf", "") + rival_attachment.attachment_uid = "attachment_rival" + rival_attachment.parse_status = "content_type_mismatch_quarantined" + rival_attachment.email = rival_email + mock_db.attachments.append(rival_attachment) + + token = _signed_session_token(_valid_session_payload(sub="owner")) + client, previous_secret, original_overrides = _with_signed_auth(mock_db, token) + try: + response = client.post( + "/api/data/attachments/attachment_rival/reparse-intent" + ) + finally: + client.close() + _restore_overrides(previous_secret, original_overrides) + + assert response.status_code == 404 + assert "attachment_rival" not in response.text + assert rival_attachment.parse_status == "content_type_mismatch_quarantined" + + +def test_data_attachment_reparse_intent_is_scoped_to_workspace(mock_db): + # Same user_id and organization_id as the caller, but a DIFFERENT + # workspace_id -- the IDOR class this same PR's new reparse-intent route + # would otherwise open: Email carried no workspace_id at all before this + # fix, so _email_scope_filter could only ever check user_id/organization_id. + other_workspace_email = _email( + "", + thread_id="thread-other-workspace", + workspace_id="workspace-rival", + ) + other_workspace_attachment = _attachment("payload.pdf", "") + other_workspace_attachment.attachment_uid = "attachment_other_workspace" + other_workspace_attachment.parse_status = "content_type_mismatch_quarantined" + other_workspace_attachment.email = other_workspace_email + mock_db.attachments.append(other_workspace_attachment) + + token = _signed_session_token(_valid_session_payload(sub="owner")) + client, previous_secret, original_overrides = _with_signed_auth(mock_db, token) + try: + response = client.post( + "/api/data/attachments/attachment_other_workspace/reparse-intent" + ) + finally: + client.close() + _restore_overrides(previous_secret, original_overrides) + + assert response.status_code == 404 + assert "attachment_other_workspace" not in response.text + assert ( + other_workspace_attachment.parse_status + == "content_type_mismatch_quarantined" + ) + + def test_data_document_webdav_materialization_executes_source_backed_write( mock_db, monkeypatch, @@ -2950,12 +3149,14 @@ async def _seed_smoke_test_data(conn, ids: dict): text( """ INSERT INTO email_records ( - user_id, organization_id, message_id, thread_id, - fingerprint, sender, recipients, subject, "date", body + user_id, organization_id, workspace_id, message_id, thread_id, + fingerprint, sender, recipients, subject, "date", body, + is_read ) VALUES ( - :user_id, :organization_id, :message_id, :thread_id, - :fingerprint, :sender, :recipients, :subject, now(), :body + :user_id, :organization_id, :workspace_id, :message_id, + :thread_id, :fingerprint, :sender, :recipients, :subject, + now(), :body, true ) RETURNING id """ @@ -2963,6 +3164,7 @@ async def _seed_smoke_test_data(conn, ids: dict): { "user_id": ids["user_id"], "organization_id": ids["organization_id"], + "workspace_id": ids["workspace_id"], "message_id": first_message_id, "thread_id": "thread-data-smoke", "fingerprint": "sha256:data-smoke", @@ -2976,12 +3178,12 @@ async def _seed_smoke_test_data(conn, ids: dict): text( """ INSERT INTO email_records ( - user_id, organization_id, message_id, sender, recipients, - subject, "date", body + user_id, organization_id, workspace_id, message_id, sender, + recipients, subject, "date", body, is_read ) VALUES ( - :user_id, :organization_id, :message_id, :sender, - :recipients, :subject, now(), :body + :user_id, :organization_id, :workspace_id, :message_id, + :sender, :recipients, :subject, now(), :body, true ) RETURNING id """ @@ -2989,6 +3191,7 @@ async def _seed_smoke_test_data(conn, ids: dict): { "user_id": ids["user_id"], "organization_id": ids["organization_id"], + "workspace_id": ids["workspace_id"], "message_id": second_message_id, "sender": "partner@example.com", "recipients": "owner@example.com", @@ -3000,12 +3203,14 @@ async def _seed_smoke_test_data(conn, ids: dict): text( """ INSERT INTO email_records ( - user_id, organization_id, message_id, thread_id, - fingerprint, sender, recipients, subject, "date", body + user_id, organization_id, workspace_id, message_id, thread_id, + fingerprint, sender, recipients, subject, "date", body, + is_read ) VALUES ( - :user_id, :organization_id, :message_id, :thread_id, - :fingerprint, :sender, :recipients, :subject, now(), :body + :user_id, :organization_id, :workspace_id, :message_id, + :thread_id, :fingerprint, :sender, :recipients, :subject, + now(), :body, true ) RETURNING id """ @@ -3013,6 +3218,7 @@ async def _seed_smoke_test_data(conn, ids: dict): { "user_id": ids["rival_user_id"], "organization_id": ids["rival_organization_id"], + "workspace_id": ids["rival_workspace_id"], "message_id": rival_message_id, "thread_id": "thread-rival", "fingerprint": "sha256:rival", @@ -3189,32 +3395,35 @@ async def _seed_smoke_test_data(conn, ids: dict): text( """ INSERT INTO email_attachments ( - email_id, filename, content, + attachment_uid, email_id, filename, content, content_type, parse_status, parse_content_type, parser_key, parse_error_code ) VALUES ( - :first_email_id, 'ready.txt', 'ready attachment', - 'text/plain', 'parsed', 'text/plain', + :first_attachment_uid, :first_email_id, 'ready.txt', + 'ready attachment', 'text/plain', 'parsed', 'text/plain', 'plain_text', NULL ), ( - :second_email_id, 'blank.txt', '', + :second_attachment_uid, :second_email_id, 'blank.txt', '', 'application/pdf', 'unsupported_content_type', 'application/pdf', 'plain_text', 'unsupported_content_type' ), ( - :rival_email_id, 'rival.txt', 'rival attachment', - 'text/plain', 'parsed', 'text/plain', + :rival_attachment_uid, :rival_email_id, 'rival.txt', + 'rival attachment', 'text/plain', 'parsed', 'text/plain', 'plain_text', NULL ) """ ), { + "first_attachment_uid": f"attachment_{uuid.uuid4().hex}", "first_email_id": first_email_id, + "second_attachment_uid": f"attachment_{uuid.uuid4().hex}", "second_email_id": second_email_id, + "rival_attachment_uid": f"attachment_{uuid.uuid4().hex}", "rival_email_id": rival_email_id, }, ) diff --git a/backend/tests/test_email_import_service.py b/backend/tests/test_email_import_service.py index 51d2a2633..474f1f64b 100644 --- a/backend/tests/test_email_import_service.py +++ b/backend/tests/test_email_import_service.py @@ -1,10 +1,11 @@ import logging import datetime from pathlib import Path -from unittest.mock import AsyncMock, patch +from unittest.mock import AsyncMock, MagicMock, patch import pytest from sqlalchemy.ext.asyncio import AsyncSession +from sqlalchemy.sql.schema import UniqueConstraint import services.email_import_service as email_import_module from services.exceptions import EmailParseError, EmbeddingGenerationError @@ -16,6 +17,42 @@ MAX_EMBEDDING_CHUNKS_PER_WINDOW, _generate_import_embeddings, ) +from db.models import Email + + +def test_email_identity_includes_workspace_scope(): + constraint = next( + item + for item in Email.__table__.constraints + if isinstance(item, UniqueConstraint) + and item.name == "uq_emails_workspace_message" + ) + assert tuple(column.name for column in constraint.columns) == ( + "user_id", + "organization_id", + "workspace_id", + "message_id", + ) + + +@pytest.mark.asyncio +async def test_duplicate_lookup_is_scoped_to_target_workspace(): + session = AsyncMock(spec=AsyncSession) + result = MagicMock() + result.scalar_one_or_none.return_value = None + session.execute.return_value = result + + await email_import_module._find_existing_email( + session, + user_id="user-1", + organization_id="org-1", + workspace_id="workspace-blue", + message_id="message@example.com", + fingerprint="fingerprint-1", + ) + + statement = session.execute.await_args.args[0] + assert "email_records.workspace_id" in str(statement) def test_import_transport_ceiling_accepts_sources_over_20_mib(): @@ -792,3 +829,49 @@ async def test_generate_import_embeddings_recovers_valid_items_after_batch_failu assert embeddings[2] == [0.75] * (EMBEDDING_DIMENSION // 2) + [0.0] * ( EMBEDDING_DIMENSION // 2 ) + + +@pytest.mark.asyncio +async def test_persist_project_graph_projection_uses_the_resolved_workspace_id(): + # _persist_project_graph_projection recomputed its own workspace_id + # (f"workspace-{organization_id}") instead of taking the caller's already + # -resolved workspace_id -- so a non-default import workspace put the + # Email row in one workspace and its derived project-graph objects in a + # different one. + class _FakeExtraction: + objects = ["placeholder-object"] + + captured = {} + + class _FakeSession: + async def commit(self): + pass + + async def rollback(self): + pass + + async def fake_extract(*args, **kwargs): + return _FakeExtraction() + + async def fake_persist(session, *, extraction, user_id, organization_id, workspace_id): + captured["workspace_id"] = workspace_id + + with ( + patch.object( + email_import_module, + "_extract_project_semantics_for_import", + fake_extract, + ), + patch.object( + email_import_module, "persist_project_graph_projection", fake_persist + ), + ): + await email_import_module._persist_project_graph_projection( + _FakeSession(), + ["segment"], + user_id="user-1", + organization_id="org-1", + workspace_id="workspace-custom", + ) + + assert captured["workspace_id"] == "workspace-custom" diff --git a/backend/tests/test_emails_api.py b/backend/tests/test_emails_api.py index 7bffa6ff7..032330791 100644 --- a/backend/tests/test_emails_api.py +++ b/backend/tests/test_emails_api.py @@ -163,9 +163,7 @@ def scalar_one_or_none(self): # Emulate the SQL window query: newest head row per thread, # ordered by date desc, LIMIT applied to heads (not raw rows). heads: dict[str, object] = {} - for item in sorted( - self.items, key=lambda email: email.date, reverse=True - ): + for item in sorted(self.items, key=lambda email: email.date, reverse=True): key = item.thread_id or item.message_id heads.setdefault(key, item) rows = list(heads.values()) @@ -249,6 +247,21 @@ def get_bind(self): return _PostgresBind() +class ScalarQueryCapturingImportSession(PostgresImportRecordingSession): + def __init__( + self, + items, + tenant_config=_DEFAULT_TENANT_CONFIG, + llm_providers=None, + ): + super().__init__(items, tenant_config=tenant_config, llm_providers=llm_providers) + self.scalar_queries = [] + + async def scalar(self, query): + self.scalar_queries.append(query) + return await super().scalar(query) + + def compiled_query_text(query) -> str: return str(query).lower() @@ -953,7 +966,9 @@ async def test_import_email_files_rejects_invalid_canonical_filename( ): response = await client.post( "/api/emails/import-files", - files=[("files", (upload_filename, b"not accepted", "application/octet-stream"))], + files=[ + ("files", (upload_filename, b"not accepted", "application/octet-stream")) + ], headers={"X-Organization-Id": "org-acme"}, ) @@ -1399,6 +1414,65 @@ async def test_import_email_files_rejects_oversized_archive_before_partial_commi assert session.commit_count == 0 +@pytest.mark.asyncio +async def test_owner_import_quota_count_is_not_scoped_to_a_single_workspace( + client: AsyncClient, monkeypatch +): + # MAX_IMPORT_EMAILS_PER_OWNER and the advisory quota lock are both scoped + # to (user_id, organization_id) only -- an owner-wide allowance, not a + # per-workspace one. If the count query underneath it additionally filters + # by workspace_id, an owner importing through N distinct workspaces gets + # N times the intended allowance instead of one shared 1000-email cap. + from db.session import get_db + + existing_email = Email( + id=91, + user_id="testuser", + organization_id="org-acme", + workspace_id="workspace-other", + message_id="", + thread_id="existing-thread-other", + sender="partner@example.com", + recipients="user@example.com", + subject="Existing in another workspace", + date=datetime.datetime(2026, 6, 11, 10, 0, tzinfo=datetime.timezone.utc), + body="Existing body", + embedding=[0.0] * 1536, + ) + session = ScalarQueryCapturingImportSession([existing_email]) + previous_db_override = app.dependency_overrides.get(get_db) + app.dependency_overrides[get_db] = lambda: session + try: + await client.post( + "/api/emails/import-files", + files=[ + ( + "files", + ( + "quota-cross-workspace.eml", + _sample_eml_bytes(message_id=""), + "message/rfc822", + ), + ) + ], + headers={"X-Organization-Id": "org-acme"}, + ) + finally: + if previous_db_override is None: + app.dependency_overrides.pop(get_db, None) + else: + app.dependency_overrides[get_db] = previous_db_override + + quota_queries = [ + query + for query in session.scalar_queries + if "count(" in compiled_query_text(query) and "email_records" in compiled_query_text(query) + ] + assert quota_queries + assert_query_is_owner_scoped(quota_queries[-1]) + assert "workspace_id" not in compiled_query_text(quota_queries[-1]) + + @pytest.mark.asyncio async def test_unique_email_thread_intent_query_is_scoped_to_current_user( client: AsyncClient, sample_email: Email @@ -1483,6 +1557,7 @@ async def cleanup_seed_rows(): Email( user_id=user_id, organization_id=organization_id, + workspace_id=f"workspace-{organization_id}", message_id="waiting-smoke-msg", thread_id="", sender="reply-smoke@example.com", @@ -1494,6 +1569,7 @@ async def cleanup_seed_rows(): Email( user_id=user_id, organization_id=organization_id, + workspace_id=f"workspace-{organization_id}", message_id="note-smoke-msg", thread_id="note-smoke-thread", sender="reply-smoke@example.com", @@ -1505,6 +1581,7 @@ async def cleanup_seed_rows(): Email( user_id=user_id, organization_id=organization_id, + workspace_id=f"workspace-{organization_id}", message_id="answered-smoke-msg", thread_id="", sender="reply-smoke@example.com", @@ -1516,6 +1593,7 @@ async def cleanup_seed_rows(): Email( user_id=user_id, organization_id=organization_id, + workspace_id=f"workspace-{organization_id}", message_id="answer-smoke-msg", thread_id="answered-smoke-thread", sender="target@example.com", @@ -2095,9 +2173,11 @@ def test_email_owner_filters(): group_ids=(), workspace_id="ws-789", ) - filters1 = Email.owner_filters(ctx1.user_id, ctx1.organization_id) + filters1 = Email.owner_filters( + ctx1.user_id, ctx1.organization_id, ctx1.workspace_id + ) - assert len(filters1) == 2 + assert len(filters1) == 3 assert ( str(filters1[0].compile(compile_kwargs={"literal_binds": True})) == "email_records.user_id = 'user-123'" @@ -2106,6 +2186,10 @@ def test_email_owner_filters(): str(filters1[1].compile(compile_kwargs={"literal_binds": True})) == "email_records.organization_id = 'org-456'" ) + assert ( + str(filters1[2].compile(compile_kwargs={"literal_binds": True})) + == "email_records.workspace_id = 'ws-789'" + ) # Test with None organization_id ctx2 = AuthContext( @@ -2115,9 +2199,11 @@ def test_email_owner_filters(): group_ids=(), workspace_id="ws-789", ) - filters2 = Email.owner_filters(ctx2.user_id, ctx2.organization_id) + filters2 = Email.owner_filters( + ctx2.user_id, ctx2.organization_id, ctx2.workspace_id + ) - assert len(filters2) == 2 + assert len(filters2) == 3 assert ( str(filters2[0].compile(compile_kwargs={"literal_binds": True})) == "email_records.user_id = 'user-123'" @@ -2127,6 +2213,7 @@ def test_email_owner_filters(): == "email_records.organization_id IS NULL" ) + def test_find_matches_for_candidates_perf_optim_handles_missing_lookups(): """ Test to guarantee 100% coverage on the bolt performance optimization in @@ -2143,7 +2230,7 @@ def test_find_matches_for_candidates_perf_optim_handles_missing_lookups(): sender="test@test.com", recipients="test2@test.com", subject="Subject", - body="Body" + body="Body", ) candidates = [candidate] diff --git a/backend/tests/test_imap_worker.py b/backend/tests/test_imap_worker.py index d2798719a..7f1a6e2a9 100644 --- a/backend/tests/test_imap_worker.py +++ b/backend/tests/test_imap_worker.py @@ -64,6 +64,7 @@ async def test_imap_worker_imports_fetched_rfc822_messages(monkeypatch): imap_username="imap-user@example.com", imap_password="imap-secret", ) + config.workspace_id = "workspace-imap" raw_message = ( b"Message-ID: \r\n" b"From: Sender \r\n" @@ -125,6 +126,7 @@ async def test_imap_worker_imports_fetched_rfc822_messages(monkeypatch): # No \Seen in the FLAGS envelope above -> imported as unread. assert kwargs["is_read"] is False assert args[3] == "org-imap" + assert args[4] == "workspace-imap" assert kwargs["owner_addresses"] == ["imap-user@example.com"] session.commit.assert_awaited_once() @@ -164,6 +166,53 @@ async def test_imap_worker_requires_credentials_without_sensitive_log_names( assert "imap-secret" not in caplog.text +@pytest.mark.asyncio +async def test_resolve_unambiguous_workspace_id_returns_the_sole_workspace(): + from services.imap_worker import resolve_unambiguous_workspace_id + + class FakeSession: + async def scalars(self, _statement): + class _Result: + def __iter__(self): + return iter(["workspace-only"]) + + return _Result() + + workspace_id = await resolve_unambiguous_workspace_id( + FakeSession(), "user-1", "org-1" + ) + assert workspace_id == "workspace-only" + + +@pytest.mark.asyncio +async def test_resolve_unambiguous_workspace_id_fails_closed_when_absent_or_ambiguous(): + from services.imap_worker import resolve_unambiguous_workspace_id + + class FakeSession: + def __init__(self, rows): + self._rows = rows + + async def scalars(self, _statement): + rows = self._rows + + class _Result: + def __iter__(self): + return iter(rows) + + return _Result() + + assert ( + await resolve_unambiguous_workspace_id(FakeSession([]), "user-1", "org-1") + is None + ) + assert ( + await resolve_unambiguous_workspace_id( + FakeSession(["workspace-a", "workspace-b"]), "user-1", "org-1" + ) + is None + ) + + def test_flags_indicate_seen_parses_seen_flag(): from services.imap_worker import flags_indicate_seen @@ -173,7 +222,7 @@ def test_flags_indicate_seen_parses_seen_flag(): no_flags = ("OK", [(b"1 (RFC822 {%d}" % len(raw), raw)]) assert flags_indicate_seen(seen[1]) is True - assert flags_indicate_seen(unseen[1]) is False # other flags, but not \Seen + assert flags_indicate_seen(unseen[1]) is False # other flags, but not \Seen assert flags_indicate_seen(no_flags[1]) is False # no FLAGS section -> unread assert flags_indicate_seen([]) is False assert flags_indicate_seen(None) is False diff --git a/backend/tests/test_import_fixtures.py b/backend/tests/test_import_fixtures.py index bad57ec13..344e8e1ea 100644 --- a/backend/tests/test_import_fixtures.py +++ b/backend/tests/test_import_fixtures.py @@ -1,8 +1,10 @@ import pytest import datetime from unittest.mock import patch, AsyncMock +from sqlalchemy.dialects import postgresql from scripts.import_fixtures import process_zip_file import import_fixtures +import scripts.import_fixtures as scripts_import_fixtures @pytest.mark.asyncio @@ -15,6 +17,115 @@ async def test_process_zip_file(): await process_zip_file("dummy.zip", AsyncMock()) +@pytest.mark.asyncio +async def test_process_zip_file_batch_insert_includes_workspace_id(): + # Email.workspace_id is NOT NULL (0020_email_workspace_scope); this batch + # insert built its own values dict independently of the root importer's + # email_obj = Email(...) construction (already fixed) and was missed -- + # any nonempty archive would fail at commit without it. + captured_batch_values = [] + + class _RecordingSession: + async def scalar(self, *args, **kwargs): + return None + + async def execute(self, statement, batch_values=None): + if batch_values is not None: + captured_batch_values.extend(batch_values) + + async def commit(self): + pass + + email_data = { + "message_id": "msg-1", + "sender": "sender@example.com", + "reply_to": None, + "recipients": ["recipient@example.com"], + "subject": "Subject", + "in_reply_to": None, + "references": None, + "date": datetime.datetime(2026, 1, 1, tzinfo=datetime.timezone.utc), + "body": "body text", + } + + with ( + patch("scripts.import_fixtures.extract_backup_async") as mock_extract, + patch("scripts.import_fixtures.parse_eml", return_value=email_data), + patch("scripts.import_fixtures.chunk_text", return_value=[]), + patch( + "scripts.import_fixtures.assign_thread_id", + new=AsyncMock(return_value="thread-1"), + ), + ): + mock_extract.return_value = ["fixture.eml"] + await process_zip_file("dummy.zip", _RecordingSession()) + + assert len(captured_batch_values) == 1 + batch_row = captured_batch_values[0] + assert "workspace_id" in batch_row + assert batch_row["workspace_id"] == ( + f"workspace-{scripts_import_fixtures.IMPORT_ORGANIZATION_ID}" + if scripts_import_fixtures.IMPORT_ORGANIZATION_ID + else f"workspace-{scripts_import_fixtures.IMPORT_USER_ID}" + ) + + +@pytest.mark.asyncio +async def test_process_zip_file_upsert_targets_workspace_scoped_identity(): + # Alembic 0020_email_workspace_scope drops the 3-column + # uq_emails_owner_message_id constraint and replaces it with the 4-column + # uq_emails_workspace_message (user_id, organization_id, workspace_id, + # message_id). An ON CONFLICT target that still names only the old + # 3-column shape matches no constraint on a real PostgreSQL database and + # PostgreSQL rejects the statement outright -- every nonempty ZIP import + # would fail. SQLite (this test's default) is lenient about this, which is + # exactly why the mismatch survived undetected; compile against the + # PostgreSQL dialect to actually exercise the constraint-matching rule. + captured_statement = {} + + class _RecordingSession: + async def scalar(self, *args, **kwargs): + return None + + async def execute(self, statement, batch_values=None): + captured_statement["statement"] = statement + + async def commit(self): + pass + + email_data = { + "message_id": "msg-1", + "sender": "sender@example.com", + "reply_to": None, + "recipients": ["recipient@example.com"], + "subject": "Subject", + "in_reply_to": None, + "references": None, + "date": datetime.datetime(2026, 1, 1, tzinfo=datetime.timezone.utc), + "body": "body text", + } + + with ( + patch("scripts.import_fixtures.extract_backup_async") as mock_extract, + patch("scripts.import_fixtures.parse_eml", return_value=email_data), + patch("scripts.import_fixtures.chunk_text", return_value=[]), + patch( + "scripts.import_fixtures.assign_thread_id", + new=AsyncMock(return_value="thread-1"), + ), + ): + mock_extract.return_value = ["fixture.eml"] + await process_zip_file("dummy.zip", _RecordingSession()) + + compiled = str( + captured_statement["statement"].compile(dialect=postgresql.dialect()) + ) + assert ( + "ON CONFLICT (user_id, organization_id, workspace_id, message_id)" + in compiled + ) + + @pytest.mark.asyncio async def test_root_importer_persists_canonical_thread_id(tmp_path): class MockResult: @@ -50,11 +161,15 @@ async def commit(self): } session = MockSession() - with patch.object(import_fixtures, "parse_eml", return_value=parsed), patch.object( - import_fixtures, "generate_embeddings", new_callable=AsyncMock - ) as mock_embeddings, patch.object( - import_fixtures, "assign_thread_id", new_callable=AsyncMock - ) as mock_assign: + with ( + patch.object(import_fixtures, "parse_eml", return_value=parsed), + patch.object( + import_fixtures, "generate_embeddings", new_callable=AsyncMock + ) as mock_embeddings, + patch.object( + import_fixtures, "assign_thread_id", new_callable=AsyncMock + ) as mock_assign, + ): mock_embeddings.return_value = [[0.0] * 1536] mock_assign.return_value = "canonical-thread" @@ -99,11 +214,15 @@ async def commit(self): } session = MockSession() - with patch.object(import_fixtures, "parse_eml", return_value=parsed), patch.object( - import_fixtures, "generate_embeddings", new_callable=AsyncMock - ) as mock_embeddings, patch.object( - import_fixtures, "assign_thread_id", new_callable=AsyncMock - ) as mock_assign: + with ( + patch.object(import_fixtures, "parse_eml", return_value=parsed), + patch.object( + import_fixtures, "generate_embeddings", new_callable=AsyncMock + ) as mock_embeddings, + patch.object( + import_fixtures, "assign_thread_id", new_callable=AsyncMock + ) as mock_assign, + ): mock_embeddings.return_value = [[0.0] * 1536] mock_assign.return_value = "duplicate-scope-thread" @@ -111,17 +230,85 @@ async def commit(self): assert imported is True query_text = str(session.queries[0]).lower() - assert "email_records.message_id" in query_text - assert "email_records.user_id" in query_text - assert "email_records.organization_id" in query_text + where_clause = query_text.partition("where ")[2] + assert "email_records.message_id = :message_id_1" in where_clause + assert "email_records.user_id = :user_id_1" in where_clause + assert "email_records.organization_id = :organization_id_1" in where_clause + # Email's identity is scoped by workspace_id too (uq_emails_workspace_message); + # omitting it from the WHERE clause (workspace_id is always in the SELECT + # list simply because select(Email) selects every column, so checking for + # its bare presence in the whole query proves nothing) would let a + # duplicate lookup in one workspace find -- and wrongly skip re-importing + # -- a row that only exists in another one. + assert "email_records.workspace_id = :workspace_id_1" in where_clause mock_assign.assert_awaited_once_with( session, parsed, user_id=import_fixtures.IMPORT_USER_ID, organization_id=import_fixtures.IMPORT_ORGANIZATION_ID, + workspace_id=import_fixtures.IMPORT_WORKSPACE_ID, ) +@pytest.mark.asyncio +async def test_root_importer_stores_email_under_configured_workspace_id( + tmp_path, monkeypatch +): + # A custom NARUON_IMPORT_WORKSPACE_ID is threaded into assign_thread_id + # (asserted above) but the stored Email row must land in that SAME + # workspace -- otherwise thread assignment and email storage disagree on + # which workspace a fixture belongs to, splitting or hiding the imported + # conversation when queried by workspace. + class MockResult: + def scalar_one_or_none(self): + return None + + class MockSession: + def __init__(self): + self.added = None + + async def execute(self, _query): + return MockResult() + + def add(self, obj): + self.added = obj + + async def commit(self): + pass + + monkeypatch.setattr(import_fixtures, "IMPORT_WORKSPACE_ID", "workspace-custom-fixture") + + eml_file = tmp_path / "custom-workspace.eml" + eml_file.write_text("Message-ID: \n\nBody") + parsed = { + "message_id": "", + "sender": "sender@example.com", + "recipients": "user@example.com", + "subject": "Custom workspace", + "date": datetime.datetime.now(datetime.timezone.utc), + "body": "Body", + "attachments": [], + } + session = MockSession() + + with ( + patch.object(import_fixtures, "parse_eml", return_value=parsed), + patch.object( + import_fixtures, "generate_embeddings", new_callable=AsyncMock + ) as mock_embeddings, + patch.object( + import_fixtures, "assign_thread_id", new_callable=AsyncMock + ) as mock_assign, + ): + mock_embeddings.return_value = [[0.0] * 1536] + mock_assign.return_value = "custom-workspace-thread" + + imported = await import_fixtures.import_eml_file(session, eml_file) + + assert imported is True + assert session.added.workspace_id == "workspace-custom-fixture" + + @pytest.mark.asyncio async def test_root_importer_uses_local_embedding_without_openai_key( tmp_path, monkeypatch @@ -157,10 +344,13 @@ async def commit(self): monkeypatch.delenv("OPENAI_API_KEY", raising=False) session = MockSession() - with patch.object(import_fixtures, "parse_eml", return_value=parsed), patch.object( - import_fixtures, - "generate_embeddings", - side_effect=AssertionError("network call"), + with ( + patch.object(import_fixtures, "parse_eml", return_value=parsed), + patch.object( + import_fixtures, + "generate_embeddings", + side_effect=AssertionError("network call"), + ), ): imported = await import_fixtures.import_eml_file(session, eml_file) @@ -203,11 +393,15 @@ async def commit(self): monkeypatch.setenv("OPENAI_API_KEY", "test-key") session = MockSession() - with patch.object(import_fixtures, "parse_eml", return_value=parsed), patch.object( - import_fixtures, "generate_embeddings", new_callable=AsyncMock - ) as mock_embeddings, patch.object( - import_fixtures, "assign_thread_id", new_callable=AsyncMock - ) as mock_assign: + with ( + patch.object(import_fixtures, "parse_eml", return_value=parsed), + patch.object( + import_fixtures, "generate_embeddings", new_callable=AsyncMock + ) as mock_embeddings, + patch.object( + import_fixtures, "assign_thread_id", new_callable=AsyncMock + ) as mock_assign, + ): mock_embeddings.return_value = [] mock_assign.return_value = "empty-embedding-thread" @@ -253,11 +447,15 @@ async def rollback(self): } session = MockSession() - with patch.object(import_fixtures, "parse_eml", return_value=parsed), patch.object( - import_fixtures, "generate_embeddings", new_callable=AsyncMock - ) as mock_embeddings, patch.object( - import_fixtures, "assign_thread_id", new_callable=AsyncMock - ) as mock_assign: + with ( + patch.object(import_fixtures, "parse_eml", return_value=parsed), + patch.object( + import_fixtures, "generate_embeddings", new_callable=AsyncMock + ) as mock_embeddings, + patch.object( + import_fixtures, "assign_thread_id", new_callable=AsyncMock + ) as mock_assign, + ): mock_embeddings.return_value = [[0.0] * 1536] mock_assign.return_value = "commit-failure-thread" diff --git a/backend/tests/test_newsdom_worker.py b/backend/tests/test_newsdom_worker.py index 0693d395c..e94d3fec5 100644 --- a/backend/tests/test_newsdom_worker.py +++ b/backend/tests/test_newsdom_worker.py @@ -8,6 +8,7 @@ import asyncio import base64 +import datetime from types import SimpleNamespace import pytest @@ -77,7 +78,18 @@ def _pending_attachment( return attachment -def _pending_document(document_id: str, *, organization_id: str = "org-1") -> Document: +def _pending_document( + document_id: str, + *, + organization_id: str = "org-1", + created_at=None, +) -> Document: + # A real, DB-flushed Document always has created_at populated (the + # column's own default); constructing one directly, outside a session, + # would otherwise leave it None -- an artifact of the test fixture that + # a bare, un-flushed cursor comparison can never hit in production. + if created_at is None: + created_at = datetime.datetime.now(datetime.timezone.utc) return Document( document_id=document_id, workspace_id="ws-1", @@ -86,6 +98,7 @@ def _pending_document(document_id: str, *, organization_id: str = "org-1") -> Do document_type="pdf", document_content=base64.b64encode(b"%PDF-1.7 fake").decode("ascii"), document_status=PDF_DOM_RECOGNITION_PENDING_STATUS, + created_at=created_at, ) @@ -100,9 +113,35 @@ def all(self): return self._rows +def _row_key(row): + """Return the id a bulk-loaded row is keyed by for ``session.get``. + + Duck-typed (not ``isinstance``) so the expired-instance test doubles + (``_ExpiredAttachment``/``_ExpiredDocument``, which only expose one of + ``id``/``document_id`` and raise ``AttributeError`` on any other read) + register under the right key without triggering that raise. + """ + document_id = getattr(row, "document_id", None) + return row.id if document_id is None else document_id + + class _SequenceSession: - def __init__(self, row_batches): + """A fake session whose ``get`` always returns a fresh, healthy instance. + + Mirrors the real sweep contract: the bulk-loaded rows only supply ids + and the cursor; every actual processing target is re-fetched via ``get`` + by id, exactly like a real ``AsyncSession`` would after an earlier + item's rollback expired the bulk-loaded instances. ``by_id`` lets a + test register a *different* object than the bulk-loaded one to prove + that re-fetch is what the sweep actually uses. + """ + + def __init__(self, row_batches, *, by_id=None): self._row_batches = list(row_batches) + self._by_id = dict(by_id or {}) + for batch in self._row_batches: + for row in batch: + self._by_id.setdefault(_row_key(row), row) self.statements = [] self.commit_count = 0 self.rollback_count = 0 @@ -111,6 +150,9 @@ async def execute(self, statement): self.statements.append(statement) return _RowsResult(self._row_batches.pop(0)) + async def get(self, _model, row_id, options=None): + return self._by_id.get(row_id) + async def commit(self): self.commit_count += 1 @@ -129,22 +171,47 @@ async def __aexit__(self, *_args): return False -class _LeaseSession: - def __init__(self, *, dialect_name="postgresql", scalar_result=True): - self.bind = SimpleNamespace( - dialect=SimpleNamespace(name=dialect_name), - ) +class _LeaseConnection: + def __init__(self, *, scalar_result=True): self.scalar_result = scalar_result self.scalar_calls = [] - - def get_bind(self): - return self.bind + self.execution_options_calls = [] + # Ordered log spanning both call types, so a test can prove + # AUTOCOMMIT was set BEFORE the advisory-lock query ran, not just + # that both happened at some point. + self.ordered_calls = [] + + async def execution_options(self, **options): + self.execution_options_calls.append(options) + self.ordered_calls.append(("execution_options", options)) + return self async def scalar(self, statement, params): self.scalar_calls.append((statement, params)) + self.ordered_calls.append(("scalar", params)) return self.scalar_result +class _FakeEngine: + def __init__(self, *, dialect_name="postgresql", connection=None): + self.dialect = SimpleNamespace(name=dialect_name) + self._connection = connection + + def connect(self): + return _LeaseConnectionContext(self._connection) + + +class _LeaseConnectionContext: + def __init__(self, connection): + self.connection = connection + + async def __aenter__(self): + return self.connection + + async def __aexit__(self, *_args): + return False + + @pytest.mark.asyncio async def test_attachment_recognized_when_configured(): attachment = _pending_attachment() @@ -407,14 +474,450 @@ async def request_fn(**_kwargs): @pytest.mark.asyncio -async def test_document_sweep_advances_and_wraps_without_starvation(): +async def test_attachment_sweep_advances_the_cursor_and_retries_the_failed_row( + monkeypatch, +): + # An earlier version capped the cursor at the first failure instead of + # advancing it, to keep that row selectable later. That protected the + # one stuck row but pinned the *whole batch window* behind it -- once + # more than batch_limit rows were stuck at once, nothing past them was + # ever reached (reproduced directly; see _sweep_attachments's + # docstring). The cursor now always advances to the batch's last row; + # the failed row is retried instead via the independent + # _attachment_retry_ids set. + first = _pending_attachment(attachment_id=1) + second = _pending_attachment(attachment_id=2) + third = _pending_attachment(attachment_id=3) + session = _SequenceSession([[first, second, third], [second]]) + + async def config_resolver(_session, _organization_id): + return _config() + + failed_once = set() + + async def fail_the_middle_item_once(*, session, attachment, config_resolver, request_fn): + if attachment.id == 2 and attachment.id not in failed_once: + failed_once.add(attachment.id) + raise RuntimeError("recognition blew up") + return await process_pending_attachment( + session=session, + attachment=attachment, + config_resolver=config_resolver, + request_fn=request_fn, + ) + + monkeypatch.setattr( + newsdom_worker_module, "process_pending_attachment", fail_the_middle_item_once + ) + async def request_fn(**_kwargs): + return _canned_response() + + worker = NewsdomRecognitionWorker( + config_resolver=config_resolver, + request_fn=request_fn, + ) + await worker._sweep_attachments(session) + + assert worker._attachment_cursor == 3 + assert worker._attachment_retry_ids == {2} + assert first.parse_status == "parsed" + assert second.parse_status == PDF_DOM_RECOGNITION_PENDING_STATUS + assert third.parse_status == "parsed" + assert session.commit_count == 2 + assert session.rollback_count == 1 + + # The next sweep reselects row 2 via the retry set, not the cursor + # (which stays at 3, well past it). + await worker._sweep_attachments(session) + + assert worker._attachment_cursor == 3 + assert worker._attachment_retry_ids == set() + assert second.parse_status == "parsed" + assert session.commit_count == 3 + + +@pytest.mark.asyncio +async def test_attachment_sweep_advances_the_cursor_and_retries_the_pending_row(): + # Same fix as the failure case: RESULT_PENDING (no active provider yet) + # no longer caps the cursor either -- it goes into _attachment_retry_ids + # instead, so rows after it in the same batch are never held hostage. + first = _pending_attachment(attachment_id=1, organization_id="org-unconfigured") + second = _pending_attachment(attachment_id=2, organization_id="org-ready") + third = _pending_attachment(attachment_id=3, organization_id="org-ready") + session = _SequenceSession([[first, second, third], [first]]) + + configured = {"org-unconfigured": False, "org-ready": True} + + async def config_resolver(_session, organization_id): + return _config() if configured[organization_id] else None + + async def request_fn(**_kwargs): + return _canned_response() + + worker = NewsdomRecognitionWorker( + config_resolver=config_resolver, + request_fn=request_fn, + ) + await worker._sweep_attachments(session) + + assert worker._attachment_cursor == 3 + assert worker._attachment_retry_ids == {1} + assert first.parse_status == PDF_DOM_RECOGNITION_PENDING_STATUS + assert second.parse_status == "parsed" + assert third.parse_status == "parsed" + assert session.rollback_count == 0 + + # Once org-unconfigured gets a provider, the retry set -- not the + # cursor, which never revisits row 1 -- is what still reselects it. + configured["org-unconfigured"] = True + await worker._sweep_attachments(session) + + assert first.parse_status == "parsed" + assert worker._attachment_retry_ids == set() + + +class _ExpiredAttachment: + """Stands in for an ORM instance ``AsyncSession.rollback()`` expired. + + Any attribute read raises, exactly like touching an expired async-mapped + instance outside an active await/greenlet context would in real + SQLAlchemy -- proving the sweep never touches this bulk-loaded object + for its *own* processing once it has already failed and been rolled + back once. + """ + + id = 1 + + def __getattr__(self, _name): + raise AttributeError( + "must not read attributes off the stale bulk-loaded instance" + ) + + +class _ExpiredDocument: + """Document counterpart of ``_ExpiredAttachment``.""" + + document_id = "doc-001" + + def __getattr__(self, _name): + raise AttributeError( + "must not read attributes off the stale bulk-loaded instance" + ) + + +@pytest.mark.asyncio +async def test_attachment_sweep_never_processes_the_bulk_loaded_instance_directly(): + poisoned_first = _ExpiredAttachment() + fresh_first = _pending_attachment(attachment_id=1, organization_id="org-ready") + second = _pending_attachment(attachment_id=2, organization_id="org-ready") + # The bulk query "sees" the poisoned stand-in for id 1 (as a real + # AsyncSession would after some earlier rollback expired it), but `get` + # returns the real, healthy row -- proving the sweep re-fetches instead + # of processing the bulk-loaded object directly. + session = _SequenceSession( + [[poisoned_first, second]], by_id={1: fresh_first, 2: second} + ) + + async def config_resolver(_session, _organization_id): + return _config() + + async def request_fn(**_kwargs): + return _canned_response() + + worker = NewsdomRecognitionWorker( + config_resolver=config_resolver, request_fn=request_fn + ) + await worker._sweep_attachments(session) + + assert fresh_first.parse_status == "parsed" + assert second.parse_status == "parsed" + assert session.commit_count == 2 + assert session.rollback_count == 0 + + +class _LivePendingAttachmentSession: + """A session whose pending-attachment query genuinely reflects the + worker's own (cursor, retry_ids) filtering, instead of replaying a + pre-scripted sequence of batches like ``_SequenceSession``. + + ``_SequenceSession`` is fine for single-sweep, single-assertion tests, + but it can't prove a *multi-sweep scheduling* claim: it would happily + "return" whatever batch a test pre-registers regardless of whether the + real query would ever actually produce it. Reproducing the Devin-review + finding this fixes (more than ``batch_limit`` consecutive permanently- + pending rows starve every row after them) needs a fake that mirrors + what a real database would return for + ``NewsdomRecognitionWorker._pending_attachment_statement`` across many + sweeps: every attachment still carrying + ``PDF_DOM_RECOGNITION_PENDING_STATUS`` whose id is either past the + worker's forward cursor or in its retry set, forward rows ordered ahead + of retry rows, capped at ``batch_limit``. + """ + + def __init__(self, worker, attachments): + self._worker = worker + self._table = {attachment.id: attachment for attachment in attachments} + self.commit_count = 0 + self.rollback_count = 0 + + async def execute(self, _statement): + cursor = self._worker._attachment_cursor + retry_ids = self._worker._attachment_retry_ids + pending = [ + row + for row in self._table.values() + if row.parse_status == PDF_DOM_RECOGNITION_PENDING_STATUS + and (cursor is None or row.id > cursor or row.id in retry_ids) + ] + pending.sort(key=lambda row: (0 if (cursor is None or row.id > cursor) else 1, row.id)) + return _RowsResult(pending[: self._worker.batch_limit]) + + async def get(self, _model, row_id, options=None): + return self._table.get(row_id) + + async def commit(self): + self.commit_count += 1 + + async def rollback(self): + self.rollback_count += 1 + + +@pytest.mark.asyncio +async def test_attachment_sweep_does_not_starve_rows_behind_many_stuck_rows(): + # The Devin-review finding this fixes: an organization bulk-imports a + # burst of PDFs before ever configuring a provider -- more than + # batch_limit consecutive rows land pending at once. Reproduced first + # against the pre-fix code (cursor capped at the first unresolved row): + # with 60 leading stuck rows and batch_limit=50, every sweep re-selected + # the same first 50 stuck rows forever and the 60 healthy rows behind + # them were never reached, even after 14 sweeps. blocked = [ - _pending_document(f"doc-{index:03d}", organization_id="org-blocked") + _pending_attachment(attachment_id=index, organization_id="org-blocked") + for index in range(1, 61) + ] + ready = [ + _pending_attachment(attachment_id=index, organization_id="org-ready") + for index in range(61, 121) + ] + all_attachments = blocked + ready + + async def config_resolver(_session, organization_id): + return _config() if organization_id == "org-ready" else None + + async def request_fn(**_kwargs): + return _canned_response() + + worker = NewsdomRecognitionWorker( + batch_limit=50, config_resolver=config_resolver, request_fn=request_fn + ) + session = _LivePendingAttachmentSession(worker, all_attachments) + + for _ in range(10): + await worker._sweep_attachments(session) + if all(a.parse_status == "parsed" for a in ready): + break + + assert all(a.parse_status == "parsed" for a in ready) + assert all( + a.parse_status == PDF_DOM_RECOGNITION_PENDING_STATUS for a in blocked + ) + assert worker._attachment_retry_ids == {a.id for a in blocked} + + +@pytest.mark.asyncio +async def test_attachment_sweep_rediscovers_a_row_reverted_to_pending_behind_the_cursor(): + # An already-recognized attachment can, in principle, be explicitly + # re-marked pending again later (the same class of external transition + # Devin Review flagged for the reparse worker's reparse-intent endpoint, + # and for documents' pdf-dom-recognition-intent endpoint) -- its id is + # then already behind the forward cursor, and it was never seen as + # unresolved so it isn't in the retry set either. A periodic full + # rescan (every FULL_RESCAN_EVERY_N_SWEEPS sweeps) bounds how long such + # a row can stay invisible. + reverted = _pending_attachment(attachment_id=1, organization_id="org-ready") + + async def config_resolver(_session, _organization_id): + return _config() + + async def request_fn(**_kwargs): + return _canned_response() + + worker = NewsdomRecognitionWorker( + config_resolver=config_resolver, request_fn=request_fn + ) + session = _LivePendingAttachmentSession(worker, [reverted]) + + await worker._sweep_attachments(session) + assert reverted.parse_status == "parsed" + assert worker._attachment_cursor == 1 + + reverted.parse_status = PDF_DOM_RECOGNITION_PENDING_STATUS + + rediscovered_at = None + for sweep_number in range(2, newsdom_worker_module.FULL_RESCAN_EVERY_N_SWEEPS + 1): + await worker._sweep_attachments(session) + if reverted.parse_status != PDF_DOM_RECOGNITION_PENDING_STATUS: + rediscovered_at = sweep_number + break + + assert rediscovered_at == newsdom_worker_module.FULL_RESCAN_EVERY_N_SWEEPS + + +class _LivePendingDocumentSession: + """Document counterpart of ``_LivePendingAttachmentSession`` -- mirrors + ``NewsdomRecognitionWorker._pending_document_statement`` for real, + including its ``(created_at, document_id)`` forward comparison, so a + test can prove the query genuinely reaches a document whose random UUID + sorts below the cursor but whose ``created_at`` sorts after it. + """ + + def __init__(self, worker, documents): + self._worker = worker + self._table = {document.document_id: document for document in documents} + self.commit_count = 0 + self.rollback_count = 0 + + def _is_forward(self, document): + cursor = self._worker._document_cursor + if cursor is None: + return True + cursor_created_at, cursor_document_id = cursor + if document.created_at != cursor_created_at: + return document.created_at > cursor_created_at + return document.document_id > cursor_document_id + + async def execute(self, _statement): + retry_ids = self._worker._document_retry_ids + pending = [ + row + for row in self._table.values() + if row.document_status == PDF_DOM_RECOGNITION_PENDING_STATUS + and (self._is_forward(row) or row.document_id in retry_ids) + ] + pending.sort( + key=lambda row: ( + 0 if self._is_forward(row) else 1, + row.created_at, + row.document_id, + ) + ) + return _RowsResult(pending[: self._worker.batch_limit]) + + async def get(self, _model, document_id): + return self._table.get(document_id) + + async def commit(self): + self.commit_count += 1 + + async def rollback(self): + self.rollback_count += 1 + + +@pytest.mark.asyncio +async def test_document_sweep_cursor_uses_created_at_not_document_id_ordering(): + # Document.document_id defaults to a random UUID (db/models.py: default= + # lambda: f"doc_{uuid.uuid4().hex}"), so it is NOT monotonic with + # insertion order -- a document inserted later can sort lexicographically + # *below* one inserted earlier. Devin Review flagged that a cursor based + # on document_id alone would then permanently miss such a row: it is + # never in the retry set (never seen before) and never satisfies + # "document_id > cursor" (it sorts lower), so it would stay pending + # forever. The cursor is now a (created_at, document_id) tuple -- + # created_at is genuinely monotonic with insertion order. + early = datetime.datetime(2026, 1, 1, tzinfo=datetime.timezone.utc) + later = datetime.datetime(2026, 1, 2, tzinfo=datetime.timezone.utc) + # "doc-zzz" sorts after "doc-aaa" as a string, despite being created + # first -- exactly the adversarial ordering the fix must survive. + already_seen = _pending_document( + "doc-zzz", organization_id="org-ready", created_at=early + ) + new_arrival = _pending_document( + "doc-aaa", organization_id="org-ready", created_at=later + ) + + async def config_resolver(_session, _organization_id): + return _config() + + async def request_fn(**_kwargs): + return _canned_response() + + worker = NewsdomRecognitionWorker( + config_resolver=config_resolver, request_fn=request_fn + ) + session = _LivePendingDocumentSession(worker, [already_seen]) + await worker._sweep_documents(session) + assert worker._document_cursor == (early, "doc-zzz") + assert already_seen.document_status == "parsed" + + # The new, later-arriving document lands in the same table only once it + # actually exists -- exactly like a real insert between sweeps. + session._table["doc-aaa"] = new_arrival + await worker._sweep_documents(session) + + assert new_arrival.document_status == "parsed" + + +@pytest.mark.asyncio +async def test_document_sweep_rediscovers_a_row_reverted_to_pending_behind_the_cursor(): + # POST .../pdf-dom-recognition-intent (backend/api/data.py) can + # explicitly re-trigger recognition on ANY existing document, including + # one whose (created_at, document_id) is already behind the forward + # cursor. Neither the cursor nor the retry set can discover that on its + # own; a periodic full rescan bounds how long it can stay invisible. + reverted = _pending_document("doc-001", organization_id="org-ready") + + async def config_resolver(_session, _organization_id): + return _config() + + async def request_fn(**_kwargs): + return _canned_response() + + worker = NewsdomRecognitionWorker( + config_resolver=config_resolver, request_fn=request_fn + ) + session = _LivePendingDocumentSession(worker, [reverted]) + + await worker._sweep_documents(session) + assert reverted.document_status == "parsed" + cursor_after_first_sweep = worker._document_cursor + + # Simulate an operator re-triggering recognition on this now-old + # document -- its (created_at, document_id) is already behind the cursor. + reverted.document_status = PDF_DOM_RECOGNITION_PENDING_STATUS + + rediscovered_at = None + for sweep_number in range(2, newsdom_worker_module.FULL_RESCAN_EVERY_N_SWEEPS + 1): + await worker._sweep_documents(session) + if reverted.document_status != PDF_DOM_RECOGNITION_PENDING_STATUS: + rediscovered_at = sweep_number + break + + assert rediscovered_at == newsdom_worker_module.FULL_RESCAN_EVERY_N_SWEEPS + assert cursor_after_first_sweep == (reverted.created_at, "doc-001") + + +@pytest.mark.asyncio +async def test_document_sweep_advances_the_cursor_and_retries_a_blocked_row(): + # A batch that fully resolves lets the cursor advance to its tail (a + # lexicographic max, since document_id is a string). A later-arriving + # row that's still blocked (no provider configured for its org) goes + # into _document_retry_ids instead of relying on the query going empty + # to trigger a full rescan -- an earlier design's wrap-to-None never + # fired under continuous inbound traffic (new rows keep landing past + # the cursor, so the query never returns zero rows), which could starve + # that row forever. See _sweep_attachments's docstring for the fuller + # rationale (identical fix, shared between both sweeps). + resolved_batch = [ + _pending_document(f"doc-{index:03d}", organization_id="org-ready") for index in range(1, 11) ] - ready_after_batch = _pending_document("doc-011", organization_id="org-ready") - ready_after_wrap = _pending_document("doc-001", organization_id="org-ready") - session = _SequenceSession([blocked, [ready_after_batch], [], [ready_after_wrap]]) + late_arrival = _pending_document("doc-011", organization_id="org-blocked") + # The re-sweep re-fetches "doc-011" by id -- exactly like a real + # database would return the current state of that single row, not a + # separate instance -- so simulate "it became configured" by mutating + # the same object rather than registering a second, distinct Document + # under the same id. + session = _SequenceSession([resolved_batch, [late_arrival], [late_arrival]]) async def config_resolver(_session, organization_id): return _config() if organization_id == "org-ready" else None @@ -428,37 +931,182 @@ async def request_fn(**_kwargs): request_fn=request_fn, ) await worker._sweep_documents(session) + assert worker._document_cursor == (resolved_batch[-1].created_at, "doc-010") + assert worker._document_retry_ids == set() + for document in resolved_batch: + assert document.document_status == "parsed" + await worker._sweep_documents(session) - worker._document_cursor = "doc-999" + # The cursor still advances (to doc-011, past doc-010), but doc-011 + # stays pending and now enters the retry set rather than forcing a + # rescan-from-scratch. + assert worker._document_cursor == (late_arrival.created_at, "doc-011") + assert worker._document_retry_ids == {"doc-011"} + assert late_arrival.document_status == PDF_DOM_RECOGNITION_PENDING_STATUS + + late_arrival.organization_id = "org-ready" await worker._sweep_documents(session) + assert worker._document_cursor == (late_arrival.created_at, "doc-011") + assert worker._document_retry_ids == set() + assert late_arrival.document_status == "parsed" - second_query = session.statements[1].compile() - wrapped_query = session.statements[3].compile() - assert "workspace_documents.document_id >" in str(second_query) - assert "doc-010" in second_query.params.values() - assert "workspace_documents.document_id >" not in str(wrapped_query) - assert worker._document_cursor == "doc-001" - assert ready_after_batch.document_status == "parsed" - assert ready_after_wrap.document_status == "parsed" assert session.commit_count == 12 assert session.rollback_count == 0 @pytest.mark.asyncio -async def test_attachment_cursor_wraps_and_empty_batches_are_stable(): - wrapped = _pending_attachment(attachment_id=1) - session = _SequenceSession([[], [wrapped], []]) +async def test_document_sweep_advances_the_cursor_and_retries_the_failed_row( + monkeypatch, +): + # Same fix as the attachment sweep, adapted for the string-keyed + # document cursor: the cursor always advances to the batch's last row + # (a lexicographic max) instead of capping at the first failure, and the + # failed row is retried via _document_retry_ids instead. + first = _pending_document("doc-001") + second = _pending_document("doc-002") + third = _pending_document("doc-003") + session = _SequenceSession([[first, second, third], [second]]) + + async def config_resolver(_session, _organization_id): + return _config() + + failed_once = set() + + async def fail_the_middle_item_once(*, session, document, config_resolver, request_fn): + if document.document_id == "doc-002" and document.document_id not in failed_once: + failed_once.add(document.document_id) + raise RuntimeError("recognition blew up") + return await process_pending_document( + session=session, + document=document, + config_resolver=config_resolver, + request_fn=request_fn, + ) + + monkeypatch.setattr( + newsdom_worker_module, "process_pending_document", fail_the_middle_item_once + ) + async def request_fn(**_kwargs): + return _canned_response() + + worker = NewsdomRecognitionWorker( + config_resolver=config_resolver, + request_fn=request_fn, + ) + await worker._sweep_documents(session) + + assert worker._document_cursor == (third.created_at, "doc-003") + assert worker._document_retry_ids == {"doc-002"} + assert first.document_status == "parsed" + assert second.document_status == PDF_DOM_RECOGNITION_PENDING_STATUS + assert third.document_status == "parsed" + assert session.commit_count == 2 + assert session.rollback_count == 1 + + await worker._sweep_documents(session) + + assert worker._document_cursor == (third.created_at, "doc-003") + assert worker._document_retry_ids == set() + assert second.document_status == "parsed" + assert session.commit_count == 3 + + +@pytest.mark.asyncio +async def test_document_sweep_advances_the_cursor_and_retries_the_pending_row(): + # Same fix as the attachment sweep: RESULT_PENDING no longer caps the + # cursor either -- it goes into _document_retry_ids instead. + first = _pending_document("doc-001", organization_id="org-unconfigured") + second = _pending_document("doc-002", organization_id="org-ready") + third = _pending_document("doc-003", organization_id="org-ready") + session = _SequenceSession([[first, second, third], [first]]) + + configured = {"org-unconfigured": False, "org-ready": True} + + async def config_resolver(_session, organization_id): + return _config() if configured[organization_id] else None + + async def request_fn(**_kwargs): + return _canned_response() + + worker = NewsdomRecognitionWorker( + config_resolver=config_resolver, + request_fn=request_fn, + ) + await worker._sweep_documents(session) + + assert worker._document_cursor == (third.created_at, "doc-003") + assert worker._document_retry_ids == {"doc-001"} + assert first.document_status == PDF_DOM_RECOGNITION_PENDING_STATUS + assert second.document_status == "parsed" + assert third.document_status == "parsed" + assert session.rollback_count == 0 + + configured["org-unconfigured"] = True + await worker._sweep_documents(session) + + assert first.document_status == "parsed" + assert worker._document_retry_ids == set() + + +@pytest.mark.asyncio +async def test_document_sweep_never_processes_the_bulk_loaded_instance_directly(): + poisoned_first = _ExpiredDocument() + fresh_first = _pending_document("doc-001", organization_id="org-ready") + second = _pending_document("doc-002", organization_id="org-ready") + session = _SequenceSession( + [[poisoned_first, second]], + by_id={"doc-001": fresh_first, "doc-002": second}, + ) + + async def config_resolver(_session, _organization_id): + return _config() + + async def request_fn(**_kwargs): + return _canned_response() + + worker = NewsdomRecognitionWorker( + config_resolver=config_resolver, request_fn=request_fn + ) + await worker._sweep_documents(session) + + assert fresh_first.document_status == "parsed" + assert second.document_status == "parsed" + assert session.commit_count == 2 + assert session.rollback_count == 0 + + +@pytest.mark.asyncio +async def test_load_pending_attachments_queries_forward_cursor_and_retry_ids(): + # No more wraparound: a persistently-stuck row is retried via an + # explicit "id IN retry_ids" filter, independent of the forward cursor, + # instead of relying on the query going empty to trigger a rescan (which + # never happens once new rows keep landing past the cursor). + row = _pending_attachment(attachment_id=5) + session = _SequenceSession([[row]]) worker = NewsdomRecognitionWorker(batch_limit=10) - worker._attachment_cursor = 999 + worker._attachment_cursor = 3 + worker._attachment_retry_ids = {1} rows = await worker._load_pending_attachments(session) - empty_rows = await worker._load_pending_attachments(session) - assert rows == [wrapped] - assert empty_rows == [] - assert worker._attachment_cursor is None - assert "email_attachments.id >" in str(session.statements[0]) - assert "email_attachments.id >" not in str(session.statements[1]) + assert rows == [row] + compiled = str(session.statements[0].compile()) + assert "email_attachments.id >" in compiled + assert "IN" in compiled.upper() + + +@pytest.mark.asyncio +async def test_load_pending_attachments_has_no_id_filter_before_the_first_sweep(): + row = _pending_attachment(attachment_id=1) + session = _SequenceSession([[row]]) + worker = NewsdomRecognitionWorker(batch_limit=10) + + rows = await worker._load_pending_attachments(session) + + assert rows == [row] + compiled = str(session.statements[0].compile()) + assert "email_attachments.id >" not in compiled + assert "IN" not in compiled.upper() @pytest.mark.asyncio @@ -497,48 +1145,104 @@ async def broken_resolver(_session, _organization_id): @pytest.mark.asyncio -async def test_postgresql_lease_helpers_and_non_postgresql_fallback(): - postgres = _LeaseSession(scalar_result=1) - sqlite = _LeaseSession(dialect_name="sqlite") +async def test_postgresql_lease_helpers_and_non_postgresql_fallback(monkeypatch): + postgres_connection = _LeaseConnection(scalar_result=1) - assert await newsdom_worker_module._try_acquire_sweep_lease(postgres) is True - assert postgres.scalar_calls[0][1] == newsdom_worker_module._SWEEP_LOCK_PARAMS - await newsdom_worker_module._release_sweep_lease(postgres) - assert len(postgres.scalar_calls) == 2 - assert await newsdom_worker_module._try_acquire_sweep_lease(sqlite) is None + assert ( + await newsdom_worker_module._try_acquire_sweep_lease(postgres_connection) + is True + ) + # AUTOCOMMIT before the lock-acquire statement: a plain (non-autocommit) + # connection would otherwise leave this connection's implicit + # transaction open and idle for the whole sweep -- a PostgreSQL + # idle_in_transaction_session_timeout could then kill this connection + # mid-sweep, silently dropping the lease. + assert postgres_connection.execution_options_calls == [ + {"isolation_level": "AUTOCOMMIT"} + ] + assert ( + postgres_connection.scalar_calls[0][1] + == newsdom_worker_module._SWEEP_LOCK_PARAMS + ) + # Ordering, not just occurrence: AUTOCOMMIT must be set before the + # advisory-lock query runs, or the query could still open an implicit + # transaction under the connection's prior isolation level. + assert postgres_connection.ordered_calls[0][0] == "execution_options" + assert postgres_connection.ordered_calls[1][0] == "scalar" + await newsdom_worker_module._release_sweep_lease(postgres_connection) + assert len(postgres_connection.scalar_calls) == 2 + + monkeypatch.setattr( + newsdom_worker_module, "engine", _FakeEngine(dialect_name="sqlite") + ) + assert newsdom_worker_module._engine_uses_postgresql() is False + + monkeypatch.setattr( + newsdom_worker_module, "engine", _FakeEngine(dialect_name="postgresql") + ) + assert newsdom_worker_module._engine_uses_postgresql() is True + + +@pytest.mark.asyncio +async def test_worker_sweep_skips_locking_when_engine_is_not_postgresql(monkeypatch): + session = object() + calls = [] + worker = NewsdomRecognitionWorker() + + monkeypatch.setattr( + newsdom_worker_module, "engine", _FakeEngine(dialect_name="sqlite") + ) + monkeypatch.setattr( + newsdom_worker_module, + "AsyncSessionLocal", + lambda: _AsyncSessionContext(session), + ) - class BrokenBindSession: - def get_bind(self): - raise RuntimeError("no bind") + async def sweep_attachments(actual_session): + calls.append(("attachments", actual_session)) - assert newsdom_worker_module._session_uses_postgresql(BrokenBindSession()) is False + async def sweep_documents(actual_session): + calls.append(("documents", actual_session)) + + monkeypatch.setattr(worker, "_sweep_attachments", sweep_attachments) + monkeypatch.setattr(worker, "_sweep_documents", sweep_documents) + + await worker._sweep() + + assert calls == [("attachments", session), ("documents", session)] @pytest.mark.asyncio @pytest.mark.parametrize( ("lease", "expected_sweeps", "expected_releases"), - [(False, 0, 0), (None, 2, 0), (True, 2, 1)], + [(False, 0, 0), (True, 2, 1)], ) async def test_worker_sweep_honors_lease_outcome( monkeypatch, lease, expected_sweeps, expected_releases ): session = object() + lock_connection = object() calls = [] releases = [] worker = NewsdomRecognitionWorker() + monkeypatch.setattr( + newsdom_worker_module, + "engine", + _FakeEngine(dialect_name="postgresql", connection=lock_connection), + ) monkeypatch.setattr( newsdom_worker_module, "AsyncSessionLocal", lambda: _AsyncSessionContext(session), ) - async def acquire(actual_session): - assert actual_session is session + async def acquire(actual_connection): + assert actual_connection is lock_connection return lease - async def release(actual_session): - releases.append(actual_session) + async def release(actual_connection): + releases.append(actual_connection) async def sweep_attachments(actual_session): calls.append(("attachments", actual_session)) diff --git a/backend/tests/test_newsdom_worker_expired_batch_regression.py b/backend/tests/test_newsdom_worker_expired_batch_regression.py new file mode 100644 index 000000000..4f910f15f --- /dev/null +++ b/backend/tests/test_newsdom_worker_expired_batch_regression.py @@ -0,0 +1,50 @@ +"""Regression coverage for NewsDOM batch identity after session rollback.""" + +import pytest + +from services.newsdom_worker import NewsdomRecognitionWorker + + +class _OneReadAttachmentId: + """Bulk row whose mapped identity becomes unreadable after first access.""" + + def __init__(self, attachment_id: int) -> None: + self._attachment_id = attachment_id + self._reads = 0 + + @property + def id(self) -> int: + self._reads += 1 + if self._reads > 1: + raise RuntimeError("bulk row was expired by rollback") + return self._attachment_id + + +class _RollbackSession: + def __init__(self) -> None: + self.rollback_count = 0 + + async def get(self, *_args, **_kwargs): + raise RuntimeError("force per-item rollback") + + async def rollback(self) -> None: + self.rollback_count += 1 + + +@pytest.mark.asyncio +async def test_attachment_cursor_uses_ids_snapshotted_before_rollback() -> None: + """Cursor advancement must not dereference bulk ORM rows after rollback.""" + bulk_row = _OneReadAttachmentId(41) + session = _RollbackSession() + worker = NewsdomRecognitionWorker() + + async def load_pending(_session): + return [bulk_row] + + worker._load_pending_attachments = load_pending # type: ignore[method-assign] + + await worker._sweep_attachments(session) # type: ignore[arg-type] + + assert session.rollback_count == 1 + assert worker._attachment_cursor == 41 + assert worker._attachment_retry_ids == {41} diff --git a/backend/tests/test_noema_agent.py b/backend/tests/test_noema_agent.py index 31a562db6..ef3b0b3d6 100644 --- a/backend/tests/test_noema_agent.py +++ b/backend/tests/test_noema_agent.py @@ -3,12 +3,14 @@ These cover three seams without needing a live LLM or a database: * tool wiring (mail read/search, content-graph, task actions, writeback) -* configuration resolved from the DB provider layer (not ``os.getenv``) -* graceful degradation when the provider or pydantic-ai runtime is absent +* contextual-orchestrator gateway configuration resolved from the tenant's + own DB row (not ``os.getenv``, and never a direct tenant LLM-provider key) +* graceful degradation when the gateway or the pydantic-ai runtime is absent """ import datetime +import httpx import pytest from db.models import ( @@ -16,16 +18,16 @@ ContentNodeRecord, Email, KnowledgeGraphEdgeRecord, - LLMProvider, + TenantConfig, TicketTask, ) -from services import noema_agent -from services.llm_provider_selection import RuntimeLLMProvider +from services import noema_agent, orchestrator_gateway from services.noema_agent import ( NOEMA_TOOL_SPECS, NoemaAgentDeps, build_noema_agent, run_noema_agent, + tool_check_calendar_conflict, tool_content_graph_query, tool_dispatch_writeback, tool_list_tasks, @@ -33,10 +35,16 @@ tool_search_mail, tool_update_task_status, ) +from services.orchestrator_gateway import OrchestratorGateway UTC = datetime.timezone.utc +async def _pass_through_url_validator(value: str | None) -> str | None: + """Stand in for real SSRF/DNS validation in hermetic unit tests.""" + return value + + class _FakeScalars: def __init__(self, items): self._items = list(items) @@ -68,10 +76,12 @@ class _QueueSession: def __init__(self, results=None): self._results = list(results or []) + self.statements = [] self.added = [] self.commits = 0 async def execute(self, _statement): + self.statements.append(_statement) if not self._results: return _FakeResult() return self._results.pop(0) @@ -121,12 +131,16 @@ async def test_search_mail_returns_owner_scoped_snippets(): assert len(results) == 1 assert results[0]["message_id"] == "msg-1" assert "budget" in results[0]["snippet"] + assert "email_records.workspace_id" in str(session.statements[0]) + assert "workspace-org-1" in session.statements[0].compile().params.values() @pytest.mark.asyncio async def test_read_mail_missing_returns_not_found(): session = _QueueSession([_FakeResult(items=[])]) result = await tool_read_mail(_deps(session), "msg-404") + assert "email_records.workspace_id" in str(session.statements[0]) + assert "workspace-org-1" in session.statements[0].compile().params.values() assert result["status"] == "not_found" @@ -172,6 +186,8 @@ async def test_content_graph_query_returns_nodes_and_edges(): assert result["status"] == "ok" assert result["nodes"][0]["uid"] == "node-1" assert result["edges"][0]["kind"] == "mentions" + assert "email_records.workspace_id" in str(session.statements[0]) + assert "workspace-org-1" in session.statements[0].compile().params.values() @pytest.mark.asyncio @@ -291,6 +307,137 @@ async def test_writeback_rejects_unknown_action(): assert result["status"] == "error" +# --------------------------------------------------------------------------- # +# Calendar conflict check: same deterministic policy as /api/calendar/conflicts +# --------------------------------------------------------------------------- # + + +def _commitment_row(commitment_id, start_at, end_at, status): + return { + "commitment_id": commitment_id, + "start_at": start_at, + "end_at": end_at, + "status": status, + } + + +@pytest.mark.asyncio +async def test_check_calendar_conflict_requires_authoritative_provider_evidence(): + session = _QueueSession([]) + result = await tool_check_calendar_conflict( + _deps(session), + proposed_commitment_id="new-1", + proposed_start_at="2026-03-01T10:00:00+00:00", + proposed_end_at="2026-03-01T11:00:00+00:00", + proposed_status="confirmed", + existing=[ + _commitment_row( + "other-1", + "2026-03-01T12:00:00+00:00", + "2026-03-01T13:00:00+00:00", + "confirmed", + ) + ], + ) + assert result["status"] == "error" + assert result["decision_code"] == "review_required" + assert result["error_code"] == "calendar_authoritative_evidence_unavailable" + + +@pytest.mark.asyncio +async def test_check_calendar_conflict_does_not_trust_conversational_overlap(): + session = _QueueSession([]) + result = await tool_check_calendar_conflict( + _deps(session), + proposed_commitment_id="new-1", + proposed_start_at="2026-03-01T10:00:00+00:00", + proposed_end_at="2026-03-01T11:00:00+00:00", + proposed_status="confirmed", + existing=[ + _commitment_row( + "other-1", + "2026-03-01T10:30:00+00:00", + "2026-03-01T11:30:00+00:00", + "confirmed", + ) + ], + ) + assert result["status"] == "error" + assert result["decision_code"] == "review_required" + + +@pytest.mark.asyncio +async def test_check_calendar_conflict_remains_review_required_without_provider_read(): + session = _QueueSession([]) + result = await tool_check_calendar_conflict( + _deps(session), + proposed_commitment_id="new-1", + proposed_start_at="2026-03-01T10:00:00+00:00", + proposed_end_at="2026-03-01T11:00:00+00:00", + proposed_status="confirmed", + existing=[ + _commitment_row( + "other-1", + "2026-03-01T10:30:00+00:00", + "2026-03-01T11:30:00+00:00", + "tentative", + ) + ], + ) + assert result["status"] == "error" + assert result["decision_code"] == "review_required" + + +@pytest.mark.asyncio +async def test_check_calendar_conflict_does_not_drop_malformed_rows_and_claim_available(): + session = _QueueSession([]) + result = await tool_check_calendar_conflict( + _deps(session), + proposed_commitment_id="new-1", + proposed_start_at="2026-03-01T10:00:00+00:00", + proposed_end_at="2026-03-01T11:00:00+00:00", + proposed_status="confirmed", + existing=[ + {"commitment_id": "bad-1"}, # missing start_at/end_at/status + _commitment_row("bad-2", "not-a-timestamp", "also-not-one", "confirmed"), + ], + ) + assert result["status"] == "error" + assert result["decision_code"] == "review_required" + assert result["error_code"] == "calendar_authoritative_evidence_unavailable" + + +@pytest.mark.asyncio +async def test_check_calendar_conflict_rejects_even_empty_unverified_evidence(): + """Conversational evidence cannot substitute for an authoritative read.""" + session = _QueueSession([]) + result = await tool_check_calendar_conflict( + _deps(session), + proposed_commitment_id="new-1", + proposed_start_at="2026-03-01T10:00:00+00:00", + proposed_end_at="2026-03-01T11:00:00+00:00", + proposed_status="confirmed", + existing=[], + ) + assert result["status"] == "error" + assert result["error_code"] == "calendar_authoritative_evidence_unavailable" + + +@pytest.mark.asyncio +async def test_check_calendar_conflict_rejects_invalid_proposed_status(): + session = _QueueSession([]) + result = await tool_check_calendar_conflict( + _deps(session), + proposed_commitment_id="new-1", + proposed_start_at="2026-03-01T10:00:00+00:00", + proposed_end_at="2026-03-01T11:00:00+00:00", + proposed_status="banana", + existing=[], + ) + assert result["status"] == "error" + assert result["error_code"] == "calendar_status_unsupported" + + def test_tool_specs_cover_declared_capabilities(): capabilities = {spec["capability"] for spec in NOEMA_TOOL_SPECS} assert { @@ -300,6 +447,7 @@ def test_tool_specs_cover_declared_capabilities(): "tasks.read", "tasks.update", "calendar.writeback", + "calendar.conflict_check", } <= capabilities @@ -308,44 +456,28 @@ def test_tool_specs_cover_declared_capabilities(): # --------------------------------------------------------------------------- # -class _ProviderScalars: - def __init__(self, items): - self._items = items - - def first(self): - return self._items[0] if self._items else None - - -class _ProviderResult: - def __init__(self, providers): - self._providers = providers - - def scalars(self): - return _ProviderScalars(self._providers) - - def scalar_one_or_none(self): - return None - - -class _ProviderSession: - """Minimal session that satisfies resolve_runtime_llm_provider.""" - - def __init__(self, providers): - self._providers = providers - - async def execute(self, statement): - text = str(statement).lower() - if "llm_providers" in text: - return _ProviderResult(self._providers) - return _ProviderResult([]) - - @pytest.mark.asyncio -async def test_run_agent_unavailable_without_provider(monkeypatch): - async def _no_provider(*args, **kwargs): - return None +async def test_run_agent_never_resolves_a_direct_tenant_llm_provider(): + """Noema's LLM path must go only through the contextual-orchestrator gateway. + + Owner architecture finding on this PR: naruon's general-purpose Noema + agent must never resolve a tenant's own configured LLM provider (or build + a direct ``openai.AsyncOpenAI`` client from one) -- that would make naruon + a second provider-routing authority instead of a `contextual-orchestrator` + consumer. Production LLM routing belongs to + `ContextualWisdomLab/contextual-orchestrator`; naruon owns the Noema + tools/authorization/context, not a second routing authority. + + ``resolve_runtime_llm_provider`` (the tenant-BYOK resolver used elsewhere + in this app for search/summaries) is not merely unused by + ``run_noema_agent`` -- it is not even imported into this module's + namespace any more, which is the strongest available proof there is + nothing left to fall back to. A workspace with no gateway configured + gets a single, structured "unavailable" result -- never a crash from + reaching for an ``LLMProvider`` row that was never queried. + """ + assert not hasattr(noema_agent, "resolve_runtime_llm_provider") - monkeypatch.setattr(noema_agent, "resolve_runtime_llm_provider", _no_provider) result = await run_noema_agent( _QueueSession([]), user_id="user-1", @@ -354,53 +486,55 @@ async def _no_provider(*args, **kwargs): prompt="hello", ) assert result.status == "unavailable" - assert result.provider_name is None + assert result.error_code == "orchestrator_gateway_unavailable" @pytest.mark.asyncio -async def test_run_agent_uses_db_provider_and_degrades_without_runtime(monkeypatch): - provider = LLMProvider( - id=7, +async def test_run_agent_uses_db_gateway_config_and_degrades_without_runtime( + monkeypatch, +): + tenant_config = TenantConfig( user_id="user-1", organization_id="org-1", - name="Local Gemma", - provider_type="ollama", - base_url="http://ollama:11434/v1", - model_identifier="gemma", - embedding_model="embeddinggemma", - api_key=None, - is_active=True, - updated_at=datetime.datetime.now(UTC), + noema_orchestrator_base_url="https://orchestrator.internal/v1", + noema_orchestrator_token="orch-token", ) - # Simulate the pydantic-ai runtime being absent: the config still resolves - # from the DB provider, and the agent degrades to a notice. + # Bypass real SSRF/DNS validation for this hermetic unit test; the fixed + # https URL above is a stand-in for whatever the tenant configured. + monkeypatch.setattr( + orchestrator_gateway, + "validate_llm_provider_base_url_async", + _pass_through_url_validator, + ) + # Simulate the pydantic-ai runtime being absent: the gateway config still + # resolves from the DB (never os.getenv), and the agent degrades to a + # notice distinct from "gateway not configured". monkeypatch.setattr(noema_agent, "_load_pydantic_ai", lambda: None) result = await run_noema_agent( - _ProviderSession([provider]), + _QueueSession([_FakeResult(scalar=tenant_config)]), user_id="user-1", organization_id="org-1", workspace_id="workspace-org-1", prompt="hello", ) assert result.status == "unavailable" - # Proves the provider name came from the DB record, not os.getenv. - assert result.provider_name is not None + assert result.error_code is None + # Proves the gateway resolved (from the DB row, not os.getenv) before the + # pydantic-ai check ran, distinguishing this from the "not configured" + # case in test_run_agent_never_resolves_a_direct_tenant_llm_provider. + assert result.provider_name == orchestrator_gateway.ORCHESTRATOR_MODEL_ALIAS assert "pydantic-ai" in (result.notice or "") @pytest.mark.asyncio async def test_build_agent_returns_none_without_runtime(monkeypatch): monkeypatch.setattr(noema_agent, "_load_pydantic_ai", lambda: None) - provider = RuntimeLLMProvider( - api_key="sk-test", - base_url=None, - chat_model="gpt-4o", - embedding_model="text-embedding-3-small", - provider_name="OpenAI", - provider_source="tenant_config", + gateway = OrchestratorGateway( + base_url="https://orchestrator.internal/v1", + inference_token="orch-token", ) - agent, closer = await build_noema_agent(provider) + agent, closer = await build_noema_agent(gateway) assert agent is None await closer() # no-op closer must be awaitable @@ -411,7 +545,7 @@ async def test_build_agent_returns_none_without_runtime(monkeypatch): @pytest.mark.asyncio -async def test_agent_runs_tools_with_test_model(): +async def test_agent_runs_tools_with_test_model(monkeypatch): # This is the ONLY test that exercises the real pydantic-ai build path # (imports OpenAIChatModel, constructs the Agent, registers the tools and # their RunContext-typed schemas). It is skipped only when pydantic-ai is @@ -421,15 +555,21 @@ async def test_agent_runs_tools_with_test_model(): from pydantic_ai import Agent as PydanticAgent from pydantic_ai.models.test import TestModel - provider = RuntimeLLMProvider( - api_key="sk-test", - base_url=None, - chat_model="gpt-4o", - embedding_model="text-embedding-3-small", - provider_name="OpenAI", - provider_source="tenant_config", + # Bypass the real SSRF-guarded/DNS-pinned HTTP client for this hermetic + # test -- the client is immediately overridden by TestModel below and + # never used to reach the network. + async def _fake_http_client(_base_url): + return "https://orchestrator.internal/v1", httpx.AsyncClient() + + monkeypatch.setattr( + noema_agent, "build_llm_provider_http_client", _fake_http_client + ) + + gateway = OrchestratorGateway( + base_url="https://orchestrator.internal/v1", + inference_token="orch-token", ) - agent, closer = await build_noema_agent(provider) + agent, closer = await build_noema_agent(gateway) # A real Agent must be built — not the graceful-degradation None. assert agent is not None assert isinstance(agent, PydanticAgent) diff --git a/backend/tests/test_owner_scope_regressions.py b/backend/tests/test_owner_scope_regressions.py new file mode 100644 index 000000000..026abbcba --- /dev/null +++ b/backend/tests/test_owner_scope_regressions.py @@ -0,0 +1,174 @@ +"""Owner-lane regressions for task authorization and Reply-SLA error routing.""" + +import datetime +from types import SimpleNamespace +from unittest.mock import AsyncMock + +import pytest +from fastapi import HTTPException +from sqlalchemy.exc import IntegrityError + +from api.auth import AuthContext +from api.tasks import ( + UpdateTicketTaskRequest, + _build_task_query, + list_ticket_tasks, + update_ticket_task, +) +from db.models import TicketTask +from services import reply_sla_escalation_service as reply_sla_service + + +class _TaskRows: + def __init__(self, rows): + self._rows = rows + + def all(self): + return self._rows + + def one_or_none(self): + if not self._rows: + return None + if len(self._rows) != 1: + raise AssertionError("expected at most one task row") + return self._rows[0] + + +class _CrossWorkspaceTaskSession: + """Return a foreign-workspace source-linked task unless SQL excludes it.""" + + def __init__(self, task: TicketTask) -> None: + self.task = task + self.commit = AsyncMock() + self.refresh = AsyncMock() + self.statement_texts: list[str] = [] + + async def execute(self, statement): + statement_text = str(statement).lower() + self.statement_texts.append(statement_text) + source_scope_enforced = ( + "ticket_tasks.email_id is null" in statement_text + and "email_records.id is not null" in statement_text + ) + return _TaskRows([] if source_scope_enforced else [(self.task, None)]) + + +@pytest.fixture +def workspace_auth() -> AuthContext: + return AuthContext( + user_id="alice", + role="member", + organization_id="org-acme", + group_ids=(), + workspace_id="workspace-a", + ) + + +@pytest.fixture +def cross_workspace_task() -> TicketTask: + now = datetime.datetime(2026, 9, 13, tzinfo=datetime.timezone.utc) + return TicketTask( + id=1, + task_uid="cross-workspace-source-task", + user_id="alice", + organization_id="org-acme", + title="foreign source task", + status="open", + priority="normal", + source_type="email", + related_email_id=991, + related_thread_id="foreign-thread", + created_at=now, + updated_at=now, + ) + + +def test_task_query_requires_authorized_source_or_no_source( + workspace_auth: AuthContext, +) -> None: + """Keep unlinked tasks while excluding links whose scoped email join vanished.""" + statement_text = str(_build_task_query(workspace_auth)).lower() + + assert "email_records.workspace_id" in statement_text + assert "ticket_tasks.email_id is null" in statement_text + assert "email_records.id is not null" in statement_text + + +@pytest.mark.asyncio +async def test_list_tasks_excludes_source_linked_task_from_other_workspace( + workspace_auth: AuthContext, + cross_workspace_task: TicketTask, +) -> None: + """Do not expose a task merely because its source-email columns were hidden.""" + database = _CrossWorkspaceTaskSession(cross_workspace_task) + + response = await list_ticket_tasks(db=database, auth_context=workspace_auth) + + assert response == [] + assert database.statement_texts + + +@pytest.mark.asyncio +async def test_update_task_rejects_source_linked_task_from_other_workspace( + workspace_auth: AuthContext, + cross_workspace_task: TicketTask, +) -> None: + """Treat a task linked to an unauthorized workspace email as not found.""" + database = _CrossWorkspaceTaskSession(cross_workspace_task) + + with pytest.raises(HTTPException) as captured: + await update_ticket_task( + "cross-workspace-source-task", + UpdateTicketTaskRequest(status="done"), + db=database, + auth_context=workspace_auth, + ) + + assert captured.value.status_code == 404 + assert cross_workspace_task.status == "open" + database.commit.assert_not_awaited() + + +@pytest.mark.asyncio +async def test_bulk_non_unique_integrity_error_never_enters_unique_conflict_recovery( + monkeypatch, +) -> None: + """Propagate FK/check failures instead of treating them as uniqueness races.""" + now = datetime.datetime.now(datetime.timezone.utc) + source_email = SimpleNamespace( + id=71, + date=now - datetime.timedelta(days=3), + ) + database = SimpleNamespace(rollback=AsyncMock()) + driver_error = RuntimeError("foreign key failure") + driver_error.sqlstate = "23503" + bulk_error = IntegrityError("insert", {}, driver_error) + + async def pending_replies(*_args, **_kwargs): + return [source_email] + + bulk_escalation = AsyncMock(side_effect=bulk_error) + fallback_escalation = AsyncMock( + side_effect=AssertionError("non-unique failure entered conflict recovery") + ) + monkeypatch.setattr(reply_sla_service, "check_missing_replies", pending_replies) + monkeypatch.setattr( + reply_sla_service, "_process_bulk_escalation", bulk_escalation + ) + monkeypatch.setattr( + reply_sla_service, "_process_fallback_escalation", fallback_escalation + ) + + with pytest.raises(IntegrityError) as captured: + await reply_sla_service.create_reply_sla_escalation_tasks( + database, + user_id="alice", + organization_id="org-acme", + workspace_id="workspace-a", + overdue_hours=48, + limit=10, + ) + + assert captured.value is bulk_error + database.rollback.assert_awaited_once() + fallback_escalation.assert_not_awaited() diff --git a/backend/tests/test_pop3_worker.py b/backend/tests/test_pop3_worker.py index d08e79f6f..b9342c830 100644 --- a/backend/tests/test_pop3_worker.py +++ b/backend/tests/test_pop3_worker.py @@ -82,7 +82,7 @@ async def test_pop3_worker_skips_disallowed_destination(): ) with patch("services.pop3_worker.poplib.POP3_SSL") as pop3_ssl: - await worker._sync_tenant(config, asyncio.Semaphore(1)) + await worker._sync_tenant(config, "workspace-pop3", asyncio.Semaphore(1)) pop3_ssl.assert_not_called() @@ -132,7 +132,12 @@ async def rollback(self): session = FakeSession() async def fake_process_fetched_email( - db_session, email_data, user_id, organization_id, owner_addresses=None + db_session, + email_data, + user_id, + organization_id, + workspace_id, + owner_addresses=None, ): imported.append( { @@ -140,6 +145,7 @@ async def fake_process_fetched_email( "email_data": email_data, "user_id": user_id, "organization_id": organization_id, + "workspace_id": workspace_id, "owner_addresses": owner_addresses, } ) @@ -162,7 +168,7 @@ async def fake_process_fetched_email( lambda host, port: pop3_client, ) - await worker._sync_tenant(config, asyncio.Semaphore(1)) + await worker._sync_tenant(config, "workspace-pop3", asyncio.Semaphore(1)) pop3_client.user.assert_called_once_with("pop3-user@example.com") pop3_client.pass_.assert_called_once_with("pop3-secret") @@ -178,3 +184,98 @@ async def fake_process_fetched_email( assert imported[0]["email_data"]["subject"] == "POP3 import" assert session.committed is True assert session.rolled_back is False + + +class _FakeTenantResult: + def __init__(self, rows): + self._rows = rows + + def scalars(self): + return self + + def all(self): + return self._rows + + +class _FakeSyncSession: + """Fakes the two queries _sync() issues in its own session: an initial + TenantConfig load, then a per-tenant distinct-workspace resolution scan + against already-imported Email rows.""" + + def __init__(self, tenant_rows, workspace_rows_by_user_id): + self._tenant_rows = tenant_rows + self._workspace_rows_by_user_id = workspace_rows_by_user_id + + async def __aenter__(self): + return self + + async def __aexit__(self, exc_type, exc, tb): + return False + + async def execute(self, _statement): + return _FakeTenantResult(self._tenant_rows) + + async def scalars(self, _statement): + rows = self._workspace_rows_by_user_id[self._tenant_rows[0].user_id] + + class _Iter: + def __iter__(self): + return iter(rows) + + return _Iter() + + +@pytest.mark.asyncio +async def test_pop3_sync_resolves_workspace_from_existing_mail(monkeypatch): + # TenantConfig has no workspace_id column at all; _sync() must derive it + # from this owner's already-imported Email rows the same way ImapSyncWorker + # does, not assume one is already attached to the loaded config (attaching + # one dynamically, as this test used to, hides that _sync() never actually + # resolved anything in production and every nonempty POP3 fetch silently + # imported zero messages). + worker = Pop3SyncWorker() + config = TenantConfig( + user_id="pop3-user", + organization_id="org-pop3", + pop3_server="pop3.example.com", + pop3_port=995, + ) + session = _FakeSyncSession([config], {"pop3-user": ["workspace-existing"]}) + monkeypatch.setattr("services.pop3_worker.AsyncSessionLocal", lambda: session) + + observed = [] + + async def fake_sync_tenant(cfg, workspace_id, _semaphore): + observed.append((cfg, workspace_id)) + + monkeypatch.setattr(worker, "_sync_tenant", fake_sync_tenant) + + await worker._sync() + + assert len(observed) == 1 + assert observed[0][0] is config + assert observed[0][1] == "workspace-existing" + + +@pytest.mark.asyncio +async def test_pop3_sync_skips_tenant_with_no_unambiguous_workspace(monkeypatch): + worker = Pop3SyncWorker() + config = TenantConfig( + user_id="pop3-user", + organization_id="org-pop3", + pop3_server="pop3.example.com", + pop3_port=995, + ) + session = _FakeSyncSession([config], {"pop3-user": []}) + monkeypatch.setattr("services.pop3_worker.AsyncSessionLocal", lambda: session) + + observed = [] + + async def fake_sync_tenant(cfg, workspace_id, _semaphore): + observed.append((cfg, workspace_id)) + + monkeypatch.setattr(worker, "_sync_tenant", fake_sync_tenant) + + await worker._sync() + + assert observed == [] diff --git a/backend/tests/test_project_graph_api.py b/backend/tests/test_project_graph_api.py index 1580f5182..22b29a9f9 100644 --- a/backend/tests/test_project_graph_api.py +++ b/backend/tests/test_project_graph_api.py @@ -1243,6 +1243,7 @@ async def _seed_source_segment( email = Email( user_id=user_id, organization_id=organization_id, + workspace_id=f"workspace-{organization_id}", message_id=f"<{uuid.uuid4().hex}@example.com>", thread_id=f"thread-{uuid.uuid4().hex}", fingerprint=f"sha256:{uuid.uuid4().hex}", diff --git a/backend/tests/test_project_graph_import_wiring.py b/backend/tests/test_project_graph_import_wiring.py index 04c201599..bbf0979ef 100644 --- a/backend/tests/test_project_graph_import_wiring.py +++ b/backend/tests/test_project_graph_import_wiring.py @@ -39,7 +39,13 @@ def test_project_source_segments_maps_content_segments(): @pytest.mark.asyncio -async def test_projection_persists_with_workspace_scope_when_objects_found(monkeypatch): +async def test_projection_persists_with_the_callers_resolved_workspace(monkeypatch): + # workspace_id is the caller's already-resolved workspace (the same value + # the imported Email row itself was stored under) -- the function must + # pass it through verbatim rather than recomputing its own default from + # organization_id/user_id, or a non-default import workspace would put + # the email and its derived project-graph objects in different + # workspaces. persist_mock = AsyncMock() monkeypatch.setattr( import_service, "persist_project_graph_projection", persist_mock @@ -50,21 +56,24 @@ async def test_projection_persists_with_workspace_scope_when_objects_found(monke ] await import_service._persist_project_graph_projection( - session, segments, user_id="user1", organization_id="org1" + session, + segments, + user_id="user1", + organization_id="org1", + workspace_id="workspace-org1", ) persist_mock.assert_awaited_once() kwargs = persist_mock.await_args.kwargs assert kwargs["user_id"] == "user1" assert kwargs["organization_id"] == "org1" - # Mirrors the scope convention enforced by the project graph repository. assert kwargs["workspace_id"] == "workspace-org1" assert kwargs["extraction"].objects # real extractor produced candidates session.commit.assert_awaited_once() @pytest.mark.asyncio -async def test_projection_falls_back_to_user_workspace_without_org(monkeypatch): +async def test_projection_passes_through_a_non_default_workspace(monkeypatch): persist_mock = AsyncMock() monkeypatch.setattr( import_service, "persist_project_graph_projection", persist_mock @@ -73,11 +82,15 @@ async def test_projection_falls_back_to_user_workspace_without_org(monkeypatch): segments = [_segment("seg1", "We must deliver the milestone by 2026-01-01.", 0)] await import_service._persist_project_graph_projection( - session, segments, user_id="user1", organization_id="" + session, + segments, + user_id="user1", + organization_id="", + workspace_id="workspace-custom-tenant", ) kwargs = persist_mock.await_args.kwargs - assert kwargs["workspace_id"] == "workspace-user1" + assert kwargs["workspace_id"] == "workspace-custom-tenant" @pytest.mark.asyncio @@ -89,7 +102,7 @@ async def test_projection_noop_when_no_segments(monkeypatch): session = AsyncMock() await import_service._persist_project_graph_projection( - session, [], user_id="u", organization_id="o" + session, [], user_id="u", organization_id="o", workspace_id="workspace-o" ) persist_mock.assert_not_awaited() @@ -107,7 +120,7 @@ async def test_projection_noop_when_no_objects_extracted(monkeypatch): segments = [_segment("seg1", "hello there, nice weather today", 0)] await import_service._persist_project_graph_projection( - session, segments, user_id="u", organization_id="o" + session, segments, user_id="u", organization_id="o", workspace_id="workspace-o" ) persist_mock.assert_not_awaited() @@ -125,7 +138,7 @@ async def test_projection_swallows_failure_and_rolls_back(monkeypatch): # Best-effort: a projection failure must not propagate to the import. await import_service._persist_project_graph_projection( - session, segments, user_id="u", organization_id="o" + session, segments, user_id="u", organization_id="o", workspace_id="workspace-o" ) session.rollback.assert_awaited_once() diff --git a/backend/tests/test_project_graph_projection.py b/backend/tests/test_project_graph_projection.py index 7598730ce..f406c06fe 100644 --- a/backend/tests/test_project_graph_projection.py +++ b/backend/tests/test_project_graph_projection.py @@ -326,6 +326,7 @@ async def _seed_source_segment( email = Email( user_id=user_id, organization_id=organization_id, + workspace_id=f"workspace-{organization_id}", message_id=f"<{uuid.uuid4().hex}@example.com>", thread_id=f"thread-{uuid.uuid4().hex}", fingerprint=f"sha256:{uuid.uuid4().hex}", diff --git a/backend/tests/test_release_governance.py b/backend/tests/test_release_governance.py index 2e197b578..0951c0acc 100644 --- a/backend/tests/test_release_governance.py +++ b/backend/tests/test_release_governance.py @@ -10,6 +10,7 @@ import json import os import re +import subprocess import sys import importlib.util from pathlib import Path @@ -665,10 +666,74 @@ def test_app_ci_runs_backend_and_frontend_checks_without_duplicate_release_pushe assert "uses: actions/setup-node@v" not in workflow push_block = workflow.split("push:", 1)[1].split("pull_request:", 1)[0] - assert "develop" in push_block assert "master" in push_block assert "release/**" not in push_block + pull_request_block = workflow.split("pull_request:", 1)[1].split("push:", 1)[0] + assert "branches:" not in pull_request_block, ( + "pull_request trigger must not exclude stacked PR base branches" + ) + + +def test_app_ci_collects_repository_root_governance_contract_tests() -> None: + """The repo-root ``tests/`` contract suite must run in CI, not only locally. + + ``tests/test_stacked_pr_workflow_contract.py`` asserts on workflow YAML but + lives outside ``backend/``, where the backend job's pytest invocation + (``cd backend && python -m pytest -q``) never collects it. Without a + dedicated step, a future trigger regression on the governed workflows + could land without CI ever running that contract. + + Parses the workflow as YAML (rather than a raw substring match) so a + comment that merely preserves the ``pytest -q tests`` text after the real + step is deleted cannot satisfy this assertion, and additionally requires + the same Timeout/Fatal/Warn/Denied output screening the backend test step + already applies, so prohibited output from this step can't slip through + Application CI unnoticed. + """ + workflow = yaml.safe_load(read_repo_text(".github/workflows/app-ci.yml")) + backend_steps = workflow["jobs"]["backend"]["steps"] + root_test_steps = [ + step for step in backend_steps if "pytest -q tests" in (step.get("run") or "") + ] + assert len(root_test_steps) == 1, ( + "app-ci.yml's backend job must have exactly one real (non-commented) " + "step running the repo-root tests/ suite" + ) + root_test_run = root_test_steps[0]["run"] + assert re.search(r"grep -qiE ['\"]timeout\|fatal\|warn\|denied['\"]", root_test_run), ( + "the repo-root tests/ step must screen its own output for " + "Timeout/Fatal/Warn/Denied, like the backend test step does" + ) + + +def test_app_ci_repository_root_governance_step_runs_clean_under_warnings_as_error() -> ( + None +): + """The repo-root ``tests/`` step must actually pass, not just be wired in. + + Live incident: the previous test confirms this step is wired into + app-ci.yml, but the step itself crashed on every real CI run with an + unhandled pytest ``INTERNALERROR`` -- the repo root had no pytest + configuration setting ``asyncio_default_fixture_loop_scope``, so + pytest-asyncio's ``pytest_configure`` hook emitted a + ``PytestDeprecationWarning`` for that unset option, and the workflow + step's ``PYTHONWARNINGS=error`` env (mirrored here) turned that warning + into a fatal exception before pytest could even collect a single test. + Reproduces the exact workflow invocation as a subprocess so this passes + only when the real CI step would too. + """ + result = subprocess.run( + [sys.executable, "-m", "pytest", "-q", "tests"], + cwd=REPO_ROOT, + env={**os.environ, "PYTHONWARNINGS": "error"}, + capture_output=True, + text=True, + ) + assert "INTERNALERROR" not in result.stdout, result.stdout + assert "INTERNALERROR" not in result.stderr, result.stderr + assert result.returncode == 0, result.stdout + result.stderr + def test_docker_publish_validates_pr_images_and_publishes_semver_images_only_on_tags() -> ( None @@ -705,8 +770,14 @@ def test_docker_publish_validates_pr_images_and_publishes_semver_images_only_on_ == 2 ) push_block = workflow.split("push:", 1)[1].split("pull_request:", 1)[0] + pull_request_block = workflow.split("pull_request:", 1)[1].split("permissions:", 1)[ + 0 + ] assert "tags:" in push_block assert "branches:" not in push_block + assert "branches:" not in pull_request_block, ( + "pull_request trigger must not exclude stacked PR base branches" + ) assert "ai_email_client-backend" in workflow assert "ai_email_client-frontend" in workflow assert workflow.count("image: naruon") == 2 @@ -1198,4 +1269,4 @@ def test_agents_records_ghcr_visibility_publication_runbook() -> None: assert "Package settings" in agents assert "Danger Zone" in agents assert "Change visibility" in normalized_agents - assert "anonymous pull/token access" in agents \ No newline at end of file + assert "anonymous pull/token access" in agents diff --git a/backend/tests/test_release_governance_output_contract.py b/backend/tests/test_release_governance_output_contract.py new file mode 100644 index 000000000..fb9b8140f --- /dev/null +++ b/backend/tests/test_release_governance_output_contract.py @@ -0,0 +1,30 @@ +"""Exact-output contract for the repository-root governance subprocess.""" + +from __future__ import annotations + +import os +import re +import subprocess +import sys +from pathlib import Path + + +REPO_ROOT = Path(__file__).resolve().parents[2] +PROHIBITED_OUTPUT = re.compile(r"timeout|fatal|warn|denied", re.IGNORECASE) + + +def test_repository_root_governance_subprocess_has_clean_output() -> None: + """Mirror Application CI's fail-closed log contract for the real root test step.""" + + result = subprocess.run( + [sys.executable, "-m", "pytest", "-q", "tests"], + cwd=REPO_ROOT, + env={**os.environ, "PYTHONWARNINGS": "error"}, + capture_output=True, + text=True, + ) + output = f"{result.stdout}\n{result.stderr}" + + assert result.returncode == 0, output + assert "INTERNALERROR" not in output, output + assert PROHIBITED_OUTPUT.search(output) is None, output diff --git a/backend/tests/test_reply_sla_batch_conflict_owner_integration.py b/backend/tests/test_reply_sla_batch_conflict_owner_integration.py new file mode 100644 index 000000000..09020911f --- /dev/null +++ b/backend/tests/test_reply_sla_batch_conflict_owner_integration.py @@ -0,0 +1,139 @@ +"""Owner-integration regressions for bounded Reply-SLA conflict recovery.""" + +import datetime +import inspect +from contextlib import asynccontextmanager, nullcontext +from types import SimpleNamespace +from unittest.mock import AsyncMock + +import pytest +from sqlalchemy.exc import IntegrityError + +from services import reply_sla_escalation_service as service + +NOW = datetime.datetime(2026, 9, 12, 10, tzinfo=datetime.timezone.utc) + + +def _mail(email_id: int) -> SimpleNamespace: + return SimpleNamespace( + id=email_id, + user_id="alice", + organization_id="org-a", + workspace_id="workspace-a", + message_id=f"mail-{email_id}", + thread_id=f"thread-{email_id}", + subject=f"subject-{email_id}", + date=NOW - datetime.timedelta(days=3), + ) + + +def _task(email_id: int, *, task_uid: str) -> SimpleNamespace: + return SimpleNamespace( + task_uid=task_uid, + related_email_id=email_id, + status="open", + priority="normal", + title="old title", + related_thread_id=f"old-thread-{email_id}", + updated_at=NOW - datetime.timedelta(days=1), + ) + + +class _AlwaysUniqueConflictSession: + """Count SAVEPOINT/flush attempts while every insert loses a unique race.""" + + def __init__(self) -> None: + self.flush_count = 0 + self.savepoint_count = 0 + self.commit = AsyncMock() + self.rollback = AsyncMock() + self.no_autoflush = nullcontext() + + def add(self, _task_record) -> None: + return None + + async def flush(self) -> None: + self.flush_count += 1 + driver_error = RuntimeError("duplicate") + driver_error.sqlstate = "23505" + raise IntegrityError("insert", {}, driver_error) + + @asynccontextmanager + async def begin_nested(self): + self.savepoint_count += 1 + yield + + +@pytest.mark.asyncio +async def test_repeated_unique_conflicts_never_fall_back_to_per_row_savepoints( + monkeypatch, +): + """Exhaust contention after three batch attempts instead of N row retries.""" + mails = [_mail(email_id) for email_id in range(1, 5)] + winners = {email.id: _task(email.id, task_uid=f"winner-{email.id}") for email in mails} + visibility = [ + {}, + {1: winners[1]}, + {2: winners[2]}, + {3: winners[3]}, + ] + reads = 0 + + async def fetch_existing(_db, _user_id, _organization_id, _email_ids): + nonlocal reads + result = visibility[min(reads, len(visibility) - 1)] + reads += 1 + return result + + def create_task(_user_id, _organization_id, email): + return _task(email.id, task_uid=f"candidate-{email.id}") + + def update_task(task, email, now): + if task.status != "done": + task.status = "blocked" + task.priority = "urgent" + task.title = f"follow up: {email.subject}" + task.updated_at = now + + monkeypatch.setattr(service, "_fetch_existing_tasks_by_email", fetch_existing) + monkeypatch.setattr(service, "_create_task_for_escalation", create_task) + monkeypatch.setattr(service, "_update_task_for_escalation", update_task) + + database = _AlwaysUniqueConflictSession() + with pytest.raises(service.ReplySlaTaskConflict) as captured: + await service._process_fallback_escalation( + database, + "alice", + "org-a", + mails, + NOW, + ) + + assert captured.value.error_code == "reply_sla_batch_retry_exhausted" + assert database.flush_count == 3 + assert database.savepoint_count == 3 + database.rollback.assert_awaited_once() + database.commit.assert_not_awaited() + + +def test_conflict_exposes_machine_readable_error_code() -> None: + """Keep HTTP conflict classification independent of localized prose.""" + conflict = service.ReplySlaTaskConflict( + "reply_sla_task_conflict", + "concurrent winner not visible", + ) + assert conflict.error_code == "reply_sla_task_conflict" + assert str(conflict) == "concurrent winner not visible" + + +def test_rollback_reload_contract_includes_workspace_scope() -> None: + """Preserve #1486 workspace ownership when batching expired-mail reloads.""" + reload_overdue_replies = getattr(service, "_reload_overdue_replies") + parameters = inspect.signature(reload_overdue_replies).parameters + assert tuple(parameters) == ( + "db", + "user_id", + "organization_id", + "workspace_id", + "email_ids", + ) diff --git a/backend/tests/test_reply_sla_escalation_edges.py b/backend/tests/test_reply_sla_escalation_edges.py new file mode 100644 index 000000000..937dcec70 --- /dev/null +++ b/backend/tests/test_reply_sla_escalation_edges.py @@ -0,0 +1,381 @@ +"""Unit edge contracts; scripted sessions do not model PostgreSQL races.""" + +import datetime +from contextlib import asynccontextmanager +from types import SimpleNamespace +from unittest.mock import AsyncMock + +import pytest +from sqlalchemy.exc import IntegrityError + +from db.models import Email, TicketTask +from services import reply_sla_escalation_service as escalation_service + +pytestmark = pytest.mark.asyncio +CURRENT_TIME = datetime.datetime(2026, 9, 5, 12, tzinfo=datetime.timezone.utc) + + +def _email_record(email_id, *, date=CURRENT_TIME - datetime.timedelta(days=3)): + return Email( + id=email_id, + user_id="alice", + organization_id="org-acme", + workspace_id="workspace-acme", + message_id=f"", + thread_id=f"", + subject=f"Reply {email_id}", + date=date, + ) + + +def _task_record(email_record, *, task_uid="existing-task_record", status="open"): + return TicketTask( + task_uid=task_uid, + user_id="alice", + organization_id="org-acme", + title="Manual follow-up", + status=status, + priority="normal", + source_type="reply_sla", + related_email_id=email_record.id, + related_thread_id="manual-thread", + created_at=CURRENT_TIME - datetime.timedelta(days=2), + updated_at=CURRENT_TIME - datetime.timedelta(days=1), + ) + + +class TaskSession: + """Script query visibility and discard staged inserts on savepoint failure.""" + + def __init__(self, row_batches, *, flush_failures=(), fail_first_commit=False): + self.pending = [] + self.committed = [] + self.fail_first_commit = fail_first_commit + results = [ + SimpleNamespace(scalars=lambda rows=rows: SimpleNamespace(all=lambda: rows)) + for rows in row_batches + ] + self.execute = AsyncMock(side_effect=results) + self.flush = AsyncMock( + side_effect=[ + IntegrityError("concurrent source task_record", {}, None) + if fail + else None + for fail in flush_failures + ] + ) + self.commit = AsyncMock(side_effect=self._commit) + self.rollback = AsyncMock(side_effect=self.pending.clear) + self.refresh = AsyncMock() + + def add(self, task_record): + self.pending.append(task_record) + + async def _commit(self): + if self.fail_first_commit: + self.fail_first_commit = False + raise IntegrityError("concurrent source task_record", {}, None) + self.committed.extend(self.pending) + self.pending.clear() + + @asynccontextmanager + async def begin_nested(self): + staged_before = list(self.pending) + try: + yield + except IntegrityError: + self.pending[:] = staged_before + raise + + +@pytest.fixture +def fixed_clock(monkeypatch): + class FixedDatetime(datetime.datetime): + @classmethod + def now(cls, tz): + return CURRENT_TIME.astimezone(tz) + + monkeypatch.setattr( + escalation_service, + "datetime", + SimpleNamespace( + datetime=FixedDatetime, + timedelta=datetime.timedelta, + timezone=datetime.timezone, + ), + ) + + +async def test_duplicate_existing_source_updates_first_task_only(): + email_record = _email_record(11) + first_record = _task_record(email_record, task_uid="first_record-task_record") + duplicate_record = _task_record(email_record, task_uid="older_record-task_record") + duplicate_record.updated_at = CURRENT_TIME - datetime.timedelta(days=2) + database_session = TaskSession([[first_record, duplicate_record]]) + + created_count, task_records = await escalation_service._process_bulk_escalation( + database_session, "alice", "org-acme", [email_record], CURRENT_TIME + ) + + assert created_count == 0 + assert task_records == [(first_record, "")] + assert (first_record.status, first_record.priority) == ("blocked", "urgent") + assert first_record.related_thread_id == "thread-11@example.com" + assert first_record.updated_at == CURRENT_TIME + assert ( + duplicate_record.title, + duplicate_record.status, + duplicate_record.priority, + ) == ( + "Manual follow-up", + "open", + "normal", + ) + assert duplicate_record.related_thread_id == "manual-thread" + assert duplicate_record.updated_at == CURRENT_TIME - datetime.timedelta(days=2) + assert database_session.committed == [] + + +@pytest.mark.parametrize( + "process_batch", + [ + escalation_service._process_bulk_escalation, + escalation_service._process_fallback_escalation, + ], + ids=["bulk", "fallback"], +) +async def test_completed_task_keeps_user_edits_and_completion(process_batch): + email_record = _email_record(12) + completed_record = _task_record(email_record, status="done") + database_session = TaskSession([[completed_record]]) + + created_count, task_records = await process_batch( + database_session, "alice", "org-acme", [email_record], CURRENT_TIME + ) + + assert created_count == 0 + assert task_records == [(completed_record, "")] + assert ( + completed_record.title, + completed_record.status, + completed_record.priority, + ) == ( + "Manual follow-up", + "done", + "normal", + ) + assert completed_record.related_thread_id == "manual-thread" + assert completed_record.created_at == CURRENT_TIME - datetime.timedelta(days=2) + assert completed_record.updated_at == CURRENT_TIME - datetime.timedelta(days=1) + assert database_session.committed == [] + + +@pytest.mark.parametrize( + "process_batch", + [ + escalation_service._process_bulk_escalation, + escalation_service._process_fallback_escalation, + ], + ids=["bulk", "fallback"], +) +async def test_empty_batch_does_not_commit(process_batch): + database_session = TaskSession([[]]) + + assert await process_batch( + database_session, "alice", "org-acme", [], CURRENT_TIME + ) == (0, []) + database_session.commit.assert_not_awaited() + assert database_session.pending == database_session.committed == [] + + +@pytest.mark.parametrize("recent_count", [0, 1], ids=["empty", "not-overdue"]) +async def test_no_overdue_mail_returns_evaluation_without_writes( + monkeypatch, fixed_clock, recent_count +): + pending = [_email_record(13, date=CURRENT_TIME)] if recent_count else [] + monkeypatch.setattr( + escalation_service, "check_missing_replies", AsyncMock(return_value=pending) + ) + database_session = TaskSession([]) + + escalation_result = await escalation_service.create_reply_sla_escalation_tasks( + database_session, + user_id="alice", + organization_id="org-acme", + workspace_id="workspace-acme", + overdue_hours=48, + limit=10, + ) + + assert ( + escalation_result.evaluated, + escalation_result.created, + escalation_result.overdue_hours, + ) == ( + recent_count, + 0, + 48, + ) + assert escalation_result.tasks == [] + database_session.execute.assert_not_awaited() + database_session.commit.assert_not_awaited() + + +async def test_naive_utc_deadline_is_inclusive_and_sorted_with_aware_mail( + monkeypatch, fixed_clock +): + boundary_record = _email_record(21, date=datetime.datetime(2026, 9, 3, 12)) + recent_record = _email_record(22, date=datetime.datetime(2026, 9, 3, 12, 0, 1)) + older_record = _email_record( + 23, + date=datetime.datetime( + 2026, 9, 3, 20, tzinfo=datetime.timezone(datetime.timedelta(hours=9)) + ), + ) + monkeypatch.setattr( + escalation_service, + "check_missing_replies", + AsyncMock(return_value=[boundary_record, recent_record, older_record]), + ) + persisted_boundary = _task_record( + boundary_record, task_uid="boundary_record-task_record", status="done" + ) + persisted_older = _task_record( + older_record, task_uid="older_record-task_record", status="done" + ) + database_session = TaskSession([[persisted_boundary, persisted_older]]) + + escalation_result = await escalation_service.create_reply_sla_escalation_tasks( + database_session, + user_id="alice", + organization_id="org-acme", + workspace_id="workspace-acme", + overdue_hours=48, + limit=10, + ) + + assert (escalation_result.evaluated, escalation_result.created) == (3, 0) + assert [task_entry.source_email_id for task_entry in escalation_result.tasks] == [ + "", + "", + ] + assert [task_entry.task.task_uid for task_entry in escalation_result.tasks] == [ + "older_record-task_record", + "boundary_record-task_record", + ] + assert boundary_record.date == datetime.datetime(2026, 9, 3, 12) + + +async def test_missing_refresh_row_preserves_task_and_original_source_reference(): + first_record, second_record = _email_record(31), _email_record(32) + stale_record = _task_record(first_record, task_uid="stale_record-task_record") + refreshed_record = _task_record(first_record, task_uid="persisted-task_record") + retained_record = _task_record( + second_record, task_uid="retained_record-task_record" + ) + task_records = [(stale_record, ""), (retained_record, None)] + database_session = TaskSession([[refreshed_record]]) + + await escalation_service._refresh_escalated_tasks( + database_session, "alice", "org-acme", [31, 32], task_records + ) + + assert task_records == [ + (refreshed_record, ""), + (retained_record, None), + ] + database_session.commit.assert_not_awaited() + + +@pytest.mark.parametrize("late_visibility", [False, True], ids=["batch", "individual"]) +async def test_all_insert_conflicts_reuse_winners_without_counting_failed_inserts( + late_visibility, +): + email_records = [_email_record(41), _email_record(42)] + winner_records = [ + _task_record(email_records[0], task_uid="winner_record-first_record"), + _task_record(email_records[1], task_uid="winner_record-second_record"), + ] + database_session = TaskSession( + [[], [] if late_visibility else winner_records, winner_records, winner_records], + flush_failures=[True, True, True, True] if late_visibility else [True], + ) + + created_count, task_records = await escalation_service._process_fallback_escalation( + database_session, "alice", "org-acme", email_records, CURRENT_TIME + ) + + assert created_count == 0 + assert task_records == [ + (winner_records[0], ""), + (winner_records[1], ""), + ] + assert [ + (task_record.status, task_record.priority) for task_record, _ in task_records + ] == [ + ("blocked", "urgent"), + ("blocked", "urgent"), + ] + assert [task_record.related_thread_id for task_record, _ in task_records] == [ + "thread-41@example.com", + "thread-42@example.com", + ] + assert database_session.pending == database_session.committed == [] + database_session.commit.assert_awaited_once() + + +async def test_late_individual_conflict_preserves_successful_sibling_insert( + monkeypatch, fixed_clock +): + first_record, second_record = _email_record(51), _email_record(52) + winner_record = _task_record(second_record, task_uid="late-winner_record") + monkeypatch.setattr( + escalation_service, + "check_missing_replies", + AsyncMock(return_value=[first_record, second_record]), + ) + database_session = TaskSession( + [[], [], [], [winner_record], [winner_record]], + flush_failures=[True, True, False, True], + fail_first_commit=True, + ) + + escalation_result = await escalation_service.create_reply_sla_escalation_tasks( + database_session, + user_id="alice", + organization_id="org-acme", + workspace_id="workspace-acme", + overdue_hours=48, + limit=10, + ) + + assert (escalation_result.evaluated, escalation_result.created) == (2, 1) + assert [task_entry.source_email_id for task_entry in escalation_result.tasks] == [ + "", + "", + ] + [inserted_record] = database_session.committed + assert escalation_result.tasks[0].task is inserted_record + assert escalation_result.tasks[1].task is winner_record + assert ( + inserted_record.user_id, + inserted_record.organization_id, + inserted_record.source_type, + ) == ( + "alice", + "org-acme", + "reply_sla", + ) + assert (inserted_record.related_email_id, inserted_record.related_thread_id) == ( + 51, + "thread-51@example.com", + ) + assert [ + (task_entry.task.status, task_entry.task.priority) + for task_entry in escalation_result.tasks + ] == [ + ("blocked", "urgent"), + ("blocked", "urgent"), + ] + assert database_session.pending == [] + database_session.rollback.assert_awaited_once() diff --git a/backend/tests/test_reply_sla_scheduler.py b/backend/tests/test_reply_sla_scheduler.py index 959ae83eb..99739624b 100644 --- a/backend/tests/test_reply_sla_scheduler.py +++ b/backend/tests/test_reply_sla_scheduler.py @@ -1,4 +1,5 @@ import asyncio +from types import SimpleNamespace import pytest @@ -6,30 +7,60 @@ from services.reply_sla_scheduler import ReplySlaScheduler, _sysrand +@pytest.fixture(autouse=True) +def lease_connection(monkeypatch): + """Keep fast unit tests separate from the real PostgreSQL lease tests.""" + + class LeaseConnection: + """Record whether a failed sweep retires its physical connection.""" + + invalidated = False + + async def __aenter__(self): + return self + + async def __aexit__(self, *_args): + return False + + async def invalidate(self): + self.invalidated = True + + connection = LeaseConnection() + monkeypatch.setattr( + "services.reply_sla_scheduler.engine", + SimpleNamespace(connect=lambda: connection), + ) + return connection + + @pytest.mark.asyncio async def test_reply_sla_scheduler_escalates_configured_mailbox_owners(monkeypatch): calls: list[dict[str, object]] = [] class MockScalars: def all(self): - return [ - TenantConfig( + return [1, 2] + + class MockResult: + def scalars(self): + return MockScalars() + + class MockSession: + async def get(self, record_type, config_id, *, populate_existing=False): + assert record_type is TenantConfig + return { + 1: TenantConfig( user_id="alice", organization_id="org-acme", smtp_username="alice@example.com", ), - TenantConfig( + 2: TenantConfig( user_id="bob", organization_id="org-beta", imap_username="bob@example.com", ), - ] + }[config_id] - class MockResult: - def scalars(self): - return MockScalars() - - class MockSession: async def __aenter__(self): return self @@ -40,16 +71,27 @@ async def execute(self, stmt): self.statement = stmt return MockResult() + async def scalars(self, stmt): + return ["workspace-a"] + session = MockSession() async def fake_create_reply_sla_escalation_tasks( - db, *, user_id, organization_id, overdue_hours, limit, tenant_config + db, + *, + user_id, + organization_id, + workspace_id, + overdue_hours, + limit, + tenant_config, ): calls.append( { "db": db, "user_id": user_id, "organization_id": organization_id, + "workspace_id": workspace_id, "overdue_hours": overdue_hours, "limit": limit, "tenant_config": tenant_config, @@ -58,7 +100,7 @@ async def fake_create_reply_sla_escalation_tasks( monkeypatch.setattr( "services.reply_sla_scheduler.AsyncSessionLocal", - lambda: session, + lambda *, bind: session, ) monkeypatch.setattr( "services.reply_sla_scheduler.create_reply_sla_escalation_tasks", @@ -74,6 +116,7 @@ async def fake_create_reply_sla_escalation_tasks( "db": session, "user_id": "alice", "organization_id": "org-acme", + "workspace_id": "workspace-a", "overdue_hours": 24, "limit": 7, "tenant_config": calls[0]["tenant_config"], @@ -82,6 +125,7 @@ async def fake_create_reply_sla_escalation_tasks( "db": session, "user_id": "bob", "organization_id": "org-beta", + "workspace_id": "workspace-a", "overdue_hours": 24, "limit": 7, "tenant_config": calls[1]["tenant_config"], @@ -99,16 +143,23 @@ async def test_reply_sla_scheduler_continues_after_owner_escalation_failure( class MockScalars: def all(self): - return [ - TenantConfig(user_id="alice", organization_id="org-acme"), - TenantConfig(user_id="bob", organization_id="org-beta"), - ] + return [1, 2] class MockResult: def scalars(self): return MockScalars() class MockSession: + async def get(self, record_type, config_id, *, populate_existing=False): + assert record_type is TenantConfig + return { + 1: TenantConfig(user_id="alice", organization_id="org-acme"), + 2: TenantConfig(user_id="bob", organization_id="org-beta"), + }[config_id] + + async def rollback(self): + return None + async def __aenter__(self): return self @@ -118,8 +169,18 @@ async def __aexit__(self, exc_type, exc, tb): async def execute(self, stmt): return MockResult() + async def scalars(self, stmt): + return ["workspace-a"] + async def fake_create_reply_sla_escalation_tasks( - db, *, user_id, organization_id, overdue_hours, limit, tenant_config + db, + *, + user_id, + organization_id, + workspace_id, + overdue_hours, + limit, + tenant_config, ): calls.append(user_id) if user_id == "alice": @@ -127,7 +188,7 @@ async def fake_create_reply_sla_escalation_tasks( monkeypatch.setattr( "services.reply_sla_scheduler.AsyncSessionLocal", - lambda: MockSession(), + lambda *, bind: MockSession(), ) monkeypatch.setattr( "services.reply_sla_scheduler.create_reply_sla_escalation_tasks", @@ -179,6 +240,9 @@ async def __aenter__(self): async def __aexit__(self, exc_type, exc, tb): return False + async def rollback(self): + return None + async def scalar(self, stmt, params=None): self.scalar_calls.append(stmt) if len(self.scalar_calls) == 1: @@ -203,7 +267,7 @@ def all(self): async def test_sync_skips_sweep_when_lease_held_elsewhere(monkeypatch): session = _FakePostgresSession(lease_acquired=False) monkeypatch.setattr( - "services.reply_sla_scheduler.AsyncSessionLocal", lambda: session + "services.reply_sla_scheduler.AsyncSessionLocal", lambda *, bind: session ) scheduler = ReplySlaScheduler() @@ -217,7 +281,7 @@ async def test_sync_skips_sweep_when_lease_held_elsewhere(monkeypatch): async def test_sync_sweeps_and_releases_lease_when_acquired(monkeypatch): session = _FakePostgresSession(lease_acquired=True) monkeypatch.setattr( - "services.reply_sla_scheduler.AsyncSessionLocal", lambda: session + "services.reply_sla_scheduler.AsyncSessionLocal", lambda *, bind: session ) scheduler = ReplySlaScheduler() @@ -228,14 +292,16 @@ async def test_sync_sweeps_and_releases_lease_when_acquired(monkeypatch): @pytest.mark.asyncio -async def test_sync_releases_lease_even_when_sweep_fails(monkeypatch): +async def test_sync_retires_lease_connection_when_sweep_fails( + monkeypatch, lease_connection +): session = _FakePostgresSession(lease_acquired=True) async def boom(self, _session): raise RuntimeError("sweep failed") monkeypatch.setattr( - "services.reply_sla_scheduler.AsyncSessionLocal", lambda: session + "services.reply_sla_scheduler.AsyncSessionLocal", lambda *, bind: session ) monkeypatch.setattr( ReplySlaScheduler, "_sweep_configured_owners", boom, raising=True @@ -245,7 +311,8 @@ async def boom(self, _session): with pytest.raises(RuntimeError): await scheduler._sync() - assert len(session.scalar_calls) == 2 # unlock still happened + assert len(session.scalar_calls) == 1 + assert lease_connection.invalidated is True @pytest.mark.asyncio @@ -398,3 +465,88 @@ async def flaky_sync(): await scheduler.stop() assert scheduler._is_running is False + + +@pytest.mark.asyncio +@pytest.mark.parametrize("unlock_result", [False, None, 1, "true"]) +async def test_unconfirmed_unlock_retires_the_connection( + monkeypatch, lease_connection, unlock_result +): + """False and truthy non-boolean replies cannot admit a connection back to the pool.""" + session = _FakePostgresSession(lease_acquired=True) + + async def scalar_result(statement, params=None): + session.scalar_calls.append(statement) + return True if len(session.scalar_calls) == 1 else unlock_result + + monkeypatch.setattr(session, "scalar", scalar_result) + monkeypatch.setattr( + "services.reply_sla_scheduler.AsyncSessionLocal", lambda *, bind: session + ) + with pytest.raises(RuntimeError, match="release was not confirmed"): + await ReplySlaScheduler()._sync() + assert lease_connection.invalidated is True + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "delete_phase", ["before_workspaces", "before_first_workspace"] +) +async def test_deleted_owner_is_not_escalated(monkeypatch, delete_phase): + """A configuration deleted after selection must not reach escalation.""" + session = _FakePostgresSession(lease_acquired=True) + lookup_count = 0 + + async def selected_owners(statement): + return SimpleNamespace(scalars=lambda: SimpleNamespace(all=lambda: [42])) + + async def deleted_owner(record_type, record_key, *, populate_existing=False): + nonlocal lookup_count + lookup_count += 1 + assert record_type is TenantConfig and record_key == 42 + if delete_phase == "before_first_workspace" and lookup_count == 1: + return TenantConfig(user_id="owner_scope", organization_id="tenant_scope") + return None + + async def selected_workspaces(statement): + return ["workspace_scope"] + + async def unexpected_escalation(*args, **kwargs): + pytest.fail("Deleted owner reached escalation.") + + monkeypatch.setattr(session, "execute", selected_owners) + monkeypatch.setattr(session, "get", deleted_owner, raising=False) + monkeypatch.setattr(session, "scalars", selected_workspaces, raising=False) + monkeypatch.setattr( + "services.reply_sla_scheduler.AsyncSessionLocal", lambda *, bind: session + ) + monkeypatch.setattr( + "services.reply_sla_scheduler.create_reply_sla_escalation_tasks", + unexpected_escalation, + ) + await ReplySlaScheduler()._sync() + assert len(session.scalar_calls) == 2 + + +@pytest.mark.asyncio +async def test_stop_handles_running_state_without_task(): + """Stopping an incompletely started scheduler clears its running state.""" + scheduler = ReplySlaScheduler() + scheduler._is_running = True + await scheduler.stop() + assert scheduler._is_running is False + + +@pytest.mark.asyncio +async def test_loop_exits_without_sleep_after_sweep_clears_running(monkeypatch): + """A sweep that stops the scheduler must not park for another interval.""" + scheduler = ReplySlaScheduler() + scheduler._is_running = True + monkeypatch.setattr(_sysrand, "uniform", lambda *_args: 0) + + async def stopping_sweep(): + scheduler._is_running = False + + monkeypatch.setattr(scheduler, "_sync", stopping_sweep) + await scheduler._run_loop() + assert scheduler._is_running is False diff --git a/backend/tests/test_reply_sla_scheduler_owner_load_regression.py b/backend/tests/test_reply_sla_scheduler_owner_load_regression.py new file mode 100644 index 000000000..918c7a401 --- /dev/null +++ b/backend/tests/test_reply_sla_scheduler_owner_load_regression.py @@ -0,0 +1,40 @@ +"""Regression coverage for owner-local Reply SLA config-load failures.""" + +import pytest + +from services.reply_sla_scheduler import ReplySlaScheduler + + +class _ScalarIds: + def all(self): + return [17] + + +class _ConfigIdsResult: + def scalars(self): + return _ScalarIds() + + +class _RecoverableOwnerLoadFailureSession: + def __init__(self) -> None: + self.rollback_count = 0 + + async def execute(self, _statement): + return _ConfigIdsResult() + + async def get(self, *_args, **_kwargs): + raise RuntimeError("owner config refresh failed") + + async def rollback(self) -> None: + self.rollback_count += 1 + + +@pytest.mark.asyncio +async def test_owner_config_load_failure_is_isolated_to_that_owner() -> None: + """A recoverable config refresh failure must roll back and end the owner lane.""" + session = _RecoverableOwnerLoadFailureSession() + scheduler = ReplySlaScheduler() + + await scheduler._sweep_configured_owners(session) + + assert session.rollback_count == 1 diff --git a/backend/tests/test_reply_sla_scheduler_postgres.py b/backend/tests/test_reply_sla_scheduler_postgres.py new file mode 100644 index 000000000..b7144f102 --- /dev/null +++ b/backend/tests/test_reply_sla_scheduler_postgres.py @@ -0,0 +1,564 @@ +"""Migrated PostgreSQL regression for scheduler connection ownership. + +The mail excerpt is from the public 2014-04-27 pgsql-general queue question: +https://www.postgresql.org/message-id/CANsFX049q7C_vJAtn2BSJy_4hQPu0%3DJNtv-Lyzb%3DgbZu-be30A%40mail.gmail.com +Its question and UTC date are preserved; names, addresses, subject and message +identity are anonymized. The clock is replayed three days later, not the mail +date rewritten. Isolated scope IDs and fault injection are test controls, not +customer workload or reply-classification accuracy evidence. + +Run scripts/migrate_db.py against an isolated PostgreSQL before this test. +No metadata-created substitute schema or unavailable-database skip is used. +""" + +import asyncio +import datetime as date_time +from email import policy +from email.parser import BytesParser +from email.utils import parsedate_to_datetime +from pathlib import Path +from types import SimpleNamespace +from uuid import uuid4 + +import pytest +from alembic.script import ScriptDirectory +from sqlalchemy import delete, select, text +from sqlalchemy.exc import DBAPIError +from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker, create_async_engine + +from core.config import settings +from db.models import Email, TenantConfig, TicketTask +from scripts.migrate_db import alembic_config +from services import reply_sla_escalation_service, reply_sla_scheduler +from services import reply_tracking_service + +pytestmark = [pytest.mark.postgres, pytest.mark.asyncio] + + +class ReplayDateTime(date_time.datetime): + """Keep the recorded mail in its original seven-day tracking window.""" + + @classmethod + def now(cls, tz=None): + """Return the fixed replay instant in the requested timezone.""" + replay_time = cls(2014, 4, 30, 19, 31, 42, tzinfo=date_time.timezone.utc) + return replay_time.astimezone(tz) if tz else replay_time.replace(tzinfo=None) + + +async def _seed_observed_mail(session_factory, owner_key, workspace_key=None): + """Persist an anonymized observed question using current ORM mappings.""" + mail_record = BytesParser(policy=policy.default).parsebytes( + (Path(__file__).parent / "fixtures/reply_sla_observed_message.eml").read_bytes() + ) + async with session_factory() as database_session: + assert ( + await database_session.scalar( + text("SELECT version_num FROM alembic_version") + ) + == ScriptDirectory.from_config(alembic_config()).get_current_head() + ) + if ( + await database_session.scalar( + select(TenantConfig.id).where( + TenantConfig.user_id == owner_key, + TenantConfig.organization_id == owner_key, + ) + ) + is None + ): + database_session.add( + TenantConfig( + user_id=owner_key, + organization_id=owner_key, + smtp_username="archive-author@example.invalid", + ) + ) + email_record = Email( + user_id=owner_key, + organization_id=owner_key, + workspace_id=workspace_key or owner_key, + message_id=str(mail_record["Message-ID"]), + sender=str(mail_record["From"]), + recipients=str(mail_record["To"]), + subject=str(mail_record["Subject"]), + date=parsedate_to_datetime(str(mail_record["Date"])), + body=mail_record.get_content(), + ) + database_session.add(email_record) + await database_session.commit() + return email_record.id + + +async def _clear_observed_mail(session_factory, owner_key): + """Remove only this test's related tasks, email and tenant configuration.""" + async with session_factory() as database_session: + for record_type in (TicketTask, Email, TenantConfig): + await database_session.execute( + delete(record_type).where( + record_type.user_id == owner_key, + record_type.organization_id == owner_key, + ) + ) + await database_session.commit() + + +async def test_real_escalation_commit_does_not_lend_lease_to_pool_reader(monkeypatch): + """A task commit must not lend its still-held lease to another request.""" + worker_engine = create_async_engine( + settings.DATABASE_URL, pool_size=2, max_overflow=0 + ) + replica_engine = create_async_engine( + settings.DATABASE_URL, pool_size=1, max_overflow=0 + ) + seed_sessions = async_sessionmaker(worker_engine, expire_on_commit=False) + replica_sessions = async_sessionmaker(replica_engine, expire_on_commit=False) + owner_key = f"lease_scope_{uuid4().hex}" + reader_connections = [] + worker_pids = [] + competing_lease_results = [] + + class InterleavedSession(AsyncSession): + """Pause only after real writes to let an unrelated pool reader enter.""" + + async def commit(self): + """Keep actual commit semantics, then borrow a competing connection.""" + worker_pids.append(await self.scalar(text("SELECT pg_backend_pid()"))) + await super().commit() + if not reader_connections: + reader_connections.append(await worker_engine.connect()) + async with replica_sessions() as replica_session: + can_lead = await reply_sla_scheduler._try_acquire_sweep_lease( + replica_session + ) + competing_lease_results.append(can_lead) + if can_lead: + await reply_sla_scheduler._release_sweep_lease(replica_session) + + replay_clock = SimpleNamespace( + datetime=ReplayDateTime, + timedelta=date_time.timedelta, + timezone=date_time.timezone, + ) + monkeypatch.setattr(reply_sla_escalation_service, "datetime", replay_clock) + monkeypatch.setattr(reply_tracking_service, "datetime", replay_clock) + monkeypatch.setattr(reply_sla_scheduler, "engine", worker_engine, raising=False) + monkeypatch.setattr( + reply_sla_scheduler, + "AsyncSessionLocal", + async_sessionmaker( + worker_engine, class_=InterleavedSession, expire_on_commit=False + ), + ) + try: + source_email_id = await _seed_observed_mail(seed_sessions, owner_key) + await reply_sla_scheduler.ReplySlaScheduler()._sync() + assert competing_lease_results == [False] + async with replica_sessions() as replica_session: + task_record = await replica_session.scalar( + select(TicketTask).where(TicketTask.user_id == owner_key) + ) + assert task_record is not None + assert task_record.related_email_id == source_email_id + assert task_record.status == "blocked" + assert task_record.priority == "urgent" + assert task_record.task_uid + replica_can_lead = await reply_sla_scheduler._try_acquire_sweep_lease( + replica_session + ) + if replica_can_lead: + await reply_sla_scheduler._release_sweep_lease(replica_session) + reader_pid = await reader_connections[0].scalar( + text("SELECT pg_backend_pid()") + ) + assert replica_can_lead is True, ( + f"completed scheduler stranded its lease: worker={worker_pids}, reader={reader_pid}" + ) + assert reader_pid not in worker_pids + finally: + for reader_connection in reader_connections: + await reader_connection.invalidate() + await reader_connection.close() + await _clear_observed_mail(seed_sessions, owner_key) + await worker_engine.dispose() + await replica_engine.dispose() + + +@pytest.mark.parametrize("conflict_path", ["scheduler_bulk", "service_savepoint"]) +async def test_concurrent_task_creation_preserves_both_source_workspaces( + monkeypatch, conflict_path +): + """A manual writer racing a sweep must not defeat conflict recovery or skip its next workspace.""" + worker_engine = create_async_engine( + settings.DATABASE_URL, pool_size=1, max_overflow=0 + ) + replica_engine = create_async_engine( + settings.DATABASE_URL, pool_size=1, max_overflow=0 + ) + worker_sessions = async_sessionmaker(worker_engine, expire_on_commit=False) + replica_sessions = async_sessionmaker(replica_engine, expire_on_commit=False) + owner_key = f"lease_scope_{uuid4().hex}" + source_email_ids = [] + competing_task_uid = uuid4().hex + competing_email_ids = [] + fetch_existing = reply_sla_escalation_service._fetch_existing_tasks_by_email + replay_clock = SimpleNamespace( + datetime=ReplayDateTime, + timedelta=date_time.timedelta, + timezone=date_time.timezone, + ) + monkeypatch.setattr(reply_sla_escalation_service, "datetime", replay_clock) + monkeypatch.setattr(reply_tracking_service, "datetime", replay_clock) + monkeypatch.setattr(reply_sla_scheduler, "engine", worker_engine, raising=False) + monkeypatch.setattr(reply_sla_scheduler, "AsyncSessionLocal", worker_sessions) + + async def race_after_lookup(database_session, user_id, organization_id, email_ids): + """Commit a competing task after the real initial lookup saw no task.""" + task_records = await fetch_existing( + database_session, user_id, organization_id, email_ids + ) + if not competing_email_ids: + competing_email_ids.append(email_ids[0]) + async with replica_sessions() as replica_session: + replica_session.add( + TicketTask( + task_uid=competing_task_uid, + user_id=user_id, + organization_id=organization_id, + title="Existing follow-up", + status="open", + priority="normal", + source_type="reply_sla", + related_email_id=email_ids[0], + ) + ) + await replica_session.commit() + return task_records + + monkeypatch.setattr( + reply_sla_escalation_service, + "_fetch_existing_tasks_by_email", + race_after_lookup, + ) + try: + for workspace_suffix in ("first", "second"): + source_email_ids.append( + await _seed_observed_mail( + worker_sessions, owner_key, f"{owner_key}_{workspace_suffix}" + ) + ) + if conflict_path == "scheduler_bulk": + await reply_sla_scheduler.ReplySlaScheduler()._sync() + else: + async with worker_sessions() as worker_session: + overdue_emails = ( + await worker_session.scalars( + select(Email) + .where(Email.id.in_(source_email_ids)) + .order_by(Email.id) + ) + ).all() + ( + created_count, + _, + ) = await reply_sla_escalation_service._process_fallback_escalation( + worker_session, + owner_key, + owner_key, + overdue_emails, + ReplayDateTime.now(date_time.timezone.utc), + ) + assert created_count == 1 + async with replica_sessions() as replica_session: + task_records = ( + await replica_session.scalars( + select(TicketTask).where(TicketTask.user_id == owner_key) + ) + ).all() + assert {task.related_email_id for task in task_records} == set( + source_email_ids + ) + assert len(task_records) == 2 + assert all( + task.status == "blocked" and task.priority == "urgent" + for task in task_records + ) + assert any(task.task_uid == competing_task_uid for task in task_records) + finally: + await _clear_observed_mail(worker_sessions, owner_key) + await worker_engine.dispose() + await replica_engine.dispose() + + +async def test_deleted_mailbox_after_first_workspace_stops_remaining_work(monkeypatch): + """A configuration removed after a successful commit must not authorize the next workspace.""" + worker_engine = create_async_engine( + settings.DATABASE_URL, pool_size=1, max_overflow=0 + ) + replica_engine = create_async_engine( + settings.DATABASE_URL, pool_size=1, max_overflow=0 + ) + worker_sessions = async_sessionmaker(worker_engine, expire_on_commit=False) + replica_sessions = async_sessionmaker(replica_engine, expire_on_commit=False) + owner_key = f"lease_scope_{uuid4().hex}" + visited_workspaces = [] + create_tasks = reply_sla_scheduler.create_reply_sla_escalation_tasks + replay_clock = SimpleNamespace( + datetime=ReplayDateTime, + timedelta=date_time.timedelta, + timezone=date_time.timezone, + ) + monkeypatch.setattr(reply_sla_escalation_service, "datetime", replay_clock) + monkeypatch.setattr(reply_tracking_service, "datetime", replay_clock) + monkeypatch.setattr(reply_sla_scheduler, "engine", worker_engine, raising=False) + monkeypatch.setattr(reply_sla_scheduler, "AsyncSessionLocal", worker_sessions) + + async def remove_after_commit(database_session, **owner_scope): + """Apply an independent committed mailbox deletion after the first actual task write.""" + visited_workspaces.append(owner_scope["workspace_id"]) + result = await create_tasks(database_session, **owner_scope) + async with replica_sessions() as replica_session: + await replica_session.execute( + delete(TenantConfig).where( + TenantConfig.user_id == owner_key, + TenantConfig.organization_id == owner_key, + ) + ) + await replica_session.commit() + return result + + monkeypatch.setattr( + reply_sla_scheduler, "create_reply_sla_escalation_tasks", remove_after_commit + ) + try: + for workspace_suffix in ("first", "second"): + await _seed_observed_mail( + worker_sessions, owner_key, f"{owner_key}_{workspace_suffix}" + ) + await reply_sla_scheduler.ReplySlaScheduler()._sync() + assert len(visited_workspaces) == 1 + async with replica_sessions() as replica_session: + task_ids = ( + await replica_session.scalars( + select(TicketTask.task_uid).where(TicketTask.user_id == owner_key) + ) + ).all() + assert len(task_ids) == 1 + finally: + await _clear_observed_mail(worker_sessions, owner_key) + await worker_engine.dispose() + await replica_engine.dispose() + + +@pytest.mark.parametrize( + "exit_mode", + ["complete", "acquire_cancel", "work_cancel", "unlock_error", "close_wait"], +) +async def test_one_slot_sweep_releases_lease_before_session_cleanup( + monkeypatch, exit_mode +): + """Real writes need one pool slot; cancellation cannot wait on session cleanup holding a lease.""" + worker_engine = create_async_engine( + settings.DATABASE_URL, pool_size=1, max_overflow=0 + ) + replica_engine = create_async_engine( + settings.DATABASE_URL, pool_size=1, max_overflow=0 + ) + worker_sessions = async_sessionmaker(worker_engine, expire_on_commit=False) + replica_sessions = async_sessionmaker(replica_engine, expire_on_commit=False) + owner_key = f"lease_scope_{uuid4().hex}" + ready_event = asyncio.Event() + cleanup_started = asyncio.Event() + cleanup_allowed = asyncio.Event() + never_ready = asyncio.Event() + acquire_lease = reply_sla_scheduler._try_acquire_sweep_lease + release_lease = reply_sla_scheduler._release_sweep_lease + sweep_owners = reply_sla_scheduler.ReplySlaScheduler._sweep_configured_owners + replay_clock = SimpleNamespace( + datetime=ReplayDateTime, + timedelta=date_time.timedelta, + timezone=date_time.timezone, + ) + monkeypatch.setattr(reply_sla_escalation_service, "datetime", replay_clock) + monkeypatch.setattr(reply_tracking_service, "datetime", replay_clock) + monkeypatch.setattr(reply_sla_scheduler, "engine", worker_engine, raising=False) + + class GatedCleanupSession(AsyncSession): + """Model blocked session teardown while retaining real SQLAlchemy operations.""" + + async def __aexit__(self, *error_state): + """Expose the point before the normal shielded close is allowed.""" + if exit_mode == "close_wait": + cleanup_started.set() + await cleanup_allowed.wait() + return await super().__aexit__(*error_state) + + monkeypatch.setattr( + reply_sla_scheduler, + "AsyncSessionLocal", + async_sessionmaker( + worker_engine, class_=GatedCleanupSession, expire_on_commit=False + ), + ) + + async def acquire_then_pause(database_session): + """Cancel after PostgreSQL acquired the lease but before caller acknowledgement.""" + acquired = await acquire_lease(database_session) + assert acquired is True + ready_event.set() + await never_ready.wait() + + async def sweep_then_pause(scheduler, database_session): + """Execute actual escalation before cancelling an in-progress cycle.""" + await sweep_owners(scheduler, database_session) + ready_event.set() + await never_ready.wait() + + async def fail_unlock(database_session): + """Raise at the release boundary while the real lease remains held.""" + raise RuntimeError("Injected release failure.") + + if exit_mode == "acquire_cancel": + monkeypatch.setattr( + reply_sla_scheduler, "_try_acquire_sweep_lease", acquire_then_pause + ) + elif exit_mode in {"work_cancel", "close_wait"}: + monkeypatch.setattr( + reply_sla_scheduler.ReplySlaScheduler, + "_sweep_configured_owners", + sweep_then_pause, + ) + elif exit_mode == "unlock_error": + monkeypatch.setattr(reply_sla_scheduler, "_release_sweep_lease", fail_unlock) + + sweep_task = None + try: + source_email_id = await _seed_observed_mail(worker_sessions, owner_key) + sweep_task = asyncio.create_task( + reply_sla_scheduler.ReplySlaScheduler()._sync() + ) + if exit_mode.endswith("cancel") or exit_mode == "close_wait": + await asyncio.wait_for(ready_event.wait(), 5) + sweep_task.cancel() + if exit_mode == "close_wait": + await asyncio.wait_for(cleanup_started.wait(), 5) + async with replica_sessions() as replica_session: + assert await acquire_lease(replica_session) is True + await release_lease(replica_session) + cleanup_allowed.set() + with pytest.raises(asyncio.CancelledError): + await sweep_task + elif exit_mode == "unlock_error": + with pytest.raises(RuntimeError, match="Injected release failure"): + await asyncio.wait_for(sweep_task, 5) + else: + await asyncio.wait_for(sweep_task, 5) + async with replica_sessions() as replica_session: + assert await acquire_lease(replica_session) is True + await release_lease(replica_session) + task_ids = ( + await replica_session.scalars( + select(TicketTask.related_email_id).where( + TicketTask.user_id == owner_key + ) + ) + ).all() + assert task_ids == ( + [] if exit_mode == "acquire_cancel" else [source_email_id] + ) + assert worker_engine.pool.checkedout() == 0 + finally: + cleanup_allowed.set() + if sweep_task is not None: + if not sweep_task.done(): + sweep_task.cancel() + await asyncio.gather(sweep_task, return_exceptions=True) + await _clear_observed_mail(worker_sessions, owner_key) + await worker_engine.dispose() + await replica_engine.dispose() + + +@pytest.mark.parametrize( + "failure_mode", ["transaction_error", "explicit_rollback", "disconnect"] +) +async def test_owner_failure_recovers_only_while_physical_lease_is_valid( + monkeypatch, failure_mode +): + """Recover healthy rollback, but never continue the sweep on a replacement backend.""" + worker_engine = create_async_engine( + settings.DATABASE_URL, pool_size=1, max_overflow=0 + ) + replica_engine = create_async_engine( + settings.DATABASE_URL, pool_size=1, max_overflow=0 + ) + worker_sessions = async_sessionmaker(worker_engine, expire_on_commit=False) + replica_sessions = async_sessionmaker(replica_engine, expire_on_commit=False) + owner_keys = [f"lease_scope_{uuid4().hex}" for _ in range(2)] + visited_owners = [] + create_tasks = reply_sla_scheduler.create_reply_sla_escalation_tasks + replay_clock = SimpleNamespace( + datetime=ReplayDateTime, + timedelta=date_time.timedelta, + timezone=date_time.timezone, + ) + monkeypatch.setattr(reply_sla_escalation_service, "datetime", replay_clock) + monkeypatch.setattr(reply_tracking_service, "datetime", replay_clock) + monkeypatch.setattr(reply_sla_scheduler, "engine", worker_engine, raising=False) + monkeypatch.setattr(reply_sla_scheduler, "AsyncSessionLocal", worker_sessions) + + async def fail_first_owner(database_session, **owner_scope): + """Inject one real database failure before letting remaining owners proceed.""" + visited_owners.append(owner_scope["user_id"]) + if len(visited_owners) == 1: + if failure_mode == "transaction_error": + await database_session.execute(text("SELECT 1 / 0")) + elif failure_mode == "explicit_rollback": + await database_session.rollback() + raise RuntimeError("Escalation operation failed.") + else: + worker_pid = await database_session.scalar( + text("SELECT pg_backend_pid()") + ) + async with replica_sessions() as replica_session: + assert ( + await replica_session.scalar( + text("SELECT pg_terminate_backend(:worker_pid)"), + {"worker_pid": worker_pid}, + ) + is True + ) + await database_session.scalar(text("SELECT 1")) + return await create_tasks(database_session, **owner_scope) + + monkeypatch.setattr( + reply_sla_scheduler, "create_reply_sla_escalation_tasks", fail_first_owner + ) + try: + for owner_key in owner_keys: + await _seed_observed_mail(worker_sessions, owner_key) + if failure_mode == "disconnect": + with pytest.raises(DBAPIError) as error_info: + await reply_sla_scheduler.ReplySlaScheduler()._sync() + assert error_info.value.connection_invalidated + assert len(visited_owners) == 1 + else: + await reply_sla_scheduler.ReplySlaScheduler()._sync() + assert set(visited_owners) == set(owner_keys) + async with replica_sessions() as replica_session: + task_owners = ( + await replica_session.scalars( + select(TicketTask.user_id).where(TicketTask.user_id.in_(owner_keys)) + ) + ).all() + assert task_owners == ( + [] if failure_mode == "disconnect" else visited_owners[1:] + ) + assert ( + await reply_sla_scheduler._try_acquire_sweep_lease(replica_session) + is True + ) + await reply_sla_scheduler._release_sweep_lease(replica_session) + finally: + for owner_key in owner_keys: + await _clear_observed_mail(worker_sessions, owner_key) + await worker_engine.dispose() + await replica_engine.dispose() diff --git a/backend/tests/test_reply_tracking.py b/backend/tests/test_reply_tracking.py index 5ff91616d..dd646c373 100644 --- a/backend/tests/test_reply_tracking.py +++ b/backend/tests/test_reply_tracking.py @@ -57,7 +57,9 @@ async def test_identifying_sent_emails_awaiting_replies(): config_mock = TenantConfig(user_id="user_1", smtp_username="my@email.com") session = ReplyTrackingSession(config_mock, [email_awaiting]) - flagged_emails = await check_missing_replies(session, "user_1", "org_1") + flagged_emails = await check_missing_replies( + session, "user_1", "org_1", "workspace-a" + ) assert len(flagged_emails) == 1 assert flagged_emails[0].id == 1 @@ -113,7 +115,9 @@ async def test_missing_reply_tracking_excludes_answered_and_non_intent_threads() [sent_answered, external_reply, sent_without_reply_intent, self_sent_note], ) - flagged_emails = await check_missing_replies(session, "user_1", "org_1") + flagged_emails = await check_missing_replies( + session, "user_1", "org_1", "workspace-a" + ) assert flagged_emails == [] @@ -147,7 +151,9 @@ async def test_missing_reply_tracking_groups_bracketed_thread_ids(): config_mock = TenantConfig(user_id="user_1", smtp_username="my@email.com") session = ReplyTrackingSession(config_mock, [sent_answered, external_reply]) - flagged_emails = await check_missing_replies(session, "user_1", "org_1") + flagged_emails = await check_missing_replies( + session, "user_1", "org_1", "workspace-a" + ) assert flagged_emails == [] @@ -159,7 +165,7 @@ async def test_missing_reply_tracking_scopes_email_query_to_user_and_org(): config_mock = TenantConfig(user_id="user_1", smtp_username="my@email.com") session = ReplyTrackingSession(config_mock, []) - await check_missing_replies(session, "user_1", "org_1") + await check_missing_replies(session, "user_1", "org_1", "workspace-a") config_query = session.queries[0] config_query_text = compiled_query_text(config_query) @@ -199,11 +205,14 @@ async def test_missing_reply_tracking_uses_provided_tenant_config_without_lookup session, "user_1", "org_1", + "workspace-a", tenant_config=config_mock, ) assert [email.id for email in flagged_emails] == [8] - assert all("tenant_configs" not in compiled_query_text(query) for query in session.queries) + assert all( + "tenant_configs" not in compiled_query_text(query) for query in session.queries + ) @pytest.mark.asyncio @@ -227,6 +236,8 @@ async def test_missing_reply_tracking_returns_empty_when_no_user_addresses(): from services.reply_tracking_service import check_missing_replies session = ReplyTrackingSession(None, []) - flagged_emails = await check_missing_replies(session, "user_1", "org_1") + flagged_emails = await check_missing_replies( + session, "user_1", "org_1", "workspace-a" + ) assert flagged_emails == [] diff --git a/backend/tests/test_search.py b/backend/tests/test_search.py index 2e89d9f56..04f22bb77 100644 --- a/backend/tests/test_search.py +++ b/backend/tests/test_search.py @@ -13,9 +13,7 @@ pytestmark = pytest.mark.usefixtures("dev_auth_dependency_overrides") -_CANDIDATE_DATE = datetime.datetime( - 2026, 4, 27, 10, 0, tzinfo=datetime.timezone.utc -) +_CANDIDATE_DATE = datetime.datetime(2026, 4, 27, 10, 0, tzinfo=datetime.timezone.utc) class MockLexicalRow: @@ -117,19 +115,11 @@ async def execute(self, stmt): return MockRowsResult([MockReplyCountRow("thread-123", 2)]) if "word_similarity" in statement_text: return MockRowsResult( - [ - MockLexicalRow( - 1, "Test Subject", "test@test.com", "Test Body", 0.9 - ) - ] + [MockLexicalRow(1, "Test Subject", "test@test.com", "Test Body", 0.9)] ) if "<=>" in statement_text: return MockRowsResult( - [ - MockDenseRow( - 1, "Test Subject", "test@test.com", "Test Body", 0.3 - ) - ] + [MockDenseRow(1, "Test Subject", "test@test.com", "Test Body", 0.3)] ) return MockRowsResult([]) @@ -278,9 +268,7 @@ async def override_scoped_db(): assert "email_records.organization_id" in statement_text query_params = channel_statement.compile().params user_scope_params = { - value - for key, value in query_params.items() - if key.startswith("user_id") + value for key, value in query_params.items() if key.startswith("user_id") } organization_scope_params = { value @@ -317,9 +305,7 @@ async def override_scoped_db(): assert response.status_code == 200 data = response.json() assert len(data["results"]) == 1 - joined_statements = " ".join( - str(stmt).lower() for stmt in session.statements - ) + joined_statements = " ".join(str(stmt).lower() for stmt in session.statements) assert "word_similarity" in joined_statements assert "<=>" not in joined_statements @@ -355,9 +341,7 @@ async def override_scoped_db(): assert response.status_code == 200 data = response.json() assert len(data["results"]) == 1 - joined_statements = " ".join( - str(stmt).lower() for stmt in session.statements - ) + joined_statements = " ".join(str(stmt).lower() for stmt in session.statements) assert "word_similarity" in joined_statements assert "<=>" not in joined_statements @@ -406,12 +390,10 @@ async def override_search_db(): "llm_providers" in str(stmt).lower() for stmt in config_session.statements ) assert all( - "word_similarity" not in str(stmt).lower() - for stmt in config_session.statements + "word_similarity" not in str(stmt).lower() for stmt in config_session.statements ) assert any( - "word_similarity" in str(stmt).lower() - for stmt in search_session.statements + "word_similarity" in str(stmt).lower() for stmt in search_session.statements ) @@ -437,9 +419,7 @@ async def override_scoped_db(): app.dependency_overrides.clear() assert response.status_code == 200 - joined_statements = " ".join( - str(stmt).lower() for stmt in session.statements - ) + joined_statements = " ".join(str(stmt).lower() for stmt in session.statements) assert "<=>" in joined_statements @@ -478,16 +458,52 @@ def test_build_reply_counts_stmt_scopes_and_groups_by_thread_key(): from api.search import build_reply_counts_stmt stmt = build_reply_counts_stmt( - ["thread-1", "thread-2"], user_id="user1", organization_id="org1" + ["thread-1", "thread-2"], + user_id="user1", + organization_id="org1", + workspace_id="workspace-a", ) sql = str(stmt).lower() assert "email_records.user_id" in sql assert "email_records.organization_id" in sql + assert "email_records.workspace_id" in sql assert "count(email_records.id)" in sql assert "group by coalesce(nullif(btrim(btrim(email_records.thread_id)" in sql +def test_lexical_attachment_statement_excludes_non_parsed_attachments(): + # Quarantined (content_type_mismatch_quarantined) and deferred-recognition + # (pdf_dom_recognition_pending) attachments store a base64-encoded raw + # payload in `content`, not parsed text -- hybrid search must not surface + # that payload as if it were a genuine content match. + from db.models import Email + from services.hybrid_retrieval.retrieval_channels import ( + build_lexical_attachment_statement, + ) + + owner_filters = Email.owner_filters("user1", "org1", "workspace-a") + + stmt = build_lexical_attachment_statement("hello", owner_filters, 20) + sql = str(stmt).lower() + + assert "email_attachments.parse_status" in sql + + +def test_dense_attachment_statement_excludes_non_parsed_attachments(): + from db.models import Email + from services.hybrid_retrieval.retrieval_channels import ( + build_dense_attachment_statement, + ) + + owner_filters = Email.owner_filters("user1", "org1", "workspace-a") + + stmt = build_dense_attachment_statement([0.1] * 1536, owner_filters, 20) + sql = str(stmt).lower() + + assert "email_attachments.parse_status" in sql + + def _make_fusion_settings(**overrides): return FusionSettings(**overrides) diff --git a/backend/tests/test_search_postgres.py b/backend/tests/test_search_postgres.py index 999f60505..b25a0cf27 100644 --- a/backend/tests/test_search_postgres.py +++ b/backend/tests/test_search_postgres.py @@ -134,6 +134,7 @@ def _make_email( return Email( user_id=user_id, organization_id=organization_id, + workspace_id=f"workspace-{organization_id}", message_id=f"<{uuid.uuid4().hex}@example.com>", thread_id=thread_id, sender="sender@example.com", diff --git a/backend/tests/test_security_api.py b/backend/tests/test_security_api.py index a00a1d57c..5791ffea9 100644 --- a/backend/tests/test_security_api.py +++ b/backend/tests/test_security_api.py @@ -660,9 +660,12 @@ async def override_get_db(): previous_secret = settings.AUTH_SESSION_HMAC_SECRET original_overrides = dict(app.dependency_overrides) settings.AUTH_SESSION_HMAC_SECRET = SecretStr(TEST_SESSION_HMAC_SECRET) - token = _signed_session_token( - _valid_session_payload(workspace="another_workspace_id") - ) + # This route's own check (_require_authoritative_workspace_scope) is + # keyed on session_verifier=="hmac" alone, not on the workspace claim's + # value, so an otherwise-valid, org-consistent workspace claim isolates + # that behavior from the org/workspace-derivation check in + # _auth_context_from_session_payload. + token = _signed_session_token(_valid_session_payload()) app.dependency_overrides[get_db] = override_get_db app.dependency_overrides.pop(get_auth_context, None) app.dependency_overrides.pop(get_current_user, None) diff --git a/backend/tests/test_tasks_api.py b/backend/tests/test_tasks_api.py index 5e690fce5..396707a93 100644 --- a/backend/tests/test_tasks_api.py +++ b/backend/tests/test_tasks_api.py @@ -815,6 +815,7 @@ async def cleanup_seed_rows(): Email( user_id=user_id, organization_id=organization_id, + workspace_id=f"workspace-{organization_id}", message_id="", thread_id="", sender="reply-sla-smoke@example.com", diff --git a/backend/tests/test_tenant_config_api.py b/backend/tests/test_tenant_config_api.py index b9d84ded6..3db4e2489 100644 --- a/backend/tests/test_tenant_config_api.py +++ b/backend/tests/test_tenant_config_api.py @@ -147,6 +147,34 @@ def fake_validate_pop3_destination(pop3_server, pop3_port, *, resolve_host=True) assert data["google_client_secret"] is None +def test_noema_orchestrator_gateway_config_round_trips_and_masks_the_token( + client, mock_db +): + # backend/services/orchestrator_gateway.py resolves + # noema_orchestrator_base_url/noema_orchestrator_token from TenantConfig, + # but nothing could ever set them without this wiring -- Devin Review + # correctly flagged that the feature was otherwise unreachable through + # any supported configuration call. + post_payload = { + "user_id": "test_user", + "noema_orchestrator_base_url": "https://orchestrator.internal/v1", + "noema_orchestrator_token": "orch-token-123", + } + response = client.post( + "/api/config", json=post_payload, headers={"X-User-Id": "test_user"} + ) + assert response.status_code == 200 + + get_response = client.get( + "/api/config", + headers={"X-User-Id": "test_user"}, + ) + assert get_response.status_code == 200 + data = get_response.json() + assert data["noema_orchestrator_base_url"] == "https://orchestrator.internal/v1" + assert data["noema_orchestrator_token"] == "********" + + def test_validate_mail_config_update_revalidates_existing_mail_hosts(monkeypatch): from api.tenant_config import validate_mail_config_update diff --git a/backend/tests/test_threading_perf.py b/backend/tests/test_threading_perf.py index 145353eed..17cd4d03f 100644 --- a/backend/tests/test_threading_perf.py +++ b/backend/tests/test_threading_perf.py @@ -33,6 +33,7 @@ async def test_assign_thread_id_batches_many_reference_lookups(): }, user_id="testuser", organization_id="org-acme", + workspace_id="workspace-a", ) assert thread_id == "ref0@example.com" diff --git a/backend/tests/test_threading_pipeline.py b/backend/tests/test_threading_pipeline.py index 570599af2..3a3eaeeee 100644 --- a/backend/tests/test_threading_pipeline.py +++ b/backend/tests/test_threading_pipeline.py @@ -3,18 +3,19 @@ from services.threading_service import generate_email_fingerprint from db.models import Email + def test_generate_email_fingerprint(): fingerprint1 = generate_email_fingerprint( subject="Test Subject", date_str="2023-10-27T10:00:00+00:00", sender="sender@example.com", - recipient="receiver@example.com" + recipient="receiver@example.com", ) fingerprint2 = generate_email_fingerprint( subject="Test Subject", date_str="2023-10-27T10:00:00+00:00", sender="sender@example.com", - recipient="receiver@example.com" + recipient="receiver@example.com", ) assert fingerprint1 == fingerprint2 assert isinstance(fingerprint1, str) @@ -24,26 +25,27 @@ def test_generate_email_fingerprint(): subject="Other Subject", date_str="2023-10-27T10:00:00+00:00", sender="sender@example.com", - recipient="receiver@example.com" + recipient="receiver@example.com", ) assert fingerprint1 != fingerprint3 + @pytest.mark.asyncio async def test_email_deduplication(): from services.imap_worker import process_fetched_email from unittest.mock import MagicMock - + from services.email_parser import EmailData - + from datetime import datetime, timezone - + session_mock = AsyncMock() session_mock.add = MagicMock() # Assume select returns nothing (no duplicate) execute_result = MagicMock() execute_result.scalar_one_or_none.return_value = None session_mock.execute.return_value = execute_result - + email_data: EmailData = { "subject": "Test Duplicate", "date": datetime(2023, 10, 27, 10, 0, tzinfo=timezone.utc), @@ -55,23 +57,27 @@ async def test_email_deduplication(): "references": None, "thread_id": None, "reply_to": None, - "attachments": [] + "attachments": [], } - - await process_fetched_email(session_mock, email_data, "user_1", "org_1") - + + await process_fetched_email( + session_mock, email_data, "user_1", "org_1", "workspace-a" + ) + # Check that session.add was called since it's not a duplicate session_mock.add.assert_called_once() added_email = session_mock.add.call_args[0][0] assert added_email.date == datetime(2023, 10, 27, 10, 0, tzinfo=timezone.utc) - + # Now simulate a duplicate session_mock.reset_mock() existing_email = Email(id=1, thread_id="thread_1") execute_result.scalar_one_or_none.return_value = existing_email - - await process_fetched_email(session_mock, email_data, "user_1", "org_1") - + + await process_fetched_email( + session_mock, email_data, "user_1", "org_1", "workspace-a" + ) + # Check that session.add was NOT called, but existing_email's thread_id remains the same # or some update happens session_mock.add.assert_not_called() @@ -113,6 +119,7 @@ async def test_email_pipeline_triggers_self_sent_knowledge_extraction(): email_data, "user_1", "org_1", + "workspace-a", owner_addresses=["user_1@example.com"], ) @@ -149,7 +156,9 @@ async def test_email_pipeline_preserves_personal_scope_as_null(): "attachments": [], } - await process_fetched_email(session_mock, email_data, "user@example.com", None) + await process_fetched_email( + session_mock, email_data, "user@example.com", None, "workspace-a" + ) added_email = session_mock.add.call_args[0][0] assert added_email.organization_id is None diff --git a/backend/tests/test_threading_service.py b/backend/tests/test_threading_service.py index 2372ceaf1..9add7a9d3 100644 --- a/backend/tests/test_threading_service.py +++ b/backend/tests/test_threading_service.py @@ -51,6 +51,7 @@ async def test_reply_before_root_uses_first_reference_as_deterministic_thread_id }, user_id="testuser", organization_id="org-acme", + workspace_id="workspace-a", ) assert thread_id == "root@example.com" @@ -69,6 +70,7 @@ async def test_reply_without_references_uses_in_reply_to_as_deterministic_thread }, user_id="testuser", organization_id="org-acme", + workspace_id="workspace-a", ) assert thread_id == "parent@example.com" @@ -87,6 +89,7 @@ async def test_existing_parent_thread_id_wins_over_deterministic_fallback(): }, user_id="testuser", organization_id="org-acme", + workspace_id="workspace-a", ) assert thread_id == "thread-123" @@ -105,6 +108,7 @@ async def test_existing_legacy_bracketed_thread_id_is_normalized(): }, user_id="testuser", organization_id="org-acme", + workspace_id="workspace-a", ) assert thread_id == "root@example.com" @@ -124,6 +128,7 @@ async def test_forwarded_subject_alone_does_not_merge_unrelated_thread(): }, user_id="testuser", organization_id="org-acme", + workspace_id="workspace-a", ) assert thread_id == "forwarded-copy@example.com" @@ -143,12 +148,14 @@ async def test_existing_thread_lookup_is_scoped_to_owner_and_organization(): }, user_id="testuser", organization_id="org-acme", + workspace_id="workspace-a", ) assert thread_id == "thread-123" query_text = str(session.queries[-1]).lower() assert "email_records.user_id" in query_text assert "email_records.organization_id" in query_text + assert "email_records.workspace_id" in query_text def test_normalize_message_id_strips_brackets_and_outer_whitespace(): @@ -201,7 +208,11 @@ def test_extract_reference_ids_falls_back_to_whitespace_split_without_brackets() async def test_find_existing_thread_ids_returns_empty_without_candidates(): session = _SequentialSession([]) result = await _find_existing_thread_ids( - session, [], user_id="testuser", organization_id="org-acme" + session, + [], + user_id="testuser", + organization_id="org-acme", + workspace_id="workspace-a", ) assert result == {} assert session.execute_count == 0 @@ -217,6 +228,7 @@ async def test_find_existing_thread_ids_dedupes_overlapping_bracket_targets(): ["", "a@x.com"], user_id="testuser", organization_id="org-acme", + workspace_id="workspace-a", ) assert result == {"a@x.com": "thread-a"} @@ -239,6 +251,7 @@ async def test_find_existing_thread_ids_skips_rows_with_blank_thread_or_message_ ["a@x.com", "c@x.com"], user_id="testuser", organization_id="org-acme", + workspace_id="workspace-a", ) assert result == {"c@x.com": "thread-c"} @@ -259,6 +272,7 @@ async def test_assign_thread_id_uses_a_later_candidate_when_the_first_has_no_thr }, user_id="testuser", organization_id="org-acme", + workspace_id="workspace-a", ) assert thread_id == "thread-older" @@ -277,7 +291,10 @@ def test_generate_email_fingerprint_is_deterministic_case_insensitive_and_field_ # 2. lower-cased + outer-whitespace-stripped components collapse to one key assert ( generate_email_fingerprint( - " QUARTERLY PLAN ", "Mon, 01 Jun 2026 09:00:00 +0000", "A@X.com", " b@Y.com " + " QUARTERLY PLAN ", + "Mon, 01 Jun 2026 09:00:00 +0000", + "A@X.com", + " b@Y.com ", ) == baseline ) @@ -305,6 +322,7 @@ async def test_assign_thread_id_generates_fresh_uuid_when_no_identifiers_present {"message_id": None, "in_reply_to": None, "references": None}, user_id="testuser", organization_id="org-acme", + workspace_id="workspace-a", ) assert len(thread_id) == 32 @@ -330,6 +348,7 @@ async def test_multi_id_in_reply_to_threads_on_any_existing_parent(): }, user_id="testuser", organization_id="org-acme", + workspace_id="workspace-a", ) assert thread_id == "thread-xyz" @@ -348,6 +367,7 @@ async def test_multi_id_in_reply_to_fallback_uses_first_parent_as_root(): }, user_id="testuser", organization_id="org-acme", + workspace_id="workspace-a", ) assert thread_id == "first@example.com" @@ -369,6 +389,7 @@ async def test_in_reply_to_with_cfws_comment_extracts_bare_msg_id(): }, user_id="testuser", organization_id="org-acme", + workspace_id="workspace-a", ) assert thread_id == "thread-123" diff --git a/backend/tests/test_webdav_api.py b/backend/tests/test_webdav_api.py index 2c31ab9a3..59be6f0d6 100644 --- a/backend/tests/test_webdav_api.py +++ b/backend/tests/test_webdav_api.py @@ -1080,6 +1080,7 @@ async def test_knowledge_materialization_intent_real_postgres_endpoint_smoke( id INTEGER PRIMARY KEY, user_id VARCHAR NOT NULL, organization_id VARCHAR NOT NULL, + workspace_id VARCHAR NOT NULL, message_id VARCHAR NOT NULL, thread_id VARCHAR ) @@ -1131,15 +1132,17 @@ async def test_knowledge_materialization_intent_real_postgres_endpoint_smoke( INSERT INTO email_records ( id, user_id, - organization_id, - message_id, + organization_id, + workspace_id, + message_id, thread_id ) VALUES ( :email_id, :user_id, - :organization_id, - :message_id, + :organization_id, + :workspace_id, + :message_id, :thread_id ) """ @@ -1148,6 +1151,7 @@ async def test_knowledge_materialization_intent_real_postgres_endpoint_smoke( "email_id": smoke_id, "user_id": user_id, "organization_id": "org-acme", + "workspace_id": "workspace-org-acme", "message_id": message_id, "thread_id": thread_id, }, diff --git a/backend/uv.lock b/backend/uv.lock index 2f0d04a76..d455f2a61 100644 --- a/backend/uv.lock +++ b/backend/uv.lock @@ -683,6 +683,19 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/7e/f5/f66802a942d491edb555dd61e3a9961140fd64c90bce1eafd741609d334d/httpcore-1.0.9-py3-none-any.whl", hash = "sha256:2d400746a40668fc9dec9810239072b40b4484b640a8c38fd654a024c7a1bf55", size = 78784, upload-time = "2025-04-24T22:06:20.566Z" }, ] +[[package]] +name = "httpcore2" +version = "2.5.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "h11" }, + { name = "truststore" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/47/06/5c12df521b5322fb1114a83d46911b2fbcb8855ddb3a635f11c01a214af5/httpcore2-2.5.0.tar.gz", hash = "sha256:88aa170137c17328d5ac44234f9fd10706466d5fb347f3edac4d39b91137b09d", size = 64808, upload-time = "2026-06-25T14:16:56.472Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/c9/a1/7564199d1a8728fe737b0a72e5b3f8d92dfe085a74ddf7cdd83bce5f206d/httpcore2-2.5.0-py3-none-any.whl", hash = "sha256:5ce35188de461d31e8d000bfb8ef8bf22c6c16587a211e5571deaa5e9bdf842a", size = 80330, upload-time = "2026-06-25T14:16:53.634Z" }, +] + [[package]] name = "httplib2" version = "0.32.0" @@ -710,6 +723,22 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/2a/39/e50c7c3a983047577ee07d2a9e53faf5a69493943ec3f6a384bdc792deb2/httpx-0.28.1-py3-none-any.whl", hash = "sha256:d909fcccc110f8c7faf814ca82a9a4d816bc5a6dbfea25d6591d6985b8ba59ad", size = 73517, upload-time = "2024-12-06T15:37:21.509Z" }, ] +[[package]] +name = "httpx2" +version = "2.5.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "anyio" }, + { name = "httpcore2" }, + { name = "idna" }, + { name = "truststore" }, + { name = "typing-extensions", marker = "python_full_version < '3.13'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/d0/e2/b5dedc0cf35aa65de5f541ccd30d2bc1fd7f1d43c9ab09f8ed9a7342317b/httpx2-2.5.0.tar.gz", hash = "sha256:e2df9cb4611021527ff8a675b1c320b610a2ec397acc8d6fe6e91df2d9b33c29", size = 83121, upload-time = "2026-06-25T14:16:57.491Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/31/22/859d8252dad9bc9adee34b52e62cde621ece07b042ccb2ab4da1be46695f/httpx2-2.5.0-py3-none-any.whl", hash = "sha256:3d2d4d9cf4b61f1a1f46a95947cfdb47e80cb56a2f91c6256ac8f58e4891df41", size = 76652, upload-time = "2026-06-25T14:16:55.23Z" }, +] + [[package]] name = "icalendar" version = "7.2.0" @@ -1019,6 +1048,7 @@ dependencies = [ [package.dev-dependencies] dev = [ { name = "coverage" }, + { name = "httpx2" }, { name = "pytest" }, { name = "pytest-asyncio" }, { name = "ruff" }, @@ -1065,6 +1095,7 @@ requires-dist = [ [package.metadata.requires-dev] dev = [ { name = "coverage", specifier = "==7.15.1" }, + { name = "httpx2", specifier = "==2.5.0" }, { name = "pytest", specifier = "==9.1.1" }, { name = "pytest-asyncio", specifier = "==1.4.0" }, { name = "ruff", specifier = "==0.15.21" }, @@ -1972,6 +2003,15 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/22/2a/5e5e750890ada51017d18d0d4c30da696e5b5bd3180947729927628fc3cb/tqdm-4.68.4-py3-none-any.whl", hash = "sha256:5168118b2368f48c561afda8020fd79195b1bdb0bdf8086b88442c267a315dc2", size = 676612, upload-time = "2026-07-07T09:58:16.256Z" }, ] +[[package]] +name = "truststore" +version = "0.10.4" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/53/a3/1585216310e344e8102c22482f6060c7a6ea0322b63e026372e6dcefcfd6/truststore-0.10.4.tar.gz", hash = "sha256:9d91bd436463ad5e4ee4aba766628dd6cd7010cf3e2461756b3303710eebc301", size = 26169, upload-time = "2025-08-12T18:49:02.73Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/19/97/56608b2249fe206a67cd573bc93cd9896e1efb9e98bce9c163bcdc704b88/truststore-0.10.4-py3-none-any.whl", hash = "sha256:adaeaecf1cbb5f4de3b1959b42d41f6fab57b2b1666adb59e89cb0b53361d981", size = 18660, upload-time = "2025-08-12T18:49:01.46Z" }, +] + [[package]] name = "typing-extensions" version = "4.16.0" diff --git a/docs/adr/0005-attachment-content-type-quarantine.md b/docs/adr/0005-attachment-content-type-quarantine.md new file mode 100644 index 000000000..4f90833be --- /dev/null +++ b/docs/adr/0005-attachment-content-type-quarantine.md @@ -0,0 +1,467 @@ +# ADR-0005: Quarantine attachments whose bytes disagree with their declared type + +> Status correction (2026-09-01): references below to an open or deferred +> `Email.owner_filters()` gap are historical. The helper now requires +> `(user_id, organization_id, workspace_id)` and all production callers must +> provide an explicit workspace; no implicit workspace scope is accepted. + +**Status:** Proposed (Naruon-local attachment ingestion policy; unmerged PR #1486) +**Date:** 2026-08-30 +**Decision owner:** Naruon maintainers +**Scope:** Naruon's email-attachment parsing boundary +(`services/attachment_parser.py`, `db.models.Attachment`, +`POST /api/data/attachments/{attachment_uid}/reparse-intent`). This ADR does +not introduce a general-purpose file-type detection library, a virus/malware +scanner, or automatic remediation of quarantined content. + +## Context + +`parse_email_attachment` classifies an attachment purely from the sender- +supplied `content_type` header (falling back to the filename extension for +generic/missing values). Nothing previously verified that the attachment's +actual bytes matched that claim. A sender — malicious or simply +misconfigured — can label arbitrary binary content (an executable, an image, +an archive) with any `content_type`/extension, and Naruon would either try to +parse it as if it truly were that type, defer it for heavy recognition, or +silently record it as an unsupported binary — three outcomes, none of which +tell an operator or a future worker that the content lied about what it is. + +Every other product-facing entity added recently (`TicketTask`, +`CalendarConflictJudgment`, `Document`) exposes an opaque `*_uid` id; `email_attachments` +was the one remaining entity addressable only by its internal sequential +integer id, so it could not be the target of a public, addressable retry +action. + +## Decision + +1. `services/attachment_parser.py` sniffs the attachment's real bytes against + a small table of cheap, well-known magic-byte signatures (PDF, PNG, JPEG, + GIF, ZIP). Text formats have no reliable magic bytes and are not sniffed. +2. When sniffing recognizes a signature that disagrees with the declared or + extension-resolved type, the attachment is quarantined rather than parsed, + deferred, or classified as a plain unsupported binary: + `parse_status = parse_error_code = "content_type_mismatch_quarantined"`. + The declared type stays in `content_type`; the sniffed (actual) type is + recorded in `parse_content_type`, so comparing the two columns on a + quarantined row shows the mismatch directly — no new column is introduced + for this. The raw bytes are retained (base64, bounded by the same + `MAX_ATTACHMENT_PARSE_SOURCE_BYTES` used for deferred PDF payloads) so a + later pass has something to act on, unlike the existing failure statuses + (`unsupported_content_type`, `parse_size_limit_exceeded`, + `invalid_pdf_payload`) which discard content. +3. `Attachment` gains an `attachment_uid` opaque id (Alembic `0019`, + backfilled for any pre-existing rows), following the same + `_` convention as `CalendarConflictJudgment.judgment_uid` + and `TicketTask.task_uid`. +4. `POST /api/data/attachments/{attachment_uid}/reparse-intent` lets a caller + scoped to the attachment's parent email (via the existing + `_email_scope_filter` join, since `Attachment` carries no + `workspace_id`/`user_id` of its own) record that a quarantined attachment + should be re-evaluated: it transitions `parse_status` from + `content_type_mismatch_quarantined` to `reparse_pending` and clears + `parse_error_code`. Calling it on any other status is rejected + (`422`). Matching the `hwp-conversion-intent`/`pdf-dom-recognition-intent` + pattern already established for `Document`, this endpoint only records + intent — it does not itself re-parse. A worker that consumes + `reparse_pending` was intentionally out of scope for this slice at initial + authorship, exactly as the NewsDOM PDF worker was a separate follow-up to + the original deferred-PDF decision — **revised below**: that worker now + ships in this same PR, after review (see Revisions). + +## Alternatives rejected + +### Reject mismatched attachments outright at ingestion + +Rejected: a genuinely legitimate attachment can carry a wrong `content_type` +header for reasons having nothing to do with intent (a misconfigured mail +client, a lossy import path). Outright rejection would silently drop customer +data the same way an unbounded parse failure would; quarantining preserves the +evidence and gives an explicit, auditable path back to normal handling. + +### Add a dedicated `quarantine_status` column + +Rejected: every other status-like concept in this codebase +(`Attachment.parse_status`, `Document.document_status`, +`CalendarConflictJudgment.status_code`) is a plain string column with no +separate state table or DB enum. Quarantine is one more value of the existing +`parse_status`/`parse_error_code` columns, not a new state dimension. + +### Store the sniffed type in a new column + +Rejected: `parse_content_type` already means "the MIME type this attachment +was or would be handled as." Repurposing it for the sniffed type on a +quarantined row keeps that meaning intact (it is still "what this attachment +actually is") while `content_type` keeps its existing meaning ("what the +sender declared"), without adding new schema surface for a single derived +field. + +### Synchronously re-parse from the reparse-intent endpoint + +Rejected for scope and consistency: no endpoint in this file performs heavy +synchronous work from an `-intent` route today — `hwp-conversion-intent` and +`pdf-dom-recognition-intent` both just flip a status and leave the actual +work to a background worker. Doing real work here would also require +deciding, synchronously, whether the reparse should trust the sender's +original claim or the sniffed type — a decision better made by a dedicated +worker pass than inline in a request handler. + +### Use a general-purpose file-type detection library (e.g. `python-magic`) + +Rejected for this slice per the Ponytail-adjacent discipline this org applies +before adding a dependency: the org has only ever needed to distinguish a +handful of binary families (currently just PDF, and this ADR's image/zip +additions) from what a sender declares. A half-dozen fixed magic-byte +prefixes checked in Python cover that need with no new dependency, no native +extension, and no supply-chain surface. If a much broader sniffing need +arises later (arbitrary container/archive formats, deep content inspection), +that is the point to re-evaluate a dedicated library against this policy. + +## Consequences + +- A sender cannot get Naruon to treat disguised binary content as its false + label by declaring a convenient `content_type` — the actual bytes decide + when a recognized signature is present. +- Operators/future tooling can address a specific attachment directly via + `attachment_uid` for the first time; nothing else changes about how + attachments are listed or displayed in aggregate. +- `reparse_pending` is consumed by `services/attachment_reparse_worker.py` + (see Revisions below) — a `reparse-intent` call now leads to an actual + re-evaluation on the next sweep, not an indefinite queue with no consumer. +- **Fixed (see Revisions below): `_get_scoped_attachment` and every other + attachment/email query in `api/data.py` now scope by `workspace_id` too.** + `Email` gained a `workspace_id` column (Alembic `0020_email_workspace_scope`), + and `_email_scope_filter` enforces it unconditionally, matching the pattern + `_owner_scope_statement` already used for `Document`/`WebdavAccount`/ + `ProjectFolder`. A session with the same `user_id`/`organization_id` but a + different signed `workspace_id` claim can no longer read or mutate another + workspace's attachments through this file's queries. +- **Known, still-open, narrower-scoped limitation, tracked as a dedicated + follow-up:** `Email.owner_filters(user_id, organization_id)` — the + classmethod backing mail list/search/ontology/threading/Noema-agent email + reads across `api/emails.py`, `api/search.py`, `api/ontology.py`, + `services/noema_agent.py`, `services/threading_service.py`, and + `services/hybrid_retrieval/retrieval_channels.py` — has the identical + missing-`workspace_id` gap `_email_scope_filter` had, and was deliberately + left unfixed here: closing it means auditing and updating every one of + those call sites, a whole-app multi-tenancy change well outside scope for + a PR whose stated purpose is a calendar-conflict-check tool. Recorded here + rather than silently worked around; the fix belongs in its own dedicated PR. + +## Research grounding + +The content-graph follow-up is grounded in Edge et al.'s GraphRAG work, which +separates graph-based indexing from later graph-guided answer construction and +reports benefits for query-focused summarization over large private corpora. +That supports preserving the same document topology when content enters the +index through reparse as when it enters through initial import; this ADR does +not claim that Naruon implements the paper's entity extraction or community +summarization pipeline. + +- Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, + Steven Truitt, and Jonathan Larson. 2024. “From Local to Global: A Graph RAG + Approach to Query-Focused Summarization.” arXiv:2404.16130. + https://arxiv.org/abs/2404.16130 + +No paper PDF is copied into this repository: the stable source citation is +linked instead, avoiding an unsupported redistribution assumption. + +## Revisions + +### 2026-09-05: Status and sibling lease evidence correction + +The former Accepted label preceded protected-branch adoption. This file is +absent from the verified `develop` head +`042b0c70531b229af3acbd0421a2f23098d848b3`; PR #1486 remains Draft. +All implementation and test statements below describe proposal history, not +protected-main, released, or deployed behavior. No valid delta is withdrawn. +The separate reply-SLA scheduler still had the physical-connection lease and +expired-record defects described below for the attachment workers. Its own +real PostgreSQL RED/GREEN, cancellation and manual-write conflict evidence is +recorded in [the scheduler decision supplement](../doctoring/reply_sla_physical_lease.md). +Worker evidence must not be transferred between implementations. + +Two real gaps were found and fixed after initial review, both narrowing rather +than reversing the original decision: + +- **OOXML/ODF/EPUB/JAR false positives.** DOCX, XLSX, PPTX, and other ZIP-based + container formats sniff as `application/zip` by construction — that is not a + disguise, it is what those formats are. `_is_genuine_content_type_mismatch` + now excludes a ZIP sniff whose declared type is itself a known ZIP-container + family (checked by MIME-type substring, not an exhaustive list, so sibling + OOXML/ODF variants are covered without enumerating every one). A ZIP sniffed + under any other declared type (e.g. `application/pdf`) is still quarantined. +- **Oversized mismatches were quarantined with no retained bytes, but + reparse-intent still accepted them.** A quarantined row whose payload exceeds + `MAX_ATTACHMENT_PARSE_SOURCE_BYTES` now gets the existing + `parse_size_limit_exceeded` status instead of + `content_type_mismatch_quarantined` — the same non-retryable terminal state + every other oversized attachment in this parser already gets — so + reparse-intent (which only accepts the quarantine status) can never be + requested for a row it has nothing left to act on. +- **The reparse-intent follow-up worker, deliberately deferred at first, now + ships.** `services/attachment_reparse_worker.py` (`AttachmentReparseWorker`, + wired into `main.py`'s lifespan next to `NewsdomRecognitionWorker`, same + jittered-loop + PostgreSQL advisory-lock-lease + starvation-free-cursor + shape) sweeps `reparse_pending` rows and replays + `parse_email_attachment` against the retained bytes and the attachment's + original declared `content_type` — deliberately *not* the sniffed type, + so the worker carries no bespoke "which type do I trust" logic of its own; + it only re-asks the same classification pipeline the same question, and + automatically benefits from any future fix to that pipeline (exactly as + the OOXML fix above already would have, had it existed first). A retained + payload that fails to decode (not valid base64) moves to a new terminal + `reparse_payload_invalid` status rather than being swept forever. +- **Two correctness gaps in the worker itself, found on review of the above.** + (1) The sweep cursor advanced to the batch's last row before any item in it + was actually processed, so a row whose processing raised an exception (and + therefore kept its `reparse_pending` status) fell below the cursor and would + never be reselected until the whole forward queue drained to empty — silent, + indefinite starvation under continuous reparse-intent traffic. The cursor + now caps at the row just before the first failure in a batch, keeping that + row in range for the next sweep. (2) The PostgreSQL advisory lease was + acquired and released through the same `AsyncSession` used for each item's + `commit()`/`rollback()`; since `AsyncSession.commit()` returns its + connection to the pool on every call, the lease's release could run on a + *different* physical backend connection than the one that acquired it — + advisory locks are scoped to the acquiring backend session, so a mismatched + unlock is a silent no-op that leaves the lease stuck until that connection + is later recycled or closed, silently halting every replica's sweep. The + lease is now acquired and released on one dedicated connection held open + for the whole sweep instead. +- **`services/newsdom_worker.py` carried the same two gaps, now fixed to + match.** It shared `AttachmentReparseWorker`'s acquire/release-through-the- + per-item-session lease shape and its pre-processing cursor advance, on + both its attachment and document sweeps. `NewsdomRecognitionWorker` now + acquires/releases the advisory lock on one dedicated connection opened for + the whole sweep (`_engine_uses_postgresql()` / `_try_acquire_sweep_lease` + / `_release_sweep_lease` now take a connection, not a session), and each + sweep caps its cursor at the point of the first failure instead of the + batch's last row. The document cursor is a string primary key + (`Document.document_id`, no `id - 1` to fall back on), so that sweep + tracks the last row actually confirmed resolved before a failure rather + than subtracting from the failed row's id — equivalent to the integer + case for a contiguous key, and correct for a non-contiguous one. Both + sweeps also re-fetch each row fresh by id before processing (mirroring + `AttachmentReparseWorker`), since `AsyncSession.rollback()` after an + earlier item's failure expires every object already loaded in that + session, and a stale, expired bulk-loaded instance's attribute read would + raise instead of just isolating that earlier failure. +- **Closed the cross-workspace gap this ADR's own Consequences section + recorded above, for this file's queries.** `Email` gained a `workspace_id` + column (Alembic `0020_email_workspace_scope`: add nullable, backfill every + existing row with `workspace-` — the same convention + already used by `services/email_import_service.py` and + `services/project_graph/`, since `Email.organization_id` is `NOT NULL` and + there is no FK from `Email` to any table that independently carries a real + `workspace_id` — then set `NOT NULL`). `_email_scope_filter` now applies + `Email.workspace_id == auth_context.workspace_id` unconditionally, mirroring + `_owner_scope_statement`'s existing pattern for `Document`/`WebdavAccount`/ + `ProjectFolder`, so every caller of `_get_scoped_attachment` and the + quality-surface stats helpers picks up the workspace predicate for free + (they already unpack `*email_scope` into `.where()`). The three production + call sites that construct new `Email` rows + (`services/email_import_service.py`, `services/imap_worker.py`, + `import_fixtures.py`) now populate `workspace_id` the same way. New test: + `tests/test_data_api.py::test_data_attachment_reparse_intent_is_scoped_to_workspace` + (same-user/same-org, different-workspace denial — the exact case this ADR + flagged as unclosed). `Email.owner_filters()` — the separate classmethod + backing mail list/search/ontology/threading/Noema-agent reads — has the + identical gap and remains a deliberately separate, tracked follow-up (see + Consequences above); it was not touched here. +- **CodeRabbit raised the `workspace-` backfill's trust + boundary; its follow-up correctly rebutted this ADR's first response, and + the underlying gap is now fixed at the authentication layer.** Its + concern: if a real signed session's `workspace` claim ever diverges from + the `workspace-` formula, the migration's backfill (and + every new row this PR's production call sites write) could misattribute + rows or make them unreachable. This ADR's first answer claimed that was + provably impossible for the HMAC session path, reasoning from this repo's + data-write call sites (no code here ever *writes* a non-conventional + `workspace_id`). That reasoning missed the actual attack surface: + `AUTH_SESSION_HMAC_SECRET`-signed sessions are *minted* by an external + control-plane token issuer (`iss=naruon-control-plane` per + `docs/operations/auth-key-management.md`) that has no code in this + repository at all, so nothing here can "prove" what `workspace` value it + puts in a token. CodeRabbit traced `_auth_context_from_session_payload` + (`api/auth.py`) directly and showed it requires the `org` and `workspace` + claims to each be present, but never checks the relationship between + them — so a session signed with a correct `org` and an arbitrary + `workspace` value was accepted as-is, for both the HMAC and OIDC paths, + in production today. Fixed: `_auth_context_from_session_payload` now + rejects (`401`) any session whose `workspace` claim is not exactly + `workspace-`, closing the gap at the one place every + session (HMAC and OIDC alike) is constructed, rather than in the OIDC + decoder alone or in this migration's backfill formula. This makes + `auth_context.workspace_id` an actually-enforced invariant instead of an + assumed convention for every workspace_id-scoped table + (`Email`, `Document`, `WebdavAccount`, `ProjectFolder`, + `CalendarConflictJudgment`, `CarddavAccount`), not just this PR's own + `Email` change. New test: + `tests/test_auth_real.py::test_build_auth_context_rejects_workspace_claim_not_derived_from_org`. + Two existing tests asserted specifically on a *different* concern + (`api/security.py`'s `_require_authoritative_workspace_scope`, which + unconditionally bars `session_verifier=="hmac"` from `/api/security/access-surface` + regardless of the workspace claim's value, and a data-quality-surface test + that only cared the workspace filter clause was present) but happened to + use a `workspace` value inconsistent with their own `org` claim; both were + updated to use an org-consistent workspace claim so each isolates the one + behavior it's actually testing. +- **Devin flagged that `services/email_import_service.py`'s + `_build_email_object` re-derives `workspace_id` from `organization_id` + instead of accepting the caller's already-verified + `auth_context.workspace_id`, reachable at `import_email_files`'s HTTP + route — confirmed accurate as an architecture observation, and now + provably inconsequential rather than merely "not currently + exploitable."** With the authentication-layer fix above, + `auth_context.workspace_id` is guaranteed equal to the derived value for + every session that reaches this code, for both the HMAC and OIDC paths — + not just "for every request path exercised today" as this ADR previously + (incorrectly) claimed. Threading the real `workspace_id` through + `import_email_files` → `import_email_uploads` → `_import_single_eml` → + `_build_email_object` (`imap_worker.py`'s background-poll path has no + `AuthContext` at all — there is no signed workspace to thread through + there, so it must keep deriving) remains a reasonable DRY improvement, + but it is a multi-call-site plumbing change with zero behavioral effect + now, not a fix for this one's narrower scope. +- **Correction: the "three production call sites" claim above missed a + fourth `Email`-inserting path, and Devin found it.** `backend/scripts/ + import_fixtures.py::process_zip_file` builds its own bulk-insert + `batch_values` dict independently of `backend/import_fixtures.py`'s + `email_obj = Email(...)` construction (already fixed) — a genuinely + separate script, not the same call site under a confusing shared + filename. Since `Email.workspace_id` is `NOT NULL`, every nonempty + archive import through this path failed at commit. Fixed the same way + (`workspace-` convention, included in both the insert + values and the `on_conflict_do_update` set). New test: + `tests/test_import_fixtures.py::test_process_zip_file_batch_insert_includes_workspace_id` + (the existing `test_process_zip_file` only ever exercised the empty-zip + path, which never reaches the insert — why this was missed originally). + Devin also re-flagged `noema_agent.py`'s `tool_search_mail`/ + `tool_read_mail`/`tool_content_graph_query` reading through + `Email.owner_filters()` without `workspace_id` — traced via `git blame` + to `b6cb4e6f` (2026-07-13, over a month before this PR): confirmed as + the identical, already-tracked `Email.owner_filters()` gap two + paragraphs above, not new exposure from this PR, so left deferred per + the same narrow-scope decision. + +- **Correction (Devin Review, `6df8f44a`/`62b74a05` round): the "Fixed: + `_auth_context_from_session_payload` now rejects (`401`) any session whose + `workspace` claim is not exactly `workspace-`" entry above + no longer describes current code — flag it as historical, not current + behavior.** Commit `b778fb69` ("fix: enforce workspace-safe Noema + identity") removed that exact-match rejection entirely: workspace + membership is now treated purely as the independently signed opaque claim + the verified session authority produces, never derived from or validated + against `organization_id`. The rationale is the same class of correction + CodeRabbit pushed on this PR's `Email` legacy-backfill thread — assuming + every real deployment mints `workspace` as `workspace-` + was itself the wrong premise; a genuinely independent external token + issuer is free to mint an opaque value that doesn't derive from `org` at + all, and rejecting such a session outright would be a false-positive + authentication failure, not a security improvement. The referenced test + was renamed accordingly: + `tests/test_auth_real.py::test_build_auth_context_rejects_workspace_claim_not_derived_from_org` + is now + `test_build_auth_context_accepts_independently_signed_workspace_membership`, + and asserts the opposite outcome (a session with `org="org-acme"`, + `workspace="workspace-project-blue"` is accepted, not rejected). The two + existing tests noted above (`api/security.py`'s + `_require_authoritative_workspace_scope` test and the data-quality-surface + test) are unaffected by this reversal — their own updates only fixed an + org-inconsistent workspace value, not this specific rejection behavior. + This ADR's downstream claims about `auth_context.workspace_id` being "an + actually-enforced invariant" for every `workspace_id`-scoped table are + narrowed by this reversal to: the claim is *present and well-formed*, not + that it's provably derived from or consistent with `organization_id`. + +- **Attachment reparse never indexed a successfully re-recognized + attachment's content into the content graph, unlike the initial import + path — flagged as informational by Devin Review on this PR ("confirm this + is intended"), confirmed real but out of scope for this PR, and closed + here as the tracked follow-up.** + `services/email_import_service.py::_append_email_content_graph` already + builds a `ContentNodeRecord`/`ContentSegmentRecord` graph for an + attachment that parses cleanly on first import, but + `attachment_reparse_worker.py::apply_reparsed_result` only ever updated + the `Attachment` row's own columns — a previously-quarantined attachment + that later reparses to `"parsed"` stayed invisible to content-graph-backed + search/AI-hub features even after successful recognition, despite + `AttachmentParseResult` carrying the same `parse_content` field the import + path indexes. Fixed by calling a new + `_append_reparsed_attachment_content_graph` from `apply_reparsed_result` + whenever the reparse result lands on `"parsed"`. It reuses the same + `services.content_graph.parse_content` helper the import path already + calls, plus a newly shared `content_graph_source_record_uid` (moved out of + `email_import_service.py`, where it was a private function, into + `services/content_graph/parser.py` as a public helper both call sites + import) — not a second indexing path, the same one with a second caller. + Since a persisted attachment's original position among its email's + siblings is not reliably reproducible after import, the reparse path keys + `source_record_uid` on the attachment's permanent `attachment_uid` alone + instead of the import path's message-id + list-position convention, and + sets the new records' `email_id` directly from the attachment's + already-loaded `email_id` column rather than through a transient `Email` + relationship append (the attachment here is already a persisted row, + unlike at import time, so there is no transient parent to defer FK + resolution through). New tests: + `test_reparse_that_lands_on_parsed_indexes_the_content_graph`, + `test_reparse_that_lands_on_parsed_with_blank_content_does_not_index_content_graph`, + `test_reparse_that_does_not_land_on_parsed_does_not_index_content_graph`. + Verification: full backend suite 1908 passed / 40 skipped, ruff clean. + +- **CodeRabbit's full review of the content-graph-indexing follow-up above + found two real correctness gaps in it, both fixed here.** (1) The reparse + embedding refresh regenerated `attachment.embedding` from + `attachment.content` rather than the resolved parse source text. + `apply_reparsed_result` only overwrites `attachment.content` when + `result.content` (a markup-stripped *display* string) is non-empty; a + `"parsed"` result whose display text strips to empty while its raw + `result.parse_content` does not (e.g. an attachment that is only markup, + no visible text nodes) left `attachment.content` at its stale, + still-base64-encoded retained value, so the embedding was generated from + base64 noise instead of the actual reparsed text — while the content graph + indexed the correct text, since `_append_reparsed_attachment_content_graph` + already resolved `result.parse_content or result.content` itself, matching + `email_import_service._extract_and_generate_embeddings`'s identical + resolution at import time. `process_reparse_pending_attachment` now + returns a `ReparseOutcome(parse_status, embedding_source_text)` instead of + a bare status string, carrying that same resolved text through to the + embedding refresh explicitly rather than re-deriving it (unreliably) from + the attachment row. New test: + `test_reparse_that_lands_on_parsed_with_markup_only_content_still_embeds_parse_content`. + (2) `0011_email_read_state.py`'s `downgrade()` dropped `emails.is_read` + unconditionally whenever the column and legacy `emails` table were both + present — including a same-named `is_read` column that predated this + revision entirely, which this revision's own `NOT EXISTS`-guarded + `upgrade()` therefore never touched, destroying that unrelated column and + its data on downgrade. `upgrade()` now tags the column it creates with a + `COMMENT ON COLUMN` provenance marker + (`_IS_READ_PROVENANCE_MARKER = "0011_email_read_state:added"`); + `downgrade()` drops the column only when that exact marker is present via + `col_description`, so it drops what this revision added and nothing else. + New real-Postgres test: + `test_legacy_email_read_state_downgrade_preserves_a_preexisting_column` + (pre-seeds a legacy `emails.is_read` column with data, runs upgrade then + downgrade, asserts both the column and its data survive). Also renamed + `email_import_service._generate_source_embedding` to the public + `generate_source_embedding` (CodeRabbit nitpick): it is a third + cross-module dependency `attachment_reparse_worker.py` imports, alongside + `content_graph_source_record_uid` and `append_knowledge_graph_edges`, so + the module boundary stays consistent when every cross-module helper is + public. Verification: full backend suite 1911 passed / 43 skipped + (`DATABASE_URL` unset, matching CI), and every test touched by this fix + passes in isolation against a real PostgreSQL 16 + pgvector database; ruff + clean. Running the full suite against that same real database in one + process reproduces one pre-existing, already-reported cross-file + test-ordering failure (`test_0001_initial_upgrade_succeeds_against_a_ + fresh_database` drops and recreates `email_records` mid-suite) — + orthogonal to this fix, not caused by it. + +## References (APA 7th) + +Freed, N., & Borenstein, N. (1996). *Multipurpose Internet Mail Extensions +(MIME) Part Two: Media Types* (RFC 2046). RFC Editor. +https://doi.org/10.17487/RFC2046 +RFC 2046 is the standard this ADR's "declared type" (the `Content-Type` +header on a MIME body part) refers to; it does not itself specify or require +content-sniffing, which is the gap this ADR addresses. diff --git a/docs/adr/README.md b/docs/adr/README.md index 4d461fff6..f27af8bb7 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -13,6 +13,7 @@ govern implementation. | [ADR-0002](0002-fitted-topic-artifact-consumption.md) | Conditionally consume only a versioned fitted topic artifact through a fail-closed adapter | Proposed | Target `PLANNED`; runtime `BLOCKED-UPSTREAM` | | [ADR-0003](0003-separate-topic-measurement-from-agenda-generation.md) | Keep statistical measurement separate from agenda generation | Proposed | Target and future capability `PLANNED`; no implementation authorization | | [ADR-0004](0004-status-weighted-calendar-conflicts.md) | Evaluate CalDAV VEVENT overlaps by occupying status; cancelled does not occupy | Accepted | `ACCEPTED-NARUON-POLICY`; advisory evaluate API only | +| [ADR-0005](0005-attachment-content-type-quarantine.md) | Quarantine attachments whose sniffed bytes disagree with their declared content type; address them via a new `attachment_uid` | Proposed | PR #1486 candidate; no protected-branch or release adoption evidence | The complete topic-intelligence requirements, architecture, contract, UML, conceptual ERD, security, test, and operability graph is indexed at diff --git a/docs/doctoring/reply_sla_physical_lease.md b/docs/doctoring/reply_sla_physical_lease.md new file mode 100644 index 000000000..b89b82bfe --- /dev/null +++ b/docs/doctoring/reply_sla_physical_lease.md @@ -0,0 +1,230 @@ +# Reply follow-up physical-connection lease + +Status: **Proposed**. Date: 2026-09-05. Decision owner: Naruon maintainers. +This is a scheduler decision record for PR #1486; it does not inherit or supplement attachment-quarantine ADR-0005. +The installed `adr-author` package lacks its required +`adr-identity.instructions.md`; its MADR context/options/consequences structure +is used without claiming allocator, state-machine or acceptance validation. + +## Context and problem statement + +PR [#1486](https://github.com/ContextualWisdomLab/naruon/pull/1486) at +`b32954dbf6066bc0d953887e8ca06820588f2c5f` contains the existing owner/workspace +sweep. Its direct base is `develop` at +`042b0c70531b229af3acbd0421a2f23098d848b3`. It is a broad, unmerged proposal; +the calendar, attachment, migration and Noema changes remain intact. Repairing +this scheduler does not validate those unrelated slices or supersede the PR. + +A user has sent a question and has not received a reply after the configured +deadline. A replica creates the source-linked follow-up task and commits. An +ordinary request then borrows that returned pool connection while the replica +continues its sweep on a different connection. The advisory lock still belongs +to the first connection. The final unlock consequently does not release it, +and the next replica cannot start follow-up work even though the first cycle +has finished. Reusing the same Python session object does not establish +physical connection ownership. + +The actual database reproduction observed `worker=[79], reader=79`, a persisted +blocked/urgent task, and `replica_can_lead=False` after `_sync()` returned. +The original 12 mocked scheduler tests passed and did not exercise this pool +interleaving. An independent read-only review also identified two existing +manual-request conflict failures: rollback expires the loaded emails before +fallback accesses them, and savepoint rollback already removes failed inserts +before the fallback calls `expunge()` again. + +## Requirements and constraints + +| Product requirement | Technical invariant | Acceptance evidence | +|---|---|---| +| Follow-ups keep progressing after a saved task | Lease remains on one physical connection across transaction boundaries | Competing replica cannot acquire during a real commit; can acquire after completion | +| One failed mailbox does not stop every mailbox | Healthy transaction failure rolls back; later owner state is reloaded | Real division-by-zero and explicit rollback cases continue to another owner | +| Lost coordination cannot authorize more work | Disconnected backend aborts this sweep | Actual `pg_terminate_backend` of the task-owned worker yields an invalidated DBAPI error and no second owner write | +| A manual request must not lose or duplicate a follow-up | Conflict recovery preserves existing task identity and processes both source workspaces | Real concurrent insert through a second session, bulk and savepoint paths | +| Small connection pools remain usable | One worker connection, no second lease checkout | Pool size 1, overflow 0, real writes and cancellation cases | + +Existing opaque task identifiers, owner/organization/workspace queries, 48-hour +default, limits and scheduling interval are retained. No schema, provider +credential, model routing, API payload or browser authorization change is made. +The current Python service is repaired in place; no new Python computation +core or framework is introduced. + +## Considered options and decision + +The chosen coordination change binds the existing session to the connection +already checked out for the sweep. Transaction ownership remains with the +escalation service. A normal cycle rolls back any trailing read transaction, +then requires a literal boolean successful unlock. Any exception, including +cancellation before acquisition acknowledgement, invalidates that connection +**inside** the session context, before shielded close/rollback can wait. + +| Option | Commits retain lease | One pool slot | Existing task transactions | Decision | +|---|---|---|---|---| +| Bind existing session to one checked-out connection | Yes | Yes | Preserved | Chosen | +| Independent lock connection plus ordinary work session | Yes | No | Preserved | Extra checkout can exhaust a one-slot pool | +| Transaction-scoped advisory lock | No | Yes | Lost at each task commit | Does not cover the whole sweep | +| Process-local mutex | Only within one process | Yes | Preserved | Does not coordinate replicas | + +An independent lock connection offers a clean separation of responsibilities +and is reasonable for a separately budgeted pool. This scheduler has no such +budget or necessity: binding the session avoids the extra resource and new +abstraction. Transaction locks have automatic cleanup, but changing all task +commits into one sweep transaction would also change failure isolation and +visibility. A local mutex is cheap but cannot protect multiple deployments. + +Conflict recovery is a separate sub-decision. The shared escalation service +refreshes each selected mail record after bulk rollback before attempting its +existing fallback. Both redundant `expunge()` calls are removed because actual +savepoint rollback already detaches failed new tasks. The scheduler reloads +configuration before each workspace using a forced database read, so a +successfully recovered conflict does not leave the next workspace reading +expired attributes and a deleted configuration cannot remain authorized by the +session cache. A real second-session deletion after the first committed task +first reproduced two workspace executions, then passed with only one. +Healthy owner failures +record only the exception class, not raw SQL, credentials or mail content. + +```mermaid +sequenceDiagram + participant Sweep as Reply scheduler + participant Conn as Checked-out DB connection + participant Reader as Ordinary request + participant Replica as Other replica + Sweep->>Conn: Acquire session advisory lease + loop Each owner workspace + Sweep->>Conn: Select pending replies and commit tasks + Reader->>Conn: Cannot borrow while checked out + Replica->>Conn: Try same lease (false) + end + alt Normal completion + Sweep->>Conn: Roll back trailing read and confirm unlock + else Cancellation or failure + Sweep->>Conn: Invalidate before session cleanup + end + Replica->>Conn: Next cycle can acquire lease +``` + +## Reproduction and verification + +Use a fresh isolated PostgreSQL 16 + pgvector instance, not an operator database. +The task runner uses a digest-pinned image, read-only root, non-root user, +`no-new-privileges`, task-owned tmpfs storage, loopback-only random port and +random generated test credentials. Its exact Compose project is +`naruon-pr1486-lease-dbimpf`; cleanup targets only that project. No global prune, +shared database reset or operator environment file is used. + +From `backend`, with only the isolated test database URL and generated test +session secret supplied: + +```sh +uv sync --locked +uv run --locked python scripts/migrate_db.py +uv run --locked python scripts/migrate_db.py +uv run --locked coverage run --branch \ + --source=services.reply_sla_scheduler,services.reply_sla_escalation_service \ + -m pytest -q -W error -ra --tb=short \ + tests/test_reply_sla_scheduler.py tests/test_reply_sla_scheduler_postgres.py \ + tests/test_reply_sla_escalation_edges.py \ + tests/test_tasks_api.py tests/test_reply_tracking.py \ + tests/test_reply_tracking_service.py tests/test_db_session.py \ + tests/test_alembic_migrations.py tests/test_bootstrap_db.py +uv run --locked coverage report -m +``` + +The new PostgreSQL test checks the actual `0022_noema_orchestrator_gateway` +migration receipt and does not create ORM metadata or skip on unavailable DB. +The older tasks API smoke still has metadata bootstrap and synthetic unit-like +inputs; its passing result is not used as migration, live-auth or realistic +customer-mail evidence. The runner migrates first, so it cannot hide a broken +fresh upgrade for this receipt. + +The test mail is a short observed question from a public mailing-list message. +Its question and original UTC date are preserved. Names, addresses, subject and +message identity are anonymized; the same source is replayed in two isolated +workspaces. Tracking/escalation clocks are fixed three days after that message. +This is a concurrency regression, not a representative inbox, classifier +accuracy, notification delivery, throughput or p95 acceptance test. + +Evidence retained under `/private/tmp/naruon-scheduler-lease.dbIMPF`: + +| Receipt | Observation | +|---|---| +| `baseline.xml` | Unchanged head: fresh/repeat migrations and 12 mocked tests pass | +| `lease_red.xml` / `lease_green.xml` | Real commit lends held connection: 1 failed → 1 passed | +| `owner_red.xml` / `owner_green.xml` | Healthy rollback and disconnect handling: 3 failed → 4 passed including original lease regression | +| `conflict_red.xml` / `conflict_green.xml` | Bulk and nested real insert races: 2 failed → 2 passed | +| `deletion_red.xml` / `deletion_green.xml` | Committed mailbox deletion between two workspaces: 1 failed → 1 passed | +| `full_candidate.xml` | Failed collection, not runtime GREEN: Starlette deprecated fallback client | +| `expanded_candidate.xml` | Intermediate 149 passed; scheduler-only 103 statements / 26 branches 100% | +| `reviewed_candidate.xml` | 165 passed, zero failures/errors/skips, 3.22 seconds; both changed runtime modules total 259/259 statements and 82/82 branches, zero exclusions | + +The final runtime definitions have docstrings on all 24 classes/functions/methods +(scheduler 10/10, escalation service 14/14). The 12 PostgreSQL cases execute +actual migrated database writes, ownership probes or cancellation boundaries; +the 12 new scripted edge cases are explicitly unit tests, not real races. +These percentages cover these two modules only, not repository-wide test, +docstring, edge-case completeness, browser behavior or protected CI. The exact +post-commit rerun and SHA-256 receipts are recorded on the PR head. + +A later mechanical naming pass accidentally renamed the existing `db.models` +import in the new unit-only file. Its isolated collection failed with +`ModuleNotFoundError`; intermediate local commit `7ce6592` is **not** a GREEN +receipt. The import is restored in a forward correction, preserving the +framework/package boundary. The earlier 165-pass receipt is historical, and +the full suite must run again on the corrected committed head before push. + +The first expanded test collection exposed Starlette 1.3.1's fallback to +deprecated `httpx` when its intended `httpx2` test client was absent. The existing +Naruon #1469 development pin `httpx2==2.5.0` is reused in the dev dependency group +and lockfile, and the obsolete warning-ignore entry is removed. Runtime HTTP +clients and production dependencies are unchanged. No warning suppression is +accepted as the repair. Later receipts belong to their own exact source and +must not inherit intermediate coverage totals. + +## Risks, consequences and follow-up + +| Risk / cost | Effect and mitigation | Owner | +|---|---|---| +| One connection remains checked out for the sweep | Bounded resource cost; actual one-slot test checks no second checkout | Naruon runtime | +| Multiple owners share the global lease | Serial sweep throughput ceiling; measure before designing per-owner coordination | Naruon runtime | +| Lost DB during an already committed operation | This does not provide exactly-once execution; existing unique source-task identity and subsequent sweep reconcile state | Naruon task service | +| Cancellation while teardown stalls | Test gates real session `__aexit__`; proves invalidation order, not a real network blackhole | Naruon runtime | +| Transaction-pooling proxy changes backend identity | Requires direct PostgreSQL or session-affine pooling; proxy compatibility is unverified | Deployment owner | +| Historical proposal text mistaken for release evidence | ADR-0005 is Proposed; exact-head CI, reviews and protected merge are still required | Naruon maintainers | + +If a rollout fails, stop the affected background worker under the operator's +normal deployment controls and preserve queued mail/task data. Do not restore +the known connection-lending implementation, delete task rows to clear a lease, +or terminate unscoped database sessions. Repair the diagnosed owner and rerun +the same migration, regression and deployment verification before resuming. + +The existing sibling import-lock repair remains in PR #1317, not copied here. +Attachment worker evidence in #1469 does not certify this scheduler or #1486's +other workers. The canonical Gap ledger remains owned by PR #1602; this branch +does not create a second competing baseline. Hosted checks, independent review, +protected merge, immutable release and deployed behavior remain separate gates. + +## References (APA 7th) + +PostgreSQL Global Development Group. (n.d.). *Explicit locking: Advisory locks* +(PostgreSQL 16 documentation). Retrieved September 5, 2026, from +https://www.postgresql.org/docs/16/explicit-locking.html#ADVISORY-LOCKS + +SQLAlchemy authors. (n.d.). *Session basics: Committing and rolling back* +(SQLAlchemy 2.0 documentation). Retrieved September 5, 2026, from +https://docs.sqlalchemy.org/en/20/orm/session_basics.html + +SQLAlchemy authors. (n.d.). *Transactions and connection management* +(SQLAlchemy 2.0 documentation). Retrieved September 5, 2026, from +https://docs.sqlalchemy.org/en/20/orm/session_transaction.html + +Public archive author [identity anonymized]. (2014, April 27). +*Postgresql the right tool (queue using advisory_locks + long transactions)* +[Mailing-list message]. PostgreSQL public archives. +https://www.postgresql.org/message-id/CANsFX049q7C_vJAtn2BSJy_4hQPu0%3DJNtv-Lyzb%3DgbZu-be30A%40mail.gmail.com + +Only the short cited question is included, not the full copyrighted message or +an assumed-redistributable paper PDF. Context7 was quota-unavailable. DeepWiki +helped locate callers but its claim that a same-session `finally` guarantees +unlock was contradicted by the exact-head PostgreSQL RED; source and executable +evidence take precedence. The tested installed SQLAlchemy version is 2.0.51; +the currently served 2.0 documentation identifies itself as 2.0.52. diff --git a/docs/doctoring/status-weighted-calendar-conflicts.md b/docs/doctoring/status-weighted-calendar-conflicts.md index 7a2b3ad3b..d32c8e866 100644 --- a/docs/doctoring/status-weighted-calendar-conflicts.md +++ b/docs/doctoring/status-weighted-calendar-conflicts.md @@ -4,7 +4,9 @@ Naruon evaluates a proposed calendar commitment against a bounded set of existing commitments and returns one of three deterministic outcomes: `available`, `blocked`, or `review_required`. The decision is advisory evidence only. It does not mutate, cancel, reschedule, accept, or decline any provider event. -The public endpoint is `POST /api/calendar/conflicts/evaluate`. It is mounted behind Naruon's existing private API authentication dependency. Inputs are either structured commitments (`commitment_id`, timezone-aware `start_at`/`end_at`, status) or iCalendar/ICS `proposed_ics` / `existing_ics` VEVENT documents. Occupying statuses are `confirmed`, `tentative`, and `desired`. RFC 5545 `STATUS:CANCELLED` is accepted and does not occupy the interval. Existing evidence is capped at 500 commitments per request. The Calendar coordination view selects a signed, source-backed writeback source for the authenticated user/workspace and does not present canned ICS pairs as production coordination evidence. Known `.ics` pairs remain test fixtures only. +The public endpoint is `POST /api/calendar/conflicts/evaluate`. It is mounted behind Naruon's existing private API authentication dependency. Inputs are either structured commitments (`commitment_id`, timezone-aware `start_at`/`end_at`, status) or iCalendar/ICS `proposed_ics` / `existing_ics` VEVENT documents. Occupying statuses are `confirmed`, `tentative`, and `desired`. RFC 5545 `STATUS:CANCELLED` is accepted and does not occupy the interval. Existing evidence is capped at 500 commitments per request (`MAX_EXISTING_COMMITMENTS` in `services/calendar_conflict_policy.py`, shared by the REST endpoint, the ICS parser, and the Noema tool below). The Calendar coordination view selects a signed, source-backed writeback source for the authenticated user/workspace and does not present canned ICS pairs as production coordination evidence. Known `.ics` pairs remain test fixtures only. + +`POST /api/calendar/conflicts/evaluate` itself remains exactly as stateless as described above -- calling it never writes anything. A separate, additive surface persists a decision as a correctable record: `POST /api/calendar/conflicts/judgments` runs the same deterministic policy and stores the result as a `CalendarConflictJudgment` row (`status_code`: `proposed` → `confirmed`/`overridden`/`dismissed`), scoped to `user_id` + `organization_id` + `workspace_id`; `GET /api/calendar/conflicts/judgments` lists the caller's own judgments (newest-first, bounded to 200) and `GET .../judgments/{judgment_uid}` fetches one by its opaque uid regardless of that bound; `POST .../judgments/{judgment_uid}/corrections` records a human override/confirm/dismiss as a `CalendarConflictCorrection` row with a full before/after JSON snapshot. Noema's `check_calendar_conflict` tool fails closed with `calendar_authoritative_evidence_unavailable` and `review_required`: the current runner has an outbound CalDAV write seam but no scoped inbound provider-calendar reader, so conversational mail/task evidence cannot establish that a time is available. ## Standards traceability @@ -26,12 +28,14 @@ This policy deliberately prevents a convenience feature from silently breaking a The decision path is deterministic and uses no LLM judgment. It accepts only scheduling evidence needed for the decision; it does not require email bodies, participant names, provider credentials, or calendar descriptions. The endpoint rejects naive timestamps, invalid/non-positive intervals, unsupported statuses, oversized evidence batches, extra request fields, and a missing proposed source through the transport/service validation layers. A missing proposal returns `calendar_proposed_source_missing` as HTTP 422; the handler does not use `assert`, so optimized bytecode cannot strip the guard. Customer-facing results include a concrete next action rather than a generic warning. -No database objects or migrations are introduced. No provider is contacted. Rollback must disable or remove the frontend integration first (`frontend/src/components/calendar/types.ts`, `constants.ts`, `helpers.ts`, and `CalendarCoordinationView` wiring in `CalendarLayout`), then remove the backend route registration, ICS parser, and policy module. Existing calendar data is unaffected because the slice is read-only for provider and database state. +`POST /api/calendar/conflicts/evaluate` itself introduces no database objects and contacts no provider; it remains read-only for provider and database state, exactly as originally shipped. The judgment/correction persistence slice does introduce database objects: Alembic `0018_calendar_conflict_judgments` creates `calendar_conflict_judgments` and `calendar_conflict_corrections` (both scoped by `user_id`/`organization_id`/`workspace_id`, with `calendar_conflict_corrections` foreign-keyed to its parent judgment). Rollback of the stateless `/evaluate` endpoint is unchanged: disable or remove the frontend integration first (`frontend/src/components/calendar/types.ts`, `constants.ts`, `helpers.ts`, and `CalendarCoordinationView` wiring in `CalendarLayout`), then remove the backend route registration, ICS parser, and policy module. Rollback of the persistence slice additionally requires removing `api/calendar_conflicts.py`'s `/judgments*` routes and `services/calendar_conflict_judgment_service.py`, then running `alembic downgrade` past `0018_calendar_conflict_judgments` to drop both tables -- downgrading discards any judgments/corrections already recorded, so it is a destructive operation once real customer corrections exist, not a no-op like the stateless endpoint's rollback. ## Verification evidence required before merge The exact unchanged PR head must prove known `.ics` pairs (cancelled allows, tentative review, confirmed blocks, adjacent allow), realistic overlap, adjacency, timezone-offset equivalence, deterministic ordering, self-update, invalid interval, unsupported status, API validation, authentication, and bounded-batch behavior. Repository-required CI, security, coverage, supply-chain, package, and independent current-head review gates remain authoritative; predecessor or queued evidence is non-passing. The policy decision is recorded in [ADR-0004](../adr/0004-status-weighted-calendar-conflicts.md). +The judgment/correction persistence slice additionally requires: `alembic heads` resolving to one head after `0018_calendar_conflict_judgments`; workspace-scoped isolation (a judgment/correction query never returns another `workspace_id`'s rows even under a matching `user_id`+`organization_id`); the row lock (`SELECT ... FOR UPDATE`) on `apply_correction`'s judgment lookup; `validate_correction_coherence()` rejecting every mismatched `status_code`/`decision_code` pair (an override with no replacement decision, or a confirm/dismiss that also tries to change the decision); an override that repeats the judgment's *current* decision is a distinct, accepted case (coherence still passes, since `decision_code` is present), but `apply_correction` must leave `reason_code`/`recommended_action` untouched for it rather than replacing them for no real change; and `default_recommended_action()` as the only place `recommended_action` text is derived from a `decision_code`, on both the original evaluation path and the correction path. + ## References (APA 7th) Daboo, C. (Ed.). (2009). *iCalendar transport-independent interoperability protocol (iTIP)* (RFC 5546). RFC Editor. https://doi.org/10.17487/RFC5546 diff --git a/pytest.ini b/pytest.ini new file mode 100644 index 000000000..cf444ed81 --- /dev/null +++ b/pytest.ini @@ -0,0 +1,4 @@ +[pytest] +asyncio_default_fixture_loop_scope = function +markers = + postgres: requires a reachable PostgreSQL DATABASE_URL diff --git a/registered_agents.json b/registered_agents.json index 312a0488d..80bc0a691 100644 --- a/registered_agents.json +++ b/registered_agents.json @@ -5,16 +5,17 @@ "framework": "pydantic-ai", "framework_license": "MIT", "entrypoint": "services.noema_agent:run_noema_agent", - "description": "General-purpose assistant that reasons over the naruon workspace (mail, content graph, tasks) on the tenant's configured LLM provider and dispatches opt-in, audit-logged writebacks through the self-hosted runner.", + "description": "General-purpose assistant that reasons over scoped naruon workspace mail, content graph, and tasks, refuses calendar availability claims until an authoritative provider read seam exists, and dispatches opt-in, audit-logged writebacks through the self-hosted runner.", "capabilities": [ "mail.search", "mail.read", "content_graph.query", "tasks.read", "tasks.update", + "calendar.conflict_check", "calendar.writeback" ], - "provider_source": "runtime_llm_provider", + "provider_source": "contextual_orchestrator_gateway", "writeback": { "opt_in": true, "audit_logged": true diff --git a/scripts/ci/pr_governance_gate.sh b/scripts/ci/pr_governance_gate.sh index a66142ca8..5dead1f0c 100644 --- a/scripts/ci/pr_governance_gate.sh +++ b/scripts/ci/pr_governance_gate.sh @@ -258,7 +258,7 @@ IS_DRAFT="$(printf '%s' "$PR_JSON" | jq -r '.isDraft')" REVIEW_DECISION="$(printf '%s' "$PR_JSON" | jq -r '.reviewDecision // ""')" if [ "$IS_DRAFT" = "true" ]; then - add_blocker 'Draft PR: merge automation is paused.' + add_waiting 'Draft PR: merge automation is paused.' fi if [ "$MERGE_STATE" = "BEHIND" ]; then diff --git a/scripts/ci/test_pr_governance_gate.sh b/scripts/ci/test_pr_governance_gate.sh index 8fb42aa9b..970f5e81f 100644 --- a/scripts/ci/test_pr_governance_gate.sh +++ b/scripts/ci/test_pr_governance_gate.sh @@ -17,6 +17,9 @@ args="$*" if [ "$1" = "pr" ] && [ "$2" = "view" ]; then case "${GH_SCENARIO:-pass}" in + draft) + printf '{"number":42,"state":"OPEN","headRefOid":"%s","isDraft":true,"mergeable":"MERGEABLE","mergeStateStatus":"CLEAN","reviewDecision":"","statusCheckRollup":[]}' "$head_sha" + ;; changes_requested) printf '{"number":42,"state":"OPEN","headRefOid":"%s","isDraft":false,"mergeable":"MERGEABLE","mergeStateStatus":"CLEAN","reviewDecision":"CHANGES_REQUESTED","statusCheckRollup":[]}' "$head_sha" ;; @@ -453,6 +456,19 @@ assert_failed_checks_create_marker_comment() { assert_not_in_file '^pr merge' "$temp_dir/gh.log" } +assert_draft_pr_waits_without_false_failure() { + local temp_dir + temp_dir="$(mktemp -d)" + run_gate draft "$temp_dir" + + assert_exit_code 0 "$temp_dir" + assert_in_file 'Draft PR: merge automation is paused.' "$temp_dir/gh.log" + assert_in_file 'status=in_progress' "$temp_dir/gh.log" + assert_not_in_file 'conclusion=failure' "$temp_dir/gh.log" + assert_not_in_file 'PR governance metadata gate is not ready' "$temp_dir/gh.log" + assert_not_in_file '^pr merge' "$temp_dir/gh.log" +} + assert_existing_marker_comment_is_patched() { local temp_dir temp_dir="$(mktemp -d)" @@ -923,6 +939,7 @@ assert_head_change_during_evaluation_skips_stale_publication assert_closed_during_evaluation_skips_stale_publication assert_startup_failure_creates_marker_comment assert_failed_checks_create_marker_comment +assert_draft_pr_waits_without_false_failure assert_existing_marker_comment_is_patched assert_resolved_marker_comment_is_updated_on_ready_gate assert_coderabbit_pending_waits_without_hard_comment diff --git a/task_agent_mapping.json b/task_agent_mapping.json index 71e58921a..90fa5b971 100644 --- a/task_agent_mapping.json +++ b/task_agent_mapping.json @@ -3,5 +3,6 @@ "mail.triage": "noema-general-agent", "mail.search": "noema-general-agent", "tasks.followup": "noema-general-agent", + "calendar.conflict_check": "noema-general-agent", "calendar.writeback": "noema-general-agent" } diff --git a/tests/test_stacked_pr_workflow_contract.py b/tests/test_stacked_pr_workflow_contract.py new file mode 100644 index 000000000..d54014e0d --- /dev/null +++ b/tests/test_stacked_pr_workflow_contract.py @@ -0,0 +1,27 @@ +"""Regression coverage for governed checks on stacked pull requests.""" + +from pathlib import Path +import re + + +REPO_ROOT = Path(__file__).resolve().parents[1] +GOVERNED_PULL_REQUEST_WORKFLOWS = ( + "app-ci.yml", + "bandit.yml", + "dependency-review.yml", + "docker-publish.yml", +) + + +def test_governed_pull_request_workflows_accept_stacked_base_branches() -> None: + """Required repository checks must run for every PR base, including stacks.""" + for name in GOVERNED_PULL_REQUEST_WORKFLOWS: + workflow = (REPO_ROOT / ".github" / "workflows" / name).read_text() + pull_request_trigger = re.search( + r"(?ms)^ pull_request:\s*$\n(?P(?:^ .*$\n)*)", + workflow, + ) + assert pull_request_trigger is not None, f"{name} must run on pull_request" + assert "branches:" not in pull_request_trigger.group("body"), ( + f"{name} must not exclude stacked PR base branches" + )