Skip to content
Open
Show file tree
Hide file tree
Changes from 39 commits
Commits
Show all changes
48 commits
Select commit Hold shift + click to select a range
cfacf01
Node: preserve the modification time when rehashing
khsrali Aug 7, 2026
cf45eb3
Collab: add the collab profile options and the per-profile state file
khsrali Aug 7, 2026
0c1ebec
Collab: add the wire protocol and the HTTP transport that carries it
khsrali Aug 7, 2026
cd79600
Collab: select and export the nodes a peer is owed since its cursor
khsrali Aug 7, 2026
175ac9f
Archive: reject a thin collab delta in verdi archive import
khsrali Aug 7, 2026
00a8cfb
Collab: negotiate the missing UUIDs and import a thin delta
khsrali Aug 7, 2026
ddf9b49
Collab: add the endpoint that serves pull negotiations and push imports
khsrali Aug 7, 2026
d257a18
Collab: supervise the collab endpoint as a watcher of the daemon
khsrali Aug 7, 2026
0a02773
Collab: add verdi collab init and the join code that founds or joins
khsrali Aug 7, 2026
139e9c9
Collab: add the immutable collab.policy option, fixed at creation
khsrali Aug 7, 2026
9b166e3
Collab: stamp peer addresses and gossip the roster on every sync
khsrali Aug 7, 2026
0a103bc
Collab: add verdi collab pull and verdi collab push
khsrali Aug 7, 2026
1644259
Collab: gate the handshake on the peer's archive format version
khsrali Aug 7, 2026
c86861e
Collab: add verdi collab log and compact the log as it grows
khsrali Aug 7, 2026
98865ba
Collab: add verdi collab rotate and rekey to replace the token
khsrali Aug 7, 2026
44de161
Collab: add a collab section to verdi status
khsrali Aug 7, 2026
2e1ed2c
Collab: record tombstones when nodes are deleted, to stop re-delivery
khsrali Aug 7, 2026
f315f76
Collab: replicate node extras between peers under the sync policy
khsrali Aug 7, 2026
94b4788
Collab: replicate curated group membership under the grow policy
khsrali Aug 7, 2026
bfa730b
Collab: map computers so a pulled calculation can be a cache hit
khsrali Aug 7, 2026
d8c9785
Collab: cap concurrent endpoint work with expiring slots
khsrali Aug 7, 2026
4e71c64
Docs: add the how-to for sharing provenance with collab peers
khsrali Aug 7, 2026
a99432d
Collab: a three-member end-to-end suite over the real wire
khsrali Aug 10, 2026
d7dfde2
🐛 Collab: hold every route against the collab UUID
khsrali Aug 10, 2026
d64cc55
🐛 Collab: one lock for every collab configuration write
khsrali Aug 10, 2026
6f0e2cb
🐛 Collab: refuse a push at its first request, and sweep what one leav…
khsrali Aug 10, 2026
168f0c5
🐛 Collab: hold a serving slot for the peer, not for the request
khsrali Aug 10, 2026
b62188c
🐛 Collab: let a peer say that its negotiation is over
khsrali Aug 10, 2026
a466f3e
🐛 Collab: one bad peer no longer ends the round
khsrali Aug 10, 2026
d93bc2a
🐛 Collab: renegotiate the extras a retried push had negotiated
khsrali Aug 10, 2026
ce03347
🐛 Collab: read the collab state at the instant a sync describes
khsrali Aug 10, 2026
cbf9f9c
🐛 Collab: the join code is asked for, not printed by `verdi status`
khsrali Aug 10, 2026
fa86096
🧪 Collab: pin why the claim carries every tombstone
khsrali Aug 10, 2026
66549fb
🐛 Collab: a withheld seed stops disabling the delta cache
khsrali Aug 10, 2026
95c5200
🐛 Collab: report the sealed processes no delta can carry
khsrali Aug 10, 2026
ea59a9c
🐛 Collab: a policy pydantic can build, a log the bug report carries, …
khsrali Aug 11, 2026
2da1799
👌 Collab: name the computers that arrived through the collab
khsrali Aug 11, 2026
eb9d122
🐛 Collab: a computer map cannot name a computer this profile lacks
khsrali Aug 11, 2026
184cdbd
📚 Collab: say that a node's PK is local to the profile it lands in
khsrali Aug 11, 2026
e00cb76
🐛 Collab: the 401 says that it closes the connection
khsrali Aug 11, 2026
3b9d60a
🐛 Collab: offer the extras a peer without a cursor may already hold
khsrali Aug 11, 2026
be5a3ca
🐛 Collab: refuse tombstoned nodes at the diff, not in the claim
khsrali Aug 11, 2026
fe65e9d
🐛 Collab: what a delta offers, and when it must be renegotiated
khsrali Aug 12, 2026
6f3e29d
🐛 Collab: the HTTP layer answers rather than hangs
khsrali Aug 12, 2026
e03825a
🐛 Collab: the locks say what they know, and the roster survives a bad…
khsrali Aug 12, 2026
8b5057f
🐛 Collab: one sync at a time, and init never founds over a membership
khsrali Aug 12, 2026
bb587e9
🧪 Collab: the convergence oracle compares content, not only shape
khsrali Aug 12, 2026
53df008
📚 Collab: the one-command rule, and what a tombstone does not gate
khsrali Aug 12, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -64,3 +64,4 @@ docker-bake.override.json

# dev files
.dev
.aep/
28 changes: 28 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
@@ -1 +1,29 @@
@AGENTS.md


## Code

The smallest change that satisfies the requirement. Prefer extending an existing function or module over adding a new one. No abstraction, configuration knob, or extensibility hook until there is a second caller that needs it.

No defensive code for conditions that cannot currently occur. No compatibility shims for versions we do not support.

Comments explain *why*, never *what*. If a comment restates the line below it, delete the comment. If the code needs a comment to be readable, first try renaming things.

Delete code that a change makes dead. Do not leave it commented out — that is what git is for.

## Tests

Test the functionality, not the implementation. The minimum suite that would actually fail if the feature broke, and no more.

Mock anything slow or external: network, filesystem beyond a tmpdir, databases, subprocesses, time, third-party APIs. CI time is a real cost.

One test per behavior. No parametrized matrices covering combinations that cannot differ. No test helper frameworks, custom assertion DSLs, or fixture hierarchies built for a single test file — a test that needs its own framework to be readable is over-engineered.

A test that cannot fail is worse than no test. Before adding one, know what break it catches.

It's better to run pytest via "-n auto" to speed up


## Development phases

Don't worry about migration between development phases. All phases in this PR are still a PR, there're no users yet.
333 changes: 333 additions & 0 deletions docs/source/howto/collaborate.rst

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions docs/source/howto/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ How-To Guides
data
query
share_data
collaborate
archive_profile
visualising_graphs
installation
Expand Down
26 changes: 26 additions & 0 deletions docs/source/reference/command_line.rst
Original file line number Diff line number Diff line change
Expand Up @@ -106,6 +106,32 @@ Below is a list with all available subcommands.
test Run tests for the given code to check whether it is usable.


.. _reference:command-line:verdi-collab:

``verdi collab``
----------------

.. code:: console

Usage: [OPTIONS] COMMAND [ARGS]...

Share provenance with the peers of a collab.

Options:
--help Show this message and exit.

Commands:
init Set up a profile as part of a collab.
link Print the code that admits a newcomer to this collab.
log Show the history of the pulls, pushes and extras refreshes of the...
map-computer Treat calculations that ran on a peer computer as if they ran on a...
peer Correct the entries of the peers of the collab.
pull Fetch the new sealed provenance of peers and import it.
push Send the new sealed provenance of this profile to peers.
rekey Adopt the new token of a collab whose token was rotated.
rotate Retire the token of the collab and mint a new one.


.. _reference:command-line:verdi-computer:

``verdi computer``
Expand Down
1 change: 1 addition & 0 deletions src/aiida/cmdline/commands/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@
cmd_bug_report,
cmd_calcjob,
cmd_code,
cmd_collab,
cmd_computer,
cmd_config,
cmd_daemon,
Expand Down
20 changes: 20 additions & 0 deletions src/aiida/cmdline/commands/cmd_archive.py
Original file line number Diff line number Diff line change
Expand Up @@ -465,6 +465,18 @@ def _gather_imports(archives: list[str], webpages: list[str] | None) -> list[tup
return final_archives


def _is_collab_thin_delta(archive_path: str) -> bool:
"""Return whether the archive is a thin collab delta, which only ``verdi collab pull`` imports completely."""
from aiida.tools.archive.abstract import get_format

try:
with get_format().open(archive_path, mode='r') as reader:
return bool(reader.get_metadata().get('creation_parameters', {}).get('collab_thin_delta'))
except Exception:
# An archive this version cannot read yet is handled (or refused) by the import machinery itself.
return False


def _import_archive_and_migrate(
ctx: click.Context, archive: str, web_based: bool, import_kwargs: dict, try_migration: bool
) -> None:
Expand Down Expand Up @@ -502,6 +514,14 @@ def _import_archive_and_migrate(
echo.echo_success('archive downloaded, proceeding with import')

archive_path = str(archive_path)

if _is_collab_thin_delta(archive_path):
echo.echo_critical(
f'{archive} is a collab transfer delta, not a general-purpose archive: importing it here would '
'silently lose the links it carries to nodes outside of it. It is imported by `verdi collab pull` '
'on the destination profile.'
)

echo.echo_report(f'starting import: {archive}')
try:
_import_archive(archive_path, archive_format=archive_format, **import_kwargs)
Expand Down
1 change: 1 addition & 0 deletions src/aiida/cmdline/commands/cmd_bug_report.py
Original file line number Diff line number Diff line change
Expand Up @@ -226,6 +226,7 @@ def _get_log_files() -> dict[str, pathlib.Path]:
'circus': filepaths['circus']['log'],
'daemon': filepaths['daemon']['log'],
'broker_service': filepaths['broker_service']['log'],
'collab': filepaths['collab']['log'],

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔒 Security & Privacy | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Look for token, authorization header, or join-code material reaching the collab log.
set -euo pipefail

echo '=== logging calls in the collab package ==='
rg -nP -C4 'LOGGER\.(debug|info|warning|error|exception|critical)' --type=py src/aiida/tools/collab

echo '=== token / auth header handling near logging ==='
rg -nP -C6 '\b(token|Authorization|bearer|Bearer|join_code|JoinCode)\b' --type=py src/aiida/tools/collab/server.py src/aiida/tools/collab/endpoint.py src/aiida/tools/collab/client.py

echo '=== does the HTTP server log request lines or headers? ==='
rg -nP -C6 'log_message|log_request|BaseHTTPRequestHandler|handle_one_request' --type=py src/aiida/tools/collab

Repository: aiidateam/aiida-core

Length of output: 24540


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo '=== collab server request and logging implementation ==='
sed -n '190,292p' src/aiida/tools/collab/server.py
sed -n '530,558p' src/aiida/tools/collab/server.py

echo '=== all logger calls in collab sources ==='
rg -n -P 'LOGGER\.[A-Za-z_]+\s*\(' src/aiida/tools/collab --type=py

echo '=== bug-report log path construction and logger configuration ==='
sed -n '190,245p' src/aiida/cmdline/commands/cmd_bug_report.py
rg -n -P -C5 'collab|filepaths|log' src/aiida/cmdline/commands/cmd_bug_report.py src/aiida --glob '*.py' | head -n 300

echo '=== request-derived values used in exceptions/logging ==='
rg -n -P -C4 'raise |LOGGER\.|log_message|send_error|detail|str\(exception\)|self\.path|self\.headers|self\.rfile' src/aiida/tools/collab/server.py src/aiida/tools/collab/endpoint.py

Repository: aiidateam/aiida-core

Length of output: 50376


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo '=== collab secret and join-code references ==='
rg -n -P '\b(join[_ -]?code|token|authorization|bearer)\b' src/aiida/tools/collab src/aiida/cmdline --type=py --glob '!cmd_bug_report.py' | head -n 250

echo '=== collab logging calls with source expressions ==='
python3 - <<'PY'
import ast
from pathlib import Path

root = Path('src/aiida/tools/collab')
for path in sorted(root.glob('*.py')):
    tree = ast.parse(path.read_text(), filename=str(path))
    for node in ast.walk(tree):
        if isinstance(node, ast.Call) and isinstance(node.func, ast.Attribute):
            receiver = node.func.value
            if isinstance(receiver, ast.Name) and receiver.id == 'LOGGER':
                args = [ast.get_source_segment(path.read_text(), arg) for arg in node.args]
                print(f'{path}:{node.lineno}: LOGGER.{node.func.attr}({", ".join(args)})')
PY

echo '=== logging configuration mentioning collab handlers or files ==='
rg -n -P -C5 'collab.*log|log.*collab|AIIDA_LOGGER|FileHandler|RotatingFileHandler' src/aiida --type=py | head -n 350

echo '=== request logging implementation references ==='
python3 - <<'PY'
import inspect
from http.server import BaseHTTPRequestHandler
for name in ('log_request', 'log_error', 'log_message', 'send_response', 'parse_request'):
    method = getattr(BaseHTTPRequestHandler, name)
    print(f'--- {name} ---')
    print(inspect.getsource(method))
PY

Repository: aiidateam/aiida-core

Length of output: 47434


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo '=== join-code transport and serialization ==='
sed -n '90,160p' src/aiida/tools/collab/protocol.py
sed -n '100,130p' src/aiida/tools/collab/client.py
sed -n '520,565p' src/aiida/cmdline/commands/cmd_collab.py

echo '=== endpoint report construction and logging ==='
sed -n '220,345p' src/aiida/tools/collab/endpoint.py

echo '=== logger argument data-flow candidates ==='
rg -n -P -C8 'report\(|reports\.append|LOGGER\.(report|debug|info|warning|error|exception)|JoinCode|join_code\(' src/aiida/tools/collab src/aiida/cmdline/commands/cmd_collab.py --type=py

Repository: aiidateam/aiida-core

Length of output: 50376


Stop logging the full request line. BaseHTTPRequestHandler.send_response() logs self.requestline, which includes query strings even for unauthenticated requests. A token or join code in the request URL is therefore copied to the collab log and bug-report archive. Log only the method and path, without the query string.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/aiida/cmdline/commands/cmd_bug_report.py` at line 229, Update the collab
request logging configuration near the filepaths entry so BaseHTTPRequestHandler
no longer records the full request line containing query parameters. Override or
adjust the relevant request-handler logging behavior to emit only the HTTP
method and URL path, preserving the existing collab log and bug-report archive
destinations.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The server's log_message writes at DEBUG only, and no request in this protocol ever carries a token or join code in a URL — the token travels exclusively as a Bearer header, the join code never travels at all. The scenario needs debug logging enabled and a client that does not exist, so we are leaving it as is.

}

for log_type, log_filepath in log_filepaths.items():
Expand Down
Loading
Loading