You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Removes the ipfs-publisher service and the IPFS/CID indirection between batcher and blockchain-anchor. batcher now XZ-compresses each detached batch, recursively splits it to fit ANCHOR_MAX_PAYLOAD_BYTES (shared default 8192, defined in @scp/core), and emits chain-ready payloads directly on telemetry.batched.v1 (batch_id, payload, uncompressed_size, compressed_size, payload_hash). Oversized single events are routed to the DLQ instead of blocking the rest of the batch.
blockchain-anchor now consumes telemetry.batched.v1 directly and submits the compressed payload bytes on-chain via cps.setPayload. Idempotency is based on a byte-for-byte comparison of the incoming payload against the current on-chain payload (api.query.cps.payload), replacing the previous CID-string comparison. Oversized payloads are defensively rejected as a permanent error.
Reworked TelemetryBatchedPayload proto, removed TelemetryIpfsPublishedPayload and the ipfs.published.v1 topic
Deleted services/ipfs-publisher entirely
Updated docker-compose.yml, kafka-init-topics.sh, integration-test.yml, and .env.example to drop IPFS publisher wiring (the ipfs container is kept, since it is still used as a libp2p GossipSub reserved peer for pubsub-broadcaster integration tests)
Updated READMEs, architecture docs, tracing.md, and WP-04/WP-05 work package docs to reflect the new pipeline
…compressed batches directly (#34)
Removes the ipfs-publisher service and the IPFS/CID indirection between
batcher and blockchain-anchor. batcher now XZ-compresses each detached
batch, recursively splits it to fit ANCHOR_MAX_PAYLOAD_BYTES (shared
default 8192, defined in @scp/core), and emits chain-ready payloads
directly on telemetry.batched.v1 (batch_id, payload, uncompressed_size,
compressed_size, payload_hash). Oversized single events are routed to
the DLQ instead of blocking the rest of the batch.
blockchain-anchor now consumes telemetry.batched.v1 directly and
submits the compressed payload bytes on-chain via cps.setPayload.
Idempotency is based on a byte-for-byte comparison of the incoming
payload against the current on-chain payload (api.query.cps.payload),
replacing the previous CID-string comparison. Oversized payloads are
defensively rejected as a permanent error.
- Reworked TelemetryBatchedPayload proto, removed
TelemetryIpfsPublishedPayload and the ipfs.published.v1 topic
- Deleted services/ipfs-publisher entirely
- Updated docker-compose.yml, kafka-init-topics.sh,
integration-test.yml, and .env.example to drop IPFS
publisher wiring (the ipfs container is kept, since it is still used
as a libp2p GossipSub reserved peer for pubsub-broadcaster
integration tests)
- Updated READMEs, architecture docs, tracing.md, and WP-04/WP-05 work
package docs to reflect the new pipeline
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Start batcher and verify the end-to-end anchoring flow
.github/workflows/integration-test.yml:218
The workflow still starts blockchain-anchor but never starts @scp/batcher, so telemetry cannot reach telemetry.batched.v1 and the new direct anchoring path is not exercised. Issue #34 explicitly requires an end-to-end byte-identity check from batcher output to set_payload; add the batcher startup/health checks and verify that anchoring flow.
Add or provision CPS metadata for encoded compressed payloads
proto/connectivity/v1/payload.proto:31
The contract says CPS metadata describes the schema, encoding, and compression, but this PR contains no metadata update/provisioning path. Without that metadata, on-chain readers cannot determine that these raw bytes are protobuf SignedEnvelopeBatch compressed with XZ, which leaves issue #34's payload-interpretation acceptance criterion unmet. Add the CPS metadata update mechanism or document and provision the required out-of-band metadata.
Count all terminal outcomes after recursive batch splitting
services/batcher/src/index.ts:243
This misses batches that were split into one fitted sub-batch plus one or more oversized events: fitted.length is then 1 even though recursive splitting occurred. Count all terminal outcomes when deciding whether the original batch split.
Publish blockchain result before committing the offset
services/blockchain-anchor/src/index.ts:515
The successful path proceeds directly to committing the consumed offset without publishing telemetry.blockchain.result.v1. Issue #34 requires a CID-free blockchain result and commit only after that result is emitted, so this currently leaves downstream consumers with no anchoring outcome. Publish the batch-based result before this commit.
Correct setup steps for the retained IPFS container
.github/copilot-instructions.md:152
docker compose up -d still starts the retained IPFS container for the pubsub reserved peer, so this setup step is inaccurate even though IPFS is no longer part of anchoring.
Document the retained IPFS container in the setup
README.md:26
make all still runs docker compose up, and the compose file intentionally retains the IPFS container as the pubsub reserved peer. Omitting IPFS here makes the setup description inaccurate.
Include trace IDs and batch IDs in produced-batch logs
docs/tracing.md:126
This lookup cannot return a batch_id for the trace: the batcher's fitting batch log includes trace_ids but no batch ID, while its batch produced log includes the batch ID but omits trace IDs. Add per-sub-batch trace IDs to the produced log (or revise the documented correlation procedure) so this command is actionable.
This only checks a standalone array length and never exercises fitBatch with a compressed result exactly equal to the limit, so the named boundary behavior and issue #34 acceptance criterion remain untested. Use a deterministic fixture or injectable compressor that makes fitBatch return an exact-limit payload.
- blockchain-anchor: BoundedVec<u8>.toU8a() includes the SCALE
compact-length prefix by default; pass toU8a(true) so the on-chain
payload bytes compared for idempotency match the bare bytes we submit.
- blockchain-anchor: compare batches by their content-addressed
payload_hash (blake2b-256, already computed by the batcher) against a
hash of the currently anchored on-chain payload, instead of comparing
raw bytes. The Kafka offset for a batch is only committed once this
check (and, if needed, the extrinsic) completes, so an uncommitted
offset already durably marks a batch as unconfirmed — no additional
persisted queue is required.
- batcher: update the produceBatch comment to reflect that re-publishing
a sub-batch is harmless now that anchoring dedupes by payload_hash,
regardless of whether the whole split lands atomically.
- tests: idempotency.test.ts now computes payload_hash the same way the
real batcher does (blake2b-256 of the payload) instead of a fixed
placeholder, so the hash-based dedup check is exercised correctly.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
…compressed batches directly (#34)
Removes the ipfs-publisher service and the IPFS/CID indirection between batcher and blockchain-anchor. batcher now XZ-compresses each detached batch, recursively splits it to fit ANCHOR_MAX_PAYLOAD_BYTES (shared default 8192, defined in @scp/core), and emits chain-ready payloads directly on telemetry.batched.v1 (batch_id, payload, uncompressed_size, compressed_size, payload_hash). Oversized single events are routed to the DLQ instead of blocking the rest of the batch.
blockchain-anchor now consumes telemetry.batched.v1 directly and submits the compressed payload bytes on-chain via cps.setPayload. Idempotency is based on a byte-for-byte comparison of the incoming payload against the current on-chain payload (api.query.cps.payload), replacing the previous CID-string comparison. Oversized payloads are defensively rejected as a permanent error.