Skip to content

About

Collaboration Tunnel Protocol - IETF-style specification for efficient AI crawler content delivery

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

Β 

History

36 Commits

Folders and files

Repository files navigation

Collaboration Tunnel Protocol Specification

Status: Individual Draft (Submitted to IETF) Version: draft-jurkovikj-collab-tunnel-01 Date: November 14, 2025 Author: Antun Jurkovikj IETF Datatracker: https://datatracker.ietf.org/doc/draft-jurkovikj-collab-tunnel/

Abstract

This document specifies the Collaboration Tunnel Protocol, an HTTP-based method for efficient, verifiable delivery of web content to automated agents. The protocol uses bidirectional URL discovery, template-invariant content fingerprinting with strong ETags, sitemap-first verification, and strict conditional request discipline to reduce bandwidth while preserving canonical URLs.

Specification

Current Version (01):

Previous Versions:

Implementation Status

Publisher Implementations

Consumer Implementations

Live Demonstrations

Test the protocol on these production sites:

Validator Tool

Measured Results

Based on 491 URLs across 4 production sites:

  • Bandwidth savings: 83% vs HTML-only crawling (103 KB average HTML β†’ 17.7 KB average JSON)
  • Token savings: 86% for AI processing (13,900 tokens β†’ 1,960 tokens)
  • Skip rate: 90%+ for unchanged content using sitemap-first verification
  • 304 response rate: 100% after initial fetch (ETag validation working)
  • RFC compliance: 100% of endpoints (RFC 9110, 9111, 8288)

Protocol Overview

The Collaboration Tunnel Protocol consists of four coordinated mechanisms:

1. Bidirectional Discovery

  • C-URL (HTML page) β†’ M-URL (JSON endpoint) via <link rel="alternate">
  • M-URL β†’ C-URL via Link: <C-URL>; rel="canonical" header
  • Prevents SEO conflicts

2. Template-Invariant Fingerprinting

  • Content normalized through 6-step pipeline (operates on JSON content field, not HTML):
    1. Decode HTML entities (& β†’ &, β€” β†’ β€”)
    2. Apply Unicode NFKC normalization
    3. Apply Unicode case folding (locale-independent lowercase)
    4. Remove control characters (Unicode category Cc), except TAB, LF, CR
    5. Collapse ASCII whitespace (SPACE, TAB, LF, CR) to single space
    6. Trim leading/trailing whitespace
  • Then compute SHA-256 hash over normalized UTF-8 bytes
  • Strong ETag format: "sha256-..." (computed over canonical JSON bytes)
  • Method A (recommended): Hash canonical JSON excluding hash field
  • Method B (allowed): Hash normalized content text (content-locked)
  • Stable across theme/template changes
  • Parity verification: Clients MUST verify contentHash == clean(ETag) == payload.hash via string equality
  • Deterministic JSON: Implementations MUST use stable key order, UTF-8, compact serialization per RFC 8785 or equivalent

3. Conditional Request Discipline

  • If-None-Match takes precedence
  • 304 Not Modified for unchanged content
  • Strict caching requirements

4. Sitemap-First Verification

  • JSON sitemap lists (cUrl, mUrl, contentHash)
  • Skip fetch if hash unchanged (zero-fetch optimization)
  • 90%+ skip rate in production

Response Format

M-URL JSON Payload

{
  "profile": "tct-1",
  "canonical_url": "https://example.com/post/",
  "title": "Article Title",
  "content": "Core article content...",
  "hash": "sha256-2c26b46b68ffc68ff99b453c1d30413413422d706483bfa0f98a5e886266e7ae",
  "modified": "2025-10-15T14:30:00Z"
}

M-URL HTTP Headers

HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8
Link: <https://example.com/post/>; rel="canonical"
ETag: "sha256-e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
Cache-Control: max-age=0, must-revalidate, stale-while-revalidate=60, stale-if-error=86400
Vary: Accept-Encoding

Sitemap JSON Format

{
  "version": 1,
  "profile": "tct-1",
  "items": [
    {
      "cUrl": "https://example.com/post/",
      "mUrl": "https://example.com/post/llm/",
      "modified": "2025-10-23T18:00:00Z",
      "contentHash": "sha256-e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
    }
  ]
}

Profile Field: The "profile": "tct-1" field enables protocol versioning. Future versions (e.g., tct-2) can introduce new fields while maintaining backward compatibility.

Quick Start

For Publishers (Website Owners)

WordPress:

# Install plugin
wp plugin install trusted-collab-tunnel --activate

# Or download from: https://wordpress.org/plugins/trusted-collab-tunnel/

Cloudflare Worker:

// Deploy worker from: https://github.com/antunjurkovic-collab/trusted-collab-worker

For Consumers (AI Crawlers)

Python:

pip install collab-tunnel
from collab_tunnel import CollabTunnelCrawler

crawler = CollabTunnelCrawler(user_agent="MyBot/1.0")
sitemap = crawler.fetch_sitemap("https://example.com/llm-sitemap.json")

for item in sitemap.items:
    if crawler.should_fetch(item):  # Zero-fetch optimization
        content = crawler.fetch_content(item['mUrl'])
        print(content['title'])

stats = crawler.get_stats()
print(f"Bandwidth saved: {stats['savings_percentage']}%")

Feedback

This is an individual draft submitted to the IETF and seeking community feedback.

Version History

  • Version 01 (November 14, 2025): Incorporated expert feedback, added Unicode normalization test vectors, enhanced security considerations
  • Version 00 (October 2025): Initial submission

Intellectual Property Rights (IPR)

This specification may be covered by US Patent Application 63/895,763 (filed October 8, 2025, status: Patent Pending).

IPR Disclosure

An IPR disclosure has been submitted to the IETF declaring that licenses under any patent claims held by the author will be made available on a Royalty-Free basis under reasonable terms and conditions for all implementations of this specification.

For All Implementers:

  • βœ… Royalty-Free licensing for all implementations
  • βœ… No restrictions on commercial or non-commercial use
  • βœ… No restrictions on scale or deployment size
  • βœ… Open and interoperable implementations encouraged

For questions about IPR: antunjurkovic@gmail.com

License

This specification document is provided under the IETF Trust Legal Provisions for IETF Documents (BCP 78, BCP 79).

Implementation License

Implementations of this protocol are licensed on a Royalty-Free basis as described in the IPR disclosure above. The specification is published to encourage widespread adoption and ensure interoperability.

Links

Contributing

Contributions, suggestions, and feedback are welcome!

Please open an issue or submit a pull request for:

  • Specification improvements
  • Clarifications
  • Additional examples
  • Error corrections

Acknowledgments

Thanks to:

  • WordPress and Cloudflare communities for implementation platforms
  • IETF HTTP Working Group for foundational HTTP standards (RFC 9110, 9111, 8288, 6596)
  • Open Archives Initiative for ResourceSync inspiration
  • Early adopters testing the protocol on production sites

About

Collaboration Tunnel Protocol - IETF-style specification for efficient AI crawler content delivery

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages