Krawlify

Two pieces that ship together: the Krawlify extension, which runs in your Chrome, and the Krawlify relay (server/), a small Node service that both your scraper and the extension can reach.

Share a Chrome DevTools Protocol connection from your own, already-logged-in Chrome profile with a scraping app — no --remote-debugging-port, no throwaway profile, no separate browser.

A Chrome extension opens tabs on your scraper's behalf, attaches chrome.debugger to them, and pipes CDP over a WebSocket to a small local relay. Your scraper connects to that relay and speaks ordinary CDP, so puppeteer.connect() and chromium.connectOverCDP() work unchanged. It sees only the tabs it opened — never the rest of your browser.

   scraping app                    relay (node, :9333)                 your Chrome
┌────────────────┐            ┌───────────────────────┐          ┌────────────────────┐
│ puppeteer      │            │  /json/version        │          │ Krawlify extension │
│ playwright     │──── ws ───▶│  /json/list           │◀── ws ───│  (service worker)  │
│ raw CDP client │            │  /devtools/browser/…  │          │         │          │
└────────────────┘            │  /devtools/page/…     │          │  chrome.debugger   │
                              │  /control             │          │         ▼          │
                              └───────────────────────┘          │   the tabs the     │
                                                                 │   crawler opened   │
                                                                 └────────────────────┘

Why a relay? An MV3 extension cannot listen on a port, so it can only dial out. The relay is the one place both the extension and your scraper can reach. It holds no browser state of its own — it is a switchboard.

Why an extension? chrome.debugger is the only way to speak CDP to a normal Chrome that was not started with --remote-debugging-port. That is the whole point: you get the real profile, with its cookies, logins, extensions and fingerprint.

Your tabs stay yours

A connected client can address exactly one thing: tabs it opened itself. Your own tabs are not filtered out of a list it could ask for differently — they are not in the protocol at all. Ownership lives in the extension, Target.createTarget is the only thing that adds to it, and closing a tab removes it. Crawler tabs are grouped and labelled so you can see them at a glance, and they close automatically when the client disconnects.

Things that were harder than they look

Written down because they are the interesting part, and because each one cost real time:

• Target ids cannot be synthesized. Chrome guarantees a page target's id is its main frame's id, and Playwright silently depends on it — with a made-up id it decides the page is a blank new tab and waits forever for a first navigation. No error, just a hang.
• Event ordering is part of the contract. Target.setDiscoverTargets must emit its targetCreated events before its own reply, because clients treat the response as "the list is complete now".
• Tenants are isolated by construction, not by checks. Each token gets a Tenant object holding its own link, tabs and sweep timer; sessions are handed that Tenant as their world, so a session physically cannot reach another user's state.
• An unknown token and an offline browser deliberately return the same 503. Telling them apart would turn the endpoint into an oracle for guessing valid tokens.
• MV3 service workers die mid-crawl. Reconnection is driven by a chrome.alarms backstop rather than a setTimeout, because the timer does not survive termination.

docs/DECISIONS.md records the evidence for these and a dozen more, mostly gathered by diffing protocol traces against Chrome's own debugging port (npm run trace:diff).

Where things are documented

File	For
this file	using it — setup, connecting a scraper, endpoints, security, limits
CLAUDE.md	picking the project up: commands, layout, invariants, gotchas
ARCHITECTURE.md	changing how it works: components, state ownership, flows
docs/PROTOCOL.md	the extension⇄relay envelope and every client endpoint
docs/DECISIONS.md	why it is built this way, with the evidence
docs/TROUBLESHOOTING.md	symptom → cause → fix, and how to use tools/

If something misbehaves, docs/TROUBLESHOOTING.md is the fastest route. If you are about to change the CDP emulation, read docs/DECISIONS.md first — several choices there look arbitrary and are not.

Setup

Run the relay from Docker Hub:

docker run -d --name krawlify-relay -p 127.0.0.1:9333:9333 \
  -e KRAWLIFY_ALLOW_HOSTS=relay.example.com \
  kamenarov/krawlify-relay:latest

Or from source:

npm install
npm start          # prints its endpoints; there is no server token to copy

KRAWLIFY_ALLOW_HOSTS is every hostname the relay will answer to. Any other Host gets a 421 — that allowlist is the DNS-rebinding guard, so it is required for anything but loopback. See Serving outside localhost before exposing it.

Then load the extension:

1. Open chrome://extensions, turn on Developer mode, click Load unpacked, and select this repo's extension/ directory.
2. Open Krawlify's Details → Extension options, set the relay URL, and press Save & connect. The extension generates its own token there — copy it; that token is how a scraper reaches this browser. Renew Token rotates it and instantly cuts off anything still using the old one.
3. That is all. The crawler opens its own tabs (grouped as Krawlify); your own tabs are never exposed to it.

Chrome shows a "Krawlify started debugging this browser" banner while a tab is attached. That is Chrome telling the truth; it cannot be suppressed.

To verify end to end:

npm run test:tab -- --token <the token from the options page>

Connecting a scraper

The relay is multi-tenant: several browsers connect to one relay, each under the token its extension generated, and a client picks which browser to drive by presenting that token as Authorization: Bearer <token>. The relay itself has no password — the token is both the identity and the credential.

Bearer is the only accepted form. A token in a query string ends up in proxy logs and error messages, and this one is a standing credential for a logged-in browser. The single exception is the extension's own /extension socket, which must use ?token= because a service-worker WebSocket cannot set headers at all.

puppeteer — use the WebSocket endpoint, since puppeteer drops the query string when given a browserURL:

import puppeteer from 'puppeteer-core';

const browser = await puppeteer.connect({
  browserWSEndpoint: 'ws://127.0.0.1:9333/devtools/browser/<guid>',
  headers: { authorization: 'Bearer <token>' },
});
const page = await browser.newPage();       // the crawler opens its own tab
console.log(await page.title());

Playwright — connectOverCDP, not connect, and pass the token as a header:

import { chromium } from 'playwright-core';

const browser = await chromium.connectOverCDP('http://127.0.0.1:9333', {
  headers: { authorization: 'Bearer <token>' },
});
const page = browser.contexts()[0].pages()[0];

npm start prints both snippets with the live guid and token filled in.

Endpoints

Endpoint	Purpose
GET /json/version	Browser version plus webSocketDebuggerUrl; the discovery entry point
GET /json/list	One entry per tab the crawler opened; your own tabs never appear
GET /json/new?url=	Open a tab (this is what makes a tab visible to the client)
GET /status	Extension state, crawler tabs, connected clients, endpoints
GET /healthz	Unauthenticated liveness probe; leaks nothing
ws /devtools/browser/<guid>	Full browser-level CDP — what puppeteer/Playwright want
ws /devtools/page/<targetId>	One tab, raw passthrough; simplest for a hand-rolled client
ws /control	Non-CDP extras: list/open tabs, read cookies

Control channel

For things CDP cannot do because they need extension APIs rather than a debugger. JSON in, JSON out: {id, method, params} → {id, result}.

ws.send(JSON.stringify({ id: 1, method: 'tabs.list' }));

status, tabs.list, tabs.create, tabs.close, tabs.activate, tabs.navigate, cookies.get (needs --allow-cookies). The share.* methods were removed — open and close tabs with Target.createTarget / Target.closeTarget instead.

Server options

--port <n>              default 9333
--host <addr>           default 127.0.0.1
--log <level>           silent | error | warn | info | debug
--allow-multi-client    let several clients drive one tab (they will fight over CDP state)
--allow-cookies         let clients read profile cookies over /control
--sweep-delay <s>       seconds before the crawler's tabs are closed after the last
                        client disconnects (default 5, 0 disables)
--allow-host <name>     accept this Host as well as loopback (repeatable); required to
                        serve anything other than localhost
--tls-cert <file>       PEM certificate; serve https/wss directly
--tls-key <file>        PEM private key for --tls-cert
--trust-proxy           believe X-Forwarded-Proto from a TLS proxy in front of the relay
--allow-origin <origin> accept browser requests from this Origin (repeatable)
--extension-id <id>     only accept the control channel from this extension id

Serving outside localhost

The relay is loopback-only until you say otherwise, and three things have to line up:

1. Bind wider — --host 0.0.0.0 (the Docker image already does; set KRAWLIFY_BIND=0.0.0.0 to publish it off the host).
2. Allow the Host — --allow-host relay.example.com. Any name not listed is refused with 421; that allowlist is what keeps the DNS-rebinding protection meaningful.
3. Terminate TLS — either --tls-cert/--tls-key, or a proxy in front plus --trust-proxy. Not optional: the token authenticates and identifies a browser, it is sent on every request, and over plain ws:// anyone on the path can lift one and drive that user's logged-in Chrome.

--trust-proxy matters for more than warnings. Behind an https proxy the relay would otherwise advertise ws:// in webSocketDebuggerUrl, and the client would follow that URL to a port with nothing on it. With the flag it reads X-Forwarded-Proto and hands out wss://. Only enable it where a proxy you control sets that header — a direct client can forge it.

In the extension, set the same address as the Server URL (https://relay.example.com); it is rewritten to wss:// for you, and the options page shows the resolved endpoint.

Security

This endpoint can drive a browser that is logged into everything you are. Treat the token like a password.

• Loopback only by default. The relay rejects any request whose Host is not loopback (421), blocking DNS-rebinding attacks where a public hostname resolves to 127.0.0.1. --allow-host opts specific names in; everything else stays refused.
• A token per browser, not per server. Each extension generates 256 bits; the relay keeps no list to check against. A request with no token at all is 401; a token with no live browser behind it is 503 — deliberately the same answer as an unknown token, so the endpoint cannot be used to discover which tokens exist.
• Tenants cannot reach each other. Per-token state lives in a Tenant, and a client session is only ever given its own; addressing another tenant's browser guid is a 404.
• Origin rejection. A real scraper sends no Origin; a web page always does. Any unexpected Origin is rejected (403), so a random page you visit cannot drive the relay. The one exception is the extension's own chrome-extension:// origin on /extension, which you can pin with --extension-id.
• The client only sees tabs it opened. The extension tells the relay about a tab only once the client created it through Target.createTarget, and drops it when it closes. Your own tabs are invisible — not listed, not addressable — and closing a crawler tab reports it to clients as a destroyed target immediately.
• Crawler tabs are cleaned up. Five seconds after the last client disconnects the extension closes them, so a crashed scraper leaves nothing behind. --sweep-delay 0 turns it off.
• One client per tab by default, so two scrapers cannot silently corrupt each other's CDP domain state. Opt out with --allow-multi-client.
• Cookie access is off unless you pass --allow-cookies, and it additionally needs a permission you grant on the options page.
• Browser.close does not close your browser. It just disconnects that client.

Known limits

• DevTools conflicts. Chrome allows one debugger per tab. If you have DevTools open on a tab, attaching fails with a message telling you to close it. The reverse also holds: opening DevTools on an attached tab detaches Krawlify.
• Target.createBrowserContext is unsupported. A live profile has exactly one browser context, so Playwright's browser.newContext() and puppeteer's incognito contexts fail by design. Use newPage() / Target.createTarget, which open real tabs.
• Browser.setDownloadBehavior is accepted and ignored. Playwright sends it while connecting, so failing it would break connectOverCDP; silently repointing your real browser's download directory seemed worse than ignoring it. Playwright's download API will not work.
• Chrome pages cannot be shared — chrome://, other extensions, and the Web Store are all off-limits to chrome.debugger, so they never appear as targets.
• MV3 worker lifetime. The extension's service worker is kept alive by the relay's 15-second ping plus a 30-second alarm as a backstop, and reconnects with backoff. A disconnect detaches every tab rather than leaving orphaned debugger sessions.
• No toolbar icon is bundled; Chrome shows its default placeholder.

Integrating with crawer_app

crawer_app currently connects with chromium.connect(PLAYWRIGHT_WS_ENDPOINT), which speaks the Playwright server protocol — a different thing from CDP, so it cannot talk to this relay as-is. src/crawler/browser.ts needs connectOverCDP instead:

-browser = await chromium.connect(endpoint, { timeout: 30_000 });
+browser = await chromium.connectOverCDP(endpoint, {
+  timeout: 30_000,
+  headers: { authorization: `Bearer ${process.env.KRAWLIFY_TOKEN}` },
+});

Two consequences worth knowing before you switch:

• connectOverCDP gives you the existing default context, so the per-crawl newContext() isolation goes away — you are driving the real profile, which is the point when you want its cf_clearance cookie, but it means crawls share state.
• The relay must be reachable from wherever the worker runs. It binds loopback, so a worker in Docker needs a tunnel or an SSH forward rather than --host 0.0.0.0.

Tests

npm run test:e2e            # spawns Chrome, drives it through the relay, 33 assertions
npm run test:e2e -- --head   # same, with a visible window
npm run test:smoke -- --token <token>   # against a relay/extension you already run

test/e2e.mjs starts Chrome without --remote-debugging-port, so the extension is the only debugger in play — the same situation as real use. It covers the token/Host/Origin guards, puppeteer and Playwright end to end, the raw per-tab endpoint, per-tab exclusivity, tab creation, and that revoking consent immediately hides tabs.

It needs a Chrome for Testing or Chromium build, because branded Google Chrome ignores --load-extension ("--disable-extensions-except is not allowed in Google Chrome"). This only affects automation; loading the extension by hand in your normal Chrome is fine. The test finds a browser in puppeteer's or Playwright's cache automatically, or you can point it at one:

npx @puppeteer/browsers install chrome@stable
KRAWLIFY_TEST_CHROME=/path/to/chrome npm run test:e2e

extension/autoconfig.json

If extension/autoconfig.json exists and nothing has been configured yet, the extension adopts it on startup — handy for automated setups, and how the e2e test configures itself. It is only honoured for loopback server URLs, and it is gitignored because it holds a token.

{ "serverUrl": "http://127.0.0.1:9333", "token": "…" }

How the emulation works

chrome.debugger exposes only page-level CDP: no Browser domain, no Target domain, and no way to hand out sessions. Clients always open a browser endpoint first and discover pages through Target.*, so server/browser-session.js synthesizes that layer — shared tabs become type: "page" targets, attaching mints a flat sessionId bound to a tab, and Chrome's own nested session ids (OOPIFs, workers) pass through untouched, which chrome.debugger accepts directly as of Chrome 125.

Two details are load-bearing and were found the hard way:

• Target ids come from Chrome, via chrome.debugger.getTargets() — never synthesized. Chrome guarantees a page target's id is its main frame's id, and Playwright relies on that invariant to recognise the page. With an id derived from the tab id, Playwright decides the page is a blank new tab and waits forever for a first navigation.
• Every page target reports a browserContextId. Playwright asserts the field is non-empty before adopting a page; because the id is not one it created, it falls back to its default context, which is exactly right for a single-profile browser.

Both were found by diffing protocol traces against Chrome's own debugging port, which npm run trace:diff now does for you. docs/DECISIONS.md records the evidence for these and a dozen other choices; ARCHITECTURE.md explains the session model in full.

Diagnostics

npm run inspect      # stream the extension service worker's console and state
npm run trace:diff   # run a client against Krawlify and real Chrome, then compare

Reach for trace:diff whenever a client connects but then hangs — that failure mode produces no error anywhere, and the diff points straight at the divergence. See docs/TROUBLESHOOTING.md.

Releasing

CI runs the e2e gate on every push and pull request. Pushes to main publish kamenarov/krawlify-relay:edge; a v* tag publishes the semver tags and :latest. The image is built for linux/amd64 and linux/arm64, and nothing publishes unless the gate passes.

npm version patch          # bumps package.json and tags
git push --follow-tags

The extension is versioned separately in extension/manifest.json and distributed as a signed CRX — see docs/DISTRIBUTION.md, which also explains why it is not on the Chrome Web Store.

Author

Yordan Kamenarov — kamenarov.dev

Licence

MIT — see LICENSE. Both the relay and the extension are covered.