Add RocksDB managed backup and restore operations

DESIGN.md

@@ -305,3 +305,60 @@ guarded against (in the scan or in tar-fs's own pack walk); that's a pre-existin
 this fix doesn't attempt to solve. `deploy_component`/`package_component` still never validate that
 declared entry points (`jsResource`/`graphqlSchema`) survived extraction — a truncation from some
 other future cause would still report success silently; that's a deferred, separate fix.
+
+## RocksDB backup/restore: the restore lock + marker protocol (`dataLayer/restoreMarker.ts`, `dataLayer/rocksdbBackup.ts`)
+
+The `restore_backup` operation restores a user database on a live server by closing it across all
+worker threads, purging its directory (`backups.restore` with `purgeAllFiles`), and reloading it.
+Three non-obvious mechanics keep that safe:
+
+- **Two files next to (never inside) the database directory**: `<db>.restore.lock`, an OS-level
+  exclusive flock (rocksdb-js `tryFileLock`, auto-released on process death), serializes restores;
+  `<db>.restoring`, a marker written+fsynced (file _and_ parent directory) after the lock and
+  before any destructive step, means "a restore started and has not finished". Startup/rescan
+  detection (`databasesBlockedByRestore` in `resources/databases.ts`) checks the **marker first**
+  and only probes the lock when the marker exists — probes take the flock and are mutually
+  exclusive across threads, so probing the (persistent) lock file of every long-ago-restored
+  database on every rescan would make concurrent rescans misclassify healthy databases as
+  in-progress. Marker-present + lock-held = restore in progress (don't load); marker-present +
+  lock-free = crashed mid-restore (don't load; rerun the restore to recover).
+- **The ITC close broadcast is best-effort, so closure is verified before the purge.** The SCHEMA
+  broadcast (`signalSchemaChange`) resolves after remote handlers complete but times out at 30s
+  "best-effort", swallows errors, and never reaches job-worker threads at all (their ports are
+  excluded from broadcasts to avoid re-entrant deadlocks). A destructive purge cannot trust it:
+  `restoreBackup` polls rocksdb-js `registryStatus()` (process-global across worker threads) until
+  the database path has no open instance, and aborts with a 409 — _cleaning up the marker, since
+  nothing was destroyed_ — if handles remain.
+- **Online restore is impossible for a database a component holds open — and that failure is
+  correct.** rocksdb-js's registry is process-global but records only a per-path refCount, with no
+  attribution to a thread or component; Harper keeps no component→database ownership map. So when a
+  loaded component (or the `system` database, which Harper itself never stops while running) holds
+  its own handle on the target database, `registryStatus()` stays non-zero, Harper can neither
+  identify nor force-close that handle, and an in-place purge would corrupt a live instance.
+  `verifyDatabaseClosed` therefore waits only a short grace period (`DATABASE_CLOSE_WAIT_MS`, for a
+  just-finished job worker's own close to drain) and then fails fast with a 409 that points at
+  running the operation offline (`harper restore_backup` with the server stopped, where no
+  components are loaded and nothing holds the database open). Offline restore is the supported path
+  for component-held and `system` databases; online restore serves databases not actively held by a
+  component. The CLI exposes each backup operation under its operation name only (`create_backup`,
+  `restore_backup`, …) — no hyphenated alias — and `bin/backup.ts` routes it to a reachable server
+  or, when the local server is stopped, to the equivalent offline function.
+- **Job workers must release their RocksDB handles on exit, or the closure check can never pass.**
+  rocksdb-js's registry is process-global across worker threads, and a thread that exits WITHOUT
+  closing leaks its handles (the refCount never drops); the only alternative, `shutdown()`, tears
+  down rocksdb for the _entire_ process. A job worker (`server/jobs/jobProcess.ts`) opens the whole
+  database graph via `getDatabases()` and exits when the job finishes — and `create_backup` is
+  itself a job, so before any `restore_backup` there is always at least one exited job worker that
+  touched the database. Without cleanup those leaked handles keep `registryStatus()` non-zero and
+  would fail the closure check even when no component holds the database. `jobProcess` therefore
+  calls `closeLoadedDatabases()` (`resources/databases.ts`) in its `finally`, closing every loaded
+  user database on that thread (the non-enumerable `system` DB is intentionally skipped), so an
+  exited job worker leaves no residual handle to be mistaken for a live holder.
+- **Every path that can (re)open or delete a RocksDB database checks the restore state**, not just
+  the `getDatabases()` scan: `database()`'s on-demand open (`create_table`/`create_schema` would
+  otherwise resurrect a half-purged directory as a fresh empty DB, defeating the recovery
+  protocol) and `dropDatabase` (whose `destroy()` racing a restore's copy would gut a completed
+  restore with the marker already gone) both throw 409 via `throwIfBlockedByRestore`.
+
+Known limitation: the flock is process-owned; if the restore job's worker _thread_ dies without
+the process exiting, the lock stays held (restores 409) until Harper restarts.

bin/backup.ts

@@ -0,0 +1,128 @@
+'use strict';
+
+import { createWriteStream } from 'node:fs';
+import { basename } from 'node:path';
+import { pipeline } from 'node:stream/promises';
+import * as YAML from 'yaml';
+import * as envMgr from '../utility/environment/environmentManager.ts';
+envMgr.initSync();
+import * as terms from '../utility/hdbTerms.ts';
+import { buildRequest, cliOperations, resolveRequestOptions } from './cliOperations.ts';
+import { loadCredentials } from './cliCredentials.ts';
+import { httpRequest } from '../utility/common_utils.ts';
+import { initConfig } from '../config/configUtils.ts';
+import { getHdbPid } from '../utility/processManagement/processManagement.js';
+import {
+	backupDirForDatabase,
+	createBackupOffline,
+	deleteBackupOffline,
+	listBackupsInDir,
+	purgeBackupsOffline,
+	restoreBackupOffline,
+	verifyBackupOffline,
+} from '../dataLayer/rocksdbBackup.ts';
+
+const { OPERATIONS_ENUM } = terms;
+
+/**
+ * Runs a backup operation from the CLI. `command` is the operation name (e.g. `create_backup`);
+ * there is no hyphenated alias. Each works whether or not Harper is running: RocksDB is
+ * single-writer, so when a server is reachable the running process must own the database handle
+ * and the operation is forwarded to it (also handling remote `target=` and auth); only when the
+ * *local* server is stopped do we open the database/backup files directly. `get_backup` is the
+ * exception — it streams a live snapshot from a running server and has no offline form.
+ */
+export async function runBackupCommand(command: string): Promise<void> {
+	const request = buildRequest();
+	const databaseName = request.database || 'data';
+
+	// We print results here (not by returning a string) so the CLI's top-level handler doesn't
+	// re-emit them through logger.notify — this keeps output identical to forwarding the operation
+	// to a running server, which the online branch below simply delegates to.
+	if (command === OPERATIONS_ENUM.GET_BACKUP) {
+		console.log(await downloadBackup(request, databaseName));
+		return;
+	}
+
+	// Fall back to direct file access only when there is no server to talk to (no target and the
+	// local instance is stopped). We never fall back on a *reachable* server's error response —
+	// e.g. restore_backup rejecting the system/component-held database (whose offline path is to
+	// stop the server and rerun this same command, which then takes the branch below).
+	if (useOperationApi(request)) {
+		request.operation = command;
+		await cliOperations(request); // prints its own result, exactly like any forwarded operation
+		return;
+	}
+
+	let result: any;
+	switch (command) {
+		case OPERATIONS_ENUM.CREATE_BACKUP:
+			result = await createBackupOffline(databaseName);
+			break;
+		case OPERATIONS_ENUM.LIST_BACKUPS:
+			result = await listBackupsInDir(backupDirForDatabase(databaseName));
+			break;
+		case OPERATIONS_ENUM.VERIFY_BACKUP:
+			result = await verifyBackupOffline(databaseName, request.backup_id, request.verify_checksum);
+			break;
+		case OPERATIONS_ENUM.DELETE_BACKUP:
+			result = await deleteBackupOffline(databaseName, request.backup_id);
+			break;
+		case OPERATIONS_ENUM.PURGE_BACKUPS:
+			result = await purgeBackupsOffline(databaseName, request.keep_count);
+			break;
+		case OPERATIONS_ENUM.RESTORE_BACKUP:
+			result = await restoreBackupOffline(databaseName, request.backup_id, request.target_database);
+			break;
+		default:
+			throw new Error(`Unknown backup command '${command}'`);
+	}
+	console.log(YAML.stringify(result).trim());
+}
+
+/**
+ * Whether to route a backup command through the operation API rather than direct file access.
+ * A configured target (explicit `target=`, env, or saved `last_target`) always implies a remote
+ * server; otherwise we use the operation API only while the local instance is actually running
+ * (`getHdbPid` verifies process liveness, so a stale pid file reads as stopped).
+ */
+function useOperationApi(request: any): boolean {
+	if (request.target || process.env.HARPER_CLI_TARGET || process.env.CLI_TARGET || loadCredentials()?.last_target) {
+		return true;
+	}
+	initConfig();
+	return Boolean(getHdbPid());
+}
+
+/**
+ * `get_backup` streams a full-snapshot tar of the database's current state from a running server
+ * into a local file. It works against a remote server (`target=…`) or the local instance, resolving
+ * the target and auth the same way as any other CLI operation; a local connection requires Harper to
+ * be running (there is no offline form). For RocksDB the stream is gzipped by default; pass
+ * `gzip=false` for a plain tar. While stopped, use `create_backup` — a local incremental directory
+ * backup — instead.
+ */
+async function downloadBackup(request: any, databaseName: string): Promise<string> {
+	// only forward gzip when the user passed it — it is a RocksDB-only option (sending it
+	// unconditionally would fail LMDB downloads), and when unset the server applies its own
+	// default (RocksDB gzips)
+	const body: any = { operation: terms.OPERATIONS_ENUM.GET_BACKUP, database: databaseName };
+	if (request.gzip !== undefined) body.gzip = request.gzip;
+	// resolveRequestOptions connects to a remote target (target=/env/saved) or the local domain
+	// socket with auth; for a local connection it enforces that Harper is running.
+	const { options } = await resolveRequestOptions(request);
+	options.streamResponse = true;
+	const response = await httpRequest(options, body);
+	if (response.statusCode !== 200) {
+		const chunks: Buffer[] = [];
+		for await (const chunk of response) chunks.push(Buffer.from(chunk));
+		throw new Error(`get_backup failed (${response.statusCode}): ${Buffer.concat(chunks).toString('utf8')}`);
+	}
+	// name the file after what the server is sending (RocksDB: <db>.tar[.gz]; LMDB: the database file).
+	// basename() strips any path from the server-supplied filename — a malicious/compromised target
+	// could otherwise return `../../…` and steer the write outside the cwd.
+	const serverFilename = response.headers['content-disposition']?.match(/filename="([^"]+)"/)?.[1];
+	const outputPath = request.out || (serverFilename && basename(serverFilename)) || `${databaseName}.backup`;
+	await pipeline(response, createWriteStream(outputPath));
+	return `Backup of database '${databaseName}' written to ${outputPath}`;
+}

bin/cliOperations.ts

@@ -180,9 +180,16 @@ function resolveTarget(req, allCredentials) {
  * @param skipResponseLog By default, the response is logged to the console. Set this to true to skip logging it, which can be useful for sensitive responses like login calls!
  * @returns {Promise<void>}
  */
-async function cliOperations(req: any, skipResponseLog = false) {
-	require('dotenv').config();
-
+/**
+ * Resolve the transport options for a CLI operation request: a remote target URL (with auth) when
+ * one is configured (`target=`, env, or a saved `last_target`), otherwise the local domain socket.
+ * Returns the `options` object ready for `httpRequest` (method + Content-Type, and an Authorization
+ * header for remote targets, refreshing an expired operation token when possible) plus the resolved
+ * `target` (undefined for a local connection). Exits the process if a local connection is required
+ * but Harper is not running or has no domain socket. Shared by `cliOperations` and the CLI's
+ * streaming `get_backup` download so both reach local and remote servers the same way.
+ */
+export async function resolveRequestOptions(req: any): Promise<{ options: any; target: any }> {
 	const allCredentials = loadCredentials();
 	req.target = normalizeTarget(resolveTarget(req, allCredentials));
 	let target;
@@ -222,58 +229,65 @@ async function cliOperations(req: any, skipResponseLog = false) {
 			process.exit(1);
 		}
 	}
-	await PREPARE_OPERATION[req.operation]?.(req);
-	try {
-		let options = target ?? {
-			protocol: 'http:',
-			socketPath: getConfigPath(terms.CONFIG_PARAMS.OPERATIONSAPI_NETWORK_DOMAINSOCKET),
-		};
-		options.method = 'POST';
-		options.headers = { 'Content-Type': 'application/json' };
-		if (target?.username) {
-			options.headers.Authorization = `Basic ${Buffer.from(`${target.username}:${target.password}`).toString('base64')}`;
-		} else if (allCredentials) {
-			let tokens = null;
-			let lookupKey = null;
-			if (target && allCredentials.targets) {
-				lookupKey = target.resolvedTarget;
-				tokens = allCredentials.targets[lookupKey] ?? null;
-			}
+	let options = target ?? {
+		protocol: 'http:',
+		socketPath: getConfigPath(terms.CONFIG_PARAMS.OPERATIONSAPI_NETWORK_DOMAINSOCKET),
+	};
+	options.method = 'POST';
+	options.headers = { 'Content-Type': 'application/json' };
+	if (target?.username) {
+		options.headers.Authorization = `Basic ${Buffer.from(`${target.username}:${target.password}`).toString('base64')}`;
+	} else if (allCredentials) {
+		let tokens = null;
+		let lookupKey = null;
+		if (target && allCredentials.targets) {
+			lookupKey = target.resolvedTarget;
+			tokens = allCredentials.targets[lookupKey] ?? null;
+		}
 
-			if (tokens?.operation_token) {
-				if (tokens.refresh_token && isJWTExpired(tokens.operation_token)) {
-					console.error('Operation token expired, attempting to refresh...');
-					try {
-						const refreshOptions = { ...options };
-						refreshOptions.headers = { ...options.headers, Authorization: `Bearer ${tokens.refresh_token}` };
-						const refreshResponse = await httpRequest(refreshOptions, {
-							operation: 'refresh_operation_token',
-						});
-						if (refreshResponse.statusCode === 200) {
-							const refreshData = JSON.parse(refreshResponse.body);
-							if (refreshData.operation_token) {
-								tokens.operation_token = refreshData.operation_token;
-								saveCredentials(lookupKey || target?.resolvedTarget, {
-									operation_token: tokens.operation_token,
-									refresh_token: tokens.refresh_token,
-								});
-								console.error('Operation token refreshed successfully.');
-								// Update the original request's authorization header with the new token
-								options.headers.Authorization = `Bearer ${tokens.operation_token}`;
-							}
-						} else if (refreshResponse.statusCode === 401) {
-							console.error('Refresh token expired or invalid. Please run harper login again.');
-							process.exit(1);
-						} else {
-							console.error(`Failed to refresh operation token: ${refreshResponse.statusCode}`);
+		if (tokens?.operation_token) {
+			if (tokens.refresh_token && isJWTExpired(tokens.operation_token)) {
+				console.error('Operation token expired, attempting to refresh...');
+				try {
+					const refreshOptions = { ...options };
+					refreshOptions.headers = { ...options.headers, Authorization: `Bearer ${tokens.refresh_token}` };
+					const refreshResponse = await httpRequest(refreshOptions, {
+						operation: 'refresh_operation_token',
+					});
+					if (refreshResponse.statusCode === 200) {
+						const refreshData = JSON.parse(refreshResponse.body);
+						if (refreshData.operation_token) {
+							tokens.operation_token = refreshData.operation_token;
+							saveCredentials(lookupKey || target?.resolvedTarget, {
+								operation_token: tokens.operation_token,
+								refresh_token: tokens.refresh_token,
+							});
+							console.error('Operation token refreshed successfully.');
+							// Update the original request's authorization header with the new token
+							options.headers.Authorization = `Bearer ${tokens.operation_token}`;
 						}
-					} catch (refreshErr) {
-						console.error(`Error refreshing operation token: ${refreshErr.message}`);
+					} else if (refreshResponse.statusCode === 401) {
+						console.error('Refresh token expired or invalid. Please run harper login again.');
+						process.exit(1);
+					} else {
+						console.error(`Failed to refresh operation token: ${refreshResponse.statusCode}`);
 					}
+				} catch (refreshErr) {
+					console.error(`Error refreshing operation token: ${refreshErr.message}`);
 				}
-				options.headers.Authorization = `Bearer ${tokens.operation_token}`;
 			}
+			options.headers.Authorization = `Bearer ${tokens.operation_token}`;
 		}
+	}
+	return { options, target };
+}
+
+async function cliOperations(req: any, skipResponseLog = false) {
+	require('dotenv').config();
+
+	const { options, target } = await resolveRequestOptions(req);
+	await PREPARE_OPERATION[req.operation]?.(req);
+	try {
 		// Streaming deploy (multipart upload + SSE progress) only works against >= 5.1 servers.
 		// When deploying to a remote target, probe its version first and downgrade to the
 		// legacy JSON deploy if it predates 5.1. Local (domain-socket) deploys always

bin/harper.ts

@@ -8,7 +8,7 @@ import * as cliOperations from './cliOperations.ts';
 import { packageJson } from '../utility/packageUtils.js';
 import checkNode from '../launchServiceScripts/utility/checkNodeVersion.js';
 import * as hdbTerms from '../utility/hdbTerms.ts';
-const { SERVICE_ACTIONS_ENUM } = hdbTerms as any;
+const { SERVICE_ACTIONS_ENUM, OPERATIONS_ENUM } = hdbTerms as any;
 if (typeof process.setSourceMapsEnabled === 'function') {
 	process.setSourceMapsEnabled(true); // this is necessary for source maps to work, at least on the main thread.
 }
@@ -25,6 +25,7 @@ By default, the CLI also supports certain Operation APIs. Specify the operation
 Commands:
 copy-db <source> <target>       - Copies a database from source path to target path
 dev <path>                      - Run the application in dev mode with debugging, foreground logging, no auth
+get_backup database=<name> [gzip=false] [out=<file>] - Download a full backup from a running server (RocksDB: gzipped tar stream, gzip=false for plain tar)
 install                         - Install harperdb
 <api-operation> <param>=<value> - Run an API operation and return result to the CLI, not all operations are supported
 login [target] [username]       - Login to a remote or local Harper instance
@@ -112,6 +113,14 @@ async function harper() {
 			let targetDbPath = process.argv[4];
 			return require('./copyDb').copyDb(sourceDb, targetDbPath);
 		}
+		case OPERATIONS_ENUM.CREATE_BACKUP:
+		case OPERATIONS_ENUM.LIST_BACKUPS:
+		case OPERATIONS_ENUM.VERIFY_BACKUP:
+		case OPERATIONS_ENUM.DELETE_BACKUP:
+		case OPERATIONS_ENUM.PURGE_BACKUPS:
+		case OPERATIONS_ENUM.GET_BACKUP:
+		case OPERATIONS_ENUM.RESTORE_BACKUP:
+			return require('./backup').runBackupCommand(service);
 		case SERVICE_ACTIONS_ENUM.DEV:
 			process.env.DEV_MODE = 'true';
 		// fall through

components/mcp/tools/operations.ts

@@ -175,6 +175,9 @@ const DESTRUCTIVE_OPERATIONS: ReadonlySet<string> = new Set([
 	'restart_service',
 	'set_configuration',
 	'remove_node',
+	'restore_backup',
+	'delete_backup',
+	'purge_backups',
 ]);
 
 /**