Document RocksDB backup & restore operations

AGENTS.md

@@ -47,6 +47,7 @@ Prefer plain ASCII characters in Markdown unless a typographic character is genu
 - Reference is versioned; current version is v5 (`reference/`). Archived v4 content is in `reference_versioned_docs/version-v4/`.
 - Do not modify `reference_versioned_docs/` for current (v5) work; edit `reference/` instead.
 - The `<VersionBadge>` component is globally registered — no import needed in `.md`/`.mdx` files.
+- The `<EngineBadge>` component (also global) tags storage-engine support inline: `<EngineBadge engines="RocksDB" />` or `<EngineBadge engines="RocksDB, LMDB" />`.
 - See the complete repository organization in `CONTRIBUTING.md`
 
 ## Versioning Content

reference/backups/operations.md

@@ -0,0 +1,140 @@
+---
+id: operations
+title: Backup Operations
+---
+
+<!-- Source: HarperFast/harper#1831 (RocksDB managed backup and restore operations) -->
+
+Operations for backing up and restoring databases. For how the backup system works, its limitations, and worked examples, see the [Backups Overview](./overview.md).
+
+All backup operations require a `super_user` role when invoked through a running server; run offline (from the CLI with the server stopped), they are governed by filesystem permissions instead. All accept a `database` parameter that defaults to `data`. The engine badge on each operation indicates which storage engines support it: the managed-backup operations require RocksDB, while `get_backup` works with both RocksDB and LMDB.
+
+| Operation                           | Description                                                         | Role Required |
+| ----------------------------------- | ------------------------------------------------------------------- | ------------- |
+| [`create_backup`](#create_backup)   | Creates a managed, incremental directory backup of a database (job) | super_user    |
+| [`list_backups`](#list_backups)     | Lists the managed backups for a database                            | super_user    |
+| [`verify_backup`](#verify_backup)   | Verifies a managed backup's integrity (job)                         | super_user    |
+| [`delete_backup`](#delete_backup)   | Deletes a single managed backup                                     | super_user    |
+| [`purge_backups`](#purge_backups)   | Deletes all but the newest `keep_count` managed backups             | super_user    |
+| [`restore_backup`](#restore_backup) | Restores a database from a managed backup (job)                     | super_user    |
+| [`get_backup`](#get_backup)         | Streams a full snapshot of a database in the response for download  | super_user    |
+
+## Running backup operations from the CLI
+
+Every operation on this page can be run from the CLI under its operation name, for example `harper create_backup database=data`. When Harper is running, the CLI forwards the operation to the server; when it is stopped, the command operates directly on the database and backup files. `get_backup` is the exception — it streams from a running server and has no offline form.
+
+Like any [CLI operation](../cli/operations-api-commands.md), backup commands also accept a remote `target=<url>` to run against another instance instead of the local one (see [Remote Operations](../cli/overview.md#remote-operations)). With a remote target the operation is always forwarded — there is no offline path.
+
+Offline invocation matters most for restore: an in-place `restore_backup` of the `system` database, or of any database a loaded component keeps open, must be run with the server stopped. See [when can a database be restored?](./overview.md#when-can-a-database-be-restored)
+
+## `create_backup`
+
+<VersionBadge version="v5.2.0" /> <EngineBadge engines="RocksDB" />
+
+Creates an incremental directory backup of the database under the configured backup root ([`storage.backupPath`](../configuration/options.md#storage), default `<rootPath>/backup`). Through a running server this runs as a background [job](../operations-api/operations.md#jobs): the operation returns a `job_id` immediately, and [`get_job`](../operations-api/operations.md#get_job) reports the outcome including the new `backup_id`, `size`, and `timestamp`.
+
+Backups of the same database share unchanged files, so the second and subsequent backups only copy what changed. Use [`purge_backups`](#purge_backups) to manage retention.
+
+```json
+{ "operation": "create_backup", "database": "data" }
+```
+
+```bash
+harper create_backup database=data
+```
+
+## `list_backups`
+
+<VersionBadge version="v5.2.0" /> <EngineBadge engines="RocksDB" />
+
+Returns the managed backups for a database, each with its `backup_id`, `timestamp`, `size`, and `file_count`. Returns an empty array if no backups have been created yet.
+
+```json
+{ "operation": "list_backups", "database": "data" }
+```
+
+```bash
+harper list_backups database=data
+```
+
+## `verify_backup`
+
+<VersionBadge version="v5.2.0" /> <EngineBadge engines="RocksDB" />
+
+Verifies a managed backup's file sizes, and optionally their checksums when `verify_checksum` is `true` (slower). Through a running server this runs as a background [job](../operations-api/operations.md#jobs). `backup_id` is required.
+
+```json
+{ "operation": "verify_backup", "database": "data", "backup_id": 1, "verify_checksum": true }
+```
+
+```bash
+harper verify_backup database=data backup_id=1 verify_checksum=true
+```
+
+## `delete_backup`
+
+<VersionBadge version="v5.2.0" /> <EngineBadge engines="RocksDB" />
+
+Deletes a single managed backup. Files shared with other backups are reference-counted and removed only when no remaining backup references them. `backup_id` is required.
+
+```json
+{ "operation": "delete_backup", "database": "data", "backup_id": 1 }
+```
+
+```bash
+harper delete_backup database=data backup_id=1
+```
+
+## `purge_backups`
+
+<VersionBadge version="v5.2.0" /> <EngineBadge engines="RocksDB" />
+
+Deletes all but the newest `keep_count` managed backups, returning the number `deleted` and the number `remaining`.
+
+```json
+{ "operation": "purge_backups", "database": "data", "keep_count": 3 }
+```
+
+```bash
+harper purge_backups database=data keep_count=3
+```
+
+## `restore_backup`
+
+<VersionBadge version="v5.2.0" /> <EngineBadge engines="RocksDB" />
+
+Restores a database in place from a managed backup. `backup_id` defaults to the latest backup. The audit/transaction log is restored alongside the data.
+
+Through a running server this runs as a background [job](../operations-api/operations.md#jobs): Harper closes the database across all worker threads, restores it, and reloads it. This works only when no loaded component is holding the database open — restoring the `system` database, or a database a component keeps open, requires running the command from the CLI with the server stopped. See [when can a database be restored?](./overview.md#when-can-a-database-be-restored)
+
+From the CLI with the server stopped, `target_database=<name>` restores into a new database instead of overwriting the source. The target must not already exist; Harper picks the new database up on the next start.
+
+```json
+{ "operation": "restore_backup", "database": "data", "backup_id": 1 }
+```
+
+```bash
+harper restore_backup database=data backup_id=1
+# restore into a new database instead of overwriting the source (server stopped)
+harper restore_backup database=data backup_id=1 target_database=data_restored
+```
+
+## `get_backup`
+
+<EngineBadge engines="RocksDB, LMDB" />
+
+Streams a full snapshot of the specified database in the HTTP response for download — there is no server-side artifact and nothing to clean up. The server must be running; this is the one backup operation with no offline form. Behavior depends on the storage engine:
+
+- **RocksDB** <VersionBadge type="changed" version="v5.2.0" /> — streams a `tar` archive of the current database state (all tables plus the transaction log), gzipped by default (`gzip` is a RocksDB-only option and compresses the snapshot substantially). Pass `"gzip": false` for a plain `tar`. The snapshot is always the current state; downloading a specific historical `backup_id` is not supported — to move a retained backup off-host, copy its backup directory.
+- **LMDB** — streams the `.mdb` file. Specify `"table"` for a single table or `"tables"` for a set, and `"include_audit": true` to include the audit store. These options are LMDB-only.
+
+```json
+{ "operation": "get_backup", "database": "data" }
+```
+
+From the CLI, pass `out=<path>` to choose the output file (defaults to a name derived from the response):
+
+```bash
+harper get_backup database=data out=./data.tar.gz
+harper get_backup database=data target=https://node-2.example.com:9925 out=./data.tar.gz
+```

reference/backups/overview.md

@@ -0,0 +1,107 @@
+---
+id: overview
+title: Backups
+---
+
+<!-- Source: HarperFast/harper#1831 (RocksDB managed backup and restore operations) -->
+
+Harper can back up and restore its databases natively. A backup is a whole-database copy — all tables plus the audit/transaction log — so a restored database is a consistent point-in-time image of everything in it. (`get_backup` on an LMDB database is the exception: it can stream a subset of tables, and includes the audit store only when requested.)
+
+Two complementary mechanisms are available:
+
+- **Managed backups** <VersionBadge version="v5.2.0" /> <EngineBadge engines="RocksDB" /> — incremental, checksum-verified backups kept in a server-side repository. Harper creates, lists, verifies, deletes, purges, and restores them through the [backup operations](./operations.md).
+- **Snapshot download** <EngineBadge engines="RocksDB, LMDB" /> — [`get_backup`](./operations.md#get_backup) streams a full snapshot of a database over HTTP for storage off-host, with no server-side artifact.
+
+Every backup operation is available through the [Operations API](../operations-api/overview.md) and as a [CLI command](./operations.md#running-backup-operations-from-the-cli) under the same operation name.
+
+## How managed backups work
+
+Managed backups use the RocksDB backup engine. Each backup is a directory under the configured backup root ([`storage.backupPath`](../configuration/options.md#storage), default `<rootPath>/backup`), with one subdirectory per database. Because RocksDB data files are immutable once written, backups in the same location share unchanged files: the first backup copies the whole database, and each subsequent backup only copies what changed since the last one. Shared files are reference-counted, so deleting a backup only removes files no remaining backup references.
+
+Every managed backup includes the database's transaction log, so a restored database keeps its `read_audit_log` history as of the backup point.
+
+The long-running operations (`create_backup`, `verify_backup`, `restore_backup`) run as background [jobs](../operations-api/operations.md#jobs) when invoked through a running server: the operation returns a `job_id` immediately, and [`get_job`](../operations-api/operations.md#get_job) reports the outcome.
+
+## Online and offline
+
+Backup operations work whether or not Harper is running. Invoked from the CLI while the server is running, the operation is forwarded to the server; while the server is stopped, the command operates directly on the database and backup files. `get_backup` is the exception — it streams from a running server and has no offline form.
+
+The offline path matters for restore. RocksDB is single-writer, so an in-place restore requires the database to be fully closed first. A running Harper server can close its own worker threads' handles, but it cannot close a handle held by a loaded component, nor stop the `system` database it depends on to run — those databases can only be restored with the server stopped. See [when can a database be restored?](#when-can-a-database-be-restored) below.
+
+## Limitations
+
+- **Managed backups require the RocksDB storage engine.** For LMDB databases, use [`get_backup`](./operations.md#get_backup) or [volume snapshots](../cli/commands.md#how-backups-work).
+- **Whole-database granularity.** There is no per-table backup or restore. (The one exception: `get_backup` on an LMDB database can stream individual tables, and includes the audit store only with `include_audit`.)
+- **One storage root per database.** A database whose tables use per-table `path` storage configs spans multiple root stores and cannot be backed up with these operations.
+- **Backups live on the node that created them.** The backup repository is a local directory. For disaster recovery, copy backup directories off-host, or use `get_backup` to pull snapshots from a running server.
+- **`get_backup` always streams the current state.** It cannot download a historical managed backup; to move a retained backup off-host, copy its backup directory.
+- **A restore is a point-in-time rollback.** In a replicated cluster, coordinate a restore with replication before bringing the node back.
+
+### When can a database be restored?
+
+An in-place restore purges and rewrites the database's files, which requires the database to be fully closed first. Whether a restore can run online depends on what is holding the database open:
+
+| Database                                           | Online `restore_backup` (server running) | Offline `harper restore_backup` (server stopped) |
+| -------------------------------------------------- | ---------------------------------------- | ------------------------------------------------ |
+| A user database not opened by any loaded component | Yes — restored in place                  | Yes                                              |
+| A user database that a loaded component keeps open | No — the job fails with a `409`          | Yes                                              |
+| The `system` database                              | No — rejected up front                   | Yes                                              |
+
+- **Online** works only when no loaded component is holding the database open. If a component is using it, the job fails fast with a `409` telling you to restore offline — Harper cannot force a component's handles closed while the server is running, and restoring under an open instance would corrupt it. (Harper does not track which component uses which database, so it cannot selectively stop one; it can only detect that the database is still open.)
+- **Offline** — the same [`restore_backup`](./operations.md#restore_backup) command run from the CLI with the server stopped — works for any database, because no components are loaded and nothing holds it open. This is the required path for the `system` database and for any database a component keeps open.
+
+Online restore always restores in place. To restore into a new database instead of overwriting the source, run `restore_backup` with `target_database` from the CLI with the server stopped.
+
+## Example: incremental backups and restore
+
+Create a managed backup of the `data` database (the default):
+
+```bash
+harper create_backup database=data
+```
+
+The first backup copies the entire database; each one after that only copies files that changed, so frequent backups are cheap. Schedule `create_backup` as often as your recovery point requires, and manage retention with `purge_backups`:
+
+```bash
+harper purge_backups database=data keep_count=7
+```
+
+List the backups to find the one to restore:
+
+```bash
+harper list_backups database=data
+```
+
+Restore the latest backup, or pass `backup_id=<id>` for an earlier one:
+
+```bash
+harper restore_backup database=data
+```
+
+With the server running, this restores the database in place — Harper closes the database across its worker threads, restores it, and reloads it — as long as nothing is holding the database open (see [when can a database be restored?](#when-can-a-database-be-restored)). With the server stopped, the same command restores the files directly and works for any database.
+
+## Example: download a snapshot and restore it manually
+
+`get_backup` streams a full snapshot of a database from a running server — local or remote — which makes it the simplest way to keep backups off-host. For a RocksDB database the stream is a `tar` archive, gzipped by default:
+
+```bash
+harper get_backup database=data out=./data.tar.gz
+# or pull from another instance
+harper get_backup database=data target=https://node-2.example.com:9925 out=./data.tar.gz
+```
+
+The archive contains the database's current state — all tables plus the transaction log — and extracts into a directory that opens directly as a RocksDB database.
+
+To restore it, stop Harper, replace the database's directory under the storage path ([`storage.path`](../configuration/options.md#storage), default `<rootPath>/database`) with the extracted archive, and start Harper again:
+
+```bash
+harper stop
+mv ~/hdb/database/data ~/hdb/database/data.old
+mkdir -p ~/hdb/database/data
+tar -xzf data.tar.gz -C ~/hdb/database/data
+harper start
+```
+
+Keeping the previous directory as `data.old` makes the swap easy to undo; delete it once the restored database checks out.
+
+For an LMDB database, `get_backup` streams the database's `.mdb` file instead; restore it by replacing the database's `.mdb` file under the same storage path (`<storage.path>/<database>.mdb`) with the downloaded file while Harper is stopped.

reference/cli/commands.md

@@ -294,6 +294,8 @@ Database files are stored in the `hdb/database` directory. As long as the snapsh
 - Cloud provider volume snapshots (AWS EBS, Azure Disk, GCP Persistent Disk)
 - Enterprise backup solutions with snapshot capabilities
 
+Harper also has a built-in backup system — managed incremental backups and snapshot downloads, runnable from the CLI under their operation names (e.g. `harper create_backup`). See [Backups](../backups/overview.md).
+
 ## Remote Operations
 
 The CLI supports executing commands on remote Harper instances. For details, see [CLI Overview - Remote Operations](./overview.md#remote-operations).

reference/cli/operations-api-commands.md

@@ -97,6 +97,13 @@ This is just a brief overview of all operations available as CLI commands. Revie
 | `get_fingerprint`                | Get instance fingerprint              | [Licensing](../operations-api/operations.md#registration--licensing)   | v4.3.0          |
 | `set_license`                    | Set license key                       | [Licensing](../operations-api/operations.md#registration--licensing)   | v4.3.0          |
 | `get_usage_licenses`             | Get usage and license info            | [Licensing](../operations-api/operations.md#registration--licensing)   | v4.7.3          |
+| `create_backup`                  | Create a managed database backup      | [Backups](../backups/operations.md)                                    | v5.2.0          |
+| `list_backups`                   | List managed backups                  | [Backups](../backups/operations.md)                                    | v5.2.0          |
+| `verify_backup`                  | Verify a managed backup's integrity   | [Backups](../backups/operations.md)                                    | v5.2.0          |
+| `delete_backup`                  | Delete a managed backup               | [Backups](../backups/operations.md)                                    | v5.2.0          |
+| `purge_backups`                  | Delete all but the newest backups     | [Backups](../backups/operations.md)                                    | v5.2.0          |
+| `restore_backup`                 | Restore a database from a backup      | [Backups](../backups/operations.md)                                    | v5.2.0          |
+| `get_backup`                     | Download a full database snapshot     | [Backups](../backups/operations.md)                                    | v5.2.0          |
 | `get_job`                        | Get job status                        | [Jobs](../operations-api/operations.md#jobs)                           | v4.3.0          |
 | `search_jobs_by_start_date`      | Search jobs by start date             | [Jobs](../operations-api/operations.md#jobs)                           | v4.3.0          |
 | `read_log`                       | Read application logs                 | [Logging](../operations-api/operations.md#logs)                        | v4.3.0          |