Document RocksDB backup & restore operations

reference/cli/commands.md

@@ -294,6 +294,73 @@ Database files are stored in the `hdb/database` directory. As long as the snapsh
 - Cloud provider volume snapshots (AWS EBS, Azure Disk, GCP Persistent Disk)
 - Enterprise backup solutions with snapshot capabilities
 
+## Backup Commands
+
+<VersionBadge version="v5.2.0" />
+
+These are the RocksDB backup [operations](../operations-api/operations.md#backup--restore) invoked from the CLI. Each uses its **operation name** — there is no hyphenated form: `create_backup`, `list_backups`, `verify_backup`, `delete_backup`, `purge_backups`, `restore_backup`, and `get_backup`. Managed backups are a server-side, incremental, checksum-verified repository under the configured backup root (`storage.backupPath`, default `<rootPath>/backup`), one subdirectory per database.
+
+Each command works whether or not Harper is running. When Harper is running, the operation is forwarded to the server; when the local server is **stopped**, the command operates directly on the database/backup files. RocksDB is single-writer, so the process holding the database open must do the work: an in-place `restore_backup` of the `system` database, or of any database a loaded component keeps open, must therefore be run with the server stopped. `get_backup` is the exception — it streams from a **running** server and has no offline form.
+
+All commands accept `database=<name>` (defaults to `data`). Like any CLI operation, they also accept a remote `target=<url>` to run against another instance instead of the local one (see [Remote Operations](#remote-operations)); with a remote target the operation is always forwarded (there is no offline path).
+
+### `harper create_backup`
+
+Create an incremental directory backup of a database.
+
+```bash
+harper create_backup database=data
+```
+
+### `harper list_backups`
+
+List the managed backups for a database (id, timestamp, size, file count).
+
+```bash
+harper list_backups database=data
+```
+
+### `harper verify_backup`
+
+Verify a managed backup's integrity. Pass `verify_checksum=true` to also verify checksums (slower).
+
+```bash
+harper verify_backup database=data backup_id=1 verify_checksum=true
+```
+
+### `harper delete_backup`
+
+Delete a single managed backup.
+
+```bash
+harper delete_backup database=data backup_id=1
+```
+
+### `harper purge_backups`
+
+Delete all but the newest `keep_count` managed backups.
+
+```bash
+harper purge_backups database=data keep_count=3
+```
+
+### `harper get_backup`
+
+Download a full snapshot of a database to a local file, from the local instance or a remote `target=<url>` (the server must be **running** — there is no offline form). For RocksDB this is a `tar` archive, **gzipped by default** (`data.tar.gz`); pass `gzip=false` for a plain `tar`. For LMDB it is the `.mdb` file. Pass `out=<path>` to choose the output file (defaults to a name derived from the response).
+
+```bash
+harper get_backup database=data out=./data.tar.gz
+harper get_backup database=data target=https://node-2.example.com:9925 out=./data.tar.gz
+```
+
+### `harper restore_backup`
+
+Restore a database from a managed backup. Pass `backup_id=<id>` to choose a backup (defaults to the latest), or `target_database=<name>` to restore into a new database rather than in place. Run it with the server **stopped** to restore the `system` database, any database a loaded component keeps open, or into a `target_database`; while Harper is running, those cases are rejected with a pointer to stop the server.
+
+```bash
+harper restore_backup database=data backup_id=1
+```
+
 ## Remote Operations
 
 The CLI supports executing commands on remote Harper instances. For details, see [CLI Overview - Remote Operations](./overview.md#remote-operations).

reference/configuration/options.md

@@ -257,6 +257,7 @@ storage:
 - `noReadAhead` — Advise OS against read-ahead; _Default_: `false`
 - `prefetchWrites` — Prefetch before write transactions; _Default_: `true`
 - `path` — Database files directory; _Default_: `<rootPath>/database`
+- `backupPath` — Directory for managed database backups (created by [`create_backup`](../operations-api/operations.md#backup--restore)), one subdirectory per database; _Default_: `<rootPath>/backup` (Added in: v5.2.0)
 - `blobPaths` — Blob storage directory or directories; _Default_: `<rootPath>/blobs` (Added in: v4.5.0)
 - `pageSize` — Database page size (bytes); _Default_: OS default
 - `reclamation.threshold` — Free-space ratio below which reclamation begins evicting from caching tables; _Default_: `0.4` (Added in: v4.5.0)

reference/operations-api/operations.md

@@ -44,7 +44,6 @@ Detailed documentation: [Database Overview](../database/overview.md)
 | `drop_table`        | Drops a table and all its records                                   | super_user    |
 | `create_attribute`  | Adds a new attribute to a table                                     | super_user    |
 | `drop_attribute`    | Removes an attribute and all its values from a table                | super_user    |
-| `get_backup`        | Returns a binary snapshot of a database for backup purposes         | super_user    |
 
 ### `describe_all`
 
@@ -133,9 +132,114 @@ Drops an attribute and all its values from the specified table.
 }
 ```
 
+---
+
+## Backup & Restore
+
+Operations for backing up and restoring databases. A backup is always a whole-database copy (all tables plus the audit/transaction log); there is no per-table granularity.
+
+Two complementary mechanisms are exposed:
+
+- **Managed backups** <VersionBadge version="v5.2.0" /> — a server-side, incremental, checksum-verified repository under the configured backup root (`storage.backupPath`, default `<rootPath>/backup`), one subdirectory per database. Create, list, verify, delete, purge, and restore are all available. These operations require the RocksDB storage engine.
+- **Download** — `get_backup` streams a full snapshot of the database in the HTTP response with no server-side artifact, for pulling a copy off-host.
+
+Every managed backup and every RocksDB `get_backup` includes the audit/transaction log, so a restored database keeps its `read_audit_log` history as of the backup point. A restore is a point-in-time rollback; in a replicated cluster, coordinate a restore with replication before bringing the node back.
+
+| Operation        | Description                                                         | Role Required |
+| ---------------- | ------------------------------------------------------------------- | ------------- |
+| `create_backup`  | Creates a managed, incremental directory backup of a database (job) | super_user    |
+| `list_backups`   | Lists the managed backups for a database                            | super_user    |
+| `verify_backup`  | Verifies a managed backup's integrity (job)                         | super_user    |
+| `delete_backup`  | Deletes a single managed backup                                     | super_user    |
+| `purge_backups`  | Deletes all but the newest N managed backups                        | super_user    |
+| `restore_backup` | Restores a database from a managed backup (job)                     | super_user    |
+| `get_backup`     | Streams a full snapshot of a database in the response for download  | super_user    |
+
+### `create_backup`
+
+<VersionBadge version="v5.2.0" />
+
+Creates an incremental directory backup of the database (RocksDB only) under the configured backup root. Runs as a background [job](#jobs): the operation returns a `job_id` immediately, and [`get_job`](#get_job) reports the outcome including the new `backupId`, `size`, and `timestamp`. `database` defaults to `data`.
+
+Directory backups in the same location share unchanged files, so the second and subsequent backups of a database only copy what changed. Use `purge_backups` to manage retention.
+
+```json
+{ "operation": "create_backup", "database": "dev" }
+```
+
+### `list_backups`
+
+<VersionBadge version="v5.2.0" />
+
+Returns the managed backups for a database (RocksDB only), each with its `backupId`, `timestamp`, `size`, and file count. Returns an empty array if no backups have been created yet.
+
+```json
+{ "operation": "list_backups", "database": "dev" }
+```
+
+### `verify_backup`
+
+<VersionBadge version="v5.2.0" />
+
+Verifies a managed backup's file sizes, and optionally their checksums when `verify_checksum` is `true` (slower). Runs as a background [job](#jobs). `backup_id` is required.
+
+```json
+{ "operation": "verify_backup", "database": "dev", "backup_id": 1, "verify_checksum": true }
+```
+
+### `delete_backup`
+
+<VersionBadge version="v5.2.0" />
+
+Deletes a single managed backup. Files shared with other backups are reference-counted and removed only when no remaining backup references them. `backup_id` is required.
+
+```json
+{ "operation": "delete_backup", "database": "dev", "backup_id": 1 }
+```
+
+### `purge_backups`
+
+<VersionBadge version="v5.2.0" />
+
+Deletes all but the newest `keep_count` managed backups, returning the number `deleted` and the number `remaining`.
+
+```json
+{ "operation": "purge_backups", "database": "dev", "keep_count": 3 }
+```
+
+### `restore_backup`
+
+<VersionBadge version="v5.2.0" />
+
+Restores a database in place from a managed backup (RocksDB only), without stopping Harper. Runs as a background [job](#jobs): Harper closes the database across all worker threads, restores it, and reloads it. `backup_id` defaults to the latest backup. The audit/transaction log is restored alongside the data.
+
+```json
+{ "operation": "restore_backup", "database": "dev", "backup_id": 1 }
+```
+
+#### When can a database be restored?
+
+An in-place restore purges and rewrites the database's files, which requires the database to be **fully closed** first. Harper closes it across its own worker threads, but it cannot close a handle held by a loaded component, nor stop the `system` database it depends on to run. So whether a restore can run online depends on what is holding the database open:
+
+| Database                                           | Online `restore_backup` (server running) | Offline `harper restore_backup` (server stopped) |
+| -------------------------------------------------- | ---------------------------------------- | ------------------------------------------------ |
+| A user database not opened by any loaded component | Yes — restored in place                  | Yes                                              |
+| A user database that a loaded component keeps open | No — the job fails with a `409`          | Yes                                              |
+| The `system` database                              | No — rejected up front                   | Yes                                              |
+
+- **Online** (this operation) works only when no loaded component is holding the database open. If a component is using it, the job fails fast with a `409` telling you to restore offline — Harper cannot force a component's handles closed while the server is running, and restoring under an open instance would corrupt it. (Harper does not track which component uses which database, so it cannot selectively stop one; it can only detect that the database is still open.)
+- **Offline** — the same [`restore_backup`](../cli/commands.md#harper-restore_backup) command run from the CLI with the server **stopped** — works for **any** database, because no components are loaded and nothing holds it open. This is the required path for the `system` database and for any database a component keeps open.
+
+Online restore always restores in place. To restore into a new database instead of overwriting the source, run `restore_backup` with `target_database` from the CLI with the server stopped.
+
+A restore is a point-in-time rollback; in a replicated cluster, coordinate it with replication before bringing the node back (see the note at the top of this section).
+
 ### `get_backup`
 
-Returns a binary snapshot of the specified database (or individual table). Safe for backup while Harper is running. Specify `"table"` for a single table or `"tables"` for a set.
+Streams a full snapshot of the specified database in the HTTP response for download - there is no server-side artifact and nothing to clean up. Behavior depends on the storage engine:
+
+- **RocksDB** <VersionBadge type="changed" version="v5.2.0" /> — streams a `tar` archive of the current database state (all tables plus the transaction log), **gzipped by default** (`gzip` is RocksDB-only and compresses the snapshot substantially). Pass `"gzip": false` for a plain `tar`. It is always the current state; downloading a specific historical `backup_id` is not supported - to move a retained backup off-host, copy the backup directory.
+- **LMDB** — streams the `.mdb` file. Specify `"table"` for a single table or `"tables"` for a set, and `"include_audit": true` to include the audit store. These options are LMDB-only.
 
 ```json
 { "operation": "get_backup", "database": "dev" }

release-notes/v5-lincoln/5.2.md

@@ -8,6 +8,12 @@ title: '5.2'
 
 All patch release notes for 5.2.x are available on the [releases page](https://github.com/HarperFast/harper/releases?q=v5.2&expanded=true).
 
+## Backup & Restore
+
+New managed-backup operations for RocksDB databases: `create_backup`, `list_backups`, `verify_backup`, `delete_backup`, `purge_backups`, and `restore_backup`. Managed backups are a server-side, incremental, checksum-verified repository under the configured backup root (`storage.backupPath`, default `<rootPath>/backup`), and every backup captures the audit/transaction log alongside the data. `restore_backup` restores a user database in place without stopping Harper, as long as no loaded component is holding that database open; a database held open by a component — and always the `system` database — is restored offline by running `restore_backup` from the CLI with the server stopped. Each of these operations can be run from the CLI under its operation name (e.g. `harper create_backup database=data`): while Harper is running the operation is forwarded to the server, and while it is stopped the command operates directly on the files. See [Backup & Restore Operations](/reference/v5/operations-api/operations#backup--restore) and [Backup Commands](/reference/v5/cli/commands#backup-commands).
+
+`get_backup` now understands RocksDB: it streams a full-snapshot `tar` of the database — gzipped by default (pass `gzip: false` for a plain `tar`) — instead of failing. The LMDB behavior (streaming the `.mdb`, with `table`/`tables`/`include_audit`) is unchanged.
+
 ## Querying
 
 ### Filtered Vector Search (Predicate-Aware HNSW Traversal)