diff --git a/polaris-synchronizer/.gitignore b/polaris-synchronizer/.gitignore index e85ca10a..883eda77 100644 --- a/polaris-synchronizer/.gitignore +++ b/polaris-synchronizer/.gitignore @@ -95,3 +95,6 @@ build # Ignore Gradle wrapper jar file gradle/wrapper/gradle-wrapper.jar gradle/wrapper/gradle-wrapper-*.sha256 + +# Generated test credentials (manual-test docker-compose seed volume) +manual-test/seed/ diff --git a/polaris-synchronizer/README.md b/polaris-synchronizer/README.md index c7e754a3..c6c1b6e5 100644 --- a/polaris-synchronizer/README.md +++ b/polaris-synchronizer/README.md @@ -184,3 +184,14 @@ java -jar cli/build/libs/polaris-synchronizer-cli.jar sync-polaris \ > nor remove or modify them or their assignments to principals/principal-roles on the target. This is to accommodate that > the tool itself will be running with the permission levels for these principals and roles, and we do not want to modify > the tool's permissions at runtime. + +> :bulb: If you only need to migrate Polaris management entities (catalogs, catalog-roles, grants) and want to skip +> synchronizing Iceberg namespaces and tables, run the tool with the `--skip-iceberg-content` flag. This is useful when +> Iceberg content is synchronized through another means, or when speeding up a sync where namespace/table content isn't needed. + +# Testing Locally + +See [`manual-test/`](manual-test/README.md) for a self-contained docker-compose setup that brings up a source and +target Polaris instance and walks through running `sync-polaris`, including with `--skip-iceberg-content`, without +needing a real Polaris deployment. + diff --git a/polaris-synchronizer/manual-test/README.md b/polaris-synchronizer/manual-test/README.md new file mode 100644 index 00000000..c1bd6c28 --- /dev/null +++ b/polaris-synchronizer/manual-test/README.md @@ -0,0 +1,122 @@ +# Manual Test Setup + +Brings up two independent Apache Polaris instances (source + target) to +manually validate `polaris-synchronizer`, including the `--skip-iceberg-content` +flag, without needing Postgres/RDS or S3/MinIO: + +* Both instances use Polaris's default **in-memory metastore** (no datasource + configured). +* Both instances share a single **FILE storage volume**, mounted at the same + path in each container — so table metadata written by source is readable + by target, and a full sync (including namespace/table content) actually + succeeds. This lets you run the full sync and the `--skip-iceberg-content` + sync back-to-back in the same session and compare the results. + +`polaris-init` seeds SOURCE with a catalog, principal, principal-role, +catalog-role, grant, namespace, and table. TARGET is left completely empty — +`polaris-synchronizer` is responsible for creating everything there. + +## Usage + +```bash +cd polaris-synchronizer/manual-test +docker compose up --build -d +docker compose ps # wait for polaris-init to show "Exited (0)" +cat seed/credentials.json +``` + +Polaris REST/management APIs are reachable at `http://localhost:8181` (source) +and `http://localhost:8183` (target). + +Build the CLI jar from the `polaris-synchronizer` root: + +```bash +cd .. +./gradlew assemble +# jar is at cli/build/libs/polaris-synchronizer-cli.jar +``` + +### Step 1: Create a read-only omnipotent principal on source + +```bash +java -jar cli/build/libs/polaris-synchronizer-cli.jar create-omnipotent-principal \ + --polaris-api-connection-properties base-url=http://localhost:8181 \ + --polaris-api-connection-properties oauth2-server-uri=http://localhost:8181/api/catalog/v1/oauth/tokens \ + --polaris-api-connection-properties credential=root:s3cr3t \ + --polaris-api-connection-properties scope=PRINCIPAL_ROLE:ALL \ + --replace +``` + +Note the `clientId`/`clientSecret` printed at the end — this is +`SOURCE_OMNI_ID` / `SOURCE_OMNI_SECRET` below. + +### Step 2: Create a read-write omnipotent principal on target + +```bash +java -jar cli/build/libs/polaris-synchronizer-cli.jar create-omnipotent-principal \ + --polaris-api-connection-properties base-url=http://localhost:8183 \ + --polaris-api-connection-properties oauth2-server-uri=http://localhost:8183/api/catalog/v1/oauth/tokens \ + --polaris-api-connection-properties credential=root:s3cr3t \ + --polaris-api-connection-properties scope=PRINCIPAL_ROLE:ALL \ + --replace \ + --write-access +``` + +Note the credentials printed here — this is `TARGET_OMNI_ID` / `TARGET_OMNI_SECRET`. + +Since TARGET starts with no catalogs, this step will report 0 catalogs +processed — that's expected. `sync-polaris` will create the target's catalog +before syncing catalog-roles/grants, so there's nothing to set up yet. + +### Step 3a: Run with `--skip-iceberg-content` + +```bash +java -jar cli/build/libs/polaris-synchronizer-cli.jar sync-polaris \ + --source-properties base-url=http://localhost:8181 \ + --source-properties credential=root:s3cr3t \ + --source-properties oauth2-server-uri=http://localhost:8181/api/catalog/v1/oauth/tokens \ + --source-properties scope=PRINCIPAL_ROLE:ALL \ + --source-properties omnipotent-principal-name= \ + --source-properties omnipotent-principal-client-id= \ + --source-properties omnipotent-principal-client-secret= \ + --target-properties base-url=http://localhost:8183 \ + --target-properties credential=root:s3cr3t \ + --target-properties oauth2-server-uri=http://localhost:8183/api/catalog/v1/oauth/tokens \ + --target-properties scope=PRINCIPAL_ROLE:ALL \ + --target-properties omnipotent-principal-name= \ + --target-properties omnipotent-principal-client-id= \ + --target-properties omnipotent-principal-client-secret= \ + --skip-iceberg-content +``` + +Catalog, catalog-role, and grant sync complete, but namespace/table sync is +skipped entirely — verify below that target has zero namespaces. + +### Step 3b: Re-run without `--skip-iceberg-content` (full sync) + +Drop `--skip-iceberg-content` from the same command and re-run it. Since the +storage volume is shared, `test_table`'s metadata is now readable by target +too, so namespace/table sync succeeds this time — target ends up with the +namespace and table that were skipped in Step 3a. + +### Verify on target + +```bash +TARGET_TOKEN=$(curl -sf -X POST http://localhost:8183/api/catalog/v1/oauth/tokens \ + -d "grant_type=client_credentials&client_id=root&client_secret=s3cr3t&scope=PRINCIPAL_ROLE:ALL" \ + | jq -r '.access_token') + +curl -sf http://localhost:8183/api/management/v1/catalogs/test-catalog/catalog-roles \ + -H "Authorization: Bearer $TARGET_TOKEN" | jq + +# After Step 3a (--skip-iceberg-content), this returns an empty list. +# After Step 3b (full sync), this returns test_ns. +curl -sf http://localhost:8183/api/catalog/v1/test-catalog/namespaces \ + -H "Authorization: Bearer $TARGET_TOKEN" | jq +``` + +Tear down with: + +```bash +docker compose down -v +``` diff --git a/polaris-synchronizer/manual-test/docker-compose.yml b/polaris-synchronizer/manual-test/docker-compose.yml new file mode 100644 index 00000000..fef1b9fe --- /dev/null +++ b/polaris-synchronizer/manual-test/docker-compose.yml @@ -0,0 +1,88 @@ +services: + + # ── Polaris instances ──────────────────────────────────────────────────────── + # Official Apache Polaris image. No Postgres/RDS — Polaris defaults to its + # in-memory metastore when no datasource is configured. Both instances share + # a single FILE storage volume mounted at the same path, so metadata written + # by source is readable by target — this lets a full sync (including + # namespace/table content) actually succeed, so you can compare it against + # a `--skip-iceberg-content` run in this example. + + polaris-source: + image: apache/polaris:1.1.0-incubating + container_name: polaris-source + hostname: polaris-source + ports: + - "8181:8181" + - "8182:8182" + environment: + POLARIS_BOOTSTRAP_CREDENTIALS: POLARIS,root,s3cr3t + JAVA_OPTS_APPEND: >- + -Dpolaris.readiness.ignore-severe-issues=true + -Dpolaris.features."SUPPORTED_CATALOG_STORAGE_TYPES"=["FILE"] + -Dpolaris.features."ALLOW_INSECURE_STORAGE_TYPES"=true + -Dpolaris.features."ALLOW_UNSTRUCTURED_TABLE_LOCATION"=true + healthcheck: + test: ["CMD", "curl", "-sf", "http://localhost:8182/q/health/ready"] + interval: 5s + timeout: 5s + retries: 20 + start_period: 20s + volumes: + - shared-storage:/tmp/polaris + networks: + - polaris-net + + polaris-target: + image: apache/polaris:1.1.0-incubating + container_name: polaris-target + hostname: polaris-target + ports: + - "8183:8181" + - "8184:8182" + environment: + POLARIS_BOOTSTRAP_CREDENTIALS: POLARIS,root,s3cr3t + JAVA_OPTS_APPEND: >- + -Dpolaris.readiness.ignore-severe-issues=true + -Dpolaris.features."SUPPORTED_CATALOG_STORAGE_TYPES"=["FILE"] + -Dpolaris.features."ALLOW_INSECURE_STORAGE_TYPES"=true + -Dpolaris.features."ALLOW_UNSTRUCTURED_TABLE_LOCATION"=true + healthcheck: + test: ["CMD", "curl", "-sf", "http://localhost:8182/q/health/ready"] + interval: 5s + timeout: 5s + retries: 20 + start_period: 20s + volumes: + - shared-storage:/tmp/polaris + networks: + - polaris-net + + # ── Init container ──────────────────────────────────────────────────────── + # Seeds SOURCE with: catalog + principal + principal-role + catalog-role + + # grant + namespace + table. TARGET is left completely empty — polaris-synchronizer + # is responsible for creating the catalog/catalog-role/grants (and, unless + # --skip-iceberg-content is used, the namespace/table) on the target. + + polaris-init: + build: + context: ./init + container_name: polaris-init + dns_search: "." + depends_on: + polaris-source: + condition: service_healthy + polaris-target: + condition: service_healthy + volumes: + - ./seed:/seed + restart: "no" + networks: + - polaris-net + +networks: + polaris-net: + driver: bridge + +volumes: + shared-storage: diff --git a/polaris-synchronizer/manual-test/init/Dockerfile b/polaris-synchronizer/manual-test/init/Dockerfile new file mode 100644 index 00000000..ebd4cdf0 --- /dev/null +++ b/polaris-synchronizer/manual-test/init/Dockerfile @@ -0,0 +1,5 @@ +FROM alpine:3.19 +RUN apk add --no-cache curl jq bash +COPY init.sh /init.sh +RUN chmod +x /init.sh +ENTRYPOINT ["/bin/bash", "/init.sh"] diff --git a/polaris-synchronizer/manual-test/init/init.sh b/polaris-synchronizer/manual-test/init/init.sh new file mode 100644 index 00000000..ae091289 --- /dev/null +++ b/polaris-synchronizer/manual-test/init/init.sh @@ -0,0 +1,162 @@ +#!/bin/bash +# Seeds SOURCE with: catalog + principal + principal-role + catalog-role + +# grant + namespace + table. TARGET is left completely empty so that +# polaris-synchronizer creates entities on target itself. + +set -euo pipefail + +SOURCE="http://polaris-source:8181" +TARGET="http://polaris-target:8181" +CATALOG="test-catalog" +ROOT_CLIENT_ID="root" +ROOT_SECRET="s3cr3t" + +wait_ready() { + local base=$1 + echo "[init] Waiting for $base..." + until curl -sf "$base/api/catalog/v1/oauth/tokens" \ + -H "Content-Type: application/x-www-form-urlencoded" \ + -d "grant_type=client_credentials&client_id=${ROOT_CLIENT_ID}&client_secret=${ROOT_SECRET}&scope=PRINCIPAL_ROLE:ALL" \ + >/dev/null 2>&1; do sleep 3; done + echo "[init] $base is ready" +} + +get_token() { + local base=$1 + curl -sf -X POST "$base/api/catalog/v1/oauth/tokens" \ + -H "Content-Type: application/x-www-form-urlencoded" \ + -d "grant_type=client_credentials&client_id=${ROOT_CLIENT_ID}&client_secret=${ROOT_SECRET}&scope=PRINCIPAL_ROLE:ALL" \ + | jq -r '.access_token' +} + +mgmt_post() { + local base=$1 token=$2 path=$3 body=$4 + curl -sf -X POST "$base/api/management/v1$path" \ + -H "Authorization: Bearer $token" \ + -H "Content-Type: application/json" \ + -d "$body" +} + +mgmt_put() { + local base=$1 token=$2 path=$3 body=$4 + curl -sf -X PUT "$base/api/management/v1$path" \ + -H "Authorization: Bearer $token" \ + -H "Content-Type: application/json" \ + -d "$body" +} + +catalog_post() { + local base=$1 token=$2 path=$3 body=$4 + curl -sf -X POST "$base/api/catalog/v1$path" \ + -H "Authorization: Bearer $token" \ + -H "Content-Type: application/json" \ + -d "$body" +} + +wait_ready "$SOURCE" +wait_ready "$TARGET" + +SRC_TOKEN=$(get_token "$SOURCE") +echo "[init] SOURCE admin token acquired" + +# ── SOURCE: catalog (FILE storage, local to the polaris-source container) ──── + +mgmt_post "$SOURCE" "$SRC_TOKEN" "/catalogs" '{ + "catalog": { + "name": "'"$CATALOG"'", + "type": "INTERNAL", + "properties": {"default-base-location": "file:///tmp/polaris/'"$CATALOG"'/"}, + "storageConfigInfo": { + "storageType": "FILE", + "allowedLocations": ["file:///tmp/polaris/'"$CATALOG"'"] + } + } +}' +echo "[init] SOURCE: catalog created" + +# ── SOURCE: principal ───────────────────────────────────────────────────────── + +PRINCIPAL_RESP=$(mgmt_post "$SOURCE" "$SRC_TOKEN" "/principals" '{ + "principal": {"name": "test-user", "type": "user"}, + "credentialRotationRequired": false +}') + +USER_CLIENT_ID=$(echo "$PRINCIPAL_RESP" | jq -r '.credentials.clientId') +USER_SECRET=$(echo "$PRINCIPAL_RESP" | jq -r '.credentials.clientSecret') +echo "[init] SOURCE: principal test-user created (clientId=$USER_CLIENT_ID)" + +# ── SOURCE: principal role ──────────────────────────────────────────────────── + +mgmt_post "$SOURCE" "$SRC_TOKEN" "/principal-roles" '{"principalRole": {"name": "analyst-role"}}' +echo "[init] SOURCE: principal role analyst-role created" + +# ── SOURCE: assign principal role → principal ───────────────────────────────── + +mgmt_put "$SOURCE" "$SRC_TOKEN" "/principals/test-user/principal-roles" \ + '{"principalRole": {"name": "analyst-role"}}' +echo "[init] SOURCE: analyst-role assigned to test-user" + +# ── SOURCE: catalog role ────────────────────────────────────────────────────── + +mgmt_post "$SOURCE" "$SRC_TOKEN" "/catalogs/$CATALOG/catalog-roles" \ + '{"catalogRole": {"name": "catalog-analyst"}}' +echo "[init] SOURCE: catalog role catalog-analyst created" + +# ── SOURCE: assign catalog role → principal role ────────────────────────────── + +mgmt_put "$SOURCE" "$SRC_TOKEN" "/principal-roles/analyst-role/catalog-roles/$CATALOG" \ + '{"catalogRole": {"name": "catalog-analyst"}}' +echo "[init] SOURCE: catalog-analyst assigned to analyst-role" + +# ── SOURCE: grant CATALOG_MANAGE_CONTENT to catalog role ───────────────────── + +mgmt_put "$SOURCE" "$SRC_TOKEN" "/catalogs/$CATALOG/catalog-roles/catalog-analyst/grants" \ + '{"grant": {"type": "catalog", "privilege": "CATALOG_MANAGE_CONTENT"}}' +echo "[init] SOURCE: CATALOG_MANAGE_CONTENT granted to catalog-analyst" + +# ── SOURCE: namespace + table (as test-user) ────────────────────────────────── + +USER_TOKEN=$(curl -sf -X POST "$SOURCE/api/catalog/v1/oauth/tokens" \ + -H "Content-Type: application/x-www-form-urlencoded" \ + -d "grant_type=client_credentials&client_id=${USER_CLIENT_ID}&client_secret=${USER_SECRET}&scope=PRINCIPAL_ROLE:ALL" \ + | jq -r '.access_token') + +catalog_post "$SOURCE" "$USER_TOKEN" "/$CATALOG/namespaces" \ + '{"namespace": ["test_ns"], "properties": {}}' +echo "[init] SOURCE: namespace test_ns created" + +catalog_post "$SOURCE" "$USER_TOKEN" "/$CATALOG/namespaces/test_ns/tables" '{ + "name": "test_table", + "location": "file:///tmp/polaris/'"$CATALOG"'/test_ns/test_table", + "schema": { + "type": "struct", + "schema-id": 0, + "fields": [ + {"id": 1, "name": "id", "required": true, "type": "long"}, + {"id": 2, "name": "data", "required": false, "type": "string"} + ] + } +}' +echo "[init] SOURCE: table test_table created" + +# ── Persist seeded credentials for the CLI steps ────────────────────────────── + +mkdir -p /seed +cat > /seed/credentials.json <