From 263bc9738b8d2d9c1def12d85cb672ae0ba14f35 Mon Sep 17 00:00:00 2001 From: Ilanit Stein Date: Sun, 31 May 2026 19:31:11 +0300 Subject: [PATCH 1/5] docs: restructure documentation with updated command references MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Reorganize documentation into docs/ with commands/ and development/ subdirectories. Add CONTRIBUTING.md for contributor onboarding. New structure: - docs/commands/ — reference for all 5 commands (export, transform, apply, validate, transfer-pvc) - docs/development/ — architecture, setup, testing, plugin development - docs/multistage-pipeline.md — multi-stage Kustomize pipeline guide - docs/plugins.md — plugin overview - docs/resource-compatibility.md — resource type compatibility matrix - docs/installation.md — installation and prerequisites Content updated to reflect current codebase: - crane apply uses embedded kustomize (no kubectl dependency) - crane apply supports --skip-cluster-scoped and --kustomize-args flags - crane validate supports offline mode via --api-resources - crane export documents CRD collection and _cluster/ subdirectory Co-Authored-By: Claude Opus 4.6 --- CONTRIBUTING.md | 109 +++++ README.md | 10 +- docs/README.md | 30 ++ docs/commands/apply.md | 106 +++++ docs/commands/export.md | 113 +++++ docs/commands/transfer-pvc.md | 84 ++++ docs/commands/transform.md | 569 +++++++++++++++++++++++++ docs/commands/validate.md | 108 +++++ docs/development/README.md | 57 +++ docs/development/architecture.md | 139 ++++++ docs/development/plugin-development.md | 187 ++++++++ docs/development/setup.md | 128 ++++++ docs/development/testing.md | 121 ++++++ docs/installation.md | 54 +++ docs/multistage-pipeline.md | 523 +++++++++++++++++++++++ docs/plugins.md | 82 ++++ docs/resource-compatibility.md | 75 ++++ 17 files changed, 2494 insertions(+), 1 deletion(-) create mode 100644 CONTRIBUTING.md create mode 100644 docs/README.md create mode 100644 docs/commands/apply.md create mode 100644 docs/commands/export.md create mode 100644 docs/commands/transfer-pvc.md create mode 100644 docs/commands/transform.md create mode 100644 docs/commands/validate.md create mode 100644 docs/development/README.md create mode 100644 docs/development/architecture.md create mode 100644 docs/development/plugin-development.md create mode 100644 docs/development/setup.md create mode 100644 docs/development/testing.md create mode 100644 docs/installation.md create mode 100644 docs/multistage-pipeline.md create mode 100644 docs/plugins.md create mode 100644 docs/resource-compatibility.md diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 00000000..8ff7b101 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,109 @@ +# Contributing to Crane + +Thank you for your interest in contributing to Crane! This guide will help you get started. + +## Quick Start + +1. Fork and clone the repository: + ```bash + git clone https://github.com//crane.git + cd crane + ``` + +2. Build the binary: + ```bash + go build -o crane main.go + ``` + +3. Run tests: + ```bash + go test ./... + ``` + +4. Submit a pull request + +## Prerequisites + +- **Go** 1.25+ (see `go.mod` for exact version) +- **kubectl** (for deploying output to clusters and for `crane validate` with live cluster mode) +- **Docker** (optional, for container-based testing) +- Access to a Kubernetes cluster (for E2E testing) + +## Development Setup + +For detailed guides on architecture, testing, and advanced development workflows, see the [Development Documentation](docs/development/README.md). + +## Code Standards + +### Go Conventions + +- Follow standard Go idioms and [Effective Go](https://go.dev/doc/effective_go) practices +- Use `gofmt` for formatting +- Prefer explicit error messages with context — include the actual type received, the API resource being processed, and enough context to debug without re-running + +### Error Handling + +Error messages must be actionable: + +```go +// Good — shows actual object type +fmt.Errorf("expected *unstructured.Unstructured but got %T", object) + +// Bad — shows nil pointer +fmt.Errorf("expected *unstructured.Unstructured but got %T", u) +``` + +### Working with Kubernetes API Objects + +- Use `*unstructured.Unstructured` for dynamic resources +- Always check type assertions and provide informative error messages +- Include the actual resource type and API resource name in errors + +## Commit Messages + +Use conventional commits format: + +- `fix:` for bug fixes +- `feat:` for new features +- `refactor:` for code restructuring +- `test:` for test additions +- `docs:` for documentation + +Include issue references where applicable: `fix: improve error messages (#197)` + +## Pull Request Guidelines + +**Title:** Clear and concise (under 70 characters) + +**Body must include:** +- **Summary** — what changed and why +- **Impact** — severity, affected components +- **Test plan** — how to verify + +**Before submitting:** + +- [ ] Code compiles: `go build ./...` +- [ ] Tests pass: `go test ./...` +- [ ] Error messages are informative +- [ ] No unnecessary refactoring in the same PR +- [ ] Backward compatible (or documented breaking change) + +## Testing + +- All new features require tests +- Bug fixes should include regression tests when possible +- Use table-driven tests for multiple scenarios +- Test files: `*_test.go` in the same package +- E2E tests: `e2e-tests/` + +See [Testing Guide](docs/development/testing.md) for detailed testing documentation. + +## Reporting Issues + +- Check existing issues before filing a new one +- Include reproduction steps, expected behavior, and actual behavior +- Include Crane version, Go version, and Kubernetes version + +## Code of Conduct + +Be respectful and constructive. We follow the [Konveyor community guidelines](https://www.konveyor.io/). diff --git a/README.md b/README.md index a616d47e..6a249d96 100644 --- a/README.md +++ b/README.md @@ -99,7 +99,7 @@ How does it work? Crane works by: * `$ cat transform/10_KubernetesPlugin/input/secret.yaml` - Contains the original exported Secret with all metadata 5. `$ crane apply` - * Runs `kubectl kustomize` on the transform stage to apply patches + * Runs embedded kustomize on the transform stage to apply patches * Outputs clean, declarative YAML to `output/output.yaml` * Example: * `$ cat output/output.yaml` @@ -117,6 +117,14 @@ How does it work? Crane works by: 6. The content in `output/output.yaml` is now ready to be deployed to the target cluster or checked into Git for GitOps workflows: * `$ kubectl apply -f output/output.yaml` +## Documentation + +For comprehensive documentation, see the [docs/](docs/README.md) directory: + +- [Installation](docs/installation.md) +- [Command Reference](docs/README.md#command-reference) — export, transform, apply, validate, transfer-pvc +- [Contributing](CONTRIBUTING.md) | [Development Guide](docs/development/README.md) + ## Further Examples Please see [konveyor/crane-runner/main/examples](https://github.com/konveyor/crane-runner/tree/main/examples#readme) for further scenarios to explore what can be done with Crane + Tekton for migrating applications. diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 00000000..8ecd7aa1 --- /dev/null +++ b/docs/README.md @@ -0,0 +1,30 @@ +# Crane Documentation + +Welcome to the Crane documentation. Crane is a Kubernetes migration tool that helps migrate workloads between clusters using a non-destructive pipeline: **export → transform → apply → validate**. + +## Getting Started + +- [Installation](./installation.md) — Prerequisites, building, and verifying the Crane binary + +## Command Reference + +- [`crane export`](./commands/export.md) — Export namespace resources from a source cluster +- [`crane transform`](./commands/transform.md) — Transform exported resources using plugins and Kustomize +- [`crane apply`](./commands/apply.md) — Apply transformations and produce final manifests +- [`crane validate`](./commands/validate.md) — Validate final manifests against a target cluster +- [`crane transfer-pvc`](./commands/transfer-pvc.md) — Transfer PVC data between clusters via rsync + +## Concepts + +- [Multi-Stage Pipeline](./multistage-pipeline.md) — How Crane's multi-stage Kustomize transform pipeline works +- [Plugins](./plugins.md) — Built-in and custom plugin overview + +## Reference + +- [Pre-Apply Validation Guide](./pre-apply-validation-guide.md) — Checklist for validating manifests before applying +- [Resource Compatibility](./resource-compatibility.md) — Supported resource types and migration boundaries + +## Development + +- [Development Guide](./development/README.md) — Architecture, setup, testing, and plugin development +- [Contributing](../CONTRIBUTING.md) — How to contribute to Crane diff --git a/docs/commands/apply.md b/docs/commands/apply.md new file mode 100644 index 00000000..7110ffec --- /dev/null +++ b/docs/commands/apply.md @@ -0,0 +1,106 @@ +# crane apply + +Apply transformations to exported resources and produce final manifests. + +## Synopsis + +```bash +crane apply [flags] +``` + +## Description + +`crane apply` runs embedded kustomize on each transform stage to apply patches, producing clean, declarative YAML ready for deployment to a target cluster. By default, all stages are applied sequentially to maintain consistency across the pipeline. The output includes both a single multi-document YAML file and individual resource files organized by namespace. + +Kustomize is embedded directly in the Crane binary (via the krusty API), so no external `kubectl` dependency is needed. + +## Flags + +| Flag | Short | Default | Description | +|------|-------|---------|-------------| +| `--export-dir` | `-e` | `export` | The path where exported resources are saved (kept for consistency; not used by apply) | +| `--transform-dir` | `-t` | `transform` | The path where transform stage directories are located | +| `--output-dir` | `-o` | `output` | The path where final manifests are written | +| `--stage` | | | Apply a specific stage only (e.g., `10_KubernetesPlugin`). If not specified, all stages are applied | +| `--kustomize-args` | | | Additional arguments for kustomize (e.g., `--enable-helm --helm-command=helm3`) | +| `--skip-cluster-scoped` | | `false` | Exclude cluster-scoped resources (ClusterRole, ClusterRoleBinding, CRD, etc.) from output. Useful for non-admin migration scenarios | + +## Output Structure + +```text +output/ +├── output.yaml # All resources in a single multi-document YAML +└── resources/ # Individual resource files + ├── / + │ ├── Deployment_apps_v1__.yaml + │ ├── Service__v1__.yaml + │ └── ConfigMap__v1__.yaml + └── _cluster/ # Cluster-scoped resources (when not skipped) + ├── ClusterRoleBinding_rbac.authorization.k8s.io_v1_clusterscoped_.yaml + └── ClusterRole_rbac.authorization.k8s.io_v1_clusterscoped_.yaml +``` + +- **`output.yaml`** — Ready for `kubectl apply -f` +- **`resources//`** — Individual files for selective review or application +- **`resources/_cluster/`** — Cluster-scoped resources (omitted when `--skip-cluster-scoped` is set) + +## Examples + +### Apply all stages (default) + +```bash +crane apply +``` + +### Apply with custom directories + +```bash +crane apply --transform-dir ./migration/transform --output-dir ./migration/output +``` + +### Apply a specific stage only + +```bash +crane apply --stage 10_KubernetesPlugin +``` + +### Skip cluster-scoped resources + +```bash +crane apply --skip-cluster-scoped +``` + +### Pass additional kustomize arguments + +```bash +crane apply --kustomize-args "--enable-helm --helm-command=helm3" +``` + +### Deploy to target cluster + +```bash +crane apply +kubectl apply -f output/output.yaml +``` + +## Common Errors + +| Error | Cause | Solution | +|-------|-------|----------| +| `kustomization.yaml validation failed` | Invalid Kustomize syntax or missing resource files | Run `crane apply --stage ` to isolate the failing stage | +| `invalid stage name` | Stage name doesn't follow `_` format | Use a valid stage name like `10_KubernetesPlugin` | +| `invalid kustomize-args` | Unsupported or malformed kustomize arguments | Check supported kustomize flags | + +## Next Steps + +After applying, validate the manifests against your target cluster: + +```bash +crane validate --input-dir output --context target-cluster +``` + +See [crane validate](./validate.md) for details. Or deploy directly: + +```bash +kubectl apply -f output/output.yaml +``` diff --git a/docs/commands/export.md b/docs/commands/export.md new file mode 100644 index 00000000..690c93b9 --- /dev/null +++ b/docs/commands/export.md @@ -0,0 +1,113 @@ +# crane export + +Export namespace resources from a source Kubernetes cluster to disk. + +## Synopsis + +```bash +crane export [flags] +``` + +## Description + +`crane export` discovers all API types in a Kubernetes cluster, lists objects in the specified namespace (plus related cluster-scoped RBAC resources), and writes manifests to an export directory. This is the first step in the Crane migration pipeline. + +Exported resources are written as individual YAML files under `export/resources//`. Cluster-scoped resources related to the namespace (ClusterRoleBindings, ClusterRoles, SCCs) are written to `export/resources//_cluster/`. Any errors encountered during listing are recorded in `export/failures//`. + +### CRD Collection + +When custom resources are found in the namespace, Crane automatically collects their corresponding CustomResourceDefinitions. Operator-managed CRDs (identified via owner references) are skipped, since they should be installed by the operator on the target cluster rather than migrated directly. If the migration user lacks permission to read CRDs, Crane logs a warning and continues — the assumption is that the CRDs already exist on the target. + +## Flags + +| Flag | Short | Default | Description | +|------|-------|---------|-------------| +| `--export-dir` | `-e` | `export` | The path where files are exported | +| `--label-selector` | `-l` | | Restrict export to resources matching a label selector | +| `--namespace` | `-n` | _(context default)_ | Namespace to export | +| `--crd-skip-group` | | | API groups to skip for CRD export (repeatable) | +| `--crd-include-group` | | | API groups to force-include for CRD export (repeatable) | +| `--as-extras` | | | Extra impersonation info (format: `key=val1,val2;key2=val3`) | +| `--qps` | `-q` | `100` | Query-per-second rate for API requests | +| `--burst` | `-b` | `1000` | API burst rate | + +Standard kubeconfig flags (`--kubeconfig`, `--context`, `--cluster`, `--as`, `--as-group`, etc.) are also available. + +## Output Structure + +```text +export/ +├── resources/ +│ └── / +│ ├── Deployment_apps_v1__.yaml +│ ├── Service__v1__.yaml +│ ├── ConfigMap__v1__.yaml +│ └── _cluster/ +│ ├── ClusterRoleBinding_rbac.authorization.k8s.io_v1_clusterscoped_.yaml +│ ├── ClusterRole_rbac.authorization.k8s.io_v1_clusterscoped_.yaml +│ └── CustomResourceDefinition_apiextensions.k8s.io_v1_clusterscoped_.yaml +└── failures/ + └── / + └── +``` + +Resource filenames follow the format: `Kind_group_version_namespace_name.yaml` + +## Examples + +### Basic namespace export + +```bash +crane export -n my-app +``` + +### Export with label selector + +```bash +crane export -n my-app --label-selector "app=frontend" +``` + +### Export with custom directory + +```bash +crane export -n my-app --export-dir ./migration/export +``` + +### Export with impersonation + +```bash +crane export -n my-app --as system:serviceaccount:my-app:deployer +``` + +### Export with extra impersonation info + +```bash +crane export -n my-app \ + --as user@example.com \ + --as-extras "scope=read,write;project=my-project" +``` + +### Export skipping specific CRD groups + +```bash +crane export -n my-app --crd-skip-group monitoring.coreos.com +``` + +## Common Errors + +| Error | Cause | Solution | +|-------|-------|----------| +| `namespace must be set` | No namespace specified and none in kubeconfig context | Use `-n ` | +| `namespaces "X" not found` | Namespace does not exist | Verify namespace name | +| `cannot verify namespace exists` | Insufficient RBAC (warning only) | Export proceeds; verify namespace exists manually | +| `extras requires specifying a user or group` | `--as-extras` used without `--as` | Add `--as` or `--as-group` flag | + +## Next Steps + +After exporting, transform the resources: + +```bash +crane transform --export-dir export +``` + +See [crane transform](./transform.md) for details. diff --git a/docs/commands/transfer-pvc.md b/docs/commands/transfer-pvc.md new file mode 100644 index 00000000..cb2b996b --- /dev/null +++ b/docs/commands/transfer-pvc.md @@ -0,0 +1,84 @@ +# crane transfer-pvc + +Transfer PersistentVolumeClaim resources and volume data between clusters. + +## Synopsis + +```bash +crane transfer-pvc [flags] +``` + +## Description + +The `transfer-pvc` subcommand transfers a PersistentVolumeClaim resource and its volume data to a destination cluster. It establishes a connection to the destination cluster by creating a public endpoint of the user's choice in the destination namespace. It then creates a PVC and an rsync daemon Pod in the destination namespace to receive data from the source PVC. Finally, it creates an rsync client Pod in the source namespace which transfers data to the rsync daemon using the endpoint. The connection is encrypted using self-signed certificates created automatically at the time of transfer. + +## Example + +```bash +crane transfer-pvc --source-context= --destination-context= --pvc-name= --endpoint=route +``` + +The above command transfers PVC (along with PV data) named `` in the namespace specified by `` context into the namespace specified by `` context. The `--endpoint` argument specifies the kind of public endpoint to use to establish a connection between the source and the destination cluster. + +## Flags + +| Flag | Type | Required | Description | +|------|------|----------|-------------| +| `--source-context` | string | Yes | Kube context of the source cluster | +| `--destination-context` | string | Yes | Kube context of the destination cluster | +| `--pvc-name` | string | Yes | Mapping of source/destination PVC names (see [PVC Options](#pvc-options)) | +| `--pvc-namespace` | string | No | Mapping of source/destination PVC namespaces (see [PVC Options](#pvc-options)) | +| `--dest-storage-class` | string | No | Storage class of destination PVC (defaults to source storage class) | +| `--dest-storage-requests` | string | No | Requested storage capacity of destination PVC (defaults to source capacity) | +| `--destination-image` | string | No | Custom image to use for destination rsync Pod | +| `--source-image` | string | No | Custom image to use for source rsync Pod | +| `--endpoint` | string | No | Kind of endpoint to create in destination cluster (see [Endpoint Options](#endpoint-options)) | +| `--ingress-class` | string | No | Ingress class when endpoint is nginx-ingress | +| `--subdomain` | string | No | Custom subdomain to use for the endpoint | +| `--output` | string | No | Output transfer stats in the specified file | +| `--verify` | bool | No | Verify transferred files using checksums | + +### PVC Options + +`--pvc-name` allows specifying a mapping of source and destination PVC names. This is a required option. + +`--pvc-namespace=` allows specifying a mapping of namespaces of source and destination PVC. By default, the namespaces in the source and destination contexts are used. When this option is specified, the namespaces in kube contexts are ignored and specified namespaces are used. + +Both `--pvc-name` and `--pvc-namespace` follow mapping format `:`, where `` specifies the name in the source cluster while `` is the name in the destination cluster. If only `` is specified, the same names are used in the destination cluster. + +#### Examples + +Transfer a PVC `test-pvc` in namespace `test-ns` to a destination PVC by the same name and namespace: + +```bash +crane transfer-pvc --pvc-name=test-pvc --pvc-namespace=test-ns \ + --source-context=source --destination-context=destination --endpoint=route +``` + +Transfer a PVC `source-pvc` in namespace `source-ns` to a destination PVC `destination-pvc` in namespace `destination-ns`: + +```bash +crane transfer-pvc --pvc-name=source-pvc:destination-pvc \ + --pvc-namespace=source-ns:destination-ns \ + --source-context=source --destination-context=destination --endpoint=route +``` + +### Endpoint Options + +Endpoint enables a connection between the source and destination cluster for data transfer. It is created in the destination cluster. The destination cluster must support the kind of endpoint used. + +By default, `nginx-ingress` is used as endpoint. For nginx-ingress, `--subdomain` and `--ingress-class` are required. + +In an OpenShift cluster, `route` endpoint can be used. A subdomain option can be specified but is not required. By default, the cluster's subdomain will be used. + +## Next Steps + +After transferring PVC data, you may want to export, transform, and apply the remaining namespace resources: + +```bash +crane export -n +crane transform +crane apply +``` + +See [crane export](./export.md) for details. diff --git a/docs/commands/transform.md b/docs/commands/transform.md new file mode 100644 index 00000000..a2f86b2f --- /dev/null +++ b/docs/commands/transform.md @@ -0,0 +1,569 @@ +# Crane Transform Directory Structure + +This document explains the structure of the `transform/` directory created by Crane's multi-stage Kustomize pipeline. + +## Quick Start + +After running `crane transform`, you'll see: + +```text +transform/ +└── 10_KubernetesPlugin/ + ├── resources/ + │ ├── ConfigMap__v1_default_nginx-config.yaml + │ ├── Deployment_apps_v1_default_wordpress.yaml + │ └── Service__v1_default_kubernetes.yaml + ├── patches/ + │ └── default--apps-v1--Deployment--wordpress.patch.yaml + └── kustomization.yaml +``` + +### What's in Each Directory? + +- **`resources/`**: Individual Kubernetes manifest files, one per resource +- **`patches/`**: Kustomize patches to apply to resources +- **`kustomization.yaml`**: Kustomize configuration file + +## Working with Transform Output + +### Viewing Final Resources + +To see what will be deployed: + +```bash +# Using crane apply (recommended — uses embedded kustomize) +crane apply --transform-dir transform --output-dir output + +# Or preview with kubectl kustomize (if kubectl is installed) +kubectl kustomize transform/10_KubernetesPlugin/ +``` + +### Applying to Cluster + +```bash +# Option 1: Use crane apply (no kubectl dependency for this step) +crane apply --transform-dir transform --output-dir output +kubectl apply -f output/output.yaml + +# Option 2: Direct apply with kubectl +kubectl apply -k transform/10_KubernetesPlugin/ +``` + +### Making Manual Changes + +You can edit resources in the `resources/` directory: + +```bash +# Edit a deployment +vi transform/10_KubernetesPlugin/resources/Deployment_apps_v1_default_wordpress.yaml + +# Preview changes +crane apply --stage 10_KubernetesPlugin + +# Apply changes +kubectl apply -f output/output.yaml +``` + +**Important**: +- **Plugin stages** (ending with `Plugin`): Automatically regenerate on rerun - your manual edits will be overwritten +- **Custom stages** (not ending with `Plugin`): Refuse to overwrite without `--force` flag to protect your edits + +## Multi-Stage Pipelines + +For complex transformations, you can create multiple stages: + +```text +transform/ +├── 10_KubernetesPlugin/ # Base Kubernetes transformations +├── 20_OpenshiftPlugin/ # OpenShift-specific changes +└── 30_ImagestreamPlugin/ # ImageStream configurations +``` + +**Sequential Consistency**: Each stage runs on the **fully applied output** of the previous stage. This means: + +- Stage 1 reads from the export directory +- Stage 1's patches are applied, producing materialized output +- Stage 2 reads Stage 1's applied output (not the raw patches) +- Stage 2's patches are applied to the already-transformed resources +- And so on... + +This ensures that: +- Structural changes from earlier stages are visible to later stages +- Resources marked as whiteout (deleted) don't appear in subsequent stages +- Each stage sees the actual state of resources, not just patch instructions + +### Working Directory Structure + +When running multistage transforms, Crane creates a working directory structure for debugging: + +```text +transform/ +├── 10_KubernetesPlugin/ # Stage 1 transform artifacts +│ ├── resources/ +│ ├── patches/ +│ └── kustomization.yaml +├── 20_OpenshiftPlugin/ # Stage 2 transform artifacts +│ ├── resources/ +│ ├── patches/ +│ └── kustomization.yaml +└── .work/ # Intermediate working artifacts + ├── 10_KubernetesPlugin/ + │ ├── input/ # Stage 1 input snapshot (from export) + │ └── output/ # Stage 1 materialized output + └── 20_OpenshiftPlugin/ + ├── input/ # Stage 2 input snapshot (Stage 1 output) + └── output/ # Stage 2 materialized output +``` + +The `.work/` directory contains intermediate snapshots useful for debugging multi-stage pipelines. + +**Important**: The `.work/` directory contains intermediate snapshots that are regenerated on each transform run. These are useful for debugging but should not be committed to version control. Add to your `.gitignore`: + +```gitignore +# Crane intermediate artifacts (regenerated on each run) +transform/.work/ +``` + +### Running Multi-Stage Transforms + +```bash +# Default: discover and run all existing stages +# If no stages exist, creates default 10_KubernetesPlugin stage +crane transform + +# Run specific stage only (creates it if doesn't exist) +crane transform --stage 10_KubernetesPlugin + +# Create a new plugin-based stage automatically +# Stage name ending with "Plugin" will use that plugin +crane transform --stage 35_OpenshiftPlugin + +# Create a new pass-through stage for manual editing +# Stage name NOT ending with "Plugin" creates empty pass-through +crane transform --stage 40_CustomManualEdits +``` + +### Applying Transforms + +```bash +# Apply all stages sequentially +crane apply --transform-dir transform --output-dir output +``` + +**Note**: The default behavior applies **all stages** sequentially to ensure each stage output is properly materialized. This maintains sequential consistency across the entire pipeline. + +#### Output Structure + +`crane apply` creates the following output structure: + +```text +output/ +├── output.yaml # Single file with all resources +└── resources/ # Individual resource files + ├── default/ # Organized by namespace + │ ├── Deployment_apps_v1_default_myapp.yaml + │ ├── Service__v1_default_myapp.yaml + │ └── ConfigMap__v1_default_config.yaml + ├── kube-system/ + │ └── Service__v1_kube-system_metrics.yaml + └── _cluster/ # Cluster-scoped resources + └── ClusterRoleBinding_rbac.authorization.k8s.io_v1_clusterscoped_myapp.yaml +``` + +- **`output.yaml`**: All resources in a single multi-document YAML file (ready for `kubectl apply -f`) +- **`resources//`**: Individual resource files organized by namespace for easier review and selective application +- **`resources/_cluster/`**: Cluster-scoped resources (omitted when `--skip-cluster-scoped` is set) + +## Automatic Stage Creation + +Crane automatically creates new stages when you reference them with `--stage`. The stage name determines whether a plugin will be used or if it's a pass-through stage for manual editing. + +### Stage Naming Convention + +Stage names follow the pattern `_`: + +**Plugin-based stages (name ends with "Plugin"):** +- `10_KubernetesPlugin` → uses plugin "KubernetesPlugin" +- `20_OpenshiftPlugin` → uses plugin "OpenshiftPlugin" +- `35_CustomPlugin` → uses plugin "CustomPlugin" + +**Pass-through stages (name does NOT end with "Plugin"):** +- `30_CustomEdits` → no plugin, resources pass through unchanged +- `40_ManualChanges` → no plugin, ready for manual editing +- `50_Tweaks` → no plugin, resources unchanged + +### Plugin-Based Stages + +Stage names ending with `Plugin` **must** have a corresponding plugin installed. Crane extracts the plugin name and runs it: + +```bash +# Creates a stage using OpenshiftPlugin +crane transform --stage 20_OpenshiftPlugin + +# Output: +# - resources/ (from previous stage or export) +# - patches/ (generated by OpenshiftPlugin) +# - kustomization.yaml +``` + +**If the plugin doesn't exist, you'll get an error:** + +```bash +crane transform --stage 20_NonexistentPlugin +# Error: stage 20_NonexistentPlugin requires plugin 'NonexistentPlugin' +# but it was not found (available plugins: KubernetesPlugin, NamespaceCleanup) +``` + +### Pass-Through Stages + +Stage names **not** ending with `Plugin` create pass-through stages where resources are copied unchanged, ready for manual editing: + +```bash +# Creates a pass-through stage for manual editing +crane transform --stage 30_CustomEdits + +# Output: +# - resources/ (copied unchanged from previous stage) +# - patches/ (empty - ready for your custom patches) +# - kustomization.yaml +``` + +You can then manually add custom patches: + +```bash +# Add custom patch +cat > transform/30_CustomEdits/patches/add-labels.yaml <-------.patch.yaml` +- **Format**: JSON patch +- **Purpose**: Apply plugin transformations + +Example patch (`default--apps-v1--Deployment--wordpress.patch.yaml`): + +```yaml +- op: remove + path: /metadata/uid +- op: remove + path: /metadata/resourceVersion +- op: remove + path: /status +``` + +### kustomization.yaml + +Kustomize configuration that ties everything together: + +```yaml +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +patches: +- path: patches/default--apps-v1--Deployment--wordpress.patch.yaml + target: + group: apps + kind: Deployment + name: wordpress + namespace: default + version: v1 +resources: +- resources/ConfigMap__v1_default_nginx-config.yaml +- resources/Deployment_apps_v1_default_wordpress.yaml +- resources/Service__v1_default_kubernetes.yaml + +# Whiteout resources are written to resources/ for complete snapshot +# but excluded from active resources list above: +# - resources/Pod__v1_default_wordpress-74b89cc84c-nm9f8.yaml +``` + + +## Common Workflows + +### 1. Review Transformations + +```bash +# Use crane apply to render and review +crane apply +cat output/output.yaml + +# Or view specific resource type +crane apply +grep -A 20 "kind: Deployment" output/output.yaml + +# Or use kubectl kustomize directly (if installed) +kubectl kustomize transform/10_KubernetesPlugin/ +``` + +### 2. Customize After Transform + +**Note**: Plugin stages (ending with `Plugin`) will be automatically regenerated on next run, overwriting manual edits. For manual customizations, create a custom stage (not ending with `Plugin`). + +```bash +# Create a custom stage for manual edits +crane transform --stage 40_CustomEdits + +# Edit resources +vi transform/40_CustomEdits/resources/Deployment_apps_v1_default_wordpress.yaml + +# Add custom patches +cat > transform/40_CustomEdits/patches/custom-patch.yaml < transform/50_CustomLabels/patches/add-labels.yaml < +``` + +## Exit Codes + +| Code | Meaning | +|------|---------| +| `0` | All checks pass — all GVKs are available on the target cluster | +| `1` | One or more checks failed, or another error occurred | + +## Examples + +### Validate against current context (live mode) + +```bash +crane validate +``` + +### Validate against a specific target cluster + +```bash +crane validate --context target-cluster +``` + +### Validate with custom input directory + +```bash +crane validate --input-dir ./migration/output +``` + +### Generate YAML report + +```bash +crane validate --output yaml +``` + +### Offline validation against captured API surface + +```bash +crane validate --api-resources api-surface.json --input-dir output +``` + +### Full pipeline example + +```bash +crane export -n my-app +crane transform +crane apply +crane validate --context target-cluster +``` + +## Common Errors + +| Error | Cause | Solution | +|-------|-------|----------| +| `input-dir "X" is not a directory` | Path doesn't exist or isn't a directory | Run `crane apply` first to generate output | +| `loading kubeconfig` | Cannot connect to target cluster | Check kubeconfig and `--context` flag, or use `--api-resources` for offline mode | +| `--api-resources and --context are mutually exclusive` | Both offline and live flags specified | Use one mode or the other | +| Validation failures | GVKs not available on target cluster | Install required CRDs/operators on target, or transform manifests to use supported API versions | + +## Next Steps + +If validation passes, deploy to the target cluster: + +```bash +kubectl apply -f output/output.yaml +``` + +If validation fails, review the report and failures directory, then adjust your transform stages or target cluster configuration. diff --git a/docs/development/README.md b/docs/development/README.md new file mode 100644 index 00000000..cf11ced9 --- /dev/null +++ b/docs/development/README.md @@ -0,0 +1,57 @@ +# Crane Development Guide + +This directory contains detailed guides for developers contributing to Crane. + +## Guides + +- [Architecture](./architecture.md) — Pipeline design, package structure, and data flow +- [Development Setup](./setup.md) — Setting up your development environment +- [Testing](./testing.md) — Unit tests, table-driven patterns, and E2E testing +- [Plugin Development](./plugin-development.md) — Writing custom transformation plugins + +## Quick Reference + +```bash +# Build +go build -o crane main.go + +# Test +go test ./... + +# Lint +gofmt -l . +``` + +## Project Structure + +```text +crane/ +├── cmd/ # CLI command implementations +│ ├── apply/ # crane apply +│ ├── export/ # crane export +│ ├── transform/ # crane transform +│ ├── validate/ # crane validate +│ ├── transfer-pvc/ # crane transfer-pvc +│ └── ... +├── internal/ # Internal packages +│ ├── apply/ # Kustomize apply logic +│ ├── kustomize/ # Embedded kustomize runner (krusty API) +│ ├── transform/ # Orchestrator, stages, writer +│ ├── validate/ # Scanner, matcher, report +│ ├── file/ # File helper utilities +│ ├── flags/ # Global CLI flags +│ └── plugin/ # Plugin loading and execution +├── e2e-tests/ # End-to-end tests +│ ├── framework/ # Test framework utilities +│ ├── tests/ # Test cases +│ ├── golden-manifests/ # Expected output fixtures +│ └── utils/ # Test helper utilities +└── docs/ # Documentation +``` + +## Related Repositories + +- [konveyor/crane-lib](https://github.com/konveyor/crane-lib) — Transformation logic library +- [konveyor/crane-plugins](https://github.com/konveyor/crane-plugins) — Community plugins +- [konveyor/crane-plugin-openshift](https://github.com/konveyor/crane-plugin-openshift) — OpenShift plugin +- [backube/pvc-transfer](https://github.com/backube/pvc-transfer) — PV migration library diff --git a/docs/development/architecture.md b/docs/development/architecture.md new file mode 100644 index 00000000..58aa58b2 --- /dev/null +++ b/docs/development/architecture.md @@ -0,0 +1,139 @@ +# Crane Architecture + +Crane follows a pipeline architecture designed around Unix philosophy: small, focused tools assembled in powerful ways. All operations are non-destructive and output results to disk. + +## Pipeline Overview + +```text +Source Cluster → export → transform → apply → validate → Target Cluster + │ │ │ │ + ▼ ▼ ▼ ▼ + export/ transform/ output/ validate/ +``` + +Each stage reads from disk and writes to disk, making the pipeline transparent, auditable, and repeatable. + +## Export Phase + +**Package:** `cmd/export/` and `internal/` + +1. **Discovery** — Queries the Kubernetes API server for all available resource types using the discovery client +2. **Listing** — Lists all resources in the specified namespace, respecting label selectors and pagination +3. **RBAC filtering** — Identifies cluster-scoped resources (ClusterRoleBindings, ClusterRoles, SCCs) related to ServiceAccounts in the namespace +4. **CRD collection** — Exports CRD definitions for custom resources found in the namespace; operator-managed CRDs (identified via owner references) are skipped +5. **Writing** — Writes individual YAML files to `export/resources//` + +Key design decisions: +- Uses `dynamic.Interface` for listing any resource type without compile-time knowledge +- Failures are recorded to `export/failures/` rather than aborting the entire export +- Cluster-scoped resources go to a `_cluster/` subdirectory +- When CRD access is denied, a warning is logged and export continues (assumes CRDs exist on target) + +## Transform Phase + +**Package:** `cmd/transform/` and `internal/transform/` + +The transform phase uses a multi-stage Kustomize pipeline: + +1. **Stage discovery** — Scans `transform/` for existing stage directories matching `_` pattern +2. **Plugin execution** — For plugin-based stages (name ending in `Plugin`), loads and runs the matching plugin to generate JSONPatch operations +3. **Resource writing** — Writes individual resource files to `/resources/` +4. **Patch writing** — Writes plugin-generated patches to `/patches/` +5. **Kustomization generation** — Generates `kustomization.yaml` linking resources and patches + +### Stage Types + +- **Plugin stages** (`10_KubernetesPlugin`): Automatically regenerated on each run +- **Custom stages** (`30_CustomEdits`): Pass-through stages for manual editing, protected from overwrites without `--force` + +### Sequential Consistency + +In multi-stage pipelines, each stage runs on the fully materialized output of the previous stage (not raw patches). Intermediate artifacts are stored in `transform/.work//input/` and `transform/.work//output/`. + +### Key Components + +- **`Orchestrator`** (`internal/transform/orchestrator.go`): Coordinates multi-stage execution, manages stage discovery, plugin loading, and sequential consistency +- **`Writer`** (`internal/transform/writer.go`): Handles writing resources, patches, and kustomization files to stage directories +- **`Stage`** (`internal/transform/stages.go`): Represents a transform stage with discovery, validation, and navigation utilities + +## Apply Phase + +**Package:** `cmd/apply/` and `internal/apply/` + +1. **Stage discovery** — Discovers all stages in the transform directory +2. **Kustomize build** — Runs embedded kustomize (via the `krusty` API from `sigs.k8s.io/kustomize`) on each stage's directory +3. **Cluster-scoped filtering** — When `--skip-cluster-scoped` is set, filters out cluster-scoped resources from output +4. **Output writing** — Writes results to `output/output.yaml` (combined) and `output/resources//` (individual files); cluster-scoped resources go to `output/resources/_cluster/` + +The `KustomizeApplier` (`internal/apply/kustomize.go`) embeds the kustomize library directly via the `krusty.MakeKustomizer` API, eliminating the external kubectl dependency. Additional kustomize arguments (e.g., `--enable-helm`) can be passed via `--kustomize-args`. + +## Validate Phase + +**Package:** `cmd/validate/` and `internal/validate/` + +1. **Scanning** (`scanner.go`): Reads manifests from the output directory, extracting GVK + namespace tuples +2. **Matching** (`matcher.go`): Queries the target cluster's discovery API to check if each GVK is served; alternatively, matches against a captured API surface JSON file for offline validation +3. **Reporting** (`report.go`): Generates a compatibility report (JSON/YAML) and writes incompatible resources to a failures directory + +Supports two modes: +- **Live mode**: Queries target cluster via kubeconfig/context +- **Offline mode**: Validates against a captured API surface file (`--api-resources`), useful for air-gapped environments + +## Plugin System + +Plugins are external binaries that: +1. Receive a Kubernetes resource on stdin +2. Return JSONPatch operations on stdout +3. Are discovered from `~/.local/share/crane/plugins/` (default) + +The built-in `KubernetesPlugin` (from `crane-lib`) removes server-managed fields like `metadata.uid`, `metadata.resourceVersion`, `metadata.creationTimestamp`, `metadata.managedFields`, and `status`. + +## PVC Transfer + +**Package:** `cmd/transfer-pvc/` + +Uses the [pvc-transfer](https://github.com/backube/pvc-transfer) library: +1. Creates a PVC on the destination cluster +2. Sets up an rsync daemon Pod with an encrypted stunnel transport +3. Creates a public endpoint (Route or Ingress) on the destination +4. Runs an rsync client Pod on the source to transfer data +5. Cleans up all transfer resources after completion + +## Data Flow + +```text + ┌──────────────┐ + │ Source │ + │ Cluster │ + └──────┬───────┘ + │ crane export + ▼ + ┌──────────────┐ + │ export/ │ Raw manifests (namespace + _cluster/) + │ resources/ │ + └──────┬───────┘ + │ crane transform + ▼ + ┌──────────────┐ + │ transform/ │ Kustomize stages with patches + │ / │ + └──────┬───────┘ + │ crane apply (embedded kustomize) + ▼ + ┌──────────────┐ + │ output/ │ Clean, deployable manifests + │ output.yaml │ + └──────┬───────┘ + │ crane validate + ▼ + ┌──────────────┐ + │ validate/ │ Compatibility report + │ report.json │ + └──────┬───────┘ + │ kubectl apply + ▼ + ┌──────────────┐ + │ Target │ + │ Cluster │ + └──────────────┘ +``` diff --git a/docs/development/plugin-development.md b/docs/development/plugin-development.md new file mode 100644 index 00000000..395b942d --- /dev/null +++ b/docs/development/plugin-development.md @@ -0,0 +1,187 @@ +# Plugin Development + +Crane uses a plugin system for transforming Kubernetes resources during migration. Plugins generate JSONPatch (RFC 6902) operations that are applied via Kustomize. + +## How Plugins Work + +1. Crane discovers plugin binaries in the plugin directory (`~/.local/share/crane/plugins/` by default) +2. During transform, each plugin receives a Kubernetes resource on stdin +3. The plugin analyzes the resource and returns JSONPatch operations on stdout +4. Crane writes the patches to the stage's `patches/` directory +5. During apply, the embedded kustomize engine applies the patches to the resources + +## Plugin Interface + +A plugin is any executable binary that: + +- Reads a Kubernetes resource (JSON) from **stdin** +- Writes JSONPatch operations (JSON array) to **stdout** +- Returns exit code `0` on success, non-zero on error +- Writes error messages to **stderr** + +### Input (stdin) + +A single Kubernetes resource in JSON format: + +```json +{ + "apiVersion": "apps/v1", + "kind": "Deployment", + "metadata": { + "name": "my-app", + "namespace": "default", + "uid": "abc-123", + "resourceVersion": "12345" + }, + "spec": { "..." : "..." }, + "status": { "..." : "..." } +} +``` + +### Output (stdout) + +A JSON array of RFC 6902 JSONPatch operations: + +```json +[ + {"op": "remove", "path": "/metadata/uid"}, + {"op": "remove", "path": "/metadata/resourceVersion"}, + {"op": "remove", "path": "/status"}, + {"op": "add", "path": "/metadata/labels/migrated", "value": "true"} +] +``` + +Return an empty array `[]` if no transformations are needed. + +## Writing a Plugin in Go + +```go +package main + +import ( + "encoding/json" + "fmt" + "os" + + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" +) + +type PatchOp struct { + Op string `json:"op"` + Path string `json:"path"` + Value interface{} `json:"value,omitempty"` +} + +func main() { + var resource unstructured.Unstructured + if err := json.NewDecoder(os.Stdin).Decode(&resource); err != nil { + fmt.Fprintf(os.Stderr, "failed to decode resource: %v\n", err) + os.Exit(1) + } + + var patches []PatchOp + + // Example: add a migration label + patches = append(patches, PatchOp{ + Op: "add", + Path: "/metadata/labels/migrated-by", + Value: "my-plugin", + }) + + // Example: remove a specific annotation + annotations := resource.GetAnnotations() + if _, ok := annotations["source-cluster-only"]; ok { + patches = append(patches, PatchOp{ + Op: "remove", + Path: "/metadata/annotations/source-cluster-only", + }) + } + + if err := json.NewEncoder(os.Stdout).Encode(patches); err != nil { + fmt.Fprintf(os.Stderr, "failed to encode patches: %v\n", err) + os.Exit(1) + } +} +``` + +Build and install: + +```bash +go build -o ~/.local/share/crane/plugins/MyCustomPlugin +``` + +## Writing a Plugin in Bash + +```bash +#!/bin/bash +# Simple plugin that adds a label to all resources + +cat <_Plugin` — Plugin-based stage (must have matching plugin) +- `_` — Pass-through stage (no plugin needed) + +## Testing Plugins + +### Manual Testing + +```bash +# Test with a sample resource +cat sample-deployment.json | ./MyCustomPlugin + +# Test in a full pipeline +crane transform --stage 20_MyCustomPlugin +crane apply +``` + +### Unit Testing + +Test your plugin with various resource types to ensure it handles edge cases: + +- Resources without the fields you're trying to modify +- Resources with deeply nested structures +- Cluster-scoped resources (no namespace) +- Custom resources (CRDs) + +## Plugin Priority + +Plugins are executed in stage order (by the numeric prefix). Lower numbers run first: + +| Priority | Typical Use | +|----------|------------| +| 10 | Core cleanup (KubernetesPlugin) | +| 20 | Platform-specific (OpenshiftPlugin) | +| 30-40 | Security, networking | +| 50-70 | Storage, images | +| 80-90 | Custom application transformations | + +## Existing Plugins + +- **KubernetesPlugin** (built-in via crane-lib): Removes server-managed fields +- [crane-plugins](https://github.com/konveyor/crane-plugins): Community plugins +- [crane-plugin-openshift](https://github.com/konveyor/crane-plugin-openshift): OpenShift-specific transformations + +## Best Practices + +1. **Idempotent**: Running the plugin multiple times should produce the same result +2. **Defensive**: Check if fields exist before removing them +3. **Focused**: Each plugin should handle one concern +4. **Documented**: Include usage instructions and examples +5. **Tested**: Cover edge cases (missing fields, different resource types) diff --git a/docs/development/setup.md b/docs/development/setup.md new file mode 100644 index 00000000..5cd56081 --- /dev/null +++ b/docs/development/setup.md @@ -0,0 +1,128 @@ +# Development Setup + +## Prerequisites + +| Tool | Version | Purpose | +|------|---------|---------| +| Go | 1.25+ (see `go.mod`) | Building and testing | +| kubectl | Latest stable | Deploying output to clusters; not required for `crane apply` (kustomize is embedded) | +| Docker | Latest stable | Container-based testing (optional) | +| kind or minikube | Latest stable | Local Kubernetes clusters for E2E tests | + +## Getting Started + +### Clone the Repository + +```bash +git clone https://github.com/konveyor/crane.git +cd crane +``` + +### Build + +```bash +go build -o crane main.go +``` + +### Install Locally + +```bash +go install . +``` + +Or move the binary to your PATH: + +```bash +sudo mv crane /usr/local/bin/ +``` + +### Run Tests + +```bash +# All unit tests +go test ./... + +# Specific package +go test ./internal/transform/... + +# With verbose output +go test -v ./cmd/export/... + +# With race detection +go test -race ./... +``` + +### Format Code + +```bash +gofmt -w . +``` + +## Project Layout + +Commands live in `cmd//` — each is a self-contained package with a cobra command constructor (`NewXxxCommand`), an `Options` struct holding flags, and `Complete/Validate/Run` methods. + +Internal packages in `internal/` contain the business logic. Commands are thin wrappers that parse flags and call into internal packages. + +## Setting Up Test Clusters + +For E2E testing, you need source and target Kubernetes clusters: + +### Using kind + +```bash +# Create source cluster +kind create cluster --name source + +# Create target cluster +kind create cluster --name target + +# Verify contexts +kubectl config get-contexts +``` + +### Using minikube + +```bash +# Create source cluster +minikube start -p source + +# Create target cluster +minikube start -p target +``` + +## IDE Configuration + +### VS Code + +Recommended extensions: +- Go (golang.go) +- YAML (redhat.vscode-yaml) + +### GoLand / IntelliJ + +The project should be auto-detected as a Go module. Ensure the Go SDK is configured to match the version in `go.mod`. + +## Common Development Tasks + +### Adding a New Command + +1. Create `cmd//.go` +2. Define an `Options` struct with `Complete`, `Validate`, and `Run` methods +3. Create a `NewXxxCommand` function returning `*cobra.Command` +4. Register in `main.go` + +### Adding a New Flag + +Add the flag in the `addFlagsForOptions` function of the relevant command. Use `mapstructure` tags for viper integration. + +### Working with Kubernetes API Objects + +Use `*unstructured.Unstructured` for dynamic resources. Always check type assertions: + +```go +obj, ok := item.(*unstructured.Unstructured) +if !ok { + return fmt.Errorf("expected *unstructured.Unstructured but got %T", item) +} +``` diff --git a/docs/development/testing.md b/docs/development/testing.md new file mode 100644 index 00000000..f5306314 --- /dev/null +++ b/docs/development/testing.md @@ -0,0 +1,121 @@ +# Testing Guide + +## Unit Tests + +### Running Tests + +```bash +# All tests +go test ./... + +# Specific package +go test ./internal/transform/... + +# Verbose output +go test -v ./internal/validate/... + +# With coverage +go test -coverprofile=coverage.out ./... +go tool cover -html=coverage.out +``` + +### Test Conventions + +- Test files are `*_test.go` in the same package +- Use table-driven tests for multiple scenarios +- Test helpers go in `test_helpers.go` files (e.g., `internal/transform/test_helpers.go`) + +### Table-Driven Test Example + +```go +func TestValidateStageName(t *testing.T) { + tests := []struct { + name string + stageName string + wantErr bool + }{ + {"valid plugin stage", "10_KubernetesPlugin", false}, + {"valid custom stage", "30_CustomEdits", false}, + {"missing number", "KubernetesPlugin", true}, + {"missing name", "10_", true}, + {"empty", "", true}, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + err := ValidateStageName(tt.stageName) + if (err != nil) != tt.wantErr { + t.Errorf("ValidateStageName(%q) error = %v, wantErr %v", + tt.stageName, err, tt.wantErr) + } + }) + } +} +``` + +### Testing with Temporary Directories + +Many tests create temporary directories for export/transform/output: + +```go +func TestSomething(t *testing.T) { + tmpDir := t.TempDir() // Automatically cleaned up + exportDir := filepath.Join(tmpDir, "export") + os.MkdirAll(exportDir, 0700) + // ... test logic +} +``` + +## E2E Tests + +### Location + +E2E tests live in `e2e-tests/`: + +```text +e2e-tests/ +├── framework/ # Test framework (app deployment, kubectl wrappers, pipeline) +├── tests/ # Test cases (mtc_XXX_*.go) +├── golden-manifests/ # Expected output fixtures +└── utils/ # Helper utilities +``` + +### Running E2E Tests + +E2E tests require running Kubernetes clusters: + +```bash +cd e2e-tests +go test -v ./tests/... +``` + +### Golden Manifests + +The `golden-manifests/` directory contains expected export and output for known applications (e.g., `redis`, `simple-nginx-nopv`). E2E tests compare actual output against these fixtures. + +### Writing E2E Tests + +Follow existing patterns in `e2e-tests/tests/`: + +1. Use `framework.DeployApp()` to set up the test application +2. Run the Crane pipeline using `framework.RunPipeline()` +3. Compare output against golden manifests or verify specific resource properties + +## CI + +- **Unit tests and build**: Run on every push via `.github/workflows/go.yml` +- **E2E tests**: Run on push to main and on a nightly cron schedule via `.github/workflows/run-e2e-tests.yml` + +## Test Coverage + +To check coverage for a specific package: + +```bash +go test -coverprofile=cover.out ./internal/transform/ +go tool cover -func=cover.out +``` + +Guidelines: +- All new features require tests +- Bug fixes should include regression tests +- Focus on testing behavior, not implementation details diff --git a/docs/installation.md b/docs/installation.md new file mode 100644 index 00000000..1ee78495 --- /dev/null +++ b/docs/installation.md @@ -0,0 +1,54 @@ +# Installation + +## Prerequisites + +- **Go** 1.25+ (only required if building from source) +- **kubectl** — required for deploying output to clusters and for `crane validate` with live cluster mode; not required for `crane apply` (kustomize is embedded) +- **Docker** (optional) — for container-based workflows + +## Install from Release + +Download a prebuilt binary from the [releases page](https://github.com/konveyor/crane/releases): + +```bash +# Download the latest release (adjust OS/arch as needed) +curl -L https://github.com/konveyor/crane/releases/latest/download/crane-linux-amd64 -o crane +chmod +x crane +sudo mv crane /usr/local/bin/ +``` + +## Build from Source + +```bash +git clone https://github.com/konveyor/crane.git +cd crane +go build -o crane main.go +sudo mv crane /usr/local/bin/ +``` + +## Verify Installation + +```bash +crane version +``` + +You should see the Crane version output. + +## Quick Test + +Verify Crane works with an accessible cluster: + +```bash +# Export resources from a namespace +crane export -n default --export-dir /tmp/crane-test + +# Check exported resources +ls /tmp/crane-test/resources/default/ +``` + +If you see YAML files for resources in the `default` namespace, Crane is working correctly. + +## Next Steps + +- [Command Reference](./README.md#command-reference) — Detailed documentation for each command +- [Contributing](../CONTRIBUTING.md) — How to contribute diff --git a/docs/multistage-pipeline.md b/docs/multistage-pipeline.md new file mode 100644 index 00000000..97ba8fb1 --- /dev/null +++ b/docs/multistage-pipeline.md @@ -0,0 +1,523 @@ +# Multi-Stage Kustomize Transform Pipeline + +This document describes the multi-stage Kustomize transform pipeline feature for Crane, which replaces the previous JSONPatch file-per-resource workflow with a more flexible, scalable approach. + +## Overview + +The multi-stage pipeline allows transformations to be organized into sequential stages, where each stage can apply a different set of plugins to resources. Stages are processed in order based on their priority, and the output of one stage becomes the input for the next. + +## Key Concepts + +### Stage Directory Structure + +Each stage is a directory following the naming convention `_`: + +```text +transform/ +├── 10_KubernetesPlugin/ +│ ├── resources/ +│ │ ├── deployment.yaml # Grouped by resource type +│ │ ├── service.yaml +│ │ └── configmap.yaml +│ ├── patches/ +│ │ ├── deployment-myapp-default.yaml +│ │ └── service-myapp-default.yaml +│ ├── kustomization.yaml # Generated Kustomize file +│ ├── whiteout-report.yaml # Resources excluded from output +│ ├── ignored-patches-report.yaml # Patches discarded due to conflicts +│ └── .crane-metadata.json # Stage metadata with content hashes +├── 20_OpenshiftPlugin/ +│ └── ... +└── 30_ImagestreamPlugin/ + └── ... +``` + +### Resource Grouping + +Resources are grouped by type (kind + API group) into multi-document YAML files: +- Core resources: `deployment.yaml`, `service.yaml`, `pod.yaml` +- Non-core resources: `route.route.openshift.io.yaml`, `imagestream.image.openshift.io.yaml` + +### Kustomization File + +Each stage contains a `kustomization.yaml` that references: +- **resources**: List of resource files from the `resources/` directory +- **patches**: Strategic merge patches or JSON patches with target selectors + +Example: +```yaml +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +resources: +- resources/deployment.yaml +- resources/service.yaml +patches: +- path: patches/deployment-myapp-default.yaml + target: + group: apps + version: v1 + kind: Deployment + name: myapp + namespace: default +``` + +### Resource Cleanup via Plugins + +Exported resources contain live/runtime metadata from the source cluster that must be removed before applying to a target cluster. Crane uses **plugins** to generate JSONPatch operations that clean these resources. + +**Built-in Kubernetes Plugin**: + +The `kubernetes` plugin (from `crane-lib`) automatically removes: +- `metadata.uid` - Cluster-assigned unique identifier +- `metadata.resourceVersion` - Etcd versioning field +- `metadata.creationTimestamp` - Original creation time +- `metadata.managedFields` - Server-side apply tracking +- `status` section - Runtime state + +**How it works**: + +1. **Transform phase**: Plugins analyze exported resources and generate JSONPatch operations +2. **Patches written**: Operations saved as Kustomize patches in `patches/` directory +3. **Apply phase**: Embedded kustomize applies patches to resources +4. **Result**: Clean, declarative manifests ready for target cluster + +**Example patch** (auto-generated by kubernetes plugin): +```yaml +- op: remove + path: /metadata/resourceVersion +- op: remove + path: /metadata/creationTimestamp +- op: remove + path: /metadata/managedFields +- op: remove + path: /status +- op: remove + path: /metadata/uid +``` + +This approach ensures: +- No conflicts with server-managed fields +- Resources can be applied to any cluster +- Idempotent operations (safe to re-apply) + +## CLI Usage + +### Transform Command + +#### Single Stage Mode (Default) + +Create a single transform stage using all available plugins: + +```bash +crane transform \ + --export-dir export \ + --transform-dir transform +``` + +This creates a `10_KubernetesPlugin` stage directory and runs all plugins (including the built-in `kubernetes` plugin) to clean exported resources. + +**Plugin Filtering** (optional): + +To run only specific plugins: + +```bash +crane transform \ + --export-dir export \ + --transform-dir transform \ + --stage-name 10_KubernetesPlugin \ + --plugin-name kubernetes +``` + +**Note**: When `--plugin-name` is omitted (default), ALL available plugins are used. This is recommended to ensure proper resource cleanup. + +#### Multi-Stage Mode + +Execute specific stages: + +```bash +# Run a specific stage +crane transform --stage 20_OpenshiftPlugin +``` + +#### Force Overwrite + +Override dirty check protection: + +```bash +crane transform --force --stage-name 10_KubernetesPlugin +``` + +### Apply Command + +#### Apply Final Stage (Default) + +Apply only the last stage in the pipeline: + +```bash +crane apply \ + --transform-dir transform \ + --output-dir output +``` + +This builds the final stage using embedded kustomize and writes the result to `output/output.yaml`. + +#### Apply Specific Stages + +Apply applies all stages sequentially by default. The apply command does not support stage-specific flags as it always processes the complete pipeline to ensure sequential consistency. + +## Priority Assignment + +### Auto-Assignment + +Plugin priorities are automatically assigned from stage directory names: + +```go +// Stage directories +10_KubernetesPlugin → plugin "kubernetes" gets priority 10 +20_OpenshiftPlugin → plugin "openshift" gets priority 20 +30_ImagestreamPlugin → plugin "imagestream" gets priority 30 +``` + +### Manual Assignment + +Override auto-assigned priorities: + +```bash +crane transform \ + --plugin-priorities kubernetes:5,openshift:15,imagestream:25 +``` + +### Recommended Priority Order + +The system provides heuristic-based recommendations: + +| Plugin Type | Recommended Priority | Keywords | +|-------------------|--------------------|------------------------------| +| Kubernetes Core | 10 | kubernetes, k8s, core | +| OpenShift | 20 | openshift, ocp | +| Namespace/Project | 30 | namespace, project | +| Security | 40 | security, scc, psp | +| Network | 50 | network, route, ingress | +| Storage | 60 | storage, pvc, pv | +| Image | 70 | image, imagestream, registry | +| Build | 80 | build, buildconfig | +| Custom | 90 | custom, app, application | + +## Stage Chaining + +Stages are chained automatically based on priority order: + +```text +export/ → 10_KubernetesPlugin/ → 20_OpenshiftPlugin/ → 30_ImagestreamPlugin/ → output/ +``` + +Each stage: +1. Reads input resources (from export or previous stage) +2. Applies transformations via plugins +3. Writes output to its stage directory +4. Next stage uses this output as input + +## Workflow Examples + +### Example 1: Simple Transform and Apply + +```bash +# Export resources from source cluster +crane export --kubeconfig source.yaml --export-dir export + +# Transform resources (single stage) +crane transform --export-dir export --transform-dir transform + +# Apply transformations +crane apply --transform-dir transform --output-dir output + +# Deploy to target cluster +kubectl apply -f output/output.yaml +``` + +### Example 2: Multi-Stage Pipeline + +```bash +# Export resources +crane export --kubeconfig source.yaml --export-dir export + +# Create Kubernetes base transformations +crane transform \ + --export-dir export \ + --transform-dir transform \ + --stage-name 10_KubernetesPlugin \ + --plugin-name kubernetes + +# Create OpenShift-specific transformations +crane transform \ + --transform-dir transform \ + --stage-name 20_OpenshiftPlugin \ + --plugin-name openshift + +# Create ImageStream transformations +crane transform \ + --transform-dir transform \ + --stage-name 30_ImagestreamPlugin \ + --plugin-name imagestream + +# Apply final stage +crane apply --transform-dir transform --output-dir output +``` + +### Example 3: Iterative Development + +```bash +# Initial transform +crane transform --export-dir export --transform-dir transform + +# Make manual edits to resources in transform/10_KubernetesPlugin/resources/ +# Edit deployment.yaml to add annotations, etc. + +# Try to re-run transform (will fail due to dirty check) +crane transform --export-dir export --transform-dir transform +# Error: contains user modifications + +# Force overwrite if needed +crane transform --export-dir export --transform-dir transform --force + +# Or preserve changes by using a different stage name +crane transform --stage-name 20_custom +``` + +## Migration from JSONPatch Workflow + +### Old Workflow + +```text +transform/ +├── namespace/ +│ └── default/ +│ └── deployment/ +│ └── myapp.json # JSONPatch per resource +``` + +### New Workflow + +```text +transform/ +└── 10_KubernetesPlugin/ + ├── resources/ + │ └── deployment.yaml # Grouped by type + ├── patches/ + │ └── deployment-myapp-default.yaml + └── kustomization.yaml +``` + +### Benefits + +1. **Reduced File Count**: Resources grouped by type instead of one file per resource +2. **Standard Format**: Uses Kustomize, a widely adopted tool +3. **Stage Chaining**: Supports multi-stage pipelines for complex transformations +4. **Better Diff**: Deterministic ordering produces stable Git diffs +5. **Dirty Check**: Prevents accidental overwrites of user modifications + +## Advanced Features + +### Stage Validation + +Validate stage names before applying: + +```go +import ( + "github.com/konveyor/crane/internal/transform" +) + +// Validate stage names +stages, err := transform.DiscoverStages(transformDir) +if err != nil { + fmt.Printf("failed to discover stages: %v\n", err) +} + +for _, stage := range stages { + if err := transform.ValidateStageName(stage.DirName); err != nil { + fmt.Printf("Invalid stage name %s: %v\n", stage.DirName, err) + } +} +``` + +### Custom Stage Naming + +Generate stage names with priority numbers: + +```go +import "github.com/konveyor/crane/internal/transform" + +// Generate a stage name with priority +stageName := transform.GenerateStageName(15, "my-plugin") +// Returns: "15_my-plugin" + +// Validate a stage name +err := transform.ValidateStageName("15_my-plugin") +if err != nil { + // Handle invalid stage name +} +``` + +## Troubleshooting + +### Issue: Transform fails with "contains user modifications" + +**Cause**: Stage directory has been manually edited after creation. + +**Solution**: +1. Use `--force` to overwrite changes +2. Use a different `--stage-name` to preserve changes +3. Commit changes to Git before re-running transform + +### Issue: Apply fails with "kustomization.yaml validation failed" + +**Cause**: Invalid Kustomize syntax or missing resources. + +**Solution**: +1. Check kustomization.yaml syntax +2. Verify all resource files exist in resources/ +3. Run `crane apply --stage ` to isolate the failing stage + +### Issue: Resources not appearing in output + +**Cause**: Resources may be whiteout (excluded) by plugins. + +**Solution**: +1. Check `whiteout-report.yaml` in stage directory +2. Review plugin configuration +3. Check plugin logs for whiteout decisions + +### Issue: Patches not being applied + +**Cause**: Patch file or target selector may be incorrect. + +**Solution**: +1. Verify patch file exists in patches/ +2. Check target selector matches resource metadata +3. Review `ignored-patches-report.yaml` for conflicts + +## Best Practices + +1. **Stage Naming**: Use descriptive names that indicate the transformation purpose + - Good: `10_KubernetesPlugin-base`, `20_OpenshiftPlugin-routes`, `30_security-context` + - Bad: `10_stage1`, `20_stage2` + +2. **Priority Spacing**: Leave gaps (10, 20, 30) to allow insertion of new stages + +3. **Version Control**: Commit transform directories to Git to track changes + +4. **Testing**: Always test transformed output before applying to production + +5. **Incremental Changes**: Use separate stages for different concerns (security, networking, storage) + +6. **Documentation**: Include README.md in transform directory explaining pipeline purpose + +## API Reference + +### Transform Package + +```go +// Stage discovery +stages, err := transform.DiscoverStages(transformDir) + +// Stage filtering +selector := transform.StageSelector{ + FromStage: "20_OpenshiftPlugin", + ToStage: "30_ImagestreamPlugin", +} +filtered := transform.FilterStages(stages, selector) + +// Stage navigation +first := transform.GetFirstStage(stages) +last := transform.GetLastStage(stages) +prev := transform.GetPreviousStage(stages, currentStage) +next := transform.GetNextStage(stages, currentStage) + +// Stage name validation and generation +err = transform.ValidateStageName("10_KubernetesPlugin") +stageName := transform.GenerateStageName(10, "kubernetes") +``` + +### Apply Package + +```go +// Kustomize apply (embedded — no kubectl dependency) +applier := &apply.KustomizeApplier{ + Log: logger, + TransformDir: transformDir, + OutputDir: outputDir, + SkipClusterScoped: false, +} + +// Apply a single stage +err = applier.ApplySingleStage("10_KubernetesPlugin") + +// Apply multiple stages with selector +selector := transform.StageSelector{ + FromStage: "10_KubernetesPlugin", + ToStage: "30_ImagestreamPlugin", +} +err = applier.ApplyMultiStage(selector) + +// Apply only the final stage (most common) +err = applier.ApplyFinalStage() +``` + +## Further Reading + +- [Kustomize Documentation](https://kubectl.docs.kubernetes.io/references/kustomize/) +- [Crane Plugin Development](./development/plugin-development.md) +- [Transform Architecture](./development/architecture.md) + +## Manual/Non-Plugin Stages + +Stages don't have to correspond to a plugin. If a stage directory name doesn't match any available plugin, the resources pass through unchanged. + +**Use case**: Manual transformation stages where you want to hand-edit resources. + +### Example + +```bash +# Create a multi-stage pipeline +crane transform --stage-name 10_KubernetesPlugin # Plugin-backed +crane transform --stage-name 50_ManualEdits # No matching plugin +crane transform --stage-name 90_FinalCleanup # No matching plugin +``` + +**What happens:** + +1. **10_KubernetesPlugin**: Resources transformed by KubernetesPlugin (removes metadata.uid, etc.) +2. **50_ManualEdits**: Resources copied unchanged to `transform/50_ManualEdits/resources/` + - No plugins match "ManualEdits" + - No patches generated + - You can manually edit resources in this stage +3. **90_FinalCleanup**: Resources from previous stage copied unchanged + - User can add manual patches or edits + +### Behavior + +When stage name doesn't match any plugin: +- `filterPluginsByStage()` returns empty list `[]` +- `runner.Run()` called with empty plugin list +- Resources written unchanged (no transformations) +- No patches generated +- **This is intentional** - allows user-controlled stages + +### Mixed Pipeline Example + +```text +export/ +└── resources/ + └── deployment.yaml (raw export with uid, resourceVersion, etc.) + ↓ +transform/10_KubernetesPlugin/ (plugin: removes server-managed fields) +└── resources/deployment.yaml (cleaned) + ↓ +transform/50_ManualEdits/ (no plugin: user edits) +└── resources/deployment.yaml (manually edited) + ↓ +transform/90_CustomPlugin/ (plugin: custom transformations) +└── resources/deployment.yaml (custom patches applied) + ↓ +output/output.yaml (final result) +``` diff --git a/docs/plugins.md b/docs/plugins.md new file mode 100644 index 00000000..a14b22b7 --- /dev/null +++ b/docs/plugins.md @@ -0,0 +1,82 @@ +# Plugins + +Crane uses plugins to transform Kubernetes resources during migration. Plugins generate JSONPatch operations that clean, modify, or adapt resources for the target cluster. + +## Built-in Plugin + +### KubernetesPlugin + +The built-in Kubernetes plugin (from [crane-lib](https://github.com/konveyor/crane-lib)) automatically removes server-managed fields that would conflict when applying to a new cluster: + +- `metadata.uid` +- `metadata.resourceVersion` +- `metadata.creationTimestamp` +- `metadata.managedFields` +- `status` + +This plugin runs as the default first stage (`10_KubernetesPlugin`) and is always available without additional installation. + +## Community Plugins + +### crane-plugins + +The [konveyor/crane-plugins](https://github.com/konveyor/crane-plugins) repository contains community-contributed plugins based on experience from real-world Kubernetes migrations. + +### OpenShift Plugin + +The [konveyor/crane-plugin-openshift](https://github.com/konveyor/crane-plugin-openshift) plugin handles OpenShift-specific migration concerns: + +- Route transformations +- ImageStream references +- SecurityContextConstraints adjustments +- DeploymentConfig to Deployment conversion + +Install: + +```bash +# Download to the default plugin directory +curl -L -o ~/.local/share/crane/plugins/OpenshiftPlugin +chmod +x ~/.local/share/crane/plugins/OpenshiftPlugin +``` + +Use in a transform stage: + +```bash +crane transform --stage 20_OpenshiftPlugin +``` + +## Managing Plugins + +### Plugin Directory + +Plugins are discovered from `~/.local/share/crane/plugins/` by default. Override with `--plugin-dir`: + +```bash +crane transform --plugin-dir /path/to/plugins +``` + +### Listing Available Plugins + +```bash +crane transform list-plugins +``` + +### Skipping Plugins + +Skip specific plugins during transform: + +```bash +crane transform --skip-plugins OpenshiftPlugin +``` + +### Plugin Optional Flags + +Pass configuration to plugins: + +```bash +crane transform --optional-flags '{"new-namespace": "production"}' +``` + +## Writing Custom Plugins + +Plugins are executable binaries that read a Kubernetes resource from stdin and write JSONPatch operations to stdout. See the [Plugin Development Guide](./development/plugin-development.md) for details. diff --git a/docs/resource-compatibility.md b/docs/resource-compatibility.md new file mode 100644 index 00000000..e9933574 --- /dev/null +++ b/docs/resource-compatibility.md @@ -0,0 +1,75 @@ +# Crane Compatibility Matrix: Namespace vs. Cluster Resources + +> **Note to Users:** Crane is primarily a namespace-scoped migration tool. However, it can migrate cluster-scoped resources if they are explicitly related to the namespace workload and the migration context has sufficient RBAC permissions. + +--- + +## 1. Executive Summary +This document defines the operational boundaries of **Crane**. Migration success depends on two factors: +1. **Functional Relevance:** Is the cluster resource strictly required by the namespace workload? +2. **Permission Context:** Does the migration service account have the required RBAC to "get" and "create" these cluster-scoped objects? + +--- + +## 2. Resource Support Visualization +The following diagram illustrates how Crane treats different resource layers based on their relationship to the application. + +```mermaid +graph TD +subgraph Cluster_Scope_Conditional ["Cluster Scope (Conditionally Supported)"] +CRD[Custom Resource Definitions] +RBAC_C[ClusterRoles / Bindings] +SCC[Security Context Constraints] +end +subgraph Namespace_Scope_Supported ["Namespace Scope (Fully Supported)"] +Workload[Deployments / StatefulSets] +Net[Services / Routes] +Cfg[ConfigMaps / Secrets] +end +Workload -->|Depends On| CRD +Workload -->|Bound To| SCC +Namespace_Scope_Supported -->|Automated Export| Target[Target Cluster] +Cluster_Scope_Conditional -.->|Migrated if Related & Permitted| Target +``` + +--- + +## 3. Detailed Compatibility Matrix + +### 3.1 Namespace-Scoped Resources (Fully Supported) +These resources are the core of any migration and are moved automatically. +* **Workloads:** Deployment, DeploymentConfig, StatefulSet, DaemonSet, Job, CronJob. +* **Networking:** Service, Route, Ingress, Endpoints. +* **Config & Secrets:** ConfigMap, Secret, ServiceAccount. +* **Storage:** PersistentVolumeClaim (PVC). + +### 3.2 Cluster-Scoped Resources (Conditionally Supported) +These resources are migrated **only if they are linked to the namespace workload** and the execution context has appropriate permissions. + +* **Definitions (CRDs):** Migrated if the namespace contains Custom Resources (CRs) that depend on these definitions. +* **Global Security (ClusterRole/Binding):** Migrated if specifically referenced by a ServiceAccount within the migrating namespace. +* **Security Context Constraints (SCCs):** Migrated if the application requires specific localized SCCs to run (common in OpenShift environments). +* **Quota & Limits:** ResourceQuotas and LimitRanges are migrated if they are defined specifically for the source namespace. + +### 3.3 Infrastructure Resources (Manual/Environmental) +These are generally not migrated as they represent the physical or cloud provider environment. +* **Nodes & MachineConfigs:** Environment-specific hardware configurations. +* **StorageClasses:** These must be mapped to the destination's available storage providers. +* **Operator Controllers:** While the *Operands* (the apps) move, the *Operators* (the managers) must be pre-installed on the target. + +--- + +## 4. Permission & Context Requirements +For "Conditionally Supported" resources to migrate successfully, the following must be true: +* **Sufficient RBAC:** The migration user/service account must have cluster-wide `get`, `list`, and `create` permissions for the specific cluster-scoped types. +* **Owner References:** Crane looks for functional links (e.g., a ServiceAccount tied to a ClusterRole) to determine what "related" infrastructure needs to move. + +--- + +## 5. Pre-Migration Checklist +* [ ] **RBAC Audit:** Ensure the migration identity has permissions to access related ClusterRoles and SCCs. +* [ ] **Storage Mapping:** Confirm that destination StorageClasses are ready to receive moved PVCs. +* [ ] **Operator Readiness:** All required Operators are installed via OperatorHub on the target cluster. +* [ ] **CRD Check:** Verify if any global CRDs need to be manually applied to the target to avoid "orphan" resources. + +--- From 8bcfe38b5ec2a24ac6b32ab7e9c9b9c75456f1fe Mon Sep 17 00:00:00 2001 From: Ilanit Stein Date: Wed, 17 Jun 2026 11:26:57 +0300 Subject: [PATCH 2/5] docs: update documentation for latest CLI changes and review feedback Update all doc files to reflect 35 commits merged to main since the original PR: --stage flag replaced with positional arguments, --overwrite added to export/apply/validate, transform default behavior changed to create stages for all plugins, --skip-plugins and --instructions-file flags added, --api-resources mutual exclusion expanded, and --plugin-priorities/--stage-name/--plugin-name flags removed. Address reviewer feedback: use verb-first commit/PR titles instead of conventional commit prefixes (aufi), add community plugin maintenance note (aufi), fix conditional Required status for transfer-pvc nginx flags (CodeRabbit), add defensive JSONPatch parent-object checks in plugin examples (CodeRabbit), add nil check for GetAnnotations (CodeRabbit), and resolve apply stage-specific flag contradiction (CodeRabbit). Co-Authored-By: Claude Opus 4.6 --- CONTRIBUTING.md | 16 +++--- docs/commands/apply.md | 11 +++-- docs/commands/export.md | 5 ++ docs/commands/transfer-pvc.md | 4 +- docs/commands/transform.md | 53 ++++++++++++-------- docs/commands/validate.md | 8 +-- docs/development/plugin-development.md | 25 +++++++--- docs/multistage-pipeline.md | 67 ++++++++++++-------------- docs/plugins.md | 6 ++- 9 files changed, 112 insertions(+), 83 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 8ff7b101..78e7a9cd 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -59,17 +59,17 @@ fmt.Errorf("expected *unstructured.Unstructured but got %T", u) - Always check type assertions and provide informative error messages - Include the actual resource type and API resource name in errors -## Commit Messages +## Commit and PR Title Conventions -Use conventional commits format: +Use verb-first titles that describe what the change does: -- `fix:` for bug fixes -- `feat:` for new features -- `refactor:` for code restructuring -- `test:` for test additions -- `docs:` for documentation +- `Fix` for bug fixes (e.g., `Fix transform error on empty namespace`) +- `Add` for new features (e.g., `Add crane validate command`) +- `Update` for enhancements (e.g., `Update crane-lib with resources whiteout`) +- `Refactor` for code restructuring +- `Remove` for removals -Include issue references where applicable: `fix: improve error messages (#197)` +Include issue references where applicable: `Fix transform error on empty namespace (#197)` ## Pull Request Guidelines diff --git a/docs/commands/apply.md b/docs/commands/apply.md index 7110ffec..7ae9acad 100644 --- a/docs/commands/apply.md +++ b/docs/commands/apply.md @@ -5,7 +5,7 @@ Apply transformations to exported resources and produce final manifests. ## Synopsis ```bash -crane apply [flags] +crane apply [stage...] [flags] ``` ## Description @@ -21,9 +21,11 @@ Kustomize is embedded directly in the Crane binary (via the krusty API), so no e | `--export-dir` | `-e` | `export` | The path where exported resources are saved (kept for consistency; not used by apply) | | `--transform-dir` | `-t` | `transform` | The path where transform stage directories are located | | `--output-dir` | `-o` | `output` | The path where final manifests are written | -| `--stage` | | | Apply a specific stage only (e.g., `10_KubernetesPlugin`). If not specified, all stages are applied | | `--kustomize-args` | | | Additional arguments for kustomize (e.g., `--enable-helm --helm-command=helm3`) | | `--skip-cluster-scoped` | | `false` | Exclude cluster-scoped resources (ClusterRole, ClusterRoleBinding, CRD, etc.) from output. Useful for non-admin migration scenarios | +| `--overwrite` | | `false` | Overwrite the output directory if it already exists | + +Stages are specified as positional arguments (e.g., `crane apply 10_KubernetesPlugin`). Stages can be specified by directory name or plugin name. If no stages are specified, all discovered stages are applied sequentially. ## Output Structure @@ -61,7 +63,7 @@ crane apply --transform-dir ./migration/transform --output-dir ./migration/outpu ### Apply a specific stage only ```bash -crane apply --stage 10_KubernetesPlugin +crane apply 10_KubernetesPlugin ``` ### Skip cluster-scoped resources @@ -87,7 +89,8 @@ kubectl apply -f output/output.yaml | Error | Cause | Solution | |-------|-------|----------| -| `kustomization.yaml validation failed` | Invalid Kustomize syntax or missing resource files | Run `crane apply --stage ` to isolate the failing stage | +| `kustomization.yaml validation failed` | Invalid Kustomize syntax or missing resource files | Run `crane apply ` to isolate the failing stage | +| `output directory "X" already exists` | Output directory from a previous run | Use `--overwrite` to replace it | | `invalid stage name` | Stage name doesn't follow `_` format | Use a valid stage name like `10_KubernetesPlugin` | | `invalid kustomize-args` | Unsupported or malformed kustomize arguments | Check supported kustomize flags | diff --git a/docs/commands/export.md b/docs/commands/export.md index 690c93b9..d065a730 100644 --- a/docs/commands/export.md +++ b/docs/commands/export.md @@ -30,9 +30,12 @@ When custom resources are found in the namespace, Crane automatically collects t | `--as-extras` | | | Extra impersonation info (format: `key=val1,val2;key2=val3`) | | `--qps` | `-q` | `100` | Query-per-second rate for API requests | | `--burst` | `-b` | `1000` | API burst rate | +| `--overwrite` | | `false` | Overwrite the export directory if it already exists | Standard kubeconfig flags (`--kubeconfig`, `--context`, `--cluster`, `--as`, `--as-group`, etc.) are also available. +> **Note:** `--context` is mutually exclusive with `--cluster`, `--server`, `--user`, and `--token`. + ## Output Structure ```text @@ -101,6 +104,8 @@ crane export -n my-app --crd-skip-group monitoring.coreos.com | `namespaces "X" not found` | Namespace does not exist | Verify namespace name | | `cannot verify namespace exists` | Insufficient RBAC (warning only) | Export proceeds; verify namespace exists manually | | `extras requires specifying a user or group` | `--as-extras` used without `--as` | Add `--as` or `--as-group` flag | +| `export directory "X" already exists` | Export directory from a previous run | Use `--overwrite` to replace it | +| Non-zero exit with aggregated error | All namespace list calls returned Forbidden | Ensure service account has list permissions on at least one namespace | ## Next Steps diff --git a/docs/commands/transfer-pvc.md b/docs/commands/transfer-pvc.md index cb2b996b..5b0d8404 100644 --- a/docs/commands/transfer-pvc.md +++ b/docs/commands/transfer-pvc.md @@ -33,8 +33,8 @@ The above command transfers PVC (along with PV data) named `` in the n | `--destination-image` | string | No | Custom image to use for destination rsync Pod | | `--source-image` | string | No | Custom image to use for source rsync Pod | | `--endpoint` | string | No | Kind of endpoint to create in destination cluster (see [Endpoint Options](#endpoint-options)) | -| `--ingress-class` | string | No | Ingress class when endpoint is nginx-ingress | -| `--subdomain` | string | No | Custom subdomain to use for the endpoint | +| `--ingress-class` | string | When endpoint is nginx-ingress | Ingress class when endpoint is nginx-ingress | +| `--subdomain` | string | When endpoint is nginx-ingress | Custom subdomain to use for the endpoint | | `--output` | string | No | Output transfer stats in the specified file | | `--verify` | bool | No | Verify transferred files using checksums | diff --git a/docs/commands/transform.md b/docs/commands/transform.md index a2f86b2f..7255c957 100644 --- a/docs/commands/transform.md +++ b/docs/commands/transform.md @@ -58,7 +58,7 @@ You can edit resources in the `resources/` directory: vi transform/10_KubernetesPlugin/resources/Deployment_apps_v1_default_wordpress.yaml # Preview changes -crane apply --stage 10_KubernetesPlugin +crane apply 10_KubernetesPlugin # Apply changes kubectl apply -f output/output.yaml @@ -128,21 +128,32 @@ transform/.work/ ```bash # Default: discover and run all existing stages -# If no stages exist, creates default 10_KubernetesPlugin stage +# If no stages exist, creates stages for all available plugins crane transform -# Run specific stage only (creates it if doesn't exist) -crane transform --stage 10_KubernetesPlugin +# Run specific stages only (creates them if they don't exist) +crane transform 10_KubernetesPlugin + +# Run multiple specific stages +crane transform 10_KubernetesPlugin 20_OpenshiftPlugin # Create a new plugin-based stage automatically # Stage name ending with "Plugin" will use that plugin -crane transform --stage 35_OpenshiftPlugin +crane transform 35_OpenshiftPlugin # Create a new pass-through stage for manual editing # Stage name NOT ending with "Plugin" creates empty pass-through -crane transform --stage 40_CustomManualEdits +crane transform 40_CustomManualEdits + +# Skip specific plugins when running all stages +crane transform --skip-plugins OpenshiftPlugin + +# Use a declarative instructions file (mutually exclusive with positional stage args) +crane transform --instructions-file instructions.yaml ``` +Shell autocompletion is available for plugin and stage names. + ### Applying Transforms ```bash @@ -176,7 +187,9 @@ output/ ## Automatic Stage Creation -Crane automatically creates new stages when you reference them with `--stage`. The stage name determines whether a plugin will be used or if it's a pass-through stage for manual editing. +When no stages exist in the transform directory, `crane transform` automatically creates stages for **all available plugins** (not just KubernetesPlugin). Plugins are sorted alphabetically and assigned priorities starting at 10, incrementing by 5. Use `--skip-plugins` to exclude specific plugins from this default behavior. + +You can also create new stages by specifying them as positional arguments. The stage name determines whether a plugin will be used or if it's a pass-through stage for manual editing. ### Stage Naming Convention @@ -198,7 +211,7 @@ Stage names ending with `Plugin` **must** have a corresponding plugin installed. ```bash # Creates a stage using OpenshiftPlugin -crane transform --stage 20_OpenshiftPlugin +crane transform 20_OpenshiftPlugin # Output: # - resources/ (from previous stage or export) @@ -209,7 +222,7 @@ crane transform --stage 20_OpenshiftPlugin **If the plugin doesn't exist, you'll get an error:** ```bash -crane transform --stage 20_NonexistentPlugin +crane transform 20_NonexistentPlugin # Error: stage 20_NonexistentPlugin requires plugin 'NonexistentPlugin' # but it was not found (available plugins: KubernetesPlugin, NamespaceCleanup) ``` @@ -220,7 +233,7 @@ Stage names **not** ending with `Plugin` create pass-through stages where resour ```bash # Creates a pass-through stage for manual editing -crane transform --stage 30_CustomEdits +crane transform 30_CustomEdits # Output: # - resources/ (copied unchanged from previous stage) @@ -334,7 +347,7 @@ kubectl kustomize transform/10_KubernetesPlugin/ ```bash # Create a custom stage for manual edits -crane transform --stage 40_CustomEdits +crane transform 40_CustomEdits # Edit resources vi transform/40_CustomEdits/resources/Deployment_apps_v1_default_wordpress.yaml @@ -361,10 +374,10 @@ crane transform crane transform --force # Run specific plugin stage (regenerates automatically) -crane transform --stage 10_KubernetesPlugin +crane transform 10_KubernetesPlugin # Run specific custom stage (requires --force if directory not empty) -crane transform --stage 40_CustomEdits --force +crane transform 40_CustomEdits --force ``` ### 4. Working with Multiple Stages @@ -374,11 +387,11 @@ crane transform --stage 40_CustomEdits --force crane transform # Automatically add more plugin stages -crane transform --stage 20_OpenshiftPlugin -crane transform --stage 30_ImagestreamPlugin +crane transform 20_OpenshiftPlugin +crane transform 30_ImagestreamPlugin # Add a manual editing stage -crane transform --stage 40_CustomEdits +crane transform 40_CustomEdits # Edit the manual stage (example: edit a specific deployment) vi transform/40_CustomEdits/resources/Deployment_apps_v1_default_wordpress.yaml @@ -402,11 +415,11 @@ crane transform # Creates: 10_KubernetesPlugin/ # 3. Add OpenShift-specific transformations -crane transform --stage 20_OpenshiftPlugin +crane transform 20_OpenshiftPlugin # Creates: 20_OpenshiftPlugin/ using OpenshiftPlugin # 4. Add a manual customization stage -crane transform --stage 50_CustomLabels +crane transform 50_CustomLabels # Creates: 50_CustomLabels/ as pass-through (resources copied from previous stage) # 5. Manually add custom labels @@ -483,7 +496,7 @@ crane transform --force git diff transform/ # Option 3: Only regenerate plugin stages (they auto-regenerate) -crane transform --stage 10_KubernetesPlugin +crane transform 10_KubernetesPlugin # Note: Plugin stages (ending with "Plugin") regenerate automatically without --force # Custom stages (not ending with "Plugin") require --force to protect manual edits @@ -496,7 +509,7 @@ crane transform --stage 10_KubernetesPlugin **Solution**: ```bash # Validate by applying a single stage -crane apply --stage 10_KubernetesPlugin +crane apply 10_KubernetesPlugin # Check for missing files ls -la transform/10_KubernetesPlugin/resources/ diff --git a/docs/commands/validate.md b/docs/commands/validate.md index bb356cb7..5b7f6cca 100644 --- a/docs/commands/validate.md +++ b/docs/commands/validate.md @@ -18,7 +18,7 @@ Incompatible resources are written to a `failures/` directory under the validate ### Offline Validation -Use `--api-resources` to validate offline against a captured API surface JSON file (produced by `scripts/capture-api-surface.sh`) when the target cluster is not directly reachable. This is mutually exclusive with `--context`/`--kubeconfig`. +Use `--api-resources` to validate offline against a captured API surface JSON file (produced by `scripts/capture-api-surface.sh`) when the target cluster is not directly reachable. This is mutually exclusive with `--context`, `--kubeconfig`, `--server`, `--token`, `--cluster`, and `--user`. ## Flags @@ -27,7 +27,8 @@ Use `--api-resources` to validate offline against a captured API surface JSON fi | `--input-dir` | `-i` | `output` | Path to the apply output directory containing final manifests | | `--validate-dir` | | `validate` | Path where validation results and failures are saved | | `--output` | `-o` | `json` | Report file format: `json` or `yaml` | -| `--api-resources` | | | Path to API surface JSON file for offline validation (mutually exclusive with `--context`/`--kubeconfig`) | +| `--api-resources` | | | Path to API surface JSON file for offline validation (mutually exclusive with `--context`/`--kubeconfig`/`--server`/`--token`/`--cluster`/`--user`) | +| `--overwrite` | | `false` | Overwrite the validate directory if it already exists | Standard kubeconfig flags (`--kubeconfig`, `--context`, `--cluster`, etc.) are also available to specify the target cluster for live validation. @@ -94,7 +95,8 @@ crane validate --context target-cluster |-------|-------|----------| | `input-dir "X" is not a directory` | Path doesn't exist or isn't a directory | Run `crane apply` first to generate output | | `loading kubeconfig` | Cannot connect to target cluster | Check kubeconfig and `--context` flag, or use `--api-resources` for offline mode | -| `--api-resources and --context are mutually exclusive` | Both offline and live flags specified | Use one mode or the other | +| `--api-resources and --context are mutually exclusive` | Both offline and live flags specified | Use one mode or the other. `--api-resources` is also mutually exclusive with `--kubeconfig`, `--server`, `--token`, `--cluster`, and `--user` | +| `validate directory "X" already exists` | Validate directory from a previous run | Use `--overwrite` to replace it | | Validation failures | GVKs not available on target cluster | Install required CRDs/operators on target, or transform manifests to use supported API versions | ## Next Steps diff --git a/docs/development/plugin-development.md b/docs/development/plugin-development.md index 395b942d..6a6d79f1 100644 --- a/docs/development/plugin-development.md +++ b/docs/development/plugin-development.md @@ -81,7 +81,12 @@ func main() { var patches []PatchOp - // Example: add a migration label + // Example: add a migration label (ensure parent object exists first) + patches = append(patches, PatchOp{ + Op: "add", + Path: "/metadata/labels", + Value: map[string]interface{}{}, + }) patches = append(patches, PatchOp{ Op: "add", Path: "/metadata/labels/migrated-by", @@ -90,11 +95,13 @@ func main() { // Example: remove a specific annotation annotations := resource.GetAnnotations() - if _, ok := annotations["source-cluster-only"]; ok { - patches = append(patches, PatchOp{ - Op: "remove", - Path: "/metadata/annotations/source-cluster-only", - }) + if annotations != nil { + if _, ok := annotations["source-cluster-only"]; ok { + patches = append(patches, PatchOp{ + Op: "remove", + Path: "/metadata/annotations/source-cluster-only", + }) + } } if err := json.NewEncoder(os.Stdout).Encode(patches); err != nil { @@ -115,9 +122,11 @@ go build -o ~/.local/share/crane/plugins/MyCustomPlugin ```bash #!/bin/bash # Simple plugin that adds a label to all resources +# Note: ensure /metadata/labels exists before adding child keys cat <` to isolate the failing stage +3. Run `crane apply ` to isolate the failing stage ### Issue: Resources not appearing in output @@ -479,9 +474,9 @@ Stages don't have to correspond to a plugin. If a stage directory name doesn't m ```bash # Create a multi-stage pipeline -crane transform --stage-name 10_KubernetesPlugin # Plugin-backed -crane transform --stage-name 50_ManualEdits # No matching plugin -crane transform --stage-name 90_FinalCleanup # No matching plugin +crane transform 10_KubernetesPlugin # Plugin-backed +crane transform 50_ManualEdits # No matching plugin +crane transform 90_FinalCleanup # No matching plugin ``` **What happens:** diff --git a/docs/plugins.md b/docs/plugins.md index a14b22b7..db8d4c43 100644 --- a/docs/plugins.md +++ b/docs/plugins.md @@ -22,6 +22,8 @@ This plugin runs as the default first stage (`10_KubernetesPlugin`) and is alway The [konveyor/crane-plugins](https://github.com/konveyor/crane-plugins) repository contains community-contributed plugins based on experience from real-world Kubernetes migrations. +> **Note:** Community plugins (e.g., OpenshiftPlugin) are not actively maintained. Verify plugin compatibility before use in production. + ### OpenShift Plugin The [konveyor/crane-plugin-openshift](https://github.com/konveyor/crane-plugin-openshift) plugin handles OpenShift-specific migration concerns: @@ -42,7 +44,7 @@ chmod +x ~/.local/share/crane/plugins/OpenshiftPlugin Use in a transform stage: ```bash -crane transform --stage 20_OpenshiftPlugin +crane transform 20_OpenshiftPlugin ``` ## Managing Plugins @@ -63,7 +65,7 @@ crane transform list-plugins ### Skipping Plugins -Skip specific plugins during transform: +By default, `crane transform` creates stages for all available plugins. Skip specific plugins during transform: ```bash crane transform --skip-plugins OpenshiftPlugin From 63622e7b85544f29b595dce758386437d435d598 Mon Sep 17 00:00:00 2001 From: Ilanit Stein Date: Wed, 17 Jun 2026 12:49:55 +0300 Subject: [PATCH 3/5] docs: fix review issues, add API surface script, document non-admin migration - Fix stale ApplyFinalStage() reference in multistage-pipeline.md (only ApplySingleStage and ApplyMultiStage exist) - Fix misleading "Single Stage Mode" heading (default creates stages for all plugins) - Fix transform.md title to "crane transform" for consistency - Add capture-api-surface.sh script to validate.md offline validation section (from PR #321 suggestion) - Document admin vs non-admin migration in export.md (graceful Forbidden handling, CRD skip behavior) - Enhance --skip-cluster-scoped docs in apply.md with non-admin pipeline example Co-Authored-By: Claude Opus 4.6 --- docs/commands/apply.md | 13 ++++++++++++- docs/commands/export.md | 14 +++++++++++++ docs/commands/transform.md | 8 ++++++-- docs/commands/validate.md | 39 ++++++++++++++++++++++++++++++++++++- docs/multistage-pipeline.md | 8 ++++---- 5 files changed, 74 insertions(+), 8 deletions(-) diff --git a/docs/commands/apply.md b/docs/commands/apply.md index 7ae9acad..88b5721a 100644 --- a/docs/commands/apply.md +++ b/docs/commands/apply.md @@ -66,12 +66,23 @@ crane apply --transform-dir ./migration/transform --output-dir ./migration/outpu crane apply 10_KubernetesPlugin ``` -### Skip cluster-scoped resources +### Skip cluster-scoped resources (non-admin migration) + +When migrating without cluster-admin privileges, use `--skip-cluster-scoped` to exclude ClusterRoles, ClusterRoleBindings, CRDs, and other cluster-scoped resources from the output. These resources typically require admin permissions to create on the target cluster. ```bash crane apply --skip-cluster-scoped ``` +For a complete non-admin migration pipeline: + +```bash +crane export -n my-app # Skips inaccessible resources gracefully +crane transform +crane apply --skip-cluster-scoped # Excludes cluster-scoped resources +crane validate --context target-cluster # Or use --api-resources for offline +``` + ### Pass additional kustomize arguments ```bash diff --git a/docs/commands/export.md b/docs/commands/export.md index d065a730..3ff01b12 100644 --- a/docs/commands/export.md +++ b/docs/commands/export.md @@ -56,6 +56,20 @@ export/ Resource filenames follow the format: `Kind_group_version_namespace_name.yaml` +### Admin vs Non-Admin Migration + +Crane supports migration for both cluster-admin and non-admin users: + +**Admin users** have full access and can export all resource types including cluster-scoped resources (ClusterRoles, ClusterRoleBindings, CRDs, etc.). + +**Non-admin users** can still use Crane with reduced permissions: +- Export gracefully handles `Forbidden` errors — resources the user cannot list are skipped with a warning, and export continues with accessible resources +- If the user cannot verify the namespace exists (no `get namespaces` permission), Crane logs a warning and proceeds +- CRD collection skips CRDs the user cannot read, with a warning that they should already exist on the target cluster +- Export only exits with a non-zero code if **all** resource types return Forbidden, indicating the user has no list permissions at all + +For the full non-admin pipeline, pair export with `crane apply --skip-cluster-scoped` (see [crane apply](./apply.md)). + ## Examples ### Basic namespace export diff --git a/docs/commands/transform.md b/docs/commands/transform.md index 7255c957..7e0e4ce9 100644 --- a/docs/commands/transform.md +++ b/docs/commands/transform.md @@ -1,6 +1,10 @@ -# Crane Transform Directory Structure +# crane transform -This document explains the structure of the `transform/` directory created by Crane's multi-stage Kustomize pipeline. +Transform exported resources using plugins and produce Kustomize stages. + +## Overview + +This document explains the `crane transform` command and the structure of the `transform/` directory it creates. ## Quick Start diff --git a/docs/commands/validate.md b/docs/commands/validate.md index 5b7f6cca..31abf431 100644 --- a/docs/commands/validate.md +++ b/docs/commands/validate.md @@ -18,7 +18,44 @@ Incompatible resources are written to a `failures/` directory under the validate ### Offline Validation -Use `--api-resources` to validate offline against a captured API surface JSON file (produced by `scripts/capture-api-surface.sh`) when the target cluster is not directly reachable. This is mutually exclusive with `--context`, `--kubeconfig`, `--server`, `--token`, `--cluster`, and `--user`. +Use `--api-resources` to validate offline against a captured API surface JSON file when the target cluster is not directly reachable. This is mutually exclusive with `--context`, `--kubeconfig`, `--server`, `--token`, `--cluster`, and `--user`. + +#### Capturing the API Surface + +Run the following script against the target cluster to capture its API surface for offline validation: + +```bash +#!/bin/bash +# capture-api-surface.sh +# Usage: capture-api-surface.sh [-o output.json] [--context name] [--kubeconfig path] +OUTPUT="api-surface.json" +KUBECTL_FLAGS="" + +while [[ $# -gt 0 ]]; do + case "$1" in + -o) OUTPUT="$2"; shift 2 ;; + --context|--kubeconfig) KUBECTL_FLAGS="$KUBECTL_FLAGS $1=$2"; shift 2 ;; + *) echo "Unknown flag: $1"; exit 1 ;; + esac +done + +kubectl api-versions $KUBECTL_FLAGS | while read gv; do + endpoint=$([ "$gv" = "v1" ] && echo "/api/v1" || echo "/apis/$gv") + kubectl get --raw "$endpoint" $KUBECTL_FLAGS 2>/dev/null || true +done | jq -s '{"apiResourceLists":.}' > "$OUTPUT" +``` + +Or as a one-liner: + +```bash +kubectl api-versions | while read gv; do kubectl get --raw $([ "$gv" = "v1" ] && echo "/api/v1" || echo "/apis/$gv") 2>/dev/null || true; done | jq -s '{"apiResourceLists":.}' > api-surface.json +``` + +Then use the captured file for offline validation: + +```bash +crane validate --api-resources api-surface.json +``` ## Flags diff --git a/docs/multistage-pipeline.md b/docs/multistage-pipeline.md index fd3e07c5..e10031be 100644 --- a/docs/multistage-pipeline.md +++ b/docs/multistage-pipeline.md @@ -104,9 +104,9 @@ This approach ensures: ### Transform Command -#### Single Stage Mode (Default) +#### Default Mode (All Plugins) -Create a single transform stage using all available plugins: +Create transform stages for all available plugins: ```bash crane transform \ @@ -454,8 +454,8 @@ selector := transform.StageSelector{ } err = applier.ApplyMultiStage(selector) -// Apply only the final stage (most common) -err = applier.ApplyFinalStage() +// Apply all stages sequentially +err = applier.ApplyMultiStage(transform.StageSelector{}) ``` ## Further Reading From efc3c7e4ba93c0ca887e67c1e4cc053878b371a8 Mon Sep 17 00:00:00 2001 From: Ilanit Stein Date: Sun, 21 Jun 2026 12:00:21 +0300 Subject: [PATCH 4/5] docs: update for directory restructure (#555) and --ordered flag (#525) - Rename resources/ to input/ in all transform stage references - Remove .work/ directory references, output/ now lives inside each stage - Update .gitignore recommendations (transform/*/output/ replaces transform/.work/) - Add --ordered flag documentation to crane apply command reference Co-Authored-By: Claude Opus 4.6 --- docs/commands/apply.md | 24 ++++++++ docs/commands/transform.md | 99 ++++++++++++++++---------------- docs/development/architecture.md | 4 +- docs/multistage-pipeline.md | 26 ++++----- 4 files changed, 88 insertions(+), 65 deletions(-) diff --git a/docs/commands/apply.md b/docs/commands/apply.md index 88b5721a..e3f42a0d 100644 --- a/docs/commands/apply.md +++ b/docs/commands/apply.md @@ -24,6 +24,7 @@ Kustomize is embedded directly in the Crane binary (via the krusty API), so no e | `--kustomize-args` | | | Additional arguments for kustomize (e.g., `--enable-helm --helm-command=helm3`) | | `--skip-cluster-scoped` | | `false` | Exclude cluster-scoped resources (ClusterRole, ClusterRoleBinding, CRD, etc.) from output. Useful for non-admin migration scenarios | | `--overwrite` | | `false` | Overwrite the output directory if it already exists | +| `--ordered` | | `false` | Prefix resource filenames with a numeric order (e.g., `300_Role_`, `310_RoleBinding_`) so that `kubectl apply -f` processes dependencies before dependents. Useful when first apply fails due to missing referenced resources | Stages are specified as positional arguments (e.g., `crane apply 10_KubernetesPlugin`). Stages can be specified by directory name or plugin name. If no stages are specified, all discovered stages are applied sequentially. @@ -46,6 +47,22 @@ output/ - **`resources//`** — Individual files for selective review or application - **`resources/_cluster/`** — Cluster-scoped resources (omitted when `--skip-cluster-scoped` is set) +### Ordered Output (`--ordered`) + +When `--ordered` is set, individual resource files in `output/resources/` are prefixed with a 3-digit order number based on resource kind. This ensures that `kubectl apply -f output/resources//` processes resources in dependency order (e.g., Role before RoleBinding, ConfigMap before Deployment). + +```text +output/ +└── resources/ + └── / + ├── 240_ConfigMap__v1__.yaml + ├── 300_Role_rbac.authorization.k8s.io_v1__.yaml + ├── 310_RoleBinding_rbac.authorization.k8s.io_v1__.yaml + └── 340_Deployment_apps_v1__.yaml +``` + +Without `--ordered` (default), filenames have no prefix and `kubectl apply -f` processes them alphabetically, which can fail when a RoleBinding is applied before its referenced Role exists. + ## Examples ### Apply all stages (default) @@ -89,6 +106,13 @@ crane validate --context target-cluster # Or use --api-resources for offline crane apply --kustomize-args "--enable-helm --helm-command=helm3" ``` +### Apply with dependency-ordered filenames + +```bash +crane apply --ordered +kubectl apply -f output/resources/default/ +``` + ### Deploy to target cluster ```bash diff --git a/docs/commands/transform.md b/docs/commands/transform.md index 7e0e4ce9..990807f8 100644 --- a/docs/commands/transform.md +++ b/docs/commands/transform.md @@ -13,20 +13,24 @@ After running `crane transform`, you'll see: ```text transform/ └── 10_KubernetesPlugin/ - ├── resources/ + ├── input/ │ ├── ConfigMap__v1_default_nginx-config.yaml │ ├── Deployment_apps_v1_default_wordpress.yaml │ └── Service__v1_default_kubernetes.yaml ├── patches/ │ └── default--apps-v1--Deployment--wordpress.patch.yaml + ├── output/ + │ └── default/ + │ └── Deployment_apps_v1_default_wordpress.yaml └── kustomization.yaml ``` ### What's in Each Directory? -- **`resources/`**: Individual Kubernetes manifest files, one per resource +- **`input/`**: Input resources for this stage (flat files) - **`patches/`**: Kustomize patches to apply to resources -- **`kustomization.yaml`**: Kustomize configuration file +- **`output/`**: Materialized output after applying kustomization (organized by namespace) +- **`kustomization.yaml`**: Kustomize configuration file (references `input/`) ## Working with Transform Output @@ -55,11 +59,11 @@ kubectl apply -k transform/10_KubernetesPlugin/ ### Making Manual Changes -You can edit resources in the `resources/` directory: +You can edit resources in the `input/` directory: ```bash # Edit a deployment -vi transform/10_KubernetesPlugin/resources/Deployment_apps_v1_default_wordpress.yaml +vi transform/10_KubernetesPlugin/input/Deployment_apps_v1_default_wordpress.yaml # Preview changes crane apply 10_KubernetesPlugin @@ -96,36 +100,31 @@ This ensures that: - Resources marked as whiteout (deleted) don't appear in subsequent stages - Each stage sees the actual state of resources, not just patch instructions -### Working Directory Structure +### Multi-Stage Directory Structure -When running multistage transforms, Crane creates a working directory structure for debugging: +When running multistage transforms, each stage contains its complete transformation pipeline: ```text transform/ -├── 10_KubernetesPlugin/ # Stage 1 transform artifacts -│ ├── resources/ -│ ├── patches/ +├── 10_KubernetesPlugin/ # Stage 1 +│ ├── input/ # Input resources (flat files) +│ ├── patches/ # Transformations +│ ├── output/ # Materialized output (organized by namespace) │ └── kustomization.yaml -├── 20_OpenshiftPlugin/ # Stage 2 transform artifacts -│ ├── resources/ -│ ├── patches/ -│ └── kustomization.yaml -└── .work/ # Intermediate working artifacts - ├── 10_KubernetesPlugin/ - │ ├── input/ # Stage 1 input snapshot (from export) - │ └── output/ # Stage 1 materialized output - └── 20_OpenshiftPlugin/ - ├── input/ # Stage 2 input snapshot (Stage 1 output) - └── output/ # Stage 2 materialized output +└── 20_OpenshiftPlugin/ # Stage 2 + ├── input/ # Input (from Stage 1 output) + ├── patches/ # Transformations + ├── output/ # Materialized output + └── kustomization.yaml ``` -The `.work/` directory contains intermediate snapshots useful for debugging multi-stage pipelines. +Each stage's `output/` becomes the next stage's `input/`, creating a sequential pipeline. -**Important**: The `.work/` directory contains intermediate snapshots that are regenerated on each transform run. These are useful for debugging but should not be committed to version control. Add to your `.gitignore`: +**Note**: `output/` directories are regenerated on each transform run and can be added to `.gitignore` if desired: ```gitignore -# Crane intermediate artifacts (regenerated on each run) -transform/.work/ +# Crane stage output directories (regenerated on each run) +transform/*/output/ ``` ### Running Multi-Stage Transforms @@ -218,7 +217,7 @@ Stage names ending with `Plugin` **must** have a corresponding plugin installed. crane transform 20_OpenshiftPlugin # Output: -# - resources/ (from previous stage or export) +# - input/ (from previous stage or export) # - patches/ (generated by OpenshiftPlugin) # - kustomization.yaml ``` @@ -240,7 +239,7 @@ Stage names **not** ending with `Plugin` create pass-through stages where resour crane transform 30_CustomEdits # Output: -# - resources/ (copied unchanged from previous stage) +# - input/ (copied unchanged from previous stage) # - patches/ (empty - ready for your custom patches) # - kustomization.yaml ``` @@ -264,15 +263,15 @@ crane transform --force **Important**: Custom stages (not ending with `Plugin`) are protected from accidental overwrites. You must use `--force` to regenerate them. **Best Practice**: Always add custom stages as the **last stage** in your pipeline. This ensures that: -- The `resources/` directory contains the most up-to-date output from all previous stages +- The `input/` directory contains the most up-to-date output from all previous stages - You're editing the final state of resources after all plugin transformations - Re-running earlier plugin stages won't leave your custom stage with stale data -If you add a custom stage in the middle of the pipeline and later update an earlier stage, you'll need to use `--force` to refresh the custom stage's `resources/` directory with the updated output. **Warning**: Using `--force` will delete the entire stage directory including any manual changes you made to `resources/`, `patches/`, and `kustomization.yaml`. +If you add a custom stage in the middle of the pipeline and later update an earlier stage, you'll need to use `--force` to refresh the custom stage's `input/` directory with the updated output. **Warning**: Using `--force` will delete the entire stage directory including any manual changes you made to `input/`, `patches/`, and `kustomization.yaml`. ## Directory Contents Explained -### resources/ +### input/ Contains individual Kubernetes manifest files, one per resource: @@ -318,13 +317,13 @@ patches: namespace: default version: v1 resources: -- resources/ConfigMap__v1_default_nginx-config.yaml -- resources/Deployment_apps_v1_default_wordpress.yaml -- resources/Service__v1_default_kubernetes.yaml +- input/ConfigMap__v1_default_nginx-config.yaml +- input/Deployment_apps_v1_default_wordpress.yaml +- input/Service__v1_default_kubernetes.yaml -# Whiteout resources are written to resources/ for complete snapshot +# Whiteout resources are written to input/ for complete snapshot # but excluded from active resources list above: -# - resources/Pod__v1_default_wordpress-74b89cc84c-nm9f8.yaml +# - input/Pod__v1_default_wordpress-74b89cc84c-nm9f8.yaml ``` @@ -354,7 +353,7 @@ kubectl kustomize transform/10_KubernetesPlugin/ crane transform 40_CustomEdits # Edit resources -vi transform/40_CustomEdits/resources/Deployment_apps_v1_default_wordpress.yaml +vi transform/40_CustomEdits/input/Deployment_apps_v1_default_wordpress.yaml # Add custom patches cat > transform/40_CustomEdits/patches/custom-patch.yaml <_` pattern 2. **Plugin execution** — For plugin-based stages (name ending in `Plugin`), loads and runs the matching plugin to generate JSONPatch operations -3. **Resource writing** — Writes individual resource files to `/resources/` +3. **Resource writing** — Writes individual resource files to `/input/` 4. **Patch writing** — Writes plugin-generated patches to `/patches/` 5. **Kustomization generation** — Generates `kustomization.yaml` linking resources and patches @@ -48,7 +48,7 @@ The transform phase uses a multi-stage Kustomize pipeline: ### Sequential Consistency -In multi-stage pipelines, each stage runs on the fully materialized output of the previous stage (not raw patches). Intermediate artifacts are stored in `transform/.work//input/` and `transform/.work//output/`. +In multi-stage pipelines, each stage runs on the fully materialized output of the previous stage (not raw patches). Each stage contains `input/`, `patches/`, and `output/` directly within the stage directory (e.g., `transform//input/`, `transform//output/`). ### Key Components diff --git a/docs/multistage-pipeline.md b/docs/multistage-pipeline.md index e10031be..2f767d4c 100644 --- a/docs/multistage-pipeline.md +++ b/docs/multistage-pipeline.md @@ -15,16 +15,15 @@ Each stage is a directory following the naming convention `_` to isolate the failing stage ### Issue: Resources not appearing in output @@ -482,7 +482,7 @@ crane transform 90_FinalCleanup # No matching plugin **What happens:** 1. **10_KubernetesPlugin**: Resources transformed by KubernetesPlugin (removes metadata.uid, etc.) -2. **50_ManualEdits**: Resources copied unchanged to `transform/50_ManualEdits/resources/` +2. **50_ManualEdits**: Resources copied unchanged to `transform/50_ManualEdits/input/` - No plugins match "ManualEdits" - No patches generated - You can manually edit resources in this stage @@ -506,13 +506,13 @@ export/ └── deployment.yaml (raw export with uid, resourceVersion, etc.) ↓ transform/10_KubernetesPlugin/ (plugin: removes server-managed fields) -└── resources/deployment.yaml (cleaned) +└── input/deployment.yaml (cleaned) ↓ transform/50_ManualEdits/ (no plugin: user edits) -└── resources/deployment.yaml (manually edited) +└── input/deployment.yaml (manually edited) ↓ transform/90_CustomPlugin/ (plugin: custom transformations) -└── resources/deployment.yaml (custom patches applied) +└── input/deployment.yaml (custom patches applied) ↓ output/output.yaml (final result) ``` From cd3c0b8f4f5fea791c253dab0d6a46c37d7960b6 Mon Sep 17 00:00:00 2001 From: Ilanit Stein Date: Tue, 23 Jun 2026 18:56:18 +0300 Subject: [PATCH 5/5] docs: clarify admin and non-admin support in resource compatibility matrix Co-Authored-By: Claude Opus 4.6 --- docs/resource-compatibility.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/resource-compatibility.md b/docs/resource-compatibility.md index e9933574..78dbf06e 100644 --- a/docs/resource-compatibility.md +++ b/docs/resource-compatibility.md @@ -1,6 +1,6 @@ # Crane Compatibility Matrix: Namespace vs. Cluster Resources -> **Note to Users:** Crane is primarily a namespace-scoped migration tool. However, it can migrate cluster-scoped resources if they are explicitly related to the namespace workload and the migration context has sufficient RBAC permissions. +> **Note to Users:** Crane supports migration for both admin and non-admin users. Admin users can export all resources including cluster-scoped types. Non-admin users can migrate namespace-scoped resources — Crane gracefully skips inaccessible cluster-scoped resources with warnings rather than failing. See [crane export](./commands/export.md) and [crane apply](./commands/apply.md) for details. ---