Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 11 additions & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -210,11 +210,21 @@ The `sdcm/` directory is the heart of the SCT framework. Here's a detailed break
Nemesis are chaos operations that test database resilience. For a comprehensive guide, see [docs/nemesis.md](docs/nemesis.md).

**Architecture:**
- `NemesisBaseClass` — Abstract base for individual disruptions (a.k.a. "Monkeys"). Each subclass sets boolean flags and implements `disrupt()`.
- `NemesisBaseClass` — Abstract base for individual disruptions (a.k.a. "Monkeys"). Each subclass sets boolean flags, implements `disrupt()`, and optionally `precheck()`.
- `NemesisRunner` — Orchestrator that contains all `disrupt_*` methods (the actual disruption logic), handles node selection, metrics, and error reporting.
- `NemesisRegistry` — Discovery mechanism that filters nemesis using boolean flag expressions (e.g. `"not disruptive"`, `"topology_changes and not limited"`).
- `NemesisNodeAllocator` — Thread-safe singleton preventing conflicting nemesis on the same node.

**Static skip checks (`precheck()`):**

`NemesisBaseClass.precheck(node) -> str | None` is evaluated **once** before the nemesis execution loop starts. Return `None` to keep the nemesis in the rotation; return a string (the skip reason) to permanently exclude it. The `node` argument is a representative live node for static version, feature-flag, and cluster-uniform attribute checks. Use this for static conditions that do not change during the test:
- Test config / backend / product edition (`cluster.params.get(...)`, `_is_it_on_kubernetes()`, `node.is_enterprise`)
- Scylla version / feature flags / cluster-uniform node attributes (`ComparableScyllaVersion`, `is_tablets_feature_enabled()`, `node.distro`)

**Do not** use `precheck(node)` for dynamic state (data presence, live node counts, target-node state) — those belong in `disrupt()`.

A pruned nemesis emits exactly one `SKIPPED` Argus row at precheck time. If every selected nemesis is pruned, one `CRITICAL` event is published and the test fails.

**Common nemesis categories:**
- Node operations (stop/start, reboot, terminate, decommission)
- Network disruptions (block, delay, partition)
Expand Down
80 changes: 78 additions & 2 deletions docs/nemesis.md
Original file line number Diff line number Diff line change
Expand Up @@ -224,7 +224,82 @@ class MyNewMonkey(NemesisBaseClass):
```


### Step 2: Implement the disruption logic
### Step 2: Add a `precheck()` for static skip conditions (optional)

If your nemesis has conditions that make it permanently infeasible on certain
backends, configs, or Scylla versions, implement `precheck()` instead of
raising `UnsupportedNemesis` inside `disrupt()`.

`precheck(node)` is called **once** before the execution loop starts, so a
nemesis excluded here costs zero health-check, node-selection, or Argus overhead
on every subsequent cycle. The `node` argument is a representative live node for
static version, feature-flag, and cluster-uniform attribute checks.

```python
class MyTabletNemesis(NemesisBaseClass):
def precheck(self, node) -> str | None:
# Static config / backend condition — known before the test starts
if self.runner.cluster.params.get("cluster_backend") == "docker":
return "MyTabletNemesis requires a cloud backend"

# Version / feature flag — uniform across the cluster, checked via a representative node
if not node.is_tablets_feature_enabled():
return "tablets feature not enabled on this cluster"

return None # runnable — keep in the rotation

def disrupt(self):
...
```

#### What belongs in `precheck()` vs `disrupt()`

| Condition type | Where to check |
|---|---|
| Test config / backend / product edition | `precheck()` |
| Scylla version / feature flags / cluster-uniform node attribute (e.g. OS distro) | `precheck()` |
| Dynamic cluster state (data presence, live node counts, target-node busy state) | **`disrupt()` only** |

> **Representative node rule:** `precheck(node)` receives a representative node
> for any cluster-wide probe (version, feature flags, OS distro). It is not the
> selected `target_node`. Do **not** check per-node dynamic state in `precheck()`.

#### What happens when `precheck()` returns a reason

1. The nemesis is permanently removed from `disruptions_list` — it never enters
the execution cycle.
2. Exactly one `DisruptionEvent` marked `SKIPPED` is published at precheck time
(one `NemesisStatus.SKIPPED` row in Argus per excluded nemesis, not per cycle).
3. A warning is logged: `"Nemesis <Name> excluded by precheck: <reason>"`.
4. If **every** selected nemesis is excluded, one `Severity.CRITICAL`
`TestFrameworkEvent` is published naming each exclusion reason, and the
nemesis thread stops cleanly. A misconfigured selector that produces an
empty rotation fails the test loudly.

#### Before / after example

**Before** — static guard inside `disrupt()`, evaluated every cycle:

```python
def disrupt(self):
if not self.runner.cluster.params.get("use_ldap_auth"):
raise UnsupportedNemesis("LDAP not configured")
# ... actual disruption
```

**After** — evaluated once before the execution loop via `precheck(node)`:

```python
def precheck(self, node) -> str | None:
if not self.runner.cluster.params.get("use_ldap_auth"):
return "LDAP not configured"
return None

def disrupt(self):
# ... actual disruption (no static guard needed here)
```

### Step 3: Implement the disruption logic

If your nemesis reuses existing logic, just call the appropriate runner method.
You can reuse method from NemesisRunner, but it is discouraged and you should make the nemesis self-contained
Expand Down Expand Up @@ -312,7 +387,8 @@ from sdcm.nemesis import NemesisRunner
class MyCustomRunner(NemesisRunner):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
# Select all non-disruptive nemesis
# Select all non-disruptive nemesis. precheck(node) is called by run()
# before the execution loop — excluded nemesis are reported and logged once.
self.disruptions_list = self.build_disruptions_by_selector("not disruptive")
self.disruptions_list = self.shuffle_list_of_disruptions(self.disruptions_list)
```
Expand Down
120 changes: 120 additions & 0 deletions docs/plans/MASTER.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,120 @@
# SCT Implementation Plans — Master Index

![Progress Roadmap](assets/progress-roadmap.svg)

This is the central index for all implementation plans in the SCT project.
Plans are grouped by domain and tracked with status metadata.

For plan writing guidelines, see [INSTRUCTIONS.md](INSTRUCTIONS.md).

## Status Legend

| Status | Meaning |
|--------|---------|
| `draft` | Plan written, not yet approved or started |
| `approved` | Plan reviewed and approved for implementation |
| `in_progress` | Active implementation underway |
| `blocked` | Implementation blocked by dependency or issue |
| `complete` | All phases implemented and verified |
| `pending_pr` | Plan exists in an open PR, not yet merged |

## Plans by Domain

### Cluster — Cluster management, node lifecycle, backends

| Plan | Status | File / PR |
|------|--------|-----------|
| Docker Cleanup for All Backends | `draft` | [docker-cleanup-all-backends.md](infrastructure/docker-cleanup-all-backends.md) |
| GCE Provisioning | `draft` | [gce-provisioning.md](infrastructure/gce-provisioning.md) |
| Amazon EMR Spark Migrator | `pending_pr` | [#13909](https://github.com/scylladb/scylla-cluster-tests/pull/13909) |
| Source-Destination Clusters | `pending_pr` | [#13908](https://github.com/scylladb/scylla-cluster-tests/pull/13908) |
| Cassandra Cluster Support | `draft` | [cassandra-cluster-support.md](infrastructure/cassandra-cluster-support.md) |
| SSH Key Decoupling | `draft` | [ssh-key-decoupling.md](infrastructure/ssh-key-decoupling.md) |
| Multi-Cloud Provisioning Resilience | `draft` | [multi-cloud-provisioning-resilience.md](infrastructure/multi-cloud-provisioning-resilience.md) |
| AWS Capacity AZ Fallback | `draft` | [aws-capacity-az-fallback.md](infrastructure/aws-capacity-az-fallback.md) |

### Nemesis — Chaos engineering, disruptors

| Plan | Status | File / PR |
|------|--------|-----------|
| Nemesis Rework (Nemesis 2.0) | `in_progress` | [nemesis-rework.md](nemesis/nemesis-rework.md) |
| Nemesis Extraction Phase 3 | `in_progress` | [nemesis-extraction.md](nemesis/nemesis-extraction.md) |
| Nemesis Pre-Execution Skip Check (`precheck`) | `in_progress` | [nemesis-precheck.md](nemesis/nemesis-precheck.md) |

### Stress Tools — Load generators

| Plan | Status | File / PR |
|------|--------|-----------|

### CI/CD — Jenkins pipelines, Groovy libs

| Plan | Status | File / PR |
|------|--------|-----------|
| Jenkins Pipeline Cluster Reuse | `draft` | [jenkins-reuse-cluster.md](jenkins/jenkins-reuse-cluster.md) |
| Jenkins Pipeline Config Linter | `draft` | [jenkins-pipeline-config-linter.md](jenkins/jenkins-pipeline-config-linter.md) |
| Pipeline Labeling and Documentation | `draft` | [pipeline-labeling-and-documentation.md](jenkins/pipeline-labeling-and-documentation.md) |
| Jenkins Uno-Choice Billing Project | `draft` | [jenkins-uno-choice-billing-project.md](jenkins/jenkins-uno-choice-billing-project.md) |
| Centralized Trigger Matrix | `draft` | [centralized-trigger-matrix.md](jenkins/centralized-trigger-matrix.md) |
| i8g Performance Jobs Migration | `draft` | [i8g-performance-jobs-migration.md](i8g-performance-jobs-migration.md) |
| Perf-Simple-Query Offline Installer Trigger | `complete` | [perf-simple-query-offline-installer-trigger.md](jenkins/perf-simple-query-offline-installer-trigger.md), [#14340](https://github.com/scylladb/scylla-cluster-tests/pull/14340) |

### Config — Configuration system

| Plan | Status | File / PR |
|------|--------|-----------|
| Resilient Test Config Dependencies | `pending_pr` | [#13982](https://github.com/scylladb/scylla-cluster-tests/pull/13982) |
| Typed Config Access Migration | `pending_pr` | [#13878](https://github.com/scylladb/scylla-cluster-tests/pull/13878) |
| SCT Config Validation and Lazy Images | `draft` | [sct-config-validation-and-lazy-images.md](sct-config-validation-and-lazy-images.md) |
| SCT Config Follow-up Refactoring | `pending_pr` | [#13845](https://github.com/scylladb/scylla-cluster-tests/pull/13845) |
| Config Type Normalization | `complete` | [config-type-normalization.md](config/config-type-normalization.md), [#14805](https://github.com/scylladb/scylla-cluster-tests/pull/14805) |
| Constraint-Based Instance Sizing | `complete` | [constraint-based-sizing.md](config/constraint-based-sizing.md), [#14576](https://github.com/scylladb/scylla-cluster-tests/pull/14576) |

### K8s — Kubernetes operator, K8s backends

| Plan | Status | File / PR |
|------|--------|-----------|
| K8s Multitenancy Dict Config | `draft` | [k8s-multitenancy-dict-config.md](config/k8s-multitenancy-dict-config.md) |

### Framework — Core framework internals

| Plan | Status | File / PR |
|------|--------|-----------|
| Health Check Optimization | `draft` | [health-check-optimization.md](infrastructure/health-check-optimization.md) |
| Feature-Aware Adaptive Timeouts for Topology Operations | `draft` | [feature-aware-adaptive-topology-timeouts.md](infrastructure/feature-aware-adaptive-topology-timeouts.md) |
| Full Version Tag Lookup | `draft` | [full-version-tag-lookup.md](config/full-version-tag-lookup.md) |
| Keystore Improvements | `pending_pr` | [#14055](https://github.com/scylladb/scylla-cluster-tests/pull/14055) |

### AI Tooling — AI skills, agent guidance

| Plan | Status | File / PR |
|------|--------|-----------|
| AI Skills Framework | `complete` | [ai-skills-framework.md](ai-tooling/ai-skills-framework.md), [#13799](https://github.com/scylladb/scylla-cluster-tests/pull/13799), [#13827](https://github.com/scylladb/scylla-cluster-tests/pull/13827), [#13836](https://github.com/scylladb/scylla-cluster-tests/pull/13836) |
| PR Review Taxonomy Analysis | `in_progress` | [pr-review-taxonomy-analysis.md](pr-review-taxonomy-analysis.md) |

### Testing — Unit/integration test infrastructure

| Plan | Status | File / PR |
|------|--------|-----------|
| MiniCloud Local Testing | `pending_pr` | [#14009](https://github.com/scylladb/scylla-cluster-tests/pull/14009) |
| Unit/Integration Test Separation | `complete` | [unit-integration-test-separation.md](testing/unit-integration-test-separation.md), [#14172](https://github.com/scylladb/scylla-cluster-tests/pull/14172) |

## Cross-Plan Dependencies

| Dependent Plan | Depends On | Relationship |
|---------------|------------|--------------|
| Nemesis Extraction Phase 3 | Nemesis Rework | Phase 3 continues the extraction started in Nemesis 2.0 |
| Pipeline Labeling and Documentation | Jenkins Pipeline Config Linter | Labeling complements structural config linting; may reuse Jenkinsfile parser |
| SCT Config Follow-up Refactoring | SCT Config Validation and Lazy Images | Follow-up work after initial config validation |
| Typed Config Access Migration | SCT Config Follow-up Refactoring | Type safety layer on top of refactored config |

## Domain Coverage Gaps

The following domains have **no plans** currently:

| Domain | Covers | Codebase Areas |
|--------|--------|----------------|
| `monitoring` | Metrics, dashboards, reporting | `sdcm/monitorstack/`, `sdcm/reporting/` |
| `events` | Event system | `sdcm/sct_events/` |
| `remote` | Remote execution | `sdcm/remote/` |

These gaps are informational — not every domain needs an active plan.
Loading
Loading