Skip to content

fix(nemesis): handle new keyspace being written to during AddRemoveDCNemesis - #15789

Open
cezarmoise wants to merge 1 commit into
scylladb:masterfrom
cezarmoise:add-dc-rf-all-ks
Open

fix(nemesis): handle new keyspace being written to during AddRemoveDCNemesis#15789
cezarmoise wants to merge 1 commit into
scylladb:masterfrom
cezarmoise:add-dc-rf-all-ks

Conversation

@cezarmoise

Copy link
Copy Markdown
Contributor

If a new keyspace is created and written to by cassandra-stress while there is another DC, because the stress command only specifies replication_factor=3, it will try to write to that DC and get a schema creation failure.

  • change the number of nodes in the new DC to 3 (matches RF in most commands)
  • look for keyspaces from outside the nemesis, and revert their RF if needed.

Closes: SCT-130

Testing

From sct.log, 4 instances where the keyspace created by the second nemesis was altered to have RF=0 on the new DC.

2026-08-14 10:48:48,990 ... Executing CQL 'ALTER KEYSPACE system_distributed WITH replication = {'class': 'NetworkTopologyStrategy', 'eu-west-1': 3}' ...
2026-08-14 10:49:10,761 ... Executing CQL 'ALTER KEYSPACE system_traces WITH replication = {'class': 'NetworkTopologyStrategy', 'eu-west-1': 2}' ...
2026-08-14 10:49:32,812 ... Executing CQL 'ALTER KEYSPACE system_distributed WITH replication = {'class': 'NetworkTopologyStrategy', 'eu-west-1': 3, 'add_remove_nemesis_dc': 3}' ...
2026-08-14 10:49:55,263 ... Executing CQL 'ALTER KEYSPACE system_traces WITH replication = {'class': 'NetworkTopologyStrategy', 'eu-west-1': 2, 'add_remove_nemesis_dc': 3}' ...
2026-08-14 10:50:02,385 ... Executing CQL 'ALTER KEYSPACE keyspace_new_dc WITH replication = {'class': 'NetworkTopologyStrategy', 'eu-west-1': ['RACK1', 'RACK2', 'RACK0'], 'add_remove_nemesis_dc': 3}' ...
2026-08-14 11:08:59,209 ... Executing CQL 'ALTER KEYSPACE system_distributed WITH replication = {'class': 'NetworkTopologyStrategy', 'eu-west-1': 3, 'add_remove_nemesis_dc': 0}' ...
2026-08-14 11:09:18,289 ... Executing CQL 'ALTER KEYSPACE system_traces WITH replication = {'class': 'NetworkTopologyStrategy', 'eu-west-1': 2, 'add_remove_nemesis_dc': 0}' ...
2026-08-14 11:09:23,177 ... Executing CQL 'ALTER KEYSPACE keyspace_new_dc WITH replication = {'class': 'NetworkTopologyStrategy', 'eu-west-1': ['RACK1', 'RACK2', 'RACK0'], 'add_remove_nemesis_dc': 0}' ...
2026-08-14 11:10:48,728 ... Executing CQL 'ALTER KEYSPACE "keyspace_stress_write_SFM9ZJ8D" WITH replication = {'class': 'NetworkTopologyStrategy', 'add_remove_nemesis_dc': 0, 'eu-west-1': ['RACK1', 'RACK2', 'RACK0']}' ...
2026-08-14 11:11:56,717 ... Executing CQL 'ALTER KEYSPACE "keyspace_stress_write_F5EDKP0N" WITH replication = {'class': 'NetworkTopologyStrategy', 'add_remove_nemesis_dc': 0, 'eu-west-1': ['RACK1', 'RACK2', 'RACK0']}' ...
2026-08-14 11:19:49,829 ... Executing CQL 'ALTER KEYSPACE system_distributed WITH replication = {'class': 'SimpleStrategy', 'replication_factor': 3}' ...
2026-08-14 11:19:53,403 ... Executing CQL 'ALTER KEYSPACE system_traces WITH replication = {'class': 'SimpleStrategy', 'replication_factor': 2}' ...
2026-08-14 11:30:39,678 ... Executing CQL 'ALTER KEYSPACE system_distributed WITH replication = {'class': 'NetworkTopologyStrategy', 'eu-west-1': 3}' ...
2026-08-14 11:31:00,542 ... Executing CQL 'ALTER KEYSPACE system_traces WITH replication = {'class': 'NetworkTopologyStrategy', 'eu-west-1': 2}' ...
2026-08-14 11:31:07,602 ... Executing CQL 'ALTER KEYSPACE system_distributed WITH replication = {'class': 'NetworkTopologyStrategy', 'eu-west-1': 3, 'add_remove_nemesis_dc': 3}' ...
2026-08-14 11:31:30,574 ... Executing CQL 'ALTER KEYSPACE system_traces WITH replication = {'class': 'NetworkTopologyStrategy', 'eu-west-1': 2, 'add_remove_nemesis_dc': 3}' ...
2026-08-14 11:31:38,952 ... Executing CQL 'ALTER KEYSPACE keyspace_new_dc WITH replication = {'class': 'NetworkTopologyStrategy', 'eu-west-1': ['RACK1', 'RACK2', 'RACK0'], 'add_remove_nemesis_dc': 3}' ...
2026-08-14 11:48:51,538 ... Executing CQL 'ALTER KEYSPACE system_distributed WITH replication = {'class': 'NetworkTopologyStrategy', 'eu-west-1': 3, 'add_remove_nemesis_dc': 0}' ...
2026-08-14 11:49:11,815 ... Executing CQL 'ALTER KEYSPACE system_traces WITH replication = {'class': 'NetworkTopologyStrategy', 'eu-west-1': 2, 'add_remove_nemesis_dc': 0}' ...
2026-08-14 11:49:30,158 ... Executing CQL 'ALTER KEYSPACE keyspace_new_dc WITH replication = {'class': 'NetworkTopologyStrategy', 'eu-west-1': ['RACK1', 'RACK2', 'RACK0'], 'add_remove_nemesis_dc': 0}' ...
2026-08-14 11:50:08,799 ... Executing CQL 'ALTER KEYSPACE "keyspace_stress_write_S6CZ3IP8" WITH replication = {'class': 'NetworkTopologyStrategy', 'add_remove_nemesis_dc': 0, 'eu-west-1': ['RACK0', 'RACK2', 'RACK1']}' ...
2026-08-14 11:50:42,517 ... Executing CQL 'ALTER KEYSPACE "keyspace_stress_write_UC3DSMMF" WITH replication = {'class': 'NetworkTopologyStrategy', 'add_remove_nemesis_dc': 0, 'eu-west-1': ['RACK1', 'RACK2', 'RACK0']}' ...
2026-08-14 11:51:13,574 ... Executing CQL 'ALTER KEYSPACE system_distributed WITH replication = {'class': 'SimpleStrategy', 'replication_factor': 3}' ...
2026-08-14 11:51:17,417 ... Executing CQL 'ALTER KEYSPACE system_traces WITH replication = {'class': 'SimpleStrategy', 'replication_factor': 2}' ...

PR pre-checks (self review)

  • I added the relevant backport labels
  • I didn't leave commented-out/debugging code

Reminders

  • Add New configuration option and document them (in sdcm/sct_config.py)
  • Add unit tests to cover my changes (under unit-test/ folder)
  • Update the Readme/doc folder relevant to this change (if needed)

…Nemesis

If a new keyspace is created and written to by cassandra-stress while
there is another DC, it because the stress command only specifies
`replication_factor=3`, it will try to  write to that DC and get a
schema creation failure.

- change the number of nodes in the new DC to 3 (matches RF in most commands)
- look for keyspaces from outside the nemesis, and revert their RF if needed.
@scylladb-promoter

Copy link
Copy Markdown
Collaborator

✅ Test Summary: PASSED

✅ Precommit: PASSED

Total Passed Failed Skipped
28 16 0 12

✅ Tests: PASSED

Total Passed Failed Errors Skipped
4790 4759 0 0 31

Full build log

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants