feature(s3_object_storage): configure rolling upgrade test to use S3-backed keyspaces

[jsmolar]

• Pre-create all keyspaces used by rolling-upgrade.yaml with S3 storage via pre_create_keyspace config param
• Call _pre_create_keyspace() in test_rolling_upgrade before prepare_keyspaces_and_tables() so S3 keyspaces exist before any hardcoded CREATE KEYSPACE IF NOT EXISTS runs
• Add S3 storage clause to hardcoded CREATE KEYSPACE statements in fill_db_data.prepare_keyspaces_and_tables for scylla_bench and keyspace_fill_db_data

Testing

• [ ]

PR pre-checks (self review)

• I added the relevant backport labels
• I didn't leave commented-out/debugging code

Reminders

• Add New configuration option and document them (in sdcm/sct_config.py)
• Add unit tests to cover my changes (under unit-test/ folder)
• Update the Readme/doc folder relevant to this change (if needed)

[scylladb-promoter]

✅ Test Summary: PASSED

✅ Precommit: PASSED

Total	Passed	Failed	Skipped
28	16	0	12

✅ Tests: PASSED

Total	Passed	Failed	Errors	Skipped
4486	4455	0	0	31

Full build log

[pehala]

This is obviously single purpose and as it stands it cannot be merged. What is the plan on how the final implementation should look?

[jsmolar]

This is obviously single purpose and as it stands it cannot be merged. What is the plan on how the final implementation should look?

1. Pre-created keyspaces can be easily moved to the config file. - Yes
2. Snapshots are not supported - Partially, this requires a change in the test function to ignore snapshots when configured.