Add matrix plan shaper for expanding plans across variable combinations

[NicholasBHubbard]

This PR adds a matrix plan shaper that lets users define variables and their values in a plan and automatically expands them into derived plans via cartesian product. Instead of duplicating near-identical plans that differ only in configuration, users write a single plan with a matrix: key and tmt generates the combinations at runtime. Each derived plan gets TMT_MATRIX_* environment variables, which are available both for variable expansion in plan configuration (e.g., $TMT_MATRIX_DISTRO in a provision step) and directly in test scripts.

I've recently started using tmt, and noticed I'd been writing a lot of repeat tests to test against different configurations. Similar to github actions feature, I thought this could be nicely solved with a tmt matrix feature. This feature makes it much nicer to write tests IMO.

#3873 — Documents the workarounds users currently rely on to parameterize tests. The matrix feature directly addresses this use case.
#3516 — The repeat plan shaper that established the plan shaper pattern this implementation follows.

Please note that I added tests and documentation for this new feature. Let me know of any feedback you have!

[gemini-code-assist]

Warning

Gemini encountered an error creating the review. You can try again by commenting /gemini review.

[LecrisUT]

My general opinion on this is that we should avoid it and instead have tools that can generate the fmf files, but as you noted we do have plan_shapers that seem to cover some similar features.

My reasoning on why to avoid it is to avoid:

• Complex matrix generation strategies, like github action's include/exclude stuff
• Syntax for variable manipulation of env variables/context
• Making sure that the fields that the variables are used are indeed substituted¹
• Secrets stuff and whatnots

An alternative solution is to use something like full-blown jinja and we do not have to worry about missing features since that is not on our side anymore. I have a project to do that with fmf-jinja, and I did have tmt in mind when I considered use-cases, maybe can give some feedback on such an approach as well.

Anyway will put it out for discussion for further eyes.

Edit: also relevant is tmt go suggestion in #3298

Footnotes

1. There were a few cases where this was a non-intuitive pit-fall but I don't remember the reproducers or issues or if things have improved in the meantime ↩

[bajertom]

From hacking session: another issue - how to store plan IDs when they are dynamically created using matrix that splits them based on some parameters.

[kkaarreell]

Hi @NicholasBHubbard , if you are using Testing Farm, consider using https://github.com/RedHatQE/newa/ . I tries to address this issue as a wrapper above the testing-farm command but is also provides additional features related to scheduling, reporting...

[NicholasBHubbard]

Thank you very much for the feedback, I really appreciate it!

I understand the suggestion to generate fmf files using Jinja, and I can see the flexibility that approach provides. My hesitation is that it introduces an additional templating and generation step and would require me to write at least some test metadata as templates rather than native .fmf files. For this use case, I would prefer a native tmt feature if the project is open to that direction.

For context, I am not using Testing Farm, so the suggested Testing Farm wrapper would not apply to my use case. Newa is very interesting though!

Before going further with the implementation, my main questions are:

1. Is native matrix support something the tmt maintainers would be interested in?
2. If so, are plan shapers the right mechanism, or would you prefer it to be implemented another way?

I’m happy to rework the implementation, but I would first like to confirm that this is a direction the project wants to pursue.

[LecrisUT]

From hacking session: we will investigate again the 4 ways to handle these:

• This proposal (native support of matrix, with limitations in what it can support)
• Jinja template backed by pre-commit
• tmt go command proposal
• external tools like newa

We'll probably look into these and update if we would go forward with this approach as a first-party supported approach. (We can always make it available as third-party plugin though 😉)

[psss]

We'll probably look into these and update if we would go forward with this approach as a first-party supported approach.

Here's the spike issue:

• Discuss the environment combinations handling #5096

Everybody feel free to add your comments / brainstorm there pros and cons.