Add generic plugin test runner script

[ehelms]

Plugins like Katello and foreman_puppet have their own wrapper scripts or Rakefile customizations to make it easier to run individual test files without manually switching to the Foreman directory and setting up cross-project load paths. This script generalizes that pattern for all plugins.

Supports running from the Foreman checkout, a plugin checkout, or anywhere on PATH with automatic Foreman/plugin discovery. Single-file runs use direct ruby invocation for faster feedback; full-suite runs delegate to the plugin's existing rake tasks.

I am submitting this as a replacement for ktest to be available for any plugin, and support different paths of usage. And instead of this living inside our development deployment tooling, to live as a script in the main repository and thus allow it's usage from more places.

In the use case of foremanctl, the development deployer would copy the script to the users local bin directory to make it broadly available within the environment.

[adamruzicka]

Every now and then, I need to run an ad-hoc plugin rake task (usually foreman_theme_satellite:validate_docs), could this be extended to be able to run arbitrary rake tasks or would that be out of scope?

[ehelms]

Every now and then, I need to run an ad-hoc plugin rake task (usually foreman_theme_satellite:validate_docs), could this be extended to be able to run arbitrary rake tasks or would that be out of scope?

I think that's fair. Do you think we can count on the pattern always including : to differentiate rake tasks from test files? Or do you think it will need an option like --rake ?

[adamruzicka]

Or do you think it will need an option like --rake ?

I'd prefer this to also cover cases like string extraction where the plugin-name-prefix is not there (it is plugin:gettext[$plugin])

[ehelms]

@adamruzicka updated from your feedback

[adamruzicka]

One last comment, otherwise lgtm

[adamruzicka]

This looks good, please squash the commits

[Copilot]

Pull request overview

Adds a generic, repository-provided wrapper script to run Foreman plugin tests/rake tasks from various working directories by auto-discovering the Foreman checkout and (optionally) the plugin location.

Changes:

• Introduces script/foreman-plugin-test bash script for running plugin test suites, single test files, or arbitrary rake tasks.
• Implements Foreman/plugin discovery logic (Foreman checkout, plugin checkout, PATH usage, direct plugin paths).
• Uses direct bundle exec ruby for single-file runs and delegates full-suite runs to bundle exec rake test:<plugin>.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[ehelms]

Green again -- @jeremylenz you might want to look at this and tell me what you think

[jeremylenz]

Thanks! Looks like an awesome replacement for ktest.

[jeremylenz]

This will be awesome for LLMs because there will be one way to run Ruby tests across the entire ecosystem. Is there a companion foremanctl PR for adding the script to bin?

[ianballou]

I haven't tested it, but from a ktest functionality standpoint I'm happy with this PR. The biggest different is the sequential run of tests, which we can revisit in case speed is a concern.

[ehelms]

@jeremylenz Updated based on your feedback

[jeremylenz]

It doesn't seem to be working right for me.

1. I seem to get all the output in one shot (after a long wait), rather than being able to follow along
2. I don't see the error you added:

$ script/foreman-test test/unit/foreman/renderer/scope/macros/host_template_test.rb --name "foo"
WARNING:  SET CONSTRAINTS can only be used in transaction blocks
Emptying /home/vagrant/foreman/jenkins/reports/unit
-- execute("SET CONSTRAINTS ALL DEFERRED;")
   -> 0.0011s

# Running tests with run options --name foo --seed 50466:

Writing XML reports to /home/vagrant/foreman/jenkins/reports/unit


Finished tests in 0.004709s, 0.0000 tests/s, 0.0000 assertions/s.


0 tests, 0 assertions, 0 failures, 0 errors, 0 skips

[ehelms]

• I seem to get all the output in one shot (after a long wait), rather than being able to follow along

That is due to having implemented ". May be worth fixing in the script by detecting the "0 tests" case and returning a non-zero exit / warning." But I think that's not a great path and am switching it back to streaming output.

[jeremylenz]

LGTM 👍 streams test output again.