Skip to content

feat(solidity-tests): scope inline config directives by test profile - #1715

Merged
popescuoctavian merged 13 commits into
mainfrom
solidity-test-profiles
Sep 23, 2026
Merged

popescuoctavian merged 13 commits into
mainfrom
solidity-test-profiles

Conversation

@schaable

Copy link
Copy Markdown
Member

Adds test profile support to inline configuration, so Hardhat can offer profiles for Solidity tests.

SolidityTestRunnerConfigArgs gains two optional fields, testProfile and declaredTestProfiles, both defaulting to default. A caller that passes neither is unaffected.

A directive's key can now carry a profile prefix:

/// forge-config: fuzz.runs = 3       // applies under every profile
/// forge-config: ci.fuzz.runs = 500  // applies only under `ci`

Where both set the same key, the prefixed one wins. Prefixes are validated against the declared profiles rather than the selected one, so a typo fails whichever profile is running.

Behaviour changes

  • A default.-prefixed directive is now scoped to the default profile instead of applying unconditionally.
  • An unprefixed directive alongside a prefixed one setting the same key is no longer a duplicate-key error.
  • InlineConfigUnsupportedProfile is renamed to InlineConfigUndeclaredProfile and now carries declaredProfiles.

The design doc has the full rationale, including why prefixes validate against the declared set and why directives for unselected profiles are still validated.

Hardhat PR: NomicFoundation/hardhat#8590

@schaable
schaable temporarily deployed to github-action-benchmark August 31, 2026 19:24 — with GitHub Actions Inactive
@changeset-bot

changeset-bot Bot commented Aug 31, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 3a489b4

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package
Name Type
@nomicfoundation/edr Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@codecov

codecov Bot commented Aug 31, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 97.57576% with 12 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.51%. Comparing base (5270347) to head (3a489b4).
⚠️ Report is 3 commits behind head on main.

Files with missing lines Patch % Lines
...edr_solidity_tests/src/inline_config/directives.rs 96.20% 3 Missing and 8 partials ⚠️
crates/edr_napi_core/src/solidity/config.rs 87.50% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1715      +/-   ##
==========================================
+ Coverage   80.41%   80.51%   +0.09%     
==========================================
  Files         464      466       +2     
  Lines       81719    82198     +479     
  Branches    81719    82198     +479     
==========================================
+ Hits        65714    66181     +467     
- Misses      13763    13766       +3     
- Partials     2242     2251       +9     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@Wodann
Wodann requested review from popescuoctavian and a balanced review from Copilot August 31, 2026 20:17

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds profile-aware Solidity inline configuration across the parser, runner, and N-API.

Changes:

  • Adds selected/declared profile configuration and scoped directive precedence.
  • Reports undeclared profiles through renamed structured errors.
  • Adds Rust and JavaScript integration coverage.

Reviewed changes

Copilot reviewed 19 out of 20 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
js/integration-tests/solidity-tests/test/unit.ts Tests undeclared-profile errors.
js/integration-tests/solidity-tests/test/fuzz.ts Tests profile-specific fuzz runs.
js/integration-tests/solidity-tests/test-contracts/InlineConfigUndeclaredProfile.t.sol Adds invalid-profile fixture.
js/integration-tests/solidity-tests/test-contracts/InlineConfigProfiles.t.sol Adds profile precedence fixture.
crates/edr_solidity_tests/tests/testdata/default/fuzz/FuzzProfileOverride.t.sol Adds Rust profile fixture.
crates/edr_solidity_tests/tests/it/helpers.rs Declares fixture profiles.
crates/edr_solidity_tests/tests/it/fuzz.rs Tests profile resolution.
crates/edr_solidity_tests/src/multi_runner.rs Passes profiles into collection.
crates/edr_solidity_tests/src/inline_config/provider.rs Propagates profile context.
crates/edr_solidity_tests/src/inline_config/profiles.rs Defines profile configuration.
crates/edr_solidity_tests/src/inline_config/overrides.rs Passes profiles to directives.
crates/edr_solidity_tests/src/inline_config/error.rs Adds profile errors.
crates/edr_solidity_tests/src/inline_config/directives.rs Implements profile scoping and precedence.
crates/edr_solidity_tests/src/inline_config.rs Exports and documents profiles.
crates/edr_solidity_tests/src/config.rs Adds profiles to runner configuration.
crates/edr_napi/src/solidity_tests/inline_config.rs Exposes renamed structured error.
crates/edr_napi/src/solidity_tests/config.rs Accepts profile arguments.
crates/edr_napi/index.d.ts Updates TypeScript declarations.
crates/edr_napi_core/src/solidity/config.rs Validates and converts profile configuration.
.changeset/olive-pugs-shave.md Documents feature and breaking change.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread crates/edr_solidity_tests/src/inline_config/profiles.rs
@schaable
schaable temporarily deployed to github-action-benchmark August 31, 2026 20:52 — with GitHub Actions Inactive
@schaable
schaable temporarily deployed to github-action-benchmark August 31, 2026 20:52 — with GitHub Actions Inactive
@popescuoctavian

Copy link
Copy Markdown
Contributor

Thanks for adding this @schaable! This conflicts with the contract-level inline config changes from #1627. I'll wait for that to merge and then refactor this, but it may take a while. Sorry for that.

@popescuoctavian
popescuoctavian temporarily deployed to github-action-benchmark September 9, 2026 10:23 — with GitHub Actions Inactive
@popescuoctavian
popescuoctavian temporarily deployed to github-action-benchmark September 9, 2026 10:25 — with GitHub Actions Inactive
@popescuoctavian
popescuoctavian temporarily deployed to github-action-benchmark September 9, 2026 10:25 — with GitHub Actions Inactive
@popescuoctavian
popescuoctavian temporarily deployed to github-action-benchmark September 15, 2026 08:23 — with GitHub Actions Inactive
@popescuoctavian
popescuoctavian temporarily deployed to github-action-benchmark September 15, 2026 08:25 — with GitHub Actions Inactive
@popescuoctavian
popescuoctavian temporarily deployed to github-action-benchmark September 16, 2026 10:44 — with GitHub Actions Inactive
@popescuoctavian
popescuoctavian temporarily deployed to github-action-benchmark September 16, 2026 10:46 — with GitHub Actions Inactive
@popescuoctavian
popescuoctavian temporarily deployed to github-action-benchmark September 16, 2026 10:46 — with GitHub Actions Inactive
@popescuoctavian popescuoctavian self-assigned this Sep 21, 2026

@Wodann Wodann left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The main concern I have is around documenting the product decisions. We should do so both at the top-level inline_config.rs's module-level doc comments and on the Hardhat 3 website. Even more so, since we deviate from Foundry.

Ideally, this is written by a human rather than Claude and remains concise, merely stating the rules.

E.g.

/// Directives are handled in this order of priority:
///  1. Function-level > Contract-level
///  2. Profile > bare
/// 
/// Custom profiles do not inherit from the "default" profile.

Comment thread crates/edr_solidity_tests/src/inline_config/profiles.rs Outdated
Comment thread crates/edr_solidity_tests/src/inline_config/profiles.rs Outdated
Comment thread crates/edr_solidity_tests/src/inline_config/error.rs Outdated
Comment thread crates/edr_solidity_tests/src/inline_config.rs Outdated
Comment thread crates/edr_solidity_tests/tests/it/fuzz.rs Outdated
Comment thread crates/edr_solidity_tests/tests/it/fuzz.rs Outdated
let DirectiveTarget::Function(function) = target else {
return Ok(());
};
let is_fuzz_test = function.starts_with("test");

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

question: Is "test" the identifier for a fuzz test?

@popescuoctavian popescuoctavian Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Partially - fuzz tests have start with "test" (same as unit tests) AND have at least one parameter. The current implementation accepts both and doesn't report failures when using fuzz configs for unit tests, they are simply ignored (same as Foundry).

However, I believe that we should be more restrictive. IIRC in the initial implementation I made, data about parameters was not immediately available, which is why I made this choice. But it is now trivial to obtain from the Slang compilation unit ast and reject fuzz configs in unit tests. I'll address this in a follow-up PR since it's not related to this change. Great catch, thanks!

Comment thread crates/edr_solidity_tests/src/inline_config/directives.rs
@popescuoctavian
popescuoctavian added this pull request to the merge queue Sep 23, 2026
Merged via the queue into main with commit 451b5d1 Sep 23, 2026
62 of 63 checks passed
@popescuoctavian
popescuoctavian deleted the solidity-test-profiles branch September 23, 2026 16:33

This branch was previously deployed

1 inactive deployment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants