[bot] Merge master/8a7cf065 into rel/dev - #1761
Merged
Merged
Conversation
…uest
generate_simulated_response() only saw the assistant's last message and the
ground-truth MAQL, and was instructed to force every clause of that MAQL to
be satisfied "even if the assistant's question doesn't explicitly ask about
it" -- so it would inject filters/constraints the user's original request
never mentioned, even when the assistant's proposal already matched it.
- Thread the original question through (metric_skill.py's
_execute_single_metric_run already has it in scope; conversation.py's
TurnDefinition.message carries the same for multi-turn conversations) and
rewrite the prompt to agree when the original request is already
satisfied, only adding a clause when it's a reasonable reading of that
request -- not an unconditional replay of expected_outputs[0].
- Add an explicit branch for the dominant real case: the assistant asking a
clarifying question with no proposal yet. Without it, the simulated user
could trivially agree ("nothing proposed yet" == "satisfied") and stall
the conversation, burning iterations without ever supplying the agent a
usable answer.
- Replace fuzzy "is this filter a reasonable reading of the request"
judgment with a deterministic _no_filter_hint(): when the ground-truth
MAQL has no WHERE clause, the prompt explicitly tells the simulated user
no filter is needed, closing the exact loophole that caused the bug.
Matches WHERE as a standalone keyword outside {type/id} identifiers and
quoted literals (reusing the existing _PROTECTED_RE / same rule as
_casefold_outside_protected), so a substring like
{metric/somewhere_sales} isn't mistaken for a real clause.
- conversation.py's metric branch (forwards to
metric_skill.generate_simulated_response) had 0% test coverage behind a
bare `except Exception: pass` -- a future signature mismatch would
silently fall through to the generic fallback prompt. Log the exception
and add a direct unit test for the branch.
- Restore the max_tokens >= 300 assertion, and reduce the new tests'
reliance on exact prompt-prose assertions in favor of checking the
interpolated data and the independently-testable _no_filter_hint() output.
Verified locally: ran the full agent_metric_skill (8 cases) and
agent_conversations (10 cases) suites against ecommerce_demo on
tavern-frank-test -- 18/18 passing with this fix.
QA-29094
…tric-scope fix(gooddata-eval): scope simulated-user pushback to the original request
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Comment |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## rel/dev #1761 +/- ##
===========================================
+ Coverage 80.61% 80.65% +0.03%
===========================================
Files 272 272
Lines 19362 19369 +7
===========================================
+ Hits 15609 15622 +13
+ Misses 3753 3747 -6 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🚀 Automated PR to perform merge from master into rel/dev with changes up to 8a7cf06 (created by https://github.com/gooddata/gooddata-python-sdk/actions/runs/32968183952).