fix: improve Data Explorer health signal accuracy

[sunghyunkang1111]

Why this change

The existing Data Explorer health signal was producing incidents that were not consistently actionable. We reviewed the frontend state machine, Portal Backend telemetry, the Geneva metric and production monitor, and historical ICM behavior before changing the semantics.

Evidence from Portal Backend/Kusto

A roughly 45-day retained sample showed that unhealthy events were dominated by scenario timeouts rather than explicit unexpected failures:

Platform / scenario	Unhealthy rate	Unhealthy events reported as timeouts
Portal / DatabaseLoad	1.453%	70.58%
Portal / ApplicationLoad	0.211%	100%
Hosted / DatabaseLoad	1.025%	91.26%
Hosted / ApplicationLoad	0.369%	48.71%

The five-minute windows used by the current Geneva monitor were also highly sensitive to sparse traffic:

• Portal had 26 windows at or above the 10% unhealthy threshold; only 16 had at least 20 samples and only one had at least 50.
• Hosted had 125 windows at or above 10%; 105 of those windows were caused by a single unhealthy event. Only five had at least 20 samples and one had at least 50.

Evidence from Geneva/ICM

The production monitor evaluates a five-minute window every three minutes, alerts above 10%, aggregates only by Platform, has no minimum event floor or persistence, and does not automatically mitigate.

Across approximately 180 days of ICM history we found 22 direct production incidents, evenly split between Portal and Hosted. At review time, 15 remained active, 14 were unacknowledged, and none were marked customer-impacting. This is consistent with a signal that is too sensitive to isolated events and expected environmental conditions.

Frontend causes identified

• Expected authentication, authorization, firewall, and policy failures were stored globally and could affect unrelated active scenarios.
• Once any expected failure was present, a later phase failure could be treated as successful completion.
• Expected-failed phases were marked complete even though the phase had not actually succeeded.
• Background/hidden-tab timeouts could be interpreted as product failures.
• Real errors were not consistently passed to the scenario boundary for classification.

New health semantics

The monitor now applies explicit scenario-local precedence:

1. Any unexpected failure → Unhealthy immediately.
2. Otherwise, expected-only evidence → Healthy with diagnostics.
3. Otherwise, all required phases complete → Healthy.
4. Otherwise, a visible timeout → Unhealthy.

Expected evidence is non-terminal: it does not complete a phase and cannot hide a later unexpected failure.

Why not every timeout is Unhealthy

A timeout means the scenario did not complete all required phases within ten seconds; it does not identify the cause. For operational health, a visible and unexplained timeout remains Unhealthy. The exceptions are cases where the timeout is explained by a condition that is not evidence of a Data Explorer service regression:

Timeout context	Health result	Rationale
Visible page with no expected blocker	Unhealthy	The application had a fair opportunity to complete and did not.
Any unexpected phase failure	Unhealthy immediately	Concrete failure evidence always overrides visibility or expected evidence.
Expected authorization, firewall, or policy condition only	Healthy with diagnostics	The user flow did not complete, but the condition is not an actionable Data Explorer service failure.
Page was hidden/backgrounded with no unexpected failure	Healthy with diagnostics	Browser timer throttling and suspended work make the ten-second deadline unreliable.

This distinction is between user-flow completion and operational service health. Expected and hidden-page timeouts remain observable through timedOut, hasExpectedFailure, phase diagnostics, and telemetry; they are not silently discarded. Treating every timeout as Unhealthy would reintroduce the sparse-window alert noise observed in Geneva, where a single event caused 105 of 125 Hosted windows above the current 10% threshold.

What changed

• Scope expected-failure evidence to one scenario and phase.
• Classify errors at the ApplicationLoad and DatabaseLoad boundaries.
• Keep expected-failed phases pending until real completion or timeout.
• Ensure unexpected failures override expected evidence.
• Treat hidden-page-only timeouts as Healthy diagnostic events.
• Guarantee exactly one terminal event per scenario.
• Clean up visibility listeners and performance entries between runs.
• Trace failures to report the final metric instead of silently dropping rejected promises.
• Preserve the original ARM error through hosted key retrieval so AuthorizationFailed remains correctly classified.

What is intentionally not included

• Login monitoring and changes to Entra or connection-string authentication are deferred.
• The Portal Backend route and authentication policy are unchanged in this PR.
• Geneva monitor thresholds and scenario-specific monitor definitions are being handled separately so frontend semantics can be validated before production cutover.

Validation

• 45 targeted Jest tests passed, including scenario isolation, expected/unexpected precedence, hidden and visible timeouts, race behavior, reset cleanup, and exactly-once emission.
• TypeScript and strict TypeScript compilation passed.
• Prettier and ESLint passed with no new errors.
• Authenticated Hosted Explorer happy path produced one Healthy terminal event for ApplicationLoad and DatabaseLoad.
• Browser-level ARM fault injection verified:
  • listKeys and readonlykeys returning 403 AuthorizationFailed produce one Healthy timed-out ApplicationLoad result with expected diagnostics.
  • listKeys returning an unexpected 500 produces one immediate Unhealthy result and no later duplicate timeout event.
• An independent adversarial review found no high-confidence correctness issues.

Rollout expectation

This PR improves the meaning of the emitted frontend signal. The companion Geneva rollout should initially run scenario-specific monitors alongside the existing production monitor, validate volume and threshold behavior in staging, and only then replace the blended Platform-only monitor.

[github-actions]

Playwright tests ✅ passed

Passed	Failed	Flaky	Duration
629	0	34	1853s

📁 Report: 32381338962-1/report.zip
Open container (Azure sign-in required) → click into 32381338962-1 folder → click report.zip → Download → unzip → open index.html · Workflow run

[github-actions]

Playwright tests ❌ failed

Passed	Failed	Flaky	Duration
640	1	42	1107s

📁 Report: 32385047545-1/report.zip
Open container (Azure sign-in required) → click into 32385047545-1 folder → click report.zip → Download → unzip → open index.html · Workflow run

⚠️ 1 shard(s) failed before tests ran (infra/auth issue). Stats below reflect only shards that executed.

[sunghyunkang1111]

Paired Portal Backend change: https://msdata.visualstudio.com/DefaultCollection/CosmosDB/_git/CosmosDB-Portal-Backend/pullrequest/2262065

[github-actions]

Playwright tests ❌ failed

Passed	Failed	Flaky	Duration
671	2	10	918s

📁 Report: 32435679941-1/report.zip
Open container (Azure sign-in required) → click into 32435679941-1 folder → click report.zip → Download → unzip → open index.html · Workflow run

⚠️ 2 shard(s) failed before tests ran (infra/auth issue). Stats below reflect only shards that executed.

[github-actions]

Playwright tests ✅ passed

Passed	Failed	Flaky	Duration
667	0	16	975s

📁 Report: 32493151305-1/report.zip
Open container (Azure sign-in required) → click into 32493151305-1 folder → click report.zip → Download → unzip → open index.html · Workflow run