PHOENIX-7995 Tag client HA metrics with the HA group and add missing HA/CRR/failover client metrics - #2609
Draft
lokiore wants to merge 2 commits into
Draft
Conversation
…th and add a connection-failed counter The HA failover observability metrics added under PHOENIX-7872 recorded HA_FAILOVER_DURATION_MS inside FailoverPhoenixConnection.failover(long). That method is only reached through wrapActionDuringFailover -> FailoverPolicy .shouldFailover(), which returns false under the default ExplicitFailoverPolicy, or through the explicit static failover(Connection, long) helper. Neither runs during an autonomous, CRR-driven failover, so the duration metric never moved in production. Move the duration measurement to the path that actually drives failovers: refreshClusterRoleRecord, where the cluster-role transition is dispatched and where HA_FAILOVER_COUNT is already gated by shouldCountFailover. The dispatch block is wrapped in a try/finally so the duration is recorded on every exit (success, timeout, policy failure, or interrupt), avoiding a silent metric miss if a future exit path is added. The now-dead timing in failover(long) is removed. Add HA_FAILOVER_CONNECTION_FAILED_COUNTER, incremented at the single SQLException throw funnel in connectActive (no active cluster, cluster demoted mid-connect, or the underlying connect threw). This tracks real active-cluster connection failures regardless of the configured failover policy. Tests: three unit tests in HighAvailabilityGroupTest -- a counted role-flip transition records both HA_FAILOVER_COUNT and an HA_FAILOVER_DURATION_MS sample on the CRR-write path; a failed connectActive increments the connection-failed counter; a successful connectActive leaves it unchanged. HighAvailabilityGroupTest 16/16, FailoverPhoenixConnectionTest 8/8. Generated-by: Claude Code (Opus 4.8) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…HA/CRR/failover client metrics
The ZK-less HA client emits all HA/CRR/failover metrics as JVM-global
GLOBAL_HA_* counters, so a JVM connected to more than one HA group cannot
attribute a failover, stale-CRR detection, or poller failure to a specific
group. It also lacks client counters for several HA/CRR/failover events that
are relevant on the ZK-less path.
Per-HA-group tagging
--------------------
Add HAGroupClientMetricsSource, a per-group Hadoop Metrics2 source (one per HA
group name) that stamps a "haGroup" tag carrying the group name and appends the
quoted group name to its JMX context so each group registers as a distinct
source/MBean. HAGroupMetricsManager is the process-wide registry: it lazily
creates a source per group (only when global client metrics are enabled),
routes per-group increment/update, and detaches the source on HA-group close so
the same group can re-register later. Emission is dual: every group-attributable
GLOBAL_HA_* increment is mirrored to the group's source, and the JVM-global
counters continue to emit unchanged.
The tag key lives in a new module-neutral MetricConstants.HA_GROUP_TAG_NAME
("haGroup") referenced by both this client source and the server-side
HAGroupStoreMetricsSource, so both sides tag with the same key and can be
filtered together downstream.
New client counters
-------------------
- HA_FAILOVER_CONNECTION_CREATED_COUNTER: FailoverPhoenixConnection instances
successfully created against the active cluster (pairs with the existing
connection-failed counter).
- HA_ROLE_TRANSITION_FAILED_COUNTER: cluster-role-transition dispatch failures
(execution error or timeout) on the CRR-write path.
- CRR_TRANSITION_COUNT: cluster-role-record transitions applied per HA policy,
including transitions into a no-active state (distinct from HA_FAILOVER_COUNT,
which counts only transitions that establish/move an ACTIVE cluster).
Group-attributable emission is wired at the existing sites in
HighAvailabilityGroup (failover count/duration, CRR refresh, connect-failed, and
the two new transition metrics), FailoverPhoenixConnection (connection created,
stale-CRR, mutation-blocked), HighAvailabilityPolicy (parallel fallback),
ParallelPhoenixContext/ParallelPhoenixUtil (parallel connection created/error,
task timeout), and GetClusterRoleRecordUtil (poller tick count/failures). The
JVM-shared parallel-executor pool metrics and the HA_CRR_CACHE_AGE_MS gauge are
deliberately excluded from the per-group source (JVM-shared / not a counter).
Tests: HAGroupClientMetricsSourceTest (6) and HAGroupMetricsManagerTest (7)
cover the haGroup tag/quoting, per-counter increment/update, per-group
isolation, distinct registered sources, and detach/re-create. 13/13 pass.
Generated-by: Claude Code (Opus 4.8)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This PR makes the ZK-less HA client's HA/CRR/failover metrics sliceable per HA group, and fills gaps in the client-side HA metric coverage. It builds on (and supersedes) #2605, which is folded in here as its own PHOENIX-7872 commit.
1. Per-HA-group metrics tag. A new per-group Hadoop Metrics2 source,
HAGroupClientMetricsSource, registers one metrics2 source per HA group. Each instance:,haGroup=<ObjectName.quote(name)>to its JMX context so every group registers as a distinct source / MBean, andhaGrouptag carrying the (unquoted) group name so the series can be sliced per HA group downstream.HAGroupMetricsManageris a small registry (one source per group name) wired intoHighAvailabilityGrouplifecycle: a source is created on group init and detached on group close. Emission is dual: the existing JVM-globalGLOBAL_HA_*counters continue to emit unchanged, and the same event is additionally recorded on the per-group source. The tag key lives in a new module-neutralorg.apache.phoenix.metrics.MetricConstants(HA_GROUP_TAG_NAME = "haGroup"), referenced by both the new client source and the server-sideHAGroupStoreMetricsSourceso both sides share one tag key.The per-group set is intentionally limited to metrics attributable to a single HA group (each emission site has a
HighAvailabilityGroupin scope). The JVM-shared parallel-executor pool metrics and theHA_CRR_CACHE_AGE_MSgauge are excluded.2. Missing client counters. New
MetricTypes and theirGLOBAL_HA_*wrappers, emitted (dual) at the relevant client sites:HA_FAILOVER_CONNECTION_CREATED_COUNTER— aFailoverPhoenixConnectionwas successfully created against the active cluster.HA_FAILOVER_CONNECTION_FAILED_COUNTER— a connect-to-active attempt threw (no active cluster, demoted mid-connect, or connect error). (from PHOENIX-7872 Addendum record HA failover duration on the CRR-write path and add a connection-failed counter #2605 / PHOENIX-7872)HA_ROLE_TRANSITION_FAILED_COUNTER— a cluster-role-transition dispatch failed while applying a new CRR.CRR_TRANSITION_COUNT— a CRR transition was applied per HA policy, including transitions into a no-active state (distinct fromHA_FAILOVER_COUNT, which counts only transitions that establish/move an ACTIVE cluster).Existing counters (
HA_FAILOVER_COUNT,HA_FAILOVER_DURATION_MS,HA_STALE_CRR_DETECTED_COUNT,HA_MUTATION_BLOCKED_COUNT,HA_CRR_REFRESH_COUNT, theHA_PARALLEL_*connection/task counters, and the poller-tick counters) are additionally recorded per-group at their existing emission sites.The #2605 change (record
HA_FAILOVER_DURATION_MSon the CRR-write path rather than the deadfailover()path, and add the connection-failed counter) is preserved as its own PHOENIX-7872-attributed commit at the base of this branch; #2605 will be closed as superseded by this PR.Why are the changes needed?
On the ZK-less HA client the JVM-global
GLOBAL_HA_*counters aggregate across every HA group in the process, so a JVM serving more than one HA group cannot attribute failover / stale-CRR / mutation-blocked / transition activity to a specific group. Tagging each series withhaGroupmakes per-group dashboards and alerting possible while keeping the existing global counters intact for backward compatibility. The additional counters close observability gaps around connection creation/failure and role-transition outcomes that had no client-side metric.Does this PR introduce any user-facing change?
No behavioral change. New client-side metrics are added (new
MetricTypeentries and theirGLOBAL_HA_*wrappers) and a new per-grouphaGroup-tagged metrics2 source is registered; the existing global HA counters are unchanged. No SQL, API, or wire-format change.How was this patch tested?
New unit tests (run under surefire):
HAGroupClientMetricsSourceTest— thehaGrouptag carries the unquoted group name; the JMX context is quoted per group; increment/update are per-counter; unknown metric types are ignored; each group is a distinct registered source; unregister frees the source name for reuse.HAGroupMetricsManagerTest—getOrCreateis idempotent and registers a source; null/empty group names are a no-op; two groups never cross-count; each group gets a distinct tagged source;updateaccumulates per group;removedetaches the source; re-create after remove rebuilds a fresh source.HighAvailabilityGroupTestadditions cover the CRR-write-path duration/connection-failed emission (from PHOENIX-7872 Addendum record HA failover duration on the CRR-write path and add a connection-failed counter #2605 / PHOENIX-7872).mvn spotless:applywas run before pushing; the full module build compiles clean.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)