[PROPOSAL] Introduce native jitter metrics (computed in streaming) for Component and e2e stream stability analysis

[GGraziadei]

Currently, Apache Storm provides comprehensive metrics for throughput and average latency (execute-latency, process-latency). However, in high-precision real-time systems, averages often mask critical performance instabilities.

This proposal introduces a native Jitter Metric calculated at two levels:

• Component level (Step Jitter): Measures the variance in execution time within individual Bolts and Spouts.
• Topology level (Global Jitter): Measures the variance in e2e completion latency for fully acked tuples.

In deterministic real-time processing, the variance of the latency is as important as the latency itself (https://ieeexplore.ieee.org/abstract/document/10877871).

Why analysing jitter matters for real-time

In deterministic real-time processing, predictability of latency is as important as latency itself. This is a constraint to building a deterministic system.

• Mcro-burst detection: high jitter reveals short spikes that average latency smooths out.
• Compliance: modern SLAs rely on percentiles (e.g., P99). Jitter is a strong leading indicator of tail-latency degradation.
• Root Cause Analysis: high component jitter means GC pressure or resource contention; instead, high global jitter with stable components suggests network congestion or shuffle bottlenecks.
• Bottleneck identification: jitter enables precise identification of where bottlenecks occur in the topology and helps distinguish their underlying causes, making performance issues easier to diagnose and resolve.

Proposed model: Exponentially Weighted Moving Average (EWMA)

To ensure negligible performance impact, I propose to use an Exponentially Weighted Moving Average (EWMA), following RFC 1889 logic https://www.rfc-editor.org/rfc/rfc1889#appendix-A.8

Mathematical Model:
J_new = J_old + (|D_current - D_previous| - J_old) / 16

GIVEN a State {ewmaJitter, lastTransit}
PROCEDURE addValue(transitMs)
    IF transitMs < 0 THEN 
        EXIT PROCEDURE

    IF lastTransit IS NOT UNINITIALIZED THEN
        // Calculate the absolute difference between the current and previous transit time
        deviation = ABS(transitMs - lastTransit)
        
        // Update the Exponentially Weighted Moving Average using the RFC 1889 smoothing factor
        ewmaJitter = ewmaJitter + (deviation - ewmaJitter) * RFC1889_ALPHA
    END IF

    // Store current transit time for the next iteration
    lastTransit = transitMs
END PROCEDURE

Performance impact

• Minimal computational overhead: by utilizing an EWMA, we avoid the need for storing large datasets or sliding window buffers. The jitter is updated via a single linear equation, requiring only basic arithmetic.
• Memory efficiency: The EWMA algorithm is extremely memory-light, requiring only a single persistent variable (8 bytes) per executor to maintain the moving average state, plus a reference for the previous latency sample.
• System calls: To eliminate redundant overhead, the metric hooks into existing latency tracking logic. This point requires additional brainstorming to evaluate already sampled metrics.

Limitations and constraints

• Clock skew: Global jitter may be affected in the case of unsynchronised nodes. However, since jitter measures variance between consecutive samples, constant skew cancels out mathematically.
• Sampling bias: Low sampling rates may miss high-frequency jitter spikes.
• Warm-up: as an EWMA-based metric, values may fluctuate initially before stabilizing.

[rzo1]

Thanks for the detailed write-up - the motivation is solid and the RFC 1889 EWMA is genuinely cheap, so the cost
side of the proposal is hard to argue with. A few things I'd like to see clarified before this moves to a PR:

1. Relationship to existing Dropwizard metrics. Storm's metrics v2 is built on Dropwizard, and the existing
   execute-latency / process-latency / complete-latency timers are already Histogram/Timer instances with reservoir
   sampling. That gives us P99/P999, stddev, and mean for free, and they flow through the existing reporters
   (Prometheus, Graphite, etc.). Could you spell out what EWMA-of-abs-diff gives operators that the histogram
   percentiles don't? The RFC 1889 form is a great fit for an RTP receiver computing inter-arrival jitter on the fly,
   but operators consuming Storm metrics typically want tail percentiles, and the two are not the same statistic. If
   the answer is "cheaper than a reservoir," it's worth quantifying - Dropwizard's exponentially decaying reservoir is
   also quite light.

2. Definition. "Jitter" gets read three different ways in practice: (a) RFC 1889 smoothed inter-arrival deviation,
   (b) latency standard deviation, (c) tail spread (P99 − P50). The proposal picks (a), which is fine, but the
   doc/metric name should be explicit so users don't assume (b) or (c), right?

3. Smoothing factor. α = 1/16 is tuned for RTP's packet cadence. Storm tuple rates span many orders of magnitude:
   at 1M tuples/sec α=1/16 averages over microseconds of wall clock, at 10/sec over seconds. This should be
   configurable per topology (or per component), not a constant.

4. Integration point. The line "hooks into existing latency tracking logic" is doing a lot of work. Could you sketch
   where exactly BoltExecutor / SpoutExecutor latency sampling paths, the stats BoltExecutorStats /
   SpoutExecutorStats, or a parallel listener? That's the part that determines whether this is a 50-line change or a
   much larger one, and it'd be good to align on it before any code lands.

5. Acked-only e2e jitter. Restricting global jitter to fully-acked tuples introduces a selection bias: failed and
   timed-out tuples are often exactly the ones with pathological latency. Worth either including them or documenting
   the bias clearly.

6. Clock skew. "Constant skew cancels mathematically" is true, but e2e measurements cross workers/nodes where the
   relevant problem is drift, not constant offset. NTP-synced clocks still drift on the order of ms, which can dominate
   the jitter signal at low latencies. Worth calling out as a real limitation rather than a dismissed one (at least in the documentation of such a feature)

Happy to help review a PR once the integration point and the EWMA-vs-histogram-percentile question are settled.

[GGraziadei]

Hello @rzo1 thank you for your comment.

1. My ultimate goal is to move towards a self-regulating system, one capable of detecting congestion in real-time and dynamically modulating throughput across different stages, increasing the determinism of the flow transmission. Any robust control system requires a feedback loop, and while this is a complex goal that will likely be achieved in multiple steps, the absolute first requirement is a reliable, high-frequency control signal. I fully recognise that similar metrics can be derived from existing histograms, which are excellent for post-hoc analysis and human-readable dashboards. However, what I am proposing is a metric calculated on-the-fly, directly within the stream of transit. There is a fundamental difference in how these signals behave: histograms with reservoirs are inherently batch-oriented or sampling-based; they require a window of data to be collected before a statistically significant percentile can be computed. This introduces a lag that is often too high for reactive control logic. RFC 1889 EWMA instead is calculated continuously (or with a very high sampling ratio). Every single tuple (or a high percentuage or tuples0 updates the state in constant time. This lack of sampling delay provides a reactive feedback signal that can detect micro-bursts or jitter the moment they occur. In short, while histograms tell us how the system performed, this EWMA signal tells us how the system is performing right now. This immediacy is what will eventually allow an executor to detect a bottleneck and propagate that signal upstream to stabilize ingestion and ensure a more deterministic flow.
2. That is a valid point. Since jitter is often overloaded, I will explicitly label this as RFC 1889 Inter-arrival Jitter (definition A) in the documentation. By naming the metric precisely (e.g., jitter_rfc1889_a), we can clearly distinguish this reactive control signal from the descriptive statistics (P99 or StdDev) already provided by Dropwizard.
3. I completely agree that the smoothing factor has to be a configurable parameter and not a constant.
4. According to point 1, the idea is to extend the BoltExecutor and SpoutExecutor to create two new classes (e.g. BoltDeterministicExecutor / SpoutDeterministicExecutor)
5. That is a great point regarding selection bias. Including failed tuples is necessary for a complete picture, but I want to avoid polluting the jitter signal with the pathological latencies typical of timeouts. I propose splitting this into two distinct metrics: one for successful tuples and one for failed/timed-out tuples. This provides full visibility without compromising the precision of the primary control signal.
6. I completely agree, it is required to add this limitation in the documentation.

[GGraziadei]

Hi @rzo1, thank you for your review on #8586.

I have prepared this draft PR #8593 as a follow-up. Before moving it to a "Ready" state, I’d like to discuss the architectural approach with you.

1. Feedback Loop
   To complete the control loop, I have implemented a mechanism to propagate a feedback tuple (containing EWMA statistics) from an task to its upstream tasks.
   This is not an acksignal; its frequency is user-configurable. I have registered a new stream specifically for this feedback.
   Upon receiving a feedback tuple, an executor updates the perceived jitter status of its downstream tasks.

I am aware that introducing these control tuples consumes bandwidth and may slightly reduce overall throughput.
Is this approach, using specialized tuples for control signals, acceptable to you, or would you suggest a different method for exposing downstream jitter status to upstream components?

2. Proactive Backpressure via reorder buffer
   The second phase of this PR involves consuming the downstream jitter status to modify the executor's queue consumption order.
   The Idea: implement a small reorder buffer (e.g., capacity of 4 tuples).
   Reorder tuples based on the jitter of the destination task (persisted in the executor stats) to proactively mitigate backpressure. THis work is done by the consumer implementation.

What do you think about this approach?

3. Validation Results
   To validate this proposal, I implemented a discrete event simulator (available in dev-tools). In a test scenario where a task's queue is filled by 4 producers and writes to 6 downstream tasks, I observed the following:

Jitter reduced by ~25–30%.
Throughput decreased by ~1.8%.
Latency increased by ~2.5%.

What are your thoughts on these trade-offs and the overall direction?

[rzo1]

Thanks @GGraziadei — happy to dig in. Three answers, mirroring your numbering:

1. Specialized feedback tuples vs. another transport.
A dedicated stream with direct task-addressed delivery is the right shape — it's the same idiom Storm uses for system tuples (ack, metrics, ticks), it survives the worker-to-worker transport, and unanchored delivery keeps it out of the acker tree. So conceptually, yes, that's an acceptable mechanism. A few constraints I'd want pinned down before it lands:

• If feedback only fires on the acked-emit path, you lose the signal exactly where backpressure pressure is most acute: non-reliable topologies, sink bolts that don't re-emit, and any tuple that fails. Predictive backpressure that doesn't cover fire-and-forget pipelines is a hard sell. I'd push for emission to be decoupled from acking — e.g. a periodic tick on the receive side, or every N processed tuples — so spouts, sinks, and unreliable flows all participate.
• Just to head this off: __metrics is shuffle-grouped into IMetricsConsumer bolts on a seconds-scale cadence with retain caps and filtering. None of that fits a sub-second, direct-addressed control signal, and changing it would break every existing metrics consumer. A separate stream is correct.
• At per-anchor frequency, the wire/GC cost matters. A narrow dedicated record (e.g. (srcTaskId, processJitter, executeJitter, completeJitter) as primitives) will be far cheaper than reusing the v2 metrics DataPoint shape, and makes the contract of this stream explicit rather than entangled with the metrics pipeline.
• A fixed sampling ratio is noisy at low rates and expensive at high rates. Either a token-bucket rate cap or an adaptive ratio tied to throughput would be more honest. Either way, please quantify the per-tuple overhead at, say, 100k tuples/s on a representative topology before promoting from draft.
• This stream consumes queue capacity and shows up in __transfer-count. Make sure it's surfaced cleanly in the UI / metrics so operators can see what fraction of executor budget the control loop is eating.

2. Reorder buffer for proactive backpressure.
This is the part I'm most skeptical about, mainly because the benefit-to-complexity ratio isn't yet demonstrated:

• A small reorder window (single-digit elements) on the receive queue is only meaningful when an executor hosts multiple downstream-relevant tasks and each has well-populated jitter stats. In the very common single-task-per-executor case the comparator is effectively a no-op, so the gain depends entirely on topology shape — worth stating explicitly.
• Reordering inside one executor's receive queue is a local heuristic. True predictive backpressure usually wants the signal to flow further upstream — i.e. the producer should slow down or pick a different downstream task, not the consumer reshuffle a small window of already-arrived tuples. Have you considered using the jitter signal at emit-time in the producer's grouping/load-balancing decision instead of consumer-side reorder? That would be more potent and avoid head-of-line issues.
• The reorder window size should be configurable, and the comparator's invariants (what task IDs it operates on, what happens when stats are missing, what stable ordering it guarantees) need to be spelled out — that's the kind of thing that tends to silently break stream ordering guarantees users rely on.
• Tuples within a stream from a single producer to a single task currently arrive in emit order. Reordering can change that. If a topology relies on intra-stream order (very common in stateful bolts), this is a correctness change, not a performance tweak. It must be opt-in, documented as such, and ideally only kick in when the per-tuple destination differs.

3. Validation results.
Encouraging trend, but I don't think the simulator is enough evidence on its own:

• A discrete-event simulator validates the control logic, not the cost of carrying the control signal through Storm's actual transport, batching, serialization, and JCQueue dynamics. Before this is "ready," I'd want numbers from a real Storm benchmark — storm-perf or a representative topology — showing the same direction.
• The metric being optimized (RFC 1889 inter-arrival jitter) is not what most operators actually tune for. They tune tail latency (P99/P999). It's entirely possible to reduce inter-arrival deviation while worsening P99 — smoothing by penalizing outliers. Please include tail-latency alongside the jitter delta. If P99 also improves, the case is strong; if P99 regresses, we should know.
• ~1.8% throughput / +2.5% latency is a reasonable cost for a real win, but those numbers will move once acking, network transport, and realistic batch sizes are in the loop. Worth re-measuring under those conditions, and probably worth a load test with feedback enabled but predictive reorder disabled, to isolate the cost of the control signal itself from the cost of the heuristic.

In short: I'm supportive of (1) with the caveats above, cautious about (2) and would want to see emit-side use of the signal explored first, and want real Storm benchmarks plus tail-latency numbers before (3) is conclusive. Everything that does land should be opt-in with sane defaults that disable it, given that this touches hot paths and ordering semantics.

[rzo1]

Would love to hear additional thoughts by @reiabreu or @jnioche too.

[reiabreu]

Hey folks, apologies for the lack of feedback. I will try to go through the thread and leave some feedback ASAP.

[reiabreu]

I mostly agree with @rzo1 analysis (thanks for the very detailed write up to both). Specially around the reordering tuples. Fields grouping uses key mod numTasks, so we wouldn't be breaking this, but we would be changing the implicit FIFO. What what would happen if we decide to hold back a tuple in an executor due to the state of the downstream task only for the next tuple that would go to the same task find a more favorable downstream jitter and be emitted. We would be changing the order of processing and breaking the sequence guarantees, despite assuring co-location.
The proposal is very interesting, jitter metrics are worth exporting, but the self regulating loop will be tricky as proposed

[GGraziadei]

Hi @rzo1 and @reiabreu,
Thank you both for the valuable insights.

Reorder buffer
You are absolutely right. Breaking the implicit FIFO behaviour is a major correctness risk. Even if co-location is preserved, altering the sequence of tuples within the same stream introduces race conditions that users shouldn't have to reason about.
Given this, I am dropping the consumer-side reorder buffer approach.

Shifting to the emission-side heuristics
I really like @rzo1's suggestion to use the jitter signal at emit-time. Utilizing downstream jitter status to influence the producer's routing decisions is a much cleaner architecture. It naturally avoids head-of-line blocking and prevents local queues from becoming saturated.
To achieve this, I am currently evaluating a strategy centred around implementing a jitter-aware CustomStreamGrouping.

Feedback stream
Regarding the control signal, I agree with the constraints outlined. The feedback contains only EWMA metrics, thus it has the advantage of being async. I completely agree to decouple from the ack logic.
I will ensure the control tuple uses a minimal primitive shape (srcTaskId, processJitter, ...) rather than the heavy v2 metrics struct.

As the next step, I will reduce the scope of the draft PR #8593 to successfully submit and persist the feedback signal updates as discussed in the Feedback stream section and according to your previous comments.