feat(storage): separate read and hedging thread pools

[ajayky-os]

This PR separates the thread pools used for primary reads and speculative hedged reads in Cloud Storage read hedging.

Previously, both primary reads and speculative hedges shared a single thread pool. This introduced two architectural issues:

1. Resource Contention / Head-of-Line Blocking: When multiple primary reads stalled or ran concurrently, the shared pool could become saturated. Consequently, speculative hedge
   attempts were blocked from executing, defeating the primary purpose of hedging (tail latency mitigation).
2. Asymmetric Sizing Needs: Primary reads block synchronously waiting on network I/O and require a higher concurrency ceiling (typically ≥64 threads or 4 × hardware concurrency),
   whereas speculative hedges are gated by rate limits and concurrency controls (typically bounded by MaxConcurrentHedgesOption or 2 × hardware concurrency).

This change introduces a general-purpose, lazily-scaling hedging_thread_pool.h:49 and composes it within hedging_thread_pool.h:145, isolating primary read execution from speculative hedges.

Key Changes

1. Dedicated ThreadPool and Embedded HedgingThreadPool

hedging_thread_pool.h:49:
• Dynamically scales workers on demand up to max_threads.
• Workers wait on a condition variable when idle and exit gracefully on shutdown.
• Automatically clamps max_threads to ≥1 to prevent deadlock/infinite hang if configured with 0.
• Supports self-destruction from within a worker thread (safely detaches rather than joining itself).

hedging_thread_pool.h:145:
• Embeds hedging_thread_pool.h:235 by value as its execution backend (declared last to guarantee worker joining before state teardown).
• Enforces the token-bucket rate limiter (ReadHedgeRateLimitOption) and maximum concurrent hedge ceiling (MaxConcurrentHedgesOption).

2. Dual Pool Configuration & Sizing Options

• Added options.h:122: Defaults to DefaultReadThreadPoolSize() (max (64,4 × cores)).
• Added options.h:135: Defaults to DefaultHedgingThreadPoolSize() (MaxConcurrentHedgesOption if set, else max (16,2 × cores)).
• Centralized sizing defaults in DefaultReadThreadPoolSize() and DefaultHedgingThreadPoolSize() so client.cc:604 and connection_impl.cc:165 remain consistent.

3. Isolation in HedgedObjectReadSource

• Updated hedged_object_read_source.cc:90 to accept separate read_pool_ and hedge_pool_.
• Primary attempt opens are scheduled onto read_pool_.
• Speculative hedged attempts are scheduled onto hedge_pool_.

[gemini-code-assist]

Code Review

This pull request refactors the storage client's thread pool management by introducing a dedicated ThreadPool for primary read attempts, distinct from the HedgingThreadPool used for speculative hedges. It adds configuration options for thread pool sizes and updates the connection implementation, read source, and tests accordingly. The review feedback identifies a namespace compilation error in client.cc, requests the use of explicit types instead of auto for primitives in connection_impl.cc to comply with the style guide, and suggests caching thread pool size calculations in hedging_thread_pool.h for better performance.

[codecov]

Codecov Report

❌ Patch coverage is 94.91525% with 12 lines in your changes missing coverage. Please review.
✅ Project coverage is 92.24%. Comparing base (5bdcb3d) to head (edddb77).

Files with missing lines	Patch %	Lines
google/cloud/storage/internal/connection_impl.cc	10.00%	9 Missing ⚠️
...storage/internal/hedged_object_read_source_test.cc	98.23%	2 Missing ⚠️
...loud/storage/internal/hedged_object_read_source.cc	83.33%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #16389      +/-   ##
==========================================
- Coverage   92.26%   92.24%   -0.02%     
==========================================
  Files        2246     2246              
  Lines      212121   212262     +141     
==========================================
+ Hits       195707   195802      +95     
- Misses      16414    16460      +46     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[ajayky-os]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces a dedicated ThreadPool class to isolate primary read attempts from speculative hedge attempts, which previously shared a single HedgingThreadPool. The HedgingThreadPool has been refactored to delegate task execution to an internal ThreadPool while retaining its throttling and rate-limiting capabilities. Additionally, new configuration options (ReadThreadPoolSizeOption and HedgingThreadPoolSizeOption) and their corresponding default sizing logic have been added to allow fine-grained control over pool sizes. Unit tests have been expanded to verify thread pool isolation, saturation behaviors, and safe destruction. There are no review comments to address, and the changes conform to the repository's style guidelines.

[ajayky-os]

/gcbrun

[v-pratap]

DefaultReadThreadPoolSize() defaults to 64 threads, but REST ConnectionPoolSizeOption defaults to a much lower ceiling (typically 4-8 connections). Should ReadThreadPoolSize be aligned with ConnectionPoolSizeOption when unset?

[ajayky-os]

Reads can be more than ConnectionPoolSizeOption, SDK itself can create more connections despite this option set when there are more concurrent reads https://docs.cloud.google.com/cpp/docs/reference/storage/latest/structgoogle_1_1cloud_1_1storage_1_1ConnectionPoolSizeOption. That is the reason to separate the threadpool size from connectionpoolsize entirely.

[v-pratap]

If TryAcquireHedgeToken() fails here, executing continue advances i and triggers another full delay_ wait (e.g. 500ms). Does this unintentionally burn one of our max_hedges_ attempt slots and delay subsequent hedges?