Repository navigation
fix: retry Thrift OpenSession and server-rejected ExecuteStatement - #605
Open
vuanhphung wants to merge 2 commits into
Open
vuanhphung wants to merge 2 commits into
vuanhphung wants to merge 2 commits into
Conversation
Thrift OpenSession was never retried, so a single 503, 429, 502, 504 or 408 failed the connection. ExecuteStatement was retried only when the connection failed before the request was sent, so a 429 or a 503 with Retry-After (the server did not run the statement) failed too. The kernel backend retries all of these. Add OpenSession to the idempotent methods (a repeated open leaves at most an unused session), retry 408 on idempotent requests, and retry ExecuteStatement on 429 and on 503 with Retry-After. Other errors on ExecuteStatement are still not retried. UnsentRequestRetryPolicy is renamed NonIdempotentRetryPolicy. Co-authored-by: Isaac <no-reply@databricks.com> Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
vuanhphung
temporarily deployed
to
azure-prod
November 13, 2026 08:20 — with
GitHub Actions
Inactive
vuanhphung
temporarily deployed
to
azure-prod
November 13, 2026 08:20 — with
GitHub Actions
Inactive
vuanhphung
temporarily deployed
to
azure-prod
November 13, 2026 08:20 — with
GitHub Actions
Inactive
There was a problem hiding this comment.
Verdict: 1 Low
Looks good: one low concern about changelog wording. The retry classification is right: 429 and 503+Retry-After retry on ExecuteStatement, the ambiguous statuses don't, and OpenSession now uses the idempotent policy. Tests cover each case at the policy level and in the end-to-end connection tests. Follow-up work outside this PR: the existing Retry-After seconds-vs-milliseconds mismatch in getRetryAfterHeader, which the description already mentions.
Co-authored-by: Isaac <no-reply@databricks.com> Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
vuanhphung
temporarily deployed
to
azure-prod
November 13, 2026 08:26 — with
GitHub Actions
Inactive
vuanhphung
temporarily deployed
to
azure-prod
November 13, 2026 08:26 — with
GitHub Actions
Inactive
vuanhphung
temporarily deployed
to
azure-prod
November 13, 2026 08:26 — with
GitHub Actions
Inactive
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
On the Thrift backend a single transient HTTP error fails the connection or the statement, while the kernel (SEA) backend retries it:
OpenSessionis not inretryableThriftMethods, so a 503, 429, 502, 504 or 408 on session open is surfaced at once (1 attempt), andretriesTimeouthas no effect on it.ExecuteStatementis retried only when the connection fails before the request is sent (fix: retry ExecuteStatement when the connection fails before the request is sent #598). A 429, or a 503 withRetry-After, means the server did not run the statement, but it is still surfaced.HttpRetryPolicydoes not retry 408 on any request.Found by the cross-driver error-recovery spec (ERRORRECOV-001, 007–012): live runs against
main(7be705e) through the test proxy showed exactly 1 Thrift attempt in every case, while the SEA leg retried and passed.Change
OpenSessionto the idempotent methods. A repeated open leaves at most an unused session, which the server expires.HttpRetryPolicy(idempotent requests, CloudFetch and feature-flag fetches).ExecuteStatementnow also retries 429 and 503 withRetry-After, the same rule as the kernel's non-idempotent strategy (src/client/retry.rs). 500, 502, 504, 408 and 503 withoutRetry-Afterare still not retried, so a statement can't run twice.UnsentRequestRetryPolicyis renamedNonIdempotentRetryPolicy.All retries use the existing
retryMaxAttempts/retriesTimeoutbudget.Verification
npm test(1764 passing),npm run lint,npm run type-check,npm run prettier.Retry-After, no retry on 503 withoutRetry-After, 500/502/504/408, and RetryError on exhaustion.XPASS), and ERRORRECOV-013/014 (Reyden KP001 recovery) still pass.Not changed here:
getRetryAfterHeadercompares the header's seconds withretryDelayMinin milliseconds, so aRetry-Afterbelow 1000 seconds is ignored and the plain exponential backoff applies.This pull request and its description were written by Isaac.