Skip to content

test(csharp): retry EnableMultipleCatalogSupport metadata query on transient server 500 #615

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
eric-wang-1990 wants to merge 1 commit into
base: main
Choose a base branch
from
Open

test(csharp): retry EnableMultipleCatalogSupport metadata query on transient server 500 #615

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 46 additions & 8 deletions csharp/test/E2E/StatementTests.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1025,14 +1025,14 @@ private async Task TestMetadataQuery(AdbcConnection connection, string queryType
{
OutputHelper?.WriteLine($"Testing {queryType} with EnableMultipleCatalogSupport={shouldAllowMultipleCatalogs}, CatalogName={catalogName}");

var statement = connection.CreateStatement();
statement.SetOption(ApacheParameters.IsMetadataCommand, "true");
statement.SetOption(ApacheParameters.CatalogName, catalogName);
// Use default as schema name, it is the default schema name
statement.SetOption(ApacheParameters.SchemaName, "default");
statement.SqlQuery = queryType;

QueryResult queryResult = await statement.ExecuteQueryAsync();
// Retry the metadata query on a transient server-side 500. The SEA backend
// intermittently returns HTTP 500 INTERNAL_ERROR "The result files are not available
// in the result metadata" for metadata statements — a server-side result-staging race,
// not a driver or test defect (the identical query succeeds on the next attempt with a
// fresh statement). Without this, the shared-warehouse merge-queue run flakes on that
// 500. Only this specific transient error is retried; any other exception (and all the
// schema/row assertions below) propagate unchanged, so real regressions still fail.
QueryResult queryResult = await ExecuteMetadataQueryWithRetryAsync(connection, queryType, catalogName);
Assert.NotNull(queryResult.Stream);

// Store SPARK catalog schema for comparison
Expand Down Expand Up @@ -1164,6 +1164,44 @@ private async Task TestMetadataQuery(AdbcConnection connection, string queryType
}
}

/// <summary>
/// Executes a metadata query, retrying only on the transient server-side 500
/// "The result files are not available in the result metadata" (a SEA backend
/// result-staging race). Uses a fresh statement per attempt. Any other exception —
/// and success — returns/propagates immediately, so genuine failures are not masked.
/// </summary>
private async Task<QueryResult> ExecuteMetadataQueryWithRetryAsync(
AdbcConnection connection, string queryType, string catalogName)
{
const int maxAttempts = 4;
const string transientMarker = "result files are not available in the result metadata";

for (int attempt = 1; ; attempt++)
{
var statement = connection.CreateStatement();

@peco-review-bot peco-review-bot Bot Aug 3, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Low — The retry loop creates a fresh statement on every attempt but never disposes the failed ones. On a transient 500 the current statement is abandoned (the catch filter delays, then the loop continues and allocates a new statement) so up to 3 statements — each holding a server-side handle / HTTP resources — leak per invocation until GC. Much of this test file uses using var statement = connection.CreateStatement(); (e.g. lines 78, 840, 1321) for exactly this reason. Consider disposing the statement per attempt, e.g. wrap the body in using var statement = connection.CreateStatement(); inside the loop so each failed attempt is released before the backoff/retry.

statement.SetOption(ApacheParameters.IsMetadataCommand, "true");
statement.SetOption(ApacheParameters.CatalogName, catalogName);
// Use default as schema name, it is the default schema name
statement.SetOption(ApacheParameters.SchemaName, "default");
statement.SqlQuery = queryType;

try
{
return await statement.ExecuteQueryAsync();
}
catch (Exception ex) when (attempt < maxAttempts &&
ex.ToString().IndexOf(transientMarker, StringComparison.OrdinalIgnoreCase) >= 0)
{
// Exponential backoff: 500ms, 1s, 2s — gives the server time to stage results.
int delayMs = 500 * (1 << (attempt - 1));
OutputHelper?.WriteLine(
$"{queryType} (catalog={catalogName}) hit transient server 500 on attempt {attempt}/{maxAttempts}; " +
$"retrying in {delayMs}ms");
await Task.Delay(delayMs);
}
}
}

private void AssertField(Schema schema, int index, string expectedName, IArrowType expectedType, bool expectedNullable)
{
var field = schema.FieldsList[index];
Expand Down
Loading