Skip to content

Commit 7543e80

Browse files
fix(io): attempt every S3 delete and report how many failed (#966)
Follow-up to #898 (comment). `ArrowS3FileIO::DeleteFiles` returned at the first credential prefix whose delete failed, so files under the remaining prefixes were never attempted. It now attempts every file unless task submission fails, logs deletion failures, and returns `Failed to delete N of M files`. Exceptions thrown by individual deletes are counted too. - Deletion is sequential by default. `InitializeS3FileIODeleteExecutor(Executor*)` initializes a caller-owned executor shared by all S3 FileIOs, including registry-created instances. Initialize it before the first delete; otherwise, the first delete freezes the default `nullptr`. Repeating the same pointer is allowed, but replacing it is not. The executor must remain alive until the last delete completes. - `s3.delete.num-threads` limits worker tasks per call, defaulting to the hardware thread count. Workers share a file index, keeping task/future storage bounded by the configured parallelism. The shared executor controls total concurrency; this property does not create or resize a thread pool. - Accepted tasks finish before the call exits, including when submission returns an error or throws. Submission failures are returned without falling back to the calling thread and bypassing the executor's concurrency limit. All workers use the same delegate snapshot and the caller's logger. Arrow has no public S3 batch-delete API, so each request still deletes one file. `ResolvingFileIO` keeps its existing fail-fast behavior across delegates. Tests cover deletion across allowed/denied prefixes, thrown deletes, rejected/throwing submissions, bounded task counts and concurrency, immutable executor initialization, and executor configuration for both direct and registry-created FileIOs.
1 parent 274d1bb commit 7543e80

7 files changed

Lines changed: 553 additions & 25 deletions

File tree

‎mkdocs/docs/file-io.md‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -66,6 +66,7 @@ each file location's scheme.
6666
| `client.region` | `us-east-1` | Region to sign requests for |
6767
| `s3.endpoint` | `https://127.0.0.1:9000` | Endpoint to use instead of the AWS one. When absent, the `AWS_ENDPOINT_URL_S3` / `AWS_ENDPOINT_URL` environment variables are consulted |
6868
| `s3.path-style-access` | `true` | Address buckets as a path (`endpoint/bucket`) instead of a virtual host (`bucket.endpoint`). Only takes effect together with a custom endpoint |
69+
| `s3.delete.num-threads` | `8` | Maximum worker tasks per `DeleteFiles` call; defaults to the hardware thread count. Requires a delete executor |
6970
7071
The following keys are specific to iceberg-cpp; they are not part of the Java
7172
Iceberg or REST specification property set:
@@ -80,6 +81,29 @@ Without credentials, the AWS default credential chain is used, which covers
8081
environment variables, the shared configuration file, and the various role and
8182
identity providers.
8283
84+
### Bulk deletion
85+
86+
S3 `DeleteFiles` attempts each file and returns the failure count, including
87+
exceptions thrown by a delete. Deletion is sequential by default. To enable
88+
concurrency, call `iceberg::arrow::InitializeS3FileIODeleteExecutor(executor)`
89+
from `iceberg/arrow/arrow_io_util.h` before the first delete. The application
90+
owns the executor; the setting applies to direct and registry-created FileIOs.
91+
92+
Initialization is permanent. Repeating the same pointer is allowed, but changing
93+
it returns an error. The first delete freezes the default `nullptr` if no executor
94+
was initialized. Keep the executor alive until the last delete finishes, and do
95+
not start more deletes after destroying it.
96+
97+
`s3.delete.num-threads` limits workers within one call, not across concurrent
98+
calls. The shared executor controls total concurrency; this property does not
99+
create or resize a thread pool. Workers share the file list, so task and future
100+
storage is bounded by the configured parallelism.
101+
102+
`DeleteFiles` waits for accepted work before returning, even when submission
103+
fails or throws. A submission failure is returned without deleting on the caller.
104+
The executor must make progress while callers wait; do not call `DeleteFiles`
105+
from its workers unless it supports nested blocking work.
106+
83107
### S3-compatible storage
84108
85109
Stores that speak the S3 API are served by the same implementation. The scheme

‎src/iceberg/arrow/arrow_io_internal.h‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,8 +20,11 @@
2020
#pragma once
2121

2222
#include <cstddef>
23+
#include <functional>
2324
#include <memory>
25+
#include <mutex>
2426
#include <optional>
27+
#include <span>
2528
#include <string>
2629
#include <vector>
2730

@@ -30,9 +33,27 @@
3033

3134
#include "iceberg/file_io.h"
3235
#include "iceberg/iceberg_bundle_export.h"
36+
#include "iceberg/util/executor.h"
3337

3438
namespace iceberg::arrow {
3539

40+
// The first initialization or lookup fixes the executor for this state.
41+
class ICEBERG_BUNDLE_EXPORT DeleteExecutorState {
42+
public:
43+
Status Initialize(Executor* executor);
44+
Executor* Get();
45+
46+
private:
47+
std::mutex mutex_;
48+
bool initialized_ = false;
49+
Executor* executor_ = nullptr;
50+
};
51+
52+
// Attempt each file unless submission fails; always wait for accepted work.
53+
ICEBERG_BUNDLE_EXPORT Status BulkDeleteFiles(
54+
std::span<const std::string> file_locations, Executor* executor, size_t parallelism,
55+
const std::function<Status(const std::string&)>& delete_file);
56+
3657
/// \brief Open a FileIO input as an Arrow input stream.
3758
///
3859
/// Uses ArrowFileSystemFileIO's native Arrow stream directly when possible and falls

‎src/iceberg/arrow/arrow_io_util.h‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,7 @@
2929
#include "iceberg/file_io.h"
3030
#include "iceberg/iceberg_bundle_export.h"
3131
#include "iceberg/result.h"
32+
#include "iceberg/type_fwd.h"
3233

3334
namespace iceberg::arrow {
3435

@@ -47,6 +48,14 @@ ICEBERG_BUNDLE_EXPORT std::unique_ptr<FileIO> MakeLocalFileIO();
4748
ICEBERG_BUNDLE_EXPORT Result<std::unique_ptr<FileIO>> MakeS3FileIO(
4849
const std::unordered_map<std::string, std::string>& properties = {});
4950

51+
/// \brief Initialize the shared executor for S3 bulk deletes.
52+
///
53+
/// nullptr runs sequentially. The first delete freezes this default if not initialized.
54+
/// Repeating the same executor is allowed; replacing it is not. The caller owns it
55+
/// and must keep it alive until the last delete completes. Includes registry-created
56+
/// FileIOs. See Executor for blocking requirements.
57+
ICEBERG_BUNDLE_EXPORT Status InitializeS3FileIODeleteExecutor(Executor* executor);
58+
5059
/// \brief Finalize (clean up) the Arrow S3 subsystem.
5160
///
5261
/// Must be called before process exit if S3 was initialized, otherwise Arrow's

‎src/iceberg/arrow/s3/arrow_s3_file_io.cc‎

Lines changed: 155 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -18,13 +18,21 @@
1818
*/
1919

2020
#include <algorithm>
21+
#include <atomic>
22+
#include <cstdint>
2123
#include <cstdlib>
24+
#include <exception>
25+
#include <format>
26+
#include <functional>
27+
#include <future>
2228
#include <memory>
2329
#include <mutex>
2430
#include <optional>
2531
#include <shared_mutex>
32+
#include <span>
2633
#include <string>
2734
#include <string_view>
35+
#include <thread>
2836
#include <unordered_map>
2937
#include <utility>
3038
#include <vector>
@@ -39,12 +47,140 @@
3947
#include "iceberg/arrow/arrow_status_internal.h"
4048
#include "iceberg/arrow/s3/s3_properties.h"
4149
#include "iceberg/logging/log_macros.h"
50+
#include "iceberg/logging/logger.h"
51+
#include "iceberg/util/executor.h"
4252
#include "iceberg/util/macros.h"
4353
#include "iceberg/util/property_util.h"
4454
#include "iceberg/util/string_util.h"
4555

4656
namespace iceberg::arrow {
4757

58+
namespace {
59+
60+
struct DeleteTasks {
61+
std::vector<std::future<void>> futures;
62+
63+
~DeleteTasks() {
64+
for (auto& future : futures) {
65+
if (future.valid()) {
66+
future.wait();
67+
}
68+
}
69+
}
70+
};
71+
72+
// Submit may throw instead of returning an error; both reject the worker.
73+
Status SubmitWorker(Executor& executor, ExecutorTask task) {
74+
try {
75+
return executor.Submit(std::move(task));
76+
} catch (const std::exception& e) {
77+
return IOError("Submit threw an exception: {}", e.what());
78+
} catch (...) {
79+
return IOError("Submit threw an unknown exception");
80+
}
81+
}
82+
83+
DeleteExecutorState delete_executor;
84+
85+
} // namespace
86+
87+
Status DeleteExecutorState::Initialize(Executor* executor) {
88+
std::lock_guard lock(mutex_);
89+
if (initialized_) {
90+
if (executor_ != executor) {
91+
return InvalidArgument("Delete executor is already initialized");
92+
}
93+
return {};
94+
}
95+
executor_ = executor;
96+
initialized_ = true;
97+
return {};
98+
}
99+
100+
Executor* DeleteExecutorState::Get() {
101+
std::lock_guard lock(mutex_);
102+
initialized_ = true;
103+
return executor_;
104+
}
105+
106+
Status BulkDeleteFiles(std::span<const std::string> file_locations, Executor* executor,
107+
size_t parallelism,
108+
const std::function<Status(const std::string&)>& delete_file) {
109+
ICEBERG_PRECHECK(parallelism > 0, "Delete parallelism must be positive");
110+
auto logger = GetCurrentLogger();
111+
std::atomic<size_t> next = 0;
112+
std::atomic<size_t> failed = 0;
113+
auto work = [&] {
114+
ScopedLogger bind(logger);
115+
while (true) {
116+
const auto index = next.fetch_add(1, std::memory_order_relaxed);
117+
if (index >= file_locations.size()) {
118+
return;
119+
}
120+
const auto& location = file_locations[index];
121+
Status status;
122+
try {
123+
status = delete_file(location);
124+
} catch (const std::exception& e) {
125+
status = IOError("Delete threw an exception: {}", e.what());
126+
} catch (...) {
127+
status = IOError("Delete threw an unknown exception");
128+
}
129+
if (!status.has_value()) {
130+
failed.fetch_add(1, std::memory_order_relaxed);
131+
ICEBERG_LOG_WARN("Failed to delete {}: {}", location, status.error().message);
132+
}
133+
}
134+
};
135+
136+
std::optional<Error> submission_error;
137+
try {
138+
if (executor == nullptr) {
139+
work();
140+
} else {
141+
const auto workers = std::min(parallelism, file_locations.size());
142+
DeleteTasks tasks;
143+
tasks.futures.reserve(workers);
144+
size_t accepted = 0;
145+
for (; accepted < workers; ++accepted) {
146+
std::packaged_task<void()> task(work);
147+
// Submit may accept the task before failing.
148+
tasks.futures.push_back(task.get_future());
149+
ExecutorTask executor_task([task = std::move(task)]() mutable { task(); });
150+
if (auto status = SubmitWorker(*executor, std::move(executor_task));
151+
!status.has_value()) {
152+
submission_error = std::move(status.error());
153+
break;
154+
}
155+
}
156+
// A rejected worker's future is only waited for, by `tasks`.
157+
for (size_t i = 0; i < accepted; ++i) {
158+
tasks.futures[i].get();
159+
}
160+
}
161+
} catch (const std::exception& e) {
162+
return IOError("Bulk delete execution failed: {}", e.what());
163+
} catch (...) {
164+
return IOError("Bulk delete execution failed with an unknown exception");
165+
}
166+
if (submission_error.has_value()) {
167+
if (failed > 0) {
168+
submission_error->message += std::format("; failed to delete {} of {} files",
169+
failed.load(), file_locations.size());
170+
}
171+
return Status::unexpected_type(std::move(*submission_error));
172+
}
173+
if (failed > 0) {
174+
return IOError("Failed to delete {} of {} files", failed.load(),
175+
file_locations.size());
176+
}
177+
return {};
178+
}
179+
180+
Status InitializeS3FileIODeleteExecutor(Executor* executor) {
181+
return delete_executor.Initialize(executor);
182+
}
183+
48184
#if ICEBERG_S3_ENABLED
49185

50186
namespace {
@@ -181,9 +317,11 @@ std::string CanonicalizeS3Scheme(std::string_view location) {
181317
class ArrowS3FileIO final : public FileIO, public SupportsStorageCredentials {
182318
public:
183319
ArrowS3FileIO(std::shared_ptr<::arrow::fs::FileSystem> arrow_fs,
184-
std::unordered_map<std::string, std::string> default_properties)
320+
std::unordered_map<std::string, std::string> default_properties,
321+
size_t delete_parallelism)
185322
: default_file_io_(std::make_shared<ArrowFileSystemFileIO>(std::move(arrow_fs))),
186-
default_properties_(std::move(default_properties)) {}
323+
default_properties_(std::move(default_properties)),
324+
delete_parallelism_(delete_parallelism) {}
187325

188326
Result<std::unique_ptr<InputFile>> NewInputFile(std::string file_location) override;
189327

@@ -237,6 +375,7 @@ class ArrowS3FileIO final : public FileIO, public SupportsStorageCredentials {
237375

238376
std::shared_ptr<ArrowFileSystemFileIO> default_file_io_;
239377
std::unordered_map<std::string, std::string> default_properties_;
378+
const size_t delete_parallelism_;
240379
// Guards everything below; shared because reads happen per file operation.
241380
mutable std::shared_mutex mutex_;
242381
std::vector<StorageCredential> storage_credentials_;
@@ -338,41 +477,35 @@ Status ArrowS3FileIO::DeleteFile(const std::string& file_location) {
338477
}
339478

340479
Status ArrowS3FileIO::DeleteFiles(const std::vector<std::string>& file_locations) {
341-
// One snapshot so the whole batch matches the same delegate generation; only
342-
// ever a handful of delegates, so a linear scan beats hashing.
480+
// Keep one delegate snapshot for the whole batch.
343481
std::shared_ptr<ArrowFileSystemFileIO> fallback;
344482
DelegatesByPrefix by_prefix;
345483
{
346484
std::shared_lock lock(mutex_);
347485
fallback = default_file_io_;
348486
by_prefix = file_io_by_prefix_;
349487
}
350-
std::vector<std::pair<std::shared_ptr<ArrowFileSystemFileIO>, std::vector<std::string>>>
351-
locations_by_io;
352-
for (const auto& file_location : file_locations) {
353-
auto file_io = MatchDelegate(fallback, by_prefix, file_location);
354-
auto it = std::ranges::find_if(
355-
locations_by_io, [&](const auto& entry) { return entry.first == file_io; });
356-
if (it == locations_by_io.end()) {
357-
locations_by_io.emplace_back(std::move(file_io),
358-
std::vector<std::string>{file_location});
359-
} else {
360-
it->second.push_back(file_location);
361-
}
362-
}
363-
for (auto& [file_io, locations] : locations_by_io) {
364-
ICEBERG_RETURN_UNEXPECTED(file_io->DeleteFiles(locations));
365-
}
366-
return {};
488+
return BulkDeleteFiles(
489+
file_locations, delete_executor.Get(), delete_parallelism_,
490+
[&](const std::string& location) {
491+
return MatchDelegate(fallback, by_prefix, location)->DeleteFile(location);
492+
});
367493
}
368494

369495
} // namespace
370496

371497
Result<std::unique_ptr<FileIO>> MakeS3FileIO(
372498
const std::unordered_map<std::string, std::string>& properties) {
499+
size_t delete_parallelism = std::max(1u, std::thread::hardware_concurrency());
500+
if (const auto* value = FindProperty(properties, S3Properties::kDeleteNumThreads);
501+
value != nullptr) {
502+
ICEBERG_ASSIGN_OR_RAISE(auto count, StringUtils::ParseNumber<int32_t>(*value));
503+
ICEBERG_PRECHECK(count > 0, "{} must be positive", S3Properties::kDeleteNumThreads);
504+
delete_parallelism = static_cast<size_t>(count);
505+
}
373506
// Uses default credentials if properties are empty.
374507
ICEBERG_ASSIGN_OR_RAISE(auto fs, BuildArrowS3FileSystem(properties));
375-
return std::make_unique<ArrowS3FileIO>(std::move(fs), properties);
508+
return std::make_unique<ArrowS3FileIO>(std::move(fs), properties, delete_parallelism);
376509
}
377510

378511
Status FinalizeS3() {

‎src/iceberg/arrow/s3/s3_properties.h‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -54,6 +54,8 @@ struct S3Properties {
5454
static constexpr std::string_view kConnectTimeoutMs = "s3.connect-timeout-ms";
5555
/// Socket timeout in milliseconds
5656
static constexpr std::string_view kSocketTimeoutMs = "s3.socket-timeout-ms";
57+
/// Maximum workers per DeleteFiles call; defaults to the hardware thread count.
58+
static constexpr std::string_view kDeleteNumThreads = "s3.delete.num-threads";
5759
};
5860

5961
/// \brief URI schemes served by the Arrow S3 FileIO, lower-case.

0 commit comments

Comments
 (0)