[FLINK-40200][Checkpoint] Checkpoint sync phase timeout

[Efrat19]

What is the purpose of the change

For TaskManagers locking up in rocksDB native calls during checkpoint sync phase, the best mitigation would be to cancel the task and restart. A lenient checkpoint timeout X {execution.checkpointing.tolerable-failed-checkpoints} means it can take days for JM to restart the job out of the deadlock. Setting checkpoint sync timeout, after which a task failure is triggered, will address thread bloackages faster and without manual intervention.

Brief change log

• Added an opt-in checkpoint sync phase timeout logic for aborting stuck checkpoint through task failure

Verifying this change

This change added tests and can be verified as follows:

Unit tests:

• SubtaskCheckpointCoordinatorTest.testSyncPhaseCompletedOnTime
• SubtaskCheckpointCoordinatorTest.testSyncPhaseWatchDogFailsStuckTask
  ITCase
• CheckpointSyncPhaseTimeoutITCase.testStuckSyncPhaseFailsJob

Does this pull request potentially affect one of the following parts:

• Dependencies (does it add or upgrade a dependency): no
• The public API, i.e., is any changed class annotated with no
• The serializers: no
• The runtime per-record code paths (performance sensitive): no
• Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
• The S3 file system connector: no

Documentation

• Does this pull request introduce a new feature? yes
• If yes, how is the feature documented? docs

---

Was generative AI tooling used to co-author this PR?

• Yes (please specify the tool below)

Co-Generated-by: claude-sonnet-5

[Efrat19]

@flinkbot run azure

[flinkbot]

CI report:

• de8c6f9 Azure: SUCCESS

Bot commands

The @flinkbot bot supports the following commands:

• @flinkbot run azure re-run the last Azure build

[Efrat19]

cc @rkhachatryan to ptal when you can