Preservation bugfix to put PreserveExpiryTime whenever machine transitions to Failed phase by r4mek · Pull Request #1147 · gardener/machine-controller-manager

r4mek · 2026-09-01T09:59:00Z

What this PR does / why we need it:

Now we add the PreserveExpiryTime on the machine if the machine has annotation: node.machine.sapcloud.io/preserve: "when-failed" and it transitions to Failed phase.
For auto-preservation flow, we add the PreserveExpiryTime on the machine if the failed machine gets the annotation: node.machine.sapcloud.io/preserve: "auto-preserved"
Miscellaneous code fix:
- Ensure objects retrieved from the lister cache are never mutated directly.
- Machine fetch in UpdateMachineWithRetries() from the lister cache, which may return a stale object.
- On update failure, requeue the object for the next reconciliation rather than retrying with a potentially stale cached value.

Which issue(s) this PR fixes:
Fixes #

Special notes for your reviewer:

Release note:

Fixed the scenario where a machine to be preserved can for brief moment not have PreserveExpiryTime set.

gardener-prow · 2026-09-01T09:59:05Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign takoverflow for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

r4mek · 2026-09-01T10:00:24Z

/kind bug

thiyyakat

Thanks for the changes and the refactoring! I haven't looked at the full PR yet. I had one case in mind for which I thought the new code may not behave as expected. Once you address that, I will look at the rest as well.

thiyyakat · 2026-09-03T07:17:34Z

 				defer trackers.Stop()
 				waitForCacheSync(stop, c)
-				err := c.updateMachineAndMachineDeploymentDeletionAnnotations(context.TODO(), testMachineDeployment)
+				err := func() error {


This may be a stupid question, but what is the point of the closure here? Can you not simply have just L2361?

thiyyakat · 2026-09-03T07:30:34Z

+			node.Annotations[PreserveMachineAnnotationKey]; ok &&
+			AllowedPreserveAnnotationValues.Has(val) {
+			return val, true


Please print a warning here if the value is not valid.

thiyyakat · 2026-09-03T07:32:38Z

+	if val, ok :=
+		machine.Annotations[PreserveMachineAnnotationKey]; ok &&
+		AllowedPreserveAnnotationValues.Has(val) {
+		return val, true


Please log a warning if the value is not valid.

thiyyakat · 2026-09-03T07:36:29Z

+	preserveAnnotationValue, exists := machineutils.GetPreserveAnnotationValue(node, machine)
+	if !exists {
 		return
-	} else if getErr != nil {
-		if !apierrors.IsNotFound(getErr) {
-			err = getErr
-			return
-		}
-		klog.Warningf("Couldn't find node %q for machine %q", nodeName, machine.Name)
-		err = nil
 	}


Please correct me if I'm wrong, but what about the case where preservation was being manipulated by only the machine object's annotations? If a machine was preserved, and the annotation was deleted to indicate that preservation should be stopped, you will end up returning early here instead of stopping preservation.

Good catch! Addressed in the latest commit ab6bf44

gagan16k

Thanks for the changes! Have some comments, PTAL

gagan16k · 2026-09-06T09:13:53Z

-			machineDeepCopy.Annotations[machineutils.MarkedForDeletionTime] = tgd.markedMachineDeletionTimes[i]
-		}
-		_, err = dc.controlMachineClient.Machines(machine.Namespace).Update(ctx, machineDeepCopy, metav1.UpdateOptions{})
+		deletionTime := tgd.markedMachineDeletionTimes[i]


Do we need this variable deletionTime?

gagan16k · 2026-09-07T06:43:38Z

+			return mcd, err
+		}
+		// TODO: not neat. refactor later.
+		for _, machineList := range machineMap {


Can we key the machineMap by the UID of the machineSet here, might save a loop
Something like

if controllerRef := metav1.GetControllerOf(updatedMachine); controllerRef != nil { if machineList, ok := machineMap[controllerRef.UID]; ok { for i := range machineList.Items {

gagan16k · 2026-09-07T07:57:43Z

+				m.Status.CurrentStatus.PreserveExpiryTime = &metav1.Time{Time: metav1.Now().Add(c.safetyOptions.MachinePreserveTimeout.Duration)}
+			}
+			return nil
+		}, true)


As per docs

The PUT and POST verbs on objects MUST ignore the status values, to avoid accidentally overwriting the status in read-modify-write scenarios. A /status subresource MUST be provided to enable system components to update statuses of resources they manage.

On annotating machine (with mcm scaled to zero to avoid it reconciling) with these keys and dummy values

(⎈|garden--aws-ha-external:garden shoot--xxx--demo)➜ machine-controller-manager git:(pr/r4mek/1147) ✗ k patch mc shoot--xxx--demo-worker-cpu-z1-58c68-qjrl2 --type=merge -p '{ "metadata":{"annotations":{"node.machine.sapcloud.io/preserve":"auto-preserved"}}, "status":{"currentStatus":{"preserveExpiryTime":"2030-01-01T00:00:00Z"}} }' machine.machine.sapcloud.io/shoot--xxx--demo-worker-cpu-z1-58c68-qjrl2 patched (⎈|garden--aws-ha-external:garden shoot--xxx--demo)➜ machine-controller-manager git:(pr/r4mek/1147) ✗ k get mc shoot--xxx--demo-worker-cpu-z1-58c68-qjrl2 -oyaml | grep "annotations" -A 2 annotations: machinepriority.machine.sapcloud.io: "3" node.machine.sapcloud.io/preserve: auto-preserved (⎈|garden--aws-ha-external:garden shoot--xxx--demo)➜ machine-controller-manager git:(pr/r4mek/1147) ✗ k get mc shoot--xxx--demo-worker-cpu-z1-58c68-qjrl2 -oyaml | grep "preserveExpiryTime" (⎈|garden--aws-ha-external:garden shoot--xxx--demo)➜ machine-controller-manager git:(pr/r4mek/1147) ✗

As the "/status" subresource is not specified here, it drops the change on it. Maybe we would need two different patches for each change as they target different resource endpoints?.

gagan16k · 2026-09-07T08:30:33Z

-				})
+			updatedMachine, err := machineutils.PatchMachine(ctx, c.Machines(machine.Namespace), &machine, func(m *v1alpha1.Machine) error {
+				m.Labels = labelsutil.AddLabel(m.Labels, v1alpha1.DefaultMachineDeploymentUniqueLabelKey, hash)
+				return nil


This will force a reconcile instead of retry on returning 409 errors, so the unit test for LDRCBST could flake even more now. Might need to find another way to correct that.

gagan16k · 2026-09-07T09:46:24Z

+	// Deep-copy to avoid mutating cache objects downstream.
+	for i, m := range filteredMachines {
+		filteredMachines[i] = m.DeepCopy()
+	}
+


Great change, but I was thinking if we could apply this while creating the filteredMachines slices, in claimMachines().
Argument for: Saves extra work on the slice, and does it in the same loop while the slice is getting constructed.
Argument against: The other callers for the function, do not seem to write to the slice like this reconciler does, and thus do not really need a deep copy. WDYT?

gagan16k · 2026-09-07T10:32:53Z

+		machine.Name,
+		machineutils.PreserveMachineAnnotationKey,
+		machine.Annotations[machineutils.PreserveMachineAnnotationKey],
+		machine.Annotations[v1alpha1.NodeLabelKey],


Suggested change

machine.Annotations[v1alpha1.NodeLabelKey],

machine.Labels[v1alpha1.NodeLabelKey],

gagan16k · 2026-09-07T10:41:02Z

-	return nil
-}
-
 func (c *controller) findEffectivePreserveValue(machine *v1alpha1.Machine) (string, error) {


This function is redundant with the introduction of GetPreserveAnnotationValue()

gagan16k · 2026-09-07T10:46:16Z

+		if val, ok :=
+			node.Annotations[PreserveMachineAnnotationKey]; ok &&
+			AllowedPreserveAnnotationValues.Has(val) {


Suggested change

if val, ok :=

node.Annotations[PreserveMachineAnnotationKey]; ok &&

AllowedPreserveAnnotationValues.Has(val) {

if val, ok := node.Annotations[PreserveMachineAnnotationKey]; ok && AllowedPreserveAnnotationValues.Has(val) {

Not a strong opinion, but I found this kind of confusing to read the first time. But that could be just me, feel free to ignore this comment.

gagan16k · 2026-09-07T10:48:17Z

-		klog.V(4).Infof("Machine %s precondition doesn't hold, skip updating it.", name)
-		retryErr = nil
 	}
+	return machineClient.Patch(ctx, machine.Name, types.MergePatchType, patch, metav1.PatchOptions{}, subresources...)


Suggested change

return machineClient.Patch(ctx, machine.Name, types.MergePatchType, patch, metav1.PatchOptions{}, subresources...)

return machineClient.Patch(ctx, machine.Name, types.StrategicMergePatchType, patch, metav1.PatchOptions{}, subresources...)

gagan16k · 2026-09-07T10:58:09Z

+	patch, err := strategicpatch.CreateTwoWayMergePatch(base, modifiedJSON, v1alpha1.Machine{})
+	if err != nil {
+		return nil, err
+	}


Suggested change

}

}

if string(patch) == "{}" {

return machine, nil

}

We could short-circuit if the patch is empty for whatever reason, as the request will still be considered valid by the API server

bugfix

37c2482

r4mek requested a review from a team as a code owner September 1, 2026 09:59

gardener-prow Bot added do-not-merge/needs-kind Indicates a PR lacks a `kind/foo` label and requires one. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Sep 1, 2026

gardener-prow Bot added kind/bug Bug and removed do-not-merge/needs-kind Indicates a PR lacks a `kind/foo` label and requires one. labels Sep 1, 2026

make check

0126245

thiyyakat reviewed Sep 3, 2026

View reviewed changes

r4mek marked this pull request as draft September 3, 2026 09:40

gardener-prow Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Sep 3, 2026

introducing PatchMachine()

ab6bf44

r4mek marked this pull request as ready for review September 4, 2026 07:19

gardener-prow Bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Sep 4, 2026

gagan16k requested a review from thiyyakat September 6, 2026 16:26

gagan16k reviewed Sep 7, 2026

View reviewed changes

	machine.Annotations[v1alpha1.NodeLabelKey],
	machine.Labels[v1alpha1.NodeLabelKey],

	return machineClient.Patch(ctx, machine.Name, types.MergePatchType, patch, metav1.PatchOptions{}, subresources...)
	return machineClient.Patch(ctx, machine.Name, types.StrategicMergePatchType, patch, metav1.PatchOptions{}, subresources...)

-	}
+	}
+	if string(patch) == "{}" {
+		return machine, nil
+	}

Uh oh!

Conversation

r4mek commented Sep 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gardener-prow Bot commented Sep 1, 2026

Uh oh!

r4mek commented Sep 1, 2026

Uh oh!

thiyyakat left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gagan16k left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gagan16k Sep 7, 2026 • edited by thiyyakat Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

r4mek commented Sep 1, 2026 •

edited

Loading

gagan16k Sep 7, 2026 •

edited by thiyyakat

Loading