-
Notifications
You must be signed in to change notification settings - Fork 4.7k
xds: suppress the duplicate errors at the client side #9185
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Changes from 4 commits
35d57a6
0e3ce43
82cbfbb
0306ac0
9db08e2
bc6a797
2e39651
c86b1c1
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -376,8 +376,20 @@ func (a *authority) handleADSResourceUpdate(serverConfig *ServerConfig, rType Re | |
| ServerURI: serverConfig.ServerIdentifier.ServerURI, ResourceType: rType.TypeName, | ||
| }) | ||
| } | ||
| state.md.ErrState = md.ErrState | ||
| isDuplicateErr := state.md.ErrState != nil && state.md.ErrState.Err != nil && state.md.ErrState.Err.Error() == uErr.Err.Error() | ||
| var errState *xdsresource.UpdateErrorMetadata | ||
| if md.ErrState != nil { | ||
| errState = &xdsresource.UpdateErrorMetadata{ | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Why do we need to do this ? Can't we directly do
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I dont think that is the right way to go. We should not change the error stored. We can instead compare the error in state with the ms error like :
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We cannot compare
Then the next update brings a duplicate error for the
As an alternative I can try to check if the concatenated error in the
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I have changed the logic to look for a duplicate resource error in the concatenated error ( |
||
| Version: md.ErrState.Version, | ||
| Err: uErr.Err, | ||
| Timestamp: md.ErrState.Timestamp, | ||
| } | ||
| } | ||
|
eshitachandwani marked this conversation as resolved.
Outdated
|
||
| state.md.ErrState = errState | ||
| state.md.Status = md.Status | ||
| if isDuplicateErr { | ||
| continue | ||
| } | ||
| for watcher := range state.watchers { | ||
| watcher := watcher | ||
| err := uErr.Err | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -22,6 +22,7 @@ import ( | |
| "context" | ||
| "fmt" | ||
| "strings" | ||
| "sync/atomic" | ||
| "testing" | ||
| "time" | ||
|
|
||
|
|
@@ -506,3 +507,97 @@ func (s) TestADS_ACK_NACK_ResourceIsNotRequestedAnymore(t *testing.T) { | |
| t.Fatal(err) | ||
| } | ||
| } | ||
|
|
||
| // TestADS_NACKError_DuplicateSuppression verifies that when a duplicate | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Can we also add a test case for the scenario you described in an earlier comment to make sure that it works for it too.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. added a test case that covers the scenario described here |
||
| // invalid LDS resource is sent by the server, the client suppresses the | ||
| // duplicate error notification to the watcher. | ||
| // | ||
| // Note: This behavior (suppress duplicate error notifications to watchers) was | ||
| // originally introduced to handle invalid EDS resources (missing localities) as | ||
|
mswierq marked this conversation as resolved.
Outdated
|
||
| // detailed in https://github.com/grpc/grpc-go/issues/8994. Since the duplicate | ||
| // suppression mechanism is generic and implemented at the authority/resource-independent | ||
| // level, it is verified here using LDS to avoid registering additional mocked | ||
| // resource types. | ||
| func (s) TestADS_NACKError_DuplicateSuppression(t *testing.T) { | ||
| nackReceivedCh := testutils.NewChannelWithSize(1) | ||
| var nackCount atomic.Int32 | ||
| mgmtServer := e2e.StartManagementServer(t, e2e.ManagementServerOptions{ | ||
| OnStreamRequest: func(_ int64, req *v3discoverypb.DiscoveryRequest) error { | ||
| if req.GetTypeUrl() != xdsresource.V3ListenerURL { | ||
| return nil | ||
| } | ||
| if req.GetErrorDetail() != nil { | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Can we add a comment here as to what it is that we are doing ?
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. added |
||
| if count := nackCount.Add(1); count >= 3 { | ||
| nackReceivedCh.Send(nil) | ||
|
mswierq marked this conversation as resolved.
Outdated
|
||
| } | ||
| } | ||
| return nil | ||
| }, | ||
| }) | ||
|
|
||
| const listenerName = "listener" | ||
| nodeID := uuid.New().String() | ||
|
|
||
| // Create an xDS client pointing to the above server. | ||
| configs := map[string]grpctransport.Config{"insecure": {Credentials: insecure.NewBundle()}} | ||
| client := createXDSClient(t, mgmtServer.Address, nodeID, grpctransport.NewBuilder(configs)) | ||
|
|
||
| // Use a custom watcher that counts invocations. | ||
| errCh := testutils.NewChannelWithSize(1) | ||
| watcher := &countingListenerWatcher{errCh: errCh} | ||
| ldsCancel := client.WatchResource(xdsresource.V3ListenerURL, listenerName, watcher) | ||
| defer ldsCancel() | ||
|
|
||
| resources := e2e.UpdateOptions{ | ||
| NodeID: nodeID, | ||
| Listeners: []*v3listenerpb.Listener{badListenerResource(t, listenerName)}, | ||
| SkipValidation: true, | ||
| } | ||
| ctx, cancel := context.WithTimeout(context.Background(), defaultTestTimeout) | ||
| defer cancel() | ||
| if err := mgmtServer.Update(ctx, resources); err != nil { | ||
| t.Fatalf("Failed to update management server with resources: %v, err: %v", resources, err) | ||
| } | ||
|
|
||
| // Verify that the watcher receives the initial error. | ||
| v, err := errCh.Receive(ctx) | ||
| if err != nil { | ||
| t.Fatalf("timeout when waiting for an endpoints resource error from the management server: %v", err) | ||
| } | ||
| gotErr := v.(error) | ||
| wantNackErr := "no RouteSpecifier" | ||
| if !strings.Contains(gotErr.Error(), wantNackErr) { | ||
| t.Fatalf("update received with error: %v, want %q", gotErr, wantNackErr) | ||
| } | ||
|
|
||
| // Wait for the management server to receive at least 3 NACKs, indicating that | ||
| // the NACK resend loop has run multiple times. | ||
| if _, err := nackReceivedCh.Receive(ctx); err != nil { | ||
| t.Fatalf("timeout waiting for 3 NACKs to be received by the server: %v", err) | ||
| } | ||
|
|
||
| // Verify that the count of error callbacks is still exactly 1, proving that | ||
| // subsequent duplicate error updates were suppressed. | ||
| if count := watcher.errCount.Load(); count != 1 { | ||
| t.Fatalf("Error callback invoked %d times, want 1", count) | ||
| } | ||
| } | ||
|
|
||
| type countingListenerWatcher struct { | ||
| errCh *testutils.Channel | ||
| errCount atomic.Int32 | ||
| } | ||
|
|
||
| func (*countingListenerWatcher) ResourceChanged(_ xdsclient.ResourceData, done func()) { | ||
| done() | ||
| } | ||
| func (c *countingListenerWatcher) ResourceError(err error, done func()) { | ||
| c.errCount.Add(1) | ||
| c.errCh.Send(err) | ||
| done() | ||
| } | ||
| func (c *countingListenerWatcher) AmbientError(err error, done func()) { | ||
| c.errCount.Add(1) | ||
| c.errCh.Send(err) | ||
| done() | ||
| } | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also instead of evaluating this as a variable and then checking can we instead do the assignment first(which I assume needs to be done for either case) and then directly in an
ifblock we can evaluate the condition and callcontinueon true. WDYT ?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
stateworks as an cache of the previously received error for the resource, so trying to find out if there was an error duplication after updatingstatewould always returntrueThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Right! but we can do the check first and if true call continue, because if we are getting the same error, we might not need to update the state? Or can the status be different ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The stored metadata
ErrStatemay differ, this is why we need to update it even if a duplicate error for a resource is detected. This is also the reason why the duplicate check has to be done before updating it. Please, look at the comment with an example that I made: #9185 (comment)