split NextHop fiback compliance testcases

[mudhireddy]

splitting the NextHop FibAck test cases when there is no NextHopGroup referencing it into a separate test suite. plan is to create a new feature profile TE compliance test suite with these split testcases.

context for this split is: openconfig/featureprofiles#4197

[robshakir]

Is there a reason to split this using this approach vs. using a new field in the Test that indicates some deviation?

[mudhireddy]

Thanks Rob for the review.

Is there a reason to split this using this approach vs. using a new field in the Test that indicates some deviation?

using deviation to mask the non-compliance failures is not recommended as per openconfig/featureprofiles#4197 (review)
and it kind of makes sense. Plan is to add a new feature profile test suite that would run the FibAckComplianceTestSuite and have visibility into failures rather than masking with deviation.

[gggsmith]

Thanks Rob for the review.

Is there a reason to split this using this approach vs. using a new field in the Test that indicates some deviation?

using deviation to mask the non-compliance failures is not recommended as per openconfig/featureprofiles#4197 (review) and it kind of makes sense. Plan is to add a new feature profile test suite that would run the FibAckComplianceTestSuite and have visibility into failures rather than masking with deviation.

Just for visibility, I don't think changing gribigo here is the right approach -openconfig/featureprofiles#4197 (comment)

[AmrNJ]

Hey @mudhireddy , @gggsmith's ask here is to split the test suite in compliance.go to two where NextHop fiback compliance will be added to the latter suite.
TE-15.1 will test the first test suite and TE-15.2 will test the second test suite (with new deviation included in the code).

[AmrNJ]

Hi @mudhireddy , Please add a test file for the newly created fiback_compliance.go file.
@robshakir / @gggsmith Please can you checka and give your approvals?

[robshakir]

Can someone explain to me why this is required please? Why are these special cases?

[AmrNJ]

@robshakir There is a PR4197 skip fiback verification subtests for unreferenced NH, NHG aft entries for an unsupported vendor. @gggsmith suggested that TE-15.1 be split into two tests one for NextHop fiback compliance and rest would all be part of the existing TE-15.1. Comment

[robshakir]

Let's discuss this internally, I'd like to understand why there are differences in implementation behaviour here, how they are being tracked, and what the roadmap for them is.