UCP/WIREUP: Don't discard CM lane

[dmitrygx]

What

Don't discard CM lane.

Why ?

Don't schedule UCT CM lane to be discarded, since UCP EP will be destroyed due to peer failure and ucp_cm_disconnect_cb() could be invoked on async thread after UCP EP is destroyed and before UCT CM EP is destroyed from discarding functionality. So, UCP EP will passed as a corrupted argument to ucp_cm_disconnect_cb().
Fixes #5875

How ?

Add if check into the ucp_worker_iface_err_handle_progress() to understand whether the lane is CM or not:

• if it is a CM lane, purge pending operations by calling uct_ep_pending_purge() and destroy UCT EP in-place by calling uct_ep_destroy()
• if it is not a CM lane, schedule UCT EP to be discarded

[dmitrygx]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[dmitrygx]

The failure is unrelated
Similar to #5840

[dmitrygx]

@brminich @evgeny-leksikov @alinask could you review pls?

[dmitrygx]

@yosefe could you review pls?

[yosefe]

can this be related?

2020-11-07T14:45:30.4314575Z [     INFO ] Testing :::49294 Interface: eno1
2020-11-07T14:45:40.4414356Z /scrap/azure/agent-01/AZP_WORKSPACE/2/s/contrib/../test/gtest/uct/ib/test_sockaddr.cc:448: Failure
2020-11-07T14:45:40.4416016Z Value of: m_state & TEST_STATE_CONNECT_REQUESTED
2020-11-07T14:45:40.4417302Z   Actual: false
2020-11-07T14:45:40.4417884Z Expected: true
2020-11-07T14:45:49.7228411Z [  FAILED  ] sockaddr/test_uct_cm_sockaddr_multiple_cms.server_switch_cm/9, where GetParam() = sockaddr/eno1 (19299 ms)
2020-11-07T14:45:49.7229324Z [ RUN      ] sockaddr/test_uct_cm_sockaddr_multiple_cms.server_switch_cm/5 <sockaddr/ens4f1>

[dmitrygx]

can this be related?

@yosefe thank you for pointing me out
Will investigate the failure

[dmitrygx]

can this be related?

@yosefe thank you for pointing me out
Will investigate the failure

@yosefe I tried to reproduce this problem for a day. It did 1522554 iterations, but no luck to reproduce

[yosefe]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[changchengx]

port below change to yosefe#223 to simplify code logic
https://github.com/openucx/ucx/pull/5882/files#diff-68f50d5c8acc6ba66c30db814074690c7af4f019e8862ee6e8c320137746e85aR1088
https://github.com/openucx/ucx/pull/5882/files#diff-1a6cd9df9fc4c2ac2f3e662ca4ce0ff54e0ad405bab393e80090c8593745351cR2736