discard unused uct_ep when the pre-created uct_ep more than needed by changchengx · Pull Request #223 · yosefe/ucx

changchengx · 2021-11-16T08:08:51Z

What

keep NUM_PATHS compatible between server (e.g. UCX_IB_NUM_PATHS=1) and client (e.g. UCX_IB_NUM_PATHS=2)
This PR is mainly ported from:

changchengx · 2021-11-19T09:53:01Z

UCP: fail if the server cannot connect to all eps addrs from the client #31 is reverted
There's no uct_ep resource leak as being reported in https://redmine.mellanox.com/issues/2284793 when verifying it on local two server.
verify path compatible function on local two serves & every server has one NIC with two ports(HCA MT4125, 40Gbps x 2)

changchengx · 2021-11-23T08:15:19Z

@yosefe
Merge all the ucp_ep_config_lanes_intersect related functions in 422a29c

yosefe · 2022-01-12T11:13:34Z

/azp run

azure-pipelines · 2022-01-12T11:13:44Z

Azure Pipelines successfully started running 1 pipeline(s).

changchengx · 2022-01-13T14:04:38Z

How to run the below io_demo Test tag match on CX4 test case locally to check the problem?
https://dev.azure.com/ucfconsort/ucx-int3/_build/results?buildId=35022&view=logs&j=f6213e06-5fba-58e7-a041-fa7d17b490d4&t=0687ad30-25a0-56ac-90ea-d6a36c068c41&l=30

2022-01-12T11:54:39.4062959Z + python /hpc/noarch/git_projects/hpc-mtt-conf/scripts/iodemo_analyzer.py -d drop_35022/tag_cx4_rc --duration 600 -t 3 --allow_list integration3
2022-01-12T11:54:39.4523334Z ==============================
2022-01-12T11:54:39.4524537Z Client Errors:
2022-01-12T11:54:39.4525117Z ==============================
2022-01-12T11:54:39.4526337Z Log: drop_35022/tag_cx4_rc/iodemo_swx-rain02_client_00.log
2022-01-12T11:54:39.4527104Z Error iodemo analyzer: [1641988008.295210] [DEMO] timeout waiting for 1 replies on 1 connections
2022-01-12T11:54:39.4527679Z 
2022-01-12T11:54:39.4528179Z ==============================
2022-01-12T11:54:39.4528814Z Server Errors:
2022-01-12T11:54:39.4529401Z ==============================
2022-01-12T11:54:39.4530206Z Log: drop_35022/tag_cx4_rc/iodemo_swx-rain01_server_00.log
2022-01-12T11:54:39.4531328Z Error iodemo analyzer: [1641987928.295231] [swx-rain01:438277:0]         wireup.c:318  UCX  ERROR ep 0x7f2042f71000: no remote ep address for lane[0]->remote_lane[0]
2022-01-12T11:54:39.4532016Z 
2022-01-12T11:54:39.4633260Z ##[error]Bash exited with code '1'.
2022-01-12T11:54:39.4648668Z ##[section]Finishing: Analyze for tag_cx4_rc

yosefe · 2022-01-13T14:08:57Z

@changchengx the relevant script is https://github.com/yosefe/ucx/blob/integration3/buildlib/pr/io_demo/az-stage-io-demo.yaml

changchengx · 2022-01-20T05:15:08Z

io_demo Test tag match on CX4

1. reserve 2 swx-ucx servers: e.g. swx-ucx03 & swx-ucx04

2. Login in swx-ucx03

3. Build ucx/integration3 with PR in swx-ucx03

4. Start network corrupter in background
$ #Confirm whether it works swx-ucx03 & swx-ucx04
$ #Start network corrupter
[swx-ucx03]$ buildlib/az-network-corrupter.sh initial_delay=20 cycles=100 downtime=5 uptime=180 interface=bond0

5. Run below command on swx-ucx03 server
[swx-ucx03]$ export UCX_NET_DEVICES=mlx5_bond_0:1
[swx-ucx03]$ export UCX_TLS=rc_x
[swx-ucx03]$ export UCX_RNDV_THRESH=4k
[swx-ucx03]$ export LD_LIBRARY_PATH=drop_35176/install/lib:
[swx-ucx03]$ LD_LIBRARY_PATH=drop_35176/install/lib:
[swx-ucx03]$ test/apps/iodemo/run_io_demo.sh -H swx-ucx03,swx-ucx04 --tasks-per-node 1 --duration 600 -v --num-clients 1 --num-servers 1 --map-by slot --log-dir drop_35176/tag_cx4_rc -i bond0 drop_35176/install/bin/io_demo -d 512:524288 -P 2 -o read,write -i 0 -w 16 -t 60

6. Run below command on swx-ucx03 server
[swx-ucx03] python /hpc/noarch/git_projects/hpc-mtt-conf/scripts/iodemo_analyzer.py -d drop_35176/tag_cx4_rc --duration 600 -t 3 --allow_list integration3

7. Kill the corrupter

8. restore interface
[swx-ucx03]$ drop_35176/buildlib/az-network-corrupter.sh reset=yes interface=bond0

take care of the direcotry

drop_35176/install/bin/io_demo
--log-dir drop_35176/tag_cx4_rc
-d drop_35176/tag_cx4_rc

yosefe · 2022-01-23T11:21:45Z

/azp run

azure-pipelines · 2022-01-23T11:21:54Z

Azure Pipelines successfully started running 1 pipeline(s).

changchengx · 2022-02-03T14:19:31Z

@yosefe Is it possible to review this PR before fixing the CI self problem?

changchengx · 2022-02-10T08:24:36Z

CI has been passed.

yosefe · 2022-02-10T09:14:11Z

@dmitrygx can you pls review?

dmitrygx

@changchengx did you use the latest code from master?
I see that some bugs came from the initial implementation.
So, I suggest you align this PR with the master branch.

changchengx · 2022-02-15T14:36:26Z

@dmitrygx I've followed your 1st round CR comments to refine the implmentation. Please check it.

dmitrygx

shall we add ucp_worker_discard_uct_ep_cleanup to cleanup discarding hash in ucp_worker_destroy?

changchengx · 2022-02-21T06:14:33Z

shall we add ucp_worker_discard_uct_ep_cleanup to cleanup discarding hash in ucp_worker_destroy?

As we've talked offline, I'll add warning if the worker->discard_uct_ep_hash isn't empty to minimize the change when doing ucp_worker_destroy

changchengx · 2022-02-21T12:12:32Z

@yosefe Dmitry has approved this PR and he's in Hackathon these days.

changchengx · 2022-03-10T10:03:26Z

@yosefe Do you have any comments about this PR?

yosefe

can we add CI to test this?
this branch has minimal ci infrastructure to run io_demo

changchengx · 2022-03-29T08:34:05Z

can we add CI to test this? this branch has minimal ci infrastructure to run io_demo

@yosefe Do you mean to add test into io-demo.yaml or MTT?

yosefe · 2022-03-29T08:36:20Z

2 short tests in io-demo.yaml , without interference, with different num_paths (2->1 and 1->2)

changchengx · 2022-03-30T14:07:41Z

@yosefe Does azure CI stop working? There's no CI triggered here.
I've added two test cases to verify path compatible function.

yosefe · 2022-03-30T14:13:42Z

@changchengx see https://dev.azure.com/ucfconsort/ucx-int3/_build/results?buildId=40860&view=results

changchengx · 2022-03-31T01:14:12Z

Two test cases have been added.
Pass all the CI test cases.

yosefe · 2022-04-06T07:33:38Z

@changchengx pls squash

Signed-off-by: Changcheng Liu <jerrliu@nvidia.com>

changchengx force-pushed the smpath_int3 branch from bb99ed5 to 8a10332 Compare November 18, 2021 10:36

dmitrygx reviewed Nov 18, 2021

View reviewed changes

Comment thread src/ucp/wireup/wireup.c Outdated

changchengx force-pushed the smpath_int3 branch 3 times, most recently from 8a10332 to 95dc3d9 Compare November 19, 2021 03:50

changchengx changed the title ~~[WIP] implement NUM_PATHS compatible~~ discard unused uct_ep when the pre-created uct_ep more than needed Nov 19, 2021

changchengx force-pushed the smpath_int3 branch from 95dc3d9 to 8bb6068 Compare November 19, 2021 10:09

changchengx mentioned this pull request Jan 27, 2022

IODEMO: support setting UCX_IB_NUM_PATHS on cmd line #226

Closed

dmitrygx reviewed Feb 10, 2022

View reviewed changes

Repository owner deleted a comment from leibin2014 Feb 14, 2022

dmitrygx reviewed Feb 18, 2022

View reviewed changes

dmitrygx approved these changes Feb 18, 2022

View reviewed changes

dmitrygx suggested changes Mar 22, 2022

View reviewed changes

Comment thread src/ucp/wireup/wireup.c Outdated

Comment thread src/ucp/wireup/wireup.c Outdated

Comment thread src/ucp/wireup/wireup.c Outdated

Comment thread src/ucp/wireup/wireup.c Outdated

Comment thread src/ucp/wireup/wireup.c Outdated

Comment thread src/ucp/wireup/wireup.c Outdated

dmitrygx approved these changes Mar 23, 2022

View reviewed changes

yosefe reviewed Mar 27, 2022

View reviewed changes

Comment thread src/ucp/wireup/wireup.h

Comment thread src/ucp/wireup/wireup_cm.c Outdated

Comment thread src/ucp/wireup/wireup_cm.c Outdated

Comment thread src/ucp/wireup/wireup_cm.c Outdated

Comment thread src/ucp/wireup/wireup.c Outdated

yosefe reviewed Mar 27, 2022

View reviewed changes

Comment thread src/ucp/wireup/wireup.c Outdated

yosefe reviewed Apr 5, 2022

View reviewed changes

Comment thread buildlib/pr/io_demo/az-stage-io-demo.yaml Outdated

yosefe approved these changes Apr 6, 2022

View reviewed changes

changchengx force-pushed the smpath_int3 branch from 66d7842 to 065ca2c Compare April 6, 2022 07:44

yosefe reviewed Apr 6, 2022

View reviewed changes

Comment thread buildlib/pr/io_demo/io-demo.yml Outdated

Comment thread buildlib/pr/io_demo/io-demo.yml Outdated

UCP: develop path compatible function

71eb214

Signed-off-by: Changcheng Liu <jerrliu@nvidia.com>

changchengx force-pushed the smpath_int3 branch from 065ca2c to 71eb214 Compare April 6, 2022 08:08

yosefe approved these changes Apr 6, 2022

View reviewed changes

yosefe merged commit 1510f76 into yosefe:integration3 Apr 6, 2022

changchengx deleted the smpath_int3 branch April 6, 2022 13:47

Conversation

changchengx commented Nov 16, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Uh oh!

Uh oh!

changchengx commented Nov 19, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

changchengx commented Nov 23, 2021

Uh oh!

yosefe commented Jan 12, 2022

Uh oh!

azure-pipelines Bot commented Jan 12, 2022

Uh oh!

changchengx commented Jan 13, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yosefe commented Jan 13, 2022

Uh oh!

changchengx commented Jan 20, 2022

Uh oh!

yosefe commented Jan 23, 2022

Uh oh!

azure-pipelines Bot commented Jan 23, 2022

Uh oh!

changchengx commented Feb 3, 2022

Uh oh!

changchengx commented Feb 10, 2022

Uh oh!

yosefe commented Feb 10, 2022

Uh oh!

dmitrygx left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

changchengx commented Feb 15, 2022

Uh oh!

dmitrygx left a comment

Choose a reason for hiding this comment

Uh oh!

changchengx commented Feb 21, 2022

Uh oh!

changchengx commented Feb 21, 2022

Uh oh!

changchengx commented Mar 10, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

yosefe left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

changchengx commented Mar 29, 2022

Uh oh!

yosefe commented Mar 29, 2022

Uh oh!

changchengx commented Mar 30, 2022

Uh oh!

yosefe commented Mar 30, 2022

Uh oh!

changchengx commented Mar 31, 2022

Uh oh!

Uh oh!

yosefe commented Apr 6, 2022

Uh oh!

changchengx commented Nov 16, 2021 •

edited

Loading

changchengx commented Nov 19, 2021 •

edited

Loading

changchengx commented Jan 13, 2022 •

edited

Loading