Add details on IPC cache - #1081
Conversation
ashesh2512
changed the title
|
@trey-ornl , @GeorgiadouAntigoni , @hagertnl - I'd like to talk about this at our next GPU MPI meeting (I'll forward the invite to @ashesh2512 ). The hope is that in the next MPI + ROCM release all the edge case memory leaks will be resolved and we can just strop back to the "here's how to enable the IPC signal cache and tune the IPC cache size" documentation. |
|
@trey-ornl Could you please give this another look? I have removed the incorrect environment variable from the PR. |
|
|
||
| Set to ``1`` by default if ``MPICH_GPU_SUPPORT_ENABLED=1``. This enables GPU IPC support for intra-node GPU-GPU communication operations. This variable has no effect if MPICH_GPU_SUPPORT_ENABLED is set to 0. Disabling IPC is known to noticeably impact intra-node MPI performance. For known issues with IPC enabled and suggested workarounds, please see :ref:`gpu-ipc-tips-tricks`. | ||
|
|
||
| ``GTL_ENABLE_HSA_IPC_SIGNAL_CACHE`` |
There was a problem hiding this comment.
Add to the title: "/ GTL_DISABLE_HSA_IPC_SIGNAL_CACHE"?
There was a problem hiding this comment.
I had that initially, but decided against it. Adding GTL_DISABLE_HSA_IPC_SIGNAL_CACHE in the heading made it seam like the two variables were interchangeable which could be misconstrued. If you feel it should definitely be there, happy to add it back in.
There was a problem hiding this comment.
How about this?
GTL_ENABLE_HSA_IPC_SIGNAL_CACHE (previously GTL_DISABLE_HSA_IPC_SIGNAL_CACHE)
I want to make sure that a search for either also shows the other one.
3 participants
This PR adds information on the combination of environment variables required to enable stable operation of IPC. Added best practices. Added a summary on known issues associated with IPC cache, and related workarounds. Also added a description on relevant environment variables.