Skip to content

troubleshooting.md

Latest commit

 

History

History
252 lines (186 loc) · 6.41 KB

troubleshooting.md

File metadata and controls

252 lines (186 loc) · 6.41 KB
Raw

🛠️ NVBandwidth Troubleshooting Guide (FAQ)

This guide helps diagnose and resolve common issues encountered when using NVBandwidth. Each question provides error messages, root causes, verification steps, and solutions.

📋 What are the system requirements for NVBandwidth?

Multi-node requirements:

  • CUDA Toolkit 12.3 & driver and above
  • IMEX setup
  • Proper MPI configuration

Single-node requirements:

  • CUDA Toolkit 11.x or above

CUDA Version Notes:

  • Multi-node features were introduced in CUDA 12.3
  • Using earlier versions will limit functionality to single-node tests
  • When building for multi-node:
# Ensure CUDA 12.3+ is in use
$ nvcc --version

# Build with multi-node support
$ cmake -DMULTINODE=1
❌ I'm getting "CUDA_ERROR_NO_DEVICE" - no CUDA device detected. How do I fix this?

Error Message:

[CUDA_ERROR_NO_DEVICE] no CUDA-capable device is detected in expression cuInit(0)

Root Cause: This error occurs when:

  • NVIDIA driver is not installed
  • GPU(s) are not properly detected
  • CUDA driver mismatch with the installed CUDA toolkit
  • Hardware issues

Verification Steps:

  1. Check if GPU is detected:
$ nvidia-smi
  1. Verify driver installation:
# Check driver version
$ cat /proc/driver/nvidia/version

# Check loaded modules
$ lsmod | grep nvidia

# Check driver installation
$ dpkg -l | grep nvidia-driver
  1. Check CUDA setup:
# Check CUDA driver version
$ nvidia-smi | grep "CUDA Version"

# Check CUDA runtime version
$ nvcc --version

You can consult this page for CUDA driver and toolkit version compatibility.

Solution:

  1. Install/reinstall NVIDIA driver/CUDA toolkit

Notes:

  • Consider running nvidia-bug-report for detailed diagnostics:
$ nvidia-bug-report.sh
⚠️ I'm getting "CUDA_ERROR_SYSTEM_NOT_READY" error. What does this mean?

Error Message:

[CUDA_ERROR_SYSTEM_NOT_READY] system not yet initialized

Root Cause: On multi-GPU or multi-node systems this error generally indicates your system requires the NVLink/NVSwitch fabric manager, and you have not installed it.

Solution: Install and start nvidia-fabricmanager: Instructions

🔧 I'm getting "cudaErrorUnsupportedPtxVersion" when building. How do I fix this?

Error Message:

[cudaErrorUnsupportedPtxVersion] the provided PTX was compiled with an unsupported toolchain

Root Cause: This error means that the PTX code you are attempting to run was generated by a CUDA toolkit version that is newer than what your currently installed NVIDIA driver supports. The driver's JIT compiler cannot understand or process the PTX generated by a more recent toolchain.

Solution: Ensure compatibility between your CUDA driver and the CUDA toolkit used for compilation.

  1. Clean build:
rm -rf CMakeCache.txt CMakeFiles
make clean
  1. Set CUDA version (replace cuda-12.8 with your installed version):
export PATH=/usr/local/cuda-12.8/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda-12.8/lib64:$LD_LIBRARY_PATH
  1. Check driver compatibility:
# Check NVIDIA driver version
nvidia-smi

# Check CUDA driver compatibility
nvidia-smi | grep "CUDA Version"

# Verify CUDA runtime vs driver compatibility
nvcc --version
  1. Rebuild with explicit CUDA version
🏗️ I'm getting "Unsupported gpu architecture" error during compilation. What should I do?

Error Message:

nvcc fatal   : Unsupported gpu architecture 'compute_52'

Solution: Use CUDA < 13.0 for older architectures (Volta, Maxwell, or Pascal):

# For Hopper
cmake -DCMAKE_CUDA_ARCHITECTURES=sm_90
🔗 I'm getting "CUDA_ERROR_NVLINK_UNCORRECTABLE" error. How do I diagnose NVLink issues?

Error Message:

[CUDA_ERROR_NVLINK_UNCORRECTABLE] uncorrectable NVLink error

Verification:

# Check NVLink status
nvidia-smi nvlink -s

Output should show active for each NVLink, inactive or missing links may indicate a problem.

# Check topology
nvidia-smi topo -m
🚫 I'm getting "CUDA_ERROR_NOT_PERMITTED" during multi-node runs. How do I fix IMEX issues?

Error Message:

$ mpirun -np 2 -N 2 ./nvbandwidth -t multinode_device_to_device_bidirectional_memcpy_write_sm
[CUDA_ERROR_NOT_PERMITTED] operation not permitted in expression cuMemCreate(&handle, roundedUpAllocationSize, &prop, 0 )

Root Cause: Missing or improper IMEX channel setup, required for multi-node runs.

Solution:

  1. Start imex daemon and set up IMEX channels. The creation of the /dev/ nodes /dev/nvidia-caps-imex-channels/channelN must be handled by the administrator, where N is the minor number.
$ sudo systemctl start nvidia-imex
# Query the major number
$ cat /proc/devices | grep nvidia-caps-imex-channels
# Create the IMEX channels
$ sudo mkdir /dev/nvidia-caps-imex-channels/
$ sudo mknod /dev/nvidia-caps-imex-channels/channel0 c <major number> 0 

# Check IMEX service status
$ systemctl status nvidia-imex

Also refer to: IMEX Setup Guide

📚 I'm getting "libmpi_cxx.so.40 => not found" error. How do I install MPI?

Error Message:

$ ldd nvbandwidth
libmpi_cxx.so.40 => not found

Solution: Install OpenMPI development packages (package names vary by distribution):

Ubuntu/Debian:

sudo apt-get install openmpi-bin libopenmpi-dev

Red Hat/CentOS/Fedora:

# RHEL/CentOS 7/8
sudo yum install openmpi-devel

# Fedora
sudo dnf install openmpi-devel
📖 Where can I find additional resources and documentation?

Additional Resources: