Ubuntu 22.04 + Linux 6.8 内核 + ConnectX-6 网卡 MLNX OFED 驱动安装

基础信息

• 系统：Ubuntu 22.04.5 LTS
• 内核：6.8.0-xx-generic
• 网卡：Mellanox ConnectX6 VPI 双端口（200Gb/s）
• 驱动：OFED 24.04 及以上（推荐 24.10）
• 核心场景：双 H200 显卡多卡训练、高速 RDMA 通信

报错：oaflg cullt graonwlz blson lianl route 3 dew chrsatiythoni distutils iibltdl dev libgfortran5 Renoving old packages… Installing new packages Installing ofed-scripts-23.04… InstaL1Lng mnx-tools-23.04..。 InstalLing mlnx-ofed-kernel-utils-23.04..。 InstalLing mnx-ofed-kernel-dkAs-23.04.。. Failed to install ma-ofed-kermel0hm DE Colecting debug nfo…See /tP/MLNX OFED LINUX,12315,l0g5/nlnx-ofed-kernel-dkns,debinstall.lorootgamax:/hOM0/9M5X/0eSktOP/HLNX OFED LINUX-23.04-1.1.3.6-ubuntu22.64-x0 an上intarima-C0re mln-ofed allReadLng package LLsts… DoneBuiLding dependency tree..: DoneReallng state infornatLon..。 DoneE:unabe to locate package ninx-ofedE:bnable to iocate package allrooteamax:/home/anax/DesKtOp/MLNX OFED LINUX-23.04-1.1.3.0-ubuntu2

核心兼容性原因：

• OFED 23.x 发布时 Linux 6.8 内核尚未推出

• ConnectX-6 网卡新固件需要 OFED 24.x 才能发挥最佳性能

• RDMA 优化对 AI 训练（如 H200 多卡通信）至关重要

 安装前需要检查什么？

# 1. 检查内核版本

uname -r # 应该显示: 6.8.0-xx-generic

# 2. 检查网卡硬件状态

lspci | grep Mellanox lspci -vvv | grep -A 30 "ConnectX-6" | grep -E "Link Width|Speed"

# 3. 检查现有驱动 dpkg -l | grep mlnx dkms statu

 如何正确安装 MLNX OFED 24.10

下载网址：

Linux InfiniBand Drivers

apt install linux-headers-$(uname -r) # 确保新内核的头文件已安装
systemctl restart openibd # 重启驱动服务即可适配
# 1. 解压安装包
tar -xzf MLNX_OFED_LINUX-24.10-3.2.5.0-ubuntu22.04-x86_64.tgz

# 2. 进入目录
cd MLNX_OFED_LINUX-24.10-3.2.5.0-ubuntu22.04-x86_64

# 3. 执行安装（推荐完整安装）
sudo ./mlnxofedinstall
root@user/user/MLNX_OFED_LINUX-24.10-3.2.5.0-ubuntu22.04-x86_64# ./mlnxofedinstall –all
Logs dir: /tmp/MLNX_OFED_LINUX.1695372.logs
General log file: /tmp/MLNX_OFED_LINUX.1695372.logs/general.log

Below is the list of MLNX_OFED_LINUX packages that you have chosen
(some may have been added by the installer due to package dependencies):

ofed-scripts
mlnx-tools
mlnx-ofed-kernel-utils
mlnx-ofed-kernel-dkms
iser-dkms
isert-dkms
srp-dkms
rdma-core
libibverbs1
ibverbs-utils
ibverbs-providers
libibverbs-dev
libibverbs1-dbg
libibumad3
libibumad-dev
ibacm
librdmacm1
rdmacm-utils
librdmacm-dev
ibdump
libibmad5
libibmad-dev
libopensm
opensm
opensm-doc
libopensm-devel
libibnetdisc5
infiniband-diags
mft
kernel-mft-dkms
perftest
ibutils2
ibsim
ibsim-doc
ucx
sharp
hcoll
knem-dkms
knem
openmpi
mpitests
xpmem-dkms
xpmem
libxpmem0
libxpmem-dev
dpcp
srptools
mlnx-ethtool
mlnx-iproute2
rshim
ibarr

This program will install the MLNX_OFED_LINUX package on your machine.
Note that all other Mellanox, OEM, OFED, RDMA or Distribution IB packages will be removed.
Those packages are removed due to conflicts with MLNX_OFED_LINUX, do not reinstall them.

Do you want to continue?[y/N]:y

到这里输入y

……

Found 1 device(s) requiring firmware update…

Device #1: Updating FW …
FSMST_INITIALIZE – OK
FSMST_INITIALIZE – OK 0%
Writing Boot image component – OK
Writing Boot image component – Done

Restart needed for updates to take effect.
Log File: /tmp/PTLDZePWvF
Writing Boot image component – OK

Restart needed for updates to take effect.
Log File: /tmp/AKiNLWUx1i
Real log file: /tmp/MLNX_OFED_LINUX.1695372.logs/fw_update.log
Device (71:00.0):
71:00.0 Infiniband controller: Mellanox Technologies MT28908 Family [ConnectX-6]
Link Width: x16
PCI Link Speed: 16GT/s

Device (71:00.1):
71:00.1 Infiniband controller: Mellanox Technologies MT28908 Family [ConnectX-6]
Link Width: x16
PCI Link Speed: 16GT/s

Device (b1:00.0):
b1:00.0 Infiniband controller: Mellanox Technologies MT28908 Family [ConnectX-6]
Link Width: x16
PCI Link Speed: 16GT/s

Device (b1:00.1):
b1:00.1 Infiniband controller: Mellanox Technologies MT28908 Family [ConnectX-6]
Link Width: x16
PCI Link Speed: 16GT/s

Installation passed successfully
To load the new driver, run:
/etc/init.d/openibd restart
看到这里就安装顺利了
systemctl status openibd
○ openibd.service – openibd – configure Mellanox devices
Loaded: loaded (/lib/systemd/system/openibd.service; enabled; vendor preset: enabled)
Active: inactive (dead)
Docs: file:/etc/infiniband/openib.conf
# lsmod | grep mlx
mlx_compat 20480 1 ib_core
没识别到ib网口
可以检查下硬件安装 或者重启下服务

# /etc/init.d/openibd restart
Unloading HCA driver: [ OK ]
Loading HCA driver and Access Layer: [ OK ]

2 张 ConnectX6 VPI 双端口网卡（共 4 个 200Gb/s 端口）已完全识别，驱动模块（mlx5_core/mlx5_ib等）加载正常

lsmod | grep mlx # 检查核心驱动模块是否加载（应有mlx5_core、mlx5_ib等输出）
mlx5_ib 548864 0
ib_uverbs 200704 2 rdma_ucm,mlx5_ib
mlx5_core 2818048 1 mlx5_ib
mlxfw 36864 1 mlx5_core
psample 16384 1 mlx5_core
mlxdevm 188416 1 mlx5_core
macsec 77824 1 mlx5_ib
tls 155648 1 mlx5_core
pci_hyperv_intf 12288 1 mlx5_core

lsmod | grep -E "mlx5_core|mlx5_ib|ib_uverbs"
mlx5_ib 548864 0
ib_uverbs 200704 2 rdma_ucm,mlx5_ib
mlx5_core 2818048 1 mlx5_ib
mlxfw 36864 1 mlx5_core
psample 16384 1 mlx5_core
mlxdevm 188416 1 mlx5_core
macsec 77824 1 mlx5_ib
tls 155648 1 mlx5_core
pci_hyperv_intf 12288 1 mlx5_core
ib_core 524288 8 rdma_cm,ib_ipoib,iw_cm,ib_umad,rdma_ucm,ib_uverbs,mlx5_ib,ib_cm
mlx_compat 20480 12 rdma_cm,ib_ipoib,mlxdevm,mlxfw,iw_cm,ib_umad,ib_core,rdma_ucm,ib_uverbs,mlx5_ib,ib_cm,mlx5_core

ibv_devinfo | grep "state:"
state: PORT_DOWN (1)
state: PORT_DOWN (1)
state: PORT_DOWN (1)
state: PORT_DOWN (1)

lspci -vvv | grep -A 30 "ConnectX-6" | grep -E "Link Width|Speed"
LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM not supported
LnkSta: Speed 16GT/s (ok), Width x16 (ok)
LnkCap2: Supported Link Speeds: 2.5-16GT/s, Crosslink- Retimer+ 2Retimers+ DRS-
LnkCtl2: Target Link Speed: 16GT/s, EnterCompliance- SpeedDis-
LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM not supported
LnkSta: Speed 16GT/s (ok), Width x16 (ok)
LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM not supported
LnkSta: Speed 16GT/s (ok), Width x16 (ok)
LnkCap2: Supported Link Speeds: 2.5-16GT/s, Crosslink- Retimer+ 2Retimers+ DRS-
LnkCtl2: Target Link Speed: 16GT/s, EnterCompliance- SpeedDis-
LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM not supported
LnkSta: Speed 16GT/s (ok), Width x16 (ok)

dpkg -l | grep mlnx
ii ibacm 2410mlnx54-1.2410068 amd64 InfiniBand Communication Manager Assistant (ACM)
ii ibverbs-providers:amd64 2410mlnx54-1.2410068 amd64 User space provider drivers for libibverbs
ii ibverbs-utils 2410mlnx54-1.2410068 amd64 Examples for the libibverbs library
ii infiniband-diags 2410mlnx54-1.2410068 amd64 InfiniBand diagnostic programs
ii knem 1.1.4.90mlnx3-OFED.23.10.0.2.1.1 amd64 userspace tools for the KNEM kernel module
ii knem-dkms 1.1.4.90mlnx3-OFED.23.10.0.2.1.1 all DKMS support for mlnx-ofed kernel modules
ii libibmad-dev:amd64 2410mlnx54-1.2410068 amd64 Development files for libibmad
ii libibmad5:amd64 2410mlnx54-1.2410068 amd64 Infiniband Management Datagram (MAD) library
ii libibnetdisc5:amd64 2410mlnx54-1.2410068 amd64 InfiniBand diagnostics library
ii libibumad-dev:amd64 2410mlnx54-1.2410068 amd64 Development files for libibumad
ii libibumad3:amd64 2410mlnx54-1.2410068 amd64 InfiniBand Userspace Management Datagram (uMAD) library
ii libibverbs-dev:amd64 2410mlnx54-1.2410068 amd64 Development files for the libibverbs library
ii libibverbs1:amd64 2410mlnx54-1.2410068 amd64 Library for direct userspace use of RDMA (InfiniBand/iWARP)
ii libibverbs1-dbg:amd64 2410mlnx54-1.2410068 amd64 Debug symbols for the libibverbs library
ii librdmacm-dev:amd64 2410mlnx54-1.2410068 amd64 Development files for the librdmacm library
ii librdmacm1:amd64 2410mlnx54-1.2410068 amd64 Library for managing RDMA connections
ii mlnx-ethtool 6.9-1.2410068 amd64 This utility allows querying and changing settings such as speed,
ii mlnx-fw-updater 24.10-3.2.5.0 amd64 Mellanox Technologies Firmware update package
ii mlnx-iproute2 6.10.0-1.2410325 amd64 This utility allows querying and changing settings such as speed,
ii mlnx-ofed-kernel-dkms 24.10.OFED.24.10.3.2.5.1-1 all DKMS support for mlnx-ofed kernel modules
ii mlnx-ofed-kernel-utils 24.10.OFED.24.10.3.2.5.1-1 amd64 Userspace tools to restart and tune mlnx-ofed kernel modules
ii mlnx-tools 24.10-0.2410068 amd64 Userspace tools to restart and tune MLNX_OFED kernel modules
ii rdma-core 2410mlnx54-1.2410068 amd64 RDMA core userspace infrastructure and documentation
ii rdmacm-utils 2410mlnx54-1.2410068 amd64 Examples for the librdmacm library
ii srptools 2410mlnx54-1.2410068 amd64 Tools for Infiniband attached storage (SRP)

安装过程中常见的错误及解决方案？

卸载安装过的旧驱动
进入旧版 OFED 安装目录（以 23.04 为例）：
cd /home/user/Desktop/MLNX_OFED_LINUX-23.04-1.1.3.0-ubuntu22.04-x86_64
执行专用卸载脚本（注意：OFED 23.04 不支持–uninstall参数）：

./uninstall.sh

# 旧依赖
modprobe ib_core # IB核心协议模块
modprobe mlx_compat # 兼容性模块（已加载，重复执行无影响）
modprobe mlx5_core # ConnectX6核心驱动（关键！）
modprobe mlx5_ib # IB功能驱动（对接网卡硬件）
modprobe ib_uverbs # 用户态IB工具接口（ibv_devinfo依赖）

先停止依赖mlx模块的服务（避免模块占用）
systemctl stop openibd # OFED默认服务，若存在则停止
systemctl stop mlnx-tools # 若有mlnx相关服务，均停止

按依赖顺序卸载mlx内核模块（先卸依赖少的，后卸核心模块）

# 顺序：mlx5_ib → ib_uverbs → ib_core → mlx5_core → mlxfw（按lsmod依赖关系调整）
rmmod mlx5_ib
rmmod ib_uverbs
rmmod ib_core
rmmod mlx5_core
rmmod mlxfw
rmmod psample # 若lsmod显示该模块依赖mlx5_core，也需卸载
rmmod tls # 若依赖mlx5_core，同样卸载

# 3. 验证模块是否已移除（无输出则成功）
lsmod | grep mlx

# 彻底清理旧版本
sudo apt remove –purge mlnx-ofed-*
sudo dkms remove mlnx-ofed-kernel/23.04 –all
sudo apt autoremove
sudo apt autoclean
apt clean
apt autoremove -y # 自动删除无用依赖
rm -rf /var/lib/dpkg/info/mlnx-* # 删除mlnx相关的dpkg配置残留

验证包是否已清理（无输出则成功）
dpkg -l | grep mlnx

检查驱动服务状态
sudo /etc/init.d/openibd status
sudo systemctl status openibd

检查 InfiniBand 设备
ibstat

ibv_devices

检查内核模块加载
lsmod | grep mlx

# 应该看到: mlx5_core, mlx4_core, ib_core 等

检查网卡信息
ibstatus

基本 RDMA 测试
ibv_rc_pingpong
ibv_uc_pingpong

检查端口状态
iblinkinfo

验证链路速度（应该是 16GT/s x16）
cat /sys/class/infiniband/mlx5_0/ports/1/rate

系统内核升级后需要重装驱动吗？
不需要重装，但需要：
安装新内核头文件
sudo apt install linux-headers-$(uname -r)
重启驱动服务
sudo systemctl restart openibd
验证驱动状态
ibstat

如何更新网卡固件
# 1. 下载最新固件从 NVIDIA 官网
# 2. 使用 mstflint 工具更新
sudo mstflint -d <device> -i <firmware_image> burn

# 3. 重启系统生效
sudo reboot

驱动加载失败怎么办
检查内核消息
dmesg | grep mlx

手动加载模块
sudo modprobe mlx5_core
sudo modprobe mlx4_core

重启服务
sudo systemctl restart openibd

检查子网管理器
sudo service opensmd status

重启硬件服务
sudo /etc/init.d/openibd restart
性能优化建议

设置巨帧 sudo ibdev2netdev sudo ip link set <interface> mtu 9000
优化内存注册 echo 1000000000 | sudo tee /proc/sys/kernel/shmmax #
设置 CPU affinity（根据实际拓扑） sudo ibdev2netdev -v

 ConnectX-6 网卡将在 Ubuntu 22.04 + Linux 6.8 环境下稳定运行，为 H多卡训练提供高速 RDMA 通信支持。