Technical Announcement on OCP Network Card or NVMe Drive Speed Slowdown in Ubuntu

I.       Product Model

The full range of H3C UniServer servers supports hot-swappable OCP network card and NVMe drives in onboard configurations.

Ubuntu 24.04.x system, kernel version below 6.14.

After performing hot swapping of NVMe drives or OCP network card on Ubuntu 24.04.x, the device speed may drop to Gen1 (2.5Gt/s). This issue is not limited to H3C servers; all server vendors using this version of Ubuntu may be affected.

After hot swapping, the Ubuntu system kernel will retrain the PCIe link. Due to a defect in the Ubuntu 24.04.x kernel code, the PCIe link retraining after hot swapping fails, forcing the link speed to be limited to 2.5Gt/s (Gen1). Once the link speed is restricted, any replacement of the connected board or NVMe device will operate at the reduced speed. The speed reduction issue is unrelated to the terminal equipment (TE) and is linked to the PCIe link.

1.     Workaround

When the problem occurs, you can restart the server to have the operating system renegotiate the operating speeds of the PCIe and NVMe devices to normal status.

2.     Solutions

The 6.14.0-27 kernel version in Ubuntu 24.04.3 has fixed this issue.

Kernel upgrade method is as follows:

(1)   Check the current system kernel version using the command "hostnamectl" or "uname -a" in the system:

(2)   Use the command "cat /etc/lsb-release" to view LSB (Linux Standard Base) related version information.

The command returns the following fields:

a)     DISTRIB_ID=Ubuntu — system name, fixed as Ubuntu

b)     DISTRIB_RELEASE=22.04 — version number

c)     DISTRIB_CODENAME=jammy — full description (including whether it is an LTS long-term support version)

d)     DISTRIB_DESCRIPTION="Ubuntu 22.04 LTS" — full description (usually includes LTS and other information)

(3)   Use the command "apt list linux-image* linux-headers*" to list all available kernels or use the command "apt search linux-image-generic-" to list kernels related to the generic kernel image.

(4)   Use the command "sudo apt install -y <generic kernel version to be upgraded>" and use the command "sudo reboot" to restart the system.

(5)   After rebooting, use the command "hostnamectl" or "uname -a" again to check the system kernel and acknowledge whether the upgraded kernel is effective. If effective, the system kernel has changed to the target kernel.

Note: Inserting a new drive into an idle slot on the server during operating system runtime will not cause link speed reduction. This issue is only triggered during hot swapping in slots with existing devices.