How to fix "NVRM: API mismatch" between client version and kernel module version when installing Nvidia drivers for a GTX 560 Ti in Ubuntu 20.04?

Question

I have installed nvidia-driver-390 after adding a GTX 560 Ti on an Intel Core i5 12600K PC running Kubuntu 20.04 LTS.

After rebooting I get the following error:

$ dmesg|grep -i nvrm -A3
[  113.647054] NVRM: API mismatch: the client has the version 460.91.03, but
               NVRM: this kernel module has the version 390.144.  Please
               NVRM: make sure that this kernel module and all NVIDIA driver
               NVRM: components have the same version.

How do I get that client version set to 390.144?

Newer drivers such as nvidia-driver-460 do not seem to support this video card. Also I get a warning:

$ ubuntu-drivers devices
WARNING:root:_pkg_get_support nvidia-driver-390: package has invalid Support Legacyheader, cannot determine support level
== /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0 ==
modalias : pci:v000010DEd00001200sv00001043sd000083ACbc03sc00i00
vendor   : NVIDIA Corporation
model    : GF114 [GeForce GTX 560 Ti]
driver   : nvidia-340 - distro non-free
driver   : nvidia-driver-390 - distro non-free recommended
driver   : xserver-xorg-video-nouveau - distro free builtin

I run a newer kernel in order to have networking with the Intel 2.5G Ethernet IGC driver. I can boot into kernel 5.4.0.91-generic, but then I won't have a network.

$ uname -a
Linux Kairos 5.13.0-22-generic #22~20.04.1-Ubuntu SMP Tue Nov 9 15:07:24 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Furthermore, dkms gives:

$ dkms status
nvidia, 390.144, 5.13.0-22-generic, x86_64: installed
nvidia, 390.144, 5.4.0-91-generic, x86_64: built

While /var/lib/dkms/ shows:

$ tree /var/lib/dkms/
/var/lib/dkms/
├── dkms_dbversion
└── nvidia
    ├── 390.144
    │   ├── 5.13.0-22-generic
    │   │   └── x86_64
    │   │       ├── log
    │   │       │   └── make.log
    │   │       └── module
    │   │           ├── nvidia-drm.ko
    │   │           ├── nvidia.ko
    │   │           └── nvidia-modeset.ko
    │   ├── 5.4.0-91-generic
    │   │   └── x86_64
    │   │       ├── log
    │   │       │   └── make.log
    │   │       └── module
    │   │           ├── nvidia-drm.ko
    │   │           ├── nvidia.ko
    │   │           ├── nvidia-modeset.ko
    │   │           └── nvidia-uvm.ko
    │   └── source -> /usr/src/nvidia-390.144
    └── kernel-5.13.0-22-generic-x86_64 -> 390.144/5.13.0-22-generic/x86_64

12 directories, 10 files

And modinfo nvidia returns:

filename:       /lib/modules/5.13.0-22-generic/updates/dkms/nvidia.ko
alias:          char-major-195-*
version:        390.144
supported:      external
license:        NVIDIA
srcversion:     D635B75826DE88984609590
alias:          pci:v000010DEd00000E00sv*sd*bc04sc80i00*
alias:          pci:v000010DEd*sv*sd*bc03sc02i00*
alias:          pci:v000010DEd*sv*sd*bc03sc00i00*
depends:        ipmi_msghandler
retpoline:      Y
name:           nvidia
vermagic:       5.13.0-22-generic SMP mod_unload modversions 
parm:           NVreg_Mobile:int
parm:           NVreg_ResmanDebugLevel:int
parm:           NVreg_RmLogonRC:int
parm:           NVreg_ModifyDeviceFiles:int
parm:           NVreg_DeviceFileUID:int
parm:           NVreg_DeviceFileGID:int
parm:           NVreg_DeviceFileMode:int
parm:           NVreg_UpdateMemoryTypes:int
parm:           NVreg_InitializeSystemMemoryAllocations:int
parm:           NVreg_UsePageAttributeTable:int
parm:           NVreg_MapRegistersEarly:int
parm:           NVreg_RegisterForACPIEvents:int
parm:           NVreg_CheckPCIConfigSpace:int
parm:           NVreg_EnablePCIeGen3:int
parm:           NVreg_EnableMSI:int
parm:           NVreg_TCEBypassMode:int
parm:           NVreg_UseThreadedInterrupts:int
parm:           NVreg_EnableStreamMemOPs:int
parm:           NVreg_EnableBacklightHandler:int
parm:           NVreg_RestrictProfilingToAdminUsers:int
parm:           NVreg_EnableUserNUMAManagement:int
parm:           NVreg_EnableIBMNPURelaxedOrderingMode:int
parm:           NVreg_MemoryPoolSize:int
parm:           NVreg_KMallocHeapMaxSize:int
parm:           NVreg_VMallocHeapMaxSize:int
parm:           NVreg_IgnoreMMIOCheck:int
parm:           NVreg_RegistryDwords:charp
parm:           NVreg_RegistryDwordsPerDevice:charp
parm:           NVreg_RmMsg:charp
parm:           NVreg_AssignGpus:charp

I have found many posts about how to upgrade the driver to the latest version, currently 495, however it does not solve my problem and that version does not support my video card. Only version 390 works, but then I get that API mismatch error.

Answer 1

I found the solution and now Kubuntu starts up normally as expected.

Before starting, I wanted to confirm this version mismatch between loaded kernel module and newly installed standalone module files. User ladar showed in his answer that the following will confirm this issue

$ cat /proc/driver/nvidia/version
$ cat /sys/module/nvidia/version

First, I purged all drivers and libraries

$ sudo apt purge nvidia-* && sudo apt purge libnvidia-*

Then I checked if there weren't any nvidia drivers installed. The following did not return anything, as it should

$ dpkg -l nvidia-*|grep ^ii

Then I downloaded the correct version of drivers for my Geforce GTX 560 Ti from https://www.nvidia.com/Download/driverResults.aspx/177153/en-us. You should select your driver from https://www.nvidia.com/Download/index.aspx?lang=en-us.

After this I rebooted into a terminal

$ sudo init 3

I logged in on the command line and went to the directory where I just downloaded the driver, then changed permissions and ran the installation file (it may ask some configuration questions), followed by a reboot

$ cd $HOME/Downloads/Nvidia/
$ chmod a+x NVIDIA-Linux-x86_64-390.144.run
$ sudo ./NVIDIA-Linux-x86_64-390.144.run 
$ sudo reboot

I now had my normal SSDM login screen and after logging in my dual screens were both in the correct resolution.

Answer 2

I ran into this problem, but it had nothing to do with CUDA (which wasn't installed on some of the systems). On my system the kernel modules were being embedded inside the compressed kernel image, then being loaded early in the boot process. These embedded, but outdated modules, would then prevent the correct, and newly installed/compiled standalone module files from being loaded. You can confirm this issue easily. Check the following:

cat /proc/driver/nvidia/version
cat /sys/module/nvidia/version

If the loaded modules loaded don't match the driver version, you could also be facing this problem. Assuming the correct kernel modules are available, which you can confirm by running (assuming your distro uses DKMS):

dkms status

For me the fix simply involved regenerating my kernel images. On Red Hat distros, and its derivatives (Fedora, CentOS, Alma, Rocky, Oracle, etc) you can run:

(rpm -q --qf="%{VERSION}-%{RELEASE}.%{ARCH}\n" --whatprovides kernel ; uname -r) | \
sort | uniq | while read KERNEL ; do 
  dracut -f "/boot/initramfs-${KERNEL}.img" "${KERNEL}" || exit 1
done

This will regenerate the image for every installed kernel. For the equivalent logic on Debian distros, and its derivatives (including Ubuntu), you can run:

for kernel in /boot/config-*; do 
  [ -f "$kernel" ] || continue
  KERNEL=${kernel#*-}
  mkinitramfs -o "/boot/initrd.img-${KERNEL}.img" "${KERNEL}" || exit 1
done

Then reboot. You can also fix the problem temporarily, by manually removing (unloading) the NVIDIA module using rmmod or modprobe, then reloading them. When you do modprobe will use the standalone kernel module which should match your installed driver version.

P.S. I hit this issue when I upgraded from the 470.x driver, to the 510.x driver, which recently became the reccomended, stable, install version. I never ran into this problem while using the 460.x and 470.x driver releases.

P.P.S. EDIT - in retrospect the commands:

# Red Hat Distros
dracut --regenerate-all --force

# Debian Distros
update-initramfs -u -k all

Requires less typing than what I initially posted, and should work for most people.