`pyusb` fails to find `libusb` when using `pyinstaller`

Question

Problem

My application works well when in python-only and on my machine. I am attempting to deploy to a machine which has lesser privileges.

The basic script that I am attempting to run at the moment:

import logging

import os
os.environ['PYUSB_DEBUG'] = 'debug'
os.environ['PYUSB_LOG_FILENAME'] = 'C:\\path\to\log.txt'

logging.basicConfig(level=logging.DEBUG)

import usb.core
import usb.util

devices = [d for d in usb.core.find(find_all=True,
                                    idVendor=0x0683,
                                    idProduct=0x2008
                                    )]

[print(d) for d in devices]

device = devices[0]
device.set_configuration()

The imports are a bit out of whack in this example file because the os.environ variable must be set before usb import or the logging won't be set up properly to get my log.txt from the usb module.

From Python

When I execute the above script on my machine, this is the resulting log.txt:

2022-05-13 06:50:02,760 DEBUG:usb.backend.libusb1:_LibUSB.__init__(<WinDLL 'C:\WINDOWS\system32\libusb-1.0.dll', handle 7ff8476e0000 at 0x20a622c18b0>)
2022-05-13 06:50:02,760 DEBUG:usb.backend.libusb1:_LibUSB.__init__(<WinDLL 'C:\WINDOWS\system32\libusb-1.0.dll', handle 7ff8476e0000 at 0x20a622c18b0>)
2022-05-13 06:50:02,766 INFO:usb.core:find(): using backend "usb.backend.libusb1"
2022-05-13 06:50:02,766 INFO:usb.core:find(): using backend "usb.backend.libusb1"
2022-05-13 06:50:02,767 DEBUG:usb.backend.libusb1:_LibUSB.enumerate_devices()
...
...(more logs past this point, but you can see that the core found the backend)

From pyinstaller executable

I am bundling using pyinstaller. I normally like to use --onefile, but I always verify functionality without --onefile. I am only doing the most basic build: pyinstaller --noconfirm path/to/script.py

The console error:

Traceback (most recent call last):
  File "usb\backend\libusb0.py", line 738, in get_backend
  File "Lib\site-packages\_pyinstaller_hooks_contrib\hooks\rthooks\pyi_rth_usb.py", line 57, in _load_libraryOSError: USB library could not be found
Traceback (most recent call last):
  File "backend_test.py", line 15, in <module>
  File "usb\core.py", line 1309, in find
usb.core.NoBackendError: No backend available
[9344] Failed to execute script backend_test

Note that the USB library failed to find a backend.

The fail log.txt when attempted to run the executable:

Traceback (most recent call last):
  File "usb\backend\libusb0.py", line 738, in get_backend
  File "Lib\site-packages\_pyinstaller_hooks_contrib\hooks\rthooks\pyi_rth_usb.py", line 57, in _load_library
OSError: USB library could not be found
2022-05-13 06:20:59,838 ERROR:usb.backend.libusb1:Error loading libusb 1.0 backend
Traceback (most recent call last):
  File "usb\backend\libusb1.py", line 961, in get_backend
  File "Lib\site-packages\_pyinstaller_hooks_contrib\hooks\rthooks\pyi_rth_usb.py", line 57, in _load_library
OSError: USB library could not be found
2022-05-13 06:20:59,840 ERROR:usb.backend.openusb:Error loading OpenUSB backend
Traceback (most recent call last):
  File "usb\backend\openusb.py", line 745, in get_backend
  File "Lib\site-packages\_pyinstaller_hooks_contrib\hooks\rthooks\pyi_rth_usb.py", line 57, in _load_library
OSError: USB library could not be found
2022-05-13 06:20:59,842 ERROR:usb.backend.libusb0:Error loading libusb 0.1 backend
Traceback (most recent call last):
  File "usb\backend\libusb0.py", line 738, in get_backend
  File "Lib\site-packages\_pyinstaller_hooks_contrib\hooks\rthooks\pyi_rth_usb.py", line 57, in _load_library
OSError: USB library could not be found

At the moment, I can tell that the failure is occurring in a pyinstaller hook. I don't understand precisely how this works, so the clue may be found somewhere in there.

Background & Research

Versioning

libusb==1.0.24b3
pyinstaller==4.3
pyinstaller-hooks-contrib==2022.4
pyusb==1.2.1
... (there is more, but I suspect that these are the critical-to-function packages)

libusb

Not sure why this installed the beta version. Interesting, but doesn't appear to be the smoking gun.

pyinstaller

Note that pyinstaller==4.3 is a bit out of date. This is a result of my overambitious IT department's antivirus. It doesn't appear to want to allow more recent versions of pyinstaller to build executables.

pyinstaller-hooks-contrib

This is a repository which contains several pyinstaller hooks. I know that these are "critical to function" for lots of modules, but I don't really know how this works. It is possible, even likely given that this is where something is failing, that my problem lies within this library OR with this library's interaction with the version of pyinstaller that I'm using. Version 2022.4 is the version that is installed when I python -m pip install pyinstaller.

pyusb

Current version.

Other Attempts Thus Far

I have been up and down the web and have spent a particularly long time with the SO question Pyusb on windows - no backend available. Many of the below attempts were based on that SO question, but there have been other sources as well.

There are more attempts than just these, but I have been at it long enough that I don't remember them all!

Bundling libusb-1.0.dll into Executable

It is possible to bundle dll files directly with the pyinstaller-created script using the datas directive. This was my first attempt and it worked... on my machine. As soon as I deployed, the script failed with a PermissionError. I suspect that "untrusted" dll's aren't allowed to be utilized on the target machine. My login on my machine has elevated privileges, explaining why it may have worked in my environment.

Installing libusb-win32-devel-filter.exe.

Simply didn't work.

Including C:\\Windows\\System32 on PATH

I verified that the PATH variable contained C:\\Windows\\System32, which I also verified contains the proper dll file.

Specify backend

When I modify the script to specify the backend:

...
import usb.backend.libusb1 as libusb1

be = libusb1.get_backend(find_library=lambda x: "C:\\WINDOWS\\system32\\libusb-1.0.dll")
devices = [d for d in usb.core.find(find_all=True,
                                    idVendor=0x0683,
                                    idProduct=0x2008,
                                    backend=be)]
...

There is no change in behavior. I do see in the log when running in python (not pyinstaller) that the backend is immediately found:

2022-05-13 07:22:12 USBN1LPGDKWXD3 usb.backend.libusb1[6880] DEBUG _LibUSB.__init__(<WinDLL 'C:\Windows\System32\libusb-1.0.dll', handle 7ff840bd0000 at 0x190eadbe820>)
2022-05-13 07:22:12 USBN1LPGDKWXD3 usb.backend.libusb1[6880] DEBUG _LibUSB.enumerate_devices()
2022-05-13 07:22:12 USBN1LPGDKWXD3 usb.backend.libusb1[6880] DEBUG _LibUSB.get_device_descriptor(<usb.backend.libusb1._Device object at 0x00000190EAE58EB0>)
2022-05-13 07:22:12 USBN1LPGDKWXD3 usb.backend.libusb1[6880] DEBUG _LibUSB.get_device_descriptor(<usb.backend.libusb1._Device object at 0x00000190EAE58F40>)
...

Install libusb

No change.

Modify pyinstaller and pyinstaller-hooks-contrib versions

Tried pyinstaller==4.10 and python-hooks-contrib==2022.3, which appear to be "compatible" releases.

I would like to try to build the pyinstaller script by excluding the pyusb from pyinstaller-hooks-contrib, but I'm not sure how to do this.

Other Clues....

While looking in pyinstaller-hooks-contrib, I'm seeing the way to access DLL files is through ctypes.WinDLL.

import ctypes

print('ctypes.WinDLL', ctypes.WinDLL('libusb-1.0.dll'))

Result when running from python: <WinDLL 'libusb-1.0.dll', handle 7ff831cc0000 at 0x21076e9adf0> Result when running from pyinstaller: ctypes.WinDLL <PyInstallerWinDLL 'libusb-1.0.dll', handle 7ff83d400000 at 0x201a9f39fa0>

Answer 1

I'm still not sure why this works the way that it does, but my initial instincts were correct when I had tried to include libusb-1.0.dll in the pyinstaller package. Unfortunately, there is apparently an issue with the windows implementation of the libusb-1.0.dll (I'm not making that assertion, I'm quoting another SO answer).

The answer, for me, was to add libusb0.dll to my pyinstaller build directory instead of the more recent libusb-1.0.dll:

a = Analysis(['examples\\backend_test.py'],
             pathex=[],
             binaries=[],
             datas=[('C:\\Windows\\System32\\libusb0.dll', '.'),],
             hiddenimports=[],
             ...])

I can't believe that it was this simple in my case, but there it is...