Testing Python C libraries - get build path

Question

When using setuptools/distutils to build C libraries in Python

$ python setup.py build

the *.so/*.pyd files are placed in build/lib.win32-2.7 (or equivalent).

I'd like to test these files in my test suite, but I'd rather not hard code the build/lib* path. Does anyone know how to pull this path from distutils so I can sys.path.append(build_path) - or is there an even better way to get hold of these files? (without having installed them first)

Answer 1

You must get the platform that you are running on and the version of python you are running on and then assemble the name yourself. This is an internal detail of setuptools (based on the bundled version of distutils), and so is not guaranteed to be stable.

To get the current platform, use sysconfig.get_platform(). To get the python version, use sys.version_info (specifically the first three elements of the returned tuple). On my system (64-bit linux with python 2.7.2) I get:

>>> import sysconfig
>>> import sys
>>> sysconfig.get_platform()
linux-x86_64
>>> sys.version_info[:3]
(2, 7, 2)

Before setuptools 62.1

The format of the lib directory is "lib.platform-versionmajor.versionminor" (i.e. only 2.7, not 2.7.2). You can construct this string using python's string formatting methods:

def distutils_dir_name(dname):
    """Returns the name of a distutils build directory"""
    f = "{dirname}.{platform}-{version[0]}.{version[1]}"
    return f.format(dirname=dname,
                    platform=sysconfig.get_platform(),
                    version=sys.version_info)

After setuptools 62.1

The directory above clashes with alternative python implementations like PyPy, so the internal build directory was made more specific using sys.implementation.cache_tag. The format of the lib directory is "lib.<platform>-<cachetag>". You can construct this string using python's string formatting methods:

def distutils_dir_name(dname):
    """Returns the name of a distutils build directory"""
    f = "{dirname}.{platform}-{cache_tag}"
    return f.format(dirname=dname,
                    platform=sysconfig.get_platform(),
                    cache_tag=sys.implementation.cache_tag)

You can use this to generate the name of any of distutils build directory:

>>> import os
# Before Setuptools 62.1
>>> os.path.join('build', distutils_dir_name('lib'))
build/lib.linux-x86_64-3.9
# After Setuptools 62.1
>>> os.path.join('build', distutils_dir_name('lib'))
build/lib.linux-x86_64-cpython39

Note that Setuptools 62.1 is only available on Python 3.7+. Also, if you set SETUPTOOLS_USE_DISTUTILS=stdlib in your environment, then you will get the old behavior even in Setuptools 62.1.

Answer 2

This is available as the build_lib member of the build command class, and it's relative to where setup.py would be. Getting at it when you're not actually running setup is rather nasty:

import distutils.dist
import distutils.command.build
b = distutils.command.build.build(distutils.dist.Distribution())
b.finalize_options()
print(b.build_lib)

Personally I'd go with SethMMorton's solution, since although it's potentially fragile if distutils changes the build directory location, I suspect playing these kinds of tricks is even more fragile if distutils structure changes.

Answer 3

I found that compiling the module in-place is best for testing purposes. To do so just use

python setup.py build_ext --inplace

This will compile all the auxiliary files in the temp directory as usual, but the final .so file will be put in the current directory.

Answer 4

@SethMMorton's approach is the way to go. It might be however safer (even if I don't know a platform for which his version doesn't work) to use get_platform() from distutils.util rather than from sysconfig.

This is how it is used in distutils-library:

import os
import sys
from distutils.util import get_platform
def get_distutils_lib_path():
    PLAT_SPEC = "%s-%d.%d" % (get_platform(), *sys.version_info[:2])
    return os.path.join("build", "lib.%s" % PLAT_SPEC)