Using git submodules with python

Question

I've read a lot of blog posts and questions on this site about the usage of git submodules and still have no idea how to better use them with python.

I mean, what is the easier way to manage dependencies if I have such a package:

├── mypkg
│   └── __init__.py
├── setup.py
└── submodules
    ├── subm1
    └── subm2

Then, what if I need to use "mypkg" as a submodule for "top_level_pkg":

├── setup.py
├── submodules
│   └── mypkg
└── top_level_package
    └── __init__.py

, I want to run pip install . and have all resolved correctly (have each submodule installed to the VENV in correct order).

What I've tried:

• Install each submodule using "pip" running in a subprocess. But it seems to be a hacky way and hard to manage (Unexpected installation of GIT submodule)
• Use "install_requires" with "setuptools.find_packages()" but without success
• Use requirements.txt file for each submodule, but I can't find a way how to automate it so "pip" could automatically install all requirements for all submodules.

Ideally, I imagine a separate setup.py file for each submodule with install_requires=['submodules/subm1', 'submodules/submn'], but setuptools does not support it.

Answer 1

I'm not saying it's impossible, but very hard and very tricky. A safer way is to turn each submodule into an installable Python module (with it's own setup.py) and install the submodules from Git.

This link describes how to install packages from Git with setup.py: https://stackoverflow.com/a/32689886/2952185

Answer 2

Thankfully to Gijs Wobben and sinoroc I came up with solution that works for my case:

install_requires=['subm1 @ file://localhost/<CURENT_DIR>/path/to/subm1']

Answer 3

I have managed to install a Python package from a git submodule together with a main package. These are proprietary and are never published to PyPI. And both pip and tox seem to work just fine.

To set the context, I have a git repo with a single Python package and a single git submodule; the git submodule also contains a single Python package. I think this structure is as generic and simple as it can possibly be, here's a visualization:

main-git-repo-name
??? mainpkg
?   ??? __init__.py
??? setup.py
??? tests
??? util-git-repo-name (this is a git submodule)
    ??? setup.py
    ??? test
    ??? utilpkg
        ??? __init__.py

I wanted to have pip install everything in a single invocation, and the utilpkg should be usable in mainpkg via just import utilpkg (not nested oddly).

The answer for me was all in setup.py:

First, specify the packages to install and their locations:

packages=find_packages(exclude=["tests"])
       + find_packages(where="util-git-repo-name/utilpkg", exclude=["test"]),
package_dir={
    "mainpkg": "mainpkg",
    "utilpkg": "util-git-repo-name/utilpkg"
},

Second, copy all the install_requires items from the git submodule package's setup.py file into the top level. In my case the utility package is an API client generated by swagger-codegen, so I had to add:

install_requires=[
    "urllib3 >= 1.15", "six >= 1.10", "certifi", "python-dateutil",
    ...],

Anyhow, when running pip3 install . this config results in exactly what I want in the site-packages area: a directory mainpkg/ and a directory utilpkg/

HTH