'How to write setup.py to include a Git repository as a dependency
I am trying to write setup.py for my package. My package needs to specify a dependency on another Git repository.
This is what I have so far:
from setuptools import setup, find_packages
setup(
name='abc',
packages=find_packages(),
url='https://github.abc.com/abc/myabc',
description='This is a description for abc',
long_description=open('README.md').read(),
install_requires=[
"requests==2.7.0",
"SomePrivateLib>=0.1.0",
],
dependency_links = [
"git+git://github.abc.com/abc/SomePrivateLib.git#egg=SomePrivateLib",
],
include_package_data=True,
)
When I run:
pip install -e https://github.abc.com/abc/myabc.git#egg=analyse
I get
Could not find a version that satisfies the requirement SomePrivateLib>=0.1.0 (from analyse) (from versions: ) No matching distribution found for SomePrivateLib>=0.1.0 (from analyse)
What am I doing wrong?
Solution 1:[1]
Note: this answer is now outdated. Have a look at this answer below from @Dick Fox for up-to-date instructions: https://stackoverflow.com/a/54794506/2272172
You can find the right way to do it here.
dependency_links=['http://github.com/user/repo/tarball/master#egg=package-1.0']
The key is not to give a link to a Git repository, but a link to a tarball. GitHub creates a tarball of the master branch for you if you append /tarball/master as shown above.
Solution 2:[2]
After digging through the pip issue 3939 linked by @muon in the comments above and then the PEP-508 specification, I found success getting my private repo dependency to install via setup.py using this specification pattern in install_requires (no more dependency_links):
install_requires = [
'some-pkg @ git+ssh://[email protected]/someorgname/[email protected]#egg=some-pkg',
]
The @v1.1 indicates the release tag created on github and could be replaced with a branch, commit, or different type of tag.
Solution 3:[3]
This answer has been updated regularly as Python has evolved over the years. Scroll to the bottom for the most current answer, or read through to see how this has evolved.
Unfortunately the other answer does not work with private repositories, which is one of the most common use cases for this. I eventually did get it working with a setup.py file that looks like this (now deprecated) method:
from setuptools import setup, find_packages
setup(
name = 'MyProject',
version = '0.1.0',
url = '',
description = '',
packages = find_packages(),
install_requires = [
# Github Private Repository - needs entry in `dependency_links`
'ExampleRepo'
],
dependency_links=[
# Make sure to include the `#egg` portion so the `install_requires` recognizes the package
'git+ssh://[email protected]/example_org/ExampleRepo.git#egg=ExampleRepo-0.1'
]
)
Newer versions of pip make this even easier by removing the need to use "dependency_links"-
from setuptools import setup, find_packages
setup(
name = 'MyProject',
version = '0.1.0',
url = '',
description = '',
packages = find_packages(),
install_requires = [
# Github Private Repository
'ExampleRepo @ git+ssh://[email protected]/example_org/ExampleRepo.git#egg=ExampleRepo-0.1'
]
)
However, with the very latest pip you'll run into issues with the EGG format handler. This is because while the egg is ignored pip is now doing direct URL matching and will consider two URLs, one with the egg fragment and the other without, to be completely different versions even if they point to the same package. As such it's best to leave any egg fragments off.
June 2021 - setup.py
So, the best way (current to June 2021) to add a dependency from Github to your setup.py that will work with public and private repositories:
from setuptools import setup, find_packages
setup(
name = 'MyProject',
version = '0.1.0',
url = '',
description = '',
packages = find_packages(),
install_requires = [
# Github Private Repository
'ExampleRepo @ git+ssh://[email protected]/example_org/ExampleRepo.git'
]
)
February 2022 - setup.cfg
Apparently setup.py is being deprecated (although my guess is it'll be around for awhile) and setup.cfg is the new thing.
[metadata]
name = MyProject
version = 0.1.1
[options]
packages = :find
install_requires =
ExampleRepo @ git+ssh://[email protected]/example_org/ExampleRepo.git
Solution 4:[4]
A more general answer: To get the information from the requirements.txt file I do:
from setuptools import setup, find_packages
from os import path
loc = path.abspath(path.dirname(__file__))
with open(loc + '/requirements.txt') as f:
requirements = f.read().splitlines()
required = []
dependency_links = []
# Do not add to required lines pointing to Git repositories
EGG_MARK = '#egg='
for line in requirements:
if line.startswith('-e git:') or line.startswith('-e git+') or \
line.startswith('git:') or line.startswith('git+'):
line = line.lstrip('-e ') # in case that is using "-e"
if EGG_MARK in line:
package_name = line[line.find(EGG_MARK) + len(EGG_MARK):]
repository = line[:line.find(EGG_MARK)]
required.append('%s @ %s' % (package_name, repository))
dependency_links.append(line)
else:
print('Dependency to a git repository should have the format:')
print('git+ssh://[email protected]/xxxxx/xxxxxx#egg=package_name')
else:
required.append(line)
setup(
name='myproject', # Required
version='0.0.1', # Required
description='Description here....', # Required
packages=find_packages(), # Required
install_requires=required,
dependency_links=dependency_links,
)
Solution 5:[5]
Actually if you like to make your packages installable recursively (YourCurrentPackage includes your SomePrivateLib), e.g. when you want to include YourCurrentPackage into another one (like OuterPackage ? YourCurrentPackage ? SomePrivateLib) you'll need both:
install_requires=[
...,
"SomePrivateLib @ git+ssh://github.abc.com/abc/[email protected]#egg=SomePrivateLib"
],
dependency_links = [
"git+ssh://github.abc.com/abc/[email protected]#egg=SomePrivateLib"
]
And make sure you have a tag created with your version number.
Also if your Git project is private and you like to install it inside the container, e.g., a Docker or GitLab runner, you will need authorized access to your repository. Please consider to use Git + HTTPS with access tokens (like on GitLab: https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html):
import os
from setuptools import setup
TOKEN_VALUE = os.getenv('EXPORTED_VAR_WITH_TOKEN')
setup(
....
install_requires=[
...,
f"SomePrivateLib @ git+https://gitlab-ci-token:{TOKEN_VALUE}@gitlab.server.com/abc/[email protected]#egg=SomePrivateLib"
],
dependency_links = [
f"git+https://gitlab-ci-token:{TOKEN_VALUE}@gitlab.server.com/abc/[email protected]#egg=SomePrivateLib"
]
)
Updated:
You have to put #egg=SomePrivateLib at the end of dependency line if you like to have this dependency in requirements.txt file. Otherwise pip install -r requirements.txt won't work for you and you wil get something like:
ERROR: Could not detect requirement name for 'git+https://gitlab-ci-token:[email protected]/abc/[email protected]', please specify one with #egg=your_package_name
If you use reuirements.txt, this part is resposible for name of dependency’s folder that would be created inside python_home_dir/src and for name of egg-link in site-packages/
You can use a environment variable in your requirements.txt to store your dependency’s token value safe in your repo:
Example row in requrements.txt file for this case:
....
-e git+https://gitlab-ci-token:${EXPORTED_VAR_WITH_TOKEN}@gitlab.server.com/abc/[email protected]#egg=SomePrivateLib
....
Solution 6:[6]
I was successful with these three options in GitLab. I am using version 11 of GitLab.
Option 1 - no token specified. The shell will prompt for username/password.
from setuptools import setup
TOKEN_VALUE = os.getenv('EXPORTED_VAR_WITH_TOKEN')
setup(
install_requires=[
"SomePrivateLib @ git+https://gitlab.server.com/abc/[email protected]#egg=SomePrivateLib"
]
)
Option 2 - user access token specified. The token generated by going to GitLab ? account top right ? settings ? access tokens. Create the token with read_repository rights.
Example:
import os
from setuptools import setup
TOKEN_VALUE = os.getenv('EXPORTED_VAR_WITH_TOKEN')
setup(
install_requires=[
f"SomePrivateLib @ git+https://gitlab-ci-token:{TOKEN_VALUE}@gitlab.server.com/abc/[email protected]#egg=SomePrivateLib"
]
)
Option 3 - repository-level token specified. The token generated by going to the repository ? settings ? repository ? deploy tokens. From here, create a token with read_repository rights.
Example:
import os
from setuptools import setup
TOKEN_USER = os.getenv('EXPORTED_TOKEN_USER')
TOKEN_VALUE = os.getenv('EXPORTED_VAR_WITH_TOKEN')
setup(
install_requires=[
f"SomePrivateLib @ git+https://{TOKEN_USER}:{TOKEN_VALUE}@gitlab.server.com/abc/[email protected]#egg=SomePrivateLib"
]
)
In all three, I was able to do simply: "SomePrivateLib @ git+https://gitlab.server.com/abc/SomePrivateLib.git" without the #egg marking at the end.
Solution 7:[7]
This solution works for me when I run python setup.py install:
setuptools.setup(
...,
install_requires=[
'numpy',
'pandas',
'my_private_pkg'
],
dependency_links=["git+https://github.com/[username]/[my_private_pkg].git@main#egg=my_private_pkg"],
...
)
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|---|
| Solution 1 | |
| Solution 2 | Dick Fox |
| Solution 3 | |
| Solution 4 | |
| Solution 5 | |
| Solution 6 | Peter Mortensen |
| Solution 7 | user2551579 |