Where does pip install packages with a virtual environment?

Question

Situation: I have created a virtual environment and use an explicit path to run pip (without sourcing activate). Does it install packages in the global dist-packages or does it install them in the virtual environment's site-packages.

Details: As Where does pip install its packages? explains, pip installs packages in <virtualenv_name>/lib/<python_ver>/site-packages when used with a virtual environment. From my experience, this is true when I activate the virtualenv. I have an existing bash script that paths directly to the pip executable without activating the virtualenv. Does this still install packages in the virtualenv's site-packages? Or does it install them in /local/lib/<python-version>/dist-packages?

Note: I'm on Ubuntu 16.04

Answer 1

This very much depends on which version (not in the semantic version sense, but of the having-multiple-"versions" of pip installed when you create a venv) of pip your script is using, as well as its configuration (including possibly your environment).

Assuming your script has a line like

/some/path/to/pip install <some package>

and assuming that that pip has installed at least one package, you can use

/some/path/to/pip show <that package>

and it'll give you output that looks like:

$ pip show numpy
Name: numpy
Version: 1.14.5
Summary: NumPy: array processing for numbers, strings, records, and objects.
Home-page: http://www.numpy.org
Author: Travis E. Oliphant et al.
Author-email: None
License: BSD
Location: /usr/lib/python3/dist-packages
Requires:

The location line second to last should help answering your question.

Answer 2

When you're using an explicit path to the Python executable associated with the venv, e.g. .venv/bin/python3 -m pip install ..., then pip will install to the virtual environment's site e.g. .venv/lib/pythonX.Y/site-packages. Whether the venv is activated or not is irrelevant in this case.

However, if you use a command which will be resolved in $PATH such as pip, instead of the unambiguous command path/to/python -m pip, then things can get complicated.

1. The executable file that a bare command such as pip references on the filesystem depends on $PATH. Look in which pip for the right one (type pip if someone was really evil and made a shell alias for it). Look in which -a pip to see what other ones are hanging around that the first one might be shadowing. Note that some shells cache the command-to-file mapping, and sometimes screw up their cache invalidation (try hash -r if you're seeing weird behaviour and you think your shell has a messed up cache).
2. A command such as ./pip or .venv/bin/pip will not search through $PATH. That's because it has a slash in it.
3. OK, so you've found your file pip, it's an executable script. The first line of this file, called the shebang, tells you which interpreter is associated to that pip script. Look in head -1 .venv/bin/pip. If pip was installed into a venv then this will always match the venv's Python, assuming you didn't edit it manually, because the installer itself writes this shebang out (fun fact: even if you put a different one directly in your source code, the installer rewrote it!).
4. pip install will put the files where the associated interpreter's sysconfig says. Point 1 tells you exactly what "pip" means and Point 2 showed you exactly what "associated interpreter" means. The sysconfig locations are platform dependent. Find out where by running path/to/python -m sysconfig | grep "Paths:" --after 10 and look for the "purelib" path.
5. There is unfortunately an extra complication for Ubuntu users! This distribution is a Debian derivative, so they have a patched distutils installation due to Debian policies on Python packaging. TL;DR: on Ubuntu the system site dir will be named dist-packages instead of site-packages as sysconfig reports.

There is nothing magical about "activating" a venv, it just sets env vars such as $PATH. If you understand how 1-3 works, you understand how activating a venv works.

So now you should be able to reason the answer without guessing: using a path like .venv/bin/pip versus activating a venv (which changes $PATH, which changes what pip means, resulting to the same .venv/bin/pip) makes no difference.

Addendum: Why do people have so much trouble with pip installations?

Bad hygiene. I think they screwed up their systems by messing with sys.path in some way or another (appending to it in startup scripts, user site, sitecustomize.py, setting PYTHONPATH env var in .bashrc, installing some crappy package that mutated it, following some crappy guide that had a sudo pip install... the possibilities are endless!)

Look at the pip "script", it's just a console entry-point written out by setuptools. This script is created from a template at install time. You won't actually find this file anywhere in pip's source, because it doesn't exist there.

#!/path/to/.venv/bin/python3
# -*- coding: utf-8 -*-
import re
import sys

from pip._internal.main import main  # <--- look at this import statement! 

if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
    sys.exit(main())

The first thing the executable script does is try to import its private APIs from a pip module, and where that from pip module import gets resolved depends on sys.path. First match wins. When pip appears randomly broken, you should always ask yourself "is this import statement still finding the same pip/__init__.py file which was written out by the installer when it created the script?"

If it's not, or the shebang looks wrong, just delete it and reinstall pip.