github action pip dependencies not working after found cache

Question

I had wrote a workflow that included caching, following the docs of action/cache@v2. I'm using docker-compose to host my postgres db and doing tests with django framework(not in docker container).

Here are my steps:

runs-on: ubuntu-latest

    steps:
    - uses: actions/checkout@v2
    - name: Set up Python 3.8
      uses: actions/setup-python@v2
      with:
        python-version: '3.8'

    - name: Cache Dependencies
      uses: actions/cache@v2
      id: cache
      with:
        path: ~/.cache/pip
        key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements.txt') }}
        restore-keys: |
          ${{ runner.os }}-pip-
    - name: Install Dependencies
      if: steps.cache.outputs.cache-hit != 'true'
      run: |
        echo "Installing dependencies and caching them."
        python -m pip install --upgrade pip
        pip install -r requirements.txt
    - name: Start containers
      run: |
        pip cache dir
        cp .env.example .env
        docker-compose -f "docker-compose.yml" up -d --build
    - name: Run Tests
      run: |
        sleep 20
        coverage run manage.py test -v 2 && coverage report
        flake8
    - name: Stop containers
      if: always()
      run: docker-compose -f "docker-compose.yml" down

my requirements.txt already included the correct package that why first time running the build without caching it's success without any problems. The problem is any build after the first one, after the cache is found:

Run actions/cache@v2   with:
    path: ~/.cache/pip
    key: Linux-pip-0066acd4cae425b2654d95aae80b2bb4fda1c99d40c709cc1fa8a1595759a2c5
    restore-keys: Linux-pip-
     env:
    pythonLocation: /opt/hostedtoolcache/Python/3.8.10/x64
    LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.8.10/x64/lib Received 0 of 63168059 (0.0%), 0.0 MBs/sec Received 63168059 of 63168059 (100.0%), 33.1 MBs/sec Cache Size: ~60 MB (63168059 B) /usr/bin/tar --use-compress-program zstd -d -xf /home/runner/work/_temp/3f7aab5a-e81f-4039-889e-c0879488e8e8/cache.tzst
-P -C /home/runner/work/base.django/base.django Cache restored successfully Cache restored from key: Linux-pip-0066acd4cae425b2654d95aae80b2bb4fda1c99d40c709cc1fa8a1595759a2c5

The test returning error due to package from requirements.txt not found:

Run sleep 20
  sleep 20
  coverage run manage.py test -v 2 && coverage report
  flake8
  shell: /usr/bin/bash -e {0}
  env:
    pythonLocation: /opt/hostedtoolcache/Python/3.8.10/x64
    LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.8.10/x64/lib
/home/runner/work/_temp/62e260b4-7d93-4a7c-a3e1-e930c4ea36b1.sh: line 2: coverage: command not found
/home/runner/work/_temp/62e260b4-7d93-4a7c-a3e1-e930c4ea36b1.sh: line 3: flake8: command not found
Error: Process completed with exit code 127.

Even though i specified the same cache path as the cache document and it's correct with the pip document https://pip.pypa.io/en/stable/cli/pip_install/#caching ?

here what pip cache dir printing out:

/home/runner/.cache/pip

i don't know why the depedencies not found even though the cache path is correct

Answer 1

After a while, i found this article to cache the env python location instead and now the cache is working correctly

- uses: actions/cache@v2
  with:
    path: ${{ env.pythonLocation }}
    key: ${{ env.pythonLocation }}-pip-${{ hashFiles('requirements.txt') }}

I don't know why caching the entire python path worked but the cache dir didn't. Maybe someone can give a clear explanation in the future, for now this is what fixed for me

Answer 2

Summary:

~/.cache/pip doesn't contain installed dependencies, site-packages/ does. Either remove if: steps.cache.outputs.cache-hit != 'true' or cache ${{ env.pythonLocation }} (as in accepted answer) to make use of the cache.

Explanation:

~/.cache/pip is actually correct, if you want to cache downloaded assets only. pip documentation specifies that it caches HTTP responses and locally built wheels. That is what you will find in ~/.cache/pip. What pip does not cache is installed packages and whatnot.

The problem arises with the following instruction:

    - name: Install Dependencies
      if: steps.cache.outputs.cache-hit != 'true'
      run: |
        ...

If you remove the if: steps.cache.outputs.cache-hit != 'true' line, everything should work. The next time the action is ran pip will not try to download the assets, and will use cached ones instead (but will still install them from cached wheels/tarballs).

In essence, the line above tells GitHub runner to do the following:

• If we restored pip cache from cache, skip installing dependencies
• If there's no cache of pip cache, install dependencies

The first time the action is ran, runner gets cache miss. Thus, it installs dependencies. However, the second time around it scores a cache hit, and skips dependency installation.

Why is that a problem? Because upon cache hit we restore files specified by path: option:

The cache action will attempt to restore a cache based on the key you provide. When the action finds a cache, the action restores the cached files to the path you configure.

Source:
https://docs.github.com/en/actions/using-workflows/caching-dependencies-to-speed-up-workflows#using-the-cache-action

Which is ~/.cache/pip. This directory is not the directory where installed Python packages reside. That directory is site-packages/, which is included in ${{ env.pythonLocation }}. Therefore, when ${{ env.pythonLocation }} is cached, we restore installed packages, too, and everything works as expected.