How to test a dockerized application in an Azure DevOps (Server) pipeline?

Question

I have a simple python dockerized application whose structure is

/src
 - server.py
 - test_server.py
Dockerfile
requirements.txt

in which the docker base image is Linux-based, and server.py exposes a FastAPI endpoint.
For completeness, server.py looks like this:

from fastapi import FastAPI
from pydantic import BaseModel

class Item(BaseModel):
    number: int

app = FastAPI(title="Sum one", description="Get a number, add one to it", version="0.1.0")

@app.post("/compute")
async def compute(input: Item):
    return {'result': input.number + 1}

Tests are meant to be done with pytest (following https://fastapi.tiangolo.com/tutorial/testing/) with a test_server.py:

from fastapi.testclient import TestClient
from server import app
import json

client = TestClient(app)

def test_endpoint():
    """test endpoint"""
    response = client.post("/compute", json={"number": 1})
    values = json.loads(response.text)
    assert values["result"] == 2

Dockerfile looks like this:

FROM tiangolo/uvicorn-gunicorn:python3.7

COPY . /app

RUN pip install -r requirements.txt

WORKDIR /app/src

EXPOSE 8000

CMD ["uvicorn", "server:app", "--host", "0.0.0.0", "--port", "8000", "--reload"]

At the moment, if I want to run the tests on my local machine within the container, one way to do this is

1. Build the Docker container
2. Run the container, get its name via docker ps
3. Run docker exec -it <mycontainer> bash and execute pytest to see the tests passing.

Now, I would like to run tests in Azure DevOps (Server) before pushing the image to my Docker registry and triggering a release pipeline. If this sounds an OK thing to do, what's the proper way to do it?

So far, I hoped that something along the lines of adding a "PyTest" step in the build pipeline would magically work:

I am currently using a Linux agent, and the step fails with

The failure is not surprising, as (I think) the container is not run after being built, and therefore pytest can't run within it either :(

Another way to solve the solve this is to include pytest commands in the Dockerfile and deal with the tests in a release pipeline. However I would like to decouple the testing from the container that is ultimately pushed to the registry and deployed.

Is there a standard way to run pytest within a Docker container in Azure DevOps, and get a graphical report?

Answer 1

Update your azure-pipelines.yml file as follows to run the tests in Azure Pipelines

Method-1 (using docker)

trigger:
- master

pool:
  vmImage: 'ubuntu-latest'
steps:
- task: Docker@2
  inputs:
    command: 'build'
    Dockerfile: '**/Dockerfile'
    arguments: '-t fast-api:$(Build.BuildId)'
- script: |
    docker run fast-api:$(Build.BuildId) python -m pytest
  displayName: 'Run PyTest'

Successfull pipeline screenshot

Method-2 (without docker)

trigger:
- master

pool:
  vmImage: 'ubuntu-latest'
strategy:
  matrix:
    Python37:
      python.version: '3.7'

steps:
- task: UsePythonVersion@0
  inputs:
    versionSpec: '$(python.version)'
  displayName: 'Use Python $(python.version)'

- script: |
    python -m pip install --upgrade pip
    pip install -r requirements.txt
  displayName: 'Install dependencies'

- script: |
    pip install pytest pytest-azurepipelines
    python -m pytest
  displayName: 'pytest'

BTW, I have one simple FastAPI project, you can reference if your want.

Answer 2

Test your docker script using pytest-azurepipelines:

- script: |
    python -m pip install --upgrade pip
    pip install pytest pytest-azurepipelines
    pip install -r requirements.txt
    pip install -e .
  displayName: 'Install dependencies'

- script: |
    python -m pytest /src/test_server.py
  displayName: 'pytest'

Running pytest with the plugin pytest-azurepipelines will let you see your test results in the Azure Pipelines UI.

https://pypi.org/project/pytest-azurepipelines/

Answer 3

You can run your unit tests directly from within your Docker container using pytest-azurepipelines (that you need to install previously in the Docker image):

    - script: |
       docker run --mount type=bind,source="$(pwd)",target=/results \
                  --entrypoint /bin/bash my_docker_image \
                  -c "cd results && pytest"
      displayName: 'tests'
      continueOnError: true

pytest will create an xml file containing the test results, that will be made available to Azure DevOps pipeline thanks to the --mount flag in the docker run command. Then pytest-azurepipelines will publish directly the results to Azure DevOps.