Pass variables from .env file to dockerfile through docker-compose

Question

project
└───app
│   │   ...
│   │   Dockerfile
│   │
└───prod.env
└───docker-compose.yml

My docker-compose looks like this:

services:
   app:
      build:
         context: .\app
         args:
            ARG1: val1
            ARG2: val2
      env_file:
         - prod.env

But I've tried this too:

services:
   app:
      build:
         context: .\app
         args:
            ARG1: ${ARG1}
            ARG2: ${ARG2}
      env_file:
         - prod.env

My prod.env file looks like this:

ARG1 = 'val1'
ARG2 = 'val2'

But I've tried this too:

ARG1=val1
ARG2=val2

I would like for either the values of args or the values from the prod.env file to be passed to the dockerfile.

This is what I've tried to get this:

ARG ARG1
ARG ARG2

RUN echo ${ARG1}
RUN echo ${ARG2}

ENV ARG1 ${ARG1}
ENV ARG2 ${ARG2}

RUN echo ${ARG1}
RUN echo ${ARG2}

ENV ARG1 "new val2"
ENV ARG2 "new val2"

RUN echo ${ARG1}
RUN echo ${ARG2}

It always end with blank values.

Any help would be greatly appreciated. I feel like no answers from other posts have worked when I tried them.

To build I use docker-compose --env-file prod.env build

Thanks

Update Sergio Santiago asked if I could run docker-compose config and show the results.

Here are the final files I used for this test.

docker-compose:

services:
   app:
      build:
         context: .\app
         args:
            ARG1: val1
            ARG2: val2
      env_file:
         - prod.env

prod.env:

ARG3 = 'val3'
ARG4 = 'val4'

And here is the output of docker-compose --env-file prod.env config

networks:
  demo-net: {}
services:
  app:
    build:
      args:
        ARG1: val1
        ARG2: val2
      context: C:\project\app
    environment:
      ENV: prod.env
      ARG3: val3
      ARG4: val4

I would like to add that clearly from here getting the variable from the .env file to the docker-compose file is not the issue. I also have a flask app running on the container and through os.environ it is able to use the variables in the .env file. I just can't figure out how to give the same access to the Dockerfile.

Update 2 More specific information in relation to ErikMD's answer

prod.env

DOMAIN = 'actualdomain.com'
ENV = 'prod.env'
ENV_NUM = 1
ARG1 = 'value1'

dev.env

DOMAIN = 'localhost'
ENV = 'dev.env'
ENV_NUM = 0
ARG1 = 'value1'

Notice that the value for ARG1 is the same but the other values are different.

docker-compose.yml

version: "3.7"
services:
  home:
    image: home-${ENV_NUM}
    build: 
      context: .\home
      args:
        ARG1: "${ARG1}"
    networks:
      - demo-net
    env_file:
      - ${ENV}
    labels:
      - traefik.enable=true
      - traefik.http.routers.home.rule=Host(`${DOMAIN}`)
      - traefik.http.routers.home.entrypoints=web
    volumes:
      - g:\:c:\sharedrive
...
...
  reverse-proxy:
    restart: always
    image: traefik:v2.6.1-windowsservercore-1809
    command:
      - --api.insecure=true
      - --providers.docker=true
      - --entrypoints.web.address=:80
      - --providers.docker.endpoint=npipe:////./pipe/docker_engine
    ports:
      - 80:80
      - 443:443
      - 8080:8080
    networks:
     - demo-net
    volumes:
      - source: \\.\pipe\docker_engine\
        target: \\.\pipe\docker_engine\
        type: npipe
networks:
  demo-net:

The dots represent other apps that would be formatted the same as home.

dockerfile

FROM python:3.10.3

ARG ARG1="default"

ENV ARG1="${ARG1}"

WORKDIR /app

ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1

RUN echo "This is argument 1 -> ${ARG1}"

output of docker-compose --env-file prod.env config

networks:
  demo-net: {}
services:
  home:
    build:
      args:
        ARG1: value1
      context: C:\MIS-Web-App\home
    environment:
      DOMAIN: actualdomain.com
      ENV: prod.env
      ENV_NUM: '1'
      ARG1: value1
    image: home-1
    labels:
      traefik.enable: "true"
      traefik.http.routers.home.entrypoints: web
      traefik.http.routers.home.rule: Host(`mis.canaras.net`)
    networks:
      demo-net: null
    volumes:
    - g:\:c:\sharedrive:rw
...
...

Then I run either docker-compose --env-file prod.env build or docker-compose --env-file dev.env build

output of build

Step 9/23 : RUN echo "This is argument 1 -> ${ARG1}"
 ---> Running in 5142850de365
This
is
argument
1
->
Removing intermediate container 5142850de365

Now I call pass the env_file in the command as well as in the actual file because there are variables in there that my docker-compose file needs and variables that my flask app needs. And there is definitely overlap.

Getting the values from the prod.env or dev.env files to docker-compose is not the issue. Neither is getting it to my flask app. The issue is getting those values to the dockerfile.

Answer 1

My solution was annoying which is why it took me so long to figure it out. My dockerfile was using powershell on a windows server, so I had to do this for every argument:

ARG ARG1
RUN echo $env:ARG1

This seems pretty niche especially since using windows containers on a windows server is not my first choice, so check out @ErikMD 's answer if your having issues with env files and whatnot.

Answer 2

I'm posting a new answer to highlight the various assumptions related to the OP's question, in particular, the fact that there's a subtle difference between the ".env" unique filename and *.env files (arguments for env_file:).

But apart from this subtlety, the process to pass arguments from docker-compose.yml to docker build -f Dockerfile . and/or docker run -e … is easy, as shown by the comprehensive example below.

Minimal working example

Let's consider the following files in a given directory, say ./docker.

File docker-compose.yml:

services:
  demo-1:
    image: demo-${ENV_NUM}
    build:
      context: .
      args:
        ARG1: "demo-1/${ARG1}"
        ARG3: "demo-1/${ARG3}"
  demo-2:
    image: demo-2${ENV_FILE_NUM}
    build:
      context: .
      args:
        ARG1: "demo-2/${ARG1}"
        ARG3: "demo-2/${ARG3}"
    env_file:
      - var.env

Remark: even if we use a build: field, it appears to be a good idea to also add an image: field to automatically tag the built image; but note that these image names must be pairwise different.

File .env:

KEY="some value"
ENV_NUM=1
ARG1=.env/ARG1
ARG2=.env/ARG2
ARG3=.env/ARG3

File var.env:

ENV_FILE_NUM="some number"
ARG1=var.env/ARG1
ARG2=var.env/ARG2
ARG3=var.env/ARG3
ARG4=var.env/ARG4

File Dockerfile:

FROM debian:10

# Read build arguments (default value if omitted at CLI)
ARG ARG1="default 1"
ARG ARG2="default 2"
ARG ARG3="default 3"

# the build args are exported at build time
RUN echo "ARG1=${ARG1}" | tee /root/arg1.txt
RUN echo "ARG2=${ARG2}" | tee /root/arg2.txt
RUN echo "ARG3=${ARG3}" | tee /root/arg3.txt

# Export part of these args at runtime also
ENV ARG1="${ARG1}"
ENV ARG2="${ARG2}"

# exec-form is mandatory for ENTRYPOINT/CMD
CMD ["/bin/bash", "-c", "echo ARG1=\"${ARG1}\" ARG2=\"${ARG2}\" ARG3=\"${ARG3}\"; echo while at build time:; cat /root/arg{1,2,3}.txt"]

Experiment session 1

First, as suggested by @SergioSantiago in the comments, a very handy command to preview the effective docker-compose.yml file after interpolation is docker-compose config:

$ docker-compose config

WARN[0000] The "ENV_FILE_NUM" variable is not set. Defaulting to a blank string. 
name: docker
services:
  demo-1:
    build:
      context: /home/debian/docker
      dockerfile: Dockerfile
      args:
        ARG1: demo-1/.env/ARG1
        ARG3: demo-1/.env/ARG3
    image: demo-1
    networks:
      default: null
  demo-2:
    build:
      context: /home/debian/docker
      dockerfile: Dockerfile
      args:
        ARG1: demo-2/.env/ARG1
        ARG3: demo-2/.env/ARG3
    environment:
      ARG1: var.env/ARG1
      ARG2: var.env/ARG2
      ARG3: var.env/ARG3
      ARG4: var.env/ARG4
      ENV_FILE_NUM: some number
    image: demo-2
    networks:
      default: null
networks:
  default:
    name: docker_default

Here, as indicated by the warning, we see there's an issue for interpolating ENV_FILE_NUM despite the fact this variable is mentioned by var.env. The reason is that env_files lines just add new environment variables for the underlying docker run -e … command, but don't interpolate anything in the docker-compose.yml.

Contrarily, one can notice that the value ARG1=.env/ARG1 taken from ".env" is interpolated within the args: field of docker-compose.yml, cf. the output line:

args:
  ARG1: demo-1/.env/ARG1
  …

This very distinct semantics of ".env" vs. env_files is described in this page of the official documentation.

Experiment session 2

Next, let us run:

$ docker-compose up --build
                                                                                         
WARN[0000] The "ENV_FILE_NUM" variable is not set. Defaulting to a blank string.                                                                     
[+] Building 10.4s (13/13) FINISHED
 => [demo-1 internal] load build definition from Dockerfile
 => => transferring dockerfile: 609B
 => [demo-2 internal] load build definition from Dockerfile
 => => transferring dockerfile: 609B
 => [demo-1 internal] load .dockerignore
 => => transferring context: 2B
 => [demo-2 internal] load .dockerignore
 => => transferring context: 2B
 => [demo-2 internal] load metadata for docker.io/library/debian:10
 => [demo-2 1/4] FROM docker.io/library/debian:10@sha256:ebe4b9831fb22dfa778de4ffcb8ea0ad69b5d782d4e86cab14cc1fded5d8e761
 => => resolve docker.io/library/debian:10@sha256:ebe4b9831fb22dfa778de4ffcb8ea0ad69b5d782d4e86cab14cc1fded5d8e761
 => => sha256:85bed84afb9a834cf090b55d2e584abd55b4792d93b750db896f486680638344 50.44MB / 50.44MB
 => => sha256:ebe4b9831fb22dfa778de4ffcb8ea0ad69b5d782d4e86cab14cc1fded5d8e761 1.85kB / 1.85kB
 => => sha256:40dd1c1b1c36eac161ab63b6ce3a57d56ad79a667a37717a31721bac3f30aaf9 529B / 529B
 => => sha256:26a2b081e03207d26a105340161109ba0f00e857cbb0ff85aaeeeadd46b709c5 1.46kB / 1.46kB
 => => extracting sha256:85bed84afb9a834cf090b55d2e584abd55b4792d93b750db896f486680638344
 => [demo-2 2/4] RUN echo "ARG1=demo-2/.env/ARG1" | tee /root/arg1.txt
 => [demo-1 2/4] RUN echo "ARG1=demo-1/.env/ARG1" | tee /root/arg1.txt
 => [demo-1 3/4] RUN echo "ARG2=default 2" | tee /root/arg2.txt
 => [demo-2 3/4] RUN echo "ARG2=default 2" | tee /root/arg2.txt
 => [demo-2 4/4] RUN echo "ARG3=demo-2/.env/ARG3" | tee /root/arg3.txt
 => [demo-1 4/4] RUN echo "ARG3=demo-1/.env/ARG3" | tee /root/arg3.txt
 => [demo-2] exporting to image
 => => exporting layers
 => => writing image sha256:553f294a410ceeb3c0ac9d252d443710c804d3f7437ad7fffa586967517f5e7a
 => => naming to docker.io/library/demo-1
 => => writing image sha256:84bb2bd0ffae67ffed0e74efbf9253b6d634a6f37c6f99bc4eedea81846a9352
 => => naming to docker.io/library/demo-2
                                     
Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
[+] Running 3/3              
 ? Network docker_default     Created
 ? Container docker-demo-1-1  Created
 ? Container docker-demo-2-1  Created

Attaching to docker-demo-1-1, docker-demo-2-1

docker-demo-1-1  | ARG1=demo-1/.env/ARG1 ARG2=default 2 ARG3=
docker-demo-1-1  | while at build time:
docker-demo-1-1  | ARG1=demo-1/.env/ARG1
docker-demo-1-1  | ARG2=default 2
docker-demo-1-1  | ARG3=demo-1/.env/ARG3

docker-demo-2-1  | ARG1=var.env/ARG1 ARG2=var.env/ARG2 ARG3=var.env/ARG3
docker-demo-2-1  | while at build time:
docker-demo-2-1  | ARG1=demo-2/.env/ARG1
docker-demo-2-1  | ARG2=default 2
docker-demo-2-1  | ARG3=demo-2/.env/ARG3

docker-demo-1-1 exited with code 0
docker-demo-2-1 exited with code 0

Here, we can see again that the ".env" values and those of file_env: [ filename.env ] play different roles that don't overlap.

Furthermore:

• Given the absence of a Dockerfile command line ENV ARG3="${ARG3}", the value of build-arg ARG3 is not propagated at runtime (see the ARG3= line in the output above).
• But the value can be exported at runtime anyway if it is defined/overidden in the environment: or env_file: sections in the docker-compose.yml file (see the ARG3=var.env/ARG3 line in the output above).

For more details, see the documentation of the ARG directive.

Remarks on the docker-compose --env-file option use-case

As mentioned by the OP, docker-compose also enjoys a useful CLI option --enf-file (which is precisely named the same way as the very different env-file: field, which is unfortunate, but nevermind).

This option allows for the following use-case (excerpt from OP's code):

File docker-compose.yml:

services:
  home:
    image: home-${ENV_NUM}
    build:
      args:
        ARG1: "${ARG1}"
      ...
    labels:
      - traefik.http.routers.home.rule=Host(`${DOMAIN}`)
      ...
    env_file:
      - ${ENV}

File prod.env:

DOMAIN = 'actualdomain.com'
ENV = 'prod.env'
ENV_NUM = 1
ARG1 = 'value 1'

File dev.env:

DOMAIN = 'localhost'
ENV = 'dev.env'
ENV_NUM = 0
ARG1 = 'value 1'

Then run:

• docker-compose --env-file prod.env build,
• or docker-compose --env-file dev.env build

As an aside, even if most of this answer up to now, amounted to illustrating that the ".env" filename and env_file: files enjoy a very different semantics… it is true that they can also be combined "nicely" this way, as suggested by the OP, to achieve this use case.

Note in passing that the docker-compose config is also applicable to "debug" the Compose specification:

• docker-compose --env-file prod.env config,
• or docker-compose --env-file dev.env config.

Now regarding the last question:

Getting the values from the prod.env or dev.env files to docker-compose is not the issue. The issue is getting those values to the Dockerfile.

it can first be noted that there are two different cases:

1. Either the two different deployment environments (prod.env and dev.env) can share the same image, so that the difference only lies in the runtime environment variables (not the docker build args).
2. Or, depending on the file passed for --env-file, the images should be different (and then a docker-compose --env-file … build is indeed necessary).

It appears that most of the time, case 1. can be achieved (and it is also the case in the question's configuration, because the ARG1 values are the same in prod.env and dev.env) and can be viewed as more-interesting for the sake of reproducibility (because we are sure that the "prod" image will be the same as the "dev" image).

Yet, sometimes it's impossible to do so and we are "in case 2.", e.g. if the Dockerfile has a specific step, maybe related to tests or so, that has to be enabled (resp. disabled) in production mode.

So now, let us assume we're in case 2. How can we pass "everything" from the --env-file to the Dockerfile? There is only one solution, namely, extending the args: map of the docker-compose.yml and include each variable you are interested in, for example:

services:
  home:
    image: home-${ENV_NUM}
    build: 
      context: .\home
      args:
        DOMAIN: "${DOMAIN}"
        ENV_NUM: "${ENV_NUM}"
        ARG1: "${ARG1}"
    networks:
      - demo-net
    env_file:
      - ${ENV}
    labels:
      - traefik.enable=true
      - traefik.http.routers.home.rule=Host(`${DOMAIN}`)
      - traefik.http.routers.home.entrypoints=web
    volumes:
      - g:\:c:\sharedrive

Even if there is no other solution to pass arguments at build time (from docker-compose to the underlying docker build -f Dockerfile …), this has the advantage of being "declarative" (only the variables mentioned in args: will be actually passed to the Dockerfile).

Drawback?

The only drawback I see is that you may have unneeded extra environment variables at runtime (from docker-compose to the underlying docker run -e …), such as ENV=prod.env.

If this is an issue, you might want to split your ".env" files like this:

File prod.env:

DOMAIN = 'actualdomain.com'
ENV = 'prod-run.env'
ENV_NUM = 1
ARG1 = 'value 1'

File prod-run.env:

DOMAIN = 'actualdomain.com'
ENV_NUM = 1

(assuming you only want to export these two environment variables at runtime).

Or alternatively, to better follow the usual Do-not-Repeat-Yourself rule, remove prod-run.env, then pass these values as docker-compose build arguments as mentioned previously:

args:
  DOMAIN: "${DOMAIN}"
  ENV_NUM: "${ENV_NUM}"

and write in the Dockerfile:

ARG DOMAIN
ARG ENV_NUM

# ... and in the end:

ENV DOMAIN="${DOMAIN}"
ENV ENV_NUM="${ENV_NUM}"

I already gave an example of these Dockerfile directives in section "Experiment session 2".

(Sorry for the significant length of this answer BTW :)

Answer 3

You can use the .env file from docker compose so you use the same time that you defined in your service definition:

services:
    app:
         build:
            context: .\app
            args:
               ARG1: ${ARG3}
               ARG2: ${ARG4}
         env_file:
            - .env

In this way both can share the same env file, but you still have the drawback of redefining the variables as a placeholder.

This is a suggestion, choose the one that fits better to you