Reading in environment variables from an environment file

Question

I'd like to run in a local environment a Python script which is normally run in a Docker container. The docker-compose.yml specifies an env_file which looks (partially) like the following:

DB_ADDR=rethinkdb
DB_PORT=28015
DB_NAME=ipercron

In order to run this locally, I would like these lines to be converted to

os.environ['DB_ADDR'] = 'rethinkdb'
os.environ['DB_PORT'] = '28015'
os.environ['DB_NAME'] = 'ipercron'

I could write my parser, but I was wondering if there are any existing modules/tools to read in environment variables from configuration files?

Answer 1

I use Python Dotenv Library. Just install the library pip install python-dotenv, create a .env file with your environment variables, and import the environment variables in your code like this:

import os
from dotenv import load_dotenv

load_dotenv()

MY_ENV_VAR = os.getenv('MY_ENV_VAR')

From the .env file:

MY_ENV_VAR="This is my env var content."

This is the way I do when I need to test code outside my docker system and prepare it to return it into docker again.

Answer 2

If your system/environment/workflow supports using shell scripts, you can create a script that wraps these 2 operations:

1. Sourcing the .env file and exporting them as environment variables
   • Using the set -a option where "Each variable or function that is created or modified is given the export attribute and marked for export to the environment of subsequent commands".
2. Calling your Python script/app that has plain os.environ.get code

Sample .env file (config.env):

TYPE=prod
PORT=5000

Sample Python code (test.py):

import os

print(os.environ.get('TYPE'))
print(os.environ.get('PORT'))

Sample bash script (run.sh):

#!/usr/bin/env bash

set -a
source config.env
set +a

python3 test.py

Sample run:

$ tree
.
??? config.env
??? run.sh
??? test.py

$ echo $TYPE

$ echo $PORT

$ python3 test.py
None
None

$ ./run.sh 
prod
5000

When you run the Python script directly (python3 test.py) without source-ing the .env file, all the environ.get calls return None.

But, when you wrap it in a shell script that first loads the .env file into environment variables, and then runs the Python script afterward, the Python script should now be able to read the environment variables correctly. It also ensures that the exported env vars only exist as part of the execution of your Python app/script.

As compared with the other popular answer, this doesn't need any external Python libraries.

Answer 3

This could also work for you:

env_vars = [] # or dict {}
with open(env_file) as f:
    for line in f:
        if line.startswith('#') or not line.strip():
            continue
        # if 'export' not in line:
        #     continue
        # Remove leading `export `, if you have those
        # then, split name / value pair
        # key, value = line.replace('export ', '', 1).strip().split('=', 1)
        key, value = line.strip().split('=', 1)
        # os.environ[key] = value  # Load to local environ
        # env_vars[key] = value # Save to a dict, initialized env_vars = {}
        env_vars.append({'name': key, 'value': value}) # Save to a list

print(env_vars)

In the comments, you'll find a few different ways to save the env vars and also a few parsing options i.e. to get rid of the leading export keyword. Another way would be to use the python-dotenv library. Cheers.

UPDATE: I setup my own envvar_utils.py to handle conversion from string etc..

"""Utility functions for dealing with env variables and reading variables from env file"""
import os
import logging
import json

BOOLEAN_TYPE = 'boolean'
INT_TYPE = 'int'
FLOAT_TYPE = 'float'
STRING_TYPE = 'str'
LIST_TYPE = 'list'
DICT_TYPE = 'dict'


def get_envvars(env_file='.env', set_environ=True, ignore_not_found_error=False, exclude_override=()):
    """
    Set env vars from a file
    :param env_file:
    :param set_environ:
    :param ignore_not_found_error: ignore not found error
    :param exclude_override: if parameter found in this list, don't overwrite environment
    :return: list of tuples, env vars
    """
    env_vars = []
    try:

        with open(env_file) as f:
            for line in f:
                line = line.replace('\n', '')

                if not line or line.startswith('#'):
                    continue

                # Remove leading `export `
                if line.lower().startswith('export '):
                    key, value = line.replace('export ', '', 1).strip().split('=', 1)
                else:
                    try:
                        key, value = line.strip().split('=', 1)
                    except ValueError:
                        logging.error(f"envar_utils.get_envvars error parsing line: '{line}'")
                        raise

                if set_environ and key not in exclude_override:
                    os.environ[key] = value

                if key in exclude_override:
                    env_vars.append({'name': key, 'value': os.getenv(key)})
                else:
                    env_vars.append({'name': key, 'value': value})
    except FileNotFoundError:
        if not ignore_not_found_error:
            raise

    return env_vars


def create_envvar_file(env_file_path, envvars):
    """
    Writes envvar file using env var dict
    :param env_file_path: str, path to file to write to
    :param envvars: dict, env vars
    :return:
    """
    with open(env_file_path, "w+") as f:
        for key, value in envvars.items():
            f.write("{}={}\n".format(key, value))
    return True


def convert_env_var_flag_to(env_var_name, required_type, default_value):
    """
    Convert env variable string flag values to required_type
    :param env_var_name: str, environment variable name
    :param required_type: str, required type to cast the env var to
    :param default_value: boolean, default value to use if the environment variable is not available
    :return: environment variable value in required type
    """
    env_var_orginal_value = os.getenv(env_var_name, default_value)
    env_var_value = ""
    try:
        if required_type == INT_TYPE:
            env_var_value = int(env_var_orginal_value)
        elif required_type == FLOAT_TYPE:
            env_var_value = float(env_var_orginal_value)
        elif required_type == BOOLEAN_TYPE:
            env_var_value = bool(int(env_var_orginal_value))
        elif required_type == STRING_TYPE:
            env_var_value = str(env_var_orginal_value)
        elif required_type == LIST_TYPE:
            env_var_value = env_var_orginal_value.split(',') if len(env_var_orginal_value) > 0 else default_value
        elif required_type == DICT_TYPE:
            try:
                env_var_value = json.loads(env_var_orginal_value) if env_var_orginal_value else default_value
            except Exception as e:
                logging.error(f"convert_env_var_flag_to: failed loading {env_var_orginal_value} error {e}")
                env_var_value = default_value
        else:
            logging.error("Unrecognized type {} for env var {}".format(required_type, env_var_name))

    except ValueError:
        env_var_value = default_value
        logging.warning("{} is {}".format(env_var_name, env_var_orginal_value))

    return env_var_value

Answer 4

You can use ConfigParser. Sample example can be found here.

But this library expects your key=value data to be present under some [heading]. For example, like:

[mysqld]
user = mysql  # Key with values
pid-file = /var/run/mysqld/mysqld.pid
skip-external-locking
old_passwords = 1
skip-bdb      # Key without value
skip-innodb

Answer 5

Dewald Abrie posted a good solution.

Here's a slight modification that ignores breaklines (\n)

def get_env_data_as_dict(path: str) -> dict:
    with open(path, 'r') as f:
       return dict(tuple(line.replace('\n', '').split('=')) for line
                in f.readlines() if not line.startswith('#'))

print(get_env_data_as_dict('../db.env'))

Answer 6

Using only python std

import re

envre = re.compile(r'''^([^=]+)\s+?=\s+?(?:[\s"']*)(.+?)(?:[\s"']*)$''')
result = {}
with open('/etc/os-release') as ins:
    for line in ins:
        match = envre.match(line)
        if match is not None:
            result[match.group(1)] = match.group(2)

Answer 7

How about this for a more compact solution:

import os

with open('.docker-compose-env', 'r') as fh:
    vars_dict = dict(
        tuple(line.replace('\n', '').split('='))
        for line in fh.readlines() if not line.startswith('#')
    )

print(vars_dict)
os.environ.update(vars_dict)

Answer 8

In situations where using the python-dotenv wasn't possible, I've used something like the following:

import os

def load_env_file(dotenv_path, override=False):
    with open(dotenv_path) as file_obj:
        lines = file_obj.read().splitlines()  # Removes \n from lines

    dotenv_vars = {}
    for line in lines:
        line = line.strip()
        if not line or line.startswith("#") or "=" not in line:
            continue

        key, value = line.split("=", maxsplit=1)
        dotenv_vars.setdefault(key, value)

    if override:
        os.environ.update(dotenv_vars)
    else:
        for key, value in dotenv_vars.items():
            os.environ.setdefault(key, value)

It reads the given file and parses lines that have the "=" symbol in them. Value before the symbol will be the key, and value after is the value.

Current environment variables with the same keys as in the env file can either be left untouched or overwritten with the override parameter.

Answer 9

I think you should leave it for external tools to manage the environment for you.

This way you can easily use secret manager like 1password cli to get the environment variables loaded from encrypted vault like so

op run --env-file=.env -- python your_script.py

Having that said load_dotenv is smart enough not to load .env variables if they are present in the environment but some of the other solutions aren't.

And if you don't have any external tool at your disposal just use 'bash':

set -o allexport; source .env; set +o allexport

Solution taken from: Set environment variables from file of key/value pairs