Python Hydra configuration for different environments

Question

Background

I currently have a script which does some API data gathering. The script is deployed to two environments, test and user. Each environment has different settings which I have created the respective configuration files. I am currently migrating my project to use the hydra-core package.

Objective

If possible I would like configure Hydra as such. Load all the configuration files from either the test or user. The default should be the test environment unless specified via CLI to load the user environment.

Current Configuration Structure

Project Root
|
├── config
│   ├── config.yaml
│   ├── test
│   │   ├── config1.yaml
│   │   ├── config2.yaml
│   │   ├── config3.yaml
│   │   └── config4.yaml
│   └── user
│       ├── config1.yaml
│       ├── config2.yaml
│       ├── config3.yaml
│       └── config4.yaml

Current Default Configuration - config.yaml

defaults:
  - test: [config1.yaml, config2.yaml, config3.yaml, config4.yaml]

How would I override the above default via the CLI?

Answer 1

You can introduce a config group, e.g. env that would contain configs with the default lists you want to use for each environment.

??? config
?   ??? config.yaml
?   ??? env
?   ?   ??? user.yaml
?   ?   ??? test.yaml
?   ??? test
?   ?   ??? user.yaml
?   ??? user
?       ??? config1.yaml

env/user.yaml:

defaults:
 - /user/config1
 - /user/config2

env/test.yaml:

defaults:
 - /test/config1
 - /test/config2

Primary config.yaml:

defaults:
 - env: test # Overridable

With the above, you should be able to override the environment like:

$ python foo.py env=user

This is similar to the configure experiment pattern. This is relying on recursive default lists, new in Hydra 1.1. You can learn more about them here