'Dot env file comparison/validation

If you have multiple .env files in a given directory for specific environments e.g. dev.env, staging.env and prod.env, using a shell, is it possible to check that each file has the same env var keys, ignoring the values?

For example, given the following files:

dev.env

KEY_1=DEV_VALUE_1
KEY_2=DEV_VALUE_2

staging.env

KEY_1=STAGING_VALUE_1
KEY_2=STAGING_VALUE_2

prod.env

KEY_1=PROD_VALUE_1
KEY_2=PROD_VALUE_2
KEY_3=PROD_VALUE_3

Can anyone suggest an awk command or similar that would return a non-zero exit code due to prod.env containing an extra key? The goal here is to validate that all .env files are aligned in the keys they declare, primarily in a CI setting.

shellawkdotenv


Solution 1:[1]

Sure, count the files for each key, and show discrepancies.

#! /usr/bin/awk -f

{
    keys[$1]++
}

END {
    for( key in keys ) {
        if( n == 0 ) {
            n = keys[key]
        }

        if( keys[key] != n ) {
            printf "%s appears in %d of %d files\n", \
            key, keys[key], n > "/dev/stderr"
        }
    }
}

$ awk -F= -f ./missing.awk *.env
KEY_3 appears in 1 of 3 files

Solution 2:[2]

Can anyone suggest an awk command or similar that would return a non-zero exit code due to prod.env containing an extra key?

I would harness GNU AWK for this task following way

awk 'BEGIN{FS="=";PROCINFO["sorted_in"]="@ind_str_asc"}FNR==1{delete arr1}{arr1[$1]}ENDFILE{keys="";for(i in arr1){keys = keys FS i};arr2[keys];if(length(arr2)>1){exit(1)}}' *.env

Explanation: I inform GNU AWK that field separator is = and that array should be traversed with sorted indices. On starting line of each line I do clear array arr1, then for each line mention key being value of 1st column of array arr1, after processing file, I do build string keys by concatenating keys of arr1 using field separator. Here order of traverse is important as it provides that 2 files with same keys in different order will give same string. Then I use this string as key of arr2 and if there more than 1 different keys in arr2 I do exit with non-zero code as stipulated by requirements.

(tested in GNU Awk 5.0.1)

Solution 3:[3]

Here's a version that not only will inform you of mis-aligned keys, but also show you what their values are, and from which .env file :

 mawk 'BEGIN { FS=OFS="="
             __++ 
     } __~FNR {++____; _=FILENAME 
     }        {
             ___[$__]=___[$__]""_""FS""($NF)" | "
     } END { 
            _____=" %s only shows up at ::: %s\n"
        for(__ in ___) {

             _=___[__] 
             sub("[ ]*[|][ ]*$","",_)
             
             if(____!=gsub(FS,"&",_)) {
                   printf(_____,__,_) } } }' test*.env | lgp3

    KEY_3 only shows up at ::: testprod.env=PROD_VALUE_3

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source
Solution 1 James K. Lowden
Solution 2 Daweo
Solution 3 RARE Kpop Manifesto

Related Questions