How to merge multiple json files in a directory with jq or any tool?

Question

I am trying to merge all json files in a directory (can be many).

I know that if there are only 2 files, I can use jq -s . file1.json file2.json. But when I tried to use the a wild card like jq -s . file*.json, it fails. Any idea?

Answer 1

The answer Here is the jq answer... for more see the explanation below:

s="*.json"; jq -s "`x=-1; n=$(ls $s | wc -l); while [ $((x++)) -lt $(($n-2)) ]; do printf ".[$x] * " ; done; printf ".[$(($n-1))]";`" $s

Step by step explanation

To merge two json files using jq use this command:

jq -s ".[0] * .[1]" file1.json file2.json

Now to merge three json files:

jq -s ".[0] * .[1] * .[2]" file1.json file2.json file3.json

Let's follow the same logic and generalise for n files using wildcards. The difficult part is to generate the string .[0] * .[1] ... .[n-1].

First let's find n. n is the number of json files in the folder. so:

ls *.json | wc -l

now let's write a loop that will build the string ".[1] * .. * .[a] * .[a+1] * .. * .[n-2] *". The kernel of this string is .[a] * so:

x=-1; n=$(ls *.json | wc -l); while [ $((x++)) -lt $(($n-2)) ]; do printf ".[$x] * " ; done;

ok and now let's add the last part .[n-1]:

x=-1; n=$(ls *.json | wc -l); while [ $((x++)) -lt $(($n-2)) ]; do printf ".[$x] * " ; done; printf ".[$(($n-1))]";

now let's pass that string to the jq command... and let's parameterise *.json so that we can replace it with whatever we want with ease:

s="*.json"; jq -s "`x=-1; n=$(ls $s | wc -l); while [ $((x++)) -lt $(($n-2)) ]; do printf ".[$x] * " ; done; printf ".[$(($n-1))]";`" $s

et voilà!

Answer 2

While waiting a jq's solution, let me address the question "... or any tool?"

Different people mean different things by "merging", thus assuming here a plain recursive JSON merging, then with jtc (another tool) the solution would be like this (for an arbitrary number of files):

• read all the JSONs into one super JSON array
• merge into the 1st child (the first read JSON) the rest of read JSONs
• output the resulting 1st JSON child in the array

jtc -J / -w[0] -mi[1:] / -w[0] *.json 

to illustrate the solution, here's how it works with couple arbitrary JSON files:

bash $ jtc -tc file1.json 
{
   "days": {
      "2020-05-18": { "seconds": 60 },
      "2020-05-19": { "seconds": 30 },
      "2020-05-20": { "seconds": 1400 }
   },
   "total": { "seconds": 1490 }
}
bash $ jtc -tc file2.json 
{
   "days": {
      "2020-05-20": { "seconds": 95 },
      "2020-05-21": { "seconds": 80 },
      "2020-05-22": { "seconds": 120 }
   },
   "total": { "seconds": 295 }
}
bash $ jtc -J / -w[0] -mi[1:] / -w[0] -tc file*.json
{
   "days": {
      "2020-05-18": { "seconds": 60 },
      "2020-05-19": { "seconds": 30 },
      "2020-05-20": {
         "seconds": [ 1400, 95 ]
      },
      "2020-05-21": { "seconds": 80 },
      "2020-05-22": { "seconds": 120 }
   },
   "total": {
      "seconds": [ 1490, 295 ]
   }
}
bash $ 

-tc flag is used only to display JSONs here in a compact format

PS. It so happens that I'm the creator of jtc tool - unix JSON processing utility.

Answer 3

Try this:

jq -s add *.json > merged.json