jq how to pass json keys from a shell variable

Question

I have a json file I am parsing with jq. This is a sample of the file

[{
"key1":{...},
"key2":{...}
}]
[{
"key1":{...},
"key2":{...}
}]
...

each line is a list containing a json (which I know is not technically a json format but jq still works on such a file)

The below jq command works:

cat file.json | jq -r '.[] | [.key1,.key2]'

The above correctly shows:

[
<value_of_key1>,<value_of_key2>
]
[
<value_of_key1>,<value_of_key2>
]

However, I want .key1,.key2 to be dynamic since these keys can change. So I want to pass a variable to jq. Something like:

$KEYS=.key1,.key2
cat file.json | jq -r --arg var "$KEYS" '.[] | [$var]'

But the above is returning the keys themselves:

[
".key1,.key2"
]
[
".key1,.key2"
]

why is this happening? what is the correct command to make this happen?

This answer does not help me. I am not getting any errors as the OP in that question.

Answer 1

Fetching the value of a jq variable doesn't cause it to be executed as jq code.

Furthermore, jq lacks the facility to take a string, compile it as jq code, and evaluate the result. (This is commonly known as eval.)

So, short of a writing a jq parser and evaluator in jq, you will need to impose limits and/or accept a different format.

For example,

keys='[ [ "key1", "childkey" ], [ "key2", "childkey2" ] ]'  # JSON
jq --argjson keys "$keys" '.[] | [ getpath( $keys[] ) ]' file.json

or

keys='key1.childkey,key2.childkey2'
jq --arg keys "$keys" '
   ( ( $keys / "," ) | map( . / "." ) ) as $keys |
   .[] | [ getpath( $keys[] ) ]
' file.json

Answer 2

Suppose you have:

cat file
[{
"key1":1,
"key2":2
}]
[{
"key1":1,
"key2":2
}]

You can use a jq command like so:

jq '.[] | [.key1,.key2]' file
[
  1,
  2
]
[
  1,
  2
]

You can use -f to execute a filter from a file and nothing keeps you from creating the file separately from the shell variables.

Example:

keys=".key1"
echo ".[] | [${keys}]" >jqf

jq -f jqf file
[
  1
]
[
  1
]

Or just build the string directly into jq:

# note double " causing string interpolation
jq ".[] | [${keys}]" file

Answer 3

You can use --argjson option and destructuring.

file.json

[{"key1":{"a":1},"key2":{"b":2}}]
[{"key1":{"c":1},"key2":{"d":2}}]

$ in='["key1","key2"]' jq -c --argjson keys "$in" '$keys as [$key1,$key2] | .[] | [.[$key1,$key2]]' file.json

output:

[{"a":1},{"b":2}]
[{"c":1},{"d":2}]

Answer 4

Elaborating on ikegami's answer.

To start with here's my version of the answer:

$ in='key1.a,key2.b'; jq -c --arg keys "$in" '($keys/","|map(./".")) as $paths | .[] | [getpath($paths[])]' <<<$'[{"key1":{"a":1},"key2":{"b":2}}] [{"key1":{"a":3},"key2":{"b":4}}]'

This gives output

[1,2]
[3,4]

Let's try it.

We have input

[{"key1":{"a":1},"key2":{"b":2}}]
[{"key1":{"a":3},"key2":{"b":4}}]

And we want to construct array

[["key1","a"],["key2","b"]]

then use it on getpath(PATHS) builtin to extract values out of our input.

To start with we are given in shell variable with string value key1.a,key2.b. Let's call this $keys.

Then $keys/"," gives

["key1.a","key2.b"]
["key1.a","key2.b"]

After that $keys/","|map(./".") gives what we want.

[["key1","a"],["key2","b"]]
[["key1","a"],["key2","b"]]

Let's call this $paths.

Now if we do .[]|[getpath($paths[])] we get the values from our input equivalent to

[.[] | .key1.a, .key2.b]

which is

[1,2]
[3,4]