Filtering at start of query breaks filtering at end of query

Question

This is my JSON:

{
   "items":[
      {
         "fieldone":"sdfsdfsdfsdf",
         "fieldtwo":{
            "subfieldTwo":{
               "aaa/bbbb":"test-app-one"
            }
         },
         "fieldthree":{
            "subfieldThree":[
               {
                  "subObjField":"mark",
                  "otherfieldA":"sdfsdfsdf",
                  "otherfieldB":"dfsdfsdfsdf"
               },
               {
                  "subObjField":"mark",
                  "otherfieldA":"sdfsdfsdf",
                  "otherfieldB":"dfsdfsdfsdf"
               },
               {
                  "subObjField":"nomark",
                  "otherfieldA":"sdfsdfsdf",
                  "otherfieldB":"dfsdfsdfsdf"
               }
            ]
         }
      }
   ]
}

I want to filter items where fieldone == "sdfsdfsdfsdf" and return subfieldThree objects where subObjField == "mark". I cannot get these two things to work at the same time.

The following correctly returns subfieldThree objects of all items where subObjField == "mark"

items[*].fieldthree.subfieldThree[?subObjField == `mark`]

The following returns all subfieldThree objects of all items where fieldone == "sdfsdfsdfsdf"

items[?fieldone == `sdfsdfsdfsdf`].fieldthree.subfieldThree

But, if I try to apply the subObjField filter I get no output:

items[?fieldone == `sdfsdfsdfsdf`].fieldthree.subfieldThree[?subObjField == `mark`]

Why does fieldthree.subfieldThree[?subObjField == `mark`] totally break when I add that first filter?

I even tried with | and it's still not working, is this a bug or limitation of jmespath?

items[?fieldone == `sdfsdfsdfsdf`].fieldthree.subfieldThree | @[?subObjField == `mark`]

Answer 1

So, your self answer raise that the solution is indeed

items[?fieldone == `sdfsdfsdfsdf`]
  .fieldthree
  .subfieldThree[] | [?subObjField == `mark`]

But that you do not understand the reason of the usage of the pipe expression.

---

TL;DR;

This is because you have a projection as a result of your first filter, [?fieldone == `sdfsdfsdfsdf`], and you need to stop that projection.
And, since stopping a projection is done with a pipe expression in JMESPath, this is the correct way to go.

---

We could actually understand that behaviour on a very simplified JSON:

{
  "items": [{
    "fox": "quick",
    "dog": "lazy"
  }]
}

On this JSON, would you do the filter items[?fox == `quick`] you would get back the exact same array of objects and could think "well this is an array, I can throw in another filter, there".
So, you will do items[?fox == `quick`][?dog == `lazy`], and that is where the story tale ends, because JMESPath will yield you a desperately empty array — [] — in return.

^{Note: of course, in this simplified situation, you would resort to using the and operator — && — and would get away with an easier query: items[?fox == `quick` && dog == `lazy`]}

Why is that happening, though?
Well, because what you could assume being an array, after applying a filter, is not an array anymore, it is what JMESPath calls a projection.

From that chapter or the tutorial we learn that:

There’s a few things to keep in mind when working with projections. These are discussed in more detail in the wildcard expressions section of the spec, but the main points are:

• Projections are evaluated as two steps. The left hand side (LHS) creates a JSON array of initial values. The right hand side (RHS) of a projection is the expression to project for each element in the JSON array created by the left hand side. Each projection type has slightly different semantics when evaluating either the left hand side and/or the right hand side.
• If the result of the expression projected onto an individual array element is null, then that value is omitted from the collected set of results.
• You can stop a projection with a Pipe Expression (discussed later). A list projection is only valid for a JSON array. If the value is not a list, then the result of the expression is null.

^{Source: https://jmespath.org/tutorial.html#list-and-slice-projections}

So, you could consider those are really like Python or SQL views that you apply on an existing array.

And because of the two steps explained in the documentation here above, this other chapter goes further:

Projections are an important concept in JMESPath. However, there are times when projection semantics are not what you want. A common scenario is when you want to operate of the result of a projection rather than projecting an expression onto each element in the array.

^{Source: https://jmespath.org/tutorial.html#pipe-expressions}

Which is exactly what you needs here, you don't want to apply the filter fieldthree.subfieldThree[?subObjField == `mark`] — the right hand side (RHS) — to each of the elements of items[?fieldone == `sdfsdfsdfsdf`], you want to apply it on a resulting array, instead.

So you need a pipe expression in order to do that, the pipe expression indicates that the projection must stop.

You can stop it where you want in your object tree, though:

• items[?fieldone == `sdfsdfsdfsdf`] | []
    .fieldthree
    .subfieldThree[?subObjField == `mark`][]
  

• items[?fieldone == `sdfsdfsdfsdf`]
    .fieldthree | []
    .subfieldThree[?subObjField == `mark`][]
  

• items[?fieldone == `sdfsdfsdfsdf`]
    .fieldthree
    .subfieldThree[] | [?subObjField == `mark`]
  

Are three flavour of a query that would give the exact same result.

And once you have stopped the existing projection, and ends up with an actual array of objects, you are back to square one, and can apply another projection as you would like.

So, as a rule of thumb, if you need to "chain" filters, then you know you have to be resetting projections at some point, and would need the pipe expression.

Answer 2

I'm not sure why but I needed to flatten the results of the first query like this:

items[?fieldone == `sdfsdfsdfsdf`].fieldthree.subfieldThree[] | [?subObjField == `mark`]

This works, but I'm still ignorant of why it behaves like this — when do you need to use | and when do you not?