Upserting documents in elasticsearch using multiple fields instead of one

Question

Further to question Upsert documents in Elasticsearch using custom ID field, now I need to upsert the documents with 2+ fields from the input.

Taking the same example as the above question -

Sample data:

TABLE="TRADE"|TradeID="1234"|Qty=100|Price=100.00|BuyOrSell="BUY"|Stock="ABCD Inc."

if we receive modification on the above record:

TABLE="TRADE"|TradeID="1234"|Qty=120|Price=101.74|BuyOrSell="BUY"|Stock="ABCD Inc."

I need to upsert based on TradeID and Stock both. I could not find any documentation on-site mentioning it. I could actually create a new field that is the concatenation of two fields but I want to avoid it.

Answer 1

You need to create a compound ID with TradeID and Stock, something like

`document_id => "%{TradeID}-%{Stock}"`

It would be better to use a stock ticker instead of the stock name, though.

Another way is to use the fingerprint filter to create a consistent hash out of the TradeID and Stock values and then use that hash as the document ID in the output section:

filter {
  ...
  fingerprint {
    source => ["TradeID", "Stock"]
    target => "[@metadata][id]"
  }
  ...
}
output {
  elasticsearch {
    ...
    document_id => "%{[@metadata][id]}"
    ...
  }
}