'How to find the number of Inserts and Updates of Merge command?
I have code similar to this in Spark(Scala). I would like to know the number of records this code updated/inserted when execute()
is complete. Is there a way?
deltaTable
.as("target")
.merge(
source.as("source"),
"target.key = source.key")
.whenMatched
.updateExpr(Map(
"value" -> "source.value"))
.whenNotMatched
.insertExpr(Map(
"key" -> "source.key",
"value" -> "source.value"))
.execute()
Solution 1:[1]
You can get this data from the Delta table history - in Scala it's just
val lastOperationDF = deltaTable.history(1)
Then you get DataFrame, where you have the operationMetrics
column that is a map from metric name to value. According to the documentation about metrics, information that you need is in the keys numTargetRowsInserted
& numTargetRowsUpdated
.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
Solution | Source |
---|---|
Solution 1 | Alex Ott |