'Azure data explorer ingestionbatching policy is not working as expected

I tried to ingest the records to the target table from the source table with the update policy and ingestion batching policy. Data records count in the target table keep on increasing in seconds even when the ingestion batching policy has been updated to one hour.

> .alter tables TestTable policy ingestionbatching
> @'{"MaximumBatchingTimeSpan":"01:00:00", "MaximumNumberOfItems": 100,
> "MaximumRawDataSizeMB": 500}'
azureazure-data-explorer


Solution 1:[1]

The documentation is quite clear about it:

The following list shows the basic batching policy triggers to seal a batch. A batch is sealed and ingested when the first condition is met:

  • Size: Batch size limit reached or exceeded
  • Count: Batch file number limit reached
  • Time: Batching time has expired

In addition:

The following list shows conditions to seal batches related to single blob ingestion. A batch is sealed and ingested when the conditions are met:

SingleBlob_FlushImmediately: Ingest a single blob because 'FlushImmediately' was set SingleBlob_IngestIfNotExists: Ingest a single blob because 'IngestIfNotExists' was set SingleBlob_IngestByTag: Ingest a single blob because 'ingest-by' was set SingleBlob_SizeUnknown: Ingest a single blob because blob size is unknown

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source
Solution 1

Related Questions