'Event hub & Azure Data Explorer not ingesting all messages
For the last 12 hours, I am noticing Outgoing Messages
in the Event hub are too few than incoming messages. & more worrying thing is that event hub Outgoing Messages
frequency is the same (Like service is intentionally doing it). You can see in the below images that after 12 PM Outgoing Messages
are 100 per half hour. But there are more messages available.
Only Outgoing messages with Zoom
What I am doing
We are pushing application logs to Event Hub & Developer can watch these logs in ADX cluster.
Event hub configuration
- Zone Redundancy: Enabled
- Pricing tier: Standard
- Throughput Unit: 1 Unit
- Auto-inflate throughput: Disabled
ADX configuration
- Engine type: V3
- Compute specifications: Standard_D11_v2
- Instance count: 2
Event hub Ingestion metrics in ADX
About batching
After some googling, I found about batching. I thought may be resources are misconfigured but this is not the case
DB batch policy
.show database altslogsdb policy ingestionbatching
Output
"PolicyName": IngestionBatchingPolicy,
"EntityName": [altslogsdb],
"Policy": null,
"ChildEntities": [
"alts_logs_table"
],
"EntityType": Database,
Table batch policy
.show table alts_logs_table policy ingestionbatching
Output
"PolicyName": IngestionBatchingPolicy,
"EntityName": [altslogsdb].[alts_logs_table],
"Policy": null,
"ChildEntities": ,
"EntityType": Table,
May be issue got resolved automatically
After more than 24 hours, it may be got automatically resolved. I also created a support ticket. I don't know it got resolved automatically or Microsoft resolved it. Waiting for a mail from a support person.
As you can see, 3 spikes at last, I think that event hub was dropping messages because no messages are available in ADX & issue is there in that period also.
Update
Created ticket for Microsoft. Event hub and Azure data explorer teams are looking into it. The ticket is open for the last 2 months.
Solution 1:[1]
There are multiple checks that you should do:
- As Alex said above, take a look at the "insights" blade to see if there are ingestion failures on Azure Data Explorer side.
- You can check the EventHub ingestion metrics in your Azure Data Explorer cluster metrics:
- Check to see if there are errors on the EventHub such as throtteling
If you can't find the information there, please open a support ticket.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
Solution | Source |
---|---|
Solution 1 | Avnera |