'Terraform and Azure Data Explorer
I am looking for recipes on how to standup a complete Azure Data Explorer stack using Terraform. So far I have found that Terraform has support to create clusters, apply access control and create databases.
I am yet to find guidance/recipes on creation of tables and table mappings.
How are others provisioning complete Azure Data Explorer stacks? What are some of the best practices to integrate create tables/table-mappings sql calls with Terraform scripts?
Thanks in advance.
Solution 1:[1]
We found much the same. Supposedly, support for Tables and Ingestion Mappings via ARM (or perhaps APIs accessible to Terraform) will be supported at some time.
However, the ADX team within Azure refers to tables and ingestion mappings as being part of the "data plane" -- not the "control plane." So their preferred approach is to use the management commands from a Kusto client to create tables and ingestion mappings.
We wound up creating a mini-migration tool in house to handle this work for us. It basically just reads through a list of files in a directory and applies any new migrations to the database that haven't previously been run (recording successful migrations to avoid re-running them). I would love to open source this tool, but we'd need sign-off from legal. Unlike other database technology you may be familiar with, I don't believe that ADX supports transactional updates to table structure, so we didn't even bother making rollback functionality for our tool.
There are some tools you can use that don't require writing up a whole migration tool of your own: https://docs.microsoft.com/en-us/azure/kusto/tools/#source-control-integration-tools
Solution 2:[2]
Also have a look at this article: ARM Deploy Azure Data Explorer DB with Event Hub connection (and the cited template).
It shows how to deploy a schema in ADX using an inline script (no longer need a storage account + SAS token). It also shows how to give permissions for ADX MSI to the Event Hub. A similar technique would be used to give permissions to ADX itself (for say, an app Service Principal).
The equivalent should be available in Terraform in a month or so.
Solution 3:[3]
Update This functionality is now released. Please check out this blog that explains how to configure a complete Azure Data Explorer with ARM templates that execute KQL configuration scripts during the deployment.
==================Previous Answer ==================
We are considering those.
Please vote for these on https://feedback.azure.com/d365community/idea/aa060248-ec24-ec11-b6e6-000d3a4f0da0
This is a known gap and we are looking at closing it. Till then you can execute control commands to create the table and mapping from a script/code.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|---|
| Solution 1 | bff1292 |
| Solution 2 | Vincent-Philippe Lauzon |
| Solution 3 |