How do you choose an unused VPC CIDR Block with Terraform?

Question

Set up: I will be programatically creating environments using terraform. There are already a few VPCs that exist, and I want to ensure that when creating a new VPC with terraform it won't clash with existing ones.

First step, I can get the current state of the world with this

data "aws_vpcs" "current" { }

But I am unsure how to choose an IP address that isn't in the list. For example, we are using 172.x.0.0/16 as a format.

In terraform, how can I choose the lowest x that is not currently in use?

Answer 1

Terraform alone is not suitable for this dynamic allocation task because the result would be unstable: if you were to hypothetically write a Terraform configuration to find the highest allocated CIDR block and then use that for a new aws_vpc then the next time you run Terraform it would then find that the highest allocated CIDR block has changed and then plan to replace the VPC with a higher number, and keep doing so until the IP address space is exhausted. Terraform data resources do not have any "memory" between runs, because they are designed to allow a Terraform configuration to adapt to a changing environment around it.

What to do instead depends on whether you prefer a top-down manual allocation approach or a dynamic "allocate on request" approach. It sounds from your question like you prefer the second of these but I will discuss the first one first because it's the one I've more commonly seen and is more straightforward.

---

For top-down manual allocation you could write a Terraform module whose only purpose is to describe allocations from your IP address space. Each time you want to allocate a new addres space you'd first change that module to include a new allocation for its address space, and then use its result to select the appropriate base address range for your current configuration. There is a module in Terraform Registry called hashicorp/subnets/cidr that aims to make it easier to build a module like that, by defining some simple rules for backward-compatible changes to allocations later.

(A variant of the above is to use a specialized system designed for cataloging network address spaces, like NetBox, but ultimately these reduce down to the same thing: a centrally-managed registry of address space allocations which humans and software can refer to.)

---

For an "allocate on request" approach you'll need some separate system to respond to allocation requests and remember which address space was allocated to each system so that it can guarantee to always return the same allocation to each caller on future Terraform runs. Such a system will presumably need a persistent data store so that it can recall for each caller whether an address range was already allocated to it and, if so, return the same allocation rather than creating a new one.

I'm not personally aware of any specialized software for this task, but you could potentially build a simple service yourself which responds to an HTTP request containing a system identifier by returning an address range which is either freshly-allocated or previously-allocated using the system identifier as the unique key:

data "http" "cidr_block" {
  url = "https://example.com/cidr-block-allocation/example-system"
}

locals {
  cidr_block = chomp(data.http.cidr_block.body)
}

In the above I assumed that this hypothetical endpoint returns a text/plain response containing just the address range in CIDR notation, but it could potentially be a JSON response parsed with Terraform's jsondecode function, for example.

The important distinction between this data block and the one in your original question is that the persistent record of what was allocated lives in the server running at example.com independently of what VPCs exist in AWS, and so creating a new VPC at the indicated address range won't change the answer for future requests.

Answer 2

Not sure if you still need this answer, but I just finished implementing a solution for it.

It creates /16 VPCs between 172.16.0.0/16 and 172.31.0.0/16.

To work properly, VPCs must have unique names and all VPCs in the 172.* must be /16. If all 16 CIDRs are taken, it will default to "10.254.0.0/16"

variable "vpc_name" {
  type            = string
  description     = "VPC name"
  default         = "test"
}

locals {
  cidr_base       = "172.16.0.0/12"
  cidr_fail       = "10.254.0.0/16"
  vpc_max         = 16
  cidr_available  = [for index, x in data.aws_vpcs.all: cidrsubnet(local.cidr_base, 4, "${index}") if length(x.ids) == 0]
}

#Check if self exists
data "aws_vpcs" "self" {
  tags = {
    Name          = var.vpc_name
  }
}

#Get self data
data "aws_vpc" "self" {
  count           = length(data.aws_vpcs.self.ids) > 0 ? 1 : 0
  id              = data.aws_vpcs.self.ids[0]
}

#Get all VPCs
data "aws_vpcs" "all" {
  count           = local.vpc_max
  filter {
    name          = "cidr"
    values        = [cidrsubnet(local.cidr_base, 4, count.index)]
  }
}

#Use self CIDR or look for unused CIDR
resource "aws_vpc" "main" {
  cidr_block      = "${
                        length(data.aws_vpcs.self.ids) > 0  ? data.aws_vpc.self[0].cidr_block : 
                        length(local.cidr_available) > 0    ? local.cidr_available[0] : 
                        local.cidr_fail
                      }"
  tags = {
    Name          = var.vpc_name
  }
}

If you need more VPCs, you can change the variables to create the VPCs in the 10.*** private block, it would give you 255 VPCs