link

Logo for g-c.dev

Terraform optional and variable objects

A bit of context...

In my current company, TomTom we implement infrastructure as code at scale, by offering blueprint and modules for all the engineering community in the company. In this way engineers can have an accelerated path for managing their infra, and leveraging well-defined, governed and standardised technologies.

This requires a central team to work on the implementation of such blueprint and architecture documentation. In my role I'm working with this team. Most of the functionalities are defined via Terraform Modules. Those are accompanied as well by (living) Architecture Documentation and Architecture Decision Records, so that engineers can understand the decisions we took while designing and implementing these capabilities.

Interfaces and variable files in Terraform

Even though Terraform is not a full fledge programming language, or an api endpoint, it still has a interfaces, and the engineering community consumes the terraform logic via these interfaces: variable and outputs files.

While dealing with global scale in a large engineering community (~2k engineers) we need to put particular attention on interfaces, specifically for a few reasons:

  1. They must be easy to consume, mostly without even reading documentation.

    Easiness comes also through a good set of sane defaults so we require the user to specify the least amount of information possible.

  2. They must be consistent
  3. They are not what we commonly know as API endpoint, nevertheless they must follow the famous 6 rules for api design

Flexible interfaces in Terraform

It was since a long time I was seeking for the Optional feature in Terraform to get out of experimental.

So far (until v1.3) the only way to enable the capability was to use the experiment 

terraform {
  experiments = [module_variable_optional_attrs]
}

With this feature (experimental since ~4 years) it was possible to define module interface by using the optional field, such as the following code: 

variable "cert_manager_cluster_issuer" {
  type = object({
    provider                        = optional(string)
    email                           = optional(string)
    subscription_id                 = optional(string)
    hosted_zone_resource_group_name = optional(string)
    hosted_zone_name                = optional(string)
    server                          = optional(string)
    assign_pod_identity             = optional(bool)
    use_workload_identity           = optional(bool)
    hydrant_id_api_key_secret       = optional(string)
    hydrant_id_api_key_id           = optional(string)
    certificate_validation          = optional(string)
    ingress_class                   = optional(string)
  })
  default     = {}
  description = "cluster issuer and pod identity configurations. Options for cluster issuer providers such as lets_encrypt and hydrant_id"
}

The way above, the user only needs to specify the elements in the object he wants to change from the default ones. For instance, to only change the ingress class:

module "my_cluster" {
  source = ... 
  cert_manager_cluster_issuer = {
    ingress_class = "traefik"
  } 
}

The only problem with this approach was (until the experiment was on) that you need to delegate to locals any non specified value you want to default.

Finally starting from v.1.3 of Terraform the experiment has been revised and merged in the core. With this new addition we are also able to specify the default value in the variable declaration, by simplifying the code and supporting out of the box documentation (we use Terraform Docs for this).

So the new implementation in Terraform >= v1.3 will be:

variable "cert_manager_cluster_issuer" {
  type = object({
    provider                        = optional(string, "lets-encrypt")
    email                           = optional(string, "n/a")
    # [...]
    ingress_class                   = optional(string, "nginx")
  })
  default     = {}
  description = "cluster issuer and pod identity configurations. Options for cluster issuer providers such as lets_encrypt and hydrant_id"
}

How this work at a scale

In our context, we offer Architecture documentation and implementation in a form of Terraform modules. Our engineers can decide to either use single modules, or opt-in for full-fledge root stack that glue together multiple modules offering a full-fledge functionality.

Terraform Optionals come very convenient in this scenario. Each module indeed has it's own set of variables, each one with it's defined type.
When it comes to offer a full-fledge functionality, we leverage consistently the optionals so the user has to configure only the customisation he needs and the code on its side is kept very clean and consistent.

Let's take as an example the full-fledge functionality of a K8S cluster containing various components and, among them, cert-manager.

The approach we offer can be summarised with the following:

graph TD
    A[Module cert-manager] --> |composed in| B(Stack K8S Cluster)
    U(user) -..-> |can consumes with variables| A 
    U(user) -..-> |or can consumes with wrapping objects and Optional variables| B

Most commonly the engineers leverage the entire full-fledge solution (so they don't have to recompose all the functionalities together). And this is an example configuration the consumer can provide to fine-tuning the full-fledge K8S Cluster stack:

region = "westeurope"

  cluster = {
    name                       = "test-cluster"
  }

  azure_native_monitoring_enabled = false

  external_dns = {
      # The hosted zone as mentioned in the prerequisites
      "domainFilters[0]" = include.root.inputs.hosted_zone.name
  }

  cert_manager_cluster_issuer = {
    # you can use your email group. This is used to receive notification from LetsEncrypt
    email = include.root.inputs.team_email

    # The hosted zone as mentioned in the prerequisites
    hosted_zone_resource_group_name = include.root.inputs.hosted_zone.resource_group
    hosted_zone_name                = include.root.inputs.hosted_zone.name

    certificate_validation = "http01"
  }

Please note the include.root.inputs comes from the pattern we use to orchestrate with Terragrunt the deployment of the terraform layers

From the excerpt above, you can see that by using the terraform object variables and optionals we achieve: