Terraform — Use of For each and Lookup
I recently worked on a ticket where I was supposed to create Kafka topics with topic level customization and the existing scripts utilized in the project were not supporting this use-case. Let us see what the use-case in detailed below
You are given a list of topics that are supposed to be created across environments along with their properties:
Here we can see that there is different configuration for each one. The existing module was available as shown below:
In order to modify this we needed to add configuration block to this kafka_topic resource.
In order to this and utilize this module for creating N number of topics there was need to implement iteration based on number of topics present in configuration. Now the scripts looks like as follows:
Here var.topic_name belongs to a list of topics that you will be creating which can be declared similarly as follows:
Now, this implementation solves the problem of creating multiple topics with the same configuration. Now let’s solve how we can utilize the same script where it can allow us to define individual topic with its own set of configuration. For this we will use lookup function available in terraform.
Using lookup we will maintain a map of KV pair where Key will be the topic name that we are going to create and Value will be the desired configuration value needed for each topic. It will be set up as shown below:
On the main script creates Kafka topics it will look like as follows:
And this solves the our use-case. If you find any better solution for this do let me know in the comments.
The complete source code for this is present at the following link:
Resources for I used for this blog in case you want to learn more about each implementation:
