Terraform - Modules

 T of Terraform:

Terraform is a toll that helps us in building, changing and versioning in the resources of an infrastructure. Not necessarily though, because, Terraform an execute configuration files and request a API to bring the state of the infrastructure or that entity close to that as desired in configuration file (as much as possible). Terraform can serve multiple platforms including AWS, Azure, GCP, IBM-cloud, Alibaba cloud, VMware. To makes things interesting, these state dependent programs can handle any platform which serves requests from API call, when the providers are written for that platform. 

Talking about GCP, we can use existing provider, already written and just write down the configuration file and request API to achieve the state as provided in the configuration file. It has to be written in HashiCorp Configuration Language (HCL), wherein the descriptions of resources using blocks, arguments and expressions are provided.

Little into depth:

It is always better to organize things in such a way that they are easy to comprehend, well structured and reusable in many cases, but also not so vague to make the user rewrite a number of parameters every time they use it. Modules serve the purpose. A module can be created, stored and called whenever required. This is more like using classes and functions in other OOPs based languages. 

They can be organized in general style or any convenient method as preferred by the user. An example is shown, wherein the modules are stored under a single directory, categorized based in the resources they refer or work with. 

The environments such as global, production etc.. are kept separately and the modules are called and used whenever required, hence organizing it systematically and also using comfortably!

Example!

Here, I have tried to create couple of modules for resources and a code to use those modules to accomplish the task. This below is a module file of creating an instance. We can declare all the variables in the same file or make a separate file for that, Terraform reads all the files in the folder with the extension .tf  .

https://gist.github.com/group4pgs/1cf6a514c0ab8d78bae18bad20451b57

This file is saved in a location and the address is noted down, this helps in locating the module for the instance creation. We call this module in the way, shown below. One must make sure that every variable required by the module is addressed correctly.

https://gist.github.com/group4pgs/3ca92c45ec7094bcc0e386e767c203a7

 

Concluding things

Just one of the few functionalities that Terraform provides which are of great use. One can check the Official website of Terraform which already has many modules, preconfigured, you can directly refer to the github repo, and make use of the publicly available modules to build their own resources in an infrastructure. 

From Excel to GCP

 Abstract:

There are a number of ways to automate things and make the work flow easier. For instance, to create a connected infrastructure, one had to manually create every resource involved and had to interlink them manually, but with the introduction of APIs, Infrastructure is being defined by a set of codes, hence the name Infrastructure as a Code (IaaC). With the help of enabled API, one can achieve such tasks by a language that he is comfortable with. Many such options include - Terraform, Ansible and many more. Here, we try to get resources and their specifications from user in an excel file (extension .xlsx) and create resources in Google Cloud Platform (GCP) with the help of Google's API client for Python. A well defined Excel file is created with sheets corresponding to some kind of resources such as VPCs, Subnets, Compute Instances, Firewall rules etc.. Such methods allows users to create a program and automate the working of the system in a very user friendly manner.

 

 GCP API client for Python:

APIs are kind-of interfaces that makes communication between two entities possible, may that be software applications or mixed hardware-software applications. There are two major ways to invoke a API when it comes to GCP - via HTTP requests and REST (Representational State Transfer). All the API client libraries makes use of such REST style of architecture. In such systems, when a client makes a request to perform some action, it is often sent in a representational form and the response is also returned after performing the task. Client must include the final desired state of the resource and the API takes care of the rest when requested and authenticated. 

All the resources are primarily stored in a data store, which are allotted to respective user or organization when a request is successfully made. The details of the final state of the system is embedded in the request and the API creates resources respectively. For instance, if one had to list all the instances in a particular zone of the project, A request has to be made to the API through client. The request must include the Project_ID and Zone. Regarding how it is done, we shall see later. For more information on API clients, you can click here.

 Compute Instance API

Google API client has a category that handles REST API requests regarding Instances, Firewalls, VPCs, Subnets, Instance groups, Images, Health checks and many more under a single class of compute. This can be fetched by referring to googleapiclient/discovery/build. Once this is fetched, one can insert, list or delete the listed features. 

For instance, if we want a Instance of certain specifications, then the REST request must comprise of few things, such as, name, machine_type, sourceImage, network, subnetwork, accessConfigs, metadata, tags etc..  

 {     'name':name,
        'machineType':machine_type,
        'disks': [
            { 'boot':True, 'autoDelete':True, 'initializeParams':                     {'sourceImage':disk_image,} }
            ],
        'networkInterfaces': [{
                    'network': vpcLinks[network],
                    'subnetwork':subnetLinks[subnet],
                    'accessConfigs': [
                        {'type': 'ONE_TO_ONE_NAT', 'name': 'External NAT'}
                ]    
                }],
               'metadata': {
                   'items': [{ 'key':'startup_script', 'value':open(startup).read()}],
               },
               'tags':{
                   'items':networkTags}}

In python, once these are specified in the form of a dictionary and this dictionary is fed to the compute.instances().insert() with projectID and zone. 

Handling Pandas:

Now when the task to connect to the GCP is done, extracting data from the Excel sheets is done using Pandas. It reads the Excel file (sheets) and creates a data-frame with the key name equivalent to the column name. we can refer to the whole column as a list by referring to the key name, like, dataframe[column] 

Every sheet of the file contains details about the resource that needs to be created, Pandas reads it and makes it a dataframe. A code is developed to segreagte and sort the things out into multiple dictionaries. These are then sent as requests to the API via the API Client and the resources are created.

Code can be found here

Scope:

It is to be noted that this was just demonstrating an example to create certain resources, but API Clients are capable doing much more than just this. This technique can be used to list out, create and even destroy destroy resources. Python being such a upcoming language, gives us more discretion to use this power to our use and customize things as we want.