From Excel to GCP

 Abstract:

There are a number of ways to automate things and make the work flow easier. For instance, to create a connected infrastructure, one had to manually create every resource involved and had to interlink them manually, but with the introduction of APIs, Infrastructure is being defined by a set of codes, hence the name Infrastructure as a Code (IaaC). With the help of enabled API, one can achieve such tasks by a language that he is comfortable with. Many such options include - Terraform, Ansible and many more. Here, we try to get resources and their specifications from user in an excel file (extension .xlsx) and create resources in Google Cloud Platform (GCP) with the help of Google's API client for Python. A well defined Excel file is created with sheets corresponding to some kind of resources such as VPCs, Subnets, Compute Instances, Firewall rules etc.. Such methods allows users to create a program and automate the working of the system in a very user friendly manner.

 

 GCP API client for Python:

APIs are kind-of interfaces that makes communication between two entities possible, may that be software applications or mixed hardware-software applications. There are two major ways to invoke a API when it comes to GCP - via HTTP requests and REST (Representational State Transfer). All the API client libraries makes use of such REST style of architecture. In such systems, when a client makes a request to perform some action, it is often sent in a representational form and the response is also returned after performing the task. Client must include the final desired state of the resource and the API takes care of the rest when requested and authenticated. 

All the resources are primarily stored in a data store, which are allotted to respective user or organization when a request is successfully made. The details of the final state of the system is embedded in the request and the API creates resources respectively. For instance, if one had to list all the instances in a particular zone of the project, A request has to be made to the API through client. The request must include the Project_ID and Zone. Regarding how it is done, we shall see later. For more information on API clients, you can click here.

 Compute Instance API

Google API client has a category that handles REST API requests regarding Instances, Firewalls, VPCs, Subnets, Instance groups, Images, Health checks and many more under a single class of compute. This can be fetched by referring to googleapiclient/discovery/build. Once this is fetched, one can insert, list or delete the listed features. 

For instance, if we want a Instance of certain specifications, then the REST request must comprise of few things, such as, name, machine_type, sourceImage, network, subnetwork, accessConfigs, metadata, tags etc..  

 {     'name':name,
        'machineType':machine_type,
        'disks': [
            { 'boot':True, 'autoDelete':True, 'initializeParams':                     {'sourceImage':disk_image,} }
            ],
        'networkInterfaces': [{
                    'network': vpcLinks[network],
                    'subnetwork':subnetLinks[subnet],
                    'accessConfigs': [
                        {'type': 'ONE_TO_ONE_NAT', 'name': 'External NAT'}
                ]    
                }],
               'metadata': {
                   'items': [{ 'key':'startup_script', 'value':open(startup).read()}],
               },
               'tags':{
                   'items':networkTags}}

In python, once these are specified in the form of a dictionary and this dictionary is fed to the compute.instances().insert() with projectID and zone. 

Handling Pandas:

Now when the task to connect to the GCP is done, extracting data from the Excel sheets is done using Pandas. It reads the Excel file (sheets) and creates a data-frame with the key name equivalent to the column name. we can refer to the whole column as a list by referring to the key name, like, dataframe[column] 

Every sheet of the file contains details about the resource that needs to be created, Pandas reads it and makes it a dataframe. A code is developed to segreagte and sort the things out into multiple dictionaries. These are then sent as requests to the API via the API Client and the resources are created.

Code can be found here

Scope:

It is to be noted that this was just demonstrating an example to create certain resources, but API Clients are capable doing much more than just this. This technique can be used to list out, create and even destroy destroy resources. Python being such a upcoming language, gives us more discretion to use this power to our use and customize things as we want.