Introduction to Structured Data – Part 3

Throughout this blog series, we have been discussing the use of structured data in network automation. Part 1 began outlining the role of structured data and our working use case. Part 2 explored different dictionaries and methods used to interact with this structured data.

In this post, we’ll build from the previous two blogs by utilizing the original SNMP use case from part 1 and the CSV library in part 2. We’ll create an SNMP configuration for each device within the inventory to prep for deployment. To do so, we’ll utilize common open-source functionalities of Python and Jinja2. Let’s first explore the rationale of using these functionalities.

Jinja2 is a templating engine with consistency and layering to provide a cookie cutter approach, thereby making us not only faster and more efficient but also greatly reducing our chances for human error, such as mistyping or fat-fingering, as well as system problems such as buffer limitations, etc. This means we can now start to move away from impacting highly critical production changes to more of a “rinse-and-repeat” feel.

Jinja2 templates have been used for many years in web and application development. We’ll employ the solution to our networking use case to render SNMP device configurations. The idea behind a Jinja template is to retain the static or “unchanged” parts of the SNMP configuration and then use certain fields from our structured data to layer in details.

Step1 – Create SNMP Template

Our first step is to identify the information and the format that are needed for our SNMP configurations. In the first blog, we identified the following commands:

snmp-server location {data}
snmp-server contact {data}
snmp-server chassis-id {data}
snmp-server community {data} RO
snmp-server community {data} RW

We’ll keep the static SNMP command text required for the system we are configuring, and replace the {data} sections in each command with variable names.

This can be seen in the Jinja2 example below:

# hostname {{ DeviceName }}
snmp-server location {{ SiteName }} {{ Address}} {{ City }}, {{ State }} {{ Zip }}
snmp-server contact {{ Support }}
snmp-server chassis-id {{ SerialNumber }}
snmp-server community public RO
snmp-server community private RW

One thing to note is that the last two lines have “public” and “private” entered as static values rather than variables. For this example, we are making a decision to use a static value across all devices in our inventory rather than specifying a variable for them. It is definitely an option to replace those static values with variables if you use different SNMP strings in different parts of your network.

Step 2 – Review Device Inventory Data

We now need to consider the following questions to ensure that our structured data meets our requirements:

1. What is our input data type? In this use case we are using CSV, though we could have used Microsoft .xls, JSON, or YAML, as Jinja is flexible.
2. Do we have all the required fields? We need to ensure our data file contains all of the field values needed within our template. In our example, the data file meets these requirements. However, if it didn’t, we’d potentially need to add additional data files.

#Device Name,Manufacturer,Model,Serial Number,Site Name,Address,City,State,Zip,Country,Mgmt IP,Network Domain,Jump Host,Support
HQ-R1,Cisco,ISR 4431,KRG645782,Headquarters,601 E Trade St,Charlotte,NC,28202,USA,192.168.10.2,Access,10.20.5.5,HQ IT 704-123-4444
HQ-R2,Cisco,ISR 4431,KRG557862,Headquarters,601 E Trade St,Charlotte,NC,28202,USA,192.168.10.3,Access,10.20.5.5,HQ IT 704-123-4444
HQ-S1,Cisco,CAT3560-48PS,GRN883274,Headquarters,601 E Trade St,Charlotte,NC,28202,USA,192.168.10.4,Access,10.20.5.5,HQ IT 704-123-4444
HQ-S2,Cisco,CAT3560-48PS,GRN894532,Headquarters,601 E Trade St,Charlotte,NC,28202,USA,192.168.10.5,Access,10.20.5.5,HQ IT 704-123-4444
PD-N-R1,Cisco,ISR 4431,FOM123124,Police Dept (North),10430 Harris Oaks Blvd,Charlotte,NC,28269,USA,192.168.100.1,Access,10.100.5.5,N IT 704-555-1234

Because we are using a CSV file, we’ll also need to review format handling. CSVs use delimiters to help separate the different fields within your file. If not for the delimiter, we’d have one very long flat file.

print("Start")
with open(sys.argv[1], 'r') as f:
    reader = csv.reader(f, delimiter =',')
    for row in reader:
        if not row[0][0] == '#':
            templateVars = { "DeviceName" : row[0],
                "Manufacturer": row[1],
                "Model": row[2],
                "SerialNumber": row[3],
                "SiteName": row[4],
                "Address": row[5],
                "City": row[6] }

Step 3 – Pulling It All Together with Python

The Python script we’ll use will utilize standard Python libraries in addition to the Jinja2 library. The Jinja2 library will need to be installed in your environment prior to running the script. To do this you’ll need to use pip install:

pip install jinja2

Then import the following into our Python script:

import jinja2
import csv
import sys
import os

The script will be the glue between our jinja2 template containing our SNMP commands and the csv file with our device data. It will read in a line from the device inventory csv and render the data in that row against our template. Once the data is joined, we’ll create a new directory per device to place our configuration in the output. At the end, the base directory will contain a folder for each device in our inventory and the SNMP deployment script for that device.

def main():
    templateLoader = jinja2.FileSystemLoader(searchpath="./")
    templateEnv = jinja2.Environment(loader=templateLoader)
    TEMPLATE_FILE = sys.argv[2]
    template = templateEnv.get_template(TEMPLATE_FILE)
  
    dir = sys.argv[3]
    if not os.path.exists(dir):
        os.mkdir(dir)
  
    print("Start")
    with open(sys.argv[1], 'r') as f:
        reader = csv.reader(f, delimiter =',')
        for row in reader:
            if not row[0][0] == '#':
                templateVars = { "DeviceName" : row[0],
                  "Manufacturer": row[1],
                  "Model": row[2],
                  "SerialNumber": row[3],
                  "SiteName": row[4],
                  "Address": row[5],
                  "City": row[6],
                  "State": row[7],
                  "Zip": row[8],
                  "Country": row[9],
                  "MgmtIP": row[10],
                  "NetworkDomain": row[11],
                  "JumpHost": row[12],
                  "Support": row[13]
                }
                outputText = template.render(templateVars)
                filename = dir+"\\"+row[0]
                file = open(filename,'w')
                file.write(outputText)
                file.close()
                print("Create Config for: " + str(row[0]))
    print("Finish")

if len(sys.argv) < 3:
    print("missing command line syntax")
else:
    main()

Step 4 – Rendering Our SNMP Configurations

To invoke our SNMP configuration, we’ll use the following command line format with our main.py Python script, the snmp.csv device inventory data, our Jinja2 snmp-template.j2, and our base directory named configs/.

~/Python-Jinja-Blog$ 
~/Python-Jinja-Blog$ python main.py snmp.csv snmp-template.j2 configs/

When we run this command, we see statements returned from the Python script showing us the script started, created a configuration for each device in our inventory, and then completed.

~/Python-Jinja-Blog$ 
~/Python-Jinja-Blog$ python main.py snmp.csv snmp-template.j2 configs/
Start
Create Config for: HQ-R1
Create Config for: HQ-R2
Create Config for: HQ-S1
Create Config for: HQ-S2
Create Config for: PD-N-R1
Create Config for: PD-N-S1
Create Config for: PD-S-R1
Create Config for: PD-S-R2
Finish
~/Python-Jinja-Blog$ 

~/Python-Jinja-Blog$ cd configs/
~/Python-Jinja-Blog/configs$ ls
'\HQ-R1'  '\HQ-S1'  '\PD-N-R1'  '\PD-S-R1'
'\HQ-R2'  '\HQ-S2'  '\PD-N-S1'  '\PD-S-R2'
~/Python-Jinja-Blog/configs$ 

And if we then dig into the \PD-N-R1 directory and open up the configuration file, we see that the necessary data from our device inventory file was added to our Jinja2 template during the Python script run to render our SNMP configuration template for this specific device:

# hostname PD-N-R1
snmp-server location Police Dept (North) 10430 Harris Oaks Blvd Charlotte, NC 28269
snmp-server contact N IT 704-555-1234
snmp-server chassis-id FOM123124
snmp-server community public RO
snmp-server community private RW

With these output files, we can now apply commands to our device manually, or through an automated process to collect and deploy the configurations from the system.