Intro to Data Structures

From an automation standpoint, it is important to understand what data structures are, how to use them, and more importantly how to access the data within the structure.

This post won’t go into how to build data structures as that is a whole topic in and of itself, but how do we obtain the data we want from a data structure that a system provides us such as Ansible, a web API, etc? We’re going to use Python when dissecting a data structure as we can use it to tell us what type the data is or what a specific type is within the data structure.

There are two main data types that we will discuss as they’re the most common.

Let’s start with looking at what a dictionary is and how we can get data from a dictionary using the built-in Python interactive interpreter.

Dictionaries

A dictionary is referred to as a mapping in other programming languages and is made up of key value pairs. The key must be an integer or a string, where the value may be any type of object. Dictionaries are mainly used when the order does not matter, but accessing specific data that can be found by a key. Let’s take a look at a dictionary.

>>> my_dict = {
...    'test_one': 'My first key:value',
...    'test': 'My second key:value',
...    10: 'Look at my key',
...}
>>> my_dict
{'test': 'My second key:value', 'test_one': 'My first key:value', 10: 'Look at my key'}
>>> my_dict.keys()
['test', 'test_one', 10]
>>> my_dict.values()
['My second key:value', 'My first key:value', 'Look at my key']
>>> type(my_dict)
<type 'dict'>

Note how the dictionary is in a different order than we created it in, which means the keys are significant in our ability to extract the values stored within the dictionary.

Let’s take a look at how to access data within a dictionary.

>>> my_dict['test_one']
'My first key:value'
>>> my_dict[10]
'Look at my key'
>>> my_dict['test_two']
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
KeyError: 'test_two'

Notice if we try and access a key that does not exist, we get a KeyError. This error can be avoided by using the .get() method on the dictionary. This will attempt to get the key from the dictionary and, if it doesn’t exist, will return None. It also accepts an argument that it will return if the key does not exist.

>>> test = my_dict.get("test_two")
>>> test
>>> type(test)
<class 'NoneType'>
>>> test = my_dict.get("test_two", "Return this value")

>>> test
'Return this value'
>>> if my_dict.get('test_one'):
...    print('It exists!')
...
It exists!

Now that we understand how dictionaries work and how we can obtain data from a dictionary, let’s move onto lists.

Lists

A list is referred to as an array in other programming languages and is a collection of different data types that are stored in indices within the list. A list can consist of objects of any type (strings, integers, dictionaries, tuples, etc.). The order of a list is maintained in the same order the list is created and the data can be obtained by accessing the indexes of the list.

Let’s take a look at creating a list.

>>> my_list = [
...     'index one',
...     {'test': 'dictionary'},
...     [1, 2 ,3],
... ]
>>> my_list
['index one', {'test': 'dictionary'}, [1, 2, 3]]
>>> for item in my_list:
...     type(item)
...
<type 'str'>
<type 'dict'>
<type 'list'>

As you can see, the list can store different data types, and the order is in the same order that we contructed the list in. We also iterated over the list to access each index, but we can access each item by the index they’re stored at as well.

>>> my_list[0]
'index one'
>>> my_list[1]
{'test': 'dictionary'}
>>> my_list[2]
[1, 2, 3]
>>> my_list[3]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
IndexError: list index out of range
>>> type(my_list)
<type 'list'>

The first index of a list starts at zero and increments up by one at each index. Let’s add a new item to the list and validate the order is still intact.

>>> my_list.append(5)
>>> my_list
['index one', {'test': 'dictionary'}, [1, 2, 3], 5]
>>> len(my_list)
4
>>> my_list[3]
5

Now that we understand how lists work and how we can obtain data from a list, let’s move on and put this all together when we encounter data structures in the wild!

How to Navigate Data Structures

After the above sections, it seems like navigating and obtaining the information from a data structure is a no-brainer, but can be intimidating when you come across a more complex data structure.

facts = {
    "ansible_check_mode": False,
    "ansible_diff_mode": False,
    "ansible_facts": {
        "_facts_gathered": True,
        "discovered_interpreter_python": "/usr/bin/python",
        "net_all_ipv4_addresses": [
            "192.168.1.1",
            "10.111.41.12",
            "172.16.133.1",
            "172.16.130.1",
        ],
        "net_filesystems": [
            "bootflash:"
        ],
        "net_filesystems_info": {
            "bootflash:": {
                "spacefree_kb": 5869720.0,
                "spacetotal_kb": 7712692.0
            }
        },
        "net_gather_network_resources": [],
        "net_gather_subset": [
            "hardware",
            "default",
            "interfaces",
            "config"
        ],
        "net_hostname": "csr1000v",
        "net_image": "bootflash:packages.conf",
        "net_interfaces": {
            "GigabitEthernet1": {
                "bandwidth": 1000000,
                "description": "MANAGEMENT INTERFACE - DON'T TOUCH ME",
                "duplex": "Full",
                "ipv4": [
                    {
                        "address": "10.10.20.48",
                        "subnet": "24"
                    }
                ],
                "lineprotocol": "up",
                "macaddress": "0050.56bb.e14e",
                "mediatype": "Virtual",
                "mtu": 1500,
                "operstatus": "up",
                "type": "CSR vNIC"
            }
        }
    }
}

The above data structure is what we get from gather facts in Ansible. We’re going to deal with the data structure outside of Ansible so we can determine breakdown each type data type in the structure. This is a great example as it’s a real world data structure and has nesting that we will need to traverse to get the data we want.

Let’s start by looking at the data type of the initial structure and then see how we can get the ansible_check_mode data.

>>> type(facts)
<class 'dict'>
>>> facts.get('ansible_check_mode')
False

As you can see, the initial data structure is a dictionary and since ansible_check_mode is in this initial dictionary it makes it a key. We can get the value of ansible_check_mode by using the .get() method.

What if we want to loop over all the IP addresses within net_all_ipv4_addresses? Let’s see how we can do that.

>>> type(facts['ansible_facts'])
<class 'dict'>
>>> facts['ansible_facts'].keys()
dict_keys(['_facts_gathered', 'discovered_interpreter_python', 'net_all_ipv4_addresses', 'net_filesystems', 'net_filesystems_info', 'net_gather_network_resources', 'net_gather_subset', 'net_hostname', 'net_image', 'net_interfaces'])
>>> type(facts['ansible_facts']['net_all_ipv4_addresses'])
<class 'list'>
>>> for ip in facts['ansible_facts']['net_all_ipv4_addresses']:
...     print(ip)
...
192.168.1.1
10.111.41.12
172.16.133.1
172.16.130.1

As we can see above, net_all_ipv4_addresses is a key within the ansible_facts dictionary. We have to navigate through two nested dictionaries to get to the list of IPv4 addresses we want to print out.

Let’s move on and obtain the IP address and subnet mask on GigabitEthernet1.

>>> type(facts['ansible_facts']['net_interfaces'])
<class 'dict'>
>>> type(facts['ansible_facts']['net_interfaces']['GigabitEthernet1'])
<class 'dict'>
>>> type(facts['ansible_facts']['net_interfaces']['GigabitEthernet1']['ipv4'])
<class 'list'>
>>> len(facts['ansible_facts']['net_interfaces']['GigabitEthernet1']['ipv4'])
1
>>> type(facts['ansible_facts']['net_interfaces']['GigabitEthernet1']['ipv4'][0])
<class 'dict'>
>>> gi1_subnet = facts['ansible_facts']['net_interfaces']['GigabitEthernet1']['ipv4'][0]['subnet']
>>> gi1_address = facts['ansible_facts']['net_interfaces']['GigabitEthernet1']['ipv4'][0]['address']
>>> f"{gi1_address}/{gi1_subnet}"
'10.10.20.48/24'

This is definitely a more complex data structure to traverse to get the data we need. We’ll walk through how to traverse this data structure.

We can see that net_interfaces is a dictionary so we’ll use a key to traverse to the next level. The data we’re interested in is in the GigabitEthernet1 key. We see that is also a dictionary so we understand to get to the next level in the hierarchy, we will use another key which is ipv4. The ipv4 data is stored within a list and the length of the list is one, which means we can access it at index zero. The data within index zero is a dictionary which means we now need to access the address and subnet via their respective keys.

We store the address and the subnet in their own variables that tell the story of how we got to the data we want.

dictionary[dictionary][dictionary][dictionary][dictionary][list][dictionary]

Now that you understand how each data type works, you can tackle any complex data structure you encounter.

Let’s take a look at another example and keep flexing this muscle memory. Let’s determine how much space has been used on bootflash: by subtracting the spacefree_kb value from the spacetotal_kb value.

>>> type(facts['ansible_facts'])
<class 'dict'>
>>> type(facts['ansible_facts']['net_filesystems_info'])
<class 'dict'>
>>> type(facts['ansible_facts']['net_filesystems_info']['bootflash:'])
<class 'dict'>
>>> space_free = facts['ansible_facts']['net_filesystems_info']['bootflash:']['spacefree_kb']
>>> space_free
5869720.0
>>> space_total = facts['ansible_facts']['net_filesystems_info']['bootflash:']['spacetotal_kb']
>>> space_total
7712692.0
>>> space_used = space_total - space_free
>>> space_used
1842972.0

As you can see, we had to navigate through four dictionaries including the intitial structure, but we didn’t have to navigate through any lists this time to get the data we wanted.

Remember that these complex data structures can be intimidating, but breaking down each data type within the structure helps us deconstruct them into smaller chunks to navigate and process until we get to the data we want.

-Mikhail