Automated Cable Verification for Junos Devices with Ansible

  • September 19, 2016

Objective

This tutorial walks through how to use Ansible to do automated and dynamic cable verification for Junos devices. Our testing will specifically use Junos vMX devices.

Topology

This tutorial was written using the Juniper vMX 5-node topology.

WorkFlow

Before we dive in to actually verifying the cabling in the 5-node topology, let's review the workflow of what's needed.

  1. Most importantly, we need to define what the desired cabling topology should be. This is often the hardest, not from a technical perspective, but from process perspective and being disciplined to do so before you start cabling things up.
  2. Perform the deployment (rack/stack/cabling) - of course, our topology is virtual, so there is no need for this.
  3. Gather the real-time neighbors from the device
  4. Perform verification checks - there can be plenty of checks here. We will focus on two checks for each device.
    • We will verify that the same quantity of numbers matches for each interface as we compare the desired cabling topology against the real-time topology (from LLDP neighbors).
    • Additionally, we'll ensure the list of neighbors ifound on each interface via LLDP matches the list of neighbors defined in our desired topology.

Define Desired Topology

There are numerous ways to define topologies. Structured ways using .dot files can be used, but because we are using Ansible, we'll simply define our topology in a YAML variables file. We'll store it in ./group_vars/all.yml. You could also break it out in host based variables files as well, but to minimize our touch points, we'll store it in a single file.

Here is the file we are using that defines the topology for the entire 5 node topology:

---

desired:
  vmx1:
    fxp0:
      neighbors: ['vmx2', 'vmx3', 'vmx4', 'vmx5']
    ge-0/0/0:
      neighbors: ['vmx3']
    ge-0/0/2:
      neighbors: ['vmx2']
    ge-0/0/3:
      neighbors: ['vmx4']
  vmx2:
    fxp0:
      neighbors: ['vmx1', 'vmx3', 'vmx4', 'vmx5']
    ge-0/0/1:
      neighbors: ['vmx3']
    ge-0/0/2:
      neighbors: ['vmx1']
    ge-0/0/3:
      neighbors: ['vmx5']
  vmx3:
    fxp0:
      neighbors: ['vmx1', 'vmx2', 'vmx4', 'vmx5']
    ge-0/0/0:
      neighbors: ['vmx1']
    ge-0/0/1:
      neighbors: ['vmx2']
  vmx4:
    fxp0:
      neighbors: ['vmx1', 'vmx2', 'vmx3', 'vmx5']
    ge-0/0/2:
      neighbors: ['vmx5']
    ge-0/0/3:
      neighbors: ['vmx1']
  vmx5:
    fxp0:
      neighbors: ['vmx1', 'vmx2', 'vmx3', 'vmx4']
    ge-0/0/2:
      neighbors: ['vmx4']
    ge-0/0/3:
      neighbors: ['vmx2']

Gather Real-Time Topology

Next, we need to gather the topology using LLDP from each device. In order to do this, we'll use the Juniper Ansible module called junos_get_table. This module leverages Juniper Tables & Views, which are used to simplify getting operational (and configuration) data out of Junos devices. Since Juniper already has a pre-built table for LLDP, we'll simply use that one.

---

  - name: TOPOLOGY AND CABLE VERIFICATION
    hosts: vmx
    connection: local
    gather_facts: no

    tasks:

      - name: GET REAL-TIME (EXISTING) NEIGHBORS
        junos_get_table:
          host: "{{ inventory_hostname }}"
          user: "{{ un }}"
          passwd: "{{ pwd }}"
          table: LLDPNeighborTable
          file: lldp.yml
        register: neighbors_list

We haven't shown our inventory file yet, but we have a group called vmx that has all 5 vMX devices. In the first task, we're simply extracting the LLDP neighbors from each device and saving them (registering) them into a new variable called neighbors_list.

Parse Neighbors List

Take note that the neighbors returned are currently a list of dictionaries. In order to ease working with this object further in the playbook, we are going to convert the list of dictionaries to a dictionary.

Here is the conversation we are making:

neighbors_list is this:

ok: [vmx1] => {
    "neighbors_list": {
        "changed": false, 
        "resource": [
            {
                "local_int": "fxp0", 
                "local_parent": "-", 
                "remote_chassis_id": "00:05:86:71:27:c0", 
                "remote_port_desc": null, 
                "remote_sysname": "vmx2", 
                "remote_type": "Mac address"
            }, 
            {
                "local_int": "ge-0/0/2", 
                "local_parent": "-", 
                "remote_chassis_id": "00:05:86:71:27:c0", 
                "remote_port_desc": null, 
                "remote_sysname": "vmx2", 
                "remote_type": "Mac address"
            }, 
            {
                "local_int": "fxp0", 
                "local_parent": "-", 
                "remote_chassis_id": "00:05:86:71:80:c0", 
                "remote_port_desc": null, 
                "remote_sysname": "vmx4", 
                "remote_type": "Mac address"
            }, 
            {
                "local_int": "ge-0/0/3", 
                "local_parent": "-", 
                "remote_chassis_id": "00:05:86:71:80:c0", 
                "remote_port_desc": null, 
                "remote_sysname": "vmx4", 
                "remote_type": "Mac address"
            }, 
            {
                "local_int": "fxp0", 
                "local_parent": "-", 
                "remote_chassis_id": "00:05:86:71:b0:c0", 
                "remote_port_desc": null, 
                "remote_sysname": "vmx3", 
                "remote_type": "Mac address"
            }, 
            {
                "local_int": "ge-0/0/0", 
                "local_parent": "-", 
                "remote_chassis_id": "00:05:86:71:b0:c0", 
                "remote_port_desc": null, 
                "remote_sysname": "vmx3", 
                "remote_type": "Mac address"
            }, 
            {
                "local_int": "fxp0", 
                "local_parent": "-", 
                "remote_chassis_id": "00:05:86:71:c2:c0", 
                "remote_port_desc": null, 
                "remote_sysname": "vmx5", 
                "remote_type": "Mac address"
            }
        ]
    }
}

As soon as we convert it (we'll be doing this with a custom Jinja2 filter), it'll end up looking like this:

ok: [vmx1] => {
    "existing": {
        "fxp0": [
            {
                "local_int": "fxp0", 
                "local_parent": "-", 
                "remote_chassis_id": "00:05:86:71:27:c0", 
                "remote_port_desc": null, 
                "remote_sysname": "vmx2", 
                "remote_type": "Mac address"
            }, 
            {
                "local_int": "fxp0", 
                "local_parent": "-", 
                "remote_chassis_id": "00:05:86:71:80:c0", 
                "remote_port_desc": null, 
                "remote_sysname": "vmx4", 
                "remote_type": "Mac address"
            }, 
            {
                "local_int": "fxp0", 
                "local_parent": "-", 
                "remote_chassis_id": "00:05:86:71:b0:c0", 
                "remote_port_desc": null, 
                "remote_sysname": "vmx3", 
                "remote_type": "Mac address"
            }, 
            {
                "local_int": "fxp0", 
                "local_parent": "-", 
                "remote_chassis_id": "00:05:86:71:c2:c0", 
                "remote_port_desc": null, 
                "remote_sysname": "vmx5", 
                "remote_type": "Mac address"
            }
        ], 
        "ge-0/0/0": [
            {
                "local_int": "ge-0/0/0", 
                "local_parent": "-", 
                "remote_chassis_id": "00:05:86:71:b0:c0", 
                "remote_port_desc": null, 
                "remote_sysname": "vmx3", 
                "remote_type": "Mac address"
            }
        ], 
        "ge-0/0/2": [
            {
                "local_int": "ge-0/0/2", 
                "local_parent": "-", 
                "remote_chassis_id": "00:05:86:71:27:c0", 
                "remote_port_desc": null, 
                "remote_sysname": "vmx2", 
                "remote_type": "Mac address"
            }
        ], 
        "ge-0/0/3": [
            {
                "local_int": "ge-0/0/3", 
                "local_parent": "-", 
                "remote_chassis_id": "00:05:86:71:80:c0", 
                "remote_port_desc": null, 
                "remote_sysname": "vmx4", 
                "remote_type": "Mac address"
            }
        ]
    }
}

It's a subtle difference, but now we can more easily access the neighbors on each interface.

This is the task to perform this conversion:

  - name: CONVERT LIST OF DICTS TO DICT (WITH LISTS) AND SET FACT
    set_fact: existing={{ neighbors_list.resource|list_to_dict('local_int') }}

Important: This Jinja2 filter, list_to_dict, is custom and still under development, but instructions on how to use it are at the bottom of this tutorial.

Perform Final Verification

As mentioned, we are going to perform two checks on each device. In order to perform these checks, we are going to use the assert module. Each check we want to perform will be explicitly listed in our task as shown below.

  - name: ASSERT THE QUANTITY OF NEIGHBORS ON EACH INTERFACE IS CORRECT AND THE NEIGHBORS FOUNDON EACH IS CORRECT
    assert:
      that:
        - existing[item.key]|length == item.value.neighbors|length
        - existing[item.key]|map(attribute='remote_sysname')|list|sort == item.value.neighbors|sort
    with_dict: "{{ desired[inventory_hostname] }}"

The first check is verifying the right quantity of neighbors is seen on each interface. The second check is verifying the correct hostnames of the neighbors are found on the proper interface that map back to our desired topology.

You can also see how valuable Jinja2 filters are. All of the filters in this task are completely built into Ansible. The length filter simply returns the length of the list object. The filter show as map(attribute='remote_sysname') takes a list of dictionaries, but returns the specific element in each dictionary that correlates to the attribute. Then we simply chain two more filters list and sort so we end up with a sorted list of just the neighbor hostnames. We then also sort the names of the neighbors found in the desired topology even though they are sorted already just as a safe guard.

In preparation to run the full playbook, let's re-cap what our working environment looks like.

Ansible Inventory File

Stored in our project directory as ./inventory:

[all:vars]
un=ntc
pwd=ntc123

[vmx]
vmx1
vmx2
vmx3
vmx4
vmx5

Desired Topology Variables

Stored in our project directory as ./group_vars/all.yml:

---

desired:
  vmx1:
    fxp0:
      neighbors: ['vmx2', 'vmx3', 'vmx4', 'vmx5']
    ge-0/0/0:
      neighbors: ['vmx3']
    ge-0/0/2:
      neighbors: ['vmx2']
    ge-0/0/3:
      neighbors: ['vmx4']
  vmx2:
    fxp0:
      neighbors: ['vmx1', 'vmx3', 'vmx4', 'vmx5']
    ge-0/0/1:
      neighbors: ['vmx3']
    ge-0/0/2:
      neighbors: ['vmx1']
    ge-0/0/3:
      neighbors: ['vmx5']
  vmx3:
    fxp0:
      neighbors: ['vmx1', 'vmx2', 'vmx4', 'vmx5']
    ge-0/0/0:
      neighbors: ['vmx1']
    ge-0/0/1:
      neighbors: ['vmx2']
  vmx4:
    fxp0:
      neighbors: ['vmx1', 'vmx2', 'vmx3', 'vmx5']
    ge-0/0/2:
      neighbors: ['vmx5']
    ge-0/0/3:
      neighbors: ['vmx1']
  vmx5:
    fxp0:
      neighbors: ['vmx1', 'vmx2', 'vmx3', 'vmx4']
    ge-0/0/2:
      neighbors: ['vmx4']
    ge-0/0/3:
      neighbors: ['vmx2']

Using the Custom Jinja2 Filter

As mentioned already, to run the tasks above that uses the custom filter, you need to have it in a certain directory. We currently have it in ./filter_plugins/list_to_dict.py. And this is the file:

#! /usr/bin/env python

from ansible import errors


def list_to_dict(data, key):
    '''Key must be passed in when calling from a Jinja template
    '''

    new_obj = {}

    for item in data:
        try:
            key_elem = item.get(key)
        except Exception, e:
            raise errors.AnsibleFilterError(str(e))
        if key_elem:
            if new_obj.get(key_elem):
                new_obj[key_elem].append(item)
            else:
                new_obj[key_elem] = []
                new_obj[key_elem].append(item)

    return new_obj

class FilterModule(object):
    '''Convert a list of dictionaries to a dictionary provided a
       key that exists in all dicts.  If it does not, that dict is omitted
    '''
    def filters(self):
        return {
            'list_to_dict': list_to_dict
        }

Complete Ansible Playbook

Finally, here is the complete playbook with a few added debug statements to see what each object looks like:

Saved as ./junos-verify-cabling.yml:

---

  - name: TOPOLOGY AND CABLE VERIFICATION
    hosts: vmx
    connection: local
    gather_facts: no

    tasks:

      - name: GET REAL-TIME (EXISTING) NEIGHBORS
        junos_get_table:
          host: "{{ inventory_hostname }}"
          user: "{{ un }}"
          passwd: "{{ pwd }}"
          table: LLDPNeighborTable
          file: lldp.yml
        register: neighbors_list

      - debug: var=neighbors_list verbosity=1

      - name: CONVERT LIST OF DICTS TO DICT (WITH LISTS) AND SET FACT
        set_fact: existing={{ neighbors_list.resource|list_to_dict('local_int') }}

      - name: REAL-TIME (EXISTING) NEIGHBORS
        debug: var=existing verbosity=1

      - name: PROPOSED TOPOLOGY (NEIGHBORS)
        debug: var=desired[inventory_hostname] verbosity=1

      - name: ASSERT THE QUANTITY OF NEIGHBORS ON EACH INTERFACE IS CORRECT AND THE NEIGHBORS FOUNDON EACH IS CORRECT
        assert:
          that:
            - existing[item.key]|length == item.value.neighbors|length
            - existing[item.key]|map(attribute='remote_sysname')|list|sort == item.value.neighbors|sort
        with_dict: "{{ desired[inventory_hostname] }}"

Executing the Playbook

It's time to run the playbook and verify our topology is as-expected.

$ ansible-playbook -i inventory junos-verify-cabling.yml

PLAY [TOPOLOGY AND CABLE VERIFICATION] *****************************************

TASK [GET REAL-TIME (EXISTING) NEIGHBORS] **************************************
ok: [vmx5]
ok: [vmx3]
ok: [vmx4]
ok: [vmx1]
ok: [vmx2]

TASK [debug] *******************************************************************
skipping: [vmx2]
skipping: [vmx1]
skipping: [vmx4]
skipping: [vmx5]
skipping: [vmx3]

TASK [CONVERT LIST OF DICTS TO DICT (WITH LISTS) AND SET FACT] *****************
ok: [vmx1]
ok: [vmx2]
ok: [vmx4]
ok: [vmx5]
ok: [vmx3]

TASK [REAL-TIME (EXISTING) NEIGHBORS] ******************************************
skipping: [vmx5]
skipping: [vmx2]
skipping: [vmx1]
skipping: [vmx3]
skipping: [vmx4]

TASK [PROPOSED TOPOLOGY (NEIGHBORS)] *******************************************
skipping: [vmx1]
skipping: [vmx5]
skipping: [vmx2]
skipping: [vmx4]
skipping: [vmx3]

TASK [ASSERT THE QUANTITY OF NEIGHBORS ON EACH INTERFACE IS CORRECT AND THE NEIGHBORS FOUNDON EACH IS CORRECT] ***
ok: [vmx1] => (item={'value': {u'neighbors': [u'vmx4']}, 'key': u'ge-0/0/3'})
ok: [vmx1] => (item={'value': {u'neighbors': [u'vmx2']}, 'key': u'ge-0/0/2'})
ok: [vmx1] => (item={'value': {u'neighbors': [u'vmx2', u'vmx3', u'vmx4', u'vmx5']}, 'key': u'fxp0'})
ok: [vmx2] => (item={'value': {u'neighbors': [u'vmx5']}, 'key': u'ge-0/0/3'})
ok: [vmx2] => (item={'value': {u'neighbors': [u'vmx3']}, 'key': u'ge-0/0/1'})
ok: [vmx1] => (item={'value': {u'neighbors': [u'vmx3']}, 'key': u'ge-0/0/0'})
ok: [vmx2] => (item={'value': {u'neighbors': [u'vmx1', u'vmx3', u'vmx4', u'vmx5']}, 'key': u'fxp0'})
ok: [vmx2] => (item={'value': {u'neighbors': [u'vmx1']}, 'key': u'ge-0/0/2'})
ok: [vmx4] => (item={'value': {u'neighbors': [u'vmx1']}, 'key': u'ge-0/0/3'})
ok: [vmx4] => (item={'value': {u'neighbors': [u'vmx5']}, 'key': u'ge-0/0/2'})
ok: [vmx3] => (item={'value': {u'neighbors': [u'vmx2']}, 'key': u'ge-0/0/1'})
ok: [vmx5] => (item={'value': {u'neighbors': [u'vmx2']}, 'key': u'ge-0/0/3'})
ok: [vmx4] => (item={'value': {u'neighbors': [u'vmx1', u'vmx2', u'vmx3', u'vmx5']}, 'key': u'fxp0'})
ok: [vmx3] => (item={'value': {u'neighbors': [u'vmx1', u'vmx2', u'vmx4', u'vmx5']}, 'key': u'fxp0'})
ok: [vmx5] => (item={'value': {u'neighbors': [u'vmx4']}, 'key': u'ge-0/0/2'})
ok: [vmx5] => (item={'value': {u'neighbors': [u'vmx1', u'vmx2', u'vmx3', u'vmx4']}, 'key': u'fxp0'})
ok: [vmx3] => (item={'value': {u'neighbors': [u'vmx1']}, 'key': u'ge-0/0/0'})

PLAY RECAP *********************************************************************
vmx1                       : ok=3    changed=0    unreachable=0    failed=0   
vmx2                       : ok=3    changed=0    unreachable=0    failed=0   
vmx3                       : ok=3    changed=0    unreachable=0    failed=0   
vmx4                       : ok=3    changed=0    unreachable=0    failed=0   
vmx5                       : ok=3    changed=0    unreachable=0    failed=0   

Take note that we are using verbosity=1 in the debug statements so you will only see the debug output if you use the -v flag when running the playbook.

Finally, we will manually SSH to vmx1 and vmx4 and shutdown each of their ge-0/0/3 interface breaking their connectivity to one another.

Let's re-run the playbook to see the output.

$ ansible-playbook -i inventory junos-verify-cabling.yml

PLAY [TOPOLOGY AND CABLE VERIFICATION] *****************************************

TASK [GET REAL-TIME (EXISTING) NEIGHBORS] **************************************
ok: [vmx5]
ok: [vmx1]
ok: [vmx2]
ok: [vmx3]
ok: [vmx4]

TASK [debug] *******************************************************************
skipping: [vmx5]
skipping: [vmx4]
skipping: [vmx1]
skipping: [vmx3]
skipping: [vmx2]

TASK [CONVERT LIST OF DICTS TO DICT (WITH LISTS) AND SET FACT] *****************
ok: [vmx1]
ok: [vmx2]
ok: [vmx3]
ok: [vmx4]
ok: [vmx5]

TASK [REAL-TIME (EXISTING) NEIGHBORS] ******************************************
skipping: [vmx1]
skipping: [vmx2]
skipping: [vmx3]
skipping: [vmx4]
skipping: [vmx5]

TASK [PROPOSED TOPOLOGY (NEIGHBORS)] *******************************************
skipping: [vmx1]
skipping: [vmx4]
skipping: [vmx3]
skipping: [vmx5]
skipping: [vmx2]

TASK [ASSERT THE QUANTITY OF NEIGHBORS ON EACH INTERFACE IS CORRECT AND THE NEIGHBORS FOUNDON EACH IS CORRECT] ***
ok: [vmx5] => (item={'value': {u'neighbors': [u'vmx2']}, 'key': u'ge-0/0/3'})
ok: [vmx5] => (item={'value': {u'neighbors': [u'vmx4']}, 'key': u'ge-0/0/2'})
ok: [vmx5] => (item={'value': {u'neighbors': [u'vmx1', u'vmx2', u'vmx3', u'vmx4']}, 'key': u'fxp0'})
fatal: [vmx1]: FAILED! => {"failed": true, "msg": "The conditional check 'existing[item.key]|length == item.value.neighbors|length' failed. The error was: error while evaluating conditional (existing[item.key]|length == item.value.neighbors|length): 'dict object' has no attribute u'ge-0/0/3'"}
fatal: [vmx4]: FAILED! => {"failed": true, "msg": "The conditional check 'existing[item.key]|length == item.value.neighbors|length' failed. The error was: error while evaluating conditional (existing[item.key]|length == item.value.neighbors|length): 'dict object' has no attribute u'ge-0/0/3'"}
ok: [vmx2] => (item={'value': {u'neighbors': [u'vmx5']}, 'key': u'ge-0/0/3'})
ok: [vmx2] => (item={'value': {u'neighbors': [u'vmx3']}, 'key': u'ge-0/0/1'})
ok: [vmx3] => (item={'value': {u'neighbors': [u'vmx2']}, 'key': u'ge-0/0/1'})
ok: [vmx3] => (item={'value': {u'neighbors': [u'vmx1', u'vmx2', u'vmx4', u'vmx5']}, 'key': u'fxp0'})
ok: [vmx2] => (item={'value': {u'neighbors': [u'vmx1', u'vmx3', u'vmx4', u'vmx5']}, 'key': u'fxp0'})
ok: [vmx3] => (item={'value': {u'neighbors': [u'vmx1']}, 'key': u'ge-0/0/0'})
ok: [vmx2] => (item={'value': {u'neighbors': [u'vmx1']}, 'key': u'ge-0/0/2'})

NO MORE HOSTS LEFT *************************************************************

PLAY RECAP *********************************************************************
vmx1                       : ok=2    changed=0    unreachable=0    failed=1   
vmx2                       : ok=3    changed=0    unreachable=0    failed=0   
vmx3                       : ok=3    changed=0    unreachable=0    failed=0   
vmx4                       : ok=2    changed=0    unreachable=0    failed=1   
vmx5                       : ok=3    changed=0    unreachable=0    failed=0 

You can now see how each assertion failed as expected.

Summary

It should be evident that you can do much more than configuration management using a platform like Ansible. With an understanding of what data comes back from every module, you will be able to unit test the network and assert certain conditions exist across the network.

Requirements