Thank you for joining me for Part 3 of the parsing strategies blog series. This post will dive deeper into using Cisco’s PyATS Genie library for parsing. For further reference in this blog post, I’ll be referring to the Genie library just by Genie. Genie also uses Regular Expressions (RegEx) under the hood to parse the output received from a device whether it’s semi-structured data, XML, YANG, etc. This is a key difference from other parsers, but for now, let’s stick with parsing semi-structured data.
Let’s move on and dive deeper into what the show lldp neighbors parser looks like, how it works, and how we need to modify our existing playbook to use the Genie parsers.
Before we get too deep into how Genie works, you can see all the available parsers. The number of parsers has increased dramatically over the last several months and is starting to include more vendors, which is great to see.
Genie uses Python classes to build two important parsing functions:
One key difference between what we’ve covered so far and Genie parsers is the ability to connect to devices and grab the necessary output, which is the default behavior, but also allows users to provide the output instead, thus working like most parsers that are separate from the device interaction.
Another key difference is the ability to use other parsing strategies within Genie such as TextFSM or Template Text Parser (TTP), but for the sake of this post, we will be covering RegEx.
You can find more detailed information on how the Genie parsers work at their developer guide.
Let’s dive into our particular parser.
"""show_lldp.py
supported commands:
* show lldp
* show lldp entry *
* show lldp entry [<WORD>]
* show lldp interface [<WORD>]
* show lldp neighbors
* show lldp neighbors detail
* show lldp traffic
"""
import re
from genie.metaparser import MetaParser
from genie.metaparser.util.schemaengine import Schema, \
Any, \
Optional, \
Or, \
And, \
Default, \
Use
# import parser utils
from genie.libs.parser.utils.common import Common
We can see this parser is declared in the show_lldp.py module and supports several variations of the show lldp commands. It then imports re, which is the built in RegEx library. The next import is the MetaParser that makes sure that the parsers’ output adheres to the defined schema. After MetaParser is imported, the schema related imports take place that helps build the actual schema we’ll see shortly. After that, the Common class that provides helper functions is imported.
The schema class provides us insight into what the output will look like. Let’s take a look at the initial definition.
class ShowLldpNeighborsSchema(MetaParser):
"""
Schema for show lldp neighbors
"""
schema = {
'total_entries': int,
'interfaces': {
Any(): {
'port_id': {
Any(): {
'neighbors': {
Any(): {
'hold_time': int,
Optional('capabilities'): list,
}
}
}
}
}
}
}
We can see that ShowLldpNeighborsSchema will be a subclass of the MetaParser class imported at the beginning of the file. Within the ShowLldpNeighborsSchema class, we define our schema attribute used to make sure our output adheres to the schema before returning it to the user.
The schema is a dictionary and is expecting a total_entries key with an integer value and an interfaces key, used to define a dictionary. Each interface will be a key within the interfaces dictionary and the data obtained from the output is defined in several other nested dictionaries. Each key value pair specifies the key and the type of value it must be. There are also Optional keys not required to pass schema validation.
Now that we see the schema, we can mock up what our potential output would be.
{
"total_entries": 1,
"interfaces": {
"GigabitEthernet1": {
"port_id": {
"Gi1": {
"neighbors": {
"iosv-0": {
"capabilities": [
"R"
],
"hold_time": 120
}
}
}
}
}
}
}
Let’s move onto the defined parser class.
class ShowLldpNeighbors(ShowLldpNeighborsSchema):
"""
Parser for show lldp neighbors
"""
CAPABILITY_CODES = {'R': 'router',
'B': 'mac_bridge',
'T': 'telephone',
'C': 'docsis_cable_device',
'W': 'wlan_access_point',
'P': 'repeater',
'S': 'station_only',
'O': 'other'}
cli_command = ['show lldp neighbors']
We can see the ShowLldpNeighbors class is inheriting the ShowLldpNeighborsSchema class we just covered. Now there is a mapping for the short form capabilities codes returned within the output when neighbors exist, and the long form the parser wants to return to the user.
The next defined variable is cli_command. It specifies the commands executed by the parser if no output is provided.
Let’s take look at the cli method to see what will be executed when a user specifies the cli parser for show lldp neighbors command.
Each type of output will be specified as a method under the parser class. For example, if the device returns xml, there will be an
xmlmethod that will parse and return structured data that adheres to the same schema as theclioutput.
Let’s explore the code in bite size chunks to show what’s happening.
def cli(self, output=None):
if output is None:
cmd = self.cli_command[0]
out = self.device.execute(cmd)
else:
out = output
parsed_output = {}
We can see the cli method takes an optional argument named output, but defaults to None. The first logic determines whether the user has provided the output or whether the parser needs to execute the command against the device. The connection the parser uses is provided by the PyATS library. This means no other library is required such as netmiko or napalm to connect to the devices.
# Total entries displayed: 4
p1 = re.compile(r'^Total\s+entries\s+displayed:\s+(?P<entry>\d+)$')
# Device ID Local Intf Hold-time Capability Port ID
# router Gi1/0/52 117 R Gi0/0/0
# 10.10.191.107 Gi1/0/14 155 B,T 7038.eeff.572d
# d89e.f3ff.58fe Gi1/0/33 3070 d89e.f3ff.58fe
p2 = re.compile(r'(?P<device_id>\S+)\s+(?P<interfaces>\S+)'
r'\s+(?P<hold_time>\d+)\s+(?P<capabilities>[A-Z,]+)?'
r'\s+(?P<port_id>\S+)')
After the parsed_output variable is instantiated the next step is to define the RegEx expressions used to find the valuable data within the device output. Since the output is tabulated, which means it’s defined as a table, all values we care about will be on the same line (row) for each neighbor.
The parser uses
re.compileto specify the RegEx expression ahead of time for use later in the code. Typically,re.compileis used when the same expression is used multiple times.
p1 will provide the total_entries within our schema, by using the Named Capturing Groups ability within the re library. Luckily, Cisco provide great documentation within the code to tell you what each RegEx is expecting to capture. p2 defines the RegEx used to capture the neighbor related information. We can see it uses mostly \S+ which captures any non-whitespace since the output is straight forward. But we can see the capabilities named capture group is a bit more complicated. It’s expecting at least one or more capital letter or comma, and then zero or one of that RegEx expression. This may be better explained by their example if we look at the capabilities column, it shows that it can capture a single capability, two capabilities with a comma, or zero capabilities.
Now let’s look at the remaining code to see how it uses these RegEx expressions.
for line in out.splitlines():
line = line.strip()
# Total entries displayed: 4
m = p1.match(line)
if m:
parsed_output['total_entries'] = int(m.groupdict()['entry'])
continue
# Device ID Local Intf Hold-time Capability Port ID
# router Gi1/0/52 117 R Gi0/0/0
# 10.10.191.107 Gi1/0/14 155 B,T 7038.eeff.572d
# d89e.f3ff.58fe Gi1/0/33 3070 d89e.f3ff.58fe
m = p2.match(line)
if m:
group = m.groupdict()
intf = Common.convert_intf_name(group['interfaces'])
device_dict = parsed_output.setdefault('interfaces', {}). \
setdefault(intf, {}). \
setdefault('port_id', {}). \
setdefault(group['port_id'], {}).\
setdefault('neighbors', {}). \
setdefault(group['device_id'], {})
device_dict['hold_time'] = int(group['hold_time'])
if group['capabilities']:
capabilities = list(map(lambda x: x.strip(), group['capabilities'].split(',')))
device_dict['capabilities'] = capabilities
continue
return parsed_output
We can see the parser performs a for loop through the output using the splitlines method to provide a list of each line within the output. It will strip any whitespace on either side of the string.
The parser will attempt to match the p1 compiled RegEx and if it captures it, will then add total_entries to the parsed_output dictionary and then continue to the next line in the output.
If the parser didn’t capture anything for p1, it will then attempt to match p2. If a match occurs, it will then the groupdict() method to return all the named capture groups and their values as a dictionary.
We can now see the parser uses the convert_intf_name method from the Common class imported at the top of the file.
You can review the code here.
Once the interface name is converted, the parser adds the interfaces dictionary to the parsed_output variable by extracting the information captured or defaulting to an empty dictionary for any non-captured data and then assigning it to the device_dict variable.
After the device_dict is specified, it adds the hold time to it.
The next step is to strip and split the capabilities into a list and add to the device_dict variable. The parser will then continue to the next line of the output.
Once all lines are parsed, it will return the parsed_output to the user as long as it passes schema validation.
I believe this can be easier to understand than something like TextFSM since it’s written in Python and Python is a popular language among network automation engineers.
Let’s move on and review the topology again.
Below is a picture of the lab topology we’re using to validate LLDP neighbors. It’s a simple topology with three Cisco IOS routers connected together and have LLDP enabled.
We’ve already covered most of the Ansible setup in part 2, but we’ll explain the small changes we have to make to use the Genie parsers within Ansible.
Here is a look at a host var we’ve defined as a refresher since there are no changes here.
---
approved_neighbors:
- local_intf: "Gi0/0"
neighbor: "iosv-1"
- local_intf: "Gi0/1"
neighbor: "iosv-2"
Now let’s take a look at the changes in pb.validate.neighbors.yml.
---
- hosts: "ios"
connection: "ansible.netcommon.network_cli"
gather_facts: "no"
tasks:
- name: "PARSE LLDP INFO INTO STRUCTURED DATA"
ansible.netcommon.cli_parse:
command: "show lldp neighbors"
parser:
name: ansible.netcommon.pyats
set_fact: "lldp_neighbors"
- name: "MANIPULATE THE DATA TO BE IN THE SAME FORMAT AS TEXTFSM TO PREVENT CHANGING FINAL ASSERTION TASK"
set_fact:
lldp_neighbors: "{{ lldp_neighbors | convert_data }}"
- name: "ASSERT THE CORRECT NEIGHBORS ARE SEEN"
assert:
that:
- "lldp_neighbors | selectattr('local_interface', 'equalto', item['local_intf']) | map(attribute='neighbor') | first == item['neighbor']"
loop: "{{ approved_neighbors }}"
Using the
ansible.netcommon.pyatsparser requiresgenieandpyatsto be installed viapip install genie pyats.
There are a few things to dissect with the playbook. First, we changed the parser to ansible.netcommon.pyats. Second, we added another task to manipulate the data we get back from the parser, into a similar format as the second blog post in this series so we don’t have to change the last task. I did the conversion within a custom filter plugin due to the structure of the data and the ease of handling this within Python. You will see the output below once we run our playbook.
Let’s go ahead and run the playbook and see what output we get.
❯ ansible-playbook pb.validate.neighbors.yml -k -vv
ansible-playbook 2.10.3
config file = /Users/myohman/Documents/local-dev/blog-posts/ansible.cfg
configured module search path = ['/Users/myohman/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
ansible python module location = /Users/myohman/.virtualenvs/3.8/main/lib/python3.8/site-packages/ansible
executable location = /Users/myohman/.virtualenvs/3.8/main/bin/ansible-playbook
python version = 3.8.6 (default, Nov 17 2020, 18:43:06) [Clang 12.0.0 (clang-1200.0.32.27)]
Using /Users/myohman/Documents/local-dev/blog-posts/ansible.cfg as config file
SSH password:
redirecting (type: callback) ansible.builtin.yaml to community.general.yaml
redirecting (type: callback) ansible.builtin.yaml to community.general.yaml
Skipping callback 'default', as we already have a stdout callback.
Skipping callback 'minimal', as we already have a stdout callback.
Skipping callback 'oneline', as we already have a stdout callback.
PLAYBOOK: pb.validate.neighbors.yml ***********************************************************************
1 plays in pb.validate.neighbors.yml
PLAY [ios] ************************************************************************************************
META: ran handlers
TASK [Parse LLDP info into structured data] ***************************************************************
task path: /Users/myohman/Documents/local-dev/blog-posts/pb.validate.neighbors.yml:9
ok: [iosv-2] => changed=false
ansible_facts:
lldp_neighbors:
interfaces:
GigabitEthernet0/0:
port_id:
Gi0/1:
neighbors:
iosv-0:
capabilities:
- R
hold_time: 120
GigabitEthernet0/1:
port_id:
Gi0/1:
neighbors:
iosv-1:
capabilities:
- R
hold_time: 120
total_entries: 2
parsed:
interfaces:
GigabitEthernet0/0:
port_id:
Gi0/1:
neighbors:
iosv-0:
capabilities:
- R
hold_time: 120
GigabitEthernet0/1:
port_id:
Gi0/1:
neighbors:
iosv-1:
capabilities:
- R
hold_time: 120
total_entries: 2
stdout: |-
Capability codes:
(R) Router, (B) Bridge, (T) Telephone, (C) DOCSIS Cable Device
(W) WLAN Access Point, (P) Repeater, (S) Station, (O) Other
Device ID Local Intf Hold-time Capability Port ID
iosv-1 Gi0/1 120 R Gi0/1
iosv-0 Gi0/0 120 R Gi0/1
Total entries displayed: 2
stdout_lines: <omitted>
ok: [iosv-0] => changed=false
ansible_facts:
lldp_neighbors:
interfaces:
GigabitEthernet0/0:
port_id:
Gi0/0:
neighbors:
iosv-1:
capabilities:
- R
hold_time: 120
GigabitEthernet0/1:
port_id:
Gi0/0:
neighbors:
iosv-2:
capabilities:
- R
hold_time: 120
total_entries: 2
parsed:
interfaces:
GigabitEthernet0/0:
port_id:
Gi0/0:
neighbors:
iosv-1:
capabilities:
- R
hold_time: 120
GigabitEthernet0/1:
port_id:
Gi0/0:
neighbors:
iosv-2:
capabilities:
- R
hold_time: 120
total_entries: 2
stdout: |-
Capability codes:
(R) Router, (B) Bridge, (T) Telephone, (C) DOCSIS Cable Device
(W) WLAN Access Point, (P) Repeater, (S) Station, (O) Other
Device ID Local Intf Hold-time Capability Port ID
iosv-2 Gi0/1 120 R Gi0/0
iosv-1 Gi0/0 120 R Gi0/0
Total entries displayed: 2
stdout_lines: <omitted>
ok: [iosv-1] => changed=false
ansible_facts:
lldp_neighbors:
interfaces:
GigabitEthernet0/0:
port_id:
Gi0/0:
neighbors:
iosv-0:
capabilities:
- R
hold_time: 120
GigabitEthernet0/1:
port_id:
Gi0/1:
neighbors:
iosv-2:
capabilities:
- R
hold_time: 120
total_entries: 2
parsed:
interfaces:
GigabitEthernet0/0:
port_id:
Gi0/0:
neighbors:
iosv-0:
capabilities:
- R
hold_time: 120
GigabitEthernet0/1:
port_id:
Gi0/1:
neighbors:
iosv-2:
capabilities:
- R
hold_time: 120
total_entries: 2
stdout: |-
Capability codes:
(R) Router, (B) Bridge, (T) Telephone, (C) DOCSIS Cable Device
(W) WLAN Access Point, (P) Repeater, (S) Station, (O) Other
Device ID Local Intf Hold-time Capability Port ID
iosv-2 Gi0/1 120 R Gi0/1
iosv-0 Gi0/0 120 R Gi0/0
Total entries displayed: 2
stdout_lines: <omitted>
TASK [MANIPULATE THE DATA TO BE IN STANDARD FORMAT] *******************************************************
task path: /Users/myohman/Documents/local-dev/blog-posts/pb.validate.neighbors.yml:16
ok: [iosv-0] => changed=false
ansible_facts:
lldp_neighbors:
- local_interface: Gi0/1
neighbor: iosv-2
- local_interface: Gi0/0
neighbor: iosv-1
ok: [iosv-1] => changed=false
ansible_facts:
lldp_neighbors:
- local_interface: Gi0/1
neighbor: iosv-2
- local_interface: Gi0/0
neighbor: iosv-0
ok: [iosv-2] => changed=false
ansible_facts:
lldp_neighbors:
- local_interface: Gi0/1
neighbor: iosv-1
- local_interface: Gi0/0
neighbor: iosv-0
TASK [Assert the correct neighbors are seen] **************************************************************
task path: /Users/myohman/Documents/local-dev/blog-posts/pb.validate.neighbors.yml:20
ok: [iosv-0] => (item={'local_intf': 'Gi0/0', 'neighbor': 'iosv-1'}) => changed=false
ansible_loop_var: item
item:
local_intf: Gi0/0
neighbor: iosv-1
msg: All assertions passed
ok: [iosv-2] => (item={'local_intf': 'Gi0/0', 'neighbor': 'iosv-0'}) => changed=false
ansible_loop_var: item
item:
local_intf: Gi0/0
neighbor: iosv-0
msg: All assertions passed
ok: [iosv-1] => (item={'local_intf': 'Gi0/0', 'neighbor': 'iosv-0'}) => changed=false
ansible_loop_var: item
item:
local_intf: Gi0/0
neighbor: iosv-0
msg: All assertions passed
ok: [iosv-0] => (item={'local_intf': 'Gi0/1', 'neighbor': 'iosv-2'}) => changed=false
ansible_loop_var: item
item:
local_intf: Gi0/1
neighbor: iosv-2
msg: All assertions passed
ok: [iosv-1] => (item={'local_intf': 'Gi0/1', 'neighbor': 'iosv-2'}) => changed=false
ansible_loop_var: item
item:
local_intf: Gi0/1
neighbor: iosv-2
msg: All assertions passed
ok: [iosv-2] => (item={'local_intf': 'Gi0/1', 'neighbor': 'iosv-1'}) => changed=false
ansible_loop_var: item
item:
local_intf: Gi0/1
neighbor: iosv-1
msg: All assertions passed
META: ran handlers
META: ran handlers
PLAY RECAP ************************************************************************************************
iosv-0 : ok=3 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
iosv-1 : ok=3 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
iosv-2 : ok=3 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
I ran this playbook with some verbosity to show what each task returns and the format of our parsed data.
If we take a closer look at the output of the first task, we can see under the parsed key as well as setting the fact (lldp_neighbors), that we have our structured data from running the raw output through Genie.
The second task shows the loop for each host and the item it’s using during the loop. If you look back at our playbook, we’re using both the local_intf and neighbor for our assertions from our approved_neighbors variable.
We converted the data using a custom filter plugin, but we could have easily adjusted the facts and final assertions to align with the output we receive back from Genie. It’s also valuable to show the possibility of having a single playbook using any parser to run operational assertions. If we were in production, we could make the convert_data custom filter plugin translate several different parser formats into a parser agnostic format.
For brevity, here is our tree output to show you what the folder structure looks like to use a custom filter plugin.
❯ tree
.
├── ansible.cfg
├── filter_plugins
│ └── custom_filters.py
├── group_vars
│ ├── all
│ │ └── all.yml
│ └── ios.yml
├── host_vars
│ ├── iosv-0.yml
│ ├── iosv-1.yml
│ └── iosv-2.yml
├── inventory
└── pb.validate.neighbors.yml
Finally the contents of the custom_filters.py.
# -*- coding: utf-8 -*-
from __future__ import absolute_import, division, print_function
__metaclass__ = type
import re
def convert_genie_data(data):
intfs = []
for k,v in data['interfaces'].items():
intf_name = "Gi" + re.search(r'\d/\d', k).group(0)
intf_dict = {}
intf_dict['local_interface'] = intf_name
neighbor_intf = list(v['port_id'].keys())[0]
intf_dict['neighbor'] = list(v['port_id'][neighbor_intf]['neighbors'].keys())[0]
intfs.append(intf_dict)
return intfs
class FilterModule:
def filters(self):
filters = {
'convert_data': convert_genie_data,
}
return filters
I hope you enjoyed this blog post and understand a little bit more about Genie parsers and how to consume them with Ansible. The next post in this series will go over Ansible Engine parsing.
-Mikhail
Does this all sound amazing? Want to know more about how Network to Code can help you do this, reach out to our sales team. If you want to help make this a reality for our clients, check out our careers page.
Share details about yourself & someone from our team will reach out to you ASAP!
