Blog Detail
The Parsing Strategies blog post series continues with the introduction of TTP Parser. TTP is a relatively new Python library that has gained some adoption within the Network Automation community. It provides a simple way to parse text into structured data with an approach similar to TextFSM but, in my opinion, has a lot more to offer such as output modifiers, macros, built-in functions, results formatting, and many other features that will be discussed throughout the blog. Join the TTP (#ttp-template-text-parser) Slack channel in our Networktocode Slack if you are interested in joining the conversation.
This blog post will provide an overview of TTP. If you are looking for some basic instructions on how to get started using parsers with Ansible (NTC-Templates, TTP, etc.), start here: Ansible Parsers
What is TTP?
TTP is a Python library which uses implied RegEx patterns to parse data, but it is also incredibly flexible. This parsing strategy provides a way to process data at runtime, as opposed to post-processing by using built-in functions, custom macros, and output formatters while parsing. We will dive deeper into this in a later section. The use of macros (Python functions) to manipulate data and generate desired output is one of my favorite features of the library, thus being my go to parsing strategy of choice when it comes to any hierarchical configuration output. TTP provides a series of output formatters to transform data into YAML, JSON, Table, CSV, and more. TTP can be used as a CLI utility, Python library or available with Netmiko and Ansible.
Groups
Capture groups are declared by using XML group tags, which allows nesting of other groups to generate a hierarchy. Any match inside a group is appended to a list of results. Groups have several attributes that can be set, but only the ‘name’ attribute is required. An important attribute to highlight is the “method” value, which can be set to “group” or “table.” When parsing CLI output, it’s recommended to set the method to table. This tells the parser to consider every line as the start of capturing for the group. Otherwise, setting the “start” indicator per match will be required if you have a variation of regular expressions to capture in the group. Although groups use XML group tags, TTP Templates have a deeper resemblance to Jinja templates and share similar characteristics.
Group Example:
<group name="some_group" method="table">
data to parse
<group name="nested">
more data to parse
</group>
</group>
RegEx Indicator Patterns
TTP offers the ability to specify RegEx patterns to capture within a match variable. If we take a look at the source code, we can review the exact RegEx patterns that are being applied to capture a data. It’s important to understand what the regular expression pattern is before applying it, to ensure you properly capture variables.
PHRASE = r"(\S+ {1})+?\S+"
ROW = r"(\S+ +)+?\S+"
ORPHRASE = r"\S+|(\S+ {1})+?\S+"
DIGIT = r"\d+"
IP = r"(?:[0-9]{1,3}\.){3}[0-9]{1,3}"
PREFIX = r"(?:[0-9]{1,3}\.){3}[0-9]{1,3}/[0-9]{1,2}"
IPV6 = r"(?:[a-fA-F0-9]{1,4}:|:){1,7}(?:[a-fA-F0-9]{1,4}|:?)"
PREFIXV6 = r"(?:[a-fA-F0-9]{1,4}:|:){1,7}(?:[a-fA-F0-9]{1,4}|:?)/[0-9]{1,3}"
_line_ = r".+"
WORD = r"\S+"
MAC = r"(?:[0-9a-fA-F]{2}(:|\.|\-)){5}([0-9a-fA-F]{2})|(?:[0-9a-fA-F]{4}(:|\.|\-)){2}([0-9a-fA-F]{4})"
Macros
As the documentation states, “Macros are python code within a macro tag. This code can contain a number of function definitions, these functions can be referenced within TTP templates.” This allows us to process data during parsing by sending the captured output from our template into a function and return a processed result. This helps eliminate the need for post-processing of the values after the data has been parsed. TTP offers the ability to use these macro functions within match variables, groups, output, and input data. Below, we will review a couple examples of using macros on matched variables.
Macro Example:
<macro>
def subscription_level(data):
data = data.replace('"','').split()
if len(data) >= 3:
return {"card-type": data[0], "subscription-level": data[2]}
return data[0]
</macro>
Structuring Output (Data Modeling)
Another awesome feature of TTP is the ability to manipulate how the data is structured and represented back to us. I won’t go into all the details and capabilities, but you can see from our example below that we have a list of neighbors under our BGP results for the IPV4 Address Family. However, under our peering neighbors, we have generated a dictionary with the key of the neighbor that was found in the subsequent result. Can you find the differences in the way the parser was structured to accomplish the different data structures? There is an ‘Awesome’ hint under the Dynamic Path example!
Simple List:
<group name="neighbor">
neighbor {{ neighbor }} {{ activate | macro("to_bool") }}
neighbor 10.1.0.1 send-community {{ send-community }}
neighbor 10.1.0.1 route-map {{ route-map }} {{ route-map-direction }}
</group>
Result Snippet:
"neighbor": [
{
"activate": true,
"neighbor": "10.1.0.1",
"route-map": "PL-EBGP-PE1-OUT",
"route-map-direction": "out",
"send-community": "both"
},
{
"activate": true,
"neighbor": "10.1.0.5",
"route-map": "PL-EBGP-PE2-OUT",
"route-map-direction": "out",
"send-community": "both"
}
Dynamic Path:
<group name="neighbor.{{ neighbor }}"> <------ Awesome!
neighbor {{ neighbor }} remote-as {{ remote-as }}
neighbor 10.1.0.1 update-source {{ update-source }}
</group>
Result Snippet:
"neighbor": {
"10.1.0.1": {
"remote-as": "65000",
"update-source": "GigabitEthernet2.1001"
},
"10.1.0.5": {
"remote-as": "65000",
"update-source": "GigabitEthernet3.1001"
}
There are several other techniques to format structure, and I highly encourage you to review the documentation to get the most out of the Forming Results Structure feature.
Parsing Hierarchical Configuration
Let’s get right to it! I know you can’t wait for the good stuff. Here is an example of using groups, macros, specified RegEx indicators ( ORPHRASE & DIGIT ), path formatters, and match variable indicators (‘start’, ‘end’, ‘ignore’) to parse a Nokia 7750 card configuration.
#--------------------------------------------------
echo "Card Configuration"
#--------------------------------------------------
card 1
card-type "iom-1" level cr
fail-on-error
mda 1
mda-type "me6-100gb-qsfp28"
ingress-xpl
window 10
exit
egress-xpl
window 10
exit
fail-on-error
no shutdown
exit
no shutdown
exit
TTP Template:
<macro>
def subscription_level(data):
data = data.replace('"','').split()
return {"card-type": data[0], "subscription-level": data[2]}
</macro>
#-------------------------------------------------- {{ ignore }}
echo "Card Configuration" {{ _start_ }}
#-------------------------------------------------- {{ ignore }}
<group name="configure.card">
card {{ slot-number | DIGIT }}
card-type {{ card-type | ORPHRASE | macro('subscription_level') }}
fail-on-error {{fail-on-error | set(true) }}
<group name="mda">
mda {{ mda-slot }}
shutdown {{ admin-state | set(false) }}
mda-type {{ mda-type | replace('"', '') }}
<group name="ingress-xpl">
ingress-xpl {{ _start_ }}
window {{ window }}
exit {{ _end_ }}
</group>
<group name="egress-xpl">
egress-xpl {{ _start_ }}
window {{ window }}
exit {{ _end_ }}
</group>
fail-on-error {{ fail-on-error | set(true) }}
no shutdown {{ admin-state | set(true) }}
</group>
exit {{ ignore }}
</group>
#-------------------------------------------------- {{ _end_ }}
Result:
[
{
"configure":{
"card":{
"card-type":{
"card-type":"iom-1",
"subscription-level":"cr"
},
"fail-on-error":true,
"mda":{
"admin-state":true,
"egress-xpl":{
"window":"10"
},
"fail-on-error":true,
"ingress-xpl":{
"window":"10"
},
"mda-slot":"1",
"mda-type":"me6-100gb-qsfp28"
},
"slot-number":"1"
}
}
}
]
Awesome! But what exactly is happening with our “subscription_level” macro? Our template includes the following match variable:
"card-type {{ card-type | ORPHRASE | macro('subscription_level') }}".
This is using “ORPHRASE” to capture a single word or a phrase and the matched text (‘“iom-1” level cr”’) is then sent into the “subscription_level” function for processing. The function is taking this captured string and manipulating the text by splitting it to produce the following list: “[‘iom-1’, ‘level’, ‘cr’]”. Finally, it’s returning a dictionary with the keys for ‘card-type’ and ‘subscription-level’.
{
"card-type":{
"card-type":"iom-1",
"subscription-level":"cr"
}
}
Let’s take a look at another example template to parse a Cisco IOS BGP configuration that’s also taking advantage of TTP Built-In Functions. The goal is to convert values that would be better represented as booleans in our data model, specifically the following lines: “log-neighbor-changes” and “activate”. Although we can use the ‘set’ function like our previous example to accomplish something similar, I really want to drive home the fact that we can use python functions to accomplish our desired state of a matched variable. Also, macros have unique behavior when returning data, which we will review in more detail. We will also use ‘is_ip’ to validate our neighbor address is in fact an IP Address and DIGIT as a RegEx indicator to match a number.
Here is the raw output of the running configuration for BGP:
router bgp 65001
bgp router-id 192.168.10.1
bgp log-neighbor-changes
neighbor 10.1.0.1 remote-as 65000
neighbor 10.1.0.1 update-source GigabitEthernet2.1001
neighbor 10.1.0.5 remote-as 65000
neighbor 10.1.0.5 update-source GigabitEthernet3.1001
!
address-family ipv4
redistribute connected
neighbor 10.1.0.1 activate
neighbor 10.1.0.1 send-community both
neighbor 10.1.0.1 route-map PL-EBGP-PE1-OUT out
neighbor 10.1.0.5 activate
neighbor 10.1.0.5 send-community both
neighbor 10.1.0.5 route-map PL-EBGP-PE2-OUT out
exit-address-family
Template:
<macro>
def to_bool(captured_data):
represent_as_bools = ["activate", "log-neighbor-changes"]
if captured_data in represent_as_bools:
return captured_data, {captured_data: True}
</macro>
<group name="bgp">
router bgp {{ asn | DIGIT }}
bgp router-id {{ router-id }}
bgp {{ log-neighbor-changes | macro("to_bool") }}
<group name="neighbor.{{ neighbor }}">
neighbor {{ neighbor | is_ip }} remote-as {{ remote-as }}
neighbor 10.1.0.1 update-source {{ update-source }}
</group>
! {{ ignore }}
<group name="afi.{{ afi }}">
address-family {{ afi }}
redistribute {{ redistribute }}
<group name="neighbor">
neighbor {{ neighbor | is_ip }} {{ activate | macro("to_bool") }}
neighbor 10.1.0.1 send-community {{ send-community }}
neighbor 10.1.0.1 route-map {{ route-map }} {{ route-map-direction }}
</group>
exit-address-family {{ ignore }}
</group>
</group>
The output of our Ansible task:
TASK [DEBUG] *******************************************************************
ok: [AS65001_CE1] => {
"msg": [
[
{
"bgp": {
"afi": {
"ipv4": {
"neighbor": [
{
"activate": true,
"neighbor": "10.1.0.1",
"route-map": "PL-EBGP-PE1-OUT",
"route-map-direction": "out",
"send-community": "both"
},
{
"activate": true,
"neighbor": "10.1.0.5",
"route-map": "PL-EBGP-PE2-OUT",
"route-map-direction": "out",
"send-community": "both"
}
],
"redistribute": "connected"
}
},
"asn": "65001",
"log-neighbor-changes": true,
"neighbor": {
"10.1.0.1": {
"remote-as": "65000",
"update-source": "GigabitEthernet2.1001"
},
"10.1.0.5": {
"remote-as": "65000",
"update-source": "GigabitEthernet3.1001"
}
},
"router-id": "192.168.10.1"
}
}
]
]
}
Well, that was easy. All we had to do was replace the values that are of interest with jinja-like syntax and define several groups with XML group tags to properly structure our results. The macro function “to_bool” was used to process the captured data and return a boolean. You may have noticed that we returned the captured_data and a dictionary in our macro, as opposed to our earlier example only returning a simple dictionary. This is because macros will behave differently according to the data that’s being returned. Here is an explanation from the Documentation:
“If macro returns True or False – original data unchanged, macro handled as condition functions, invalidating result on False and keeps processing result on True If macro returns None – data processing continues, no additional logic associated If macro returns single item – that item replaces original data supplied to macro and processed further If macro return tuple of two elements – fist element must be string – match result, second – dictionary of additional fields to add to results”
Parsing Show Commands
Let’s continue the pattern of the series and parse the output of a simple “show lldp neighbors” command output for IOS.
Raw Output:
Capability codes:
(R) Router, (B) Bridge, (T) Telephone, (C) DOCSIS Cable Device
(W) WLAN Access Point, (P) Repeater, (S) Station, (O) Other
Device ID Local Intf Hold-time Capability Port ID
R3.admin-save.com Gi0/1 120 R Gi0/0
R2.admin-save.com Gi0/0 120 R Gi0/0
Total entries displayed: 2
Now, let’s see how simple the TTP Template is to parse operational show output commands:
<group name="LLDP_NEIGHBORS" method="table">
Device ID Local Intf Hold-time Capability Port ID {{ignore}}
{{DEVICE_ID}} {{LOCAL_INT}} {{HOLD_TIME | DIGIT}} {{CAPABILITY}} {{ PORT_ID }}
</group>
<group name="TOTAL_ENTRIES">
Total entries displayed: {{ COUNT | DIGIT}}
</group>
That’s it! Let’s review some important pieces to make this a successful template.
- Method
- “method=’table’” is applied to the group “LLDP_NEIGHBORS” as we are parsing operational show commands in table format.
- Ignore
- ”” is used to tell the parser to discard the lines that we don’t care about inside of our capture group. Any lines outside of the group are simply ignored and discarded by default.
Example Playbook:
---
- name: "EXAMPLE TTP PLAYBOOK"
hosts: R1
connection: network_cli
tasks:
- name: "10. PARSE LLDP NEIGHBORS WITH TTP"
ansible.netcommon.cli_parse:
command: "show lldp neighbors"
parser:
name: ansible.netcommon.ttp
set_fact: lldp
- name: DEBUG
debug:
msg: "{{ lldp }}"
The above playbook is referencing the template at the following relative location: “templates/ios_show_lldp_neighbors.ttp”. The templates directory contains the template starting with the ansible_network_os followed by the command.
Parsed Output:
ok: [R1] => {
"msg": [
[
{
"LLDP_NEIGHBORS": [
{
"CAPABILITY": "R",
"DEVICE_ID": "R3.admin-save.com",
"HOLD_TIME": "120",
"LOCAL_INT": "Gi0/1",
"PORT_ID": "Gi0/0"
},
{
"CAPABILITY": "R",
"DEVICE_ID": "R2.admin-save.com",
"HOLD_TIME": "120",
"LOCAL_INT": "Gi0/0",
"PORT_ID": "Gi0/0"
}
],
"TOTAL_ENTRIES": {
"COUNT": "2"
}
}
]
Finally, one thing to keep in mind is the several nested lists that were produced. It’s as simple as ensuring you are accessing the correct list when evaluating the results.
Example Ansible debug task:
- name: DEBUG
debug:
msg: "{{ lldp[0][0]['TOTAL_ENTRIES'] }}"
Output:
TASK [DEBUG] *******************************************************************
ok: [R1] => {
"msg": {
"COUNT": "2"
}
}
Conclusion
Although we have barely scratched the surface, you can see that TTP offers many great features. I find it to be very accommodating when parsing full hierarchical running configuration outputs, more so than other available parsers. The library is constantly evolving and implementing new features. Take a second to join the Slack channel to keep up with development and ask any questions you may have!
-Hugo
Does this all sound amazing? Want to know more about how Network to Code can help you do this, reach out to our sales team. If you want to help make this a reality for our clients, check out our careers page.
Share :
Contact Us to Learn More
Share details about yourself & someone from our team will reach out to you ASAP!
