NTC Templates – Data Model Update

Blog Detail

The NTC Templates library has a major overhaul for the templates going on that is going to create the need for a major release. As part of the planning for the 4.0.0 release of NTC Templates, there is an updated set of data models that are being applied for network device templates. The project is moving to a single common data model, which will allow for the transformation of the data into a single model that is common across the various vendor textFSM template. This is going to make all of your data structures look the same whether you are using Arista, Cisco, Juniper, or other vendor CLI within your Ansible or Python automations. The planned release date is October 10th.

Thank you to Michael Bear who helped significantly spearhead this effort, with the full support from the NTC Team.

NTC Templates History

NTC Templates is one of NTC’s oldest and most popular open source libraries. It is used across many automations to provide a structured data format from CLI unstructured text. It is available and incorporated with many common tools used to automate networks every day, including the capability to use it with AnsibleNetmikoScrapli, and others.

There have been some changes over time, but overall, the library is the go-to place for taking unstructured CLI output and converting it into structured data for use in your Python applications (Ansible as well, which is a Python library).

Data Model

With the 4.0.0 release of NTC-Templates, many of the individual templates are being updated to a common data model for various components. Anywhere there is the use of VLANs, for example, previously you may have seen any of the following textFSM keys:

  • VLAN
  • VLAN_ID
  • VLAN_TAG
  • TAG
  • VLAN_NUMBER
  • VLAN_NAME

Now, each of these will be migrated into a single data model with VLAN_ID being anything that represents a VLAN numerical representation, and VLAN_NAME wherever a string identification of a VLAN is made.

The same can be said for MAC addresses and prefixes. With MAC addresses there were places that would just be MACMAC_ADDRMAC_ADD, and so on. The names of the keys are no longer going to be shortened. In sticking with the Zen of Python, these are going to be much more explicit.

What Do I Need to Do?

At this point, there are two options for being prepared properly. If you are writing your first automation applications, let’s move forward to 4.0.

Stay at 3.x

First, if everything is working well for you and you wish to remain on the 3.x templates, you need to make sure to pin your dependencies for the NTC Templates library to 3.x. If you are using requirements.txt:

ntc-templates>=3.0,<4.0

With poetry:

ntc-templates=^3

Update to 4.0

If you are looking to move forward to the common data model, you will need to update your automation applications and playbooks that leverage the keys. Take a look at the updated templates if you are working on Layer 1 – 3 components, such as VLANs, MAC addresses, and IP addresses. Put the automation through the paces and update as necessary for the keys.

Summary

The updated NTC Templates have had some strong updates over the years. Many contributions from the entire Network Automation Community. The move to this while somewhat painful at this particular time will reap many benefits to the years to come. This now will help enable automation capabilities such as:

  • Pre-post comparison of different device types
  • Comparison between different device types (including those from different vendors)
  • Consistency for new onboarded templates
  • Simplified downstream automations

Conclusion

Thank you, and let us know if you have any questions. In the #networktocode channel on the Network to Code Slack workspace.

-Josh



ntc img
ntc img

Contact Us to Learn More

Share details about yourself & someone from our team will reach out to you ASAP!

Getting Started with Python Network Libraries for Network Engineers – Part 3

Blog Detail

This blog post is the third in a series covering common Python libraries that can be used to interact with network devices. In this post we will cover the Scrapli Python library by Carl Montanari. Per its documentation, Scrapli is the words “scrape” and “cli” (i.e. screen scrape) squished together. Its goal is to provide a thoroughly tested, well typed, well documented, simple API that supports both synchronous and asynchronous interaction with network devices.

Differentiation from Other Libraries

Scrapli is different from other libraries in the following ways:

  1. Scrapli provides multiple forms of transport. It defaults to using a system’s local SSH binary. This is in contrast to Netmiko, which uses Paramiko for transport. The forms of transport it supports can be found here along with a justification for the decision to allow for multiple transports and default to a system’s local SSH binary.
  2. Scrapli supports the same platforms as NAPALM out of the box. This is a subset of the platforms supported by Netmiko. The scrapli community project provides the ability for the scrapli community to contribute to and use drivers beyond those included in the base project. The platforms supported out of the box are:
    • Cisco IOS-XE
    • Cisco NX-OS
    • Juniper JunOS
    • Cisco IOS-XR
    • Arista EOS
  3. If you’re looking to write unit/integration tests for your code, scrapli has an adjacent library called scrapli-replay that can be used to do so.

Installation

You can install Scrapli via pip install scrapli. Or, if you are using Poetry, you can use poetry add scrapli. Other methods for installing scrapli are documented here.

Getting Connected

Note: We will only cover connecting via SSH in this blog post.

To start, you’ll want to import and instantiate one of the scrapli “core” drivers from scrapli.driver.core.

Scrapli requires a hostname (IP or DNS name) and an authentication method (generally username and password) to get connected. In addition to this, you will also want to specify whether or not to use SSH strict host key checking (we pass in False below).

>>> from scrapli.driver.core import IOSXEDriver
>>>
>>> conn = IOSXEDriver(
...     host="192.0.2.3",
...     auth_username="cisco",
...     auth_password="cisco",
...     auth_strict_key=False,
... )
>>> type(conn)
scrapli.driver.core.cisco_iosxe.sync_driver.IOSXEDriver
>>> conn.open()

Note: If host strict key checking is enabled (the default), an SSH session will be permitted only to hosts that have a key defined inside of your system’s “known_hosts” file.

You can also use the Scrapli class to dynamically select and instantiate a driver (much like ConnectHandler in Netmiko) in the following way:

>>> from scrapli import Scrapli
>>>
>>> conn = Scrapli(
...     host="192.0.2.3",
...     auth_username="cisco",
...     auth_password="cisco",
...     auth_strict_key=False,
...     platform="cisco_iosxe",
...  )
>>> type(conn)
scrapli.driver.core.cisco_iosxe.sync_driver.IOSXEDriver
>>> conn.open()

A list of the basic driver arguments can be found here.

Sending Commands

Once you have instantiated your Driver object, you can send single show commands via the .send_command() method. You will use the same command syntax you would type in if you were directly connected to the device via SSH:

>>> response = conn.send_command("show ip interface brief")
>>> print(response.result)
Interface              IP-Address      OK? Method Status                Protocol
FastEthernet0          unassigned      YES NVRAM  down                  down
GigabitEthernet1/0/1   192.0.2.3       YES unset  up                    up
GigabitEthernet1/0/2   unassigned      YES unset  up                    up
GigabitEthernet1/0/3   unassigned      YES unset  up                    up
>>> print(response.failed)
False

Likewise, you can send multiple commands using the .send_commands() method.

>>> response = conn.send_commands(["show ip interface brief", "show running-config | include hostname"])
>>> print(response.result[0])
Interface              IP-Address      OK? Method Status                Protocol
FastEthernet0          unassigned      YES NVRAM  down                  down
GigabitEthernet1/0/1   192.0.2.3       YES unset  up                    up
GigabitEthernet1/0/2   unassigned      YES unset  up                    up
GigabitEthernet1/0/3   unassigned      YES unset  up                    up
>>> print(response.result[1])
hostname cisco

You can also send commands from a file using the .send_commands_from_file() method.

# commands.txt
show ip int br
show run | i hostname
response = conn.send_commands_from_file("commands.txt")
>>> print(response.result[0])
Interface              IP-Address      OK? Method Status                Protocol
FastEthernet0          unassigned      YES NVRAM  down                  down
GigabitEthernet1/0/1   192.0.2.3       YES unset  up                    up
GigabitEthernet1/0/2   unassigned      YES unset  up                    up
GigabitEthernet1/0/3   unassigned      YES unset  up                    up
>>> print(response.result[1])
hostname cisco

If you want to run a command to edit the configuration, you would use the .send_configs() method instead. This method takes care of entering configuration mode for you, and it requires the commands be in a list or set:

>>> response = conn.send_configs(["interface Gi1/0/3", "no description"])
>>> print(response.result)
configure terminal
Enter configuration commands, one per line.  End with CNTL/Z.
cisco(config)#interface Gi1/0/3
cisco(config-if)#no description

Note: The send_configs() method doesn’t exit config mode like Netmiko does. It, instead, relies on the send_command() method to acquire the user exec mode prompt.

As is the case with .send_commands(), you can also send configurations from a file using the send_configs_from_file() method.

# config_changes.txt
interface Gi1/0/3
 no description
>>> response = conn.send_configs_from_file("config_changes.txt")
>>> print(response.result)
configure terminal
Enter configuration commands, one per line.  End with CNTL/Z.
cisco(config)#interface Gi1/0/3
cisco(config-if)#no description

Finally, there is a send_config() method that will parse the configuration provided as input and split it into lines, each line being sent to the device. This is useful for copying full configs over to the device.

Command Output Parsing

We’ve covered parsing strategies here on this blog before. Just as with Netmiko, Scrapli supports TextFSMTTP, and Genie. Let’s take a quick look on how to use them with Scrapli.

TextFSM

Scrapli defines TextFSM as an “extra”. As such, the first step to using TextFSM templates with scrapli is to install the TextFSM library. You can use pip or Poetry to do so.

pip install 'scrapli[textfsm]'
poetry add 'scrapli[textfsm]'

Once TextFSM is installed, you can use the .textfsm_parse_output() method on a Scrapli response object to return the output as parsed by the NTC template. This method uses the platform inferred by the driver to set the textfsm-platform. It combines this with the command sent to the device to guess as to which template it should use to parse the returned data.

>>> response = conn.send_command("show interfaces")
>>> print(response.textfsm_parse_output())
[{'abort': '',
  'address': '381c.1ae6.cd81',
  'bandwidth': '100000 Kbit',
  'bia': '381c.1ae6.cd81',
  'crc': '0',
  'delay': '100 usec',
...

You can see the template used for this command here.

Scrapli also supports defining the template that should be used to parse the response manually. You can import and use the textfsm_parse function from scrapli.helper to do so.

>>> from scrapli.helper import textfsm_parse
>>>
>>> response = conn.send_command("show interfaces")
>>> structured_result = textfsm_parse("/path/to/template", response.result)

TTP

Scrapli also supports the TTP parsing library. As with TextFSM, you will need to install it via pip install ttp or poetry add ttp before it can be used. The TTP installation does not currently include any templates, so you will need to find or create your own and provide the path to those templates.

>>> response = conn.send_command("show interfaces")
>>> structured_result = response.ttp_parse_output(template="show_interfaces_template.ttp")
>>> pprint(structured_result)
[[[{'description': 'CAM1',
    'interface': 'GigabitEthernet1/0/1',
    'link_status': 'up',
    'protocol_status': 'up'},
   {'description': 'CAM2',
    'interface': 'GigabitEthernet1/0/2',
    'link_status': 'up',
    'protocol_status': 'up'},
...

Genie

The last parser that Scrapli currently supports is Genie. As with TextFSM and TTP, scrapli does not install Genie nor its required library, pyATS, by default, so you will need to install them separately via pip install 'pyats[library]' or poetry add 'pyats[library]'. Athough Genie parsers exist for non-Cisco platforms, In Scrapli, Genie parsers can be used only for Cisco devices.

>>> response = conn.send_command("show interfaces")
>>> structured_result = response.genie_parse_output()
>>> pprint(structured_result)
{'GigabitEthernet1/0/1': {'arp_timeout': '04:00:00',
                          'arp_type': 'arpa',
                          'bandwidth': 100000,
                          'connected': True,
                          'counters': {'in_broadcast_pkts': 41240,
...

Note: Genie does not support custom templates.

Next Steps

  • Perhaps you’d like to use asyncio for interactions with network devices, or learn more advanced concepts? The documentation is excellent, and a great place to start.
  • If you’d like more examples of common interactions using scrapli, Carl has created a directory with such examples here.
  • Scrapli has an ecosystem including a few adjacent tools that are worth checking out (scrapli_netconf, scrapli_nornir…etc.).

Conclusion

I hope you enjoy Scrapli. It’s an excellent tool and is my personal go-to tool for new projects in which I need to automate interactions with devices for which it has a driver.

-Phillip

New to Python libraries? NTC’s Training Academy is holding a 3-day course Automating Networks with Python I on September 26-28, 2022 with 50% labs to get you up to speed.
Visit our 2022 public course schedule to see our full list.



ntc img
ntc img

Contact Us to Learn More

Share details about yourself & someone from our team will reach out to you ASAP!

Getting Started with Python Network Libraries for Network Engineers – Part 1

Blog Detail

This blog post will be the first in a series covering common Python libraries that can be used to interact with network devices. In this post we will cover the Netmiko Python library by Kirk Byers. Netmiko is based on the Paramiko Python library, but whereas Paramiko was designed to interact with standard OpenSSH devices (like Linux), Netmiko was designed to interact with network devices. It has a large number of supported platforms included for connecting via SSH, and it can also accommodate limited Telnet or serial connections as well as Secure Copy (SCP) for file transfers.

Installation

You can install Netmiko via pip install netmiko. Or, if you are using Poetry, you can use poetry add netmiko.

Getting Connected

Note: We will only be covering connecting via SSH in this blog post.

Like all SSH connections, Netmiko requires a hostname (IP or DNS name) and an authentication method (generally username and password) to get connected. In addition to this, you will also need to specify the device type you will be connecting to.

>>> from netmiko import ConnectHandler
>>> 
>>> conn = ConnectHandler(
...     host="192.0.2.3",
...     username="cisco",
...     password="cisco",
...     device_type="cisco_ios"
... )

There are two ways of determining the device type: looking it up in a list or having Netmiko try to detect the device type automatically. You can see the list of current device types by digging into the code on GitHub, specifically the CLASS_MAPPER_BASE dictionary in the ssh_dispatcher.py file. If, however, you aren’t exactly sure which device type you need to choose, you can use the SSHDetect class to have Netmiko help:

>>> from netmiko import ConnectHandler, SSHDetect
>>> 
>>> detect = SSHDetect(
...     host="192.0.2.3",
...     username="cisco",
...     password="cisco",
...     device_type="autodetect"  # Note specifically passing 'autodetect' here is required
... )
>>> detect.autodetect()  # This method returns the most likely device type
'cisco_ios'
>>> detect.potential_matches  # You can also see all the potential device types and their corresponding accuracy rating
{'cisco_ios': 99}
>>> conn = ConnectHandler(
...     host="192.0.2.3",
...     username="cisco",
...     password="cisco",
...     device_type=detect.autodetect()
... )

Common Methods

Once you have instantiated your ConnectHandler object, you can send single show commands via the .send_command() method. You will use the same command syntax you would type in if you were directly connected to the device via SSH:

>>> output = conn.send_command("show ip int br")
>>> print(output)
Interface              IP-Address      OK? Method Status                Protocol
FastEthernet0          unassigned      YES NVRAM  down                  down
GigabitEthernet1/0/1   unassigned      YES unset  up                    up
GigabitEthernet1/0/2   unassigned      YES unset  up                    up
GigabitEthernet1/0/3   unassigned      YES unset  up                    up
...

Note: You can send multiple show commands back-to-back with the .send_multiline(["command1", "command2"]).

If you want to run a command to edit the configuration, you would use the .send_config_set() method instead. This method takes care of entering and exiting configuration mode for you, and it requires the commands be in a list or set:

>>> output = conn.send_config_set(("interface Gi1/0/3", "no description"))
>>> print(output)
configure terminal
Enter configuration commands, one per line.  End with CNTL/Z.
cisco(config)#interface Gi1/0/3
cisco(config-if)#no description
cisco(config-if)#end
cisco#

And since we are good network engineers, we know we should always save our configuration after making changes with the .save_config() method:

>>> output = conn.save_config()
>>> print(output)
write mem
Building configuration...
[OK]
cisco#

Command Output Parsing

We’ve covered parsing strategies here on this blog before, including the three currently supported parsers Netmiko supports which are TextFSMTTP, and Genie. Let’s take a quick look on how to use them with Netmiko.

TextFSM

If you are just starting out, the easiest way to get structured output data would be to use the included TextFSM parser. By default, Netmiko includes the TextFSM library for the parsing as well as NTC Templates to use as the default templates. To get structured output, simply add use_textfsm=True to the parameters of the .send_command() method:

>>> output = conn.send_command("show interfaces", use_textfsm=True)
>>> pprint(output)
[{'abort': '',
  'address': '381c.1ae6.cd81',
  'bandwidth': '100000 Kbit',
  'bia': '381c.1ae6.cd81',
  'crc': '0',
  'delay': '100 usec',
...

You can see the template used for this command here.

TTP

Netmiko also supports the TTP parsing library, but you will need to install it via pip install ttp or poetry add ttp first. It also does not currently include any templates, so you will need to find or create your own and then provide the path to those templates when you send your command.

Creating TTP templates yourself is definitely more of a manual process, but it gives you the freedom to pare down to only the information that you need. For example, if you just need the interface name, status, and description you can have a template like so:

{{ interface }} is {{ link_status }}, line protocol is {{ protocol_status }} {{ ignore }}
  Description: {{ description }}

And then you would reference the template path using ttp_template:

>>> output = conn.send_command("show interfaces", use_ttp=True, ttp_template="templates/show_interfaces.ttp")
>>> pprint(output)
[[[{'description': 'CAM1',
    'interface': 'GigabitEthernet1/0/1',
    'link_status': 'up',
    'protocol_status': 'up'},
   {'description': 'CAM2',
    'interface': 'GigabitEthernet1/0/2',
    'link_status': 'up',
    'protocol_status': 'up'},
...

Genie

The last parser that Netmiko currently supports is Genie. Netmiko, however, does not install Genie nor its required library, PyATS, by default, so you will need to install them separately via pip install 'pyats[library]' or poetry add 'pyats[library]'. Once they are installed, it is again very similar to enable the Genie parsing:

>>> output = conn.send_command("show interfaces", use_genie=True)
>>> pprint(output)
{'GigabitEthernet1/0/1': {'arp_timeout': '04:00:00',
                          'arp_type': 'arpa',
                          'bandwidth': 100000,
                          'connected': True,
                          'counters': {'in_broadcast_pkts': 41240,
...

Note: Genie does not support custom templates.


Conclusion

As you can see, it doesn’t take much to get started with Netmiko. If you’d like to learn more advanced interactions with Netmiko, such as transferring files via SCP, connecting via SSH keys, or even handling commands that prompt for additional input, the best place to start would be the Netmiko Examples page in the GitHub repository. Another great resource is the #netmiko channel in the Network to Code Slack.

-Joe

New to Python libraries? NTC’s Training Academy is holding a 3-day course Automating Networks with Python I on September 26-28, 2022 with 50% labs to get you up to speed.
Visit our 2022 public course schedule to see our full list.



ntc img
ntc img

Contact Us to Learn More

Share details about yourself & someone from our team will reach out to you ASAP!