Learning Salt with Ansible References

Blog Detail

Have you seen or taken the 2020 NetDevOps survey? If not, don’t worry, it is held yearly from Sept 30th to Oct 31st, so you will get your chance. To quickly recap, the survey’s intention is to understand which tools are most commonly used by Network Operators and Engineers to automate their day-to-day jobs. It is a very interesting survey, and I encourage you to read it.

If we take a look at this data, particularly the questions about tools used in automation, we will see that Ansible is the most popular one. At the same time, there are other tools that are used, but not so much—like SaltStack (aka Salt). If I would speculate why SaltStack is not so popular among Network Engineers, I would say that’s because Salt is more friendly with managing the servers and not so much the networking devices. So I thought, if I write a blog about Salt with references to Ansible it might encourage engineers to learn this tool a bit more.

NOTE
I will do my best to address situations where features available to Salt do not have direct counterparts in Ansible.

Salt Terminologies

Before you start using Salt, you need to get familiar with its terminologies. The below table summarizes the ones that I will use in this blog:

SaltStack Name	Description	Ansible Name	Description
Master	Server that commands-and-controls the minions.	Server	The machine where Ansible is installed.
Minion	Managed end-hosts where the commands are executed.	Hosts	The devices that are managed by the Ansible server.
Proxy Minion	End-hosts that cannot run the `salt-minion` service and can be controlled using proxy-minion setup. *These hosts are mostly networking or IoT devices.*	N/A	N/A
Pillars	Information (e.g., username, passwords, configuration parameters, variables, etc.) that needs to be defined and associated with one or more minion(s) in order to interact with or distribute to the end-host.	Inventory and Static variables	The Ansible Inventory and variables that need to be associated with the managed host(s).
Grains	Information (e.g., IP address, OS type, memory utilization etc.) that is obtained from the managed minion. Grains also can be statically assigned to the minions.	Fact	Information obtained from the host with the `gather_facts` or similar task operation.
Modules	Commands that are executed locally on the master or on the minion.	Modules	Commands that are executed locally on the server or on the end-host.
SLS Files	(S)a(l)t (S)tate files, also known as formulas, are the representation of the state in which the end-hosts should be. SLS files are mainly written in YAML, but other languages are also supported.	Playbook/Tasks	The file where one or more tasks that need to be executed on the end-host are defined.

Salt and Ansible Playground

Let’s take a look at my playground. These are the hosts and related information involved in this blog:

Salt

Role: Hosts Salt’s master and proxy-minion configuration.
OS: Ubuntu 18.04.3
Salt: Version 3003.1
IP: 172.29.224.100

Ansible

Role: Hosts Ansible execution environment
OS: Ubuntu 18.04.3
Ansible: Version 2.10.7
IP: 172.29.224.99

IOS-XE

Role: End host for manipulation
OS: 16.12.3
IP: 172.29.224.101

NX-OS

Role: End host for manipulation
OS: 7.0.9
IP: 172.29.224.102

Junos

Role: End host for manipulation
OS: 21.2R1.10
IP: 172.29.224.103

Salt and Ansible Installation

The Salt installation can vary based on the OS that you are using, so I would suggest that you read the installation instructions that match your OS. At the end, you should have salt-master and salt-minion installed on your machine.

The Ansible installation is a bit more straightforward; use the pip install ansible==2.10.7 command to install the same version that I use.

Salt Environment Configuration

The time has come to configure the Salt environment. Confirm that salt-master service is active. If not, start it with sudo salt-master -d command.

$ service salt-master status
● salt-master.service - The Salt Master Server
   Loaded: loaded (/lib/systemd/system/salt-master.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2021-07-06 23:29:24 UTC; 7min ago
>>>
Trimmed for brevity
<<<

Using your favorite text editor, open the /etc/salt/master configuration file and append the below text to it:

file_roots: # Allows minions and proxy-minions to access SLS files with salt:// prefix
  base: # Name of the environment
    - /opt/salt/base/root/ # Full path where state and related files will be stored

pillar_roots: # Instructs master to look for pillar files at the specified location(s)
  base: # Name of the environment
    - /opt/salt/base/pillar # Full path where pillar files will be stored

Next, configure the proxy file in the /etc/salt/master directory and point it to the Salt master’s IP or FQDN, like so:

NOTE
In my setup, I am running the proxy service on the same machine where the salt-master is configured, so I will use the localhost.

$ cat /etc/salt/proxy
master: localhost

Save the changes and restart the salt-master service using sudo service salt-master restart command. After the restart, you will need to create necessary folders. For that, use the sudo mkdir -p /opt/salt/base/{root,pillar} command.

$ sudo mkdir -p /opt/salt/base/{root,pillar}
$ ll /opt/salt/base/
total 16
drwxr-xr-x 4 root root 4096 Jul  7 00:17 ./
drwxr-xr-x 3 root root 4096 Jul  7 00:17 ../
drwxr-xr-x 2 root root 4096 Jul  7 00:17 pillar/
drwxr-xr-x 2 root root 4096 Jul  7 00:17 root/

Great, the salt-master is ready. Now you can configure the pillar files for the proxy-minion service. This will contain instructions on how to communicate with the networking devices. Create junos-pillar.sls file in the /opt/salt/base/pillar/ folder and populate with the following data:

NOTE
Some vendors, such as Juniper and Cisco, have their own proxy types. Cisco currently has proxy module only for NXOS platform. For IOS, you would need to use either netmiko or NAPALM. To avoid complications, we will stick with netmiko for all three platforms.

proxy:
  proxytype: netmiko # Proxy type
  device_type: juniper_junos # A setting required by the netmiko
  ip: 172.29.224.103 # IP address of the host
  username: admin
  password: admin123

Next, create the nxos-pillar.sls file in the same folder and populate with following data:

proxy:
  proxytype: netmiko
  device_type: cisco_nxos
  ip: 172.29.224.102
  username: admin
  password: admin123

Finally create the ios-pillar.sls:

proxy:
  proxytype: netmiko
  device_type: cisco_xe
  ip: 172.29.224.101
  username: admin
  password: admin123

NOTE
You can create other pillar files for the same devices and use different proxytypes. For Juniper devices you can use junos proxy and for Cisco NXOS devices you can use nxos or nxos_api proxies.

To compare with Ansible, this is similar to the inventory file where you need to define the ansible_host, ansible_user, ansible_password and ansible_network_os variables.

[base]
IOS-XE ansible_host=172.29.224.101 ansible_user=admin ansible_password=admin123 ansible_network_os=ios ansible_connection=network_cli
NX-OS ansible_host=172.29.224.102 ansible_user=admin ansible_password=admin123 ansible_network_os=nxos ansible_connection=network_cli
Junos ansible_host=172.29.224.103 ansible_user=admin ansible_password=admin123 ansible_network_os=junos ansible_connection=netconf

Okay, let’s take a quick break to understand how these files were constructed. The most important one is the proxy key. Salt will look for this key, and the information within it, to set up the the proxy-minion service. The rest of the key value pairs were defined based on the official documentation that can be found here.

Now you need to create the pillar top.sls file (main file) where each pillar file will be associated with a respective proxy-minion name. The top.sls file should be located in the /opt/salt/base/pillar/ folder and should contain the following data:

base: # The name of the environment
  'Junos': # The name of the proxy-minion. Can be any name, but it is a good idea to put the device's hostname.
    - junos-pillar # This is the name of the previously created pillar file. Note the `.sls` suffix is not specified.
  'NX-OS':
    - nxos-pillar
  'IOS-XE':
    - ios-pillar

It is time to start the proxy-minion service for each host. For that, use the following commands:

NOTE
The value passed to the --proxyid argument should match the name of the proxy-minion defined in the top.sls file.

$ sudo salt-proxy --proxyid=Junos -d
$ sudo salt-proxy --proxyid=NX-OS -d
$ sudo salt-proxy --proxyid=IOS-XE -d

Finally, if everything was done properly, you should see three unaccepted keys after you issue the sudo salt-key --list-all command. These are the requests that the proxy services made to register the proxy-minions against the master node.

$ sudo salt-key --list-all
Accepted Keys:
Denied Keys:
Unaccepted Keys:
IOS-XE
Junos
NX-OS
Rejected Keys:

To accept all keys at once, issue sudo salt-key --accept-all command. When prompted press y.

$ sudo salt-key --accept-all
The following keys are going to be accepted:
Unaccepted Keys:
IOS-XE
Junos
NX-OS
Proceed? [n/Y] y
Key for minion IOS-XE accepted.
Key for minion Junos accepted.
Key for minion NX-OS accepted.

SaltStack and Ansible Ad Hoc Commands

You made it to the fun part of the blog. Here you will start executing some ad hoc commands against the networking devices. To start, you will need to install some Python packages on both Salt and Ansible hosts. Using sudo pip install netmiko junos-eznc jxmlease yamlordereddictloader napalm command install the packages.

$ sudo pip install netmiko junos-eznc jxmlease yamlordereddictloader napalm
Collecting napalm
  Downloading napalm-3.3.1-py2.py3-none-any.whl (256 kB)
     |████████████████████████████████| 256 kB 91 kB/s 
Collecting junos-eznc
  Downloading junos_eznc-2.6.1-py2.py3-none-any.whl (195 kB)
     |████████████████████████████████| 195 kB 113 kB/s 
>>>
Trimmed for brevity
<<<

Now you can run sudo salt "*" test.ping command on the salt-master

$ sudo salt "*" test.ping
IOS-XE:
    True
Junos:
    True
NX-OS:
    True

The Ansible counterpart command will be: ansible -i inv.ini base -m ping.

Now let’s take a look at the grains that salt master gathers from the devices. The sudo salt "*" grains.items command will reveal the below information. Similarly, you can do the same thing from the Ansible host using ansible -m ios_facts -i inv.ini IOS-XE, ansible -m nxos_facts -i inv.ini NX-OS, and ansible -m junos_facts -i inv.ini Junos commands.

NOTE
Unfortunately, netmiko does not pull grain information from the devices, but hopefully in the future this gets fixed.

Junos:
    ----------
    cpuarch:
        x86_64
    cwd:
        /
    dns:
        ----------
        domain:
        ip4_nameservers:
            - 192.168.65.5
        ip6_nameservers:
        nameservers:
            - 192.168.65.5
        options:
        search:
        sortlist:
    fqdns:
    gpus:
    hwaddr_interfaces:
        ----------
    id:
        Junos
<<<
TRIMMED
>>>
NX-OS:
    ----------
    cpuarch:
        x86_64
    cwd:
        /
    dns:
        ----------
        domain:
        ip4_nameservers:
            - 192.168.65.5
        ip6_nameservers:
        nameservers:
            - 192.168.65.5
        options:
        search:
        sortlist:
    fqdns:
    gpus:
    hwaddr_interfaces:
        ----------
    id:
        NX-OS
<<<
TRIMMED
>>>
IOS-XE:
    ----------
    cpuarch:
        x86_64
    cwd:
        /
    dns:
        ----------
        domain:
        ip4_nameservers:
            - 192.168.65.5
        ip6_nameservers:
        nameservers:
            - 192.168.65.5
        options:
        search:
        sortlist:
    fqdns:
    gpus:
    hwaddr_interfaces:
        ----------
    id:
        IOS-XE
<<<
TRIMMED
>>>

Below data were gathered to demonstrate what grains will look like if different proxytypes were used:

$ sudo salt "IOS-XE" napalm_net.facts
IOS-XE:
    ----------
    comment:
    out:
        ----------
        fqdn:
            IOS-XE.local.host
        hostname:
            IOS-XE
        interface_list:
            - GigabitEthernet1
            - GigabitEthernet2
            - GigabitEthernet3
            - GigabitEthernet4
        model:
            CSR1000V
        os_version:
            Virtual XE Software (X86_64_LINUX_IOSD-UNIVERSALK9-M), Version 16.12.3, RELEASE SOFTWARE (fc5)
        serial_number:
            ABC123ABC123
        uptime:
            5820
        vendor:
            Cisco
    result:
        True
$ sudo salt "NX-OS" nxos.grains
NX-OS:
    ----------
    hardware:
        ----------
        Device name:
            NX-OS
        bootflash:
            4287040 kB
    plugins:
        - Core Plugin
        - Ethernet Plugin
    software:
        ----------
        BIOS:
            version 
        BIOS compile time:

        NXOS compile time:
            12/22/2019 2:00:00 [12/22/2019 14:00:37]
        NXOS image file is:
            bootflash:///nxos.9.3.3.bin
$ sudo salt "Junos" junos.facts
Junos:
    ----------
    facts:
        ----------
        2RE:
            False
        HOME:
            /var/home/admin
        RE0:
            ----------
            last_reboot_reason:
                Router rebooted after a normal shutdown.
            mastership_state:
                master
            model:
                VSRX RE
            status:
                Testing
            up_time:
                9 minutes, 20 seconds
        RE1:
            None
        RE_hw_mi:
            False
        current_re:
            - master
            - fpc0
            - node
            - fwdd
            - member
            - pfem
            - re0
            - fpc0.pic0
        domain:
            None
        fqdn:
            Junos
        hostname:
            Junos
>>>
Trimmed for brevity
<<<

You have probably noticed that for the example data, I used different modules like junos.facts or nxos.grains. So why can’t we use the vendor-specific modules? This is due to the proxytype assigned to the device. Even though the salt "Junos" junos.facts is a correct command, Salt will check whether it is associated with proxytype: netmiko and, if not, it will report that it is not available for use.

To understand the syntax of the salt shell command, I would refer to the image below. The salt allows for the commands to be executed on remote systems (minions) in parallel. The documentation for all supported salt modules can be found here.

Here are some more commands for you to try:

sudo salt "Junos" netmiko.send_command "show version"
sudo salt --out json "IOS-XE" netmiko.send_command "show ip interface brief" use_textfsm=True
sudo salt "NX-OS" netmiko.send_config config_commands="['hostname NX-OS-SALT']"

SaltStack State Files and Ansible Playbooks

Ad hoc commands are very useful for checking or obtaining information from multiple devices, but they are not practical because they can’t act upon that information the way SaltState files (formulas) can. Similarly, the Ansible Playbooks are used.

You can start by creating a new pillar file, which should contain NTP server information. The file should be placed in the /opt/salt/base/pillar/ folder, named as ntp_servers.sls and contain the following data:

ntp_servers:
  - 172.29.224.1

Next, the top.sls file needs to be updated with instructions permitting all proxy-minions to use the information contained in the ntp_servers.sls file.

$ cat /opt/salt/base/pillar/top.sls
base: # Environment name
  'Junos':    # proxy-minion name
    - junos-pillar    # pillar file
  'IOS-XE':
    - ios-pillar
  'NX-OS':
    - nxos-pillar
  '*': # All hosts will match.
    - ntp_servers

Now you need to refresh the pillar information by using sudo salt "*" saltutil.refresh_pillar command.

IOS-XE:
    True
Junos:
    True
NX-OS:
    True

Confirm that information is available to all three hosts with sudo salt "*" pillar.data or sudo salt "*" pillar.items commands:

$ sudo salt "*" pillar.data
IOS-XE:
    ----------
    ntp_servers:
        - 172.29.224.1
    proxy:
        ----------
        driver:
            ios
        host:
            172.29.224.101
        password:
            admin123
        provider:
            napalm_base
        proxytype:
            napalm
        username:
            admin
NX-OS:
    ----------
    ntp_servers:
        - 172.29.224.1
    proxy:
        ----------
        connection:
            ssh
        host:
            172.29.224.102
        key_accept:
            True
        password:
            admin123
        prompt_name:
            NX-OS
        proxytype:
            nxos
        username:
            admin
Junos:
    ----------
    ntp_servers:
        - 172.29.224.1
    proxy:
        ----------
        host:
            172.29.224.103
        password:
            admin123
        port:
            830
        proxytype:
            junos
        username:
            admin

Next, you need to create a Jinja template for NTP configuration. The IOS-XE and NX-OS devices share the same configuration syntax, so all you need to do is to check the ID value stored in grains and identify the Junos device and, based on that, construct proper configuration syntax for it. Name the file as ntp_config.set and store it in the /opt/salt/base/root/ folder. The contents of the file should look like this:

{% set ntp = pillar['ntp_servers'][0] %}
{% set device_type = grains['id'] %}
{% if device_type == "Junos" %}
set system ntp server {{ ntp }}
{% else %}
ntp server {{ ntp }}
{% endif %}

Finally, you need to create the Salt State file (aka formula) and specify what it should do. The file needs to be located in the /opt/salt/base/root/ folder and named as update_ntp.sls with the following contents:

UPDATE NTP CONFIGURATION: # The ID name of the state. Should be unique if multiple IDs are present in the formula.
  module.run: # state function to run module functions
    - name: netmiko.send_config  # Name of the module function
    - config_file: salt://ntp_config.set # Argument that needs to be passed to the module function. The salt:// will convert to `/opt/salt/base/root/` path.

You might ask: “Why did we use module.run to run netmiko.send_config?” That’s a great question! SaltStack separates modules by their purposes, and execution modules cannot be used to maintain a state on the minion. For that, the state modules exist. Since netmiko does not have a state module, the module.run allows you to run execution modules in the Salt State file.

Okay, let’s compare the state file with an Ansible playbook. This is how a playbook that would do the same thing would look:

---

- name: UPDATE NTP CONFIGURATION
  hosts: base
  gather_facts: no
  vars:
    pillar:
      ntp_servers:
        - 172.29.224.1

  tasks:
    - name: 5 - UPDATE NTP CONFIGURATION FOR JUNOS
      junos_config:
        src: ntp_config.set
      vars:
        grains:
          id: Junos
      when: "ansible_network_os == 'junos'"

    - name: 10 - UPDATE NTP CONFIGURATION FOR NX-OS
      nxos_config:
        src: ntp_config.set
      vars:
        grains:
          id: NX-OS
      when: "ansible_network_os == 'nxos'"

    - name: 15 - UPDATE NTP CONFIGURATION FOR IOS
      ios_config:
        src: ntp_config.set
      vars:
        grains:
          id: IOS-XE
      when: "ansible_network_os == 'ios'"

The salt "*" state.apply update_ntp command will apply the configuration:

$ salt "*" state.apply update_ntp
IOS:
----------
          ID: UPDATE NTP CONFIGURATION
    Function: module.run
        Name: netmiko.send_config
      Result: True
     Comment: Module function netmiko.send_config executed
     Started: 02:26:17.021615
    Duration: 858.925 ms
     Changes:
              ----------
              ret:
                  configure terminal
                  Enter configuration commands, one per line.  End with CNTL/Z.
                  IOS-XE(config)#ntp server 172.29.224.1
                  IOS-XE(config)#end
                  IOS-XE#

Summary for IOS
------------
Succeeded: 1 (changed=1)
Failed:    0
------------
Total states run:     1
Total run time: 858.925 ms
NXOS:
----------
          ID: UPDATE NTP CONFIGURATION
    Function: module.run
        Name: netmiko.send_config
      Result: True
     Comment: Module function netmiko.send_config executed
     Started: 02:26:17.074178
    Duration: 5612.827 ms
     Changes:
              ----------
              ret:
                  configure terminal
                  Enter configuration commands, one per line. End with CNTL/Z.

                  NX-OS(config)# ntp server 172.29.224.1
                  NX-OS(config)# end
                  NX-OS#

Summary for NXOS
------------
Succeeded: 1 (changed=1)
Failed:    0
------------
Total states run:     1
Total run time:   5.613 s
SRX:
----------
          ID: UPDATE NTP CONFIGURATION
    Function: module.run
        Name: netmiko.send_config
      Result: True
     Comment: Module function netmiko.send_config executed
     Started: 02:26:17.083703
    Duration: 11961.207 ms
     Changes:
              ----------
              ret:
                  configure
                  Entering configuration mode

                  [edit]
                  admin@Junos# set system ntp server 172.29.224.1

                  [edit]
                  admin@Junos# exit configuration-mode
                  Exiting configuration mode

                  admin@Junos>

Summary for SRX
------------
Succeeded: 1 (changed=1)
Failed:    0
------------
Total states run:     1
Total run time:  11.961 s

Conclusion

SaltStack is a very powerful tool, and covering all of its aspects, such as reactors and syslog engines, here would be impossible. But I hope that the information presented above was enough for you to get started learning it. If you have any questions, feel free to leave a comment below, and I will do my best to answer them!

-Armen

Tags :

ansible automation netdevops salt tutorial

Does this all sound amazing? Want to know more about how Network to Code can help you do this, reach out to our sales team. If you want to help make this a reality for our clients, check out our careers page.

Contact Us to Learn More

Share details about yourself & someone from our team will reach out to you ASAP!

Parsing XML with Python and Ansible

Blog Detail

Which data type is more popular, JSON or XML? I believe the overwhelming majority will say “JSON”, which is understandable, because JSON is easier to read, easier to understand, and more human friendly. Besides that, there are a lot of reading materials around JSON and how to handle such data. But what to do when only XML is supported?

The default solution for most of us today is to convert XML to JSON or Python dictionary. But this approach has some significant drawbacks because XML and JSON are not 100% compatible. XML doesn’t have the same distinction between a list and a dictionary. Depending on what you are processing you can end up with a different datastructure if you have one element returned, an interface for example, or multiple.

There is a better way to process XML data in Python by using the native construct and libraries available. This blog is intended to give you tips and tricks on how to parse XML data with Python and Ansible.

XML Data Structure

In this section, we will briefly go over the XML Data Structure and the terminology used in it, which will help us better understand the XML data.

The image below depicts a sample XML data in a tree format.

The corresponding XML data will look like this:

<rpc-reply>
    <interface-information>
        <physical-interface type='eth'>
            <name>ge-0/0/0</name>
            <admin-status>up</admin-status>
            <oper-status>up</oper-status>
            <logical-interface>
                <name>ge-0/0/0.0</name>
                <admin-status>up</admin-status>
                <oper-status>up</oper-status>
                <filter-information>
                </filter-information>
                <address-family>
                    <address-family-name>inet</address-family-name>
                    <interface-address>
                        <ifa-local>172.16.0.151/24</ifa-local>
                    </interface-address>
                </address-family>
            </logical-interface>
        </physical-interface>
    </interface-information>
</rpc-reply>

NOTE
Some XML data will have namespaces, and we will address them a bit later.

Now let’s explain the terminology depicted in the image:

root element – The element at the root of the XML tree. XML data will always have one root element. This element can also be referred to as parent element.
child element – Any element right below the root element is a child element. The root element may contain one or more child elements.
attribute – A tag for an element providing some information about it.
data – The data the element contains.

NOTE
This is not a complete overview of the XML, but rather a quick introduction to it. The complete information about the XML can be found here.

Parsing XML with Python

Python has a very sophisticated built-in library called xml.etree.ElementTree to deal with XML data. Before we jump into the Python interpreter and start parsing the data, we will need to address XML XPath and the methods available to the xml.etree.ElementTree class.

XML XPath Support

XPath uses path expressions to find element(s) and relative data in the XML document. The xml.etree.ElementTree supports the following XPath expressions¹:

Syntax	Meaning
tag	Selects all child elements with the given tag. For example, spam selects all child elements named spam, and spam/egg selects all grandchildren named egg in all children named spam. `{namespace}` selects all tags in the given namespace, `{}spam` selects tags named spam in any (or no) namespace, and `{}*` selects only tags that are not in a namespace.
*	Selects all child elements, including comments and processing instructions. For example, `*/egg` selects all grandchildren named egg.
.	Selects the current node. This is mostly useful at the beginning of the path, to indicate that it’s a relative path.
//	Selects all subelements, on all levels beneath the current element. For example, .//egg selects all egg elements in the entire tree.
..	Selects the parent element. Returns None if the path attempts to reach the ancestors of the start element (the element find was called on).
[@attrib]	Selects all elements that have the given attribute.
[@attrib=’value’]	Selects all elements for which the given attribute has the given value. The value cannot contain quotes.
[tag]	Selects all elements that have a child named tag. Only immediate children are supported.
[.=’text’]	Selects all elements whose complete text content, including descendants, equals the given `text`.
[tag=’text’]	Selects all elements that have a child named `tag` whose complete text content, including descendants, equals the given `text`.
[position]	Selects all elements that are located at the given position. The position can be either an integer (1 is the first position), the expression `last()` (for the last position), or a position relative to the last position (e.g., `last()-1)`.

xml.etree.ElementTree Methods and Attributes

These are the methods and attributes that we will be using:

NOTE
All elements of the loaded XML data will be objects of the Element Python class and will support these methods and attributes.

Methods:

iter("<element>") – recursively iterates over all the sub-tree below the root element. Requires the element name.
findall("<xpath-expression>") – finds elements using XPath expression.
find("<element>") – finds the first child element with a particular tag.
get("<attribute-tag>") – gets the element’s attribute value.

Attributes:

tag – shows the tag name of the element (rpc-reply).
attrib – shows the value of the attribute ({'type': 'eth'}).
text – shows the data assigned to the element (ge-0/0/0.0).

Now we are ready to work with the XML data. Let’s start by opening the Python interpreter and importing the xml.etree.ElementTree class as ET for brevity.

>>> import xml.etree.ElementTree as ET

We will need to load the data now. It can be done two ways. Either load the XML file with the parse() method then obtain the root element with getroot() method, or load it from string using fromstring().

NOTE
We will be using the XML data shown in the XML Data Structure section, but with more interfaces.

>>> tree = ET.parse('interface_data.xml')
>>> root = tree.getroot()

>>> root = ET.fromstring(interface_data)

To print the tag of the root element we can use the tag attribute

>>> root.tag
'rpc-reply

Let’s iterate of the child elements of the root element and print its tag names:

>>> for child in root:
...      print(child.tag)
...
interface-information

As we can see from the output, the root element has only one child element, with interface-information tag name. To see more data, we can iterate over the interface-information element and print its elements’ tag names and attributes:

>>> for child in root:
...     for element in child:
...         print(f"Element Name: {element.tag}  Element Attribute: {element.attrib}")
...
Element Name: physical-interface  Element Attribute: {'type': 'eth'}
Element Name: physical-interface  Element Attribute: {'type': 'eth'}
Element Name: physical-interface  Element Attribute: {'type': 'eth'}
Element Name: physical-interface  Element Attribute: {'type': 'eth'}
Element Name: physical-interface  Element Attribute: {'type': 'eth'}

Even though the nested iteration works, it is not optimal, since the XML tree can have several layers. Instead, we can use iter() method and provide an element name as a parameter. In this case, we will iterate over only the found elements. For example, let’s iterate over all name elements and show their contents.

>>> for element in root.iter('name'):
...     print(element.text)
...
ge-0/0/0
ge-0/0/0.0
ge-0/0/1
ge-0/0/1.0
ge-0/0/2
ge-0/0/2.0
ge-0/0/3
ge-0/0/3.0
ge-0/0/4
ge-0/0/4.0

In the same way, we can iterate over the elements in question, but with the help of the findall() method. For it, we will need to provide XPath expression as an argument. This time, we would like to print all IP addresses that are assigned to the ifa-local element:

NOTE
The period before the forward slashes indicates that the search needs to happen from current element.

>>> for element in root.findall('.//ifa-local'):
...     print(element.text)
...
172.16.0.151/24
192.168.10.1/24
10.10.10.1/24
172.31.177.1/24
192.168.0.1/24

The XPath expression to find a particular element with a specific attribute will be a little bit different. For it, we will need to use findall(.//*[@type]) syntax to search from current element against all the elements that have an attribute tag of type:

>>> for element in root.findall('.//*[@type]'):
...     print(element.tag, element.attrib)
...
physical-interface {'type': 'eth'}
physical-interface {'type': 'eth'}
physical-interface {'type': 'eth'}
physical-interface {'type': 'eth'}
physical-interface {'type': 'eth'}

Parsing XML Data ith Namespaces

XML Namespaces are used to group certain elements identified by Uniform Resource Identifier (URI) and avoid any element name conflicts in the XML document. Detailed information about XML Namespaces can be found here.

Let’s look at an XML data with name spaces.

<rpc-reply xmlns:junos="http://xml.juniper.net/junos/20.2R0/junos">
    <interface-information xmlns="http://xml.juniper.net/junos/20.2R0/junos-interface" junos:style="terse">
        <physical-interface type='eth'>
            <name>ge-0/0/0</name>
            <admin-status>up</admin-status>
            <oper-status>up</oper-status>
            <logical-interface>
                <name>ge-0/0/0.0</name>
                <admin-status>up</admin-status>
                <oper-status>up</oper-status>
                <filter-information>
                </filter-information>
                <address-family>
                    <address-family-name>inet</address-family-name>
                    <interface-address>
                        <ifa-local>172.16.0.151/24</ifa-local>
                    </interface-address>
                </address-family>
            </logical-interface>
        </physical-interface>
    </interface-information>
</rpc-reply>

If we try to find a specific element using one of the methods that we discussed before, soon we will find out that no elements are matched.

>>> tree_xmlns = ET.parse("interface_data_with_xmlns.xml")
>>> root_xmlns = tree_xmlns.getroot()
>>> for element in root_xmlns.findall('.//ifa-local'):
...     print(element.text)
...
>>>

The reason is that each element of the interface-information child element is appended with {URI} tag. To see how it looks, we can print the tag name of the root’s child-element:

NOTE
The root element does not have the URI tag appended since it has only XML name definition rather than assignment.
>> print(root_xmlns[0].tag)
{http://xml.juniper.net/junos/20.2R0/junos-interface}interface-information
>>
Now, if we change the syntax and search for the all ifa-local elements, we should get the desired result:
>> for element in root_xmlns.findall('.//{http://xml.juniper.net/junos/20.2R0/junos-interface}ifa-local'):
...     print(element.text)
...
172.16.0.151/24
192.168.10.1/24
10.10.10.1/24
172.31.177.1/24
192.168.0.1/24
Python recommends a different way to search for elements with namespace. For that, we will need to create a dictionary and map a key to the namespace URI, then, using the desired class method, provide the dictionary as a second argument. The syntax looks like this:
>> ns = dict(interface="http://xml.juniper.net/junos/20.2R0/junos-interface")
>> for element in root_xmlns.findall('.//interface:ifa-local', ns):
...     print(element.text)
...
172.16.0.151/24
192.168.10.1/24
10.10.10.1/24
172.31.177.1/24
192.168.0.1/24
In Python 3.8 and above, the namespace can be referenced with an asterisk character(*):
>> for element in root_xmlns.findall('.//{*}ifa-local'):
...     print(element.text)
...
172.16.0.151/24
192.168.10.1/24
10.10.10.1/24
172.31.177.1/24
192.168.0.1/24

Parsing XML with Ansible

Parsing XML with Ansible can be done using two methods. The first method involves using community.general.xml module which is very well documented. The second, more interesting, one using the parse_xml filter. Here we will be showing the second option.

`parse_xml`Ansible Filter

To use the filter, we will need to create a specification file. The file has two parts: the first part defines the elements that need to be extracted from the XML data. The second part defines the variables and their values that will be available in the Ansible Playbook. We will start by looking at the first part.

keys:
  result:
    value: "{{ dev_intf }}"
    top: interface-information/physical-interface
    items:
      name: name
      admin_status: admin-status
      oper_status: oper-status
      ifl_name: logical-interface/name
      ip_addr: .//ifa-local

Let’s break it all down:

keys – This is a predefined root level key name.
result – The name of this key can be set to anything.
value – This is a predefined key which holds a Jinja2 variable which will be mapped to a variable that will be defined in the second part of the specification file.
top – This key holds the path to the element that will be iterated over for data extraction. Note that the path starts from the child element not the root element.
items – This is a predefined key which will hold multiple items.
- name – Key that will hold the value that is assigned to the <name></name> element.
- admin_status – Key that will hold the value that is assigned to the <admin-status></admin-status> element.
- oper_status – Key that will hold the value that is assigned to the <oper-status></oper-status> element.
- ifl_name – Key that will hold the value that is assigned to the <name></name> element which is a child elment of the <logical-interface></logical-interface> element.
- ip_addr – Key that will hold the value that is assigned to the <ifa-local></ifa-local> element. Note that here we are using XPath expression to locate the desired element.

Now let’s look at the second part of the file and break it down:

vars:
  dev_intf:
    name: "{{ item.name }}"
    admin_status: "{{ item.admin_status }}"
    oper_status: "{{ item.oper_status }}"
    ifl_name: "{{ item.ifl_name }}"
    ip_addr: "{{ item.ip_addr }}"

vars – This is a well-known Ansible key for defining variables.
dev_intf – The root key named by us that will hold the data extracted from the XML.
- name – Variable key that will hold the value assigned to the name key defined under items.
- admin_status – Variable key that will hold the value assigned to the admin_status defined key under items.
- oper_status – Variable key that will hold the value assigned to the oper_status defined key under items.
- ifl_name – Variable key that will hold the value assigned to the ifl_name defined key under items.

The entire specs file looks like this:

keys:
  result:
    value: "{{ dev_intf }}"
    top: interface-information/physical-interface
    items:
      name: name
      admin_status: admin-status
      oper_status: oper-status
      ifl_name: logical-interface/name
      ip_addr: .//ifa-local
vars:
  dev_intf:
    name: "{{ item.name }}"
    admin_status: "{{ item.admin_status }}"
    oper_status: "{{ item.oper_status }}"
    ifl_name: "{{ item.ifl_name }}"
    ip_addr: "{{ item.ip_addr }}"

Now let’s look at the simple Ansible Playbook in which we will use the parse_xml filter.

- name: READ XML
  connection: local
  gather_facts: no
  hosts: localhost
  tasks:
    - set_fact:
        parsed_xml_data: "{{ lookup('file', 'interface_data.xml') | parse_xml('interface.specs') }}"
    - debug:
        var: parsed_xml_data

Finally, let’s run the Playbook against the XML data and see the output.

$ ansible-playbook pb_parse_xml.yml

PLAY [READ XML] *******************************************************************************************************

TASK [set_fact] *******************************************************************************************************
ok: [localhost]

TASK [debug] **********************************************************************************************************
ok: [localhost] => {
    "xml_data_parsed": {
        "result": [
            {
                "admin_status": "up",
                "ifl_name": "ge-0/0/0.0",
                "ip_addr": "172.16.0.151/24",
                "name": "ge-0/0/0",
                "oper_status": "up"
            },
            {
                "admin_status": "up",
                "ifl_name": "ge-0/0/1.0",
                "ip_addr": "192.168.10.1/24",
                "name": "ge-0/0/1",
                "oper_status": "up"
            },
            {
                "admin_status": "up",
                "ifl_name": "ge-0/0/2.0",
                "ip_addr": "10.10.10.1/24",
                "name": "ge-0/0/2",
                "oper_status": "up"
            },
            {
                "admin_status": "up",
                "ifl_name": "ge-0/0/3.0",
                "ip_addr": "172.31.177.1/24",
                "name": "ge-0/0/3",
                "oper_status": "up"
            },
            {
                "admin_status": "up",
                "ifl_name": "ge-0/0/4.0",
                "ip_addr": "192.168.0.1/24",
                "name": "ge-0/0/4",
                "oper_status": "up"
            }
        ]
    }
}

PLAY RECAP ************************************************************************************************************
localhost                  : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Conclusion

I hope this post will help you understand how to parse XML data natively in Python and Ansible without converting it to JSON automatically.

-Armen

Tags :

ansible automation netdevops parsing python structured-data xml

Contact Us to Learn More

Share details about yourself & someone from our team will reach out to you ASAP!

Cookie	Duration	Description
__hssc	30 minutes	HubSpot sets this cookie to keep track of sessions and to determine if HubSpot should increment the session number and timestamps in the __hstc cookie.
__hssrc	session	This cookie is set by Hubspot whenever it changes the session cookie. The __hssrc cookie set to 1 indicates that the user has restarted the browser, and if the cookie does not exist, it is assumed to be a new session.
cookielawinfo-checkbox-advertisement	1 year	Set by the GDPR Cookie Consent plugin, this cookie records the user consent for the cookies in the "Advertisement" category.
cookielawinfo-checkbox-analytics	11 months	This cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Analytics".
cookielawinfo-checkbox-functional	11 months	The cookie is set by GDPR cookie consent to record the user consent for the cookies in the category "Functional".
cookielawinfo-checkbox-necessary	11 months	This cookie is set by GDPR Cookie Consent plugin. The cookies is used to store the user consent for the cookies in the category "Necessary".
cookielawinfo-checkbox-others	11 months	This cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Other.
cookielawinfo-checkbox-performance	11 months	This cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Performance".
CookieLawInfoConsent	1 year	CookieYes sets this cookie to record the default button state of the corresponding category and the status of CCPA. It works only in coordination with the primary cookie.
viewed_cookie_policy	11 months	The cookie is set by the GDPR Cookie Consent plugin and is used to store whether or not user has consented to the use of cookies. It does not store any personal data.

Cookie	Duration	Description
__cf_bm	30 minutes	Cloudflare set the cookie to support Cloudflare Bot Management.
li_gc	5 months 27 days	Linkedin set this cookie for storing visitor's consent regarding using cookies for non-essential purposes.
lidc	1 day	LinkedIn sets the lidc cookie to facilitate data center selection.
UserMatchHistory	1 month	LinkedIn sets this cookie for LinkedIn Ads ID syncing.

Cookie	Duration	Description
__hstc	5 months 27 days	Hubspot set this main cookie for tracking visitors. It contains the domain, initial timestamp (first visit), last timestamp (last visit), current timestamp (this visit), and session number (increments for each subsequent session).
_ga	1 year 1 month 4 days	Google Analytics sets this cookie to calculate visitor, session and campaign data and track site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognise unique visitors.
_gat_gtag_UA_*	1 minute	Google Analytics sets this cookie to store a unique user ID.
_gid	1 day	Google Analytics sets this cookie to store information on how visitors use a website while also creating an analytics report of the website's performance. Some of the collected data includes the number of visitors, their source, and the pages they visit anonymously.
AnalyticsSyncHistory	1 month	Linkedin set this cookie to store information about the time a sync took place with the lms_analytics cookie.
CONSENT	2 years	YouTube sets this cookie via embedded YouTube videos and registers anonymous statistical data.
hubspotutk	5 months 27 days	HubSpot sets this cookie to keep track of the visitors to the website. This cookie is passed to HubSpot on form submission and used when deduplicating contacts.
ln_or	1 day	Linkedin sets this cookie to registers statistical data on users' behaviour on the website for internal analytics.

Cookie	Duration	Description
bcookie	1 year	LinkedIn sets this cookie from LinkedIn share buttons and ad tags to recognize browser IDs.
bscookie	1 year	LinkedIn sets this cookie to store performed actions on the website.
li_sugr	3 months	LinkedIn sets this cookie to collect user behaviour data to optimise the website and make advertisements on the website more relevant.
VISITOR_INFO1_LIVE	5 months 27 days	YouTube sets this cookie to measure bandwidth, determining whether the user gets the new or old player interface.
YSC	session	Youtube sets this cookie to track the views of embedded videos on Youtube pages.
yt-remote-connected-devices	never	YouTube sets this cookie to store the user's video preferences using embedded YouTube videos.
yt-remote-device-id	never	YouTube sets this cookie to store the user's video preferences using embedded YouTube videos.
yt.innertube::nextId	never	YouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.
yt.innertube::requests	never	YouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen.

Blog Detail

Salt Terminologies

Salt and Ansible Playground

Salt and Ansible Installation

Salt Environment Configuration

SaltStack and Ansible Ad Hoc Commands

SaltStack State Files and Ansible Playbooks

Conclusion

Tags :

Share :

Contents

Recent Posts

December 5, 2024

November 25, 2024

November 15, 2024

October 4, 2024

September 5, 2024

Contact Us to Learn More

Blog Detail

XML Data Structure

Parsing XML with Python

XML XPath Support

xml.etree.ElementTree Methods and Attributes

Parsing XML Data ith Namespaces

Parsing XML with Ansible

parse_xmlAnsible Filter

Conclusion

Tags :

Share :

Contents

Recent Posts

December 5, 2024

November 25, 2024

November 15, 2024

October 4, 2024

September 5, 2024

Contact Us to Learn More

`parse_xml`Ansible Filter