Manipulating Data with Jinja Map and Selectattr

Manipulating data with the Jinja filters, Map, Selectattr, and Select provides quick and efficient ways to create new data structures. This can simplify creating lookup dictionaries and avoids complex loops or Jinja templating options.

In many instances in ansible, I’ll use an ordered list of dictionaries as my data model; this specifically keeps the preferred ordering for the device and type of configuration as well as makes it easy to do operations on all the members of that list. For some operations that rely on specific dictionaries, it can be convoluted to dig that information out of the list, or work on a subset of the list, without going through loops and conditionals, without these tools.

Like all good overviews, it’s best explained with an example. We’ll start with this example data structure for some selected interfaces, a commonly used pattern.

---
interfaces:
  - name: "mgmt"
    enable: true
    dhcp: true
    ip_address: no
    description: "management"
  - name: "1/1/1"
    enable: true
    ip_address: 10.1.1.2/24
    description: "uplink to core-1"
  - name: "1/1/2"
    enable: true
    ip_address: 10.1.2.2/24
    description: "uplink to core-2"
  - name: "1/1/3"
    enable: true
    ip_address: 10.1.3.1/24
    description: "peerlink to agg-2"
  - name: "vlan42"
    ip_address: 10.42.0.2/24
    ip_helper: false
    description: "inband management"
  - name: "vlan44"
    ip_address: 10.44.0.3/24
    ip_helper: true
    fhrp: true
    description: "wifi aps"

Selecattr

Selectattr takes in a list of dictionaries and applies a test to each dictionary.

One of the first use cases might be to get all of the interfaces that have an IP address defined:

{{ interfaces | selectattr('ip_address') }}

With the selectattr filter, a Jinja test can be used as the second argument; this gives us a lot of flexibility when selecting our data. Additional arguments can be specified as inputs to the test specified in the second arg.

One thing to be careful of with the selectattr filter with this example: The attribute you use must be defined, but not all of our interfaces have ip_address defined. When the default test is applied to the value of the attribute (bool), the var doesn’t exist, which will result in an error. We’ll hit that error wherever ip_address is not defined in our interfaces data structure.

To get around this in our data, we specify the ‘defined’ parameter to test whether the attribute is there or not:

{{ interfaces | selectattr('ip_address', 'defined') }}

This returns a list of dictionaries where the ip_address attribute is present. Easy enough.

[
    {
        "enable": true,
        "ip_address": "10.1.1.2/24",
        "name": "1/1/1"
    },
    {...},
]

To be even more granular, we can chain the output to another selectattr filter and further test the ip_address or another attribute. For example, we can pass the list of interfaces with IP addresses and test whether the IP address on the interface is in another list of IP addresses:

{{ interfaces | selectattr('ip_address', 'defined') | selectattr('ip_address', 'in', '[10.1.3.1/24]') }}

Which gives us this output:

    [
        {
            "description": "peerlink to agg-2",
            "enable": true,
            "ip_address": "10.1.3.1/24",
            "name": "1/1/3"
        }
    ]

Map

Map allows us to turn a list of dictionaries into a simpler list or flatten the list of dictionaries. We specify the values of the list by giving Map the attribute we want to take from the dictionaries.

Now, let’s get a list of the IP addresses. We could easily use this in a route-map, prefix-list, or ACL.

{{ interfaces | map(attribute='ip_address', default='n/a') }}

Gives the output:

    [
        "n/a",
        "10.1.1.1/24",
        "10.1.2.1/24",
        "10.1.3.1/24",
        "10.42.0.1/24",
        "10.44.0.1/24"
    ]

Similar to the selectattr, Map expects the attribute to be defined. In this case we have to specify a default value in case the dictionary doesn’t contain the specified attribute, ip_address in this case.

It might be useful to ONLY have IP addresses in our list of IPs. Here we can combine the selectattr by filtering down the dictionaries where the ip_address attribute is defined and then using Map to filter the dictionaries down to the values of the ip_address attribute.

{{ interfaces | selectattr('ip_address', 'defined') | map(attribute='ip_address') }}

Output:

[
    "10.1.1.1/24",
    "10.1.2.1/24",
    "10.1.3.1/24",
    "10.42.0.1/24",
    "10.44.0.1/24"
]

The Map filter also gives us the ability to input another filter to use on the data in a list; this allows us to use a filter that doesn’t support working on lists to work on each element of the list without using a loop. Most of the Jinja built-in filters will work without the Map filter, however.

Other Useful Combinations

If we need to look up IP addresses by interface name, we can instead use the items2dict filter. This gives us the ability to map attributes of the dictionary to new keys and values. This is useful when specific addresses are used in specific ways, such as network-specific ACLs, prefix-lists, bgp neighbors, and others that need to reference interfaces in specific ways or different orders.

We first use selectattr to filter down to which dictionaries have an ip_address, then use items2dict to output a new dictionary where the interface name is the key and the ip_address is the value.

{{ interfaces | selectattr('ip_address', 'defined') | items2dict(key_name='name', value_name='ip_address') }}

This gives us a nice tidy lookup dictionary without any loops and a relatively readable call:

{
    "1/1/1": "10.1.1.1/24",
    "1/1/2": "10.1.2.1/24",
    "1/1/3": "10.1.3.1/24",
    "vlan42": "10.42.0.1/24",
    "vlan44": "10.44.0.1/24"
}