Filter JSON Data in Ansible Using json_query

Parsing structured JSON data in Ansible playbooks is a common task. Nowadays, the JSON format is heavily used by equipment vendors to represent complex objects in a structured way to allow programmatic interaction with devices. JSON utilizes two main data structures:

• object – an unordered collection of key/value pairs (like Python dict)
• array – an ordered sequence of objects (like Python list)

Ansible provides many built-in capabilities to consume JSON using Ansible specific filters or the Jinja2 built-in filters. The extensive filters available, and what to use and when, can be overwhelming at first and the desired result can often require multiple filters chained together. This can lead to complex task definition, making playbook maintenance more difficult.

The built-in json_query filter provides the functionality for filtering, shaping, and transforming JSON data. It uses the third-party jmespath library, a powerful JSON query language supporting the parsing of complex structured data.

Setup

The jmespath third-party library must be installed on the host for the json_query filter to operate.

pip install jmespath

Data

Those familiar with Palo Alto firewalls will recognize a modified version of an application content update query. This result dataset will be used to demonstrate how to manipulate JSON data using the json_query filter.

response:
      result:
        content-updates:
          entry:
          - app-version: 8368-6520
            current: 'no'
            downloaded: 'no'
            filename: panupv2-all-apps-8368-6520
            version: 8368-6520
          - app-version: 8369-6522
            current: 'no'
            downloaded: 'no'
            filename: panupv2-all-apps-8369-6522
            version: 8369-6522
          - app-version: 8367-6513
            current: 'no'
            downloaded: 'no'
            filename: panupv2-all-apps-8367-6513
            version: 8367-6513
          - app-version: 8371-6531
            current: 'no'
            downloaded: 'no'
            filename: panupv2-all-apps-8371-6531
            version: 8371-6531
          - app-version: 8366-6503
            current: 'no'
            downloaded: 'no'
            filename: panupv2-all-apps-8366-6503
            version: 8366-6503
          - app-version: 8370-6526
            current: 'no'
            downloaded: 'no'
            filename: panupv2-all-apps-8370-6526
            version: 8370-6526
          - app-version: 8373-6537
            current: 'no'
            downloaded: 'yes'
            filename: panupv2-all-apps-8373-6537
            version: 8373-6537
          - app-version: 8365-6501
            current: 'no'
            downloaded: 'no'
            filename: panupv2-all-apps-8365-6501
            version: 8365-6501
          - app-version: 8364-6497
            current: 'no'
            downloaded: 'no'
            filename: panupv2-all-apps-8364-6497
            version: 8364-6497
          - app-version: 8372-6534
            current: 'yes'
            downloaded: 'yes'
            filename: panupv2-all-apps-8372-6534
            version: 8372-6534

On Palo Alto devices the default stdout response is returned as a JSON encoded string. This string can be passed into the from_json filter to provide a valid JSON data structure. A result variable was set to simplify the readability of the following examples. The json_query filter expects valid JSON as an input, so the Jinja2 expression below can also be passed directly into the json_query filter.

- name: "SET FACT FOR DEVICE STDOUT RESPONSE"
  set_fact:
    result: "{{ (content_info['stdout'] | from_json)['response']['result'] }}"

The yaml callback plugin is configured in ansible.cfg for the following examples. This will render output to the console terminal in YAML format, which can be slightly easier to read. See references at the end for Ansible callback usage.

# Use the YAML callback plugin for output
stdout_callback = yaml

Practical Examples

The JMESPath Operators table below summarizes some of the most common operators used in the jmespath query language. See references at the end for the jmespath specification.

JMESPath Operators

Basic Filter

Using the result data, the filter is applied to the valid JSON result with an additional query string argument. The query string provides the expression that is evaluated against the data to return filtered output. The default data type returned by json_query filter is a list.

In the Basic Filter example below, the query string selects the version key for each element in the entry array.

- name: "BASIC FILTER"
  debug:
    msg: "{{ result['content-updates'] | json_query('entry[*].version') }}"

TASK [BASIC FILTER - VERSION] ***************************************************
ok: [demo-fw01] =>
  msg:
  - 8368-6520
  - 8369-6522
  - 8367-6513
  - 8371-6531
  - 8366-6503
  - 8370-6526
  - 8373-6537
  - 8365-6501
  - 8364-6497
  - 8372-6534

Return Array Element

A pipe expression can be used to select the value of an array by declaring the desired index location. This operates in much the same way as the unix pipe, within the query string.

In the Array Index example below, the first element is selected. Array indexing begins at zero.

- name: "ARRAY INDEX VALUE"
  debug:
    msg: "{{ result['content-updates'] | json_query('entry[*].version | [0]') }}"

The response output is a string value for the first element in the array.

TASK [ARRAY INDEX VALUE] *********************************************************
ok: [demo-fw01] =>
  msg: 8368-6520

Filter

Using a filter expression within the json_query string, the query result can be filtered using standard comparison operators. A filter expression allows each element of the array to be evaluated against the expression. If the result evaluates to true, the element is included in the returned result.

The equality comparision operator is used in the following example to retrieve the filename(s) of the software version(s) downloaded on the device. Elements in the array that have downloaded=='yes' will have the filename included in the returned result. This provides the ability to use certain keys for the selection criteria, and return the value(s) of other keys for the result.

- name: "FILTER EXACT MATCH"
  debug:
    msg: "{{ result['content-updates'] | json_query('entry[?downloaded==`yes`].filename') }}"

The output shows the list of filenames returned.

TASK [FILTER EXACT MATCH] ********************************************************
ok: [demo-fw01] =>
  msg:
  - panupv2-all-apps-8373-6537
  - panupv2-all-apps-8372-6534

Function

The jmespath library provides built-in functions to assist in transformation and filtering tasks, for example the max_by function that returns the maximum element in an array. The following task selects the filename of the maximum app-version value from the entry array. The & provides the ability to define an expression which will be evaluated as a data type value when processed by the function.

Using an Ansible variable for the query string can be a cleaner approach to the query definition. It also helps with string quotations that are necessary when the key name is hyphenated. When using an expression data type, the jmespath library requires the key name within quotation marks, e.g., &"key-name". Where key names are not hyphenated, quotation marks are not required, e.g., &keyname.

- name: "MAX BY APP-VERSION"
  set_fact:
    content_file: "{{ result['content-updates'] | json_query(querystr) }}"
  vars:
    querystr: 'max_by(entry, &"app-version").filename'

The resulting filename value is returned.

TASK [JMESPATH FUNCTION] *********************************************************
ok: [demo-fw01] =>
  msg: panupv2-all-apps-8373-6537

Using single quotes in the previous example instructs the filter to treat the expression as a string, returning the first element of the array.

vars:
  querystr: "max_by(entry, &'app-version').filename"

TASK [MAX BY APP-VERSION] *******************************************************
ok: [demo-fw01] =>
  msg: panupv2-all-apps-8368-6520

Query String with Dynamic Variable

Ansible facts can be substituted into the query string. Again, quotation marks are relevant, so it is preferable to define a separate string as a variable within the task.

In this example the filter expression is used with a built-in function, contains. This function provides a boolean result, based on a match with a search string, on any element within the version array. The search string in this case is an Ansible variable passed into the query using a Jinja2 template.

- name: "FUNCTION WITH VARIABLE"
  debug:
    msg: "{{ result['content-updates'] | json_query(querystr) }}"
  vars:
    querystr: "entry[?contains(version, '{{ content_version }}')].filename | [0]"
    content_version: "8368-6520"

Again, the first filename, within the returned list using the indexing capability, has been selected.

TASK [FUNCTION WITH VARIABLE] **************************************************
ok: [demo-fw01] =>
  msg: panupv2-all-apps-8368-6520

Multiple Expressions

Multiple expressions can be evaluated using the logical AND operation, following the normal truth table rules. Along with the filter operator, two expressions are provided to be evaluated. This will filter the resulting data based on two selection criteria and provide a list of version values.

- name: "MULTIPLE FILTER EXPRESSIONS"
  debug:
    msg: "{{ result['content-updates']  | json_query(querystr) }}"
  vars:
    querystr: "entry[?contains(filename, '{{ version }}') && downloaded==`yes`].version"
    version: "8373-6537"

In this case, one element is returned in the list.

TASK [MULTIPLE FILTER EXPRESSIONS] ***********************************************
ok: [demo-fw01] =>
  msg:
  - 8373-6537

Tips

Here are some general tips that could be useful while developing a playbook using json_query.

• Get familiar with basic jmespath expressions using the interactive tutorial.
• Ensure the data provided as input to the json_query filter is a valid JSON object.
• Use a test playbook, using the same data as your original playbook, with tasks dedicated to printing query string evaluation output to the console.
• Be mindful of quotes within a query string, especially when using linters that specify a standard that may conflict with the jmespath specification.