Introducing Design Builder: Design Driven Network Automation

Most people involved in network automation are familiar with the concept of a Source of Truth (SoT). The SoT is usually some form of database that maintains intended state of objects as well as their interdependency. The SoT provides a way to quickly ascertain what a network’s intended state should be, while often providing a way to see what the network’s state actually is. A new concept is emerging, known as Design Oriented Source of Truth. This idea takes network designs and codifies them, attaching additional meaning to the objects within the SoT. Nautobot is a source of truth that contains all sorts of information about a network’s state. Although many of the pieces of information within Nautobot are related, they are discretely managed. A new Nautobot App aims to simplify the process of codifying network designs and populating Nautobot objects based on these designs.

Introduction

It is very common to have a small set of standardized designs that are used to deploy many sites and services in enterprise networks. For example, branch office sites may have a few different designs depending on their size. There could be a design that uses a single branch office router for small sites. Another design could have two routers and an access switch for sites with a moderate user base. A third design could include a more complex switching infrastructure for sites with many employees. When companies do tech refreshes or new site builds, these standardized designs are used and new data must be created in the source of truth. The newly open-sourced Design Builder application was created to address this problem, and fulfills the idea that a standardized design can be taken from a network engineer and transformed into a format that can be consumed and executed by Nautobot. Design Builder can expand a minimal set of inputs into a full-fledged set of configuration objects within Nautobot. This includes any kind of data object that Nautobot can model. Everything from Rack and Device objects to IP addresses and BGP peering information.

Design Builder provides powerful mechanisms that make simple designs possible. The first is the ability to represent interrelated data in a meaningful hierarchy. For example, devices have interfaces and interfaces have IP addresses. Conceptually this seems like a very simple structure. However, if we were to manually use the REST API or ORM to handle creating objects like this, we would first have to create a device object and keep its ID in memory. We would then have to create interfaces with their device foreign-key set to the device ID we just created. Finally, we’d have to save all of the interface IDs and do the same with IP addresses. Design Builder provides a means to represent objects in YAML and produce their representation within the Nautobot database. A typical design workflow follows the following diagram:

Following this process, we can produce YAML files that intuitively represent the structure of the data we want to create. An example of a Design Builder YAML design can be seen in the following YAML document:

devices:
  - name: "Router 1"
    status__name: "Active"
    interfaces:
      - name: "GigabitEthernet0"
        type: "1000base-t"
        status__name: "Active"
        ip_addresses:
          - address: "192.168.0.1/24"
            status__name: "Active"

This YAML document would produce a single device, with a single Gigabit Ethernet interface. The interface itself has a single IP address. As demonstrated in the example, Design Builder automatically associates the parent/child relationships correctly, and there is no need to keep copies of primary and foreign keys. We can visually represent this YAML design with the following diagram:

Design Builder also provides a system to query for existing related objects using some attribute of the associated object. In the above example, the status field is actually a related object. Statuses are not just simple strings, they are first-class objects within the Nautobot database. In this case, the Status object with the name Active is predefined in Nautobot and does not need to be created. It does, however, need to be associated with the Device, the Interface, and the IPAddress objects.

This object relationship is actually a foreign-key relationship in the database and ORM. If we were using the Django ORM to associate objects, we would first need to look up the status before creating the associated objects. Design Builder provides a way to perform that lookup as part of the model hierarchy. Note that we’re looking up the status by its name: status__name. Design Builder has adopted similar syntax to Django’s field lookup. The field name and related field are separated by double underscores.

Use Cases

There are many use cases that are covered by the Design Builder, but we will highlight a very simple one in this post. Our example use case handles the creation of edge site designs within Nautobot. This use case is often seen when doing tech refreshes or new site build-outs.

Engineers commonly need to add a completely new set of data for a site. This could be the result of a project to refresh a site’s network infrastructure or it could be part of deploying a new site entirely. Even with small sites, the number of objects needing to be created or updated in Nautobot could be dozens or even hundreds. However, if a standardized design is developed then Design Builder can be used to auto-populate all of the data for new or refreshed sites.

Consider the following design, which will create a new site with edge routers, a single /24 prefix and two circuits for the site:

---
sites:
  - name: "LWM1"
    status__name: "Staging"
    prefixes:
      - prefix: "10.37.27.0/24"
        status__name: "Reserved"
    devices:
      - name: "LWM1-LR1"
        status__name: "Planned"
        device_type__model: "C8300-1N1S-6T"
        device_role__name: "Edge Router"
        interfaces:
          - name: "GigabitEthernet0/0"
            type: "1000base-t"
            description: "Uplink to backbone"
            status__name: "Planned"
      - name: "LWM1-LR2"
        status__name: "Planned"
        device_type__model: "C8300-1N1S-6T"
        device_role__name: "Edge Router"      
        interfaces:
          - name: "GigabitEthernet0/0"
            type: "1000base-t"
            description: "Uplink to backbone"
            status__name: "Planned"

circuits:
  - cid: "LWM1-CKT-1"
    status__name: "Planned"
    provider__name: "NTC"
    type__name: "Ethernet"
    terminations:
      - term_side: "A"
        site__name: "LWM1"
      - term_side: "Z"
        provider_network__name: "NTC-WAN"

  - cid: "LWM1-CKT-2"
    status__name: "Planned"
    provider__name: "NTC"
    type__name: "Ethernet"
    terminations:
      - term_side: "A"
        site__name: "LWM1"
      - term_side: "Z"
        provider_network__name: "NTC-WAN"

This is still quite a bit of information to write. Luckily, the Design Builder application can consume Jinja templates to produce the design files. Using some Jinja templating, we can reduce the above design a bit:

---
sites:
  - name: "LWM1"
    status__name: "Staging"
    prefixes:
      - prefix: "10.37.27.0/24"
        status__name: "Reserved"
    devices:
    {% for i in range(2) %}
      - name: "LWM1-LR{{ i }}"
        status__name: "Planned"
        device_type__model: "C8300-1N1S-6T"
        device_role__name: "Edge Router"
        interfaces:
          - name: "GigabitEthernet0/0"
            type: "1000base-t"
            description: "Uplink to backbone"
            status__name: "Planned"
    {% endfor %}
circuits:
  {% for i in range(2) %}
  - cid: "LWM1-CKT-{{ i }}"
    status__name: "Planned"
    provider__name: "NTC"
    type__name: "Ethernet"
    terminations:
      - term_side: "A"
        site__name: "LWM1"
      - term_side: "Z"
        provider_network__name: "NTC-WAN"
  {% endfor %}

The above design file gets closer to a re-usable design. It has reduced the amount of information we have to represent by leveraging Jinja2 control structures, but there is still statically defined information. At the moment, the design includes hard coded site information (for the site name, device names and circuit IDs) as well as a hard coded IP prefix. Design Builder also provides a way for this information to be gathered dynamically. Fundamentally, all designs are just Nautobot Jobs. Therefore, a design Job can include user-supplied vars that are then copied into the Jinja2 render context. Consider the design job for our edge site design:

class EdgeDesign(DesignJob):
    """A basic design for design builder."""
    site_name = StringVar(label="Site Name", regex=r"\w{3}\d+")
    site_prefix = IPNetworkVar(label="Site Prefix")

#...

This design Job collects a site_name variable as well as a site_prefix variable from the user. Users provide values for these variables through the normal Job launch entrypoint:

Once the job has been launched, the Design Builder will provide these input variables to the Jinja rendering context. The variable names, within the jinja2 template, will match the attribute names used in the Design Job class. With the site_name and site_prefix variables now being defined dynamically, we can produce a final design document using them:

---

sites:
  - name: "{{ site_name }}"
    status__name: "Staging"
    prefixes:
      - prefix: "{{ site_prefix }}"
        status__name: "Reserved"
    devices:
    {% for i in range(2) %}
      - name: "{{ site_name }}-LR{{ i }}"
        status__name: "Planned"
        device_type__model: "C8300-1N1S-6T"
        device_role__name: "Edge Router"
        interfaces:
          - name: "GigabitEthernet0/0"
            type: "1000base-t"
            description: "Uplink to backbone"
            status__name: "Planned"
    {% endfor %}
circuits:
  {% for i in range(2) %}
  - cid: "{{ site_name }}-CKT-{{ i }}"
    status__name: "Planned"
    provider__name: "NTC"
    type__name: "Ethernet"
    terminations:
      - term_side: "A"
        site__name: "{{ site_name }}"
      - term_side: "Z"
        provider_network__name: "NTC-WAN"
  {% endfor %}

The design render context is actually much more flexible than simple user entry via script vars. Design Builder provides a complete system for managing the render context, including loading variables from YAML files and providing dynamic content via Python code. The official documentation covers all of the capabilities of the design context.

In addition to the YAML rendering capabilities, Design Builder includes a way to perform just-in-time operations while creating and updating Nautobot objects. For instance, in the above example, the site prefix is specified by the user that launches the job. It may be desirable for this prefix to be auto-assigned and provisioned out of a larger parent prefix. Design Builder provides a means to perform these just-in-time lookups and calculations in the form of something called an “action tag”. Action tags are evaluated during the object creation phase of a design’s implementation. That means that database lookups can occur and computations can take place as the design is being implemented. One of the provided action tags is the next_prefix action tag. This tag accepts query parameters to find a parent prefix, and also a parameter that specifies the length of the required new prefix. For example, if we want to provision a /24 prefix from the 10.0.0.0/16 parent, we could use the following:

prefixes:
  - "!next_prefix":
      prefix: "10.0.0.0/16"
      length: 24
    status__name: "Active"

The next_prefix action tag will find the parent prefix 10.0.00/16 and look for the first available /24 in that parent. Once found, Design Builder will create that child prefix with the status Active.

Several action tags are provided out of the box, but one of the most powerful features of Design Builder is the ability to include custom action tags in a design. Action tags are implemented in Python as specialized classes, and can perform any operation necessary to produce a just-in-time result.

There is quite a lot to understand with Design Builder, and we have only touched on a few of its capabilities. While there are several moving parts, the following diagram illustrates the high-level process that the Design Builder application uses to go from design files and templates to an implemented design.

Design Builder starts with some optional input variables from the Nautobot job and combines them with optional context variables written either in YAML or Python or both. This render context is used by the Jinja2 renderer to resolve variable names in Jinja2 templates. The Jinja2 templates are rendered into YAML documents that are unmarshaled as Python dictionaries and provided to the Builder. The Builder iterates all of the objects in this dictionary and performs necessary database creations and updates. In the process of creating and updating objects, any action tags that are present are evaluated. The final result is a set of objects in Nautobot that have been created or updated by Design Builder.

Roadmap

Our plans for Design Builder are far from over. There are many more features we’re currently working on, as well as some that are still in the planning stages. Some of the near-term features include design lifecycle and object protection.

The design lifecycle feature allows the implementations of a design to be tracked. Design instances can be created (such as an instance of the edge site design above) and can be subsequently decommissioned. Objects that belong to a design instance will be reverted to their state prior to the design implementation, or they may be removed entirely (if created specifically for a design). Designs can also track inter-design dependencies so that a design cannot be decommissioned if other design instances depend on it. The design lifecycle feature will also allow designs to be versioned so that an implementation can be updated over time.

The ability to protect objects that belong to a design is also planned. The idea is that if an object is created as part of a design implementation, any attributes that were initially set in this design cannot be updated outside of that design’s lifecycle. This object protection assures that our source of truth has data that complies with a design and prevents manually introduced errors.