Nautobot SSoT Contrib – Intro to SSoT Contrib

Nautobot is a powerful tool for driving data-driven network automation. It is an enterprise-grade network Source of Truth (SoT) built on Django and is both extensible and flexible.

Enterprise networks, as mentioned in a previous post, do not always contain truthful data in a single location and are often spread across multiple systems called Systems of Record (SoRs). Each system contains distinct sets of information about the network which can provide its own set of challenges during automation efforts.

The Nautobot Single Source of Truth (SSoT) application addresses those challenges by providing the ability to programmatically aggregate and enrich network data from SoRs into Nautobot. Each implementation of SSoT to sync data from a system is called an integration.

The application comes with several integrations out of the box, but many organizations require integrations not provided in the General Availability application. Instead, they must create a custom application extending the base SSoT application to meet their needs.

This is the first post in a series about creating custom SSoT integrations using the contrib submodule. In this post, we’ll be discussing the basics of integrations as well as an intro to Contrib and what challenges it solves.

The examples in this and future posts will be focused on an integration from an external source syncing to your own instance of Nautobot. For more information on building an integration for the source data, read Building a Nautobot SSoT App by Justin Drew. It also provides good foundational information which would be useful for this series.

For simplicity and for readers to follow along with the same data, the System of Record used will be a CSV file. It’s important to note, however, that CSV files do not make good SoRs due to the challenges of maintaining them and the risks of incompatible versions.

The CSV file used in the example will contain a list of locations and several attributes to sync, including standard attributes, standard relationships, and custom fields.

Elements of an SSoT Integration

SSoT integrations are Nautobot Jobs and additional code containing the processes for interacting with and syncing source and target systems. Custom integrations are done in a separate Nautobot application.

It is important to have a basic understanding of the existing process for creating custom SSoT integrations before diving into the details of the contrib module. The relevant elements are included in this post, but a previous post, Building a Nautobot SSoT App, can provide additional details not included here.

The key elements we will discuss are:

• Systems of Record
• DiffSync Library
  • DiffSync Models
  • DiffSync Adapters
• SSoT Job

Systems of Record

Most enterprise networks don’t have all their data in a single location. Instead, various systems may contain different data points or even fragments of data.

A System of Record (SoR) is a system that is considered “authoritative” for a certain data set, which means the data it holds is “truthful” about the data set even if the data is duplicated in other systems. For example, ServiceNow might contain device inventory data while InfoBlox contains IPAM data. Each system is considered an SoR, as each contains a section of authoritative data about the network.

The important takeaway is that an SoR must contain “authoritative data” about the network for it to be synchronized into Nautobot as the Single Source of Truth, otherwise Nautobot can’t fulfill its role as the Source of Truth.

DiffSync Library (Legacy Method without Contrib)

The DiffSync library is the heart of the SSoT synchronization process. It is an open source library created and maintained by Network to Code and is used to compare and synchronize different datasets.

There are two primary components of the DiffSync library used in SSoT synchronization: models and adapters.

DiffSync Models

DiffSync Models are created by extending the DiffSyncModel class with required configurations and logic. They represent the target data format and how to perform create, update, and delete operations on the target system. Any class inheriting from DiffSyncModel includes:

• _modelname is a string name of the model.
• _identifiers is a tuple containing what fields of the model represent a unique combination for an entry. 
• _attributes is also a tuple representing additional fields to sync with the target system.
• Each entry in _identifiers and _attributes must also be represented as an attribute in the class with type hinting.
• Methods for creating, updating, and deleting objects from the target system.

including identifiers to represent unique entries along with any additional attributes to sync. Attributes not explicitly configured are ignored. 

Creating a simple DiffSync model for Nautobot Location() objects without using contrib can look something like this:

from diffsync import DiffSyncModel

class LocationModel(DiffSyncModel):
    _modelname = "location"
    _identifiers = ("name",)
    _attributes = (
        "location_type__name",
        "status__name",
    )

    # _identifiers and _attributes as class attributes with type hinting.
    name: str
    location_type__name: str
    status__name: str

    # Create, update, and delete methods requred when not using contrib
    @classmethod
    def create(cls, diffsync, ids, attrs):
        ...

    def update(self, attrs):
        ...

    def delete(self):
        ...

DiffSync Adapters

DiffSync adapters are responsible for programmatically connecting with the source and target systems to collect and load data conforming to the data modeling done in the previous step. Each entry is saved as a separate instance of the associated model in a predetermined variable named store.

A separate adapter is required for source and target systems, and each adapter inherits from diffsync.Adapter configured with several required elements:

• Class Attributes:
  • An attribute for each model used in the SSoT sync.
  • top_level – A list of top level model names as defined by the _modelname attribute from the DiffSync models.
• def load(self) – A required method called by the Nautobot Job and entry point for loading data from the source or target system.

A simple adapter for a target system of Nautobot would look something like this:

from DiffSync import Adapter
from myssotapp.myintegration.models.nautobot import LocationModel


class NautobotTargetAdapter(Adapter):
    location = LocationModel

    top_level = [
        "location",
    ]
 
    def load(self):
        # Logic for connecting to and retrieving information from system
        data = [
            {"name": "My Location", "type": "Basic Site", "status": "Active"},
        ]

        for row in data:
            self.add(self.location(
                name=row["name"],
                location_type__name=row["type"],
                status__name=row["status"],
            ))

An adapter for the source system would look similar as well, but we will only be looking at the target adapter in this example as it relates to Nautobot.

Nautobot SSoT Job

The SSoT job acts as a controller between the previous elements of an integration. 

Nautobot jobs are a way for users to execute custom logic on demand from the Nautobot UI. The SSoT job essentially acts as a controller where it calls and executes code from the created DiffSync adapters and models.

Most Nautobot jobs inherit a single class from nautobot.extras.jobs.Job. SSoT jobs, on the other hand, inherit the Job object through SSoT specific base classes:

• nautobot_ssot.jobs.base.DataSource – When synchronizing from a source system into Nautobot
• nautobot_ssot.jobs.base.DataTarget – When synchronizing from Nautobot to another target

Each SSoT job class must have some basic components to configure:

• name attribute at the file level for grouping jobs together in the Nautobot Jobs UI
• Meta class with the following attributes (only name is required, but the others are helpful for effective documentation):
  • name – Name of the SSoT job
  • description – Description of the SSoT job
  • data_source – Source location of the data to sync
  • data_target – Target location to sync data to
• Required Methods

• load_source_adapter(self) – Instantiates source adapter and calls load() method
• load_target_adapter(self) – Instantiates target adapter and calls load() method

This example SSoT job is using an external source syncing to Nautobot:

from nautobot_ssot.jobs import DataSource
from myssotapp.myintegration.adapters import CSVSourceAdapter, NautobotTargetAdapter

# Name attribute for grouping jobs in Nautobot UI
name = "My SSoT Jobs"


class SyncCSVToNautobot(DataSource):
    class Meta:  # pylint: disable=too-few-public-methods
        name = "Sync Locations -> Nautobot"
        description = "A simple SSoT job."
        data_source = "CSV File"
        data_target = "Nautobot (local)"

    def load_source_adapter(self):
        self.source_adapter = CSVSourceAdapter(job=self)
        self.source_adapter.load()

    def load_target_adapter(self):
        self.target_adapter = NautobotTargetAdapter(job=self)
        self.target_adapter.load()

Once the job has loaded all relevant data from the adapters, it runs a diff between the source and target by comparing the created models to determine what actions need to be taken to ensure the target is in sync with the source.

Intro to SSoT Contrib (New Method)

One main challenge in custom SSoT integrations is the need to create several versions of Nautobot models and adapters to interact with the Nautobot infrastructure. It wouldn’t be uncommon to have similar code between projects in their CRUD operations.

The nautobot_ssot.contrib submodule addresses this issue by providing reusable code to perform CRUD operations with the local Nautobot system. This means you no longer need to code each CRUD action for each model and adapter, saving development time.

The primary changes are within the defined models and NautobotTargetAdapter while the SSoT job has no changes when using contrib.

Model Changes

Instead of inheriting from DiffSyncModel, we will now inherit from a new class provided by the Nautobot SSoT application at nautobot_ssot.contrib.NautobotModel. This new class provides the required create, update, and delete methods so you no longer need to provide them yourself.

A new required attribute, _model, is added which must directly reference the Nautobot class object.

from nautobot.dcim.models import Location
from nautobot_ssot.contrib.model import NautobotModel

class LocationModel(NautobotModel):
    _model = Location  # New attribute, direct reference to `Location` class
    _modelname = "location"
    _identifiers = ("name",)
    _attributes = (
        "location_type__name",
        "status__name",
    )

    name: str
    location_type__name: str
    status__name: str

    # create(), update(), and delete() provided by parent class

Adapter Changes

Much like the model, contrib provides the required background methods to interact with Nautobot to retrieve data and load it into the DiffSync models. We will inherit from nautobot_ssot.contrib.NautobotAdapter and only provide basic configurations for the class attributes as shown below. 

from nautobot_ssot.contrib.adapter import NautobotAdapter
from myssotapp.diffsync.models import LocationModel


class NautobotAdapter(NautobotAdapter):
    location = LocationModel

    top_level = [
        "location",
    ]

    # load() and all required methods for interacting with Nautobot provided by contrib