Advanced Options for Building a Nautobot SSoT App

In the first part of this series, we reviewed the building blocks of an SSoT app for Nautobot. We reviewed the design of DiffSyncModel classes, the CRUD methods on those classes, building your System of Record adapters to fill those models, and finally the Nautobot Job that executes the synchronization of data between your Systems of Record. In this second half, we’ll review advanced options available to you when architecting an SSoT app like controlling the order of processing for your data and handling special requirements for object deletion.

Please note: it is expected that you’ve read the Nautobot Plugin: Single Source of Truth (SSoT) and Building a Nautobot SSoT App posts and understand the framework terminology, such as Data Source and Data Target.

In the designing of your SSoT application you might find yourself in a situation where you want to define the processing order of your SoR objects. The standard method of processing a set of objects in the DiffSync process can’t be guaranteed but is typically a simple first in, first out queue defined by the order the objects were presented by your adapters. To change this behavior you can extend the Diff class itself and define the processing order. One option might be to process each class of your objects alphabetically, as shown in the example below:

from collections import defaultdict
from diffsync.diff import Diff


class CustomOrderingDiff(Diff):
    """Alternate diff class to list children in alphabetical order, except devices to be ordered by CRUD action."""

    @classmethod
    def order_children_default(cls, children):
        """Simple diff to return all children in alphabetical order."""
        for child_name, _ in sorted(children.items()):
            yield children[child_name]

In some cases you wish to have this done for a single object type, like Devices. This can be done by having a method in your custom Diff class named after the type of the object in the pattern order_children_<type>. It will utilize the order_children_default method for any other object classes that haven’t been explicitly defined. This option also allows you to control the order of CRUD operations that happen on a particular object, as shown in the example below:

    @classmethod
    def order_children_device(cls, children):
        """Return a list of device sorted by CRUD action, starting with deletion, then create, and update, along with being in alphabetical order."""
        children_by_type = defaultdict(list)

        # Organize the children's name by action create, update, or delete
        for child_name, child in children.items():
            action = child.action or "skip"
            children_by_type[action].append(child_name)

        # Create a global list, organized per action, with deletion first to prevent conflicts
        sorted_children = sorted(children_by_type["delete"])
        sorted_children += sorted(children_by_type["create"])
        sorted_children += sorted(children_by_type["update"])
        sorted_children += sorted(children_by_type["skip"])

        for name in sorted_children:
            yield children[name]

Once you’ve defined your custom Diff ordering class you simply need to pass it to the appropriate diff_from/diff_to or sync_from/sync_to methods, as shown below:

from diffsync import DiffSyncFlags
from diffsync.exceptions import ObjectNotCreated

    def sync_data(self):
        """SSoT synchronization from Device42 to Nautobot."""
        client = Device42API()
        d42_adapter = Device42Adapter(job=self, sync=self.sync, client=client)
        d42_adapter.load()
        nb_adapter = NautobotAdapter(job=self, sync=self.sync)
        nb_adapter.load()
        diff = nb_adapter.diff_from(d42_adapter, diff_class=CustomOrderingDiff)
        if not self.kwargs["dry_run"]:
            try:
                nb_adapter.sync_from(d42_adapter, diff_class=CustomOrderingDiff)
            except ObjectNotCreated as err:
                self.log_debug(message=f"Unable to create object. {err}")
            self.log_success(message="Sync complete.")

Custom Diff classes can come in handy when you need to ensure that an obsolete version of an object has been removed before a newer version being installed to prevent possible conflicts.

In addition to controlling the flow of your object processing, you might have a situation where the synchronization fails or you only want to consider objects that exist in one of or both of your Systems of Record. In these cases you would want to utilize a DiffSync Flag. The core DiffSync engine provides two sets of flags, allowing for modifying behavior of DiffSync at either the Global or Model level. As the name implies, global flags apply to all data and while model flags apply to a specific model. A list of the included global flag options (as of DiffSync 1.3) has been provided below:

Like your custom Diff ordering class, utilizing the global flags simply requires applying them to the appropriate diff and sync methods in your Job, as below:

from diffsync.enum import DiffSyncFlags
flags = DiffSyncFlags.CONTINUE_ON_FAILURE
diff = nb_adapter.diff_from(d42_adapter, diff_class=CustomOrderingDiff, flags=flags)

Model flags are applied to individual DiffSyncModel instances, for example, you could apply them from the adapter’s load method, as shown in the example below:

from diffsync import DiffSync
from diffsync.enum import DiffSyncModelFlags
from models import Device

class NSOAdapter(DiffSync):

    device = Device

    def load(self, data):
        """Load all devices into the adapter and add the flag IGNORE to all non-ACI devices."""
        for device in data.get("devices"):
            obj = self.device(name=device["name"])
            if "ACI" not in device["name"]:
                obj.model_flags = DiffSyncModelFlags.IGNORE
            self.add(obj)

The DiffSync library, as of version 1.3, currently includes two options for Model Flags, as shown in the table below:

Both global flags and model flags are stored as a binary representation. This allows for storage of multiple flags within a single variable and allows for additional flags to be added in the future. Due to the nature of each flag being a different binary value it is necessary to perform a bitwise OR operation when utilizing multiple flags at once. Imagine the scenario where you want to skip objects that don’t exist in either Systems of Record and log all object records regardless of their being changed. You would first need to define one flag and then perform the bitwise OR operation, as shown in the example:

>>> from diffsync.enum import DiffSyncFlags
>>> flags = DiffSyncFlags.SKIP_UNMATCHED_BOTH
>>> flags
<DiffSyncFlags.SKIP_UNMATCHED_BOTH: 6>
>>> bin(flags.value)
'0b110'
>>> flags |= DiffSyncFlags.LOG_UNCHANGED_RECORDS
>>> flags
<DiffSyncFlags.LOG_UNCHANGED_RECORDS|SKIP_UNMATCHED_BOTH|SKIP_UNMATCHED_DST|SKIP_UNMATCHED_SRC: 14>
>>> bin(flags.value)
>>> '0b1110'

Now that you’ve defined exactly how you want your SSoT application to handle the data from your Systems of Record you might have a requirement to perform some action on the data once the sync has completed. Luckily, the SSoT app makes this easy by looking for a sync_complete method in your DataTarget adapter and running it if found. A case where this could be used is one where deletion of objects in your Systems of Record needs to be handled in a specific manner due to inter-object dependence. An example of this would be something like a Site in Nautobot that can’t be deleted until all devices, racks, and other objects in that Site have been deleted or moved. To perform this operation you would need to define your object’s delete method, as below:

def delete(self):
    """Delete Site object from Nautobot.

    Because Site has a direct relationship with many other objects, it can't be deleted before anything else.
    The self.diffsync.objects_to_delete dictionary stores all objects for deletion and removes them from Nautobot
    in the correct order. This is used in the Nautobot adapter sync_complete function.
    """
    self.diffsync.job.log_warning(message=f"Site {self.name} will be deleted.")
    super().delete()
    site = Site.objects.get(id=self.uuid)
    self.diffsync.objects_to_delete["site"].append(site)  # pylint: disable=protected-access
    return self

The delete method is marking the object as deleted, but instead of deleting it immediately from Nautobot’s database, it is adding it to a list of objects to be removed once the synchronization has completed and the appropriate order of deleting objects can be performed, as shown in the following example:

from diffsync import DiffSync
from django.db.models import ProtectedError

class NautobotAdapter(DiffSync):
    """Nautobot adapter for DiffSync."""

    objects_to_delete = defaultdict(list)

    def sync_complete(self, source: DiffSync, *args, **kwargs):
        """Clean up function for DiffSync sync.

        Once the sync is complete, this function runs deleting any objects
        from Nautobot that need to be deleted in a specific order.

        Args:
            source (DiffSync): DiffSync
        """
        for grouping in (
            "ipaddr",
            "subnet",
            "vrf",
            "vlan",
            "cluster",
            "port",
            "device",
            "device_type",
            "manufacturer",
            "rack",
            "site",  # can't delete a site until all of its dependent objects, above, have been deleted
        ):
            for nautobot_object in self.objects_to_delete[grouping]:
                try:
                    nautobot_object.delete()
                except ProtectedError:
                    self.job.log_failure(obj=nautobot_object, message="Deletion failed protected object")
            self.objects_to_delete[grouping] = []

        return super().sync_complete(source, *args, **kwargs)

Just be aware that any changes made to your Systems of Record through the sync_complete method should be ones that won’t impact the data sets between your Systems of Record. This is essential to minimize unnecessary updates on subsequent runs of your sync Job. An example of this would be performing a DNS query of your devices and creating IP Addresses and interfaces on devices after the sync is complete. Doing so would cause these same IP Addresses and interfaces to be absent in the comparison of your Systems of Record and would thus be deleted and then re-added again after the sync finished. This would cause a repeated cycle of objects being removed and re-added. The best practice is to have any manipulation of your data sets in your Systems of Record performed within your adapters and prior to the diff and sync are performed.