Fresh Updates for Data Validation Engine

A fresh release brings several exciting new features to the Data Validation Engine Nautobot app!

For those who have not yet used the Data Validation Engine app in their Nautobot deployment, I’ll start with a quick overview. Data Validation Engine (I’ll refer to it as ‘DVE’ for short) provides users with a simple, customizable way to define and enforce business rules on their data within Nautobot. Rather than having a limited set of predetermined constraints that are hardcoded into Nautobot’s source code, DVE allows you to define what rules are important and create any data boundaries you’d like right within the Nautobot UI. No programming or coding required! This also enables the capability to set additional validation rules on top of your own database constraints. The rule validation applies to the Nautobot GUI, API, and even the ORM when validated_save is called.

Say your enterprise has standards for how a new device should be named. For example, the device name must follow a specific pattern like aaa00-descriptor-01, where aaa is a three-character location, 00 is an optional location number, descriptor is some text for the device of varying length, and 01 is the unit number (e.g., nyc00-rtr-03 or sfo01-fw-01). With DVE, this requirement can be easily created and enforced whenever someone goes to create a new device. With Nautobot as your company’s Source of Truth (SoT), the importance of clean and consistent data to achieving network automation cannot be overstated! The Data Validation Engine app provides a simple, user-friendly way to ensure data that is added meets all your standards.

In version 1.0 of Data Validation Engine, we introduced two rule types—Min/Max Rules and Regular Expression Rules. You can read more about these and see additional examples in the app’s User Guide and the Data Validation Engine introductory blog post.

In the most recent 2.X releases, we added:

• Required Rules
• Unique Rules
• Data Compliance

Required Rules

Required Rules allow users to make specific fields within data objects required in just a few simple clicks. Imagine your manager wants to ensure that all devices being added to Nautobot have a serial number. By default within the Device object, the serial number field is optional. However, with the extensibility power that the Nautobot app ecosystem offers, you can install Data Validation Engine and create a Required Rule that enforces this custom requirement.

The DVE rules are found under the ‘Extensibility’ dropdown, where you can click to add a new Required Rule.

We’ll name this rule Device Serial Number, select the dcim | device content type, and enter serial as the field that we want to make required. We’ll also add a helpful error message, should the user not add a value.

After clicking “Create”, we can now see our newly added Required Rule.

Let’s try it out! We’ll go add a new device and will leave the Serial number field empty when we click “Create”. And, just like that, we’ve successfully made the field required and get our helpful reminder to add the serial number!

The DVE rules are also enforced in the use of the Nautobot API, in addition to the GUI. With the Device Serial Number required rule enabled, if we try to send this API request without including a value for serial number:

POST /api/dcim/devices/ HTTP/1.1
Host: localhost:8080
Content-Type: application/json
Authorization: Token 0123456789abcdef0123456789abcdef01234567

{
    "name": "ams01-leaf-09",
    "device_type": {
        "model": "DCS-7150S-24"
    },
    "device_role": {
        "name": "leaf"
    },
    "site": {
        "name": "AMS01"
    },
    "status": "Active"
}

The following 400: BAD REQUEST response is returned:

{
    "serial": [
        "Please enter the serial number for the device!"
    ]
}

Unique Rules

Unique Rules enable you to enforce uniqueness on object fields of your choice. Let’s continue with the example from before. Pretend your manager also wants to enforce data validation that prevents duplicate device serial numbers within Nautobot.

Under the ‘Extensibility’ dropdown again, we’ll add a new Unique Rule.

Let’s call this Unique Serial Number. It will again be applied to the dcim | device content type and the serial field. We’ll set the Max instances value to 1, but it’s important to note that this can be modified for your specific implementation. For example, you could create a Unique Rule that limits the allowed occurrences of a specific value for a field to no more than 3 or whatever number you’d like. We’ll then add an optional error message and click “Create”.

Also, please note that although these examples apply rules to the dcim | device serial number field, the available object/field options you can create rules for are not limited.

Now that we have our Unique Serial Number rule created, we can test it. I added the serial number SN123ABC to an existing device and will now try adding that same value to another one.

As expected, we’re prevented from adding a duplicate serial number thanks to our Unique Serial Number rule!

Data Compliance

One of the more exciting features that was released in Data Validation Engine v2.1.0 is Data Compliance. Data Compliance can audit any object within Nautobot according to a set of rules that you can define programmatically with simple, straightforward Python code. Unlike the other rule types within DVE that only check for adherence to specified rules during the creation or modification of objects, Data Compliance will run a job that produces compliance statuses across all objects, including already existing ones (such as all preexisting devices).

This is ideal for implementing some kind of business logic or standardization requirement into Nautobot after data is already populated within the platform. Data Compliance will allow you to identify valid or invalid objects based on your specified data compliance rules. Additionally, Data Compliance has the capability to implement more complex rules using the full power of programming logic. For the technical details of how it works, check out this section in the docs.

Let’s walk through a step-by-step example of how you can implement and use it.

Step 1. Create the Data Compliance Rules

First, you’ll want to create your desired data compliance rules following whatever programming logic you want. These compliance rules will be included as methods within one or more classes that implement DataComplianceRule.

For our example, let’s create these two rules that check devices for the following:

• audit_device_name_chars – will mark a device invalid if the device name contains any special characters other than a dash (-), underscore (_), or period (.)
• audit_device_rack – will mark a device invalid if it is not assigned to a rack

There are two options on where to include the data compliance rule classes:

1. In a remote Git repository (recommended)
2. In the app’s code

Writing Data Compliance Rules in a Remote Git Repository

A Git repository can be configured to add the data compliance rules context to store DataComplianceRule classes in source control. The app looks for a folder in your remote repo called custom_validators, and any Python files within that folder containing classes that implement DataComplianceRule will be imported. No code within the app itself needs to be added, changed, or removed.

Below is the example data compliance rule class that I’ll store as custom_validators/my_data_compliance_rules.py in my remote Git repository named dve-datacompliance-demo:

import re
from nautobot_data_validation_engine.custom_validators import DataComplianceRule, ComplianceError

class DeviceDataComplianceRules(DataComplianceRule):
    model = "dcim.device"
    enforce = False
    
    # Checks if a device name contains any special characters other than a dash (-), underscore (_), or period (.) using regex
    def audit_device_name_chars(self):
        if not re.match("^[a-zA-Z0-9\-_.]+$", self.context["object"].name):
            raise ComplianceError({"name": "Device name contains unallowed special characters."})
    
    # Checks if a device is not assigned to a rack
    def audit_device_rack(self):
        if not self.context["object"].rack:
            raise ComplianceError({"rack": "Device should be assigned to a rack."})
    
    def audit(self):
        messages = {}
        for fn in [self.audit_device_name_chars, self.audit_device_rack]:
            try:
                fn()
            except ComplianceError as ex:
                messages.update(ex.message_dict)
        if messages:
            raise ComplianceError(messages)

After the Git repo is configured and rule class(es) written, add the repository to Nautobot from Extensibility -> Data Sources -> Git Repositories. Include the remote repo URL, as well as credentials if it’s not public (recommend using Nautobot Secrets for this). Also select data compliance rules for the ‘provides’ field. Then, we just use Nautobot’s Git Repositories Sync feature, and our Data Compliance rules will be automatically imported.

Writing Data Compliance Rules within the App

To write data compliance rules within the app’s code itself, add the classes that implement DataComplianceRule within nautobot_data_validation_engine/custom_validators.py.

You can use the same code snippet from above, and the only additional task will be to modify the existing custom_validators variable by casting CustomValidatorIterator() to a list and then appending the classes to it:

custom_validators = list(CustomValidatorIterator()) + [DeviceDataComplianceRules]

Step 2. Run theRunRegisteredDataComplianceRulesJob

We’ll now go to Nautobot Jobs and run the RunRegisteredDataComplianceRules job. In the pre-job settings, you can select the individual data compliance rule classes you’d like to run at that time. Otherwise, not selecting/highlighting any will default to running them all.

The job can be used to run the audit method for any number of registered DataComplianceRule classes in an ad hoc fashion. This can be used to run the data compliance rules for the first time over a set of objects or to rerun the rules after an update to the compliance logic.

Step 3. Viewing Data Compliance Results

All data compliance result objects can be found on the navigation bar under Extensibility -> Data Validation Engine -> Data Compliance. This view lists all available data compliance results produced from the RunRegisteredDataComplianceRules job.

You can add filters such as showing only invalid objects or only ones from a specific compliance rule class. Let’s filter on only the objects that are out of compliance.

Additionally, the nautobot_data_validation_engine app automatically creates template extensions to add a Data Compliance tab to the detail view of all objects. This tab makes it easy to check an individual object’s compliance with any applicable data compliance rules.

We can see that this device has an invalid name and is not assigned to a rack. Let’s fix these two things. After editing the device to correct the noncompliance, it’s automatically rechecked and is now valid and in compliance:

Note: A second job, DeleteOrphanedDataComplianceData, associated with Data Compliance can be run to remove/clean up any data compliance results that might be left dangling over time due to the parent object having since been deleted.