Writing Your First Nautobot Job, Pt.1

Welcome to our blog series “Writing Your First Nautobot Job.” The goal of this series is to provide Nautobot users with everything they need to start writing Nautobot Jobs from scratch. Now, we assume you have a basic understanding of the Python programming language and you have Nautobot up and running in your environment.

The first entry in this series will review the Django ORM and Django data models, which are fundamental to understanding how data is managed in Nautobot. This entry will be followed by a deep dive into the inner workings of the Nautobot Job framework to better our understanding of how Jobs are constructed and executed. The final entry in this series will be a step-by-step guide to writing your first (and useful) Nautobot Job.

Introduction

When starting a new project or doing something for the very first time, taking that first step is always difficult. We typically don’t know how to get started and sometimes we don’t even know where to look to find guidance. The objective of this blog series is to lower the barrier of entry and help guide you through the essentials needed for writing your first Nautobot Job. This initial entry in our series essentially serves as a primer so we can review some foundational topics that need to be understood before we start writing our first Nautobot Job.

What are Nautobot Jobs anyway? Nautobot provides a Job framework that we can use to execute a task or a set of tasks on demand or at a scheduled time. Nautobot Jobs can be used to execute any custom logic with direct access to the Source of Truth data using an object-based permissions framework. All changes can be captured in the change log.

A valid use case might be that your Nautobot Job could take some inputs from the user, manipulate it in some way, and then display some output in the Job logs. The more interesting use cases often involve interacting directly with Nautobot data or even external systems to accomplish various data creation, modification, and validation tasks. However, before we dive into how we do this within our Job, we need to know where we can practice interacting with the Nautobot data.

Nautobot’s Python Shells

Nautobot includes a command-line (CLI) management utility called nautobot-server, which includes various tools to assist with common administrative tasks. Today we’re going to focus only on the two Python shells that are included, but all of the commands are documented in detail in the administration guide. First tool we’re going to take a look at is the Nautobot shell or nbshell. The nbshell is a lightly customized version of the built-in Django shell with all of the relevant Nautobot models preloaded. This means we have a Python environment where we don’t have to import any of the Django models defined in the Nautobot source code and we can access the data in Nautobot directly. We can use this shell to practice interacting with data in Nautobot or test some of the logic in our Job the same way we test the logic of our Python code in the Python shell. Additional documentation about the Nautobot shell can be found here. Thanks to the django-extensions package, we have access to a second and slightly more advanced Python shell called shell-plus. This shell preloads all of the models across all installed plugins as well as the core Nautobot models. The shell also includes the ability to use several different types of interactive Python shells such as IPython, bpython, and ptpython. Additional options and configurations can be found in the official django-extensions documentation.

These Python shells afford users direct access to Nautobot data and function with very little validation in place. As such, it is crucial to ensure that only authorized, knowledgeable users are ever granted access to it. Never perform any action in the management shell without having a full backup in place. You can find a full guide on replicating Nautobot’s database here.

To enter the shell, you need to be sure you are in the same execution environment that Nautobot is running in, and then run one of the following commands:

nautobot-server nbshell

or

nautobot-server shell_plus

Note: Depending on your deployment, you may need to use the nautobot user and activate the virtual environment in the /opt/nautobot directory.

I’m going to be using shell_plus for the examples throughout this blog, so let’s jump into the shell and take a look at it.

user@host % invoke shell-plus
# Shell Plus Model Imports
from constance.backends.database.models import Constance
from django.contrib.admin.models import LogEntry
from django.contrib.auth.models import Group, Permission
from django.contrib.contenttypes.models import ContentType
from django.contrib.sessions.models import Session
from django_celery_beat.models import ClockedSchedule, CrontabSchedule, IntervalSchedule, PeriodicTask, PeriodicTasks, SolarSchedule
from django_rq.models import Queue
from nautobot.circuits.models import Circuit, CircuitTermination, CircuitType, Provider, ProviderNetwork
from nautobot.dcim.models.cables import Cable, CablePath
from nautobot.dcim.models.device_component_templates import ConsolePortTemplate, ConsoleServerPortTemplate, DeviceBayTemplate, FrontPortTemplate, InterfaceTemplate, PowerOutletTemplate, PowerPortTemplate, RearPortTemplate
from nautobot.dcim.models.device_components import ConsolePort, ConsoleServerPort, DeviceBay, FrontPort, Interface, InventoryItem, PowerOutlet, PowerPort, RearPort
from nautobot.dcim.models.devices import Device, DeviceRedundancyGroup, DeviceRole, DeviceType, Manufacturer, Platform, VirtualChassis
from nautobot.dcim.models.locations import Location, LocationType
from nautobot.dcim.models.power import PowerFeed, PowerPanel
from nautobot.dcim.models.racks import Rack, RackGroup, RackReservation, RackRole
from nautobot.dcim.models.sites import Region, Site
from nautobot.extras.models.change_logging import ObjectChange
from nautobot.extras.models.customfields import ComputedField, CustomField, CustomFieldChoice
from nautobot.extras.models.datasources import GitRepository
from nautobot.extras.models.groups import DynamicGroup, DynamicGroupMembership
from nautobot.extras.models.jobs import Job, JobButton, JobHook, JobLogEntry, JobResult, ScheduledJob, ScheduledJobs
from nautobot.extras.models.models import ConfigContext, ConfigContextSchema, CustomLink, ExportTemplate, FileAttachment, FileProxy, GraphQLQuery, HealthCheckTestModel, ImageAttachment, Note, Webhook
from nautobot.extras.models.relationships import Relationship, RelationshipAssociation
from nautobot.extras.models.secrets import Secret, SecretsGroup, SecretsGroupAssociation
from nautobot.extras.models.statuses import Status
from nautobot.extras.models.tags import Tag, TaggedItem
from nautobot.ipam.models import Aggregate, IPAddress, Prefix, RIR, Role, RouteTarget, Service, VLAN, VLANGroup, VRF
from nautobot.tenancy.models import Tenant, TenantGroup
from nautobot.users.models import AdminGroup, ObjectPermission, Token, User
from nautobot.virtualization.models import Cluster, ClusterGroup, ClusterType, VMInterface, VirtualMachine
from social_django.models import Association, Code, Nonce, Partial, UserSocialAuth
# Shell Plus Django Imports
from django.core.cache import cache
from django.conf import settings
from django.contrib.auth import get_user_model
from django.db import transaction
from django.db.models import Avg, Case, Count, F, Max, Min, Prefetch, Q, Sum, When
from django.utils import timezone
from django.urls import reverse
from django.db.models import Exists, OuterRef, Subquery
Python 3.8.17 (default, Jul 4 2023, 05:29:51) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.34.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]:

As you can see, there’s a lot of activity when we first load into the shell; but there’s a lot of good-to-know information exposed to us. First, we see all of the models from across the application being imported for us so we no longer have to jump into the source code to find their import locations. Next, several common Django utility functions are imported to help us with any advanced tasks we may be doing. Lastly, we have some metadata about the shell itself including what Python version we’re running and what type of interactive shell it is. This output can be very useful later, since we will more than likely have to perform similar imports in our Nautobot Job.

Before we start interacting with the data in Nautobot, we need to understand how that data is being exposed to us and what methods are available for interacting with it.

Django ORM

Nautobot is our Source of Truth (SoT) for information relevant to network automation, so we need to make sure the information in Nautobot stays updated and accurate. The information is stored in the database and users can create/update/delete the information in a variety of ways with the Nautobot GUI and API being the most popular way to do so. A brave soul could also use their SQL skills to perform CRUD operations on the database directly. But luckily, thanks to the Django ORM, we don’t need to learn SQL. Remember Nautobot is an application built on the Django framework, and Django provides us with powerful tools like the ORM to simplify data management.

ORM stands for Object Relational Mapper and the Django ORM provides a Pythonic way to interact with the Models defined in our application and the data stored in our database. This gives us an abstract and programmatic method for accessing, creating, updating, and deleting information without needing to know anything about SQL or even the database itself. The ORM is technically a database abstraction API that translates our Python code into SQL commands that are sent to the database.

Querying Objects

Let’s take a look at how we can retrieve data from Nautobot using the ORM. There are three main components used to construct and execute a query in the ORM.

• Model Class: Represents the database table being queried.
• Manager: QuerySet generator (typically “.objects”).
• Method(s): Functions that represent different SQL operations that are chainable.

In the example code below, DeviceRole is a Model class defined in our Nautobot source code and ultimately represent the table in our database. objects is the Manager of DeviceRole. Both filter() and order_by() are Methods that modify our query. This statement will construct a QuerySet object that executes the SQL operations on our database and returns the results.

DeviceRole.objects.filter(...).order_by(...)

The following table contains some of the most commonly used methods. You can find a complete list in the QuerySet API reference.

Let’s go through a few examples, starting with a query for all of the Manufacturers.

Python 3.8.17 (default, Jul 4 2023, 05:29:51) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.34.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: Manufacturer.objects.all()
Out[1]: <RestrictedQuerySet [<Manufacturer: Arista>, <Manufacturer: Cisco>, <Manufacturer: F5>, <Manufacturer: Juniper>, <Manufacturer: Palo Alto>]>

As you can see, Manufacturer is our Model class; objects is the model manager that provides an interface for query operations; and all() is a method of our model manager that returns a QuerySet object that contains all of the records in our Manufacturer database table. Now, we’re not going to deep dive into relational databases in this blog series, but in over-simplistic terms, a database table can visually be represented as a basic table in a spreadsheet. Each record in our database table is a row in the table in our spreadsheet. What the Django ORM has done for us is create a Python object for each record in our database table, and that Python object has attributes that map to each column in the database table. Sometimes those columns contain a relationship to one or more records in another database table.

Let’s use first() to return the first Manufacturer object in our QuerySet.

In [2]: vendor = Manufacturer.objects.first()

In [3]: vendor
Out[3]: <Manufacturer: Arista>

Database query performance is very important when talking about the performance of an application as a whole, and optimizing our queries can greatly improve the responsiveness of Nautobot. For that purpose, QuerySets are lazy and only specific actions will trigger an actual database query. In our example above, line 2 constructs the QuerySet object and allows us to make further modifications to it. But it is line 3 that triggers the query to be executed on the database. This allows us to construct the perfect QuerySet in multiple statements without wasting valuable IO resources.

Now, let us take a look at a simplified version of our Manufacturer table for reference:

In our shell, we can access the data in these columns via attributes on our returned Manufacturer object:

In [4]: vendor.name
Out[4]: 'Arista'

In [5]: vendor.slug
Out[5]: 'arista'

In [6]: vendor.description
Out[6]'Arista Networks is an American computer networking company headquartered in Santa Clara, California.'

What if we didn’t have the table of the attributes of a Manufacturer object above? How can we find all the available attributes and relationships of a given Model? Luckily, each model has a Meta class object inside it that can be used to access some useful information about that model. Using the get_fields() method of the related Meta class object, we can display a list of available attributes and relationships along with their types.

In [7]: Manufacturer._meta.get_fields()
Out[7]: 
(<ManyToOneRel: dcim.inventoryitem>,
 <ManyToOneRel: dcim.devicetype>,
 <ManyToOneRel: dcim.platform>,
 <django.db.models.fields.UUIDField: id>,
 <django.db.models.fields.DateField: created>,
 <django.db.models.fields.DateTimeField: last_updated>,
 <django.db.models.fields.json.JSONField: _custom_field_data>,
 <django.db.models.fields.CharField: name>,
 <nautobot.core.fields.AutoSlugField: slug>,
 <django.db.models.fields.CharField: description>,
 <django.contrib.contenttypes.fields.GenericRelation: source_for_associations>,
 <django.contrib.contenttypes.fields.GenericRelation: destination_for_associations>)

Unfortunately, this only gives us part of the info we need for reverse relationships. As you can see in the output, the first three items are of type ManytoOneRel which indicates there is a Reverse One-to-Many Relationship between the Manufacturer model and the three models listed. So how do we access the related objects? Well, the Meta comes to the rescue again with the attribute fields_map, which returns a dictionary of the reverse relationship accessors and their types.

In [9]: Manufacturer._meta.fields_map
Out[9]: 
{'inventory_items': <ManyToOneRel: dcim.inventoryitem>,
 'device_types': <ManyToOneRel: dcim.devicetype>,
 'platforms': <ManyToOneRel: dcim.platform>}

Model relationships are an advanced topic and out of scope for this blog series. However, if you would like to learn more about model relationships, check out this section in the Official Django Documentation.

Now let’s use what we’ve learned to query all of the Device Types associated with a manufacturer. There are two ways to do this, and we will explore both of them to show how powerful the Django ORM is. First, continuing with our previous examples, we can use the device_types.all() to return another QuerySet object containing all of the DeviceType objects with a relationship to our Manufacturer object.

In [10]: vendor.device_types.all()
Out[10]: <RestrictedQuerySet [<DeviceType: DCS-7150S-24>, <DeviceType: DCS-7280CR2-60>, <DeviceType: vEOS>]>

This is a very complex and powerful tool that is only possible due to the relational database and the Django ORM. We can also obtain the same resulting QuerySet by using filter() on the model manager for the DeviceType model to select all Device Types that are related to our Manufacturer object vendor.

In [11]: DeviceType.objects.filter(manufacturer=vendor)
Out[11]: <RestrictedQuerySet [<DeviceType: DCS-7150S-24>, <DeviceType: DCS-7280CR2-60>, <DeviceType: vEOS>]>

In the above example, we are passing in the previously set variable vendor, containing the manufacturer named 'Arista', as a keyword argument called manufacturer. This is equivalent to the following more explicit example:

In [12]: DeviceType.objects.filter(manufacturer__name='Arista')
Out[12]: <RestrictedQuerySet [<DeviceType: DCS-7150S-24>, <DeviceType: DCS-7280CR2-60>, <DeviceType: vEOS>]>

The double underscore (__) allows us to reference attributes of related models and lets Django connect the dots for us.

Creating Objects

To create a new Manufacturer, we can use its class constructor to instantiate a new Manufacturer object and either pass in the attributes we want to set as keyword arguments. For the Manufacturer model, the only required attribute is name.

In [13]: new_vendor = Manufacturer(name="MyManufacturer")

or

In [13]: new_vendor = Manufacturer()

In [14]: new_vendor.name = "MyManufacturer"

Up to this point, we’ve only created a Manufacturer object in our Python shell and nothing has been saved to our database. We now have to tell the ORM that we’re ready to insert that object in the database. We do this by calling the method validated_save(). This method calls self.full_clean() and self.save() on the object to enforce any model validation rules before saving the object.

Updating Objects

Similarly, if we want to update our new Manufacturer by adding a description, then we follow the same process of setting the attribute and executing validated_save() on the object.

In [15]: new_vendor.description = "My new Manufacturer"

# We need to refresh our Python object to show the updates made to the database by update().
In [16]: new_vendor.validated_save()

In [17]: new_vendor.description
Out[17]: 'My new Manufacturer'

Advanced Methods

Django provides a few advanced methods for scenarios when you’re not entirely sure whether or not something exists.

get_or_create(): This method will query the database for an object that matches the attribute you provide it, or it will create that object if it doesn’t exist. This method will return a tuple of the object and a boolean indicating whether or not it was created. However, it is important to note that this method does not call full_clean() and can bypass the validators defined on the model.

You can find all of the available methods in the QuerySet API reference.

Deleting Objects

Now that we’re done practicing our ORM skills on our test Manufacturer object, let’s delete it.

In [18]: new_vendor.delete()

In [19]: Manufacturer.objects.filter(name="MyManufacturer")
Out[19]: <RestrictedQuerySet []>

Here we are calling the delete() method on the instance itself. We can also call the delete() on a QuerySet. And if we want to be truly destructive, we can chain all() and delete() together to remove all manufacturers.

In [20]: Manufacturer.objects.filter(name="MyManufacturer").delete()

In [21]: Manufacturer.objects.filter(name="MyManufacturer")
Out[21]: <RestrictedQuerySet []>

There are no protections when manipulating data directly via the ORM, as all data validation and cleaning are bypassed; so please don’t do these things in a production environment!

What’s Next?

In Part 2, we will explore the Nautobot Job framework and how we can use our Django ORM skills to get the most out of our Jobs!

-Allen