Why did Network to Code Fork NetBox?

Blog Detail

[This post was reviewed and edited by many of the people who have contributed to the Nautobot project at Network to Code.]

Over the last five years, NetBox has gained adoption as an IP Address Management (IPAM) and Data Center Infrastructure Management (DCIM) platform while also serving as a Source of Truth for documenting networks and network automation projects. In fact, Network to Code has been heavy users, supporters, and contributors of NetBox for years. While we started offering NetBox professional services in 2018 and support for NetBox in 2019, we have continued to support an ever-growing customer base that has evolving requirements around Source of Truth and more broadly around creating holistic network automation solutions. Because of this, and in order to better serve our customers and the community, Network to Code made the difficult decision to fork NetBox and drive it in new directions.

Let’s dive into the details.

Background

Network to Code has been a power user of NetBox for the past 3+ years. We’ve designed and deployed numerous solutions that revolve around and integrate with NetBox. We have used the existing models, extensibility features, worked on a myriad of private customer-specific forks, created ways to manage settings and URLs, as well as created plugins for various use cases. In short, over the course of countless projects, we have seen first-hand what works well for users and what has room for improvement.

There have been questions raised in different public forums if Network to Code has been a contributor to NetBox. The answer is yes. Let’s explore some of those contributions.

First and foremost, as part of the NetBox community, Network to Code has contributed numerous blog articlespublished videoshosted a NetBox Day, presented at conferences, enabled and participated in innumerable Slack discussions hosted on the Network to Code Slack workspace, participated in podcasts, hosted low-cost training, and more. While all of this was not hand-to-keyboard development, NTC has undoubtedly contributed thousands of hours toward NetBox in this capacity in recent years.

For contributions to the core of NetBox, NTC employees (not counting the lead maintainer of NetBox, who was employed by NTC for 18 months and spent the majority of his time actively maintaining NetBox during that time) have contributed 78 issues and 35 PRs, two of which are still being worked on. Some of these contributions have been significant in scope, such as the initial implementation of the NetBox plugin API that was released in NetBox 2.8.

When you look at code and development, there are NTC projects that also build on top of NetBox. We have built a Device Onboarding pluginPrometheus metrics plugin, a number of other plugins that we are working to open source, as well as projects like network-importer to simplify adding data into NetBox. Each of these projects individually are the result of hundreds of hours, adding up to many months of dedicated development effort.

While we have never claimed to be an official sponsor of NetBox, we were effectively an unofficial and informal sponsor of the project. Lastly, one of the core committers on our fork had also been a maintainer of NetBox for a number of years while at Network to Code.

We have always had candid discussions on feature requests, support, and on the direction that we believed was in the best interest of the long term success of NetBox. As time went on and after numerous discussions, there became a growing divergence in our vision about what a Source of Truth for networking should look like and how to get there, and the NetBox project team suggested that we should consider forking. Late in 2020, we took a step back to think about the options.

Taking what we’ve learned deploying and integrating NetBox solutions for customers over the past few years, what we have seen and heard from the community, coupled with discussions with the NetBox project team, Network to Code came to the conclusions that it was best to fork NetBox with the vision and direction necessary for powering network automation platforms. Next, from a business and technical perspective, let’s explore the key reasons we considered and what drove us to make the decision.

We need to provide world class support for our clients.

We need to offer high-touch support models with Service Level Agreements (SLAs) we can guarantee. We need flexibility to offer Long-term Support (LTS) for customers who can’t upgrade at the pace of a fast moving open source project. In addition, we need to be able to deliver the features required to get a system into production in an enterprise environment. Some examples include simplifying the ability to integrate with Single Sign-On (SSO) services such as SAML or OIDC and adding support for additional database backends such as MySQL.

Define a strategic vision for Source of Truth that enables network automation.

We firmly believe that a Source of Truth is a fundamental requirement to enable data-driven network automation. This is where we are able to leverage our collective experience at Network to Code and add functionality that simplifies integrating with network automation solutions. While the new platform can still be used for network documentation, its key focus is on enabling network automation. After all, network automation is our core business at Network to Code. It has become apparent through our customer projects that there is a need for more flexible APIs and more seamless ways to integrate with other data sources.

Build a Network Automation App Platform.

This is one area that we think is quite unique. The goal is to leverage the rich data stored in the Source of Truth and build high-value apps that use and consume that data while interacting with other systems and network devices as needed. The best metaphor to consider is a smartphone. It will always be a “phone.” However, this “phone” is a delivery mechanism for high-value apps that we all get to use on a daily basis (should you choose to). In this example, the Source of Truth is the phone, i.e. the delivery mechanism. Our goal is to enable the Source of Truth as a platform through which high-value apps are deployed. These apps can be lightweight (creating new models and APIs) or be used to deliver solutions (pre/post change, integrate with monitoring, perform backups). We believe it should be up to the users to decide.

Community Committed

Network to Code has deep roots in the community. As mentioned earlier, we have spent thousands of hours on community work around NetBox. If you look at every other project we’ve been a part of, that number goes up significantly. From open sourcing various projects of our own to contributing to other such projects to engaging in discussions on numerous forums, community is extremely important to who we are as a company.

We know there is A LOT of work to do to gain the respect and trust of the community as we move forward with Nautobot. We are committed to fostering user and developer communities that are fully transparent. We will be hanging out in the #nautobot channel on the Network to Code Slack. Please come join and let us know what you think! You can also view the initial Nautobot roadmap.

If you have ideas, enhancements, or feature requests for Nautobot, please don’t hesitate to open an issue on GitHub.

Going Forward

Just to get us where we are with Nautobot, we have invested close to a “work year” of effort across our team just over the past few months. This effort and focus is not going to stop. While Nautobot is much more than NetBox with additional features, it’s actually not even about what you see today. This is just the first release. If you haven’t already done so, please check out the roadmap. You’ll see that our goal is to empower flexibility and extensibility while truly being a Source of Truth foundation for network automation. Our vision will see Nautobot continue to diverge from NetBox in such a way that the separation of the two will become much more distinct.

Git History

When we first released the Nautobot project on GitHub, we had made a decision, based on technical considerations, to not keep the complete NetBox commit history. It was never our intent to “erase” the NetBox commit history because that history exists within the NetBox project and we are making every attempt to be fully transparent about the forked status and attribute proper credit. We have heard your feedback that this can give the appearance of dismissing the history and community engagement that has brought NetBox to where it is today, and we apologize for that. We are grateful and we are re-adding the NetBox Git history to Nautobot. Below are the technical reasons we had in mind when we chose to reset the commit history.

First, we thought that it would make it possible to draw a clear dividing line between the development of NetBox and the development of Nautobot so there would be no ambiguity as to whether a given commit in the Git history is a shared NetBox commit that was inherited by Nautobot, or whether a commit is specific to Nautobot and distinct from NetBox. This would make it easier to see over time how the two projects have diverged.

Second, we wanted to avoid any ambiguity about version numbers. Creating a new repository from our own version 1.0 allowed us to avoid these collisions and confusion without needing to do anything messy like deleting old tags and releases from the history.

Third, we have made a large number of invasive changes related to packaging and renaming which greatly reduce the historical relevance of the history from a purely technical standpoint.

Fourth, it reduced the size of the repository substantially. A fresh clone of nautobot.git is only about 28 MB in size, versus a fresh clone of upstream netbox.git weighing in at about 185 MB.

We remain grateful to all developers of and contributors to NetBox, and hope that this provides appropriate clarity as to the reasons why we initially chose this particular approach.


Conclusion

The decision to fork and create something new was not a decision we took lightly at Network to Code. We are committed to creating something great and to ensuring Nautobot will become a thriving, transparent, and engaging project and community. We’re already seeing a lot of positive feedback from the community based on the direction planned for Nautobot.

We welcome you on the journey and hope you come along for the ride. We look forward to the evolution of Nautobot and seeing Nautobot used to power network automation solutions across the world.

-Jason



ntc img
ntc img

Contact Us to Learn More

Share details about yourself & someone from our team will reach out to you ASAP!

Using NetBox for Ansible Source of Truth

Blog Detail

This content was originally posted on the Ansible Blog on 2020-12-08. Here you will learn about NetBox at a high level, how it works to become a Source of Truth (SoT), and look into the use of the Ansible Content Collection, which is available on Ansible Galaxy. The goal is to show some of the capabilities that make NetBox a terrific tool and will be using NetBox as your network Source of Truth for automation!

Source of Truth

Why a Source of Truth? The Source of Truth is where you go to get the intended state of the device. There does not need to be a single Source of Truth, but you should have a single Source of Truth per data domain, often referred to as the System of Record (SoR). For example, if you have a database that maintains your physical sites that is used by teams outside of the IT domain, that should be the Source of Truth on physical sites. You can aggregate the data from the physical site Source of Truth into other data sources for automation. Just be aware that when it comes time to collect data, then it should come from that other tool.

The first step in creating a network automation framework is to identify the Source of Truth for the data, which will be used in future automations. Oftentimes for a traditional network, the device itself has been considered the SoT. Reading the configuration off of the device each time you need a configuration data point for automation is inefficient, and presumes that the device configuration is as intended, not simply left there in troubleshooting or otherwise inadvertently left. When it comes to providing data to teams outside of the network organization, exposing an API can help to speed up gathering data without having to check in with the device first.

NetBox

For a Source of Truth, one popular open source choice is NetBox. From the primary documentation site https://netbox.readthedocs.io, “NetBox is an open source web application designed to help manage and document computer networks”. NetBox is currently designed to help manage your:

  • DCIM (Data Center Infrastructure Management)
  • IPAM (IP Address Management)
  • Data Circuits
  • Connections (Network, console, and power)
  • Equipment racks
  • Virtualization
  • Secrets

Since NetBox is an IPAM tool, there are misconceptions at times about what NetBox is able to do. To be clear, NetBox is not:

  • Network monitoring
  • DNS server
  • RADIUS server
  • Configuration management
  • Facilities management

Why NetBox?

NetBox is a tool that is built on many common Python based open source tools, using Postgres for the backend database and Python Django for the back-end API and front-end UI. The API is extremely friendly as it supports CRUD (Create, Read, Update, Delete) operations and is fully documented with Swagger documentation. The NetBox Collection helps with several aspects of NetBox including an inventory plugin, lookup plugin, and several modules for updating data in NetBox.

NetBox gives a modern UI from the point of view of a network organization to help document IP addressing, while keeping the primary emphasis on network devices, system infrastructure, and virtual machines. This makes it ideal to use as your Source of Truth for automating.

NetBox itself does not do any scanning of network resources. It is intended to have humans maintain the data as this is going to be the Source of Truth. It represents what the environment should look like.

Ansible Content Collection for NetBox

You will find the Collection within the netbox-community GitHub organization. Here you find a Docker container imagedevice-type librarycommunity generated NetBox reports, and source code for NetBox itself.

If you are unfamiliar with what an Ansible Content Collection is, please watch this brief YouTube video.

The Galaxy link for the Collection is at https://galaxy.ansible.com/netbox/netbox.

The NetBox Collection allows you to get started quickly in adding information into a NetBox instance. The only requirements are to supply an API key and a URL to get started. With this Collection, a base inventory, and a NetBox environment you are able to get a Source of Truth populated very quickly.

Let’s walk through the base setup to get to a place where you are starting to use the NetBox Inventory Plugin as your Ansible inventory. First is the example group_vars/all.yml file that will have the list of items to be used with the tasks.

---
site_list:
  - name: “NYC”
    time_zone: America/New_York
    status: Active
  - name: “CHI”
    time_zone: America/Chicago
    status: Active
  - name: “RTP”
    time_zone: America/New_York
    status: Active
manufacturers:  # In alphabetical order
  - Arista
  - Cisco
  - Juniper
device_types:
  - model: “ASAv”
    manufacturer: “Cisco”
    slug: “asav”
    part_number: “asav”
    Full_depth: False
  - model: “CSR1000v”
    manufacturer: “Cisco”
    slug: “csr1000v”
    part_number: “csr1000v”
    Full_depth: False
  - model: “vEOS”
    manufacturer: “Arista”
    slug: “veos”
    part_number: “veos”
    Full_depth: False
  - model: “vSRX”
    manufacturer: “Juniper”
    slug: “vsrx”
    part_number: “vsrx”
    Full_depth: False
platforms:
  - name: “ASA”
    slug: “asa”
  - name: “EOS”
    slug: “eos”
  - name: “IOS”
    slug: “ios”
  - name: “JUNOS”
    slug: “junos”

The first step is to create a site. Since NetBox models physical gear, you install equipment at a physical location. Whether that is in your own facilities or inside of a cloud, this is a site. The module for this is the netbox.netbox.netbox_site module. A task in the playbook may be:

- name: "TASK 10: SETUP SITES"
  netbox.netbox.netbox_site:
    netbox_url: "{{ lookup('ENV', 'NETBOX_URL') }}"
    netbox_token: "{{ lookup('ENV', 'NETBOX_API_KEY') }}"
    data: "{{ item }}"
  loop:{{ site_list }}

The next two pieces are the base to add devices to NetBox. In order to create a specific device, you also need to have the device type and manufacturer in your NetBox instance. To do this there are specific modules available to create them. Platforms will help to identify what OS the device is. I recommend that you use what your automation platform is using—something like IOS, NXOS, and EOS are good choices and should match up to your ansible_network_os choices. These tasks look like the following:

- name: "TASK 20: SETUP MANUFACTURERS"
  netbox.netbox.netbox_manufacturer:
    netbox_url: "{{ lookup('ENV', 'NETBOX_URL') }}"
    netbox_token: "{{ lookup('ENV', 'NETBOX_API_KEY') }}"
    data:
      name: "{{ manufacturer }}"
  loop: "{{ manufacturers }}"
  loop_control:
    loop_var: manufacturer

- name: "TASK 30: SETUP DEVICE TYPES"
  netbox.netbox.netbox_device_type:
    netbox_url: "{{ lookup('ENV', 'NETBOX_URL') }}"
    netbox_token: "{{ lookup('ENV', 'NETBOX_API_KEY') }}"
    data:
      model: "{{ device_type.model }}"
      manufacturer: "{{ device_type.manufacturer }}"
      slug: "{{ device_type.slug }}"
      part_number: "{{ device_type.part_number }}"
      is_full_depth: "{{ device_type.full_depth }}"
  loop: "{{ device_types }}"
  loop_control:
    loop_var: device_type
    label: "{{ device_type['model'] }}"

- name: "TASK 40: SETUP PLATFORMS"
  netbox.netbox.netbox_platform:
    netbox_url: "{{ lookup('ENV', 'NETBOX_URL') }}"
    netbox_token: "{{ lookup('ENV', 'NETBOX_API_KEY') }}"
    data:
      name: "{{ platform.name }}"
      slug: "{{ platform.slug }}"
  loop: "{{ platforms }}"
  loop_control:
    loop_var: platform
    label: "{{ platform['name'] }}"

At this stage you are set to add devices and device information to NetBox. The following tasks leverage the ansible_facts that Ansible automatically gathers. So for these particular device types, no additional parsing/data gathering is required outside of using Ansible to gather facts. In this example for adding a device, you will notice custom_fields. A nice extension of NetBox is that if there is not a field already defined, you can set your own fields and use them within the tool.

- name: "TASK 100: NETBOX >> ADD DEVICE TO NETBOX"
  netbox.netbox.netbox_device:
    netbox_url: "{{ lookup('ENV', 'NETBOX_URL') }}"
    netbox_token: "{{ lookup('ENV', 'NETBOX_API_KEY') }}"
    data:
      name: "{{ inventory_hostname }}"
      device_type: "{{ ansible_facts['net_model'] }}"
      platform: IOS  # May be able to use a filter to define in future
      serial: "{{ ansible_facts['net_serialnum'] }}"
      status: Active
      device_role: "{{ inventory_hostname | get_role_from_hostname }}"
      site: “ANSIBLE_DEMO_SITE"
      custom_fields:
        code_version: "{{ ansible_facts['net_version'] }}"

- name: "TASK 110: NETBOX >> ADD INTERFACES TO NETBOX"
  netbox.netbox.netbox_device_interface:
    netbox_url: "{{ lookup('ENV', 'NETBOX_URL') }}"
    netbox_token: "{{ lookup('ENV', 'NETBOX_API_KEY') }}"
    data:
      device: "{{ inventory_hostname }}"
      name: "{{ item.key }}"
      form_factor: "{{ item.key | get_interface_type }}"  # Define types
      mac_address: "{{ item.value.macaddress | ansible.netcommon.hwaddr }}"
    state: present
  with_dict:
    - "{{ ansible_facts['net_interfaces'] }}"

Once you have the interfaces you can add in IP address information that is included in the ansible_facts data, I show three steps. First is to add a temporary interface (TASK 200), then add the IP address (TASK 210), and finally associate the IP address to the device (TASK 220).

- name: "TASK 200: NETBOX >> Add temporary interface"
  netbox.netbox.netbox_device_interface:
    netbox_url: "{{ lookup('ENV', 'NETBOX_URL') }}"
    netbox_token: "{{ lookup('ENV', 'NETBOX_API_KEY') }}"
    data:
      device: "{{ inventory_hostname }}"
      name: Temporary_Interface
      form_factor: Virtual
    state: present

- name: "TASK 210: NETBOX >> ADD IP ADDRESS OF ANSIBLE HOST"
  netbox.netbox.netbox_ip_address:
    netbox_url: "{{ lookup('ENV', 'NETBOX_URL') }}"
    netbox_token: "{{ lookup('ENV', 'NETBOX_API_KEY') }}"
    data:
      family: 4
      address: "{{ ansible_host }}/24"
      status: active
      interface:
        name: Temporary_Interface
        device: "{{ inventory_hostname }}"

- name: "TASK 220: NETBOX >> ASSOCIATE IP ADDRESS TO DEVICE"
  netbox.netbox.netbox_device:
    netbox_url: "{{ lookup('ENV', 'NETBOX_URL') }}"
    netbox_token: "{{ lookup('ENV', 'NETBOX_API_KEY') }}"
    data:
      name: "{{ inventory_hostname }}"
      device_type: "{{ ansible_facts['net_model'] }}"
      platform: IOS
      serial: "{{ ansible_facts['net_serialnum'] }}"
      status: Active
      primary_ip4: "{{ ansible_host }}/24"

Ansible Inventory Source

At this point you have NetBox populated with all of your devices that were in your static inventory. It is now time to make the move to using NetBox as the Source of Truth for your Ansible dynamic inventory plugin. This way you don’t have to keep finding all of the projects that need to get updated when you make a change to the environment. You just need to change your Source of Truth database – NetBox.

You define which inventory plugin to use with a YAML file that defines the characteristics of how to configure your intended use of the plugin. Below is an example, showing you are able to query many components of NetBox for use within your Ansible inventory. You may wish to only make an update to your access switches? Use the query_filters key to define what NetBox API searches should be executed. Take a look at the plugin documentation for updated supported parameters on GitHub or ReadTheDocs. The compose key allows you to pass in additional variables to be used by Ansible, as such the platform from above would be used with the ansible_network_os key. This is where you see the definition and what would get passed from the inventory source.

This definition also has groups created based on the device_roles that are defined in NetBox and the platforms. So you would be able to access all platforms_ios devices or platforms_eos as an example, based on the information in the Source of Truth.

---
plugin: netbox.netbox.nb_inventory
api_endpoint: http://netbox03
validate_certs: false
config_context: false
group_by:
 - device_roles
 - platforms
compose:
 ansible_network_os: platform.slug
query_filters:
 - site: "minnesota01"
 - has_primary_ip: True

Extending NetBox with Plugins

One of the more recent feature additions to NetBox itself is the ability to extend it via your own or community driven plugins. From the wiki: “Plugins are packaged Django apps that can be installed alongside NetBox to provide custom functionality not present in the core application” GitHub Link. You can find some of the featured plugins in the community at that link. Some include:

There are many plugins available to the community for you to choose from—or you can write your own add ons! Search on GitHub for the topic NetBox Plugin.


Conclusion

NetBox and Ansible together are a great combination for your network automation needs!

NetBox is an excellent open source tool that helps make it easy to create, update, and consume as a Source of Truth. The APIs are easy to use and make updates to the DB with, even if you did not want to use the NetBox Collection available for Ansible. Having a tool that is flexible, capable, and accurate is a must for delivering automation via a Source of Truth. NetBox delivers on each of these.

This post was inspired by a presentation done in March 2020 at the Minneapolis Ansible Meetup. For additional material on this, I have many of these tasks available as a working example on GitHub. The YouTube recording of the presentation from the Ansible Meetup is available.

-Josh



ntc img
ntc img

Contact Us to Learn More

Share details about yourself & someone from our team will reach out to you ASAP!

NFD21 – NetBox Automation Integrations

Blog Detail

Wrapping up our presentations at Networking Field Day 21 last week, I was fortunate to spend some time discussing NetBox and its role as a robust source of truth within a network automation framework. Most people who use NetBox are familiar with its web interface and to some extent its REST API, but there are a variety of ways NetBox enables the import and export of data to support highly automated workflows. If you haven’t had a chance to check out NetBox yet, you can find the project on GitHub and the documentation here.

CSV Import/Export

NetBox supports CSV-formatted data import for most objects. This is very convenient for migrating data from spreadsheets to NetBox. All that’s needed is to ensure the required columns are present and named appropriately, and that the data is in a valid format. NetBox also provides a means to export data in CSV format for use by other applications.

Export CSV form

Export Templates

While the built-in CSV export function is handy, there is often a need to provide more control over the format of exported data. This can be done using export templates, which leverage the Jinja2 templating language to render arbitrary output. For instance, you might write an export template to generate a Nagios configuration file from a set of NetBox devices. Within the NetBox UI, you can then navigate to the devices list and select your custom template under the “export” button. The generated document will be provided for direct download.

Export template form

Webhooks

Sometimes it’s necessary to automatically update related systems whenever data in NetBox changes. For example, maybe you need to update a network monitoring system every time a new device is added in NetBox. This can be accomplished by creating a webhook under NetBox’s admin UI. Simply select the type(s) of object that you’re interested in, the actions on which to trigger (creation, modification, and/or deletion), and the attributes of the remote system to be notified. When the relevant changes occur, NetBox will automatically generate an HTTP POST request to the remote system with all the details of the change and the object’s new state.

Webhooks form

To see a demonstration of webhooks in action, check out Jason Edelman’s presentation.

REST API

The most robust mechanism for exchanging data with NetBox is its REST API. The API allows for nearly the same degree of functionality as the web UI (with a few caveats) and supports the creation, retrieval, modification, and deletion of objects. All objects are represented in the API using JSON serialization. NetBox provides a human-friendly browsable version of the API locally, as well as live documentation using OpenAPI (Swagger). Client libraries such as pynetbox are available to simplify the consumption of NetBox’s API by external systems.

Browsable REST API

Custom Scripts

Custom script are a new feature in NetBox, introduced in v2.6.3, which allows an administrator to extend NetBox with arbitrary Python code. These scripts are rendered as forms in the web UI, prompting the user for data which the script processes to perform some action. Scripts can create new objects in NetBox, manipulate existing objects, import data from or send data to remote systems – anything, really. This feature is especially powerful as it allows for the extension of NetBox’s core functionality without introducing the need to maintain a custom fork of the project.

Custom script form

NetBox and Network to Code

While NetBox offers plenty of avenues for integration with other systems, we understand that actually writing the code to engage these features is often easier said than done. Fortunately, Network to Code is proud to offer commercial support for NetBox as well as assistance with custom development. Leveraging our expertise with NetBox and related network automation tools, we are uniquely positioned to deliver extremely high value solutions with quick turnaround. Let us know what we can do for you!

The full series of videos from our NFD21 presentation is posted on Tech Field Day.



ntc img
ntc img

Contact Us to Learn More

Share details about yourself & someone from our team will reach out to you ASAP!