One Parser to Rule All Your Circuit Maintenance Notifications

A few months ago, Network to Code released two open source projects (more info) to contribute to solving a common problem in modern networks: understand when a circuit is going through a planned maintenance. On one side, the circuit-maintenance-parser, to parse notifications into a common data structure, and on the other side, the nautobot-circuit-maintenance plugin, that uses the parser library to automatically populate the data into Nautobot. Following months of development, we are happy to announce the release of version 2.0.0 of the parser, which comes with a lot of improvements based on existing customer deployments and now covers 19 different ISPs!

circuit-maintenance-parser Library

In the mentioned blog, we acknowledged that we were not the first ones trying to solve this issue. We decided to adopt the format proposed here (using the iCalendar format) as our gold standard.

Now, you could be wondering: why do we need a parser library when there is a well-defined format? The answer is well-known… being just a recommendation it is not fully adopted by all the Network Service Providers (NSPs). So there is still a need to parse arbitrary data formats in order to obtain something conforming to a standard Maintenance containing the following attributes:

• provider: identifies the provider of the service that is the subject of the maintenance notification.
• account: identifies an account associated with the service that is the subject of the maintenance notification.
• maintenance_id: contains text that uniquely identifies the maintenance that is the subject of the notification.
• circuits: list of circuits affected by the maintenance notification and their specific impact.
• status: defines the overall status or confirmation for the maintenance.
• start: timestamp that defines the start date of the maintenance in GMT.
• end: timestamp that defines the end date of the maintenance in GMT.
• stamp: timestamp that defines the update date of the maintenance in GMT.
• organizer: defines the contact information included in the original notification.

Please, refer to the BCOP to more details about these attributes.

This library aims to fill the current gap between an ideal representation of a circuit maintenance notification and the current providers’ formats, enabling automation to be built around these notifications.

The first version of the circuit-maintenance-parser had a simple workflow that eventually became a blocker to solve more complex use-cases and this required a new middleware that could combine multiple data using custom logic to process composed notifications, and be able to accommodate future new use-cases. More details about the logic workflow is available in the library Readme.

Supported Providers

Obviously, one of the key success indicators of the library is how many Providers are supported, and thanks to multiple examples seen from early adopters, the supported providers list has increased to 19 providers and it’s growing quickly.

• AquaComms
• AWS
• Cogent
• Colt
• EuNetworks
• GTT
• HGC
• Lumen
• Megaport
• Momentum
• NTT
• PacketFabric
• Seaborn
• Sparkle
• Telia
• Telstra
• Turkcell
• Verizon
• Zayo

Moreover, the gold format is supported by default with the GenericProvider, so any NSP that sends the notification with the iCalendar format is supported by default.

How to Use It?

The circuit_maintenance_parser library requires two things:

• The notificationdata: this is the data that the library will check to extract the maintenance notifications. It can be simple (only one data type and content, such as an iCalendar notification) or more complex (with multiple data parts of different types, such as from an email).
• The provider identifier: used to select the required Provider. Each Provider contains the logic to process the notificationdata using associated parsers.

Python Library

First step is to define the Provider that we will use to parse the notifications. By default, the GenericProvider (used when no other provider type is defined) will support parsing of iCalendar notifications using the recommended format.

from circuit_maintenance_parser import init_provider

generic_provider = init_provider()

type(generic_provider)
<class 'circuit_maintenance_parser.provider.GenericProvider'>

However, usually some Providers don’t fully implement the standard. Or perhaps some information is missing, for example the organizer email. We also support custom defined Providers that can be used to tailor the data extraction based on the notifications your organization receives :

ntt_provider = init_provider("ntt")

type(ntt_provider)
<class 'circuit_maintenance_parser.provider.NTT'>

Once we have the Provider ready, we need to initialize the data to process. We call it NotificationData and can be initialized from a simple content and type or from more complex structures, such as an email.

from circuit_maintenance_parser import NotificationData

raw_data = b"""BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//Maint Note//https://github.com/maint-notification//
BEGIN:VEVENT
SUMMARY:Maint Note Example
DTSTART;VALUE=DATE-TIME:20151010T080000Z
DTEND;VALUE=DATE-TIME:20151010T100000Z
DTSTAMP;VALUE=DATE-TIME:20151010T001000Z
UID:42
SEQUENCE:1
X-MAINTNOTE-PROVIDER:example.com
X-MAINTNOTE-ACCOUNT:137.035999173
X-MAINTNOTE-MAINTENANCE-ID:WorkOrder-31415
X-MAINTNOTE-IMPACT:OUTAGE
X-MAINTNOTE-OBJECT-ID;X-MAINTNOTE-OBJECT-IMPACT=NO-IMPACT:acme-widgets-as-a-service
X-MAINTNOTE-OBJECT-ID;X-MAINTNOTE-OBJECT-IMPACT=OUTAGE:acme-widgets-as-a-service-2
X-MAINTNOTE-STATUS:TENTATIVE
ORGANIZER;CN="Example NOC":mailto:noone@example.com
END:VEVENT
END:VCALENDAR
"""

data_to_process = NotificationData.init_from_raw("ical", raw_data)

type(data_to_process)
<class 'circuit_maintenance_parser.data.NotificationData'>

Finally, we retrieve the maintenances (it is a List because a notification can contain multiple maintenances) from the data calling the get_maintenances method from the Provider instance:

maintenances = generic_provider.get_maintenances(data_to_process)

print(maintenances[0].to_json())
{
"account": "137.035999173",
"circuits": [
{
"circuit_id": "acme-widgets-as-a-service",
"impact": "NO-IMPACT"
},
{
"circuit_id": "acme-widgets-as-a-service-2",
"impact": "OUTAGE"
}
],
"end": 1444471200,
"maintenance_id": "WorkOrder-31415",
"organizer": "mailto:noone@example.com",
"provider": "example.com",
"sequence": 1,
"stamp": 1444435800,
"start": 1444464000,
"status": "TENTATIVE",
"summary": "Maint Note Example",
"uid": "42"
}

Notice that, either with the GenericProvider or NTT provider, we get the same result from the same parsed data, because they are using exactly the same Processor and Parser. The only difference is that NTT Provider will provide some custom default values for NTT in case the notification doesn’t contain this data. In this case, the notification contains all the information, so the custom defaults for Provider are not used.

ntt_maintenances = ntt_provider.get_maintenances(data_to_process)
assert ntt_maintenances == maintenances

CLI

There is also a cli entrypoint circuit-maintenance-parser which offers easy access to the library using few arguments:

• data-file: file storing the notification.
• data-type: ical, html or email, depending on the data type.
• provider-type: to choose the right Provider. If empty, the GenericProvider is used.

circuit-maintenance-parser --data-file "/tmp/___ZAYO TTN-00000000 Planned MAINTENANCE NOTIFICATION___.eml" --data-type email --provider-type zayo
Circuit Maintenance Notification #0
{
  "account": "my_account",
  "circuits": [
    {
      "circuit_id": "/OGYX/000000/ /ZYO /",
      "impact": "OUTAGE"
    }
  ],
  "end": 1601035200,
  "maintenance_id": "TTN-00000000",
  "organizer": "mr@zayo.com",
  "provider": "zayo",
  "sequence": 1,
  "stamp": 1599436800,
  "start": 1601017200,
  "status": "CONFIRMED",
  "summary": "Zayo will implement planned maintenance to troubleshoot and restore degraded span",
  "uid": "0"
}

How to Extend the Library?

Even though the library aims to include support for as many providers as possible, it’s likely that not all the thousands of NSP are supported and you may need to add support for some new one. Adding a new Provider is quite straightforward, and in the following example we are adding support for an imaginary provider, ABCDE, that uses HTML notifications.

First step is creating a new file: circuit_maintenance_parser/parsers/abcde.py. This file will contain all the custom parsers needed for the provider and it will import the base classes for each parser type from circuit_maintenance_parser.parser. In the example, we only need to import Html and in the child class implement the methods required by the class, in this case parse_html() which will return a dict with all the data that this Parser can extract. In this case we have to helper methods, _parse_bs and _parse_tables that implement the logic to navigate the notification data.

from typing import Dict
import bs4  # type: ignore
from bs4.element import ResultSet  # type: ignore
from circuit_maintenance_parser.parser import Html

class HtmlParserABCDE1(Html):
    def parse_html(self, soup: ResultSet) -> Dict:
        data = {}
        self._parse_bs(soup.find_all("b"), data)
        self._parse_tables(soup.find_all("table"), data)
        return [data]

    def _parse_bs(self, btags: ResultSet, data: Dict):
      ...

    def _parse_tables(self, tables: ResultSet, data: Dict):
      ...

Next step is to create the new Provider by defining a new class in circuit_maintenance_parser/provider.py. This class that inherits from GenericProvider only needs to define two attributes:

• _processors: is a list of Processor instances that uses several data Parsers. In this example, we don’t need to create a new custom Processor because the combined logic serves well (the most likely case), and we only need to use the new defined HtmlParserABCDE1 and also the generic EmailDateParser that extract the email date. Also notice that you could have multiple Processors with different Parsers in this list, supporting several formats.
• _default_organizer: this is a default helper to fill the organizer attribute in the Maintenance if the information is not part of the original notification.
• _include_filter: mapping of data_types to a list of regex expressions that if provided have to match to parse the notification. This feature removes noise from notifications that are received from the same provider, but that are not related to circuit maintenance notifications.
• _exclude_filter: antagonist mapping to define via regex which are the notifications that must not parsed.

class ABCDE(GenericProvider):
    _processors: List[GenericProcessor] = [
        CombinedProcessor(data_parsers=[EmailDateParser, HtmlParserABCDE1]),
    ]
    _default_organizer = "noc@abcde.com"
    _include_filter = {EMAIL_HEADER_SUBJECT: ["Scheduled Maintenance"]}

And expose the new Provider in circuit_maintenance_parser/__init__.py:

from .provider import (
    GenericProvider,
    ABCDE,
    ...
)

SUPPORTED_PROVIDERS = (
    GenericProvider,
    ABCDE,
    ...
)

Last, but not least, you should update the tests!

• Test the new Parser in tests/unit/test_parsers.py
• Test the new Provider logic in tests/unit/test_e2e.py

… adding the necessary data samples in tests/unit/data/abcde/.