A few months ago, Network to Code released two open source projects (more info) to contribute to solving a common problem in modern networks: understand when a circuit is going through a planned maintenance. On one side, the circuit-maintenance-parser, to parse notifications into a common data structure, and on the other side, the nautobot-circuit-maintenance plugin, that uses the parser library to automatically populate the data into Nautobot. Following months of development, we are happy to announce the release of version 2.0.0 of the parser, which comes with a lot of improvements based on existing customer deployments and now covers 19 different ISPs!
circuit-maintenance-parser LibraryIn the mentioned blog, we acknowledged that we were not the first ones trying to solve this issue. We decided to adopt the format proposed here (using the iCalendar format) as our gold standard.
Now, you could be wondering: why do we need a parser library when there is a well-defined format? The answer is well-known… being just a recommendation it is not fully adopted by all the Network Service Providers (NSPs). So there is still a need to parse arbitrary data formats in order to obtain something conforming to a standard Maintenance containing the following attributes:
Please, refer to the BCOP to more details about these attributes.
This library aims to fill the current gap between an ideal representation of a circuit maintenance notification and the current providers’ formats, enabling automation to be built around these notifications.
The first version of the circuit-maintenance-parser had a simple workflow that eventually became a blocker to solve more complex use-cases and this required a new middleware that could combine multiple data using custom logic to process composed notifications, and be able to accommodate future new use-cases. More details about the logic workflow is available in the library Readme.
Obviously, one of the key success indicators of the library is how many Providers are supported, and thanks to multiple examples seen from early adopters, the supported providers list has increased to 19 providers and it’s growing quickly.
Moreover, the gold format is supported by default with the
GenericProvider, so any NSP that sends the notification with theiCalendarformat is supported by default.
The circuit_maintenance_parser library requires two things:
notificationdata: this is the data that the library will check to extract the maintenance notifications. It can be simple (only one data type and content, such as an iCalendar notification) or more complex (with multiple data parts of different types, such as from an email).provider identifier: used to select the required Provider. Each Provider contains the logic to process the notificationdata using associated parsers.First step is to define the Provider that we will use to parse the notifications. By default, the GenericProvider (used when no other provider type is defined) will support parsing of iCalendar notifications using the recommended format.
from circuit_maintenance_parser import init_provider
generic_provider = init_provider()
type(generic_provider)
<class 'circuit_maintenance_parser.provider.GenericProvider'>
However, usually some Providers don’t fully implement the standard. Or perhaps some information is missing, for example the organizer email. We also support custom defined Providers that can be used to tailor the data extraction based on the notifications your organization receives :
ntt_provider = init_provider("ntt")
type(ntt_provider)
<class 'circuit_maintenance_parser.provider.NTT'>
Once we have the Provider ready, we need to initialize the data to process. We call it NotificationData and can be initialized from a simple content and type or from more complex structures, such as an email.
from circuit_maintenance_parser import NotificationData
raw_data = b"""BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//Maint Note//https://github.com/maint-notification//
BEGIN:VEVENT
SUMMARY:Maint Note Example
DTSTART;VALUE=DATE-TIME:20151010T080000Z
DTEND;VALUE=DATE-TIME:20151010T100000Z
DTSTAMP;VALUE=DATE-TIME:20151010T001000Z
UID:42
SEQUENCE:1
X-MAINTNOTE-PROVIDER:example.com
X-MAINTNOTE-ACCOUNT:137.035999173
X-MAINTNOTE-MAINTENANCE-ID:WorkOrder-31415
X-MAINTNOTE-IMPACT:OUTAGE
X-MAINTNOTE-OBJECT-ID;X-MAINTNOTE-OBJECT-IMPACT=NO-IMPACT:acme-widgets-as-a-service
X-MAINTNOTE-OBJECT-ID;X-MAINTNOTE-OBJECT-IMPACT=OUTAGE:acme-widgets-as-a-service-2
X-MAINTNOTE-STATUS:TENTATIVE
ORGANIZER;CN="Example NOC":mailto:noone@example.com
END:VEVENT
END:VCALENDAR
"""
data_to_process = NotificationData.init_from_raw("ical", raw_data)
type(data_to_process)
<class 'circuit_maintenance_parser.data.NotificationData'>
Finally, we retrieve the maintenances (it is a List because a notification can contain multiple maintenances) from the data calling the get_maintenances method from the Provider instance:
maintenances = generic_provider.get_maintenances(data_to_process)
print(maintenances[0].to_json())
{
"account": "137.035999173",
"circuits": [
{
"circuit_id": "acme-widgets-as-a-service",
"impact": "NO-IMPACT"
},
{
"circuit_id": "acme-widgets-as-a-service-2",
"impact": "OUTAGE"
}
],
"end": 1444471200,
"maintenance_id": "WorkOrder-31415",
"organizer": "mailto:noone@example.com",
"provider": "example.com",
"sequence": 1,
"stamp": 1444435800,
"start": 1444464000,
"status": "TENTATIVE",
"summary": "Maint Note Example",
"uid": "42"
}
Notice that, either with the GenericProvider or NTT provider, we get the same result from the same parsed data, because they are using exactly the same Processor and Parser. The only difference is that NTT Provider will provide some custom default values for NTT in case the notification doesn’t contain this data. In this case, the notification contains all the information, so the custom defaults for Provider are not used.
ntt_maintenances = ntt_provider.get_maintenances(data_to_process)
assert ntt_maintenances == maintenances
There is also a cli entrypoint circuit-maintenance-parser which offers easy access to the library using few arguments:
data-file: file storing the notification.data-type: ical, html or email, depending on the data type.provider-type: to choose the right Provider. If empty, the GenericProvider is used.circuit-maintenance-parser --data-file "/tmp/___ZAYO TTN-00000000 Planned MAINTENANCE NOTIFICATION___.eml" --data-type email --provider-type zayo
Circuit Maintenance Notification #0
{
"account": "my_account",
"circuits": [
{
"circuit_id": "/OGYX/000000/ /ZYO /",
"impact": "OUTAGE"
}
],
"end": 1601035200,
"maintenance_id": "TTN-00000000",
"organizer": "mr@zayo.com",
"provider": "zayo",
"sequence": 1,
"stamp": 1599436800,
"start": 1601017200,
"status": "CONFIRMED",
"summary": "Zayo will implement planned maintenance to troubleshoot and restore degraded span",
"uid": "0"
}
Even though the library aims to include support for as many providers as possible, it’s likely that not all the thousands of NSP are supported and you may need to add support for some new one. Adding a new Provider is quite straightforward, and in the following example we are adding support for an imaginary provider, ABCDE, that uses HTML notifications.
First step is creating a new file: circuit_maintenance_parser/parsers/abcde.py. This file will contain all the custom parsers needed for the provider and it will import the base classes for each parser type from circuit_maintenance_parser.parser. In the example, we only need to import Html and in the child class implement the methods required by the class, in this case parse_html() which will return a dict with all the data that this Parser can extract. In this case we have to helper methods, _parse_bs and _parse_tables that implement the logic to navigate the notification data.
from typing import Dict
import bs4 # type: ignore
from bs4.element import ResultSet # type: ignore
from circuit_maintenance_parser.parser import Html
class HtmlParserABCDE1(Html):
def parse_html(self, soup: ResultSet) -> Dict:
data = {}
self._parse_bs(soup.find_all("b"), data)
self._parse_tables(soup.find_all("table"), data)
return [data]
def _parse_bs(self, btags: ResultSet, data: Dict):
...
def _parse_tables(self, tables: ResultSet, data: Dict):
...
Next step is to create the new Provider by defining a new class in circuit_maintenance_parser/provider.py. This class that inherits from GenericProvider only needs to define two attributes:
_processors: is a list of Processor instances that uses several data Parsers. In this example, we don’t need to create a new custom Processor because the combined logic serves well (the most likely case), and we only need to use the new defined HtmlParserABCDE1 and also the generic EmailDateParser that extract the email date. Also notice that you could have multiple Processors with different Parsers in this list, supporting several formats._default_organizer: this is a default helper to fill the organizer attribute in the Maintenance if the information is not part of the original notification._include_filter: mapping of data_types to a list of regex expressions that if provided have to match to parse the notification. This feature removes noise from notifications that are received from the same provider, but that are not related to circuit maintenance notifications._exclude_filter: antagonist mapping to define via regex which are the notifications that must not parsed.class ABCDE(GenericProvider):
_processors: List[GenericProcessor] = [
CombinedProcessor(data_parsers=[EmailDateParser, HtmlParserABCDE1]),
]
_default_organizer = "noc@abcde.com"
_include_filter = {EMAIL_HEADER_SUBJECT: ["Scheduled Maintenance"]}
And expose the new Provider in circuit_maintenance_parser/__init__.py:
from .provider import (
GenericProvider,
ABCDE,
...
)
SUPPORTED_PROVIDERS = (
GenericProvider,
ABCDE,
...
)
Last, but not least, you should update the tests!
Parser in tests/unit/test_parsers.pyProvider logic in tests/unit/test_e2e.py… adding the necessary data samples in tests/unit/data/abcde/.
Give it a try!, as the community is growing more and more Providers are going to be added and you can benefit from all of them. Also, developing a new Provider or Parser is straightforward, and you can contribute to the library via Pull Requests or Issues, providing notifications samples to develop.
As showed in How to use it?, you can easily integrate it with any automation application only passing the notification data and selecting the Provider that should be used to parse it, and then do what you want with the structured output.
And don’t forget that you could get this integrated with Nautobot SoT using the Circuit Maintenance Plugin.
-Christian
Does this all sound amazing? Want to know more about how Network to Code can help you do this, reach out to our sales team. If you want to help make this a reality for our clients, check out our careers page.
Share details about yourself & someone from our team will reach out to you ASAP!
