Developing Batfish – Converting Config Text into Structured Data (Part 3)

This is part 3 of a blog series to help learn how to contribute to Batfish.

The previous posts in this series:

• Developing Batfish – Developer Summary (Part 1)
• Developing Batfish – Extending a Grammar (Part 2)

In this post I will be covering how to take the parsed data and apply that “text” data into a vendor-specific (VS) datamodel. I will also demonstrate how to take the extraction test we created in part 2 and extend it to test the extraction logic.

Basic Steps

1. Create or Enhance a Datamodel
2. Extract Text to Datamodel
3. Add Extraction Testing

What Is the Vendor-Specific Datamodel?

The title of this blog post is Converting Config Text into Structured Data. Throughout this blog post I will be talking about the vendor-specific (VS) datamodel, which is the schema for the structured data. Modeling data is complicated; fortunately, the maturity of the Batfish project offers an extensive number of datamodels that already exist in the source code that can help with enhancing the datamodel I need to extract the route target (RT) data for EVPN/VxLAN.

The VS datamodel is used to map/model a feature based on how a specific vendor has implemented a technology. These datamodels tend to line up closely with how that vendor’s configuration stanzas line up for that technology.

As far as terminology, within Batfish I’ve noticed the names datamodel and representation are used somewhat freely and interchangeably. I will stick to datamodel throughout the blog post to avoid confusion.

Create or Enhance a Datamodel

As I finished up part 2 of this blog series, we had updated the parsing tree to support three new commands. We added simple parsing Testconfig files to ensure that ANTLR could successfully parse the new commands. In this post I will build upon what we did previously. I will start with extending the switch-options datamodel to support the features we added parsing for. To rehash, the commands we added parsing for are below:

set switch-options vrf-target target:65320:7999999
set switch-options vrf-target auto
set switch-options vrf-target import target:65320:7999999
set switch-options vrf-target export target:65320:7999999

The current switch-options model is comprised of:

public class SwitchOptions implements Serializable {

  private String _vtepSourceInterface;
  private RouteDistinguisher _routeDistinguisher;

  public String getVtepSourceInterface() {
    return _vtepSourceInterface;
  }

  public RouteDistinguisher getRouteDistinguisher() {
    return _routeDistinguisher;
  }

  public void setVtepSourceInterface(String vtepSourceInterface) {
    _vtepSourceInterface = vtepSourceInterface;
  }

  public void setRouteDistinguisher(RouteDistinguisher routeDistinguisher) {
    _routeDistinguisher = routeDistinguisher;
  }
}

This file is located in the representation directory.

The datamodel is describing what Batfish supports within the Junos switch-options configuration stanza. I need to extend this to support and add vrf-target. To do this, I need to define the type of the data and create getters and setters.

The next step is to identify how to use this data and the best way to represent the data. The easiest of these would be the auto. This command will either be on or off. If we parse the configuration and we have the ANTLR token for auto, we can set that in the datamodel as true; otherwise we would have it set to false and would expect to see one of the other commands. The other commands would be of type ExtendedCommunity, which is already defined as part of the Batfish vendor-independent datamodel.

In this command stanza the auto keyword can be used OR the community can be provided. For this I will create an representation for ExtendedCommunityorAuto which has already been created for this exact scenario in the Cisco NX-OS representations.

Enhance the Datamodel

Before I can extract the text data from the parsing tree and apply it to a model, the datamodel must be updated to support the additional feature set. For this example I will be adding a support for vrf-target and the three different options that are possible. The result of the update is shown below:

public class SwitchOptions implements Serializable {

  private String _vtepSourceInterface;
  private RouteDistinguisher _routeDistinguisher;
  private ExtendedCommunityOrAuto _vrfTargetCommunityorAuto;
  private ExtendedCommunity _vrfTargetImport;
  private ExtendedCommunity _vrfTargetExport;

  public String getVtepSourceInterface() {
    return _vtepSourceInterface;
  }

  public RouteDistinguisher getRouteDistinguisher() {
    return _routeDistinguisher;
  }

  public ExtendedCommunityOrAuto getVrfTargetCommunityorAuto() {
    return _vrfTargetCommunityorAuto;
  }

  public ExtendedCommunity getVrfTargetImport() {
    return _vrfTargetImport;
  }

  public ExtendedCommunity getVrfTargetExport() {
    return _vrfTargetExport;
  }

  public void setVtepSourceInterface(String vtepSourceInterface) {
    _vtepSourceInterface = vtepSourceInterface;
  }

  public void setRouteDistinguisher(RouteDistinguisher routeDistinguisher) {
    _routeDistinguisher = routeDistinguisher;
  }

  public void setVrfTargetCommunityorAuto(ExtendedCommunityOrAuto vrfTargetCommunityorAuto) {
    _vrfTargetCommunityorAuto = vrfTargetCommunityorAuto;
  }

  public void setVrfTargetImport(ExtendedCommunity vrfTargetImport) {
    _vrfTargetImport = vrfTargetImport;
  }

  public void setVrfTargetExport(ExtendedCommunity vrfTargetExport) {
    _vrfTargetExport = vrfTargetExport;
  }
}

In this example I have added a getter and a setter for each new dataset. This will give me the ability to extract the data from the configuration and instantiate the switch-option vendor-specific object. One important thing to notice is the use of ExtendedCommunityOrAuto. This did not exist in the Junos representation, since it existed in Cisco NX-OS I used the same representation code.

This representation is shown below:

public final class ExtendedCommunityOrAuto implements Serializable {

  private static final ExtendedCommunityOrAuto AUTO = new ExtendedCommunityOrAuto(null);

  public static ExtendedCommunityOrAuto auto() {
    return AUTO;
  }

  public static ExtendedCommunityOrAuto of(@Nonnull ExtendedCommunity extendedCommunity) {
    return new ExtendedCommunityOrAuto(extendedCommunity);
  }

  public boolean isAuto() {
    return _extendedCommunity == null;
  }

  @Nullable
  public ExtendedCommunity getExtendedCommunity() {
    return _extendedCommunity;
  }

  //////////////////////////////////////////
  ///// Private implementation details /////
  //////////////////////////////////////////

  private ExtendedCommunityOrAuto(@Nullable ExtendedCommunity ec) {
    _extendedCommunity = ec;
  }

  @Override
  public boolean equals(Object o) {
    if (this == o) {
      return true;
    } else if (!(o instanceof ExtendedCommunityOrAuto)) {
      return false;
    }
    ExtendedCommunityOrAuto that = (ExtendedCommunityOrAuto) o;
    return Objects.equals(_extendedCommunity, that._extendedCommunity);
  }

  @Override
  public int hashCode() {
    return Objects.hashCode(_extendedCommunity);
  }

  @Nullable private final ExtendedCommunity _extendedCommunity;
}

This allows for the VS model to have one field, and when it is set to auto or a specific community, it clears the other by changing the value of that single field.

Extract Text to Datamodel

In this section I will explain how to extract data from the parsing tree and assign it to the vendor-specific datamodel. This work is completed within the ConfigurationBuilder.java file.

ConfigurationBuilder.java is located in the grammar directory.

Note: For hierarchical configurations (Junos OS and PanOS) it’s ConfigurationBuilder.java. For most other vendors it’s actually <vendor>ControlPlaneExtractor.java. In order to see this, visit the <vendor>ControlPlaneExtractor.java (CPE) file within the grammar directory mentioned above.

The first extraction I’m going to focus on is the vrf-target auto command. In order to extract this command I need to create a Java method that takes the parser context as an input, and I will extract and analyze the data in order to assign it to the datamodel I enhanced earlier.

The first step is to import the parsing tree context.

import org.batfish.grammar.flatjuniper.FlatJuniperParser.Sovt_autoContext;

Next we can create an enter or an exit rule to extract and assign the data.

@Override
public void exitSovt_auto(Sovt_autoContext ctx) {
  if (ctx.getText() != null) {
    _currentLogicalSystem.getOrInitSwitchOptions().setVrfTargetCommunityorAuto(ExtendedCommunityOrAuto.auto());
  }
}

In this method I am accessing the Sovt_autoContext from the parser. And if the ctx variables getText() method is not null, I’m assigning the value of VrfTargetCommunityorAuto in the switch-option model to auto, meaning that feature is turned on.

This is something that confused me when I was initially learning how the conversions worked. I had to sit back and remember that in cases like set switch-options vrf-target auto, it will either exist in the configuration or it won’t; therefore, the parsing context would be null when it does not exist in the configuration.

It is also worth mentioning that this is an exit rule, which is the most common. If some processing is needed (e.g., set variable values) before the child rules are processed, an enter rule can be used.

To expand on an enter rule, imagine a similar configuration stanza in Junos, which is set protocols evpn vni-options vni 11009 vrf-target target:65320:11009. In this case I’d need to set a variable for the VNI that is being configured so that I can reference it later when I need to assign the route target for the VNI. This is an example where an enter rule could be used to assign the VNI as a variable that the child rules can use.

These concepts are followed in a similar manner for each extraction you need. I will not cover every different extraction for the commands in the post in order to keep it as terse as possible; however, below is an example of the extraction created for the set switch-option vrf-target target:65320:7999999 command.

The interesting data from this command is the route target community. In order to extract that, I have the following method:

import org.batfish.grammar.flatjuniper.FlatJuniperParser.Sovt_community_targetContext;

@Override
public void exitSovt_community(Sovt_communityContext ctx) {
  if (ctx.extended_community() != null) {
    _currentLogicalSystem
        .getOrInitSwitchOptions()
        .setVrfTargetCommunityorAuto(ExtendedCommunityOrAuto.of(ExtendedCommunity.parse(ctx.extended_community().getText())));
  }
}

First I validate the context is not null. Then I set the vrfTargetCommunity to the value that was parsed. One thing to notice in the code snippet above is that since my datamodel set VrfTargetCommunityorAuto to the type of ExtendedCommunity, I’m parsing the getText() value into an ExtendedCommunity. For the remaining few commands, the extraction methods will be very similar; so I will not be showing the remaining two conversions for import and export targets.

Add Extraction Testing

Now that I have the conversions written, I need to update the tests that I wrote in part 2 of this blog series. The test I created to validate the parsing of the Testconfig file is shown below:

@Test
public void testSwitchOptionsVrfTargetAutoExtraction() {
  parseJuniperConfig("juniper-so-vrf-target-auto");
}

Now I must extend this test in order to test the extraction of the vrf-target auto configuration. The test as shown above simply validates that the configuration line can be parsed by ANTLR. It does not validate the code snippets we wrote in the previous section that are taking the “text” data and saving it to the datamodel. The test I want to write is to validate that the context extraction is working and I can assert that when the command is found it is set to auto.

@Test
public void testSwitchOptionsVrfTargetAutoExtraction() {
  JuniperConfiguration juniperConfiguration = parseJuniperConfig("juniper-so-vrf-target-auto");
  ExtendedCommunityOrAuto targetOrAuto = juniperConfiguration.getMasterLogicalSystem().getSwitchOptions().getVrfTargetCommunityorAuto();
  assertThat(ExtendedCommunityOrAuto.auto(), equalTo(targetOrAuto));
  assertThat(true, equalTo(targetOrAuto.isAuto()));
  assertThat(juniperConfiguration.getMasterLogicalSystem().getSwitchOptions().getVrfTargetExport(), nullValue());
  assertThat(
      juniperConfiguration.getMasterLogicalSystem().getSwitchOptions().getVrfTargetImport(), nullValue());
}

In order to test the conversion, I’m using the same function and just extending it to pull data out of the parsed Juniper configuration. For this Testconfig I only have the set switch-options vrf-target auto command. As seen in the extraction test, I’m asserting that isAuto is true, and that the value of targetOrAuto is ExtendedCommunityOrAuto.auto(). The remaining options are not located in that Testconfig file, and therefore I am asserting their values are null.

Since I also created and explained the vrfTargetCommunity, the test for this extraction is shown below:

@Test
public void testSwitchOptionsVrfTargetTargetExtraction() {
  JuniperConfiguration juniperConfiguration = parseJuniperConfig("juniper-so-vrf-target-target");
  ExtendedCommunityOrAuto extcomm = juniperConfiguration.getMasterLogicalSystem().getSwitchOptions().getVrfTargetCommunityorAuto();
  assertThat(ExtendedCommunity.parse("target:65320:7999999"), equalTo(extcomm.getExtendedCommunity()));
  assertThat(false, equalTo(extcomm.isAuto()));
  assertThat(
      juniperConfiguration.getMasterLogicalSystem().getSwitchOptions().getVrfTargetImport(), nullValue());
  assertThat(
      juniperConfiguration.getMasterLogicalSystem().getSwitchOptions().getVrfTargetExport(), nullValue());
}

The logic I’m using is very similar. In this case I’m testing that the extracted ExtendedCommunity matches what I have in the Testconfig file, but I’m also validating that the rest of the switch-options that do not exist in the Testconfig files are null. For the remaining import and export rules, I created similar tests to validate the extraction of those ExtendedCommunity values.

Note: Batfish developers tend to use more Matchers in their tests, they almost never use assertTrue/assertNull; often it’s assertThat(getFoo(), nullValue()). Hamcrest Matchers tend to do a better job of explaining the mismatches than JUnit (e.g., assertThat(someList(), hasSize(5)) will be much better than assertTrue(someList().size() == 5).

Summary

In this post I provided more details on what a vendor-specific datamodel is and how it fits within the Batfish application. I identified that the switch-options datamodel/representation needs to be extended to support the new variables I needed. Next, I wrote and explained how to extract the “text” data and assign it to the datamodel. And finally, I explained and showed how to write some extraction tests to validate the extractions are working as intended.