This is part 3 of a blog series to help learn how to contribute to Batfish.

The previous posts in this series:

• Developing Batfish – Developer Summary (Part 1)
• Developing Batfish – Extending a Grammar (Part 2)

In this post I will be covering how to take the parsed data and apply that “text” data into a vendor-specific (VS) datamodel. I will also demonstrate how to take the extraction test we created in part 2 and extend it to test the extraction logic.

Basic Steps

1. Create or Enhance a Datamodel
2. Extract Text to Datamodel
3. Add Extraction Testing

What Is the Vendor-Specific Datamodel?

The title of this blog post is Converting Config Text into Structured Data. Throughout this blog post I will be talking about the vendor-specific (VS) datamodel, which is the schema for the structured data. Modeling data is complicated; fortunately, the maturity of the Batfish project offers an extensive number of datamodels that already exist in the source code that can help with enhancing the datamodel I need to extract the route target (RT) data for EVPN/VxLAN.

The VS datamodel is used to map/model a feature based on how a specific vendor has implemented a technology. These datamodels tend to line up closely with how that vendor’s configuration stanzas line up for that technology.

As far as terminology, within Batfish I’ve noticed the names datamodel and representation are used somewhat freely and interchangeably. I will stick to datamodel throughout the blog post to avoid confusion.

Create or Enhance a Datamodel

As I finished up part 2 of this blog series, we had updated the parsing tree to support three new commands. We added simple parsing Testconfig files to ensure that ANTLR could successfully parse the new commands. In this post I will build upon what we did previously. I will start with extending the switch-options datamodel to support the features we added parsing for. To rehash, the commands we added parsing for are below:

set switch-options vrf-target target:65320:7999999
set switch-options vrf-target auto
set switch-options vrf-target import target:65320:7999999
set switch-options vrf-target export target:65320:7999999

The current switch-options model is comprised of:

public class SwitchOptions implements Serializable {

  private String _vtepSourceInterface;
  private RouteDistinguisher _routeDistinguisher;

  public String getVtepSourceInterface() {
    return _vtepSourceInterface;
  }

  public RouteDistinguisher getRouteDistinguisher() {
    return _routeDistinguisher;
  }

  public void setVtepSourceInterface(String vtepSourceInterface) {
    _vtepSourceInterface = vtepSourceInterface;
  }

  public void setRouteDistinguisher(RouteDistinguisher routeDistinguisher) {
    _routeDistinguisher = routeDistinguisher;
  }
}

This file is located in the representation directory.

The datamodel is describing what Batfish supports within the Junos switch-options configuration stanza. I need to extend this to support and add vrf-target. To do this, I need to define the type of the data and create getters and setters.

The next step is to identify how to use this data and the best way to represent the data. The easiest of these would be the auto. This command will either be on or off. If we parse the configuration and we have the ANTLR token for auto, we can set that in the datamodel as true; otherwise we would have it set to false and would expect to see one of the other commands. The other commands would be of type ExtendedCommunity, which is already defined as part of the Batfish vendor-independent datamodel.

In this command stanza the auto keyword can be used OR the community can be provided. For this I will create an representation for ExtendedCommunityorAuto which has already been created for this exact scenario in the Cisco NX-OS representations.

Enhance the Datamodel

Before I can extract the text data from the parsing tree and apply it to a model, the datamodel must be updated to support the additional feature set. For this example I will be adding a support for vrf-target and the three different options that are possible. The result of the update is shown below:

public class SwitchOptions implements Serializable {

  private String _vtepSourceInterface;
  private RouteDistinguisher _routeDistinguisher;
  private ExtendedCommunityOrAuto _vrfTargetCommunityorAuto;
  private ExtendedCommunity _vrfTargetImport;
  private ExtendedCommunity _vrfTargetExport;

  public String getVtepSourceInterface() {
    return _vtepSourceInterface;
  }

  public RouteDistinguisher getRouteDistinguisher() {
    return _routeDistinguisher;
  }

  public ExtendedCommunityOrAuto getVrfTargetCommunityorAuto() {
    return _vrfTargetCommunityorAuto;
  }

  public ExtendedCommunity getVrfTargetImport() {
    return _vrfTargetImport;
  }

  public ExtendedCommunity getVrfTargetExport() {
    return _vrfTargetExport;
  }

  public void setVtepSourceInterface(String vtepSourceInterface) {
    _vtepSourceInterface = vtepSourceInterface;
  }

  public void setRouteDistinguisher(RouteDistinguisher routeDistinguisher) {
    _routeDistinguisher = routeDistinguisher;
  }

  public void setVrfTargetCommunityorAuto(ExtendedCommunityOrAuto vrfTargetCommunityorAuto) {
    _vrfTargetCommunityorAuto = vrfTargetCommunityorAuto;
  }

  public void setVrfTargetImport(ExtendedCommunity vrfTargetImport) {
    _vrfTargetImport = vrfTargetImport;
  }

  public void setVrfTargetExport(ExtendedCommunity vrfTargetExport) {
    _vrfTargetExport = vrfTargetExport;
  }
}

In this example I have added a getter and a setter for each new dataset. This will give me the ability to extract the data from the configuration and instantiate the switch-option vendor-specific object. One important thing to notice is the use of ExtendedCommunityOrAuto. This did not exist in the Junos representation, since it existed in Cisco NX-OS I used the same representation code.

This representation is shown below:

public final class ExtendedCommunityOrAuto implements Serializable {

  private static final ExtendedCommunityOrAuto AUTO = new ExtendedCommunityOrAuto(null);

  public static ExtendedCommunityOrAuto auto() {
    return AUTO;
  }

  public static ExtendedCommunityOrAuto of(@Nonnull ExtendedCommunity extendedCommunity) {
    return new ExtendedCommunityOrAuto(extendedCommunity);
  }

  public boolean isAuto() {
    return _extendedCommunity == null;
  }

  @Nullable
  public ExtendedCommunity getExtendedCommunity() {
    return _extendedCommunity;
  }

  //////////////////////////////////////////
  ///// Private implementation details /////
  //////////////////////////////////////////

  private ExtendedCommunityOrAuto(@Nullable ExtendedCommunity ec) {
    _extendedCommunity = ec;
  }

  @Override
  public boolean equals(Object o) {
    if (this == o) {
      return true;
    } else if (!(o instanceof ExtendedCommunityOrAuto)) {
      return false;
    }
    ExtendedCommunityOrAuto that = (ExtendedCommunityOrAuto) o;
    return Objects.equals(_extendedCommunity, that._extendedCommunity);
  }

  @Override
  public int hashCode() {
    return Objects.hashCode(_extendedCommunity);
  }

  @Nullable private final ExtendedCommunity _extendedCommunity;
}

This allows for the VS model to have one field, and when it is set to auto or a specific community, it clears the other by changing the value of that single field.

Extract Text to Datamodel

In this section I will explain how to extract data from the parsing tree and assign it to the vendor-specific datamodel. This work is completed within the ConfigurationBuilder.java file.

ConfigurationBuilder.java is located in the grammar directory.

Note: For hierarchical configurations (Junos OS and PanOS) it’s ConfigurationBuilder.java. For most other vendors it’s actually <vendor>ControlPlaneExtractor.java. In order to see this, visit the <vendor>ControlPlaneExtractor.java (CPE) file within the grammar directory mentioned above.

The first extraction I’m going to focus on is the vrf-target auto command. In order to extract this command I need to create a Java method that takes the parser context as an input, and I will extract and analyze the data in order to assign it to the datamodel I enhanced earlier.

The first step is to import the parsing tree context.

import org.batfish.grammar.flatjuniper.FlatJuniperParser.Sovt_autoContext;

Next we can create an enter or an exit rule to extract and assign the data.

@Override
public void exitSovt_auto(Sovt_autoContext ctx) {
  if (ctx.getText() != null) {
    _currentLogicalSystem.getOrInitSwitchOptions().setVrfTargetCommunityorAuto(ExtendedCommunityOrAuto.auto());
  }
}

In this method I am accessing the Sovt_autoContext from the parser. And if the ctx variables getText() method is not null, I’m assigning the value of VrfTargetCommunityorAuto in the switch-option model to auto, meaning that feature is turned on.

This is something that confused me when I was initially learning how the conversions worked. I had to sit back and remember that in cases like set switch-options vrf-target auto, it will either exist in the configuration or it won’t; therefore, the parsing context would be null when it does not exist in the configuration.

It is also worth mentioning that this is an exit rule, which is the most common. If some processing is needed (e.g., set variable values) before the child rules are processed, an enter rule can be used.

To expand on an enter rule, imagine a similar configuration stanza in Junos, which is set protocols evpn vni-options vni 11009 vrf-target target:65320:11009. In this case I’d need to set a variable for the VNI that is being configured so that I can reference it later when I need to assign the route target for the VNI. This is an example where an enter rule could be used to assign the VNI as a variable that the child rules can use.

These concepts are followed in a similar manner for each extraction you need. I will not cover every different extraction for the commands in the post in order to keep it as terse as possible; however, below is an example of the extraction created for the set switch-option vrf-target target:65320:7999999 command.

The interesting data from this command is the route target community. In order to extract that, I have the following method:

import org.batfish.grammar.flatjuniper.FlatJuniperParser.Sovt_community_targetContext;

@Override
public void exitSovt_community(Sovt_communityContext ctx) {
  if (ctx.extended_community() != null) {
    _currentLogicalSystem
        .getOrInitSwitchOptions()
        .setVrfTargetCommunityorAuto(ExtendedCommunityOrAuto.of(ExtendedCommunity.parse(ctx.extended_community().getText())));
  }
}

First I validate the context is not null. Then I set the vrfTargetCommunity to the value that was parsed. One thing to notice in the code snippet above is that since my datamodel set VrfTargetCommunityorAuto to the type of ExtendedCommunity, I’m parsing the getText() value into an ExtendedCommunity. For the remaining few commands, the extraction methods will be very similar; so I will not be showing the remaining two conversions for import and export targets.

Add Extraction Testing

Now that I have the conversions written, I need to update the tests that I wrote in part 2 of this blog series. The test I created to validate the parsing of the Testconfig file is shown below:

@Test
public void testSwitchOptionsVrfTargetAutoExtraction() {
  parseJuniperConfig("juniper-so-vrf-target-auto");
}

Now I must extend this test in order to test the extraction of the vrf-target auto configuration. The test as shown above simply validates that the configuration line can be parsed by ANTLR. It does not validate the code snippets we wrote in the previous section that are taking the “text” data and saving it to the datamodel. The test I want to write is to validate that the context extraction is working and I can assert that when the command is found it is set to auto.

@Test
public void testSwitchOptionsVrfTargetAutoExtraction() {
  JuniperConfiguration juniperConfiguration = parseJuniperConfig("juniper-so-vrf-target-auto");
  ExtendedCommunityOrAuto targetOrAuto = juniperConfiguration.getMasterLogicalSystem().getSwitchOptions().getVrfTargetCommunityorAuto();
  assertThat(ExtendedCommunityOrAuto.auto(), equalTo(targetOrAuto));
  assertThat(true, equalTo(targetOrAuto.isAuto()));
  assertThat(juniperConfiguration.getMasterLogicalSystem().getSwitchOptions().getVrfTargetExport(), nullValue());
  assertThat(
      juniperConfiguration.getMasterLogicalSystem().getSwitchOptions().getVrfTargetImport(), nullValue());
}

In order to test the conversion, I’m using the same function and just extending it to pull data out of the parsed Juniper configuration. For this Testconfig I only have the set switch-options vrf-target auto command. As seen in the extraction test, I’m asserting that isAuto is true, and that the value of targetOrAuto is ExtendedCommunityOrAuto.auto(). The remaining options are not located in that Testconfig file, and therefore I am asserting their values are null.

Since I also created and explained the vrfTargetCommunity, the test for this extraction is shown below:

@Test
public void testSwitchOptionsVrfTargetTargetExtraction() {
  JuniperConfiguration juniperConfiguration = parseJuniperConfig("juniper-so-vrf-target-target");
  ExtendedCommunityOrAuto extcomm = juniperConfiguration.getMasterLogicalSystem().getSwitchOptions().getVrfTargetCommunityorAuto();
  assertThat(ExtendedCommunity.parse("target:65320:7999999"), equalTo(extcomm.getExtendedCommunity()));
  assertThat(false, equalTo(extcomm.isAuto()));
  assertThat(
      juniperConfiguration.getMasterLogicalSystem().getSwitchOptions().getVrfTargetImport(), nullValue());
  assertThat(
      juniperConfiguration.getMasterLogicalSystem().getSwitchOptions().getVrfTargetExport(), nullValue());
}

The logic I’m using is very similar. In this case I’m testing that the extracted ExtendedCommunity matches what I have in the Testconfig file, but I’m also validating that the rest of the switch-options that do not exist in the Testconfig files are null. For the remaining import and export rules, I created similar tests to validate the extraction of those ExtendedCommunity values.

Note: Batfish developers tend to use more Matchers in their tests, they almost never use assertTrue/assertNull; often it’s assertThat(getFoo(), nullValue()). Hamcrest Matchers tend to do a better job of explaining the mismatches than JUnit (e.g., assertThat(someList(), hasSize(5)) will be much better than assertTrue(someList().size() == 5).

Summary

In this post I provided more details on what a vendor-specific datamodel is and how it fits within the Batfish application. I identified that the switch-options datamodel/representation needs to be extended to support the new variables I needed. Next, I wrote and explained how to extract the “text” data and assign it to the datamodel. And finally, I explained and showed how to write some extraction tests to validate the extractions are working as intended.

This is part 2 of a blog series to help learn how to contribute to Batfish. Here’s the previous post in this series: Developing Batfish – Developer Summary (Part 1).

In this post I will be covering how to extend a grammar definition in detail, which includes updating the ANTLR files and adding proper testing to validate the parsing of the new configuration commands is working as expected. There are entire books and blog posts that cover the in-depth details of ANTLR, so this blog will focus on the pieces needed for a successful Batfish contribution.

What Is a Grammar?

In part 1 of this series I covered a generic definition of what a grammar is. Below covers some of the more in-depth grammar related concepts:

• Lexer – A lexer (often called a scanner) breaks up an input stream of characters into vocabulary symbols for a parser, which applies a grammatical structure to that symbol stream. I think of this as the different command elements, whether that is a parent line or sub parent (e.g., interface, vlan, prefix-list). The lexer needs to have all the elements built out for the parsing to work accurately. Lexer rules define token types. They start with an uppercase letter to distinguish them from parser rules.
• Parser Rules – A parse tree is made up of the structure of parser rules. Parser rules are written in lowercase. ANTLR has detailed documentation on parser rules.

Basic Steps

1. Create a testconfig file with the lines to be parsed
2. Update grammar definition and lexer
3. Add extraction tests

Extending a grammar merges concepts between part 2 and part 3 of this blog series. Specifically with the structured data representation and extraction testing, which commonly requires the representations to be created.

Command Additions

In this blog post I will be covering the steps to add parser rules for a few commands that I need access to for future enhancements. These commands live in the switch-options Junos stanza.

The commands I’d like to add parsing support for are:

set switch-options vrf-target target:65320:7999999
set switch-options vrf-target auto
set switch-options vrf-target import target:65320:7999999
set switch-options vrf-target export target:65320:7999999

More information on these commands can be found in the Juniper documentation.

At the start of this blog, the FlatJuniper_switch_options.g4 file has the contents:

Note: This file lives here in the Batfish directory structure.

parser grammar FlatJuniper_switch_options;

import FlatJuniper_common;

options {
   tokenVocab = FlatJuniperLexer;
}

s_switch_options
:
  SWITCH_OPTIONS
    (
      so_vtep_source_interface
      | so_route_distinguisher
      | so_vrf_target
      | so_vrf_export
      | so_vrf_import
    )
;

so_vtep_source_interface
:
  VTEP_SOURCE_INTERFACE iface = interface_id
;

so_route_distinguisher
:
  ROUTE_DISTINGUISHER route_distinguisher
;

so_vrf_target
:
  VRF_TARGET null_filler
;

so_vrf_export
:
  VRF_EXPORT null_filler
;

so_vrf_import
:
  VRF_IMPORT null_filler
;

You can see that vrf-target, export, and import sections of the commands are going to null_filler, which is implemented in Batfish specifically for Juniper to raise the error about the command syntax not being supported. null_filler is defined in FlatJuniper_common.g4 and shown below:

null_filler
:
  ~( APPLY_GROUPS | NEWLINE )* apply_groups?
;

Note that null_filler is specific to Juniper within Batfish, and each vendor has its own definition/implementation of similar logic.

We will work on revamping this file and adding the necessary support for these command sections.

Create a Testconfig

As is true with most large-scale projects, testing is required and is an integral piece to be included in any contribution to a project. The first step we need to complete is to create a simple Testconfig file that Batfish’s testing environment can pull in and attempt to parse. The commands in our Testconfigs are very simple. I will create a file per command; this will give me added flexibility when adding testing.

First, I will create a Testconfig file called juniper-so-vrf-target-auto. This file is located in this directory.

I will test the auto vrf-target first.

#
set system host-name juniper-so-vrf-target-auto
#
set switch-options vrf-target auto
#

These files are simple test-configuration files, and they don’t need to be complicated. Their purpose is to test that a command can be parsed as expected. In this example I have a host-name and the command I want to parse out.

Note: A good way to determine how many Testconfig files are needed is to think about whether or not certain configuration lines will overwrite each other. If the configurations can be grouped or scoped a single Testconfig can be used.

Updating the Grammar

Updating this grammar should be straightforward. I want to extend what is mentioned above to become our starting point for the switch-options grammar.

The updated grammar definition is shown below. I will be explaining the changes in the next section.

parser grammar FlatJuniper_switch_options;

import FlatJuniper_common;

options {
   tokenVocab = FlatJuniperLexer;
}

s_switch_options
:
  SWITCH_OPTIONS
    (
      so_route_distinguisher
      | so_vrf_target
      | so_vtep_source_interface
   )
;

so_vrf_target:
   VRF_TARGET
        (
          sovt_auto
          | sovt_community
          | sovt_export
          | sovt_import
        )
;

so_vtep_source_interface
:
  VTEP_SOURCE_INTERFACE iface = interface_id
;

so_route_distinguisher
:
  ROUTE_DISTINGUISHER route_distinguisher
;

sovt_auto
:
  AUTO
;

sovt_community
:
   extended_community
;

sovt_export
:
   EXPORT extended_community
;

sovt_import
:
   IMPORT extended_community
;

Understanding the Grammar Updates

To help with the visualization, a diff is shown below:

▶ sdiff -bBWs before.g4 after.g4
      so_vtep_source_interface				      |	      so_route_distinguisher
      | so_route_distinguisher				      <
      | so_vrf_export					      |	      | so_vtep_source_interface
      | so_vrf_import					      |	   )
							      >	;
							      >
							      >	so_vrf_target:
							      >	   VRF_TARGET
							      >	        (
							      >	          sovt_auto
							      >	          | sovt_community
							      >	          | sovt_export
							      >	          | sovt_import
so_vrf_target						      |	sovt_auto
  VRF_TARGET null_filler				      |	  AUTO
so_vrf_export						      |	sovt_community
  VRF_EXPORT null_filler				      |	   extended_community
so_vrf_import						      |	sovt_export
  VRF_IMPORT null_filler				      |	   EXPORT extended_community
							      >
							      >	sovt_import
							      >	:
							      >	   IMPORT extended_community
							      >	;

I’ll go through the changes from top to bottom:

• First, I show the change from so_vrf_target to VRF_TARGET. This one was actually an overall fix from the initial implementation I did in a previous PR into Batfish. Instead of defining so_vrf_target from scratch, I looked through the lexer and noticed that VRF_TARGET was already defined with a Lexer mode: M_VrfTarget push rule that I will be covering in the testing section below. In this case it made sense to reuse what was already built.
• Next, I added the OR block to catch the multiple different command syntaxes that are supported.
• sovt_auto will support the syntax set switch-options vrf-target auto.
• The sovt_community definition is next; it supports set switch-options vrf-target target:65320:7999999. The rule also reuses extended_community which is defined in FlatJuniper_commmon.g4.
• Finally, the last two are catching statically defined import or export community targets. These simply allow for the command set switch-options vrf-target import target:65320:7999999 or the identical command replacing import with export.

Note that the name of the rules has changed to follow the batfish standard. sovt in this case would be switch-options vrf-target. I also rearranged the rules to be alphabetical within each context block.

Add Boilerplate for Testing

The actual testing of the extraction code is going to be covered in part 3 of this series. For the purposes of validating the grammar I will demonstrate how to create a simple test. This will validate that Batfish can parse the configuration lines. In order to do this we create our Testconfig files and attempt to run the parseJuniperConfig class. At a bare minimum this can ensure the parsing of the ANTLR tree is successful.

This extraction test will be created in FlatJuniperGrammarTest.java. It will test the auto option via the config line set switch-options vrf-target auto.

  @Test
  public void testSwitchOptionsVrfTargetAutoExtraction() {
    parseJuniperConfig("juniper-so-vrf-target-auto");
  }

As seen above, I simply call the parseJuniperConfig class and pass my Testconfig filename into it.

When I run this test I should get PASSED.

Using IntelliJ I can easily execute the single test right within the application. When I run the test, the output below is shown.

Testing started at 2:57 PM ...
<omitted>

INFO: Elapsed time: 6.295s, Critical Path: 5.87s
INFO: 5 processes: 1 internal, 3 darwin-sandbox, 1 worker.
INFO: Build completed successfully, 5 total actions
//projects/batfish/src/test/java/org/batfish/grammar/flatjuniper:tests   PASSED in 2.6s

<omitted>

I see the test has a status of PASSED, which validates the Testconfig file could be parsed by ANTLR.

To demonstrate a failure and how to use the output to help troubleshoot, I’m purposely updating the Testconfig file to have the command set switch-options vrf-target nauto notice nauto instead of auto. I realize this is not a valid config in Junos, and it would never be in show configuration | display set output. This is used for demonstration purposes only.

Executed 1 out of 1 test: 1 fails locally.
INFO: Build completed, 1 test FAILED, 4 total actions
INFO: Build Event Protocol files produced successfully.
INFO: Build completed, 1 test FAILED, 4 total actions

Parser error
org.batfish.main.ParserBatfishException: 
  <omitted>
Caused by: org.batfish.common.DebugBatfishException: 
lexer: FlatJuniperLexer: line 4:30: token recognition error at: 'n'
Current rule stack: '[s_switch_options s_common statement set_line_tail set_line flat_juniper_configuration]'.
Current rule starts at: line: 4, col 4
Parse tree for current rule:
(s_switch_options
  SWITCH_OPTIONS:'switch-options')
Lexer mode: M_VrfTarget
Lexer state variables:
markWildcards: false
Error context lines:
   1:      #
   2:      set system host-name juniper-so-vrf-target-auto
   3:      #
>>>4:      set switch-options vrf-target nauto
   5:      #

  <omitted>
	at org.batfish.grammar.flatjuniper.FlatJuniperCombinedParser.parse(FlatJuniperCombinedParser.java:12)
	at org.batfish.main.Batfish.parse(Batfish.java:410)
	... 36 more

Quite a bit of output here was omitted, but some helpful parts remain.

1. There is a DebugBatfishException getting raised, and on the next line we get some details on where the issue lies. lexer: FlatJuniperLexer: line 4:30: token recognition error at: 'n'.
2. The current rule stack can help you trace the grammar resolution order that was followed. In this case, the most recent grammar where the failure occurred is s_switch_options.
3. The parser tree rule details are shown, and they include helpful information about Lexer mode and the current rule that failed.
4. Error context lines shows the exact line that failed to parse.

According to this information we can determine that the parsing tree got into the Lexer mode: M_VrfTarget.

Taking a step back to the VRF_TARGET token, which resolves in FlatJuniperLexer.g4, it defines the new mode to push into.

VRF_TARGET
:
   'vrf-target' -> pushMode ( M_VrfTarget )
;

This means that if vrf-target is found in the command output, it will push the lexer into a new mode called M_VrfTarget. In order to troubleshoot that further, I look in the FlatJuniperLexer.g4 for M_VrfTarget.

mode M_VrfTarget;

M_VrfTarget_COLON: ':' -> type ( COLON );
M_VrfTarget_DEC: F_Digit+ -> type ( DEC );
M_VrfTarget_AUTO: 'auto' -> type ( AUTO );
M_VrfTarget_EXPORT: 'export' -> type ( EXPORT );
M_VrfTarget_IMPORT: 'import' -> type ( IMPORT );
M_VrfTarget_L: 'L' -> type ( L );
M_VrfTarget_NEWLINE: F_NewlineChar+ -> type(NEWLINE), popMode;
M_VrfTarget_PERIOD: '.' -> type ( PERIOD );
M_VrfTarget_TARGET: 'target' -> type ( TARGET );
M_VrfTarget_WS: F_WhitespaceChar+ -> channel ( HIDDEN );

This lexer mode allows for additional flexibility in the parser tree when the outputs are more complex using sublexers. More details on mode, types, channels can be found in the ANTLR README.

At a high level this mode allows for any of these types to be found until the NEWLINE is found, in which the popMode will return the lexer back to the previous context.

Now that I see the mode definition for vrf-target clause, it’s evident why my Testconfig is failing. This mode does not allow for the token nauto; therefore, it’s raising the exception seen in the stack trace. If I needed to add additional types, I would define them in the same manner as the others. For this example I could add M_VrfTarget_AUTO: 'nauto' -> type ( NAUTO ); and defined NAUTO: 'nauto'; in FlatJuniperLexer.g4.

It’s important to understand that I’m showing a Lexer mode in this example to show its flexibility. I want to be clear that lexer modes are not the common case. They’re needed only when you want to limit what is lexed after a command; most commands do not need their own lexer mode.

Summary

In this post I provide more details on what a grammar is and how we can define and/or update an existing parser file. I explained the new commands that I wanted to add, along with updating an existing parser file to support the additional commands. In order to to validate the parsing additions I created a simplified test which we will add on to in the next blog post. Finally, I touched on how to utilize push and pop modes to add more flexibility to the lexer context.

In the next post I will be covering how to extract and use the parsed token data to create structured data in a Junos vendor datamodel. Once the datamodel is enhanced to support the additional command data, I will cover how to extend the conversion test we wrote in this blog post to test the datamodel instead of just the simple file parsing capabilities.