NTC Rosetta and YANG for Dummies Tutorial
  |   Source

YANG for Dummies!

Most people are excited to hear about the new projects from NetworkToCode, but quickly discover that there are a lot of moving parts that present a steep learning curve. Community contributions will be critical to the success and adoption of these projects. The groundwork has already been laid for IOS and JunOS drivers, so I want to document the path to onboarding completely new drivers. This is standalone work that you can replicate without making changes to upstream libraries. I'll demonstrate this based on David Barroso's great article titled "YANG for dummies". Most of the setup files for this tutorial were plagarized from ntc-rosetta. I am definitely standing on the backs of giants!

To play along with this notebook, first run:

git clone git@github.com:dgjustice/ntc-dummies.git && cd ntc-dummies
make build_test_container
make jupyter

Then open your browser to localhost:8888.

Let's start by creating some file structure... (This is already done in the container)

  • Create yang model in data/yang/dummies/models/star-wars/napalm-star-wars.yang

    • dummies
      └── models
         ├── napalm-star-wars
         │  └── napalm-star-wars.yang
         └── napalm-star-wars-library.json
    • Create the library file (minus the comment): cat napalm-star-wars-library.json
      {
          "ietf-yang-library:modules-state": {
              "module-set-id": "6bd894f2-9168-484e-a0bf-f3ed38d864f9",
              "module": [
                  {
                      "name": "napalm-star-wars",
                      "revision": "2019-08-31",  # <- This key *must* be present, see RFC8040
                      "conformance-type": "implement"
                  }
              ]
          }
      }
      RFC8040
    • head napalm-star-wars.yang

      // module name
      module napalm-star-wars {
      
          // boilerplate
          yang-version "1";
          namespace "https://napalm-yang.readthedocs.io/yang/napalm-star-wars";
      
          prefix "napalm-star-wars";
          revision "2019-08-31" {  # <- This *must* match the revision above.
            description
                "initial version";
            reference "0.0.1";
          }
          ...

The dummy device

Our dummy device is going to implement configuration through YAML files. That's absurd you say! YAML is a horrible format! I don't disagree, but it's suitable for pet examples and it lives up to the "dummy" device name. A more practical example might be to parse a text file for device configuration or interface state. If you are using something like textfsm, you will end up with a dictionary just like the following example. The end goal regardless of the input is to start our parsing process with some kind of structured data - most commonly JSON/Python dictionaries. This could be easily replicated with a dictionary returned from a TextFSM process, for example.

In [9]:
def read_file() -> str:
    """Read a file and return the text.
    
    This method will be attached to our parser as a helper below."""
    with open("data/star_wars/universe.yml") as f:
        config = f.read()
    return config

print(read_file())
---
universe:
  individuals:
  - affiliation: REBEL_ALLIANCE
    age: 57
    name: Obi-Wan Kenobi
  - affiliation: REBEL_ALLIANCE
    age: 19
    name: Luke Skywalker
  - affiliation: EMPIRE
    age: 42
    name: Darth Vader
  - affiliation: REBEL_ALLIANCE
    age: 896
    name: Yoda

The parser

This is the code that will parse the model. Please refer to the official Yangify docs for a complete explanation and walkthrough. Also see existing implementations at the ntc-rosetta Github page. As you are developing and debugging your own parsers, keep in mind that pdb is your friend!

In [10]:
from typing import Any, Dict, Iterator, Optional, Tuple, cast
import json

from ruamel.yaml import YAML

from ntc_rosetta.helpers import json_helpers as jh

from yangify import parser
from yangify.parser.text_tree import parse_indented_config
from yangify.parser import Parser, ParserData


def to_yaml(config: str) -> Dict[str,Any]:
    yaml = YAML()
    config_data = yaml.load(config)
    return config_data


class IndividualData(Parser):
    class Yangify(ParserData):
        path = "/napalm-star-wars:individual"

        def extract_elements(self) -> Iterator[Tuple[str, Dict[str, Any]]]:
            for person in jh.query("individuals", self.native, default=[]):
                yield "individual", cast(Dict[str, Any], person)

    def name(self) -> Optional[str]:
        return jh.query("name", self.yy.native)

    def age(self) -> int:
        return jh.query("age", self.yy.native)

    def affiliation(self) -> bool:
        return jh.query("affiliation", self.yy.native)


class Universe(Parser):
    class Yangify(ParserData):
        path = "/napalm-star-wars:universe"

    individual = IndividualData


class DummyParser(parser.RootParser):
    """
    DummyParser expects as native data a dictionary where the `universe`
    key is reserved for the device configuration.
    """

    class Yangify(parser.ParserData):
        def init(self) -> None:
            self.root_native = to_yaml(self.root_native)
            self.native = self.root_native["universe"]

    universe = Universe

The driver

We need to register the driver with rosetta so we can make use of some its utility methods. We'll need to overload the get_datamodel function so we can load our custom models.

YANG models

ntc-rosetta is a framework, so we are able to bring-our-own-yang models to the party. Simply tell yangson where to find the data, and we're off to the races (or the next step of debugging)!

In [11]:
from ntc_rosetta.drivers.base import Driver
from yangson.datamodel import DataModel
import pathlib

# This part looks just like the ios and junos drivers in ntc-rosetta
class DummyDriverNapalmStarWars(Driver):
    parser = DummyParser
    translator = None
    datamodel_name = "napalm_star_wars"
    
    # These are the overloads.  Please see the file structure and notes
    # in the introduction
    @classmethod
    def get_data_model(cls) -> DataModel:
        base = pathlib.Path("/ntc_dummies/data/yang")
        lib = base.joinpath("dummies/models/napalm-star-wars-library.json")
        path = [
            base.joinpath("dummies/models/napalm-star-wars"),
        ]
        return DataModel.from_file(lib, path)
    
    @classmethod
    def get_datamodel(cls) -> DataModel:
        if cls._datamodel is None:
            cls._datamodel = cls.get_data_model()
        return cls._datamodel

Check the datamodel

Let's take a peek at the YANG model to make sure the above worked correctly

In [12]:
# If everything worked
dummy_driver = DummyDriverNapalmStarWars()
# Let's see if we properly loaded the DataModel from the new YANG file
print(dummy_driver.get_datamodel().ascii_tree())
+--rw napalm-star-wars:universe
   +--rw individual* [name]
      +--rw affiliation? <identityref>
      +--rw age? <age(uint16)>
      +--rw name <string>

Parse it

Finally, we should be able to parse our "configuration" file and get a yang datamodel of our star-wars characters!

In [13]:
config_data = read_file()

# Alternate options
# my_universe = DummyParser(dummy_driver.get_datamodel(), native=config_data)
# print(my_universe.process().raw_value())

parsed = dummy_driver.parse(native=config_data)
print(json.dumps(parsed.raw_value(), indent=4))
{
    "napalm-star-wars:universe": {
        "individual": [
            {
                "name": "Obi-Wan Kenobi",
                "age": 57,
                "affiliation": "napalm-star-wars:REBEL_ALLIANCE"
            },
            {
                "name": "Luke Skywalker",
                "age": 19,
                "affiliation": "napalm-star-wars:REBEL_ALLIANCE"
            },
            {
                "name": "Darth Vader",
                "age": 42,
                "affiliation": "napalm-star-wars:EMPIRE"
            },
            {
                "name": "Yoda",
                "age": 896,
                "affiliation": "napalm-star-wars:REBEL_ALLIANCE"
            }
        ]
    }
}

We did it!

At this point, we have a populated data model that we can manipulate using Yangson. For a deeper dive into manipulating populated data models, please see my other tutorial at the ntc-rosetta project

Do I need to use Rosetta?

Strictly speaking, no. You can look at the commented lines in the previous cell for an alternative route (you'll have to separate the get_datamodel method). Rosetta is an interface that nicely binds these dependencies together, but your particular use case may have different requirements.

Translate it

This about wraps up the story. We'll write a simple translator to spit out yaml back out into a "native config" format. Keep in mind that this is a pet example. In production, you will need to handle replaces and merges carefully as well as add tests.

In [14]:
from yangify import translator
from yangify.translator import Translator, TranslatorData
from yangify.translator.config_tree import ConfigTree


class Individual(Translator):
    """
    Implements /napalm-star-wars:universe
    """
    class Yangify(translator.TranslatorData):
        def pre_process(self) -> None:
            self.result = self.result.new_section("")

    def name(self, value: Optional[str]) -> None:
        self.yy.result.add_command(f"  - {value}")

    def age(self, value: Optional[int]) -> None:
        self.yy.result.add_command(f"    {value}")

    def affiliation(self, value: Optional[str]) -> None:
        self.yy.result.add_command(f"    {value[0]}")

class Universe(Translator):
    class Yangify(translator.TranslatorData):
        def pre_process(self) -> None:
            self.result.new_section("individuals:")
    individual = Individual
    
class DummyTranslator(translator.RootTranslator):
    class Yangify(translator.TranslatorData):
        def init(self) -> None:
            self.root_result = ConfigTree()
            self.result = self.root_result
            self.result.add_command("---")
            self.result.add_command("universe:")

        def post(self) -> None:
            self.root_result = self.root_result.to_string()

    universe = Universe

Validate the translation

This is a sanity check to see if we're on the right path. So far, everything looks good.

In [15]:
# update the driver
DummyDriverNapalmStarWars.translator = DummyTranslator
dummy_driver = DummyDriverNapalmStarWars()

# Alternate options
# my_universe = DummyTranslator(dummy_driver.get_datamodel(), candidate=parsed.raw_value())
# print(my_universe.process())

translated = dummy_driver.translate(candidate=parsed.raw_value())
print(translated)
---
universe:
individuals:
  - Obi-Wan Kenobi
    57
    REBEL_ALLIANCE
  - Luke Skywalker
    19
    REBEL_ALLIANCE
  - Darth Vader
    42
    EMPIRE
  - Yoda
    896
    REBEL_ALLIANCE

Play with the model

At this point, we've built a large pet example. Does this have any practical merit? The time I've spent on this surely hopes so! Why don't we do what we've always wished for in a Star Wars movie - Yoda the mercenary!!!

In [17]:
irt = parsed.datamodel.parse_resource_id("/napalm-star-wars:universe/individual=Yoda")
current_data = parsed.root.goto(irt)
print("Current Yoda: ", json.dumps(current_data.raw_value(), indent=2))
modify_data = current_data.raw_value()
modify_data['affiliation'] = 'napalm-star-wars:MERCENARY'

stub = current_data.update(modify_data, raw=True)
print("Candidate Yoda: ", json.dumps(stub.raw_value(), indent=2))
stub.validate()
Current Yoda:  {
  "name": "Yoda",
  "age": 896,
  "affiliation": "napalm-star-wars:REBEL_ALLIANCE"
}
Candidate Yoda:  {
  "name": "Yoda",
  "age": 896,
  "affiliation": "napalm-star-wars:MERCENARY"
}
---------------------------------------------------------------------------
YangTypeError                             Traceback (most recent call last)
<ipython-input-17-d3e1f770506c> in <module>
      7 stub = current_data.update(modify_data, raw=True)
      8 print("Candidate Yoda: ", json.dumps(stub.raw_value(), indent=2))
----> 9 stub.validate()

/usr/local/lib/python3.6/site-packages/yangson/instance.py in validate(self, scope, ctype)
    334             YangTypeError: If the value is a scalar of incorrect type.
    335         """
--> 336         self.schema_node._validate(self, scope, ctype)
    337 
    338     def add_defaults(self, ctype: ContentType = None) -> "InstanceNode":

/usr/local/lib/python3.6/site-packages/yangson/schemanode.py in _validate(self, inst, scope, ctype)
    953         """Extend the superclass method."""
    954         if isinstance(inst, ArrayEntry):
--> 955             super()._validate(inst, scope, ctype)
    956         else:
    957             if scope.value & ValidationScope.semantics.value:

/usr/local/lib/python3.6/site-packages/yangson/schemanode.py in _validate(self, inst, scope, ctype)
    779         if scope.value & ValidationScope.semantics.value:
    780             self._check_must(inst)        # must expressions
--> 781         super()._validate(inst, scope, ctype)
    782 
    783     def _default_instance(self, pnode: "InstanceNode", ctype: ContentType,

/usr/local/lib/python3.6/site-packages/yangson/schemanode.py in _validate(self, inst, scope, ctype)
    475             self._check_schema_pattern(inst, ctype)
    476         for m in inst:
--> 477             inst._member(m).validate(scope, ctype)
    478         super()._validate(inst, scope, ctype)
    479 

/usr/local/lib/python3.6/site-packages/yangson/instance.py in validate(self, scope, ctype)
    334             YangTypeError: If the value is a scalar of incorrect type.
    335         """
--> 336         self.schema_node._validate(self, scope, ctype)
    337 
    338     def add_defaults(self, ctype: ContentType = None) -> "InstanceNode":

/usr/local/lib/python3.6/site-packages/yangson/schemanode.py in _validate(self, inst, scope, ctype)
    779         if scope.value & ValidationScope.semantics.value:
    780             self._check_must(inst)        # must expressions
--> 781         super()._validate(inst, scope, ctype)
    782 
    783     def _default_instance(self, pnode: "InstanceNode", ctype: ContentType,

/usr/local/lib/python3.6/site-packages/yangson/schemanode.py in _validate(self, inst, scope, ctype)
    845                 inst.value not in self.type):
    846             raise YangTypeError(inst.json_pointer(), self.type.error_tag,
--> 847                                 self.type.error_message)
    848         if (isinstance(self.type, LinkType) and        # referential integrity
    849                 scope.value & ValidationScope.semantics.value and

YangTypeError: [/napalm-star-wars:universe/individual/3/affiliation] invalid-type: not derived from napalm-star-wars:AFFILIATION

What broke?!

Nothing!!! This is by design. If you'll take a look at the beginning of the article, we only loaded the first Napalm-Star-Wars model. We didn't load the extended universe. Fine, we'll simply push him all the way to the dark side. :)

In [18]:
modify_data['affiliation'] = 'napalm-star-wars:EMPIRE'

stub = current_data.update(modify_data, raw=True)
print("Candidate Yoda: ", json.dumps(stub.raw_value(), indent=2))
stub.validate()
print("Validated!")

translated = dummy_driver.translate(candidate=stub.top().raw_value())
print(translated)
Candidate Yoda:  {
  "name": "Yoda",
  "age": 896,
  "affiliation": "napalm-star-wars:EMPIRE"
}
Validated!
---
universe:
individuals:
  - Obi-Wan Kenobi
    57
    REBEL_ALLIANCE
  - Luke Skywalker
    19
    REBEL_ALLIANCE
  - Darth Vader
    42
    EMPIRE
  - Yoda
    896
    EMPIRE

Thank you!

As I mentioned earlier, I stand on the backs of giants. Many thanks to Ken Celenza and the folks at Network To Code, David Barroso, and others involved in these community projects!!!

Feedback

If you find any errors or want to leave any kind of feedback, feel free to leave a comment as an issue or find me lurking on the NTC Slack.