Network Automation Code Testing with pytest

Network Automation Nerds podcast. Hello and welcome to Network Automation Nerds, a podcast about Network Automation, Network Engineering, Python, and a whole lot of technology topics. I'm your host, Eric Chou. Today on the show, we'll be welcome, Adam, from Network to Code to talk about testing in Network Automation. I think this is a very important topic that is often overlooked, especially when you're just getting started out with Network Automation, for example you don't really think about your code quality, your testing. It doesn't happen until later, but it is very important topic nonetheless. And I was really happy to read about… Adam's three-part blog post on this specific topic. So very happy and excited to welcome Adam to the show. So Adam, how are you?

Thanks, Eric. I'm doing really well I'm glad you enjoyed the blog post. I totally agree with you that. It's something that's overlooked a lot and that's kind of what led me to make those blog posts uh.. to help beginners kind of.

Yeah, I think it's also a very wide topic. I mean, they're actually positions within companies that are test engineers. So therefore, a lot of times you talk about testing and you could be having a conversation with somebody for 10 minutes and you actually talk about different kind of testing. Yeah, and

just the number of stages that testing is implemented in. It's just. Astounding like, you know, regression testing, unit testing, integration testing, like system testing. It just goes on and on. And the more you learn, as with many things, you know, the more you, you learn, you don't know, because you just keep snowballing into more and more subjects.

Yeah, for sure, for sure. So that was Yeah, that was in reference to actually some sometimes, you know, I feel people anyways, like, let's let's table that because I am very excited to dive into the topic. But I always want to start with, you know, your origin story, your background and how did you get into networking and technology and all of that. So so Adam, you know, before we dive into Python testing, can you tell us a little bit about yourself? How did you get into technology? and your origin story basically. Yeah.

I'll start talking about… I guess how I got into networking. Like many people, I… I wasn't a network automation engineer my first job. I was just a regular network engineer. And I was probably about… 19 And my uncle worked for Cisco for like 22 years.

Oh my god, wow.

Yeah, he was there from like the 80s to 2000 or something like that. He was there when the bridge was just red. Yeah, he was. And I have like, he gave me a Cisco letterman jacket that he got for his like 10 year anniversary, but it was really cool. Anyway. Yeah. I was kind of deciding what I want to do post high school. He said, "You should go get your CCNA and venture into that." I was like, I don't know. But I did, so I did. So I got my CCNA and then. From there, I started getting into very, help desk was kind of my first job. Yeah. And then from there, you know, you kind of talk with some network engineers like, oh, can I look over your shoulder while you do this configuration or add a route or whatever? Yeah. So I did that for a while and then. I actually got a, it's like a tier one network position for Whole Foods in Denver, Colorado. And I was there for a few years and I wanted to increase my networking knowledge. So I studied up for CCNP and got that CCNP. And while I was there, actually.. finished my undergrad in computer science because I wanted to be a software developer, you know, I thought this networking stuff wouldn't last, I guess. I was like, this is cool. Like, I want to, you know, I want to be the cool guy making, making video games or, you know, the next Excel or I don't know, something.

Did you tell that to your uncle? This networking thing is not going anywhere. This internet fad is just going to fade really soon.

Yeah, this isn't going to go anywhere. Exactly. It did and I kept getting better jobs. And then.. after Whole Foods actually moved to where I live now. I live in Tulsa, Oklahoma and I got a job for one of the Indian nations here. We have the reservation land and stuff. Right. And for one of their casinos here, and it's a pretty small shop. There's only, you know, five network engineers. Sure. But from there. I was Because there were so few of us, we touched everything. I touched WLCs, I touched MDSs in the data center. I touched Nexus 9Ks, 2950s and access switches, just everything. Like Cisco ISE, ASAs, and I was there for.. four and a half years and that's where I really.. learned about everything Cisco had offered at a very Cisco shop. Right, right, but somewhere in there. Yeah. I was just going to say somewhere in there maybe two and a half years in, it was probably 2017, early 2017. There's a company here called QuikTrip, QT. That's a gas station company, but Network to Code actually had a contract with QT and QT had a lunch and learn about their automation journey with Network to Code up to that point, what they're planning to do, how they were doing it. I went to that. And I was like… I can combine some programming I've learned with.. this networking stuff and spent the next three years just playing with stuff every chance like every chance I got at my job you know automate everything from like updating IP phone to updating VLANs to So anything, anything I could try and automate, I did. Right. And that was kind of where that journey started That's nice.

Nice, nice. Yeah, so I don't know, our family donates buckets of money to Whole Foods every week. So it's nice to see we contribute a little bit to your college education, where we hand the money to. That's like a black hole to us. Now I know it's Amazon, but–

Yeah, now it's Amazon. Not my day. It was just Whole Foods.

Yeah, no, but it's interesting you also mentioned the the resorts, right, the casino reserve resorts. Because back when I was in Palm Desert, so of course, there's like a huge casino and Indian reserves over there. In fact, you couldn't get by, you couldn't get to the desert cities without crossing a large chunk of the reserve land. So we had to deal with that. But it's interesting you mentioned Cisco because for their network, because I talked to their network admin a lot too, back when I was working for the ISP and we had to given fiber access and all of that, it's a very closed loop system. It's never connected to the internet. I wonder why. No, but it's a closed loop. Then they're doing odds and all the slot machines or whatnot, they're monitoring, very stat driven. I was surprised that you mentioned there are Cisco shop. It's like the software wise is closed loop, but the transport is.. you know, just, just data pipes, right? Yeah.

And to speak to that, like everything Everything we got and each any appliances we stood up, you know was on prem. It was always yeah on prem Right.

That's what I heard too. Yeah.

We couldn't use any. We used. Cisco Webex or teams or whatever. It was Webex uh… That was like the only cloud thing we used. We had a DNA center instance on-prem. Microsoft Exchange was on-prem. It has to be on reservation land. So that's kind of.. kind of why they have it that way. But it allows me to touch a lot of stuff, you know, that might look like. I can stand up ISE in VMware now, I guess, if I needed to.

Right. Yeah, so I would say I would compare this to the federal services. So back in a lot of the cloud providers or even just service provider in general, they have a dedicated team that service the government and the federal government because you have had certain clearance some kind of practices that are very unique to that vertical. And I would say it's even more restrictive in the casino and gambling gaming space. But it's cool, right? I mean, you've been there for like three years, so it's not a short time. It's definitely good exposure, it sounded like.

Yeah, it was really good and helped me. Um you know, and with but kind of integrating automation and that it allowed me to do automation on other things other than just routers and switches and stuff that allowed me to play with. Cisco Unified Communications stuff and firewalls and even automating things in ISE and stuff like that which is good, I think,

Yeah, for sure. I mean, so before this podcast started, we actually go into a little bit about your college education and you wanted to and you also alluded to earlier that you wanted to be a programmer. But what? I guess what besides just kind of the flow and life, I guess, but did you I guess how did that impact your decisions later on regarding automation and how come you never went into application development or just a developer job full time in general? Yeah.

It might sound a little cheesy, but I really liked the idea that like. I had the power to connect multiple systems, I guess. You know, one. One computer system can only do so much by itself. For sure. And you need that connectivity. That's what really makes you know, computers work. So that idea behind the synergy of computers just really intrigued me. So I liked the networking stuff. I also liked that. It was kind of a black box to a lot of people. I would even say I know software developers who are just like, "Yeah, I don't know. I just hope I have internet when I plug this in. I don't know really what's going on behind the scenes." So I really like knowingly how these routers get routing peers and what protocols are used. I kind of liked having that knowledge. Yeah.

Yeah, for sure. Me too. So I got my first internship at ISP and I remember just thinking about you know That's how it works. That's just bubblegums and duct tapes, right? Like I'm surprised even like passing traffic. There we are. We're selling it for money. And no, I hear you completely. I couldn't agree more. It's fun to really know about peering exchanges it's just not

It's so many people take the internet for granted. You know, I like to know what I'm using and kind of like how it's working. Even like everything from driving a car, you know, like I'm not a mechanic, but I'll watch YouTube videos about an alternator in the car or how the engine components work just because I think it's good to know if you're using it. You never know when you would need it, right? Yeah, you never know. And then you're knowledgeable when you need to talk about it, at least a little bit. Right. Right, so like it's one those things.

Yeah, me too. So I remember The first time I wanted to buy my wife a wedding ring, I went in and they didn't know anything about it. And they were just talking about clarity. I thought it was just a matter of size, right? It's really not about that. It's about the cut. It's about clarity. It's about like… Anyway, so I went home. I did maybe 20 minutes of research. And it was back on Blue Nile too. And it was like 20 minutes of research. Next time I went in, I feel so much more knowledgeable and I feel… Like the BS level just went down by 50%. That's the thing, right? Like it doesn't take that much time, but just get your foot in there about some subject and you're, you're miles ahead. I think. Right. Which is a nice gateway to talk about pytest. So you mentioned, you know, you're a networking guy, CLI, but you were thirst for knowledge. You really wanted to know how things work, but how does that translate into pytest testing and Python? Yeah, so.

I fell victim to sorry, there's an airplane going by I fell victim to you know that As someone who starts coming from a network engineer position to an automation engineer, And even computer science class I took, testing wasn't a really thing. Yeah, I had to take an elective on software development and it was talked about. But I never really implemented it computer science is all math. All I took was math. It feels like all I took was math. But so when I first started, I didn't, I didn't test I knew it, but You just want to get stuff working. You put hours into– understanding the syntax and.. how things work, you just want to see some kind of result, you know. Yeah. You want to be able to, even if you're just changing the description on an interface. I know can tell you how many hours that took me to initially get, including research, you know, you'll read about Netmiko and how things work. And then you once you start reading about that, you start deep diving into like, Oh, man, how's this working? And what what happens when I send a list of commands? But yeah. Testing wasn't really a forethought in my mind and it was only really once I got to network to code that was they're like, oh, yeah, you know, you kind of you need this, you know, testing is part of of developing software. And that's, I think, the line between kind of someone who just scripts and someone who just who actually makes automation suites for.. for networking, at least one of the differentiator.

I mean, and it's good to see that they're preaching good hygiene, right? you're right about initially, of course, you want to get things to work. There's nothing to test if you don't have working code. But once you first get that code working, you do have to start thinking about maintainability, testability– well, not even testability, but just having other people to review your code and how can you be sure that it's working after the third generation or even a month later, you come back to it and you're not really sure what you have done and then you make a change you know, then do you just pray and that everything breaks? How can you statistically verify that it's working? Yeah and

if anyone is like me you know you write your code but you're like how do i know for sure that this is doing. There's some black magic in there. And how do I know that every time I run this, it's going to do that? That was a thought in my head as I got introduced to testing how how to think about it, I guess. You're just validating Yeah, it's going to work 99% of the time. Yeah, but it's it's Assurance to have that test there and it's just good hygiene you know Yeah.

Yeah, for sure. Also when you have external dependencies, right? So a lot of times when we do write scripts or when we write our functions, we actually are talking to the switch, we're talking to the router or UCS, whatever, APIs. When you have that dependency, it introduces variables, right? So how do you know? that when something breaks, it's not you, it's the other guy or vice versa, right? Like it is you. So, you know, better go straighten up your function or whatnot. So those are all the different things that testing could give you. Yeah, and I think

to that point in the fact that we do interact with switches and routers, they don't natively send us. Okay, now as automation develops, they do, but they don't natively send us structured data. You query at 2950, you're not querying some awesome Nexus API. You're querying, you're getting a text file with text back and There can be so many variations of.. what you could possibly get back. Right those tests, you can account for that. Yeah.

For sure. And there's more and more abstraction that you're introducing, right? Because like you said, at the very, very low level, you're actually getting a text file and output back. And it's up to the program to terminate and watch for those keywords to say, "These are the outputs that you grab back." But they abstract that for you. So you do get structured data if you use something like… Netmiko or NAPALM or whatever. But what if those modules don't work? What if you need to troubleshoot even lower than that? But anyways, so those are the testing, like I guess the whys of why we need testing. And it seems like that's once you get your network to code, they preach good hygienes. And that was an area you start to dive into.

I think I just want to add one more reason that came to my mind. Of course. Was the fact that we are so open source when you have when anyone can contribute something. You need to verify, I guess you would call regression testing that any changes or anything they add doesn't break the functionality of what's already there. So that's another important one and why for testing.

Yeah, for sure. So I think we should keep in mind that everybody who's listening to the podcast may have different familiarities and different levels of coding experience. So maybe we should dive into, so what is regression testing? Can you give us a 5,000 feet overview? Yeah, so.

regression testing, at least the way I learned it and kind of view it is it's there and it can be typically What I see is our unit test which we can talk about in-house. They test functionality of the code post, post change almost. And they are run every time. Typically they're running a CI pipeline to kind of check that validation that the code is still running. In short and sweet it's there to test that your code still functions exactly how you thought it did given any changes in the code. It's kind of a Yeah.

Yeah, for sure. So when somebody… For example, when I was working for Microsoft, typically when you release a software version, the first thing you do is just want to make sure you didn't break anything that was already existing. To me, that's my definition of testing. A regression testing was just like you didn't break anything that was working before. If you had, You better go back and just see what broke. Figure out why.

Exactly. And good tests would tell you, we're pointing to the right direction if not the precise point, right? Yeah. I mean, Adding tests in general is good hygiene, but adding effective tests and efficient tests is a whole nother iceberg. That's a new whole career path. It really is. It's crazy.

Yeah, yeah, I would say until I started working for a software company like Microsoft, I didn't know there were actual career paths for project managers, as well as testing test engineers, right? Like there's you could actually go all the way up. So typically, what I saw was in other companies I worked for it was you get developers, which is like engineering track, you get management track, and that was it, right? So you could you have your clear path tour, you know, maybe to a principal engineer or to you know, CEOs and VPs. So those are the two tracks. But when I get to Microsoft, at least back then, it was like, no, you can actually get to partners of being a PM, you can actually get to, you know, very high level as a test engineer. So there was really those were like the four verticals you can actually get to. And and it's like a testament to how deep you could get for testing. It's I would say it's

more like it's prevalent in network automation, obviously, but you know, the testing foundation comes from software development, you know, that's where it started. And I think as network automation becomes more standard. Yeah and adopted, there will be positions purely for testing network automation suites.

Yeah, at least at a certain level, right? Because I do see the ecosystem evolving really fast and just, and part of it is, as you alluded to, was because software development has been doing this for a long time. So we're kind of just taking some of the practices over adapting into the network in contact as you have done for your blog post. So Before we dive into pytest, can you just tell us the difference between what you said before, unit tests, integration testing, end-to-end testing, or system-level testing? What are the differences between those? Yeah, so.

Unit tests are testing a single point of functionality. And my initial blog post, I think it's my first one. The example I pulled was a function to standardize MAC addresses, you know, Arista, Cisco can have different formats of MAC addresses. Sure. So you may write a function. Say your boss wanted a list of all the MAC addresses or something of all the switches at some site. Sure you anyone them in the same for me you know you might need to write a function that takes input of all these different format and mac addresses and makes makes them into one and the unit has in that case is ensuring that given expected input to that function, it produces the expected output. And that's a self-contained function typically at least what I've seen in our unit tests Tidal functions. That's kind of– the way I thought of it as I first started getting into it. Sure. So.

How is that different from say integration? Oh, by the way, we're going to link all the– all of the Adam's blog posts and all the resources that we talk about in this blog post. But if you're watching it over YouTube, you have the benefit of following along the blog post as what… I always said. I was referring to your function and so on. You know, unit tests for sure. So you're testing like a little piece of code and or function or just, you know, scope wise is just code level. So what has that different from integration testing or system level testing?

Yeah, integration testing I see as testing to two things that interact in my head. An example of that would be like testing your MAC address function, but prior to that testing.. that you're able to get the MAC address table from a switch. Sure, and you might write a test that test those in conjunction or those integrated, you know, and you have an integration test. That's kind of how I view it right Yeah, so yeah, go ahead. I think. And part of the reason I wrote the blog is to kind of to think of these tests in terms of networking, because like we mentioned earlier, these do come from software development. And I think it's a little easier to think about it in software development world. An integration test might be you're testing You know, you have a webpage or whatever and you click a button and it contacts the database and send something back, you know. That's an integration test initially was easier for me to wrap my head around because you're like, Oh yeah, it's, you know, different parts of a system that are integrated and you're just making sure it works. Yeah.

Yeah, for sure. For sure. So I would agree to that 100% is just your unit testing and then the next, you know, your kind of low level. So, you know, I mean, we're a network engineer, so you're almost looking at kind of layer one or two, right? So those are your low level blocks, you're looking at very specific point that you're testing for, like does this return the MAC address I'm looking for in a good format. And then integration testing will be one level up where you're testing multiple components where maybe, like you said, a web API plus some switch and so you're integrating. And then the end-to-end testing is where you're moving up even further. So it will be bunch your integration testing tied together and you're behaving almost like a user where you're just testing the feature end to end, make sure it doesn't nothing break. So that's my own thoughts and personal opinion on different levels. But your blog specifically, all three parts of that blog deals with, you know, kind of the ins and outs of the unit test portion, correct?

Correct. That's… I think the most prevalent, at least what I've had experience with. So that's kind of why the blogs cater to that.

Yeah. Yeah. So I think that's also why I wanted to point out is because it's always easier for me to compartmentalize when I'm trying to learn something new. So let's just say you're new to testing. If you're listening to this podcast, you're new to testing. And here are all these tools about Batfish and SuzieQ or pytest and unittest module and Python. All of these, so you kind of want to just compartmentalize. So the Batfish and SuzieQ of the world, they deal with higher levels of testing. I'll go ahead and include a link in the show notes where the two founders of You know, Batfish and SuzieQ, Dinesh Dutt and Ratul Mahajan, they actually wrote a blog post about this whole overview. So that's good to compartmentalize. So if you're listening to those tools, then that's that level. But what we're talking about here is really about very detailed testing of specific code on unit tests, but in a network engineering context.

Yeah, and I almost see, you know, Batfish and SuzieQ as if we were to kind of go back to our three testing categories, almost a system test, because you're testing, you know, if you think of.. you know, X number of network devices as a system. That's kind of what those are catering to.

Yeah, I agree. And you define what that system is, but it's a system nonetheless, multiple components. Yeah. You could be a data center or it could just be a rack of switches, but nonetheless, it is interconnected. It is part of that system. Yeah, so you mentioned you specifically used pytest for your examples. So maybe just as a general overview, so there's three parts to your blog series. So can you just tell us what's the general overview, what the first part covers, what the second part is, and then what the third part is? Do you have the first one pulled up? Yeah, so fresh ones right here. So I think you talked about just first one testing that function and then fixtures. Go ahead.

Yeah, so the idea behind the first one was to one, introduce– pytest and kind of what the idea of a test function looks like. Bye. And then on top of that, there is a concept called parameterizing something. Sure. And I run over that in this blog post and the idea.. behind parameterization is actually really cool. And I didn't know this at first you know i was i was being my head when i was trying to write a test and you know And I thought to myself this function isn't always going to expect this input. How do I– test multiple inputs into the same function. And then, you know, your copy and paste, of course, right? My naive first was like, I copy this, I write it again and just change the input and I'm done. Right but then you have like a 500 line test Python file that tests like two functions. Right. But yeah, I mean, that's, that's what I, what I started. And then you learn about this parameterization. What this does is allows you to use a single test but input different sets of data into that test. Right. It's a really neat feature. The best part about it is that the assert statements and what you're testing, as you loop through this dataset, each test is its own test. Right another point i do make in the blog is that you could Pass in a list of values to your test uh- when that single test fails on the second item in that list. It's just going to stop and fail. It's not going to test all the other ones after that because you're fundamentally just running one test just with a list of data rather than one data. And I try and give those examples in that blog post because that really sets the foundation for why parameterization is so important and so widely used.

Yeah, it is a cool concept, right? And if you're just starting out in, I guess, coding, the parameters might be not as intuitive. But essentially, it's– IO OUT] You're doing 90% the same stuff, but you just want to switch up the input data. And that is what a parameter gives you in a function. And so if you overlay that concept with testing is your testing, your test stays the same, but you want to feed in different types of data to make sure like they all behave correctly without writing. If you want to feed in 50 values, you don't want to write 50.. functions or copy and paste them, you just want to swap that out and over and over again and just run them. You know, one after another really quick. Yeah. And, and like you said, right, you want them to behave as individual tests. And that's why you alluded to your, in your blog is you want to run them as different tests as opposed to, so what's it, I guess, what's the difference between running it running all of them at the same time versus running individually as different tests.

I guess you mean as when you run. Like when you use parameterization versus when you don't.

Yeah, like when you don't use parameterization, you just feed in a bunch of values versus treating each of the value as an individual test as far as coverage reports or– IO OUT]

Okay, yeah, you know like yeah, yeah, so when you obviously when you do when you just do one test with You could send in a list of 50 items, but coverage is going to treat that as one test. And if that fails, if– IO OUT] the first 49 items you pass in pass your test but the 50th fails like that whole test fails. Right as opposed to each individual test, you know, cover would say, oh, you had 49 that worked and one that didn't. Bye.

Yeah, so for example, if you're like your your example was about testing MAC address formats, right? Like so you normalize your MAC address and you want to make sure like all of the MAC addresses are formatted in a certain way just because like if you think of your host, your MAC address is probably represented in hex, hex, colon, hex, hex, whatever, right? And versus it's like four hex, colons. They're all a little different. Sometimes they're capitalized, they're not capitalized. And that's what your function does to normalize into the same way. But then you want to test out all these different scenarios that you encounter to make sure these values are feeding the same way. I turn out to normalize, actually indeed we normalize them.

Yeah, because– And I think in the blog post I might even put.. an example of a Mac address like If you were to add something to parameterization and your function doesn't– doesn't normalize that. Like it'll tell you and When you parameterize it, it's in a single test and it tells you the exact value that it failed on. And it's just rather than like, oh yeah, this whole test failed. So it's a really good. I think that's probably one of the more important blog posts. I mean, I only have three, but–

No, but imagine you have 300, right? So instead of telling you your code sucks, go fix it, it's telling you, no, your code 90% is OK. But this value, like when you treat– I don't know. I don't know what's hex go up to F, right? So G, you know, if you have a G in there, then it's, you know, anyways, yeah, it would tell you specifically where it's failing, but as opposed to just how you go, go look it up, it failed, but it will tell you where it failed.

And that kind of lends to what we said we're Writing tests is good hygiene, but also writing good tests is efficient tests and informative tests. It's kind of the same train of thought when you write you know exception handling. You know, you don't want to just like, yeah, you know, something went wrong. Sorry. I don't, I don't know what luck. Yeah. You want to try and try and send something back. So it's kind of the same idea with test

Yeah, I agree. It's more like an art than I mean, writing good ones is really more like an art. It really is. And I think.

I think that's probably one of the hardest parts of adopting running tests. And I think one of the reasons why I put it off for so long is it's not as simple as Like I have this MAC address and I want it to look like this. Like I know exactly my end goal. I know what I need to do. I can get on Stack Overflow and find how to manipulate a string in Python. And you're like, all right, that works. When you think about tests, you know– you can't really… You can't really Google how to test a function you wrote that you have to critically think about. How am I going to test this? Um and that in the beginning, that's really scary. It was very scary for me.

No scary for me too. I think that's what makes it a fantastic tester versus just somebody who's so-so. He or she is able to do stuff really elegantly and you can't really judge that, elegancy especially in testing because at the end of day, you just get.. fail or pass, right? But you know, it's only when you understand it completely, like you said, that you could start appreciate the elegance of a good test or elegance of a good test code I don't know Yeah, it It's crazy stuff. It's an art for Yeah, which brings us to a good Good question in the next question I have is.. you know. If we already have unit test module in Python, so why do we need to use pytest? So can you give us a reason why you pick pytest over just standard library unit tests? Yeah.

The main reason is when I started you know, learning unit tests and using the unit test library. Yeah. It was confusing for me. Yeah. There's a lot of boilerplate code. You know, you have to have classes it's not as succinct as pytest or Extensible for what I found. It's a lot cleaner. And you'll notice like– in the blog post when I have, to write a test function, you're more or less– writing a python function like you don't necessarily need to understand how classes work and, you know, inheriting, and you don't have to really deal with that. You're, you're writing a Python test and pytest. While yes, it adds more abstraction. Mm hmm. I think it's easier to learn and follow. Right. Yeah, I would agree with that.

Yeah, definitely. I think a lot of people would start, or I don't know if a lot of people, but many people would start with nose or unittest, which are two other kind of testing libraries or frameworks to begin with. But they'll quickly find it not as intuitive and I think I would agree with you like the learning curve is certainly higher, right? Just like it's almost like the difference between Java and Python, right? In order to just do a hello world in Java. You got to do a public or private class. You got to do the main function. You got to call that main function. You got to worry about these curly brackets. And before you even start, you just got defeated. You're just like, what is all this?

And at least for me, it tells you, like you said in Java, you have to write all this stuff. I want to know why I'm writing this stuff. So there goes another 10 hours reading about why I'm writing this class here in private versus static versus public. Yeah using the pytest gets you immediately into testing, I think. Yeah, and also

Like you said, you want to understand why. So if you're using a class, you have to use– so here's that kind of anatomy of a test function. And you will see this in Adam's blog post. So you will get– you will define something. And then you will get some kind of input. And then you will kind of compare that– input, compared to results of that input when you feed it back in, and you would either do like an equal or not equal or something to determine the success or failure of that test. Right. So those are the three kind of main parts of a test. And in unit tests, if you use class, you have to use like self dot assert, right? Self dot equals, have that, you know a certain article, whatever. So if you're truly wanting to understand, then you have to go into, you know, like object or in your programming, why you use a self versus not. So definitely that learning curve as opposed to what you will see in Adam's blog post, just a bunch of functions which is a little easier to understand.

Yeah, definitely. And one great thing of pytest is that it also does support.. unit test, you know, if if you find, you know, if you're contributing to open source or whatever, you know, high test can run unit tests classes and files and that kind of stuff. Right.

Right. So, um… So we talked about– parameters in the testing function. And we talked about pytest, which there's a lot more advantages to pytest. So I'll put a show of notes in Brian Okken's book. It's called Python Testing with pytest. The second edition is coming out pretty quickly, I think, in the next– month or two if they hold up to the January 2022. So we're recording this end of 2021. So that's a good starting point if you wanted to learn more about it. There's a lot more.. advantages to pytest over some of the other libraries and framework like integration with coverage or you could have plugins. So if you're specifically testing for Django, for example. Django is one we use heavily because of Nautobot. Yeah, for sure. Do you want to talk a little bit about pytest and plug-ins, what they are and what's the advantage?

I honestly haven't ventured too much into that. Yeah, you know, because we have our Nautobot code here. Sure. I kind of I've ran tests and kind of understand it, but I've never really developed my own Django tests or plugins for pytest. Yeah. No worries.

Yeah, no problem. Me neither. So all I know was if you're reading pytest, then there's a plugin for that. So pulling pytest dash Django or whatnot. So install that plugin and automatically it understands a lot of the native Django-based view controller or models and stuff like that. So it saves you a lot of work. So but the point to take take back is pytest is extendable. Now that the other frameworks are not, but it has a vibrant ecosystem, lots of plugins that you could just install, which ultimately saves you time and effort, right?

Yeah, and and I think it helps also. I mean, like I said, I don't have a time experience, but when I do work on Nautobot and run tests, like if you look at the test files. If you understand Django, because you have that wrapper, I guess, around objects in Django, your tests are written in a way that look like Django. You're testing Django object, and you're like, OK, that makes sense. And I don't know how it would be done without the plugin. It's probably more lines of code and not as clear. Yeah. But it gives that layer of abstraction which.. helps people. That's how it was me read the test. Yeah, yeah, for sure.

And that's just one example, right? So if you're using, I don't know, Twisted, so there's probably a plugin for that. So it's a lot closer aligned with like the formats and the syntax and Twisted and other projects. So. Yeah, for sure. So even if I just look at pytest– let me pull up that screen right here. Yeah. If I just pull up pytest, it's plug-in. And you can see there's the timeout, pep8, flakes, and all of that. So it– IO OUT] I mean, we don't pick pytest because of a popularity test. We pick it because it has a lot of ecosystem and plugins and.. that we could leverage.

Yeah, another another really cool one actually. Yeah, mine is We have a library called netutils and that's what the code does and… We it's it's basically it's a lightweight utility library for network operations adding IP addresses together, getting the last host in a subnet range, stuff like that and vote. We wanted to keep it really lightweight, so what we ended up doing.. is using doc tests within pytest. Okay, so, you know, and this kind of lends itself to us having good hygiene, all our functions require doc strings that actually test the function of that, of that test. And you know, one of the plugins of Python so that it runs that function in your in your doc string as the example. And if it's not right, you know, it's going to tell you.

Yeah. So for people who's not as familiar with DocTest, can you just tell us, explain just to the overview of how that works? Yeah, so it…

in a typical docstring, you know, there's, we use the Google. Format there might be more proper name for that. I don't know but sure you have you know, Docstring basically tells you about the function and its inputs and its outputs. So you have kind of a top section that I think is args, which are the arguments into your function. You define them and what type you expect. And then you also have a section on that, which is return, which is what the function returns and what type and a little description about it. And under that, you have an example section. And this is what that plugin looks at. And you– IO OUT] It's almost like a little.. lambda function is it just reads that runs it in the environment and and you just you pass in your input in there and the output. And it just checks against what you actually, what you're actually seeing in the doc string. Yeah, yeah.

Yeah, so that's what's really cool about it, right? You kind of self-contained both testing and.. your code. You included the expected output in your code and pytest automatically picked that up. Yeah, it's slick. I mean, we have unit tests in our own test suite.

Sure. And I think that helps with.. you know, being able to, sometimes you can't find examples for these things, you know. Right. So it's nice you just look at the docstring, you know, like, oh, okay, that makes sense. Yeah.

So that's like documentation on steroids, right? Yeah, it really is. So you get your document and you're running. Running tests on your document examples. Correct. So this is your expected output. That's your assert equal within your documentation. Yeah, so we discussed why we pick pytest because of popularity, customization, integration. And then we talked about pytest has a really cool feature on parameterizing. I guess that's a word. And you cover that in the second part of your blog. And when you talk about the third part of your blog series, you actually talk about– mocking. So can you tell us what, I guess, along with mocking, can you tell us what like fixtures are like tear down, you know, startup fixtures and then what mocks are used for? Yeah, so.

tests, any tests you run, you know the idea, especially in unit tests. There are some main points that I try and drive with unit test one you want to make sure that your test is actually testing a single unit of something. I think it's really easy to write a test but it ended up actually testing a few things, you know? Like it's, you want to make sure it's testing one thing that you're doing. And another point is you want unit tests to be fast. All right. You don't want to sit there waiting minutes for your unit test to run. And so that kind of lends to fixtures. So what fixtures are and kind of a set up tear down processes. You can define I see it as defining kind of static data that you can pass to these functions. An example being… You know, sometimes an example being– I might be writing a plugin for Nautobot and I'm expecting.. like JSON data from Nautobot, right? Right. I don't want to spin up an instance of Nautobot run that, you know, populate data and get that data out and then pass it to like a testing function. Sure. I can make a fixture that that just returns that structured data that I would be expecting. And… That's it. It just returns that and holds that. And the great thing about fixtures is that they can be used anywhere. You know, you define them somewhere and they can be used in multiple tests. So if you needed that data.. for multiple tests in your test suite. You just have to define it once and you're able to use that basically wherever you want. Yeah.

Yeah, no, that's, that's, I'm glad you brought that up, right? Because that's something that we should have covered before, but we did not, which was kind of the zen of testing. You want to always test just one thing. You want to have– that's the name unit testing. You want to test just one unit. And then you want to make it fast so that you could– you scope them correctly. And then you run them fast so that you could run lots and lots of tests without spending a lot of time wasted on– IO OUT] perhaps unnecessary stuff. So back to what that parameterization again is you don't want just one test that fail, you want to run 50 tests and specifically knowing which one failed if any of them failed. Um

Yeah, speed and unit tests is a really big one. You always get the, I don't know, when I was in school and stuff, you get the people that, you would say your code's compiling, like why aren't you working? Oh, my code's compiling, you know? And then other people say, yeah, it's running its test suite. I don't know, I can't do anything. Yeah, yeah, yeah. You wanna make a unit test test. Yeah, for sure.

You throw that over the wall and until it comes back, you can't do anything about it. Yes, yeah, go ahead. But that's kind of

what the fixtures do is they allow you to pass data into your different test functions. And I use a fixture, I believe in the second blog post. One of them, I do use a fixture. Right a very, very basic one. But the idea is that you can.. use it for anything. And I think the setup teardown I think is from unit test. I think that's a unit test thing but obviously can be using pytest and the idea that is these set up tear down functions can be used to set something up kind of like kind of like an environment for your tests. And the way I've seen it used is to instantiate like a dummy database. Yeah. We're like integration testing, you know, if you need to stand up a dummy database, you know, you want to set that you don't want to set up a dummy database for every test you want to set it up once for your test suite and then tear it down after. Yeah, so in what comment I see man

Right. So if we're overlay, just concrete example. So Adam, it's the first one that you use the fixture. So essentially, you have a non-normalized MAC address, and then you have a normalized expected MAC address. So you want to have those always available to every single test that you have. So your fixture was– Just add a decorator in front of it, and then you'll always have them for the rest of your test.

Yeah, and those are very simple ones. Obviously, they just return a string. Sure. But yeah, the idea is there. And that's one thing that pytest, one pro of pytest, you put a decorator on it and voila, it's a fixture. Yeah. It's a really cool part.

decorator is almost like magic, right? Like you put it in front of it and voila, there it is, you know. Yeah. And now it's like an API. It's nuts, some black magic there for sure. Right. And mocking is like what Adam was talking about was your you're trying to fake like a transaction. So if you imagine, as he was saying, you're going out there, you're trying to get… I don't know, like a Nexus device output back, right? But you don't always have a Nexus device. Imagine if you're running this test in a self-contained environment where you want to keep everything else constant but your own code as the variable that you're testing. You can't always go out and get that Nexus output back, what if it's down, what if it's not working or what if it's just flat out broken, has a new version, has nothing to do with what you're testing. So your marking allows you to just fake that response to keep that consistent. So first, it keeps it consistent and also it keeps it fast. You don't always have to go out and make the API call.

Definitely one of the huge pros making it fast. Yeah, mocking is a whole another iceberg thing it's not only can it be used. And I guess in the testing world, at least in the pipeline, you'll hear mocking and patching and They're kind of the same. I guess the idea behind them is the same you're faking something marking kind of like you said you're mocking I see it as mocking something like a call or.. database as you say database or something like that. Right. And that just makes it so you don have to rely on something really, you're just mocking that. Right. Hatching at least the way I think of it is that It's your… You're changing, you're mocking like like a function or class within your code. It's pretty confusing, honestly. The more you start reading about it. And that's actually my idea for my part four. Here we come. Oh, wow. Kind of the magic magic mock and how magic mock works in in in testing because the number of things you can do when you mock a function or class is insane. Okay. Nice. OK.

Yeah, so no, I was just trying to read back and I couldn't swear I saw that somewhere and said, this is the final part. But you know, you're right. I didn't say part three was the final. So I'm going to hold you up for it. Yeah. Magic mocking. All right. And that, you know, when you do that.

you can assert that that functions, you can, I don't know if I did it in one of my blog posts, but you can mock returns from specific functions within the class or you can check that when you, that a function was ran X amount of times when you called something. It's honestly like.. the amount of things you can test when you mock a function or class within your code is insane.

Yeah, yeah, for sure. Because it is a mature you know, technology is widely used. So that means it's solving all kinds of problems for people. So that means it's a lot of different branches and different edge cases that people already encounter and they kind of integrate it back into the framework or library. So definitely. There's a lot to it for sure. Yeah, so at.

Go ahead. I was going to say, yeah, there is a hopefully blog post where we'll help someone. I get that. All right.

I'm going to hold you to it and I'll look forward to that part four and part five and six. And yeah, exactly. Until you have like a whole book worth. Yeah. You just publish it. Nice. That's probably how Brian did his. Probably. Yeah. Just to start having all these examples or go, well, you know, somebody should just publish the end look in the mirror and that's what I did. But Adam, I mean, we talked a lot about why we use pytest and we throw all these end-to-end testing, integration testing and… We talked about mocking, parameterizing. So it sounded all very complicated, right? But I don't want to scare people in thinking testing needs to be this whole complicated thing. So how, you know, in your opinion, how can people get as network engineers, how can people get started with writing more tests for their code? Because like you said, it's good hygiene. Right.

I would think, well, one. With the help of pytest, you don't need to know as much. We said it has a lower barrier of entry into there. So I want you to start learning a little bit of Python. You should be pretty soon looking towards pytest. I think what I would say is think of little functions that are applicable in the network world. Mac addresses the one that always came in my head say you wanted to I will change change like a description on an interface, you might be able to.. write a test that takes in a dictionary of how Netmiko or something returns interface values and then spit it out and just make a test. The idea is, and I think with anything, is you just, especially with testing, is you just need to do it. Yeah, when you first start you're not testing, you know, you're not testing for I'm just going to think like the newest Halo because Halo came out like two days ago, but you're not writing a test that is going to be shipped out of Microsoft. You're getting familiar with it. Yeah. So even if it's just, you know, write a test that your function adds two numbers together. Bye. That's a great one to start, you know? Yeah. In fact, I think that's one of the examples. I mean, the starter guide, right? Yeah, it is actually the sum, like your sum function. Yeah. But definitely, as network engineers, you want to try and think about how you would do that our pyntc library. Yeah, it was really helpful for me as I learned more about pytest. The coverage on that repository is pretty high and The tests are a little complicated. There's a lot of mocking. But once you understand it… It really you want to kind of know what's going on it you can look at that and see. Oh yeah, like you know, this is how testing something in the networking world looks like. Right, and that helps I think.

Yeah, and I especially like the fact that you put this into the network engineering world, right? Because I think context does matter a whole lot here. It's like, I could read 10,000 Python examples, but none of them deal with what I have to deal with on a daily basis. IP address, MAC address, interface description, BGP, that sort of stuff. But once you– put those into the context that it's relatable and you could see how you could help you.

Yeah, and now, you know, when I was getting into it, you know, I would Google pytest and like you said, everything's make sure your sum function works, make sure your square function works. That doesn't– OK, I kind of understand, but I don't– I want to, you know, I'm a network engineer at Core. I need better examples that help me netutils or other maybe you can you know put that link is those tests are actually pretty simple Those would be great tests to look at and kind of understand because the functions are so simple. Yeah. Yeah, the idea was to get it in the networking world because it's hard enough wrapping your head around it when the context doesn't even apply. It only adds more difficulty to that.

Right. You're like, Hey, I have a calculator. I don't need to test that sum function. I know it works. Yeah. Like, I can add, you know, in fact, my 80 year old could add better than that. You know, like, why am I spending hours, you know, filling in with the sum function? But, but no, you're right. So I think what I liked two parts what I like about Adam's blogs and just NTC blogs in general. One is that it's put into network engineering context. And second is it scope it down enough so that it's understandable and relevant. So like you said, you don't have to be a programmer in order to get started with testing. So for example, in his first blog post, he has a what is it four lines of code to define a function to normalize your your MAC address and then he has another three lines of code for that test function to excluding your fixtures and all that, but another three lines. So for a total of six, seven lines, you're able to do tests. And those are the kind of getting started that we need is putting into the correct context and make it simple enough to understand so that we could explore the new stuff, right? Which is pytest, but we already know MAC address. We already know… Python functions, but we just need to learn pytest that's relevant to us. Yep, and that. You know, that helped me kind of wrap my head around it.

I guess another goal would be to IP addresses, maybe changing a CIDR to an actual.. actual octet of values and testing that.

Right. Hey, you know, it's a good idea. So changing like slash notation, right? So a lot of times, even like, I think it was Nexus or maybe it was IOS XE, like they used a slash notation, right? Slash 24, slash 27, 28, versus like a subnet mask, 255, 255, 248, whatever. So maybe we could, or not we, you, maybe you could write some test function and illustrate how pytest could validate that your function is working. But I'm not assigning more work to you, I'm just brainstorming. Yeah.

The more you think about it, you actually start realizing, wow, there's a lot of things I could actually test for, you know, because there's a lot of ambiguity and the networking especially when you get like cross vendors. Yeah, maybe everything in the city, even within Cisco, you know, like Nexus versus IOS versus IOS XR all that stuff.

Yeah, yeah, they're just they're a big umbrella called Cisco, but underneath it's like individual little mushrooms and they'll look very different. Yeah, kind of annoying at times. You said it, I didn't. But no, I agree. Definitely. It's like, yeah. And you're not even talking about like Meraki or UCS, right? Like we're just talking about routers and switches and there's already much differentiation there. Yeah, so Adam, we cover a whole bunch, pytest, we cover about different levels of testing, different features, mocking. So is there any call to action for you on just testing in general?

What do you mean?

I think for people we talk about getting started with testing, is there any call to action? Once you get past that, do you want to direct people to write their own test? Is it like you said, looking at more NTC libraries?

call to action, like just get started testing? I think, well, one, get started testing locally. Sure. And then, you know, get stuff working. But the next best thing I think would be.. to try and actually make a contribution to one of our repositories. Okay, we have a ton of open source repositories. Right? That's a good call to action. And making a PR, you're going to get, you know, we're going to look at your code and there's going to be people with way more experience and in pytest than myself looking at your code and you'll learn and I think that's a really.. really good thing. I even do that here, you know. We're big enough now that I don't know every repository we have going on, you know, and someone else might be working on something. But if I don't know that technology or want to better understand what that repository is doing, you know, I'll try and add a feature or refactor some code or something and make a PR because one, you have to understand what's going on to typically you have to write some kind of test for that as well.

Yeah, I would be you know probably an NTC one, but obviously it doesn't have to be one of our open sources, hundreds of open sources projects out there. And they all test, I can assure you, they all test. Alright, that would be my don't submit your PR without some tests, right? And then that's a good practice ground to actually bounce the idea. So I think that's a Another thing that it's intimidating for people to start with is making a PR and like you said, having people look at your code, but don't be, right? Like that's another way to learn and that's another way to just communicate is just pick out those brain and you guys have a common thing to work at. I know I learned the most when I, I learned the most about Ansible when I finally submit my PR or modules to merge upstream. Writing that module was a very good learning experience to begin with. But talking to the maintainer, that was an even better way to know their thought process, their structure. So don't worry about it. We're all in the same boat. So make a PR. Maybe Adam will review your code. And you know how he's a nice guy.

That's the thing, you know, you open a PR, you're not going to break anything. You know, you're not going to merge a PR that's broken.

Right, hopefully. Hopefully, yeah, I'm sure there have been make a different branch and then you're for sure you're not merging. Yeah. Yeah.

But that's the thing, you will learn. And it is intimidating, but– 99% of the people that have reviewed my PRs are nice about it you know they don I'm not mean about it. They just want you to learn. And they were in that same position at some point too.

Yeah. Yeah. I would roger that for like just open source project in general, right? 99% of the time it's good. And for that 1%, the benefit far outweighs that 1%. Yeah, it does.

So yeah, that'd be my call to action, I I even do that here internally to repos. I don't know that I'm not personally working on, but I want to learn more about.

Nice, nice. No, that's a good call to action. Sorry to put you on the spot. That was a nice save. I think that's, that's good. That's why. That's what I would Okay. Cool. Hey, Adam, I had a great time chatting about testing. Like I said, I think it's one of the overlooked subject, but it's important to at least know about these different terms and stuff, right? So thanks for taking the time and thanks for being on the show.

Yeah, no, thank you for having me. It was a pleasure.

So thanks for listening to a Network Automation Nerds podcast today. Find us on Apple Podcasts, Google Podcasts, Spotify, and all the other podcast platforms. Until next time. Bye bye. Bye Adam. Bye.