Josh VanDeraa on Starting Your Network Automation Journey, Part 1

Network Automation Nerds podcast. Hello and welcome to Network Automation Nerds podcast, a podcast about network automation, network engineering, Python, and a lot of technology topics. I'm your host, Eric Chou. Today on the show, we'll be talking Josh VanDeraa, managing consultant at Network to Code, and my fellow network automation enthusiasts. Josh shared his passion for network automation with open source project, speaking at various conferences, and also his book, Open Source Network Management. I am super excited to have Josh on the show today and I know we'll have a great time chatting. So let's dive right in. Welcome Josh.

Thank you for having me. Awesome day here, so.

Yeah, you know, I think we've, you know, first of all, I bought your book and so and then we also, you know, kind of chatted here and there over Twitter. Obviously, it's a small community. So I'm very excited to finally, you know, kind of meet you in person virtually and kind of pick your brain a little bit.

Thank you. Well, first, thank you for the book purchase there. And yeah, I think I started to get involved in heavily network automation community just as the COVID timeframe kind of showed up. So getting together at a conference or something just wasn't one of those things that was going to happen too soon. But hopefully soon we can meet in person yet. So yeah, absolutely. Right. Like, absolutely. I think.

So my third edition of the book came out just right around Covid. It was like… 2020, like February or 2020, and the last conference I went to happened to be PyCascades. And I was able to actually have Guido van Rossum. I'm like, hold up my book, likely. But that was it, right? And then like three years after, boom, there's no in-person, at least for me, you know, I have young kids. So I try not to, try to be on the cautious side. So, you know, looks like it's coming back up and hopefully it'll stay that way.

We all kind of took that in-person thing for granted for a while, I think. Indeed, indeed. Very, very similar. It's again, got a young family at home myself and it's get the opportunity to take the right opportunities. So yeah, for sure.

So so Josh, you mentioned you got into like the network automation bit right around early 2020, I would say. I think if that were remembered correctly. But before we get started, tell us a little bit about yourself. How did you get into technology? What's your origin story?

Yeah, going back, I think I've always been that kid that's been interested in tech going going way back ahead. I have been fortunate enough to be able to have exposure to an Apple IIe. And growing up in grade school and at home a little bit, so I I've always been into like sheets as well and whatever the early Apple. predecessor was. I still remember doing things in that as a youngster. So what was going through did a lot of programming and then all of a sudden found out about networking. And I said, Oh my gosh, networking is awesome. Why have I not heard about this before?

So how was that like, right? So I think, you know, for the guests on the show, there's like kind of two different paths. One is I would say majority of them, you know, goes from network engineering and maybe not even like a tech background before then. Right. Like, so that's like me, you know, I actually took my computer to Best Buy to have my RAM replaced. Right. Like that's how bare I was before I even know about networking and then go into programming. And then on the other side, we have maybe like 30% of the guests, like yourself, they have computer. maybe programming or maybe system integration as a background, then goes into networking. So how was that like?

Yeah, so I did go to college for programming or not for programming, it was a telecommunication systems, but it was the Cisco Networking Academy, more or less. And so got exposure to many Cisco systems right from the get-go. It has CS minor. I always did well, but I… Once I got into the industry, I basically said, "All right, skip the CS stuff that I had done," because it was C++ at the time. It wasn't something that was of huge interest at the time. And so got into a large retailer and did second level support there for a while, supported all pieces of hardware. servers, network gear, desktops, you name it, it was supported by us, even some serial based items. –

Like printers, like your parallel printers. –

No, I'm serially connected. just the items that give you price updates out on the retail floor. Those had some serial connections and had to do some troubleshooting. Some people, good people on the street took care of actually being in front of it, but it wasn't quite IP.

Yeah, I know what you're talking about because I did. I did. So so I don't know what. So you mentioned computer minor, right? So what was your major in college? It was telecommunication systems. OK, so interesting. So it was like actually telecom was your major and then CS was in the minor. So yeah, so this is a little different. So I don't think my school actually even. have a telecom major, but I was a finance major. Yeah. Very few places, so. Yeah, yeah, so. So yeah, so I was going to say like, yes, I did an internship in various locations and one of it was like retail and I was like, it's this place all Kmart. I don't know. I don't know if you're familiar with it. And that was like bankrupt and there's like no Kmart's anymore. But it's like Walmart, right? But it's just called Kmart.

Yeah, Kmart is a nice two block walk for me growing up. So.

Oh, okay. Nice. Yeah. I don't know if you remember it, but it was like a big K, right? It was like a big K and then like the I think they call it super K or whatever. But Yeah, so I'm familiar with what you're talking about, like the scanners and also like the pricing check. All of these are proprietary pinouts and these stuff that have to be in a particular place. And yeah, it was painful.

Yeah, it brings some nightmares to me. That's for sure.

All right, then let's quickly move on from there. So you mentioned network engineering. It was in college. You did help desk, second level support, and go into engineering. So how did you go from maybe second level support into network engineering proper?

Yeah, so there it was about a three, three and a half year journey for me. Okay. Short. So kudos. Thank you. I mean, it was a long time in the seat myself. It felt like a long time. But in the in the grand scheme of things, yeah, still pretty short. Yeah. And so at the job, there was an opportunity to when they needed some additional help internally, hey, let's do an internal internship type of thing. So got some experience of doing a replacement of a metro area network. So DWDM was definitely not something that I learned about originally. And so I got a little bit of DWDM experience. And after the project was done. and it became a promotion type thing of once the role became available. So a little bit of waiting the time. And then I got a lot of exposure on all things except for data centers, I think was at that place where I got into the wide area network team and took care of responsibility for 2000 stores, the wide area network and what that design looked like. jumped over to a wireless team and so, you know, totally different technologies and jumping around. And then finally– –

That's very interesting. How would you recommend people? Because I would imagine a lot of people listening are thinking about career advancement, professional development, that sort of stuff. And you obviously, if I'm hearing that correctly, it's kind of you have a diverse interest, right? Like you're not worrying about pay or you're not worrying about job description, but you just kind of branch out. So how would you recommend people if they wanted to do that? How would you network and then get those exposure and opportunity?

Yeah. you know, it's, it's the people side of things so that, you know, so talking about people, it's always been. keeping an eye on what else is out there in the environment saying, okay, there's gonna be usually an opening that's gonna be coming up. And so it's not, you know, it's not just around that time of the opening, but being able to be around and be present as part of the organization. So I think that's one my… One of the things that I've usually brought is that even though I've got my narrow focus of what I should be looking at, I'm keeping an eye on the big picture. And that means interacting back in things in an office in person, hey, maybe going and catching a happy hour and being able to converse and just meet with other people. as well that might not be on your team directly. Oh yeah, like being on a podcast with some stranger from across the nation. Always glad to do so, you know, get gained out and yeah, it's you never know what doors will be opened by just going out and talking to other people. Right.

I love that. I love that. I mean, that's the same way with me. It's always like, you know, people talk about like a T shape person. So you go, you know, wide and then you go very deep on that. And I've also heard about like an M shape person, right? areas like for yourself, maybe telecom and, you know, Python or maybe some other open source network, project Ansible, for example, then yeah, so then you kind of have those business sense as well, because being a consultant, I imagine that's a lot of the language that you need to get equated to.

Yeah, indeed. It's, you know, that's one of the things that actually just has helped me out tremendously as a consultant was it was that retail industry was 13 years of my life. Right. So it was a bit, quite a bit of time and understanding at the enterprise, what's going on at that level. And then how can, how can I help and be able to bring network automation along. You know, it's, want to eventually talk a little bit more about crawl, walk, run type things. And it's, I know the first thing that as I look at is I know in large enterprises of, hey, all of a sudden something, something breaks something. There's usually a reaction and you want to make sure that you're not the one. breaking something early. We'll fix it really quickly. Indeed. With automation specifically, AAA is one of the things that you, everybody knows that got AAA and how you log into devices, but then when you apply automation, Okay, you're usually thinking about how you connect to the devices. Yeah, but not thinking about the triple A server on the back end and making sure that doesn't tip over. And so that's one of those fine balances. I have seen triple A tip over a few times in my career. And you know, that doesn't recover very quick. So yeah, yeah, make sure you have that out of bounds.

I mean, I'll be the first to admit that I have to drive to like hub sites more than one time just because I locked myself out with the console and you know, God forbid that those dial-in servers would work and you haven't checked. So for sure. So you mentioned network automation, right? And you also mentioned you have this diverse interest. You're always keeping an eye out for what's on the horizon, open-minded, talk to people, fixing problems. So how did you go from second-level support into network engineering, doing DWDM stuff, then now getting into network automation? So how did that get started for you?

Yeah, it does go back. There was a strong focus to want to bring automation. This is back 2015 into the environment that we're at. I definitely want to give a shout out to one Jason Davis at Cisco. He's often you see him at Cisco Live and doing a few other things nowadays as well. he was brought in not once but twice to kind of cover Python. And at that point, I'm like. That looks awfully awesome. I can actually, the code that was presented on the screen, I could read and understand what it was doing. And so it wasn't so much just C++ you have over here, you have to set up all the memory, all the variables. –

Oh my God, garbage collection. –

You see a whole bunch in Python, it's just here and it does that. And so it… That was kind of my introduction to Python. And it's been just all the way since then, just keep on diving in into various areas. Did a lot of Python originally at the retail organization. And I was on the firewall team at that point. So. I'll get into one of my favorite automations that I wrote early in my career. There was just looking at the We had a table that was overflowing and causing impact. Just all of a sudden in the firewall.

Table meaning like a TCAM table or like a routing table or access list.

Yeah, like a TCAM table. It was around RPCs. RPC table is filling up. We have a filed a bug with the vendor. It's definitely a bug. It's going to be fixed eventually. It has to go through all the QA and developments of how to fix this. And there is no–

Like six months later, right? Right. There is no–

There wasn't anything and I said, "Hey, I've got this fresh skill of network automation type thing that I'm doing. Why don't I just go ahead and write an automation that logs into the firewall every half hour, checks the size of the table." That was the first iteration. I said, "All right, awesome. I'm getting counters so I can know when it's about to tip over." Then eventually, I just moved into hey, I'm already logged in. I've issued this command every time that I've seen it in an errored state. Why don't I just have automation issue that command? Right. And so that helps save from having to do a code upgrade. It probably would have been an emergency patch, so not one that has the rigorous QA testing. It's a vendor supplied special patch. And that was just not something I wanted to do at the time.

You know, like that kind of reminded me of the Atomic Habits book, right, by James Clear. It's like what he recommended was just do something incrementally. And so, for example, the example he gave was just going to the gym, like go to the gym five minutes and just get out, right? Like you don't even have to sit. You just get there. So now it kind of matches what you were saying. You know, what he talked about was going to the gym is like, oh, you're there already. Why don't you do like some… some bench press, right? And then the bench press turns into some curls and some curls turn into some squats. And so there you were, you know, you log into the device and you're just checking the table. But yeah, why don't I just like clear that table real quick?

Yeah, that's exactly what it was. And now, I mean, now I probably also look at, okay, how can we monitor this table? Watch it, watch it go. But that's the iteration, right? Yes. In the journey of that. I wasn't at the, I wasn't much past the crawl stage. I was still just at my infancy on this. And so I was fortunate enough to have, have learned that just in time. cause it saved me from many of weekends having to log in. I'll say that much. So it was, and then also set up the notification, you know, I never did get to sign up notification. Hey, that I cleared the table and tell someone that I did it. But it was all just, I heard years later, hey, do we need to do anything special for this type thing? Even though I wasn't there anymore. So that was a you know Five years later, still one of my favorite stories.

Oh, totally. That was a great story. I think that kind of win is really important, especially early on to encourage you to go through that process, see the validation. I mean, I won't name any names, but I did at one point, you know, use automation to actually clear BGP's, right? Because, or like bounce BGP's here and there just to have it into… a clean state. I mean, that's that's where most networks fall over is when the state– I mean, not just networks, right? OS systems, right? For example, like your memory leaks or whatever. That's why reboot always works, because it goes back into a known state, like a clean state. So the state out of whack is really the case. And like you said, you know– you just have to, sometimes you just have to make things work. So clearing table is fine, resetting BGP peers that if that gets you some like extra sleep at night, go for it.

Exactly. It's about, it helps to provide that work life balance, right? That is still very, very important for me and for that, to be able to have that family time and then be able to get away from the family as you need or when the time is right, right? So you're not disturbed on that side of things. So.

Yeah. Or just your personal life, family life, personal life. It's all lumped together. Yeah. Whatever is important to you, right? Yes. Yeah. If you just go relax and kick back, that's all the power to you as soon as you get things done. And automation is one of these things that will probably get you there faster. So you mentioned both network automation and Python. And it's kind of interesting that you're probably like the fifth person from Network to Code. I haven't done my count, right? But probably like four or five times already that we have people from Network to Code. So can you tell us a little bit about your day-to-day at Network to Code as a consultant for Network Automation specifically? –

Yeah, definitely. So my day, so I'm a management consultant. So I've got a team of engineers, awesome team. I absolutely love working with them. And we're got multiple customers right throughout the day of just helping them along in their automation journey. And so every client's at a different stage. Some are just a little bit above crawling. Some are already in the run phase before the fly stage of crawling is just getting started, walking, hey, you've got some scripts that are running and maybe being passed along. And then running being, hey, you've got these scripts that are now in a system that is there before you get into the full self-service and full loopback. or feedback loops type things. It's going to be in the best part is it's. pretty much every automation technology that's out there. Quite the breadth. Have some Ansible projects. We've got some Python projects. Dig into a lot with Network to Code and our sponsorship of the Nautobot projects. A lot of Django as well as far as providing APIs and starting to work with data. starting to look at the data side. The data is the new oil and the network is full of lots of information. Right.

Yeah, I mean, that's something that I, interestingly enough, it's like every road goes back to data. So for me, you know, I attend doing security analysis that's networking related. Definitely, you know, like we actually have like millions and millions of record just on a daily basis. And, you know, it's close to like a bill, right? Like I couldn't even imagine back then. And so a lot of things breaks at scale. as you know. So so it's like keeping data and networking, like you said, it's just data but actionable data, right? Like, you know, what does this data mean? And what does it mean for me? Those are the true benchmark of how you're using that data, right? Data by itself probably means less.

Indeed. And, you know, I mentioned data and looking at things on scale, right? Yeah. When you're talking about the size of the records you're seeing there at eight, at eight 10, right? It's you can't really load that into Excel. Right? The 65,000 row limitation. Yeah, I think would be, yeah. So anyways, yeah. Yeah, you have to, you have to, yeah. put an exception into the app to allow it to go bigger. And then even then you don't know if your machine will be able to still run after you try to run it. So, or load it up. And so that's another place that I've seen and, you know, ask where do we get started at some point and using Python just to parse through that data.

Yeah. Right. So yeah. So you talked about helping clients and having these three different stages. So maybe depending on their journey of automation maturity. So no right or wrong. It was basically just this continuum. And how would you help people? How would you determine first? Like if I'm just a– I don't know what anybody else does, right? Like I know we have some scripts and coming in and I would just talk to this person called Josh about helping me automate my network. What are the steps and how would you go approach that? –

Yeah, so steps there is, it gets back to the human side of things. –

Back to that retail skills that you developed. –

Even before you get into the tech side of things, you need to know what's important to the organization. What are they measured on? Because usually measurements should come back and get rolled up in the grander scheme of the organization, providing value. And so how can we… one that measure what those metrics are at and then how do we… How do we measure it and then how do we improve it? So you talk about process, you have to document the process. Is there a process documented? Should you be making a change record in an ITIL system? Sometimes changing a VLAN on a switch at some organizations might require a change record. Other organizations might have to go through a full CAB process for… just a VLAN change depending on where it is. Right. So it's understanding that process and seeing how they do that. Is that someone you give it to the intern to log into the switch and make sure you have switchport VLAN add or is this an automated script somewhere that's already in place? So you never know what. where the journey someone is.

Yeah, like you just reminded me of like adding a VLAN to the trunk, whether that you have to add the keyword or you just override everything. And it's like, oh, well, you know, I guess that's a, I hope, I hope that, you know, the bro is clear because I'm just going to go, go drive to the console or something.

Yes, I know I've been on, you know, I'm sure I've probably done that early in my career, perhaps, and even not so long ago when I had an L2 WAN provided and all of a sudden every circuit except for one circuit, and that's the new one that's being provisioned, went down and every other one, you know, so just that one is up. All right, that's a… All right, it happened, you know, we've all been there. Yeah. Or we've all at least heard about that. My friend had told me. Yes. Yes. Yes. Yeah. Those are like interesting war stories that you hear. But you know, hopefully it's just like you do that once, right? And then from there on, you know, you got to add the keywords and

Yeah, even as a young engineer, I didn't even know that you have to clean the air filter of the routers. So back then, GSR was this beast, right? I learned something now here and I know I haven't done that. Yeah, that's not something you could really automate, right? Because you had to go physically pull that out. Yeah, so that's just kind of interesting. Brings me back of all these memories. And just to finish that thought, yeah, so GSR was like this either quarter of a rack or half a rack. And it goes up to the full rack and this beast. But there's the airflow. And so of course, it was, I think it was front– back to front airflow or something like that. And so there's obviously air filters. And data center tend to be pretty dirty. And so once in a while, you actually have to go pull out the filter and then just like your HVAC system at home. So yeah, if you don't clean those, then eventually it's going to cause a problem and it's going to clog up and overheat and so on. And then that's when you have some interesting intermittent outage. And the tech people, you open a tech case and they'll be like– I don't know, you know, like this just goes out here every once in a while. We don't really know what's going

Yeah, that might be the actual static electricity piece. But, you know, then just to finish the thought on how we get in and go through the process, right? And then we're always thinking with the business in mind. Yeah. Right. As much, you know, one of my personal lines is you can have the greatest network in the world, but if you don't have any applications running over it, there isn't necessarily why do you have that network? Yeah, even your home network. That's how we're doing this particular podcast is over home networks. So there are applications absolutely in most, every network there is. So we are very focused on getting outcomes that provide value as part of our engagement.

Nice. I love that. I think that's something that maybe being a pure technical engineer wouldn't be primarily focused on, right? Because you're actually several layers removed from the business side of things. So you're worried about adding VLANs. You're worrying about having bits or traverse over your wires. But at the same time, at the end of the day, it's– you're not getting, the business doesn't derive value from the connectivity. It derives value from the service that running on top of connectivity. So it's absolutely critical as you stated to get those, what are those metrics that you're measuring and then make sure you're improving those key metrics. –

Yep, and metrics speak volumes, right? From whether it's a financial metric of what is the cost of this particular piece, or if it's a metric around, hey, we need to have packets getting round trip in 15 milliseconds from point A to point Z, right? And so that's where the various levels of engineering and automation come in of just, okay, whether it's measuring metrics or if it's, you know, reporting on the metrics.

I was going to say like speaking from the true telecom engineer, right? Like typically, if you say from A to B, like point A to point B, but A to Z, that's like, you know, that's your telecoms coming up.

Yep. And in Minneapolis, I'm in Minneapolis, which I'd say is a tech edge type thing. Like Cloudflare just turned up a new speed test. And my, my place for it is I'm connecting out to Chicago. Right. So yeah, there's then usually a whole bunch of other hops in between there. Right. So the Z part is absolutely further, not just to A and B.

Yeah. Yeah. Yeah. No, I remember when I was like buying fibers and, you know, getting it's always like A to Z and, you know, like they're just skipping and obviously, you know, there's a bunch of stuff in the middle. DWDM maybe, right? Indeed. Yeah. So what are some of the big obstacles in your path to talking to the customers and the different maturity levels? What are some of the obstacles you see from people getting into automation themselves?

Yeah, I think one of the biggest things is the unknown, right? That's I like to say that, especially for our new hires coming in, we've seen quite a bit already. And so we that's where the sound of the experience comes in. But then it's it's just it is getting that experience. You know, I was fortunate enough early in my automation journey to have some lab devices, I think either lab or still some GNS3 type thing. Just and test things, right? Yeah. And be able to do debug commands if you want to get real down into lower levels of what's going on. And so it's been… that assuasion that the first thing and then they also oftentimes a lot of people think too big. It's one of the big things, network automation is big. It's a big elephant. There's so that it's okay. How do you eat an elephant? Just one bite at a time. So it starts somewhere. You got to get out and start. And I think a lot of times the big, I just talked about that. I like the big picture. But automation is the big picture and sometimes you do have to bring it into focus and just try to tackle the first thing in front of you.

So what do you think that's like would be a good first thing to tackle? Right. So I've asked this question multiple times to different people and I like to get your perspective on that.

My first perspective is definitely going to be in the read only. Being able to log into a device if you're using Python, we'll just use Python, say Netmiko from Kirk Byers. Just go in and do a show version. There's always vulnerabilities coming out. Maybe the first thing you do is create a CSV. that logs into all the devices. It's not something you need to do in parallel. This is something you can do serially because Python and it's being able to spread out and multi-process that is in threading. That is definitely a very advanced topic. And so, just start with gathering the versions and say, hey, the next time there's a… a PSIRT that comes out against your devices and they say these versions are vulnerable, these are not. You have that data. Right, and you can run that every quarter if you're doing quarterly code upgrades or what the– what that path looks like.

So yeah, I think it's also important that you mentioned about practice and doing read only. Right. So now it's not just a complete unknown. You're actually turned the read only bit into a known, right? Like you now know this is how you could do it. Not impacting anything. You're just doing a show version, but even that bit is helpful, as you said, like when you do a quarterly audits, right? I don't know, you're probably in the retail space, but also especially in like healthcare space, there's a strict amount of audits and so on. And I don't know about you. I don't like logging into device a thousand times and just doing show version. So there you go.

Agreed. That is one of the biggest pieces driving the network industry is regulation, right? Yeah. I think there's even if I remember reading right, it's been a little while since I've had to worry about PCI deeply. I think I read– Well, lucky you. Lucky us. Indeed. I think I read in the PCI 4, if I'm not– I could be very wrong about this. Somebody will correct us in the comments or whatnot. Very much so. But I think it says if you're running end of support software, that's a failing item on top of PCI there. Yeah, for sure, right. And so now, all right, knowing your software versions, maintaining that software versions becomes a very big important part of the day-to-day operations of the network.

So yeah, yeah, yeah, for sure. I think, I think that's definitely a red flag when you're doing a end of support. And in fact, that's like, I think a lot of the, I don't want to say excuse, but the reason for a lot of companies to get over the hurdle moving from Python 2 to Python 3 is like, Python 2.7, it's no longer supported. And you better just get on with it. And just going to be like security patches and failing that part is where it's unacceptable for a lot of companies. And therefore, they're forced to go to Python 3 after dragging their feet for 10 years. –

Yep, and actually I think I just comment in some conversation today, Python 3.7 is also in to support this year. Oh, no kidding. Oh, okay. You know, so if I'm trusting– 3.8, 3.10, and 11, and yeah. Yep, I'm trusting my peers that sent that message right here on the show. So I'm sure I'll be corrected again, so.

Nah, it's all good. I mean, if it's not, it should be going up to 3 anyways, right? Like it's got these all the async… goodies from three three seven and on so you should be going up there Thank you for listening to part one of the interview with Josh. Stay tuned for next week where we talk more Python and more network automation. See you there.