How Cox Gets Network Automation Right, and Proved It at DEF CON

Hello and welcome to the Network Automation Nerds Podcast, the podcast where we explore the latest in network automation and the stories that connect us all. I'm your host Eric Chou and co-host with me is Ethan Banks. In today's sponsor episode by Network to Code, we're joined by esteemed guests Josh Watkins, Eric Hansen, and David Ezell from Cox Communications, the largest private broadband company In the United States, they will shed lights on their motivation for network automation, lessons learned, some of the memorable moments, and imagine this, the story behind maintaining the network at DEF CON Hacker Convention, where the users hacked networks for a living. Stay tuned as we uncovered the trials, tribulations, and triumphs of network automation at Cox Communications. Welcome to the show, gentlemen. First, can you tell us a bit about yourself and your company?

Yeah, thanks for having us. I think it's awesome to be able to talk about network automation with you guys. It's a passion that my team has and we share. There's a lot of opportunity at the MSO space. I'm David Ezell, I'm the Director of Network Automation at Cox. I've been doing this for about five years now. I historically have been in the software technology space. I've been doing consulting most of my life from a lot of enterprise integration, app dev, that sort of thing. And I came into Cox They were getting to a point where they needed to deploy the next generation of remote physical devices out in the node. We had this thing called RPHY and it was kind of a transformation in Cox itself because they were basically going from an analog network into more of a digitized network. And that required us to be more sophisticated in the way that we manage software. And organizationally, we weren't aligned to it because effectively we were moving more intelligence out closer to the home. And that required an entire organization and our outside plant team to kind of start collaborating closely with our main telecom centers. Organizational, those teams never really kind of talk to each other. So a lot of the initial engagements we had to kind of design software solutions that kind of orchestrated our ability to provision these new digitized type of network devices into the network. And kind of the genesis of our team was off the heels of us actually implementing something like that because Cox took a very automation first mindset to it where, hey, we're not going to be able to scale the deployments of these things without going out the gate with automation. We're not going to be able to increase our workforce at the same rate that we're going to need to do this. We're talking thousands upon thousands of provisioned network devices. And so that was kind of the catalyst that started this kind of network automation team that we have here. And a lot of the software best practice blended with a lot of the network engineering knowledge and best practices were kind of combined into a joint team and really was allowing us to be very successful in that deployment and allowed us to effectively scale out, you know, with automation and not increase kind of our workforce at the same time. And then I'll let Josh and Eric kind of get into their intros and stuff. They have a lot of history in some of our brownfield network. So if you think about everything that we're doing is kind of net new greenfield style. There's a lot of different considerations and challenges there. But in the brownfield side and what Eric and Josh kind of talk about, it's orders of magnitude harder to kind of influence things that are already there, introduce new concepts, new practices. There's a lot of challenges there. I'm sure we'll get into later in the talk.

So before we meet Josh and Eric, David, I did have a question for you that you said kind of an analog to digital legacy network and move into new stuff. Can you shine a little light on what that was?

Yeah, yeah, absolutely. You know, at a very high level, right? When we think about the converged cable access platforms were voice video and data. that gets fed out from our main telecom centers across the country into effectively, you know, mux/demux, DWDMs down to, you know, nodes in the network and amplifiers down to your home. There are two aspects of it. There's like your HFC hybrid fiber coaxial network. So you got fiber down to a node and then coax down to the house at your DOCSIS kind of side of things. And then you have your fiber to the home, which is your PON, you know, optical network. And, you know, we work in both aspects of that. So when we talked about remote, where we were trying to take what was traditionally and just what we call the CCAP in the MTCs, they were trying to break out some of that and move it into a digital node. So think about that kind of green box at the end of your street or near in your neighborhood that is connected with fiber and then your fed coax from there. That's what we're talking about. Provisioning those, putting more smarts in that and then eventually putting more smarts in the amplifiers and getting more information on the network and all that sort of thing. So that's what I was referring to.

Yeah, that's awesome. As I was telling you guys before the mic was on, I was in the MSO network and that's why I'm so excited to hear about the latest. I mean, I was part of that triple play launch for the MSO I was with. So David, you mentioned, you know, Josh was here from day one. So yeah, tell us a little bit about what that was like. And Brownfield is ultimately probably X amount of times harder than Greenfield. So excited to hear about, you know, how'd you get here, Josh? I mean, did you always dream up of being like a network engineer when you were What I want to be when I grow up.

I would say completely opposite of what I thought I would be doing for my career at Cox. I'll start with my introduction. I'm Josh Watkins. I'm an enterprise network automation developer here at Cox. And I've worked across nearly every corner of the business from front lines and the call center doing tech support to data analytics and eventually finding that place where I thrive. And that was network engineering. So a lot of my I didn't see myself doing either of those two things, to be honest. I don't know what I saw myself doing, but I think that's one of the great things about an MSO is there's so many different positions and facets to running a business like this and being a service provider that you have that flexibility to find that career path that makes sense for you. For me, network automation came naturally as things progressed with network engineering. Frankly, it got hard.

It just blossoms from there. Yeah, I was part of that crew that do that late-night maintenance. I mean, I remember even in the early 2000s, the maintenance window was between like midnight to like 6 a.m. And it just gradually shrunk and shrunk and shrunk. By the time four and a half years later, it was like three hours. And, you know, it's mid of the night you're taking all these equipment, go to the hub site where these hub sites are so remote that you can't even find it. Now you're like in there. doing the maintenance. Yeah, so all of those, man.

Yeah, that was a great point. You know, Eric and I both work down in regions of the country that Cox actually services, right? So we're both employees and customers. And some of these sites where our gear is at is just out in the woods, you know? It's like a building that is down a dirt road two hours away. So, you know, it's three o'clock in the morning and you got your compact flash card to go into one of these buildings and do upgrades. And before you know it, that maintenance window is shrinking, right? To your point here that you have validation to do. So it might take two hours just to get the upgrades done, but then you have another hour of validations on top of that and automation is going to help you out on both ends there.

Yeah. What about you, Eric? Were you similar to Josh where network automation came natural to you or was it something that

They said, come shadow for a little while, see if it's something you want to do. So fast forward a couple of years and then I got a job doing network engineering. So that was working with Nokia gear, Cisco gear. I mean, at the MSOs, we just have a ton of different equipment. It's a lot to learn and I don't think you'll ever be proficient in anything, but it was just awesome to have that opportunity. So with working on this whole variety of gear, there's a definite need for automation. Like just trying to keep up with these maintenance windows that you're talking about, you know, you only have X amount of time and dealing especially with Cox business customers like Josh and I did. You got to get it done in that window. These customers do not care if you tell them you ran out of time. That's not their problem. Yeah. Yeah. So it came out of a need to ensure that we're consistent, accurate and quick with our maintenance is. And I remember one time Josh could tell you he was doing the maintenance and I was helping

Yeah, it was just dead silence probably 3.30 in the morning. It's like, you know what you just did, right? I did know and I was just frozen and just panicking. It really is the largest network outage I've ever caused was by not having automation. I simply hit the right-click of my mouse and it pasted into my terminal. I wiped out an entire city's business services in one click of the mouse. So working together though, that's one thing that I will say about Eric and I is that you kind of hear this term of like peer programming a lot of times when it comes to developing developer topics. And I think that really is part of our DNA in the network engineering space. We always were watching each other's backs where one person's doing the work, one person's doing the thinking so that we have kind of that full brain capacity to work on a network this large. And while I was frozen, Eric

Maybe the next day when everything's fixed and you're just like, you know, with that halo on top. Yeah. Yeah. Yeah. I may or may not have cried a bit when I heard that story because I was that dude, right? Like sitting at the floor of the data center where I know I just caused an outage and now I'm trying to recover, trying to do the dial in for the console for, you know, management interface. But something that I heard, it's kind of interesting to me when you're just talking about Josh is that you guys do actually pair programming-ish network engineering operations. And that is not. And who's have that hero mentality where he goes in and do that. Is that something that is unique to Cox or is that something unique to your team that you develop David or how did that work?

Well, like organizationally, like Josh and Eric are kind of the gold standard is like how to work together effectively. Right. And not everybody has the depth of experience in both the network and software that these two have. We call them unicorns, right? And it's hard to find unicorn. So a lot of the model that we have in this is obviously we're very passionate.

You said software development and describe Josh and Eric as unicorns because they have the network engineering experience, but they're also software developers. Is that truly what we're talking about here, software development? Cause network automation is really broad, right? It could be software development where you're truly writing code and building a platform that's unique to your company, but it could also be stitching together a bunch of platforms with glue code, let's say, to, to bring it all together.

I think that's a great question, Ethan, because I think at different parts of our career, both Eric and I, you know, at the beginning and what it looks like today, it starts with like scripts and we're maybe not software development. We're really automating the network, Ansible, Python, Netmiko, right? We're getting real hands-on with the gear. That led to more real… tangible software platforms. Eric and I both were actually co-authors of a US patent for software that Eric and I wrote that basically efficiently discovers the topology of our fiber network for Cox business. And we're talking about a network that spans 65,000 nodes, over 6,000 fiber rings that– these nodes live on and you need to know the order of the devices, how many devices are they online, are they offline? And there was no tool in the space that really could provide that level of performance. So we wrote it. And I would say that both sides of the coin, whether it's getting down and dirty with the devices themselves and wrangling them into compliance or writing software that we need to meet the scale and demand that we have as a service provider.

And that was like really iterative on that. It wasn't an overnight type thing. So that was, we were working maintenance windows, mainly with Nokia rings at the time. And, you know, one of the steps was to make sure the ring was in good status. The neighborhoods were all formed, all the protocols that should be up are up that sort of thing. And my background, I've got a bachelor's of science in game development of all things. How I ended up here is a story for another day, but use kind of my background in that to start with a C sharp app that we actually. worked on and let techs use so they could run it. It would reach out to the network, do a couple checks and kind of greenlight them before maintenance. So that evolved into like a larger C sharp app that then had different modes where you could upgrade firmware, migrate equipment, that sort of thing. Then I was talking to Josh and he floated the idea of, Hey, if we put this on a server and have it run all the time, do you think we could set up some sort of monitoring with that? You know, something that's useful. So then that's where. and then it turns into a thing. The scope creep never ends.

Yeah. And that's an interesting thing. You talked about pair work and pair programming and kind of starting there, right? And it's a tale that I've seen not just in the network space, but in other places where talented people decide to solve a problem and they use software to do that. And they eventually get to a point where they're delivering, they're adding a lot of value and success leads to success and people want more. And a lot of times they end up an engineering hero building the software and stuck for years maintaining the thing that they don't want anything to do with after a little while. How can we move that into more of a sustaining and enterprise function that allows us to kind of continue to do that type of great work, but do it in a more scalable and sustainable way that's more team-based versus individual-based?

I think that's a great point. And if you guys allow me for a moment to congratulate you, because that was the first time that I heard from my guests that they actually have written the software that's US patent. Build versus buy has always been like an issue, right? Everybody I talked to, that's always the first thing that they talked about. So tell us what is kind of your thought process on Build versus buy and what is kind of your tool stack today?

I guess we'll give the frontline guys perspective and then we'll let the boss tell you. So it's one of those things where Build versus buy, you might not always get the opportunity to do either one of them. You know, it might be budget season. becomes the thing. I think we have to really take stock in how much scale we're dealing with and what the maintenance of that is. It's easy to build something and turn it on that first time, right? To keep it going, keep it sustained, keep it maintained, security vulnerabilities, these types of things, that takes effort. When we're looking for platforms, whether we're thinking about building or buying, we have to really look at, does this solve the problem, but also what resources is going to take to sustain and maintain that?

Yeah, it's not necessarily one decision that Cox has made to either build or buy. It's all going to be situation dependent and some stuff that I've built, I look back and wish we would have bought it because, you know, our end users are is a huge benefit of buy, but at the same time, the flexibility you get with build is just something that our users love as well. So we really have to evaluate it case by case.

Eric, is it a headcount thing? That is, if you're going to commit to doing the development in-house, you got to commit some number of man hours to it in perpetuity, really a platform like this. But does it also get to a point where it's like, okay, we don't need one or two developers. We need more.

Yeah. And how do you come up with that number upfront, right? Like you could say that this thing's only going to support a limited number of devices and have a handful of users and then word of mouth gets out. And now you're in a situation where you don't have the staffing to support it or your one guy, your Superman guy is stuck working on that and you need them for other things. So how you do that upfront calculation is the hard part.

And can you get the head count? How hard has it been to hire people when you need them? Cause every place I've worked, that's always been a challenge. Like we'll buy you whatever software.

Not going to happen. Yeah. I mean, that's a great call out. I see that as a large deficiency in a lot of companies that say, Hey, we bought the thing. Right. But the thing doesn't really matter as much as the people doing the thing and the organizational alignment to allowing those people to do that thing the right way into perpetuity. Right. That's where they always fail. They get so tied up in the tech and the platform that they just don't even think about the operating model behind it. Right. And that's where it constantly just cycles itself and people burn out. I'm very much an open source advocate. I think that if you just bring some talented individuals to talk about the problem space and understands the technology nuances to say, I know this thing does this thing very well. And as long as I can integrate it with this other thing that does it pretty well, then it makes it easier for me to introduce new modular bits of code and services and functions to kind of stitch it together based off the use case that we may have in the network. that you're not gonna find a product on the shelf that already does that for you. 'Cause even if you buy that off the shelf thing, they call it custom off the shelf for a reason. You need a vendor to come in and implement and customize that solution for your needs anyway. So there's just a lot of opportunity to reduce your cost and increase your velocity when you do that. But obviously you have that sustainability. that support kind of concern. So as long as when I'm making decisions, if we can come out with, yes, I know we have the skill sets to build it and we have the best of breed kind of open source technologies to build it with, that's all great. Can I support it and sustain it, make sure it's secure, it's scalable, it's monitored, and how are we gonna operate this thing going into the future if, for whatever reason, the people that built it? leave, right? That's where my focus is.

Yeah. And talking about skill sets, I mean, David, I know you've seen this challenge firsthand, but when we've got these applications that are so network centric, if you're looking for someone to do the software side and you pull in a full stack developer, I mean, how much of the time of your network engineers goes to training these full stack developers, how the network works and how to write software to interact with it.

Yeah. A hundred percent.

I think that's what we struggle with. I mean, I'm an employee for network to code, and we deal with network source of truth, and we actually go into companies to consult them, help them out and stuff. So I think that's kind of what goes into what you were talking about, David, on you want to have pieces that work well together by default, but also have the option of swapping them out. if need to be. And the bonus would be if this is a skill set, you could actually– it's not so specialized that you could go out to the marketplace and kind of swap out those pieces. Because as you mentioned, the unicorns are very hard to find. We call them unicorns for a reason. And they're very hard to replace, really, to even do the knowledge transfer because it requires a certain amount of baseline to build upon.

I was just going to mention that, you know, that's why in my viewpoint, when I think about, you know, the service provider network and where do we grow, where do we invest, I've always been an advocate of you have to start with the network engineer and grow them into the software space because you're not going to be able to bring a bunch of software engineers and have them grow into the network space. It's just not as effective. And so we were passionate about like that culture of automation, kind of teaching people to fish and kind of growing them into network engineers that have that software discipline.

That's awesome because I've been in an environment where the attitude was the opposite. They try to bring software engineer into the network engineering space. And, you know, I will have to say the success rate was like 50%. If that is really hard to make the software engineers that were there, they're kind of the plumbers and they don't want to specialize in one thing because they just wanted no C sharp, for example, or Python in that makes their skillset transferable where they view network engineers as being very silo, very specialized. And so there's always that resistance of going in. So it's very.

That goes back to one of the things we were talking about on the mentality of the MSO we work for on the buy versus build type thing. I mean, they've got their idea of what system they want to use for a source of truth. But the issue that they've had since the whole time I've been working for the company is a lot of their sources of truth are ones that are built from human input. So the accuracy there is questionable. And that leads to a source you really can't trust and we can't really call it a source of truth at that point. No. So, Nautobot, as far as being used as a source of truth, we've got a couple ways that we kind of solve that gap because a lot of our engineers have made side of the desk type tools that pull live data from the network. So, we can correlate that with some of these static tools that do have human input and kind of come up with some sort of consensus on which has the correct data. And that's several different systems. So luckily Nautobot's got the whole source of truth plugins, single source of truth plugins or extensions or apps or whatever they're called now. You guys have done a lot of work with those.

They've really evolved the drivers. Yeah.

Yeah, exactly. And we pull data from a bunch of different systems to build what we've got in Nautobot right now. And I like to think that Nautobot has become the most accurate source of truth that we have. So, and since we've got that running, a lot of these other teams are taking interest in that. It's like, Hey, can we pull inventory listing from you guys? Can we pull compliance? And I think that, I mean, Josh was the person who kicked that all off. I mean, we spoke about NetBox briefly. That's, I think where we started. Josh, do you want to talk about that?

Yeah. So we were evaluating tools and Within the company, we had a tool that we like to, from a corporate standpoint, think was the source of truth. But as Eric mentioned, the data integrity and the upkeep of it was definitely a questionable side of things that people lost trust in. And we were at a point where we needed a tool that was flexible enough to think of things more out of sources of record, right? sources of record of IP addresses or of devices or a segment of the network. We're many businesses under one umbrella. We have our residential service side, our Cox business service side from fiber, DOCSIS, and everything in between. There's sources of record. What we liked about Nautobot was that single source of truth framework and plugins that really allow us to pull in what transformations and really, Nautobot gives us that flexibility with the API and having that common data model that all the devices live under to make integration with other systems much easier and people integrating with us much easier. So we've got all this information we're pulling from all these different sources of record into Nautobot, your network source of truth.

Okay. And it's your single source of truth you said because of that, because you're pulling all that information in. But I would argue that depends. Are you also using Nautobot as a repository for network state? What you intend the network state to be?

Yeah, absolutely. Great question, Ethan, because most of our external and internal customers look at Nautobot as providing a very specific thing to them. And that is Golden Config compliance, right? That app, that plugin within the Nautobot ecosystem. is what they know it as. What we know it as is you have to have an inventory and backups of these devices. There's things that you need to provide those capabilities and naturally having that ability to do really both of those things, right? Not if I can take backups to reach out to the devices, but getting that source of truth of all the devices not only enables our compliance capabilities, but it acts as a for maintenance, for security vulnerability type work. Everything really centers around your inventory, right? You can't automate the network without knowing what the network is and how to get to these devices. So yeah, if you asked someone at Cox, what do we use Nautobot for? The very first thing they're gonna tell you is compliance. We see it though as all of its parts and pieces.

Yeah, mainly they feel that way because they're getting graded on it. There's a report card. There is a carrot and a stick thing there, right? Like so the stick is security definitely says we need to be able to see this and our team's done a great job of enabling security to do the things that they've been wanting to do for a very long time. And Nautobot helped us kind of allow them to see very clearly and confidently that these devices are in compliance in these security portions of the config, right? we would like to be able to manage the life cycle of these devices with CICD, with source control, infrastructure as code. And you know, Nautobot is definitely a foundational platform for us to do those great things. But in the meantime, we need to be able to show a lot of value and we need those sticks in place in harder to kind of allow the organizational inertia to move forward. It's a huge lift to get us to do anything the same way. And this was a great opportunity for us to say, hey, we got this solution that we're doing anyway as the basis of why don't we partner and kind of push that as a standard? And that's really gotten us a very long way towards that idealistic future that not only kind of shows that, hey, this is what can be done, but also allows our network engineers to get an operating space in which they can understand what is Jinja, what is YAML, like what is this CI/CD thing? And why are you keeping me from changing my configs? It's like, oh, well, our pipeline ran this test and it said it failed. So you need to change X, Y, and Z. And these are all things that network engineers aren't used to really work.

David, you mentioned the having to learn Jinja and these other things, Ansible and some of the other topics related to network engineering. Josh, you had started a program several years back that you call peer to peer. It was kind of like an outreach for our team where we invite some of our engineering partners to join in and we'll show them a technology or a concept, things like Jinja, YAML, Ansible. And we even had some of our engineers showcase some of their products or projects that they've been working on. So that was one of the ways that we could get them kind of up to speed on some of the things that they would need when they're interacting with Nautobot.

It's a great point. And Eric, I think you mentioned we don't want one-trick ponies, right? When we buy a platform or we're buying into a technology, if those things have very domain specific languages and proprietary ways of working with them, that means we're trying to teach a group of folks. one trick ponies that don't enrich the rest of their work. Things like Jinja, YAML, Ansible, those core components around just network automation tools in general, having that within the Nautobot ecosystem means they're learning a skill that they can apply in many different places that helps their daily work in and out of Nautobot. It's key to cultivating and open source tools that enable these guys to do more and not have to have the burden of one-trick ponies.

Would you guys walk us through a workflow, some more straightforward automation task that's routine that you do within Cox that your network source of truth, Nautobot in this case, is a part of?

Yeah. This week specifically, part of config compliance for us has been decommissioning SNMPv2. A workflow for us this week involved pulling the compliance, the remediation data from the compliance app output and running that through a playbook on all 65,000 devices and decommissioning SNMPv2. So while they're getting graded on compliance, they're taking that data and able to… run it back against the network and remediate the devices. That has been, I would say, the disparity between the tools and the maturity here in our space is that we would heavily focus on tools that could tell you we were out of compliance, but have nothing that could help you get it in compliance. So we just stay out of compliance. I think that we're finally reaching that point where they're trusting that the information coming back from the network and they're writing the intent. The engineers are stating this is what my config should look like for SNMPv3. If it's not that, the level of trust goes up. Now they're using that data, pushing it out towards the network. in ways that work for them and maturity and a barrier to entry that is good for them, whether that's copy and pasting, Ansible, Python. We don't try to get into the specifics yet in terms of here's how you should do it. We just want them to want to do it at this point and Nautobot's kind of a cornerstone of that.

The process for this is so much better than it used to be. Back in the day, I went through a SNMPv2 to v3 project once as well. and along with a bunch of other compliance and updates and standards that were defined for the switches and routers. The way you'd know if the device was out of compliance was I'd log into it and saw someone change something that's like, "Guys, we talked about this. Don't do that." Or I might get a diff in the NMS that would alert me. I say, "Oh, hey, we see this line was changed." If I'm paying attention and caught it in my deluge of email that day, I go, "Oh, I better fix that whatever happened there." This is just, It's just so much more robust. It's so much more reliable.

Yeah, I mean, I would say the way we find out usually is a customer's calling, right? We're troubleshooting the network and we're like, okay, well, on one side of the network on the A side, the QoS looks like this, but on the Z side, it seems like it has a legacy QoS and they have performance issues. So we would find those things and be surprised by them. where the customer unfortunately is the alarm sometimes. For us to get to a point where things are consistent, we're having to really look at it at scale and go and pick those things out ahead of time and not about let's us do that.

Yeah. I think what I really enjoy about what you just said, Josh and also Eric and David, is that you're able to visualize, you're getting graded on it, and for anything I ever achieved, was the visualization plays such a big part to it. Whatever goal that you want, you need to be able to track it and visualize it. And so you normalize the data. And that's no small task either, right? You're normalizing these data from these asynchronous and differentiated sources. Then you normalize the data. Then you're able to visualize it. You pass it on to other teams and make it easy for them to visualize, getting graded on. Then you're able to do all these awesome things, but you lay down the foundation in the data normalization and the storage in a scalable way. So that's all awesome. But you guys are not getting away with telling us about your DEF CON network story. Because that is kind of a unique story, right? Because everybody's so friendly at DEF CON, nobody trying to hack you, right? So must be like a super easy task or am I wrong there? Tell us a little bit about that.

So DEF CON, I think it probably doesn't need a lot of introduction, right? It's a place where you have hackers and security researchers that have free roam of the network. a lot of the Las Vegas strip. So all the casinos and hotels that you're likely familiar with, these guys are probably plugged in there. And Las Vegas Convention Center, again, another place that probably doesn't need a lot of introduction, they host CES. So in terms of the actual network itself, the wifi, the capabilities, we all felt confident about that because this is a proven network when it comes to delivering performance. Now, when you introduce having a group of hackers and security researchers in the room for three days, things get a little tense, right? And Cox as a company is built that way. Our security of our customers' data is foremost when it comes to the decisions that they're making. A lot of times it makes things extra difficult, but they would always fall into that category of data safety first. and prepping for the event meant looking at the compliance of the devices, but also meant defining the compliance of the devices. So we have a golden config that operates the network. Is it secure when you have hackers all in the same room with physical access? That's the ultimate stress test.

That's crazy. Yeah, like, I mean, these guys are pulling out like, you know, Wi-Fi pineapple devices, Bluetooth.

If they can give physical access to that device, if we aren't being very diligent about how we locked it down, I think anyone's going to be nervous, right? And even just down to their physical location, are they behind lock and key? Are they up in the ceiling? What's our vectors and our posture across that? And DEF CON allowed us to… I would say the full life cycle of compliance, right? Defining compliance, finding out how far away we were from compliance, getting even more shook at that, and putting it all back together, remediating it, getting it ready before that main show, and really providing a network. And as we understand it, this is the first time that the DEFCON organizers used a network that was not theirs. Traditionally, they were bringing their own gear, standing up their own Wi-Fi networks, and you would just be a backhaul. The convention center is too large for that, so we were both the provider and the access for all the actual event attendees and running the admin of it, and they were impressed with that. So not only were we looking at this as like our Super Bowl of cybersecurity, it was just a badge of honor, right? That we could show up and meet the demands of an audience like this. I mean, these are true technology folks that really are going to be discerning. And it was a proud moment. It took all of us, right? Nautobot team, network to code, as you know, is embedded in our team with the professional services. Eric and I, our security team, hospitality networks, engineers, DEF CON, engineers, we're all involved in getting this thing to the success that it was.

Josh, so you would say Nautobot's role was really twofold, right? So you talked about… prior to the event, there was, we don't know much about this equipment. Let's pull in some information, some backups, and let's see. are the controls in place that security says we need for this and how many devices need to be changed if so. So that was one part. And then the other major concern and our ask from security was… Hey, can you continually look at these configs to see if there's any drift during the event? So that was something that kind of pushed all of our tools to the max, right? They wanted that done.

What do we end up with hourly or sooner? Hmm. Yeah. So continuous compliance, meaning if that port was shut down and it came up in the middle of the event, someone needed to look at it, right? We were not expecting new devices to get plugged in. We'll say that. So yeah, I think really that full life cycle is really what we were able to accomplish and see. effort it was going to be, right? Everyone was wondering, do we have enough manpower to get this done? And really the visibility of the compliance, regardless of the score ahead of time, is what gave us the plan that we were going to need to put in place the resources we needed to get it where we wanted it. And it was a fantastic event that really all of us just enjoyed being part of, I think. And the Network to Code team is really You guys have that herd of unicorns, right? I feel like they are one of us. It doesn't feel like a vendor or a contractor. It really just feels like one of us. And that level of trust and partnership lets us do big events like a DEFCON.

I just want to interrupt real quick. I think it's very important to maybe establish this as a term. But apparently… According to AI, Google, a group of unicorns is called a blessing.

No kidding. Wow. Do they spit out like rainbows and candies?

Makes everything better. Makes everything better.

David thinks we do, that we spit out rainbows.

The testament to Josh and Eric, I mean, literally they've been with me for many years now. There is nothing that I can't ask them to help lead and they don't deliver beyond expectations every time. And I think anybody who gets an opportunity to work with them, leads happy, leaves impassioned because they bring a lot of passion to their work. You know, we talk about the unicorn and like they may have a lot of rigor and experience and discipline for the network stuff, but they are a testament to… even holding themselves that same standard on the software side in terms of how they write code, how they manage code, how they try to convey best practices to people they work with. It's quite awesome and I'm blushing on them, but they're fantastic.

Yeah, I mean, like what you said, Josh, I think this is like the Super Bowl of, you know, just the ultimate stress test for your skill set, for your tool chains, for your security compliance, that continuous monitoring. I also want to ask you, David, how hard was it? I mean, even listening to it, I was sweating bullets, man. I was just like, Oh my God, can you push this through? Was there any roadblocks in pushing this through or were you confident? I mean, even with a successful result at the end, I want to trace back to that beginning and what were some of your

Well, should I do it? Or, you know, inaction is always an option, right? I mean, it wasn't some kind of grand opening, right? Hey, here organization aligned to this big thing. It's literally a call from one of our security leaders and saying, Hey man, we got this big thing we got to do. Can you help us out? And, you know, I had a quick powwow with my team. H&N was a net new domain for us. Like we had, it was completely unknown what we were getting ourselves into, but you know, a lot of times this stuff takes courage and you just have to kind of say, Hey, It's important to the business. We're here to help. Let's make it happen, right? And that's kind of what our team did and partnered heavily with security and H&N and a lot of the teams, like you mentioned, and got it done. And my job is to give them the space to do that. So we've got thousands of things that our team does every day. My job is to kind of be as agile, I'll use the term, right? As flexible as we can be to say when these business priorities come into us, which they do all the time. We have to be able to adapt to it and change. And so this is a testament to our ability to take in that kind of work on a moment's notice and make sure that we're doing what we need to do.

Yeah. I mean, you guys mentioned so many wins and it's exciting and, you know, just remarkable to see the progression in that you've had made so far. But I do want to ask you about what does the future hold for Network Automation Nerds at Cox? What are some of the challenges? and aspirations. We mentioned, you know, we talked about culture, we talked about skill set gaps, and all of these, I imagine, would be become part of that, that challenge in the future, right? So maybe we start with you, Eric, on what does the future hold for network automation, at Cox Communications?

One of the biggest things in a concept that was just so foreign to us until we met with NTC And they explained it was the whole infrastructure as code. I mean, Josh and I coming as network engineers for many years, it was just kind of like mind blowing. It was obvious the whole time, you know, it's like, this is how you should manage things, but it's not something that's in practice. So I think the future is seeing how much of our Brownfield stuff and any new products getting deployed. Can we do them in a modern infrastructure as code type manner to where a ensuring that we stay at that with the equipment, which unfortunately means difficult things like having network engineers not log into a device directly to make a change. They need to start by defining their intent and then the automation will take care of that. That's something that the field teams, it's going to be hard for us to get everybody on board. It's not going to be an overnight type thing just with our scale. So I think that's what it holds for us in the future, but it is going to be the challenge.

What do you think, Josh? I mean, that's like taking candy away from kids, right? You're disallowing, allowing it. But yeah, Josh, go ahead.

I was saying they literally see it like that too, right? And you have to present it in a way that they want it, right? So it has to have so much value and maybe that values on the edge, like their maintenance work that they see the value in starting at the front of the process with it. So finding the and really just our leaders, right? David is very automation focused. He understands software. He's come from that space. Not all the leaders really have that exposure, right? So we have to get them to buy into it as well so that we get the support and the resources we need. And I think really going forward, building that culture and getting buy-in is where it's going to make or break. how successful we are, like what level of automation we achieve.

What do you think David? Should we just clone Josh and Eric once it's legal?

Already working on it.

Oh, okay.

I would say my challenge as Josh mentioned is more organizational than technology, right? So like coming from the software world, managing software through source control and systems, it's just standard, right? And so coming here and realize like, oh, this is the way that they manage these devices. We need to start this. you know, say, hey, we're going to, you know, set these objectives as a company in order to get to infrastructure as code. Here's the organizational model and engagement models we're going to establish to allow us to kind of come together as a company to coalesce around these kinds of systems and best practices. And then if we can get them to prioritize that in conjunction with all the other things that they care about, even more so with I need to increase my network speeds, I need to improve my reliability, I need to be more agile in the marketplace. It's taking the things that we know already can We know that exposure to folks internally at MSOs is always limited. So we would just ask if you or Ethan have questions for us that you think your audience might be interested in. We'd love to field a couple of questions from you.

Yeah, especially since everything is not unicorns and rainbows as some people would think. Yeah, I think I'm good here coming from, you know, like the DOCSIS background, even though it was DOCSIS 1.0 or pre-DOCSIS, but you know, I do understand the different So I'm kind of biased toward that, but I'm wondering if Ethan, you have anything that you want to ask that we haven't covered.

I have one. Is QoS still a thing? And if it is, do you automate it?

Yeah, I mean, QoS is definitely a thing, especially on the Cox business network. So we're providing transparent LAN services, MPLS based network and QoS to control bandwidth and quality of service and… I mean, everything in between, still a big thing, still something that we're automating as it goes in, a new device goes in, trying to catch up on all the legacy devices. So anything we can touch that involves QoS because that is an easy way to get a truck roll or have an alarm from a customer is, "Hey, I'm not getting the type of performance that I'm expecting. And you can pull your hair out, right? With traversing multiple networks and QoS tagging labels, everything's getting popped on, popped off. And before you know it, we're confused as engineers where the automation is something that we need. We just have to get there faster in a lot of cases.

I've taught enterprise QoS and my first slide is usually, you probably need more bandwidth, not QoS. So if you think you're going to solve a bandwidth problem with QoS, you're not. I was wondering if in 2024 we'd finally gotten to the point where, yeah, we hardly ever use QoS anymore because we have so much bandwidth. It's fine. It sounds like no.

No. I would definitely say even with huge amounts of bandwidth available to our network. there's still that prioritizing packets and getting it across the network in a way that is low latency and really delivering services that customers basically can feel connected from business to business like they're in the same data center across the country.

I think for us, QoS is still going to be a thing. We can deliver big bandwidth though. That is something that we're very good at and something that I think we should be very proud of with our network is the scaling is so proud of.

protecting important services too. Like we use it on voice and the core to make sure that nothing can stop on voice. And even one of the latest things was going back and ensuring it's on all management traffic. Cause it's not a good situation if you have a management system that suddenly clobbers the network because it can. So there's a lot of spots where QoS is still your last line of defense if done correctly.

That's the key if done correctly. It's hard. It is hard. And in fairness, a network like yours is a different animal with different requirements than even a large enterprise network. Really big enterprise networks, I think tend to look a lot more like service provider networks, but you know, mid and so on enterprises just have different concerns where QoS you can do it, but because it's hard and the technical debt that comes up and all the differences between devices and needing to understand queuing theory behind every chip set and all the rest, it's like, if you don't have to, just don't.

I think you're right, Ethan. I think the MSO is such a different network and just a different piece with great power with the semi-monopoly that you have, right? Like, especially for voice, for 911 calls and for the spectrums that were reserved by emergency services, then, you know, signal leakage. I would still remember it was like a 7.1 if anybody have detected like a leakage of signals and all of that. So definitely true. I think what I took away from our conversation today and I'm very impressed about is I think your team and your organization, but you know, I've exposed you to your team is that it's such a knowledge based team and it's such a open team for whether it's, you know, peer to peer, that movement that you started or it's pair programming. It's just a knowledge based team and that's going to prepare you well for I think whatever you guys venture into. So that's very impressive.

Yeah, that gives us the confidence that we've talked about before, just like racing into the darkness and helping us figure out problems.

Yeah, absolutely. So before we close, can you tell us where we could follow your work, all the projects, all the best work that you guys do, maybe on social, maybe not. So where would we start with you, David? Where can people follow your work or best way to find you on social?

Not really a big social media guy. I don't do a lot of presence there. You follow the work by working for me. We love really talented network engineers that are passionate about automation. So if you guys are interested in that and the stars align and hit me up on LinkedIn.

Fair enough. What about you, Josh?

Yeah. For me, you can find me on GitHub under NetOps engineer. That's the easiest way to contact me. Same for me. I don't do a whole lot of social media, but find me on there. You can find my e-mail. And pretty much any of the network to code Slack channels, you'll find me lurking there. And if you have questions, reach out. We'd love to hear from you.

Josh is accepting PRs. Cool. What about you, Eric? Are you the odd man now in this group that actually does social even LinkedIn?

No, I'm pretty heavy on my LinkedIn. Okay. And anyone's welcome to message me on LinkedIn. I welcome conversation. And Josh said, check my commit history. Badge of honor there. Uh-huh, for sure. But yeah, just hit me up on LinkedIn.

Awesome. Hey, I just want to thank you guys again for being here. And thank you for listening to this episode of "Network Automation Nerds." If you have any feedbacks for "Network Automation Nerds," our guest today, or this episode, send us some follow-ups at packetpushers.net/followup. And we really want to hear from you. Again, I want to thank our guests, Josh, Eric, and David today, and our sponsor, Network to Code for this episode. So thank you guys. If you want to learn more about Network to Code, Cox communication, the links are in the show notes. And I want to thank my co-host today, Ethan Banks. You've brightened up this episode. Thank you for your presence. And last but not least, remember that too much network automation will never be enough.