What Works, and What Doesn’t, in Network Automation Projects

Hello and welcome to the Network Automation Nerds Podcast where we explore the latest in network automation from a practitioner's perspective. I'm your host Eric Chou, the network engineer who loves everything about network automation. Today we're joined by Matt Remke who has spent years in the trenches of network automation projects as a consultant. I have witnessed firsthand how Matt tackles tough challenges of scaling automation in real world environments for some of the world's largest companies. Matt will share what works, what backfire, and some of the hard-earned lessons, which is my favorite part. So let's jump in. Welcome to the show, Matt.

Hey, Eric. Thank you so much. Yeah, my name is Matt Remke. I'm a senior engagement manager with Network to Code and I think the… For those unfamiliar, Network to Code is a network automation services provider and also the parent and owner of Nautobot who a lot of folks are. I think growing increasingly familiar with. My role is more than a project manager. I'm sort of like a project delivery orchestrator. So project management to me is box checking following a rote process. The type of delivery leadership that we're talking about is engaging with clients under pressure, under stress. They need results. assembling teams, our team, their team, bridging gaps, crushing silos and orchestrating the overall delivery.

Yeah, speaking like a true leader, Matt, like I'm sure I saw your Zoom background. Matt plus one. Oh, that's awesome. Yeah. So I think oftentimes I feel like, you know, how that story goes where, you know, maybe four blind guys touching an elephant and everybody that kind of sees a partial of that picture, right? So as a, as a network engineer, you know, oftentimes I feel like I'm one of those guys who I'm limited my visibility by, you know, my experience, my exposure. But in your role, it's sort of like you take in a higher level perspective, right? And then you need to be both details as well as looking at the big picture, seeing wherever all the pieces fit. Is that where you're trying to get at?

Yeah, that's right. So I'm a technical layperson. I am not a hands to keyboard engineer, although I deeply, deeply admire and respect this community. I worked with many hundreds of you and it's fascinating. I take a view in terms of… more of a, from a business owner standpoint. But like you said, the big picture of where the value is, what we're trying to achieve. for success, but also I need to understand at least in broad strokes, the technical goings on the technical discussions, the decisions that are needed to get made, consider options, success risks, understand. timeline. you know, just being, but for me, it's just being there and understanding and putting the pieces together of the technical. Big picture that's happening and fitting that again, like you said with with you know, the overall view of okay What is the person who writes the checks for these projects want to see? That's right. Well, how do we present what we're going through and what we're learning and how we're delivering in a way that they can understand? And it just doesn't sound like the, you know, the cartoon teacher from the Peanuts cartoons.

Yeah, exactly. They don't care about VLANs or they don't care about, you know, trunking or with this dot1q or not. They really care about delivering business results. And you're there to translate that to engineers like me, as well as, you know, vice versa, right? Translate that technical bit and say, well, you know, we do want that X technology because it delivers business results.

That's right. That's probably why I'm probably different from a lot of guests on this podcast because I'm kind of like you've mentioned, I'm kind of that clutch. And the management operation, the business enterprise vision that's happening, the gray area.

Yeah. And that's why I'm excited to have you here because you do offer that different perspective, right? Like a lot of times we're very focused on technical details. So I'm here to listen to your thoughts and the lessons learned on what kind of value that we could bring to our audience so that they are aware of… a person like you is not trying to, you know, I don't know, rain on the parade or something, right? Sometimes that's the case, but they're trying to make it better.

That's really what I would hope to accomplish here is just to enlighten that piece of perspective for those who might be curious about it.

Nice. So now that we got out of the way, I am curious a little bit about how you got here, Matt, because you know, You're kind of in that bridge, right? So how does someone like you came to this place on having this transition points? So what's your background? Have you always been interested in technology or were you a business-oriented person?

Yeah, I mean, I am, you know, full disclosure. I'm almost 50. So I was going to, I was growing up. You know, I got, you know, it was a big deal to get a boom box in the early eighties with the police or Hall & Oates, you know, so I was at the stage where things went from three channels and rabbit ears and kind of to a cable box with 20 foot cord with, you know, two rows of 36 channels that turned into the Atari 2600. And then, you know, going from rotary dial phones to cordless phones. And so a lot of that technological development hockey stick was my formative years. So I don't think I was nearly as hands on and aware of the technical genius or the technical machinery that was happening in front of me. I just kind of accepted it as… You know, my world's different than my parents and grandparents, but I was aware anyway. so, you know, going up, I was a sharp kid, but went to, went to college, thought I was, you know, took a look at engineering, you know, Ran into engineering, Calc, you know, had to hustle up a tutor to squeak out. I would go from a 50 to a passing grade, but I was like, you know, I was always one of those, I'm maybe an arrogant person, but, you know. Some of the math made me flinch, but I'm also had a real mind for just kind of how things work. So I've always been a process guy. It was, I was very interested and I really excelled it like. biochemistry and physiology. And so I was really kind of hard on the pre-med track and I really enjoyed that. But then as I got out of school. uh… You know, I was looking at it and I was like, okay, what's it gonna cost me to get to go get a doctor and do I really want to do that? And it's, you know, 100 K then seven years later, even, even back then you're looking at eight years and $250,000 and I was like, I could do better than, Oh, I could do better than that. Anyway, so off I go. it would actually do was graduate college year early because I wanted to go be a ski instructor. And, and my mom was like, you know, the privilege that you have to go to school is you can't be taken, taken like that. So I said, okay, great. I'm just going to finish it. I'm going to, and then I'm going to go be a ski instructor. Anyway, I did get out of there with to be a ski instructor came back and then through a staffing firm got into. the startup. In the year 1998, that will become known as art.com. So what happened with dot-com bubble is. Everyone was throwing the capital at everything. Art.com was one of those. It was owned by some outfit out of Chicago, but they… crashed So we ended up kind of buying that domain and intellectual property assets as from what we were all wall. Now we're art.com and that started a. true experience, just explosive. 300% year over year growth for a company that sold posters. art prints and custom framed art on the Internet back in the early days when eBay was going crazy and Amazon was still selling books. Right. Yeah, so from that point. I just kind of applied myself, just saw ambitious hard charging. So I built out human resources. I worked in the customer service. I worked call center. I ran facilities. I did construction build out management projects. We expanded the plant. I did process quality management for basically a distribution outfit as well as a small manufacturing outfit, which was custom framing. So we have all these art prints. You choose what frame you want, how you want it, and then we custom size cut it, package it, ship it. That was my deal. So when I started with that company, it was, I was employee 12. And when I left, there was over 350 employees and we had just done a venture capital merge and things went sideways. That's a separate podcast after that. But I accumulated a ton of experience and a lot of it around process, understanding process, fixing things that are broken, fixing them fast. And then we need to accomplish this sort of build out this new capability and I would be the spearhead of those projects from in my early 20s. So. Long Story Longer Went to grad school, took a startup job in DC. I was a director of ops for a startup brewing company when the The housing bubble burst and diesel fuel went from $3 a gallon to $5.75 a gallon. We were selling a perishable product that needed to be transported long distances and refrigerated the whole way. That ended, I kind of build back out. back into Raleigh from DC and then I found NTC through Optum where a former coworker had graduated from Optum to Network to Code. I followed the coattails and I've been applying kind of my project delivery skills to a network automation projects ever since, but that didn't take too long.

No, no. I mean, I think that's what I also miss about, you know, a startup or, you know, cause I've been to both worlds, but like you, I started my career in a startup and during the dot-com, tail end of the dot-com bubble. that really were very, very different hats, right? And because the company was growing so fast that they were allowing me to just a junior guy to do all kinds of stuff. And you make mistakes, but an ambitious, driven guy like yourself could get a lot of experience and hard-earned lessons from there. And I am just interested in how you went from all of that, brewing and framing artwork and all that. So that was very interesting to me, but like you said, that's like to get into that, that's like another whole different episode. So fast forward to it.

Yes. I came from a, my, you know, my core mindset. I love the startup. I love the action. And I love the mindset of we'll give you as much responsibility as you can be accountable for. You know what I mean? So that's, that's the, the, it's like, you know, a kid in a candy shop or someone that's like really super hyper. motivated for, you know, however they define achievement, if it's, you know, business success, then go crazy.

Yeah, I think I've read a lot of business books on leaders I respect and a lot of times they recommend going to big companies to learn their process and see how it works. But don't overstay, I mean, as a young person, right? Like I'm not putting judgments. I'm just kind of paraphrasing what they're saying, but don't overstay for one or two or three years. You want that exposure. You want to go to a place where they will allow you to make mistakes. And that could be an established company because even like you said, Amazon, when I was there, even though they were more or less established, but they were still, quote unquote, always day one.

So five years. Yeah.

So, yeah. So I think it just kind of depends on the environment, but definitely take kind of looking at your experience as a mirror. that there are certain advantages of being in a startup and being in an environment where it allows you to wear many different hats.

I agree. That's something that we may talk about later, but I believe in reps. It's the same in business as it is in education, as it is in athletics. those who actually really get their hands dirty and get those reps, there's an accumulative benefit that comes from it.

Yeah, definitely. Yeah, definitely. I mean. Just a paraphrasing in athletic world, right? Remember the first time you do a bench press? Even the motion was unfamiliar to you, right? Just having even like an empty bar, 45 bar. Can't balance. Yeah, can balance. It's like the weight falling off. Bar wobble. But, but, you know, just do four sets, right? And then on the fifth set, you're just like, okay, now I get familiar with it. Now I can focus on. you know, adding on the weights, you know, making the motions accurately and you're finding that 1% improvement, then, you know, those reps really count as opposed to repeating the same rep 100 times. You're trying to make yourself 1% better every single time where you lift that bar. And that's what counts. So let's go and move toward the first one, maybe not the first, but If you could walk us through a recent network automation project, right? Like think back in your high level view, just looking at from start to finish, what was kind of the process and the most surprising part as a person in your role?

Okay, well, probably the one that comes to mind, we did a major application migration for a major client that we've been with for a number of years. from an older on-prem version of the application up to the updated version and also a cloud install. So there was an element of establishing the cloud and the footprint and the connectivity, and then there is the element of… the migration from the old version of the on-prem app to the new version in the cloud that's been working. And this was a big deal. This was a major, major version. So we knew there was breaking changes. And we also knew that the client had a lot. a lot of devices, a lot of legacy data that have been operating for years. on you know. The start to finish, I guess the TLDR is there's so much hurry up and wait, especially with the cloud piece. And from our viewpoint, it was. a lot of waiting for them to… identify what the design path is, what the architectural connectivity, how are they going to perform, and then having the approval of that. It probably took four months. Then on the other side, once that was done, us handling the actual moving of dirt of getting the migration completed, that would really, really well. but it took another six months. again waiting and you know I guess what I want to reflect on here is From our standpoint and from the network engineer standpoint, this is unbelievably frustrating, unbelievably frustrating when we're saying we've been waiting, we've been following up every single day for three months. And then we finally get a break. But this is a big deal for these big companies. And the risk for them of a misstep is existential. So, you know, the perspective is we, you know, from my standpoint, knowing this client and knowing what we're going into, I was a surprise that a, five to six month effort took 10 to 11 months. That's what it took. I mean, in the meantime, we had to, our team had to complete our work. The client wasn't ready to do their migration. They didn't need us for it. That at that point, we are the doctors in the white coats, right? So they want us there. So we actually had to kind of pause the engagement, put our pencils down. and then come back two months later to complete the final go live piece of this. So anyway, I guess that this is characterized by the work was understood. It took almost twice as long as it should, but… It was all client side delays, but I'm not putting shade on the client necessarily. There's just so much that goes into these companies as far as this is the first time they're doing things like, no, they're not adept. Migrating anything to the cloud is not a new thing. How do we handle the connectivity to support the underlying business that has never been in the cloud? These aren't necessarily straightforward decisions. And even when we escalate, say this decision, you have this option, you have this option, let's just execute. Even though let's just execute part can be complicated.

What can be done better though? So are we just always try, should we just always build in like a buffer time? Like, you know, you would take twice as long or it's going to be over budget or is there something that could have been done better?

I think, well, we were very proactive on this and jumping on it on, on on day one. My take personally is that You need to be proactive. You need to be communicating about it transparently. You need to have acknowledgement of what the situation is on the client side for an idea like this. that I think that there's going to just be adoption in the industry once this becomes more commonplace, once this is more standard operating procedure. I think that we are Breaking ground with these initiative of these automation initiatives with a lot of these big clients. And I think there's just a level of inertia and hesitance to change that we're going to have to just kind of grind through. And I think it'll, it'll kind of them organically improve. That would be my, would be my take on that.

So I guess the lesson learned is for the first time that you're doing this in your company, you should plan for, you know, some of this hiccups or, you know, you should build in that buffer time. And then when, when the second time is then, then that might be a better experience.

That's a brilliant takeaway. And I, and it's, it's, I think it's one of the most important things that I always talk to teams about when we talk about forming up a backlog, establishing an action plan. that forecasting what this is going to take. The first time is the worst time. Every time you build new versus increment on existing, it can be three times more time consuming, complicated, challenging. you know, more impediments, the, that are maybe simple, but you're still relying on somebody else to do something that they don't, they're unfamiliar with. So I'd almost say. You know, as an engineer, when you're looking at something that you've never done before, take your most conservative. Think about it for 10 minutes. They take your most conservative. estimate and triple it. And if that causes paralysis, double it, but explain why it's so heavy. Now, once you've established it and done it the first time, then you really get that. logarithmic slope, improvement in speed and refinement of the process. It always works that way, especially with these complicated projects. And there's nothing that isn't complicated about network automation years taking all of this chaos and madness and making it work every time.

Yeah, but also what I heard Matt, it's kind of interesting you said backlogs, right? So documentation, document your process so that don't waste a struggle, right? Don't waste the challenge. Like if you ran into the challenge and you solved it, but document the process, what was taking long? And then so you can look back and learn from those lessons, right? If it all… you know, kind of just live in one person's mind, then that's no use to organization. So another lesson I guess from, you know, what took away for the, you know, the five minutes is that, yeah, yeah, you're gonna, you're gonna run into these, expect them, but also document them so you could share that knowledge later on. So the next project would be better.

Yes. And that's, you know, that's a, that's a core part of what we do. It's kind of one of the, that's one of the, the core lean principles is what we're trying to do in everything is to build the quality. You want to identify what the… what the challenge is, what the solution path looks like, what the acceptance criteria, what would delight the client. And then you you know Once you understand these things, it's easy to keep adding breadcrumbs to it. And then every time box, if you're running an Agile or even if you're just running a Kanban periodically, you want to review what happened and pick out the things that were painful, pick up things that really worked well that you can iterate on that information out. And so there's this process of inspection, adaptation, and continuous improvement. But you can only really do that if you're building the quality in to begin with. So yes, you need to be, yeah, you need to mark the charts that way. It's a good habit. Takes a few seconds longer, saves thousands of hours for the…

For sure. I think, and also, it gets you to prioritize the low-hanging fruit, and which one was the most painful part, right? Because when, if you ask me on some of the difficult projects, It's almost like I could list five or six things that are really painful in my mind. But when I look at. the time it actually took, or if I just sit down and kind of put a point system on, this was a 8 out of 10, this was like a 5 out of 10. Then I quickly realized the 5 to 10 things I list, or maybe may or may not accurately reflect the actual pain, right? But if you document it, like you said, continuously reiterate the process, then assigning a… a data science way or scientific way to evaluate them, then you could come to an agreement as a team that, yes, we need to tackle X first because, you know, look at last project.

Yes. And you build, you build that, you definitely, that's to the extent that you get, that this is a team building conversation, to the extent that you can maintain team continuity for any given period of time, it really starts paying big dividends after a team that's well-formed can operate more than six months together. But yeah, anyway, these are all critical best practices that it builds the foundation upon which, you know, when you ask questions like, What does it take to really get done? Having these fundamental blocks as part of your standard behavior gives you the opportunity to move quick and be flexible.

Yeah, I mean, how many NFL teams pick your sport of choice, right? When the championship on the first year, the core teams are together, and how many of them are like after two or three years, they fell together, they've grown through those bumps and the tears and sweats and blood, and then, you know, the fourth year they went through. So, I mean, like I said, pick your sport of choice. But typically, I think there are teams that… you know, won the first year, but the typical process is that you have to go through those bumps. So back to what you talked about team continuity.

And Eric, let me, and that's, thank you for that. And you asked me what was in a recent project, what was the biggest surprise, right? Yeah. All right. My, despite all the delays that that's, I expected that that's, you know, just my advice on the delays and things is. Think it through upfront, understand what these are, communicate all the various steps because there are pitfalls out there. Communicate them, but be patient and have perspective. Don't let it frustrate you as an automation engineer. Now, big as a prize, the whole concept of the 10,000 hours concept, if you heard about it, it's a Gladwell concept. Basically, the idea is… In general, you can, you will be objectively considered a master at any craft for 10,000 hours into it. Right. I believe in that. strongly and I think that not only is the 10,000 hours concept a legit thing. It doesn't stop at 10,000 hours. I'm seeing, it pops to me when I see 20,000 hour people operating. And most of the guys on this job that I was talking about this big migration with a lot of risk. are that caliber. I mean, unbelievable. They were doing the, have you ever seen the Kung Fu movies where, you know, the two two teammates are, are fighting off from their backs are together and then they kind of lock arms and swap and they're fighting the other side. That's what these guys were doing as they were transitioning in and out because of the delays. And then they still just nailed it and it was beautiful. And so my biggest surprise was it's, it's still stunning. Every time you see a great Just a great performance come together. And I'm saying the accumulation of their experience was just slapping me in the face.

Nice. Nice. I like that analogy. Yeah. I like that analogy, you know? And I don't know how many, you know, if I listened to them watch Kung Fu movies, right? But, yeah, I could recommend a few. Some of the Jackie Chans, the, even the Bruce Lee's, you know, they, they do, they do back to back and then it's almost like it's the same person you carbon copy them and then, you know, they just swapped and yeah,

that's true operators. You know, as I'm saying that, that, you know, From my vantage point, it is the real world-class people who love this and do it, and they seek all the challenges like we were talking about earlier. They seek the reps, and they can get this good, and they accumulate all this knowledge. I mean… It's amazing what these guys can do.

Yeah, I mean, I'm a fan of 10,000 hours as well. I've read most of Malcolm Gladwell's books and also attended his talks. But I do want to also compliment what you just said with the fact that it's a deliberate practice. So it's 10,000 hours, but it's now one hour repeated 10,000 times, right? It's looking at– back to what you were talking about, finding out the logs and putting in the reps. But in every hour, incremental hour of practice, you want to go correct that weakest point or whatever that process that was the weakest link so that the overall process becomes stronger. So I like what you talked about that project. But if you look at all the projects that you've worked on throughout the years, including non-network engineering So what are some of the common mistake or the anti-pattern, quote unquote, in the automation projects?

Yeah, um… There's a lot of them and I like that term "anti-pattern" but I'm just approaching it from the standpoint of my experience with automation projects because that's also an agile buzzword. deep-seated in agile, but that's, that's a different kind of conversation, but I would say. I would say the biggest things that I'm always looking at and I'm always… contending with are silos, defended silos within client. So the audience here is anybody who's implementing automation projects within a commercial enterprise. uh… defended silos and inter-silo friction and inertia. understanding you know, having these chains that don't work together or they're defending ground or like they, they're There's politics involved, especially with a lot of older, these companies that we're working with, they've been around a long time. They've got a lot of established culture that goes back into the 1800s, some of them. Talk about telecoms, you know. So think about a telecom like a Cox communication, their business, they've got. They've got business, high speed internet. They've got local cable and internet. They're now getting into wifi and streaming. And so they've got this equipment of different types in different markets. and it used to be where These were standalone business units, but we get into network automation by its nature. Networking is all about. kind of breaking silos, connecting silos together that previously weren't connected. And so this is kind of what David Ezell, this is the thing I mentioned to you before. He says, what we're doing. is This automation and technological industry as it's growing is transforming. our entire networking industry, it's turning upside down because it brings into question the question of ownership. You plant manager run a DC in Austin, Texas. Are those your servers? Because we need to be able to have the network interact with those servers. from other business units. It's an exit thing, you have to have it. So this is the big challenge of this is people are entrenched with the silo. I live in the silo, these are my machines. I own this organization, you're not going to tell me what to do with anything, you know, that sort of thing. Versus the other way of saying… You've got the physical material and the physical plant, but it's purpose, it's owned by the network. It's owned by the network owns that that's how it's. single source of truth is accounted for everywhere. on Anyways, a big anti-pattern is… understanding the, it's important to at least understand from as a consultant. What players are working together, which players are at odds with each other? Because if we're asked to deliver a project where the goal line is defended by a silo that is really, really entrenched. That from a strategic delivery standpoint is something that we need to understand and we need to have. that'd be well understood by our client sponsor as well. You know what I mean? big. Big silo problem is a big anti pattern and automation projects. It's almost ever present. There are, there are outfits running it right away. I don't want to say it's 99.9% that it's prevalent. Another one would be um… If your leadership. Ultimately, the top leadership is completely disengaged, unavailable, uninformed. won't want to be informed when you're informed and they don't stay informed. If they're disengaged like that, there's a lot of risk there. A lot of automation projects. do so much great work only to be politically. undercut by turn over and changes, things like that. So. from a network automation engineer standpoint, one who wants to be on the grind delivering the hardcore coded value. Be aware of things going on like that in terms of who's the big boss? Is the big boss enthused about this automation to be delivered? They should be because that's the great power that we're in it for. They're disconnected or if there are other cooks in the kitchen. That could be just an unstable employment situation. Just be aware of that one. and then taking it back to the teams. biggest problem we have is We make assumptions without vetting and validating that given approach is going to work. We say, okay, we're coming into an engagement. We're going to do, we're going to do it this way. We got to get quickly with the client about what this way means because when we talk about firewalls and we talk about databases and we talk about OS versions, we talk about things like, you know, a lot of the devil is in the details and we need to stop making assumptions about that. Well, you need to make assumptions to get the ball rolling, but you need to validate those assumptions methodically and they need to be validated on a priority in terms of critical patterns.

Yeah. So trust, but verify. Right. And be diligent about it. Like actually do the, if this is an unpleasant work for somebody who, you know, the engineer that's really on the front line doing this. how somebody will pick it up. Maybe, if you're an ink, tell somebody to follow up on it. Like, tell your Matt Remke. that you need to track this as a risk. We need to actually facilitate conversations with the client, make sure that we run this.

Mm-hmm. Because if you because you can spend months and put months into something and then you finally get into a demonstration presentation and you guys take a look saying totally a different format. That's totally one that won't work for this. It's real. It's real. So ask it's better to take your medicine up front. Yeah, we, you know, you're not, you're not in the operating table demo, done the line.

Right. And by the way, Cox was on episode 83 for, you know, Network Automation Nerds. And I really do enjoy that conversation as well. So if any of our listeners are interested, you know, they were having the show notes and they could go back to, you know, listen to from, you know, from their own words, right? Like how they solve these problems or if they encounter these problems and how did they come about in. in their solution. But Matt, I think you mentioned that a little bit, along with what you talked about on make assumptions, but value the assumptions, the breaking the silos. So I want to ask, is there anything, any solutions that you would take? Let's just take defending silos, for example. How do we break that? If you observe that, how do we conquer that problem?

The first, my first prong of attack on this one is to make sure that, you know, hands above the table, we're communicating up through our sponsor client level chain of command. We're seeing this, we have this experience, this is our perception. There's a different few different ways we can approach that, but we're going to only approach it if we're agreed on how that needs to happen. Okay. So that's the strategic angle. You need to make sure that if you need cover fire or if there's, if there's some discussions need to be had at the, at the, at the leadership level, they're the ones having it and we're not the people getting chopped up in the meat grinder. But the other thing to do… is we, okay, let's just hypothetically say we, we know that we have an absolute delivery dependency on a, on a, you know, siloed team over there. If we have that dependency on them, the way that we have to continue to make contact with them. So if they're a dependency, we're going to be in contact, but that contact needs to be congenial and needs to be constructive. It needs to be appreciative. Like we're not just some, you know, mercenary contractor that's come in here to do this. We're trying to help make the whole system better and, you know, let's talk about it, but we're here for it. You can gain a lot of respect by actually showing up face to face. And even if you have to show up face to face for several weeks at a time, typically if your heart is in the right place and you have good positive intent. on You can at least at the tactical, at the level that the teams are interacting, they can establish rapport. And that rapport can go a long way because the trust goes right back up the chain to somebody who may not be in the room when that relationship's getting strengthened.

Got it. Yeah. So I think, Matt, you're mostly coming from an outside consultant perspective on talking about that. But I think it's also useful even if you're pushing an internal project so everybody's under the same roof. But if you just treat yourself as a quote unquote consultant, those recommendations still applies, right? Because you're– You may not have a direct influence, you know, you don't have formal authority over this person, but recognizing it, you know, maybe the defending the silo would be the first step. And then a genuine communication or having that conversation with the leadership, even as an internal employee pushing for those projects still applies and valid per, you know, your approach.

Yeah, and honestly, you know, when I talk about this friction, It doesn't come from the technologists. The language of network automation is understood by all that can read it. It's easy to build good relationships when you're good, you mean well. The big picture is you're trying to improve. Outcomes. So, you know, it's good. It's easy to be strong and come into our relationship. You got to bear that in mind. You know, that's why I think one of the important things to do when setting up projects is let's do our homework on the front side and let's understand what relationships we need to be paying attention to and strengthening. Also, what are the big decision points that this client has no idea about that we need to get them thinking about soon because this is going to be a tricky decision to get made here.

So I like that. So having a good intention, right? Because that would be pretty obvious on your intention and when you're doing the communication. And what you talked about, assessing pressure points and hairy issues upfront. That's a perfect segue into my next question, right? Because coming from your perspective, if you were to start a new network automation project today– Right in. What are some of the things that you would do? And one thing you already mentioned was just to take a moment to not jump in, you know, like, like the engineer in me always want to just get hands on keyboard and start like hemming out ordering parts and doing network designs, right? But you say, Hey, from your perspective, like slow down a little bit, do assessment first. So tell me a little bit more about that.

Great, yeah. So. just skip quickly over because I touched on it. Understand your organization and the technical landscape that you're about to start operating in. That means who are the critical stakeholders to the outcome? What's the technical landscape? What are they working with? Do we know for sure where they're headed? What are their standards and practices? What hardware do they have got? What's their footprint look like? Are they a regional outfit with one data center or do they have 13 data centers all over the globe? you know, anything that we can find out, even if we just pose the questions, it's good to get an understanding of that. as soon as possible. because the scale of your approach, will be informed by it. Okay. The other one is, I, and I mentioned, you know, See you Understand the value under where's the value at you know as an MBA we're talking about what's the value stream map that essentially You don't have to go crazy with this, but it is worth a diligent effort between you and any other stakeholders that are giving hands on or decision that appears in the process. And it's basically let's visualize whatever, let's just, you know, there may be weird outfit that has, you know, a hundred processes we do on a month to month basis. Start automating. Okay. Pareto analysis. Give me the most common. What's the most basic? Any of the ones that are the big ones. Give me the ones where, you know, 20% of your use cases are 80% of your, the volume of the work or the value you deliver. Get a sense of what, you know, what, what the inputs and the outputs are. Ideally and you know, my guys don't like doing this and I'm curious why even if it's a whiteboard And it could turn into an architectural diagram that draw the picture. What is the workflow that you currently are, what's your current existence look like from a workflow standpoint, even if it's rudimentary? And then what is a future state? And just the diff between those two will start illuminating. Well, if we want to go here, you know, we need to understand what you, we have to map the dependencies and what the whole flow looks like, because then you can actually communicate to the business people to hold the purse strings. So once you understand the various pieces, then you can just ballpark, but do it as an earnest estimate. Understand time, lag, and time on keyboard. How long is it, say, how many steps, so you understand what the cost? of the overall output is. And that way you can understand where you. can Save costs. Or it gives you through, you know, once you understand where you're at and where you're headed, you communicate it broadly fast, which is essential. to get common understanding, because you're always going to be preaching it. all the time. so you can get common understanding, but you can also make the decision so you can look at the future state and say, all right, we'll look. We can knock out this. step and automate this step. And, you know, we could spend 200 hours and it would save them 200 grand. Or we could knock out this debt and we could spend a hundred hours and it would save them $15 million. You can prioritize and knock out which steps of the value chain you need to optimize in a orchestrated plan with that. So. Just understand the big picture and be able to have honest… and efficient communications with your leadership about the takes and puts. Do we do this or do we do that? Or if we're going to do this, our option is this. But maybe this, but we better look into it because we not be able to get approvals to whatever. Anyway, but that helps you map out the approach to the you don't go charging 80% down a path that ends. And now all of a sudden you suck all that time and effort into something that is never going to fly. And then you got to double back. to that little methodical. patient approach. up front and then periodically you're going to have to do this because you know scope changes as well as I do, but at least you know where your true north is and you know, you can make sure as you go through the days and weeks, are you still more or less headed true north here? So I think that's an important piece. And then the last piece. that I think is real important, aside from the shared objectives, which that whole exercise covers. is to have our engineers be proactive. Let's, you know, it's not just the team, you know, management consultant or the leader of the project leader, the director. Reach out to your, you know, build relationships. If you have a question about something, don't wait till the next stand up. If you have the availability. just ping directly a point of contact that you think might be able to point you in the right direction and sort of establish and be proactive in seeking information because the more chatter and dialogue and flow you can get going back and forth. The more instant taking as power and more like collective knowledge is able to apply to any given discussion is on this you bringing attention to it So I would encourage people to be proactive. Get your, you know, roll up your sleeves and just. get going as long as you know that the general right direction. Let it fly and fail fast. Get out there, get it going. Don't have analysis paralysis. Looking at something for three weeks. If you're in that position, ask for help. But otherwise, when in doubt, you know. Go and move in the right direction, dig start taking.

Yeah, I like the part, especially about the planning part, because even if I just planned a trip from Seattle to LA, I do need to know where I'm going. And then I got to know, knowing where I'm going to LA, but I got stopped by Portland first. And maybe I go to San Fran and check the weather and everything else. So I do need to have my true north, but periodically assess if I'm off by a little bit, because as we know, you know, even if you just off by a little bit, if you continue down that path, you're going to end up in the very, very different place. I end up in Phoenix or something, right?

And that's why I say fail fast, because what we're trying to do is deliver functionality that we can show. Because no matter what, when your stakeholders see something actually working, they're going to be filled with ideas. But if you start and you build it and you show it, you can course correct without having to stop and go backwards. So fail fast, get functional. Perfection is your enemy here. Functionality is your friend. You know, what you want to do Build it to work, show it off, and then iterate to perfection.

Yeah, I just looked it up because one of my favorite quote was from President Eisenhower, right? So he said, "Plans are worthless, but planning is everything." So what he meant was, you know, the fact that you're planning is good, but then don't stick to this rigid static plan, right? Because they don't just don't survive reality. But it's great and then it fits perfectly into what you talked about in the automation planning. Planning is everything, but it's also worthless. You just got to go course correct and validate.

I tell you, in my experience, that… There is just, there's so much complexity dealing with large enterprises that we just. It's okay to make assumptions, but say that assumption out loud and front in front of other knowledgeable people. And you know, if you all still admit that everyone knows it's an assumption, nobody can validate and that's just a task because you just got to close these things out because you know, you can get. We can take approaches that take quite a lot of effort to work up and then it gets to an approval state approval phase. And then there's, you know, it's under the purview or somebody has visibility. That's a totally separate organization, but there's some legitimate infrastructure reason that they, that can't. So understand– What it takes to get it done, communicate, but also understand technically what your options are. I think the planning value is there because. Even if you're good at understanding it, sorting out your options, even if you. Like you said, you. day one happens and your plan's out the window, at least you have a skill set at sourcing, vetting, and deciding on options.

Yeah. Can you over communicate though, Matt? Do you think, because, you know, we've been to these projects where the conference calls like a hundred people, right? And then it becomes nobody's problem. And so can you actually, like, what's your solution? Like, watch your middle ground and include the right people, but no more.

I agree with you a hundred percent. Yes. You can never communicate. My, my old thing is I think that people are so. What is the word, you know, when, when you just, you become. There's so much static that you become deaf to it, right? You get, you get, you get, you get this tolerance, right? Yeah. Like emails, you know, here's 11 bullets about what we did. Be accountable. Every team member. we need to do that just on a, on a weekly at, at most, because nobody consumes email like that. So, but you need to do it with. earnestness. That's, I guess my thing is, Hey, this is where we're at guys. We're, we're, we talked about the goal. We understand the value of the big picture. This is where we're at. This is what we need communicate to only the people that are directly like in the, in, in the chain of action and really leave the other noise for a person like you know I meet somebody who is customer-facing as well. Yeah, you don't want cooks in your kitchen as you can help, but you have to get all the key stakeholders involved. So that's the line really to understand. Who are the stakeholders, who, the whole customer, you know, the customer leadership and then, you know, other key dependency stakeholders is where I'd keep it. And I agree with you. You'd really get yourself in trouble by broadcasting too much information because they just bog you down with trivial questions and things like that, if nothing else.

Yeah, I guess that comes from experience, wouldn't it? That identifying the key holders, but if you're doing it for the first time, I could see somebody having a difficulty, right? Like somebody who's been dealing with Cisco's and network design their whole career or most of their career. Now you're asking them to look at from their cubicle and identifying the key stakeholders, how to talk business terms, how to… tell them about the value that you bring around. I can see that's kind of difficult for the first time, but what you're saying is it's worth that practice to learn from the maths of their world to do all these necessary steps because the payoff is great at the end.

Yes, and you know, I'm speaking about that's a great distinction. I'm really talking about Automation teams as a whole not not in it. And if your job is automation, your love is automation. I'm always saying. ask these questions to the people who can be accountable for those answers. You don't have to go chasing off and do that, but be, think on the plane where you're asking the questions that you can at least have somebody. where you're waiting on an answer to validate something that you're not making a dangerous assumption.

Yeah, yeah, yeah. I think that's a great point. So if you're looking at, if you're a gamer, right? Like you have those like street fighter characters, right? You have like strength is 10, but the capacity, like being a charisma, yeah. So you're looking at your whole team. So you're looking at that person who has the communication skills, 10, and then you're the technical skill as you know, eight. So then combine your great team.

That's right. Delegate that to the person who… who within the team can handle that. That, you know, when you build good teams, you have a regular working cadence established these relationships. And a lot of times they'll be somebody, you know, there'll be a project manager, a scrum master. If you're an agile, a scrum master or a product owner is who those who needs to be accountable for getting you the team member, the answer to that question. Yeah, but I mean, don't having said that right, don't let us stop you from getting to this film. Especially once you if you're if you're proactive and you're engaged beyond just to your Scrum Master and PO and you're engaged with the client. That will be super organic and you can get your own answer. But if you're not in that situation. You have a path to get that.

Yeah. And I think that kind of corresponds to what we talked about in the beginning of our conversation, right? If you're in a startup scenario, there are more likely of chances that you start wearing different hats, that you could be the Scrum Master and you could be the guys on the keyboard. You could be, you know, doing the procurement. Then maybe perhaps in the bigger organization where, you know, you're very focused on this vertical and you have to go out of your way to learn those skillsets.

That's 100%. Yeah. is So many permutations of it, but ultimately, you know, that's I've seen I've seen Engineering teams that that rotate scrum master and PO roles that they just all just straight up engineers But they're following the practice because there's some value in… eliciting feedback from different perspectives so that everyone's informed and you know, ideas grow organically. Hey, if that's the problem you're having, it explains why I'm dealing with this over here. you know, let's meet in the middle and now let's start solving the solution of the problem of the symptom. That's just good team camaraderie, which is important to have.

That's a great point. I mean, back to we talked about Amazon, right? So I know for a fact, I've witnessed firsthand, right? Jeff Bezos actually, and the senior team actually appear out here to go take customer calls, right? So like the next time, I mean, they do chatbots nowadays, right? But the next time if you do get a human being, maybe you'll be Andy Jassy. I don't know. But back to your point about, you know, you're rotating. responsibility. So you get a glimpse of what other people's are doing. And then who knows, right? You may be a great Scrum master and you're just like, you know, and you're just been doing CLIs. So, yeah.

Yeah. And I guess the, you know, the, the point to the audience is if you're on a team that's doing network automation. It's understand that a team will work together and identify the challenges that they need to be in. They'll self delegate and they will self actualize and they will put that together. That's not everyone's case, but even if you have a dedicated role in a certain way. you're definitely going to come into new problems and great teams are. habitually engaged with each other and the problem they're trying to solve to the point that they can actually manifest solutions that can work so fast. And you don't need all this emails and instructions and everything else because they're riding the wave of the effort. They're not trying to figure out where the surfboard is and catch the next wave in. Anyway, I don't know if that knowledge makes any sense.

No, it definitely does. I'm just picturing this well-oiled machine. When it's good, it's so good.

Yeah, when it works, it's beautiful. Beautiful.

I love when you play and come together. Again, plug in whatever your favorite sports team is. When everybody's doing their part, when– The linemen are blocking, the running bag is running and your wide receiver running the right route blocking as needed. Man, it's beautiful. And that, I believe is why I watch American football, right? Because you're in the zone. It's like, remember that was the NBA 2K when the basketball got on fire and you're on fuego and then you get on the street. That's, that's a great, that's a great flow to get for the slam. Yeah, for sure, like people people who play into NBA 2K knows what we're talking about. But if you don't, you know, go go look up some YouTube videos, you'll see what we're talking about. So Matt, it's been a pleasure. I really enjoy our conversation. I know we cover a wide range of topics, but is there any, like before we go, is there any call to action that you want people to take away from our conversation? Either something I miss, you know, you want to bring up or anything that you want to just mention before we wrap it up.

You know, no, I just I would say, well, I say, I say no, then I start explaining. I would say, Pay attention to the bigger picture. Just understand thematically the big moving parts and pieces, whether it's industry changes, whether it's within a client firm changes, or if there's competing factions, or if we're moving in a technological direction that it's a this or that, we could go this way or that way. If the… Be willing to talk about that, at least within your team and say, I'm noticing this. I'm seeing that get, get some dialogue going because. You know, a lot of times we don't, we take it for granted that, you know, leadership folks are super happy to engage in that discussion. That's very insightful for them. So be proactive about talking about what you're seeing. from that standpoint. um… And, and have just positive intent, have positive intent, because you get, you can break silos and make such good relationships. If you're coming at it in a way that's on here to make the world a better place, I'm trying to do good. you know, you can cut through resistance and it's pretty rewarding as well. And you can make things happen so much faster with, by building good relationships, even on short notice with, you know, external people you'd never met go out there and, And don't be shy about it.

Yeah. I like that. I think you could be, you could afford to be vulnerable if you have good intention because people recognize that. So. I enjoy that piece of recommendation and thanks so much for being here, Matt. I really enjoyed our conversation.

Blast, Eric, thank you so much. We'll talk to you later.

Yeah, of course Thank you for listening to this episode of Network Automation Nerds. Do you have any feedbacks for network automation, our guest mat today, or this episode in general? Please send us some follow-ups at packetpushers.net/followup. We do want to hear from you. And please check out our website at packetpushers.net to find other network engineering focused blogs, newsletters, podcasts, YouTube videos, and so much more. If you enjoyed this episode and found it helpful, I really appreciate it if you take a moment to leave us a review wherever you get your podcasts. Your review really helps other network engineers discover the show and keep us motivated and bring you more great content like this one. We thank you the wonderful listeners for your support. Last but not least, remember that too much network automation would never be enough.