Network Source of Truth: From Design to Intended State

Welcome everybody to this recording of episode two of For The Journey, which is IP Fabric's podcast around, well, all things, I suppose, network automation, going on this journey towards a self-driving network. Now I'm joined by some really super special guests today. Hopefully these guys will need our introduction, but if you do need to find out who they are. Go look them up afterwards. Dave Donohue here, Cloud Architect with BlueCat. Hi, Dave. Good to see you. Great to be here. We've got Tim Schreyack, Solution Architect with Network to Code. Hi, Tim. And the other Tim, Tim Fiola, also with Network to Code. Good to see you, Tim. And today we're going to be talking about the place that we store the intent for an intent-based network, the much-fabled source of truth. A lot's been made of that need for source of truth in the journey to build this meaningful network automation. So what we're going to do is we're going to dig in to the why, the how, and what with some folk who really know their stuff rather than just me. Gents, I guess the obvious place to start here is why? Why do we need a source of truth? What benefit does it bring us? And I know, Tim S., when we were talking before about this, you got some thoughts on this, so you might want to start us off. What do you think?

Sure, absolutely. So yeah, I mean, when you think about automation in general, Here on the network side, we feel like we're kind of a little bit behind and we're catching up. The other, our partners over on the other side of compute and storage, they've been doing this for a long time. And, but their data requirements are so far less than ours. We have a really unique problem on the network side with the amount of data that we have to track, store and input into any form of automation. And traditionally we've done this by hand, you know, spreadsheets and whatever. And while that works fine when we were managing via the CLI and even as we moved into UIs and GUIs, it all still works that way. We still always had problems with this. A, we'd end up with overlapping IPs because we handed out things twice. We forget that VLAN A was being used over in this other data center. And we always had problems with this data tracking. But when we get to the automation, it now becomes a hard requirement. We cannot proceed at scale with automation if we don't have some form of source of truth. So our problem is unique. And I think that's why– We haven't seen a solution really prior to the network getting into automation for source of truth because it just wasn't that much of a requirement for the server side. Now on the network side, we're starting to see really this growing area of understanding and realizing source of truth and its importance guys thoughts

Yeah, I completely agree. Not only do you get it from the system side, but you also get it from the security side as well. That requirement to be able to say, hey, this is the exact device that's on this IP address is really critical when it comes down to it from a security posturing perspective. Like if you don't know what it is, you certainly can't go intervene on something going on on that device without knowing exactly who it is, where it is, what it is, and how it is. So I mean, that's certainly highly critical on that area as well.

There's a point I'd like to cover here. Yeah, real quick. Tim Schreyack mentioned the sense of scale. It's impossible to use it. It's impossible to scale without it. And that's very true. And now if you step back a moment to, let's say the engineer who's just starting down the automation path you know, in your typical day to like provision of VLAN, for instance, you might, you know, pull something from a spreadsheet, pull something from a database, synthesize that data into a config and go and paste that into the device. And that's fine. Now, when you start simple kind of abstraction, like let's say a simple Python script or a simple Ansible playbook to do that same thing, you can still use those same data sources. Oh, let me just go here to, you know, here and get this. Let me go here and get this. And you can just put those as arguments, simple arguments into that script or playbook, and it'll still work okay where it becomes more important now is when you start getting into the orchestration piece. It's kind of further down the spectrum on the automation journey there, where now you're dealing with step two depends on the output of step one. Step three depends on something else. And you're starting to deal with multiple network elements. At that point, you need a single source of data for your automation infrastructure or else your infrastructure is just gonna be used to find data instead of doing true automation. So it becomes a requirement when you start like down the orchestration path.

It's interesting. I remember before, I guess we're all sort of suitably of an age here in terms of network engineering, right? There was a time before network automation was really there, you know, have the focus it has now, where people would talk about the self-documenting network. The idea is that your documentation wasn't stored, right? You would go to the network to find out what the state was. But there's all kinds of problems that I can see with that, with the sort of thing you're describing there Tim. Oh, look at all those nodding heads.

So yeah, this is the confusion of actual state versus intended state. These are completely different things. And absolutely, we need to monitor and understand the actual state, but that is not how we should be driving configuration into the environment. We should be driving it from our intended state. And these are not the same. And in fact, this is something to point out really too, that our source of truth should not be reflective of actual state. Actual state should be reflective of intended state. It is exactly the opposite. And this is something I think for people going down this journey that really struggle with at first, they're like, yeah, but the actual network, that's the authoritative source of information. And 100%, no, it should not be. But that's a challenge to wrap your head around. Because you think, yeah, but it's in production. This is how I keep my job, because I have my network up and running. But when we're thinking of it from a philosophical perspective, our intended state is actually where we're driving from, not actual state.

Yeah, absolutely. It's the consistency piece, no go-to.

Okay, sure. And to put it a slightly different way, as a network engineer, if you're going to engineer your network, you'll come up with an architecture document and figure out, we need this many routers here, this many edge routers in each site and whatnot. And you create an architecture document. And then if the network doesn't match that state, that architecture, the network is wrong. Because same way with the source of truth, which holds data on the intended state of the network. Truth kind of means intent, really. In that if your network state doesn't match what's reflected in your data, in your source of truth– The network is wrong. It's your source of reality. It's not your source of truth though. Your intent is stored in your source of truth data.

Yeah, we tend to use the term observed truth rather than intended truth, right? And because obviously from, and again, sort of minor plug, but from the thing that IP Fabric does is goes and fetches that observed truth from the network. So, you know, that's where that comes in. And I guess, Dave… You've seen this, you know from a very specific angle, but you'll see this regularly and not in just on-prem networks, right? It's a broader problem than just the simple networks that we've deployed over the years.

Yeah, I mean, one of the big challenges is, well, I've got all my stuff on premise and– That's great. And I have all my existing stuff that in fact set up. But where it becomes difficult is when you have multiple different groups operating everything. So I've got my cloud ops group that doesn't talk to my network group. And now they've gone and created overlapping IP space and like GCP. And now they're like, Hey, we need to make these two things talk. And now you're sitting there being the poor network guy who's like, wait, now I have to double NAT this to make this work So now you've created a problem in the network because you didn't plan ahead. Right. And, and for me, I came from the network world and I spent a lot of days fixing those types of problems. That's like near and dear to my heart because I hate having them put in band-aids or bad band-aids to fix problems that could have been solved by just making a good decision upfront. So I'm always very… Very, very vocal about like, hey, look, if we're going to be talking about moving to the cloud or anything like that. We need to ensure that we've got one, the network team to the DDI team, which is the DNS DHCP, IPAM team. And then we also need the whole cloud team involved. It can't just be the cloud team working on their own over in a bubble by themselves because in the day it impacts the whole business.

Yeah, for me, again, it is that that that breaking down of silos thing, isn't it? And having that that central source of reference. And I like as the guys have said already, it is vitally important that we use the same reference regardless who we are and regardless of the of the resource, you know addressing VLANs, connections, whatever it is, right? So, I mean, what sort of other data, I mean, these are the obvious things, right, Tim S, what else would we put in the source of truth, do you think?

Yeah. So, I mean, this is an area like on the network side, we often cross the line between, or sorry, we are on both sides of fence of the physical side of the world and the logical side of the world. So we have to store all the information about the hardware and the equipment and where it's installed. And often our database is also storing all the other things. So all the servers, all the storage, all of that data is in our database, essentially, if you want to call it, if it belongs to somebody. And then we've also got all the logical side of it. So everything that we're doing, we really want to be tracking, whether it's VLANs and IPs, IRPs, ASNs, and all of our circuits, and power management. And the list just goes on and on of the data that we need to store on our side, essentially because we're the underpinnings of the entire infrastructure. Everyone else is relying on the services that we deliver. So that's where this notion starts to come in. And then particularly as you start to move into now, you know, kind of more modern design, you've got the underlay and the overlay. So now we have two essentially different networks that we're operating. So now we're, we just double the amount of data that we need to store. So we have so much of this data that we're trying to track, maintain and use, right? Because if we're not having it accurate enough to date that we can't use it, so what's the point, right? So it needs to be, you know, really how we initiate all of our change. From a practical perspective though, totally understand, you know, for many folks starting down this journey, there's going to be some necessary requirement, especially in the early stages of being able to ingest data into source of truth. So how do we start from there? And this is an area like you said earlier, just, you know, small plug at network to code that we put a lot of effort into, you know, with IP fabric, we have our plugins now that allow to ingest some actual state into the intended state as a way to kind of start Because while we can say in our ivory tower that, hey, you should only be coming from intended state, there is the real practical world, right? Like it's a journey, you gotta get there. And of course we're talking about the ideal end state. There's a path along the way.

No network is a greenfield, basically. Yeah. Yeah.

understood. And Tim, that's a great example. Like for BlueCat, we natively will grab everything that's DHCP'd on the network. All that ends up in the IPAM and then anything that's got a DNS entry, typically we'll have an entry for. Beyond that, we also extend that same kind of ability up to the cloud. So we slurp in all the data from say AWS GCP or Azure. And we put all that information into kind of much as I hate saying it, a single source of truth because been in the office for long enough that that's one of those words. It's like, it's kind of, but there's other parts missing and I do have that realization, but I mean to be able to provide exactly that information without some of the difficulties associated to it. Obviously, hard code of static IP stuff has to be put in manually. Just that's the nature of the beast. But all the DHCP stuff, so I know exactly what's on my LANs or here's all my book devices. So I can easily start go pushing out. Like here's the options I need for all my deals or for all my phones, for instance, right? And you can do that automated, obviously.

I guess the other thing is what we're talking about here. I mean, we're talking about pulling data from loads of different places and bringing those together that day. It's normalization, isn't it? It's being able to sort of almost abstract out the fact that you're deploying to different networks, different vendors, different domains underneath the hood, but ultimately what you need is, and it comes back to your point from before, Tim, that intent for the network has to be expressed in an abstract, normalized way, otherwise it doesn't make any sense, Tim, if you're nodding there.

Yeah, absolutely. The source of truth, you can think of it as an aggregation layer across all your data source, your authoritative data sources. And we should probably loop back around and discuss an authoritative source in a moment here. But yeah, your automation infrastructure should spend time automating stuff on the network, not searching for data. Let the source of truth aggregate all your data from all your different sources and present that upwards. Additionally, having all this data in one place also has a lot of other benefits. One of those is it exposes metadata that might not be apparent when the data is scattered around. And, you know, as an example of that, you know, if you have your IP address, your IPAM information, your circuit information, your interface information, those are all different kinds of data. But when you put them on the same data source, you start to be able to do things like move from circuit to interface to related IP address to a cable trace. So all this kind of extra metadata that's not apparent right away becomes becomes apparent and that alone is another benefit of the source of truth because it exposes other relationships that might not be obvious.

Yeah, I was just going to use the word relationship exactly that, isn't it? Because the model isn't just a flat copy of anything that you would have had on paper or on an Excel sheet. You've got relationships between those pieces of data which actually have a meaning. Certainly when it comes to even even to a troubleshooting scenario I suppose when you're trying to work out what you know where a circuit appears and which interface and which cable like you say that it's going to use You've got all that data to hand which is is going to be hugely helpful And I guess some of this data is going to be stuff that you have to manually put in there and some of it's going to be automatable. And this is, Tim, you mentioned there the aggregation piece and being able to bring data from other data sources. In that sense, then a source of truth would be almost a proxy for data from other locations, right?

Yeah, it has to do with authoritative source. And by authoritative source, we mean a final source of record that communicates intent. So for instance, you might have an IPAM that's holding your IP addressing information. That's great. Other systems might need that information. So that information gets spread out amongst other systems for them to process it as they need. But when it comes down to it, what should this be? you look to an authoritative source to solve that problem, to get the specific final answer. That is an authoritative source. So, you know, with the source of truth, you can keep your authoritative sources and you can pull the data into a source of truth. You can also do the opposite. You can also hold authoritative data in your source of truth, but push that information out to other systems to let them do what they need to do with the data. But there can only be one authoritative source for each type of data. So different types of data, you could have multiple authoritative sources as long as you're using multiple different types of data, IPAM versus DNS versus, you know, circuit information. Those are all different authoritative sources.

And I suppose the, yeah, of course, the issue that you don't want to have is multiple things claim to be the authoritative source because that's where the point at which you're going to get copies. You're going to, they're going to go out of sync and all of a sudden you've got issues, right?

Yeah, you want your network to run in the engineered state, the engineered optimized state. You put that information in one place, not multiple places.

which comes back to Tim's point, I suppose, again, about the expression of an intent not being that being effectively representing your design for the network, right? And then what you're doing is you're just using that data from that authoritative source to say, this is how I want the network to be. Now. I guess then you end up with a drift from that potentially. If you've got a network that's being operated by loads of different people, ultimately, somewhere down the line, someone's going to put something in that they've not gone to the authoritative source for. They're going to be, "Oh, if I just put this in, that'll fix this problem." And again, I can start to see you all smiling at this one because that's never happened ever. I don't know what you're talking about. So again, I guess then what we need to worry about is how we keep that up to date and how we go about that. And I guess, Dave, again, you're nodding. BlueCat has a particular.. approach to doing that? What way do you go about that?

Yeah. So BlueCat actually does that mostly natively. So like DHCP as something gets a new address, it automatically updates the actual server itself. So much to the intent, right? So as DHCP is grabbing addresses, it's going to run through the normal DORA process like discover, offer, request, acknowledge.

It's all early, it's not early.

Yeah, it's a little early, sorry. DORA's not there just yet. As DHCP happens, it's going to basically say, Hey, here's the address that I got. Here's my Mac address. And now I know exactly what physical devices there as well as the IP address. Right, so we do all that natively and then with the cloud, as things change in the cloud, it will actually automatically update itself. Right. Cause let's be honest an IPAM is worthless if it's not taken care of itself. Sure cause. Cloud stuff changes faster than we can keep up with period. Like, if you have some boxes off.

I guess with that one Dave, it's a balance, isn't it? Because you've got certain aspects of the intended state that are dynamic and certain that are static. And so I guess you what you and so I'm thinking back to what the other the teams have said about the about that intended state. And it's the static stuff you don't want to just learn it and whatever from the network. You want it to be as you intended it to be. But from a dynamic perspective, you've designed that into the fact that it's going to get allocated out. So at least by retrieving that state, you've got that to update your record. Right. So go on, sorry.

Sorry, my bad. I mean, in reality, you can't go tell AWS, oh, you're going to give me this IP they're never going to do that for you ever. Right. So you have to observe what their actual state is and then update your authoritative saying this is what that is of time being. And it really.. doing that at scale is difficult, right? I've got 400 accounts. How do I keep everything in sync? Right. That becomes really difficult, really fast. So the dynamic parts to that point. Yeah, absolutely. You kind of have to observe and be like, all right, cool. This is the intended state. Now the static stuff, absolutely. You need to say this is the intended state and these boxes should be at this point. I mean, every box I've ever hard-coded by hand, it's all right. Here's the IP we're gonna give it. Then you walk over to the box and you put it on the box. You're not like, hey, it's just gonna magically work. Right.

Yeah, it's all right. We've been saying for years not to hard code IPs on servers. So every time someone uses the word hard code, it makes me shiver. But yeah, I know what you mean. Sorry, Tim.

No worries. I think this underscores one of the challenges that we have faced as we move from the traditional method as traditionally as network engineers, we always wanted to have our hands in every little piece of the network. We wanted super handcrafted and bespoke and artisanal and we want to look at every packet as it goes by. But that doesn't scale, first of all. And secondly, there's no need. So why are we worrying about the details about a /32 that Amazon is giving us? Who cares? But yet traditionally we did. Like we cared about like what VLAN is it, you know, we use 100 for something very specific and only for that thing. But why? It's just an identifier. So let's only care about the framework. Let's care about the prefixes. Let's not care about the /32s in this example, right? Like in IPAM, or in VLANs, like let's care about ranges and big broad strokes. Let's not care about details. And I think that's kind of part of the transition as you think in your head, from an engineering perspective.

In support of the network engineer there because I was one myself. We had to. I think this is part of the problem. I think that people will talk about complexity that was introduced into the network unnecessarily, but often it was necessary because for whatever reason, the tools that we had available to us, we had to have firewall rules with IP addresses or we had to have, you know, we had to know which port things were plugged into because that's what we had to do. We had to be able to span subnets across data centers and those sorts of things that we had to have those kinds of tools available to us, but we don't need to do that anymore. And I think this is for me is the biggest shift is that as cloud has come along and taught us so that we can build applications differently, it means that we don't have to be as possessive about the minutiae. So we can normalize things and we can abstract things and we can create these incredible automation platforms that will use that data that you guys are talking about to actually do a lot of the grunt work that we used to have to do manually ourselves. Is that fair?

I think, yes, I would definitely agree. And it also is just the nature of that our roles have really expanded. We went from large networks used to, by today's standards, not very large. And we've really grown. And so what used to be manageable by hand is no longer in many environments.

And of course you've got not just one network to manage, but a network of networks, haven't you? Because you've got your campus network, you've got your wireless network, you've got your WAN, you've got your SD WAN because it sits over the top. You've got your clouds, you've got your etc, etc, etc. And so you have to understand how all of those things hang together. And I guess this is where the source of truth really comes in because it gives you that one place to go to see everything and to know how everything hangs together. That's really important. So we've talked obviously a lot about the networking side of things, but there are other areas that might have benefits from us maintaining this data as well, right? So I often think of how we can push our network data into other operational areas, perhaps who can use that. Any thoughts on that?

Yeah, this is always kind of one of those sort of fun things that happens, I think, from the network side because we are the underpinning of a lot of the infrastructure as we start to build our source of truth. Other teams use it sometimes in unexpected ways, something you didn't even realize they might think of or the problem they're facing as you allow them access to the information, they can start to do really creative and interesting things. In the early days of source of truth when in an environment I was working in we had built a network source of truth and exposed it to the rest of the organization and Some folks over on I think it was the Sort of like the application side that were managing DNS. They went off and built their own little app that handled DNS registrations So when you added a device into the source of truth, it would auto create a DNS record for you. And I was like, oh, didn't even think of that. That's cool. So yeah, just sometimes these sort of things just happen naturally, organically.

Guys, have you seen anything else that you can think of?

I haven't specifically, but I like Tim's example there because it tells you what happens when you're not focused on finding right data and when you're just, when you don't know what data is out there, when it's all right here, this is what we have. This is all our consolidated data. You can look at it holistically and go. Even if it's not your data, you can look at it holistically and then go, okay, there's a lot we can do because now we're dealing with, we know what's there. We can get it from an, you know, a single well understood interface that I can see how that would open up doors. So I really like Tim S's example there.

I think the thing for me, I always think of when I was in a network team, people would come to you asking questions that you thought were a bit oblique, really. How do I know all these circuits are up? Or I've got a bill for all of these, all of these support contracts for all this kit. How do I know that these are all the devices that are on the network? And so you can see how just having a simple inventory of everything that's in the network all of a sudden you've got access to commercially useful data as well that you might want to farm out to your support people or your contracts people or whoever in order to validate the amounts they're being charged by by suppliers. I mean this is the kind of thing that really helps for me. We call it data democratization. All of a sudden, you've got this big boatload of really interesting data that you can do new things with that you've never chosen to do before. And it's useful to lots of different people. So I think it's– I think it's incredibly useful. We just like you understood. Sorry, go ahead.

I was gonna say, we've seen customers take.. the single source of truth and then go wrap it into service now. So like people are requesting things about tickets and it's like, Hey, I need a ticket for a server for here's the server and Hey, everything's already right. Right. It removes some of the guesswork. So all that do is go find the name and it just works so we've seen a lot of that type of stuff where they're extending it beyond just as standard it scope and now putting in the hands of people. We don't typically have access to that information but that information is incredibly valuable for making sure that. We as ops guys have the right information to do what they're asking us to do for the business. Because in the day, if it doesn't tie back. It's worthless, right?

joy of APIs, right? You can integrate all this stuff directly now, which is incredible.

As Tim S was alluding to earlier, like we've done a lot of work in this area as well. As far as being able to use a tool like Nautobot as an aggregation layer to do syncing between these various sources of truth. Yeah, to exactly Dave's point of, you know, that we want to share the data. We want to get it out there. So when someone opens a ticket in ServiceNow or whatever the ticket system is, they have accurate things to choose from, you know, in those dropdowns. Those are actually reflective of what really is out there a great example. And even from just like the contract management side, like how do I handle all my maintenance contracts? How do I know that I'm paying for stuff I should be paying for? You know versus I'm paying Cisco a bunch of money for stuff. I don't even have anymore is a common example.

I couldn't possibly comment about being common, but I know exactly what you mean Chaps I've just realized we're already half an hour in And I'm thinking, are there any other nuggets that you'd like to share before we wind things up?

I think I would say from my perspective, the most common challenge we hear is like, that's great. You guys talk about this, you know, perfect end states. You know, you have all of these tools integrated and working and everything. And I just can't do that. It's okay start somewhere small, get a source of truth operating in your environment and pick one IP prefix to start working with or start somewhere. And you don't have to take the whole thing all at once. Take it as a journey, take it piece by piece. And we're needed, reach out for help. That's why we all exist. We exist here to help you. So sometimes it's just a matter of convincing your management or whatever, throw some dollars at it, get some help.

I know Tim F when we were talking before you talked about that journey from a slightly different way, right? The idea that.. you can start network automation without, but there comes a point where you really have a real cutoff.

Right. And don't let the lack of a source of truth currently prevent you from starting it because it's got to start somewhere and you don't need everything to start. Just start. And you'll realize great benefits from just abstracting, you know, managing scripts that manage hundreds of devices. But eventually you'll, you'll see the need to get there and you'll probably realize once you're there that you need it. And then that's what we're here for again.

Perfect. And Dave, any thoughts before we close?

I think these two gentlemen hit the nail squarely on the head. Yeah. BlueCat kind of enables this, the automation side of it just because we do behave as that single source of truth natively. So for us, it's pretty easy to kind of help walk customers towards automation that what I talked about earlier, that's cloud discovery and visibility. That is exactly an automation we built ourselves to go out and fix those cloud portions of the business where we have disconnects.

I think you've highlighted something, though, that people don't have to do everything themselves. There are capabilities in all of the platforms we've talked about today that people can lean on to take a lot of the pain out of some of those requirements, which is hard to do. A lot of this network automation stuff that people are constantly sold is all about programming Python to do this, to do this. You don't have to learn how to do it all yourself. That's the point here, isn't it? There is a community around. There are people to ask and there are tools that allow you to move along down that path. Gentlemen, I'm going to wrap it up there. Thank you very much for joining us. What we'll do is we'll share in the notes all your contact details if everybody wants to be contacted. Very good. If you've got any pointers, any posts, any blog posts or anything like that that people might be worth them following up with, let me know and we'll just pop it in the post, the notes afterwards. Thank you very much for joining us. I appreciate how early it is for all you folks. So yeah, much appreciated. Good to see you all. Good to see you too.