Application Dictionaries: Better Security Policy Management With Automation

Hey everyone, glad we were all available for this meeting. My name is Ethan and I'm the security engineer for this project. You folks are the application owners, the lead developer and business stakeholder. Okay, starting with you, Business Human, can you tell me which departments need access to the app? You are not sure. Okay, well, should we expose it to the internet? Not sure of that either. Ah, okay. Well, can we agree on whether or not remote VPN users should be able to hit this app? You are not really sure on that. Okay. You're not helping me out much here, but maybe you know if the app's data is governed by PCI, SOX or HIPAA. Not certain, but maybe HIPAA, you think. Got it, that's a start. That's something we'll turn to you then, developer human. Can I at least build the firewall policy because you know the specifics about the app because, hey, you built it. So I'm sure you know the answers, right? Well, I mean, one of those things is ports. What are the TCP or UDP ports your app talks on into which hosts? Not sure what I'm asking about. Okay, there's a database backend for the thing, right? What is the port profile there? Is it standard SQL stuff or probably SQL, but again, you're not sure. Can I run a sniffer and just figure it out? You ask. Yeah. Yeah. I guess I can fire up the sniffer. I'd love, I love that. Good job, everybody. I guess I will just get that sniffer fired up. And if you need anything, follow the sound of quiet sobbing punctuated by the occasional scream. Of course, eventually you get there, right? You figure out the app, you get a firewall policy stood up and it mostly works, but who owns this policy now? Networking, security, the business, the dev team. And where does this policy live? Does it live at the edge of the network? In the middle, maybe in both places, 'cause do we trust the edge? What if the app is deployed in containers and there's a lot of container to container chatter that never even sees that edge firewall? Well, how do you manage that? And then what happens when the application profile changes? Cause it's gonna, do you run a sniffer again? There's gotta be a better way to keep track of all this, right? Our guests today areKen Celenza and Brett Lykins from Network to Code. They've been thinking hard about how to manage security policy in modern IT infrastructure. We're gonna get into sources of truth, application modeling, approval workflows, and more all in the context of automation. So Ken, I wanna start with you because this is all your fault, buddy. You wrote a post on the Network to Code blog entitled Application Dictionary. So just so that people have context about what we're talking about here. Give us the the elevator pitch the 10,000 foot view of what you mean by application dictionary.

Yeah, thanks, Ethan. It's essentially the source of truth and data model that's associated with your applications more so than the firewall policy. Essentially what I'm trying to do is disaggregate the model for the intent of applications and the connections in between them versus the actual policy, which is essentially a build artifact, meaning it's just the configuration that's pushed based on the source of truth. But not currently we really take a firewall security policy centric view of the world. So I know that's a lot to kind of digest everyone at high level, but that's the elevator pitch there.

Well, you said firewall centric view of the world that we do, those of us that do firewall management tend to think in the sense of we have a firewall, there's a policy we have to build that often comes in five tuple format. And so we were trying to get those questions answered, but in actuality, what's happening is an application to application communication happening at a higher level. So if I understand what you're saying, you're taking an application dictionary is a big idea of. breaking down. these communications into the component parts that you can build a policy out of those component parts.

Exactly. And those two primary points are application definitions and then communications between those applications. And you quickly run into issues where what defines an application for a user community within a campus environment, as example. And there's a lot of data modeling issues and tricks that you have to kind of work through. But nonetheless, it's something that's solvable given enough time, I guess. –

Okay, so let me play contrarian here and ask you kind of a hard question. How solvable is it actually? And what I mean by that is now we have zero trust. Now we have, like I brought up in the intro, things like container to container security where those are comp– with one another, there's maybe not a security surface that they traverse. Well, there's always something, right? There's always a place you can put a What? Not the brand in general, the checkpoint. You know what I'm saying. Yeah. So how solvable is it? Because it seems like where we do security is gotten spread out all over the place now.

Yeah, I think, and the spread, I think one of the things that I've seen is when you have an application that's spread into all these different enclaves. It's in a container, it's in the cloud, it's in these different locations. Under the hood, the same networking protocols are still getting used. Those applications are still establishing TCP, UDP ports, and sending traffic across that. And so where the application lives is sort of independent of What communication flows does the application need? Because until that is– understood and documented. Even if you could, you can insert something on the control plane of your container solution to firewall traffic between the containers. you wouldn't know where you needed to put it or what it needed to do once you inserted it if you didn't have that information to begin with of how are these containers in your application interacting? Is there a database sitting over here and it needs SQL ports open? Is there whatever it needs? And this is the front end web container and it needs 80 and 443, whatever it may be. If you haven't done the work up front to profile your application and built this dictionary of details, And we'll get into who the you in that conversation is because that's a whole separate point. But if you haven't done that upfront, all of the bells and whistles fancy solutions for securing your container infrastructure. They don't do you a lot of good without that

So I just want to just kind of inject there for like how solvable is it I think is definitely a good question. One counter question to that which is not solving it, right? There's going to cost– there's an extreme amount of technical debt because– and I'll just give some examples here. So, you know, containers like, you know, Kubernetes and whatever, whatever orchestration systems and clouds, you know, cloud systems provide their own security. You know, you have AWS security groups and various different methods of actually control and having control points. Traditionally, the method was just to rely on the networking team and the firewalls in between to house that. And that's not gone away. And so you have this disaggregated choke points or checkpoints, as you mentioned, Ethan, where you can actually do policy enforcement. Now the problem is, is no one's giving up the reins in terms of, it's just duplicated. So if there's policy enforced in Kubernetes, it's also enforced on the network as well. Right. And it's redundant. It's kind of, you know, it's a tremendous amount of technical debt that's kind of associated with it. But to get back to your question, though, is, you know, how reasonable. I really think, I mean, there is a hard data modeling challenge there, but I truly think the hardest thing is the cultural change and the cultural shift or moving ownership of the rules from solely on the security engineer. to maintain all this data across all facets of the infrastructure, and pushing it back out into the business, into the application owners, the business owners, the programmers, et cetera, and having that shared ownership there. –

We're up to three problems now then. Okay, this is interesting. 'Cause we've got the application profile, we've got to understand the communications that needs to happen. We need to understand secondarily, the endpoints across those ports who should be communicated with. So that endpoint to endpoint communication. Now a third problem Ken you've brought up is the human side of it, organizational side of it. Now who's responsible for all of this because historically in most of the shops that I've been tied in with, It's been, you know, for lack of a better word, my or the IT infrastructure team's problem to figure out that security profile, to create that firewall rule set or whatever that profile is made up of, that policy is made up of, and then maintain that thing going forward. Ah, it's time for the annual firewall rule audit. Gosh, are we using the rules in this group? Well, they haven't been hit for a while. Does anybody actually know? And it seems to be incumbent on the IT infrastructure team to figure that stuff out. When in actuality, we aren't the ones that actually know the answer to that question. Someone else in the organization does, but how do you get those answers? So, okay, now we've defined a lot of challenges here, three different problems. I don't know if it was Ken or Brett, but one of you guys made the point of the redundancy that you can have. Because if you've got firewall rules on the endpoints, do you also need the same firewall rules in the middle? And if so, why? You said technical debt. Right, now if something's broken, there's a communication that's broken. We've got many more places that we need to check and find out where the problem is. Well, Ken and Brett, I'm glad you guys are here to solve all of this for us today on this podcast. So, okay, let's start at the beginning then. How do you begin breaking this down? Do we want to start having the component, the human aspect, that part of the solution first? Or do we want to start talking about how to model an application and coming up with its component parts?

talk about the modeling aspect first. The cultural, there's just more tips and tricks and aligning incentives. But fundamentally, they need organizational buy-in one way or another. I think it's really most of the answer there is just– generic nature, right? How do you bend an organization to your will?

that sounds really manipulative. Ken, evil villain. Well, let's start. Let's start with modeling. in, my, my imaginary scenario in the intro, I made mention of using a sniffer because sometimes that's what I've had to resort to. If I can't find any clear documentation, you know, fire up Wireshark, what is this box talking on anyway? Yuck. What do you recommend for modeling? How do I figure out ports and IP addresses and such?

Yeah, the hard work is still there. That's the unfortunate truth. You still– The idea should be in that cultural shift, you should educate your consumers to understand. If they're going to have to take ownership, ideally, they're going to actually want to know because they're actually associated with this risk profile of what they're doing. Right now, I think there's such a disaggregation. It's lobbed over a fence and they're not associated with it. I often talk with customers about– Simple things like, no longer allowing the firewall request forms to come in as an Excel spreadsheet. That's not. You can decide. There's no kind of data enforcement, model enforcement in that spreadsheet. You could put a subnet where there should be an IP address, an IP address where there's subnet, et cetera, et cetera. And really what that does is to, what I want to see is the requests are saying, I don't know how to fill this ticket out. So I'm going to call you and we're going to fill it out together. But the firewall. The security engineer shouldn't be the one always just fixing all the issues. But nonetheless, you're still fundamentally left with having to help people. profile their traffic. –

Okay, I think there's an important point here. You're saying the security engineer shouldn't be the one responsible for fixing all the problems. They may have the technical know how to do so, but you're saying don't just let someone toss you a kind of a half-baked request. Work with them to get it right. So is that an educational thing? You want them to kind of understand what they're doing?

Yes, it's education with enforcement and the easiest way to think about it is like in your ITSM or ServiceNow, basically saying here's the format of what you can put the data in. You can't just put Excel spreadsheet, which doesn't enforce anything. as much as you try, it doesn't actually enforce anything. And if you can't fill this out, here's the other ticket that says request an hour with a security engineer. But fundamentally, the security engineer is not going to type anything in to these requests anymore. Right. And so just getting that person to change it from the, you know, like who's groveling for who's, you know, help. Right? And so when the security engineer has to grovel for the requesters help to, you know, to fill out, fill out correctly and to say, well, you put in an IP address. It doesn't make sense here. That's a public IP. Okay. It's not it. Okay. Let's figure that out. You know, whatever those, you know, hundreds of different questions. It's really just saying literally who's typing it. And in that, in that process of having the requests or typing out, ideally you have people that are requesting not all the time, but somewhat regularly and they learn throughout that process.

Yeah, I will say as network engineers, IT infrastructure engineers, one of the things you always, in most organizations, there's some form of disconnect between that part of the organization and the application developers or the people purchasing and acquiring applications and installing them. and One of the things that I've seen work well is just assuming good intent and not in the sense of when if you do get that form that's not completely filled out, you go back to the other party and you ask them to populate it. Their answer a lot of times would be, I don't know, how can I find this out? And being able, and how to find that out may vary depending on the organization, but being prepared inside your organization to say, oh, you need to go talk to Joe so-and-so who stood up an application like this a couple months back, go talk to them and see how they figured out what this looked like, or those kinds of things where– You are It's organizationally dependent, but you're building those breadcrumbs and you're building those relationships across with the other parts of the organization. Traditionally, it's been a lot of Excel sheet flinging of we just need these ports open or this application is getting installed on Tuesday, open all the ports on all protocols. And you're like, wait, wait a minute, a security engineer. You're like, that's my butt on the line now. We need to have a conversation. And so it's starting to shift some of that Excel sheet tossing that's been happened to really being more conversational.

Something just clicked for me here when we're making the point about okay, don't fill it out for them. Go back to them I'm still thinking in my mind I'm gonna call them and we're gonna fill it out together because I'm the one who's got the answers and I'm gonna say yeah Here's what you meant you meant, you know this you meant whatever and then help them fill it out But you're you guys are saying even push it back further and say don't give them the answers even if you think you know You really want them to find resources when the organization so that there is an ownership. coming from someone else, giving you that authoritative information of what should be used to build a policy.

Yeah, exactly. If it's coming from somewhere else in the organization or the vendor, I mean, in a lot of cases in large enterprise data, you're seeing applications that are acquired from a vendor. If it's whatever that large SaaS provider may be. They usually will give you that whoever's bringing that application in a cut sheet of ports and protocols, be like, hey, go find that cut sheet of relevant ports and protocols for your SaaS product that you're trying to bring into the organization so you can fill this form out. I mean, so it's about directing them to those resources rather than being the resource.

Yeah, absolutely. I think, you know, definitely directing them and ultimately you will get into the point. I don't want to sit there and say you're, you know, you're going to be dogmatic about it and say, no, I'm not going to answer your question. If it's just reminding them, you know, you need to put the network to subnet, not the IP. You can't use zero or can't use that one for a slash 29 network. You need to put the dot zero because that doesn't make sense in the networking world. Right. So simple things like that. Of course, actually, interestingly, I kind of hit it a little probably about two or three organizations in a row. where they had specific people around, like it was like a cross knowledge, cross domain team that was there for things like this exact problem, like the security problem, but also like cloud, like how to build your applications, like how to build applications in the cloud world. And one of the parts that they were owning was this idea of security. Right? So how to actually, this is the group you go to, to advise. They're just an advisory board, essentially. And they just produce content like for the application teams to say, this is how you develop the cloud. This is how you secure your cloud applications and, you know, in private or public cloud, that's less relevant, but more of the understanding. But just this cross domain team that has a network resource, a programmer, you know, a security, like all those different kinds of components. And they all had like absolutely like they all raved about it. Like, you know, even when things weren't perfect, they all just said. things are much better with this kind of dedicated model to help people and just knowledge share basically with them.

Practically speaking, you can get into some different conversations in a group like that because depending on what viewpoint you're coming from, you don't think in terms of IP addresses. And some of that's even percolated up into filtering products, right? Like if you look at VMware NSX, the way they filter is with groups tagging, you know, it's happening at a much higher level that eventually that system will translate into IP addresses and policy that they shove into endpoints for you. But policy is not written or expressed in terms of IP addresses. Does that– happen, does that kind of conversation happen in groups like this?

Absolutely. You're hitting one of the points here, which is Kubernetes, the intercommunication between an application is not really expressed in IPs, the IPs can be arbitrary. That's not even expressed there. A lot of these things don't actually have IPs or even FQDNs associated with them. This is a lot of part of the data model challenges. It's no longer just a five-tuple. There's a lot of other things to consider, things like if your application should be able to scale out, how does the rule adjust with it? What is that kind of, again, in the container world where there are no IPs preemptively, they're not statically assigned. It can be brought on a different cluster and it can have different IPs locally. You can no longer express it within a standard IP. Right. And so there's, you know, I know like in, you know, in Palo Alto, they have like EDLs, external dynamic lists, which are just like, you just point to it and then you manage that, that And these are all things that are, you know, maybe to get back onto the tech, some of the technical modeling challenges. These are all things that make it very difficult, right? In the new world, it's making it very hard to kind of navigate.

So when you, you know, just to kind of tack on to those, you know, you can have things like URL based ACLs, right? Like you want to block it to a specific URI because sometimes you have like a load balancer and there's 10 applications behind that load balancer. Just your URI, you know, load balancer, I'm forgetting to wear it there, but, but here's your app, you know, like I know most vendors, most firewall vendors now have beyond the app ID. They have not just TCP ports, not just 53, it's DNS. It's just totally different making sure that you're not able to kind of tunnel over that protocol. So all of these layers, seven awareness, you're saying layer seven awareness. Yeah. Yeah. Yeah. Of course that's, that's probably the more succinct way of saying it. And so all these things complicate what that data model looks like.

Well, let's, let's, let's take a simple example then of a data model rather than, you know, we're kind of modeling what models are super meta right now. If we're to take an example, a simple example, maybe we should start with something as straightforward as, as five tuple and then build from there. What would that data model look like?

Yeah, I want to say away from a little bit of the five-tuple for a second. Right. And so let's, but let's go to, the three parts of it, meaning how I would identify an application may be something like just your traditional three tier application, web app and database, right? And so what I would have to do there is I would have to say, I'm just defining myself as an endpoint. And so I would define my, just start with the database, and say the database says 1536. And my IP or my hosts of IPS, right? This is the cluster of IPS, which I could be on. So, so again, but the big thinking here is endpoint and it's, it's very hard. I don't know. Still in my brain, I always think about IPS, but we need to forget about IPS from, because They're sort of irrelevant from they matter.

The way application development has evolved in the last decade or so… all the struggles that like I've been dealing with these problems for 15 years now. And what I've seen is. Do application developers have cared less and less about IP addressing each year? Like every year it's iteratively become, and it didn't start out big to begin with. And so like their way they develop the applications is just assuming that. the network level, that transport will be there and there will be an IP address and they don't care what it is. And where I think we're getting to now as security practitioners and networking engineers is trying to adapt our toolset to that mindset of how they've been developing applications where the IP doesn't matter. And all that matters is the service level things of this port and this protocol. It doesn't matter where it lives on the network because the application developers don't care. When they're building locally, they just spin it up and it's on local host or they're developing in their Kubernetes cluster somewhere and it gets an arbitrary IP and they get an FQDN and they go to town. And that's where I think now it's really hard to break that mindset of thinking about the five-tuple. I need an IP or a network and I need this where you get to, we're just playing catch up in some ways to where the application development has gone based on the technology that's available. –

You said service discovery and that was the magic word to me as I've been digging into Kubernetes. That's where so much of the magic lies. It's like, oh, this is service discovery. If your service discovery process were to break down within your Kubernetes application platform, You're screwed. Everything relies on service discovery, probably DNS and pulling things out of etcd that direct you to where the thing is, but what the thing is that you're getting hold of happens at a much higher level than IP. That's like, oh, well yeah, it has an IP. I don't care what it is. I'm gonna use service discovery to tell me that. It's probably gonna change a lot over time. –

Yeah, I mean, I go back to the old analogy of my name is Brett. I will answer to Brett. regardless of if I live in this house or I move 10 houses down the street, I'm still Brett. My mail will come addressed to Brett and I'll open it and answer it, but I'll just be living at a different address. I mean, it's that very simple. disaggregation that we in the, in looking at firewall five tuple and putting in permit IP one dot blah, blah, blah. We've just been very monofocus on that for so long. It's hard to pull up and think about it in that way.

But still going back to modeling then and trying to express what we're so used to that that IP oriented mindset can, you're describing something that's a thing, an endpoint generically that has attributes of some IP addresses probably. Correct. I mean, of course. –

It eventually equates to some IP at a minimum layer seven, as you kind of mentioned before. depending on the level of depth. But I think when you first get introduced to networking, you talk about IP classes and classful routing, nobody actually uses classful routing. That's just the way of the dodo. But it's just the idea just to simplify it, we tend to talk about it. If we just go back to simplify and go back 15 years when ESX was first introduced, that virtual virtualized hosts were introduced. And we still stick with the IP and port just to keep things simple. And then just know that anytime there's an IP in port, there's all these other challenges because that could be very dynamic from FQDNs, app IDs, et cetera, et cetera. But if you go back to the idea of like you have an SQL server and… You define your application to sit there and say, "My SQL server is 1536 on IP 10.111." That is part to define this application, just saying, this is who I am. This is how I receive access. And then on the app server, you're going to sit there. You're going to have maybe some HTTP, HTTPS for an API server or some sort. And you start saying, this is– we'll call it widget app. And so widget app– sorry for the redundancy, but widget app as an application server, talk to widgets SQL. You define those two as two separate things under one larger ecosystem of this defines the application holistically. Here are the different components within that application. And now I can sit and say from my widget app server, which is this IP 10.1, that one, that two. I want to make a connection to the widget app or sorry, widget SQL, which is 10.1.1 on 1536.

And where I've seen, and so what I've seen in the past is organizations take the approach of… For example, if they're trying to build this information, they either sniff the network for the traffic that's going, or they port scan this infrastructure and say, "What ports are listening? If I sweep every IP in there, all right, this must be a SQL server, it's listening on this port. This must be a web service listening on that port." But it still only gets you half. the answer because the other half of the answer is not just what to comprise that application, what ports is this particular piece of the infrastructure listening on. What is that piece of the infrastructure talking to? Where is it going?

So in our model, Brett, we've got… All the listeners, if you will, on one side, would you say that's one part? are one I don't know, one set of key value pairs maybe to describe it that way in our policy. These are the services that we're listening for in this portion of the application. And then it also keys up to here are the services we need to reach out to. What here are the services we communicate out to or the other portions of the application structure

depends on the model exactly how you can break that down. You can say. A lot of it sometimes gets into, you can really get down into like a database modeling problem of how I wanted these keys to relate or whatever. It's more generic than that. And a lot of people dive into that kind of level of it, but it's more generic of almost even. I have this unit of infrastructure. It listens for these services. It needs to talk to these other units of infrastructure on these ports and services. and how that gets expressed in the model is usually about the endpoints themselves, how do they, the elements of the infrastructure, how do they communicate? That's what you're trying to model.

Yeah, really that's what it is. You have just two primary large bucket item components, which is the service or application definition, and then the connectivity between them. If we go back and just take that example, so the application server, you need to define even if you have a zero trust model, you actually need to define between this app, between components of the app, how do they communicate? You have two application definitions, right? In a app server, in an SQL server within the same app ecosystem. And then you can connect them and say, well, this one should talk from A to B, not B to A, right? I don't need to talk 1536 to the app server. I need to talk 1536 to the SQL server, right? But then the application has to, the app server, presumably everything needs LDAP. So you need to define someone had to have already defined LDAP as a service or as an application, expose that and you need to say from widget app, I need to talk to company LDAP service.

So does my endpoint Model have just listeners and then it's a separate table in my separate. Hierarchy in my model that describes the relationships, the in and out that's a lot. Yeah, actually.

Exactly. And there are endless amounts of complications in that because Right away, it's like, okay, I have already a hierarchy within the application. And the things that I've already kind of touched on one, which is the different components of it. Like I shouldn't allow my end users to talk to SQL directly. So they should only be able to talk to the front end web service. So I actually have to break out and then sit there and say, here's my definition. Within the same app ecosystem, this is all the things that make up the app. So I have a hierarchy already. Right? And to say, this is the entry point for users, this is the entry point for developers to actually go directly to the SQL server and have to define them differently. I have to, even within the app, I have to define the relationships assuming a zero trust model. I've seen organizations just. or go to that and say, in intro, they just put an app per subnet or an app per cluster and just leave it open. Because it's a bit more realistic for the amount of data you have to consume. But nonetheless, just go to the full– or some filtering but you know

Limiting it to, scoping it to local subnets, but not being more granular, like down to ports or individual endpoints within those subnets.

Yeah, within them. Yeah, within it. It's like you have an app, like space, say it's like this is your unit of work. You know, this is your place to do your playground. your new app, new wedge app, we sign you a new slash 25 or something like that. All your everything goes within that slash 25. And you have a playground in which you can communicate unrestricted within there. Nothing goes in or out of that subnet without being vetted through some kind of process, some filtering component. –

And in practice, I've seen sort of combinations of that model. And there are pros and cons to each of those depending on the compliance requirements in place and the security structure in place. And so you see some things where… Even in a single application, if there's a lab, a QA, or like a dev, a QA, and a production instance, there may be the production may actually have more stringent port and have the blocking among the elements of the application and the need to document that. Whereas your development in your QA environment may just be that kind of all in one, you're behind a security enclave. And what goes on behind there, we don't care. And that's where sometimes even a single application, there might be different entries in your application dictionary for the different environments, for the QA1 or the production or what have you. –

Which is a trade-off position. There's an element of operational, being able to run this in an operational environment, being able to just keep up with day-to-day with… Well, that is not zero trust. That is not, you know, true micro segmentation. We're pushing a firewall rule set into each endpoint and being super granular about it. So they talk about the, the crunchy outside and the nuggety center. It's a little bit of a nuggety center there with this model, but the but on balance, it's a bit of a trade-off where it's a little more practical to run in that world. But malware and such becomes a little riskier.

Yeah, you start to have to do a lot of the organizations where I've seen this kind of thing succeed have very mature security models because they've had to because they've had breaches. They've had security incidents where they need to. They really, they understand the stakes here of how to operate and the aspirational goals are that zero trust style model, but they still have a legacy estate that is in the state it's in and it has these open rules that no one owns or whatever. And it's about that trying to make a transition of having a plan of, I want to get from A to Z. This is a framework for navigating that transition. –

Yeah, I absolutely think the rate of failure with the amount of data goes up substantially, right? Just by saying zero trust or not, that could, you know, two, three, four, five, 10X, the amount of data you need to capture and control. And thus it's that much more fragile, that much more, like no one actually understands the system. So, you know, in the pursuit of more security, you may end up less secure. And so, you know, it's a level of maturity. I often see… I've seen every organization go through the segmentation path, or many organizations say, and they parse out all the apps to /28, /29, this, that, the other thing. And we're like, okay, we're going to get it in place in the new data center, and then we're going to turn on filtering. And it never happens because it's just too much. No one could ever do it. And they try to take on too much at once. Right? I was rather saying like, let's just at least put the proper choke point in front of the data center. Step one, step two, let's segment it to the subnet, step three, zero trust. And that's realistically a multi-year journey.

So we've talked about a lot of the challenges here building the data model because of the realities of operating a data center, operating a computing environment. All right. With the model, that's just a model. All we've done is express endpoints, listeners, and then relationships between these endpoints, what should be allowed to and from these various endpoints on these various ports. And we've done it as a model. So we would describe that model in this context as our source of truth. Where does that live? Or maybe it doesn't matter for purposes of this conversation, but what does that look like? Is that a JSON? sitting in, I don't know, some kind of a database repository or NetBox or something.

Yeah, so I think you fundamentally need a database to back end it. How you're going to approach this. It's too much data to– to use in traditional infrastructure as code YAML style. It's just way too much data to manage. –

And the relationships are key, which is where it's kind of one of those perfect use cases for everybody loves keeping these in YAML or talking about NoSQL databases. But this is really an instance where we care about the relationships and sticking this in a SQL database of some kind is. Key to making this function because when you go to do things with this, you have this data, we want to take action and build policy. You care about the relationships when you're building that policy.

So we're not, well, we could be talking about JSON at least as part of what's happening. You can store JSON in SQL, but I mean, really we're talking about traditional SQL tables with a schema, keys, and then relationships between those tables.

Absolutely. And strict enforcement of all that data because it's too much data to have without, like I would argue to like never pursue a non, you know, a non-relational database in this effort.

What we talked about though, now, to your point, I think we were going into this, we talked about the data problem. We talked about this. What do we do with this now that we have it? How do we express policy with this? Because you can't sell this to your leadership in an organization without the– where's the rubber meet the road? And I think that's where– Once you have the data in a state, in a system where you can trust it, that's where automation starts to really come into play for expressing policy. And you can do that with something you build yourself. You can do that with off-the-shelf vendor tools. I've been in environments where we've done sort of a mixture of the both, where there are products out there by third-party vendors that will help you. set firewall policy, they'll look at all your firewalls, look at the cloud infrastructure and all that and be able to push policy to it. And what you can do is use this data. Perhaps you have to build something in the middle to. you know, create your policy from the data and push it to those systems, but you can essentially offload some of that heavy lifting to a third party system if your automation in your environment isn't that far advanced yet and things like that. There's a lot of options with what to do with the data once you get it. –

Well, okay, but once I've got it, it's living in my SQL database. And of course I've designed the schema very well and I've got my relationships just nailed down in between the tables in my database. Now I've got a structure that I can programmatically query. You can pull data out of a SQL database in a million ways. Python, if you wanted. Okay. I do that. And now I've got this structured data that I can use. And then I've got some abstraction, a library I'm using within Python to turn this SQL data and relationships into a firewall policy that I could expressly push into some specific device I've got on my network at Juniper SRX or, Some kind of a Cisco box. They've got 20 different firewalls now and so on

Yeah, absolutely. And what I'll say is that you have to translate from one model to another, right? So you have your standardized model. And if you think about having different adapters for different types of endpoints. And so, I mean, by that size endpoints are Palo Alto, firewalls or Kubernetes, whatever, or AWS, those become like the model translations you need to create. So I have some definition of an application. I have some definition of connectivity. but that means nothing to the actual configuration policy. So you need that model that gets created and used to generate configuration either via– Jinja-style pushing configurations like on a Cisco IOS ACL on just a router, you can detect that's a perfectly reasonable security enforcement choke point for many use cases. Or it might be an API call to a Palo Alto firewall or whatever the various different methods are. So you essentially need a model translation, which is that abstraction point you're talking about. You translate from the model created for your internal applications connectivity to translate to the configuration. in the specific adapter to the specific vendor implementation. –

I think the magic words here that one of you guys wrote in the notes here, build artifact. Build artifact. I gotta create a build artifact. John Anderson, I gotta go. –

I got credit John Anderson who and I work with, you know, principle of Network to Code. We were talking about, it wasn't exactly this, we were talking about it. And he just said, it just sounds like a build artifact you're talking about, like just nonchalantly. the perfect term for it. So it really is. It's just the, you're not managing the configuration. That's just the configuration. It's just the build artifact of what the model that you've built, you're building. The configuration is the boring part. It's the thing. It really is. It is.

We've got to express what's in the model, the relationships, the policy that's been agreed upon into a build artifact, but It's kind of like it's Palo or it's checkpoint or it's Fortinet or it's Cisco, whatever. We've got to get that policy in there so that it's enforced. Yeah, who cares how? Give me the right adapter. Here's the data. Go forth and make your build artifact and push it out. And then we can make sure it's– would you do that with a pipeline or what would your process be here?

Yeah, absolutely. I mean, it's going to be, you I think we all love to get to that pipeline, auto deploy feature or capability, I should say. It's very hard to get to, I think more realistic is, and just, sorry, quick plug, like I did a presentation at NANOG recently where I talked about pretty much what you said, which is configuration is the boring part. Don't automate that first anyway. And part of the reason is culturally, no one's going to jump into the pipeline, right? So crawl, walk, run here. and You just need to publish the configuration. In my opinion, you need to publish the configuration to the engineer to actually go deploy. That's mainly deploying at first. It really is the smallest amount of the effort. It's just applying the configuration.

That's so true.

The greatest trick I've ever pulled is that automation is about configuration management or primarily even. To me, it's- That's the last mile. But anyway, so I'll get up my soapbox for a second. Culturally, I just think it makes sense to build towards the auto deployment. Once you're so bored of just pasting in whatever the tool tells you to paste in, but you need to build up that confidence. And this is a journey. This is realistically one of the big holes in this strategy. is there is it's hard to get. actually get value out of it. This is very much a waterfall delivery model. Not completely, there's wins along the way, but there is a bit of a waterfall mentality here.

It's a, to your point, it's a journey and the organization, it takes organizational discipline and it takes organizational buy-in to do this. And in a lot of cases, I've only seen organizations really… take off a bite of this and decide that they're going to go down this road when they have a very cogent reason. They've had a security breach, they're coming under a new compliance regime of some kind. There's some external impetus that says to the IT organization, We need to fix this and we're committed to this multi-year journey to fixing how we manage our security policy. without that kind of external pressures. Usually you don't see it succeeding in an organization because there can be an engineering team that says, this is how we're going to shift our management of firewall policy. We're going to do these things, but without some sort of pressure or buy in from the highest levels in that organization. You don't see it go to fruition in the same way.

There's another way to sell this though, that I think maybe helps with that organizational buy-in. One of the challenges, we alluded to earlier in the show. Over time, these policies need to change and grow and, and, Hey, is anyone using firewall rule 142 anymore? I don't know. We haven't had any hits against it for a while, but et cetera, et cetera. How do you get that ownership established? And what I'm seeing here. When you go through this process to get this data model built, one of the bits of metadata you could assign to that is either a group or some human that owns this thing. And so when you do your periodic security audits, 'cause of course we're all doing those, aren't we? Of course we are. You can check in with that human and go, I'm imagining something here. You could generate a report that in a human way expresses what's in that database table set of relationships and go, is this still true? Do we need to change anything? Can we restrict this change it? What needs to be updated so that we're current?

And that's the kind of thing where the automation is key. And Ken's talking about pushing the config to the device is the boring part, the very last thing to automate. Before then, you need to automate that part of the policy process of here are the 10,000 firewall rules on this firewall. Here are the 3,000 different owners inside the business. go email them and wait for a response. If there's no response, email their boss. Those kinds of things where you have, not so email is necessarily the best way, but there's all kinds of ways to automate that, but that's where you put process, automation process to those policies of shifting that load out where people know that, oh, security review time is coming. It's no longer just the security engineer. you know, shaking his head and pounding it on the desk, that email or that effort goes out to everyone when every time that process comes around. –

And it comes back to organizational buy-in as well because you have to have some teeth in that review process, something like… This bit of metadata says you folks are responsible for this. give us some feedback that this is or is not needed. We're gonna assume it's not needed. We're removing this from the policy and these communications will be denied. If you're okay with that, you don't have to do anything. If you're not okay with that, you better let us know what's up.

Yeah, absolutely. I think you had a few things there. So I really think one of the things to understand is one of the goals to get towards is to in a journey of educating people at the end state, you ideally can remove a lot of education or it's not needed, I should say, fairway meaning. Really, application owners want to speak in terms of apps. They don't want to speak in terms of five tuples. This actually allows you to get to that and to express things in the way businesses speak, which is, I need access to LDAP, not I need 10.20, 20.20, 3.389, or whatever the LDAP for this I'm forgetting. That is fundamentally not the way the business works and that's not their goal is to speak in those terms. And so this is getting you towards that back to that original ask, which is, I talked to many application people, I said, I don't know, it's hard. I simply want to, I just want all that access. It's like, well, who owns that? Who's the owner of that information? And to the application folks, it's the security team because they're the ones they're interfacing. But it shares that ownership, it assigns ownership to people. And one of those things that in that lifecycle ownership is things to consider, things like who allows approval? LDAP is a wide open system. Everyone's going to need it. They don't want to be sitting there saying yes every time somebody comes across. But other applications are going to be very much controlled, and they want approval. They want you to sit there and come to them. and they want to review. And so having this data allows you to actually build in those processes based on all this metadata that we're talking about. –

And one of the things we're sort of dancing around in and where I've seen the successor failure of some of these kinds of projects happen is… folks with a very well-defined and mature ITSM solution in place, if they're using ServiceNow or whatever it may be, where they are already using the policy and process enforcement mechanisms of that tool, this is just another thing that gets added in where, you know, in addition to approving, say, group membership or this change ticket, there's an approval flow of, I'm approving this firewall rule to continue to work or whatever, where… You know, that's very helpful where you hook into any existing ITSM approval flows like that. So you're not having to reinvent that wheel as well.

Yeah, in addition, that's, you know, like, I don't want to make it seem like this is just so pie in the sky. Like, you know, one of the points I want to get to is it is definitely a crawl walk run thing to get to this point. And I think one of the most obvious points is to Just to see if you stay with the five tuple kind of methodology and actually deploy your configurations via automation or generate your configurations via automation and then push them. That really goes a long way in solving a lot of challenges, which is standardized configuration. One of the huge benefits for any automation is that standardized configurations. And you can't parse something, you can't reasonably parse something. if the configurations aren't standardized. And so simple things like Brett uses dashes and I use underscores, that is going to break your parsing or you're going to make it very fragile process to kind of understand how people would come in. And so introducing the path I tend to talk to customers with is, this is the future, but you need to get your… your basic configuration in shape and ready to be prepared to actually make this a reasonable venture in which you could go and say within three to six months, I could actually do the basic things via an application dictionary. But the problem is where it becomes so difficult is when your configuration, you have to– you know, deconstruct your configuration first, just to normalize your configuration to actually make it automatable.

You just said you're saying so much right there. This is such a big chunk. I'm like, I'm sighing and shaking my head and so on thinking about this part. We've all been through it, right? The big like. Let's say you're a new and new to an environment and in your, you're an engineer walking into this environment. I've been there several times and you, you, okay, time to look at the, the firewalls and it's a train wreck of rules and, and different ways they were expressed and the order doesn't make a lot of sense. And a lot of them have no hits and trying to get it to a point where you can actually do something with it and it's predictable and you trusted and so on. That's Huge to do that normalization.

Absolutely. It's a foundational block. A term I often use is stop the bleeding, which is like, from here on out, no more bad rules. Unfortunately, some poor soul has to go back and decompose 50,000 rules out there and say, are they valid or not? You have to make some reasonable guesses and you have to try to find some owners. If you don't stop the bleeding, if you don't build that auditing process for the configuration itself, That's really where the guts of, or the bulk of the time to actually make something like an application dictionary useful. Right, I mean, I think you can get in a… Let's say post once you actually build your data model, you just kind of UML diagrams and so forth. from that point to actually getting some values, probably closer about six months. But if you have to go and do the data at the same time, that's 18 months. And it's hard to show that ROI. And so often the strategy we kind of talk with our customers are, is stop the bleeding, let's just get the ITSM to automation workflow worked out. We know the configurations are normalized and you learn a lot along that process. Meaning you learn where all the edge use cases are. This is our policy, this is our SAM policy, full stop until you get to the fifth rule. And then they go, oh yeah, I forgot to tell you about that. Oh, you forgot to tell me about that, right? And so there's a lot, and then there is a clear ROI there, meaning all the reporting you could build with just the idea of the five tuple, all of the analysis that you can do is sit there and say, what's definition of a good rule? And every organization I talk to says the same thing. We want to make sure it's a good rule. And like, what does that mean to you? It's like, well, you know, I'm like, well, I know what I would say is, but it's not gonna be what you're going to say. So why don't you go ahead and tell me? And then it's always different. It's always different, right? Which is PCI, non QA, non QA, et cetera, et cetera.

You're hitting on a problem that's common to automation broadly, which is standardization. When, when it's a lack of standardization, you got all these exceptions and weird little corner cases that you're, you've got buried in your configurations trying to standardize it as a nightmare. You've got to have that you've got to be standardized.

I don't want to say automation is impossible, but you've just made your job so much more complex. And I think and what people tend to forget when they think about the stance looking at this, you look at this Wild West config that has had, you know, keyboard cowboys typing things in in every manner for 10, 15 years. you don't have to tackle all that at once either. And that's where you can start to automate, you look at what's your… what element of the configuration, like, oh yeah, we'll start with all these rules that have rule comments in them with underscores. We're going to change all those to dashes. And you start to take bytes out of those kind of things. And that's where you can apply– automation to help you do that because if it was the olden times even if somebody hand jamming through line 101 this and that no line 102 this and that and you know you don't have to do that by hand I think a lot of people think that They can't use the automation until it's standardized. You can use automation to get yourself to a more standard state. –

Sure, sure. –

It goes back to the point of like configuration management is the least fun or at least whatever, you know, important like different reporting to make the configuration is 10 times more. So actually just going through with a read-only counts, doing all your analysis and then two months later, one time deploying that configuration. You're going to update that report dozens of times in between. And that's really what the automation and the programmatic access is crucial.

Okay, we've gone through a lot of material here, gentlemen. There's one thing that we didn't touch on going back to the data model that I do wanna get a sense of. When we talked about building out the application profile at a high level and like the listener ports and the relationships and there's some IPs in there, what about policies that don't care about those things that were, you know, IPs and poor specifically, because they're filtering maybe up at layer seven. Does that somehow become part of our data model too?

Absolutely. It's a crucial part and it becomes like, you know, it could be, you have like this idea of required one of and those attributes could be being like an external dynamic list, an NSX tag ID or tag. an IP, a container service discovery, that's the crux of part of the solution. Right, which is. Removing that as a required field is fundamentally one of those things that you have to get to. So an IP address is probably, for suitable features, always going to be part of one of those points. one of those service definitions that more abstract, not talking any specific product, but more abstract service definitions. So like a widget SQL, as I mentioned before. But the important part is that you have the service definition and that's just a specific attribute or a specific type on it. Okay. –

Yeah, and in the model, you can have layer seven attributes that you really care about. So for example, worked with a client in the past that they've managed a lot of WordPress installs for people and they wanted no IPs, but their IPs to be accessible to the /wpadmin URI. And so that's a policy you can enforce with some vendors security products that had to be modeled into how we were going to model the policy was understanding layer seven URI elements so that we could express firewall policy that way. I mean, when you start to talk about what defines an application, you can get very precise in layer seven information like that. And you need to account for that in the model.

It's just one of the things too is that one point to hit on here is I really feel like the- The firewall, a solution like this is very personal. The only thing that's going to come to the market, quote unquote, air quotes there, is something that allows tremendous amount of flexibility in that data model. Because people, organizations just look at the way their apps are modeled or distributed and built very differently. And so, the RQ, like different lanes and environments, it's all very different. So I really think organizations, I talked to so many customers that say, why isn't this out there ready? And it's like, because how do you want it? Like you want it this way? The 12 other organizations had nothing close to that. What so it wasn't even similar at all. And everyone's, what most customers are saying, or most organizations are saying is they'll realize it is. I want a product that acts the way we operate. not understanding that there's additional products not gonna work like that, and what you need to kind of move towards is things that are frameworky. Right? And so things that are framework or Ansible is very framework. You get nothing out of the box. ServiceNow is very frameworky. It has, you know, Terraform. These things are not giving you, you know, ServiceNow gives you some things, but really the crux of what ServiceNow is like, you can do custom forms and you can do custom CMDB and all this other fun stuff. There is a point of commonality to all of the data models, but the deeper you get into how an organization operates, what their application needs, what their concerns are, it becomes somewhat bespoke. the deeper you get.

Which really leads me to another question. I kind of thought this was the answer I was going to get from you two about the data model as it gets more complex, which begs the question, who builds the data model? Is it me, the security engineer? Is it a developer? Is it a database administrator? Who does that?

It's maybe a very biased answer. But I fundamentally– in my self-serving opinion, you need someone that bridges the gaps between those worlds. Right, which is they need context of how firewalls have been managed. and how programs, how to build models and how to build programmatic access and what those requirements are. Now the actual builder is going to be like the actual hands to keyboard, like a programmer, right? Somebody that knows, you know, Django flask, whatever your kind of. choice of tool is there, but the to build the UML diagrams, you need all sides of that and you need it. And fundamentally what I've seen, I think we may have talked about this, Ethan, on our historically long podcast we did several years ago, which is The idea that if you ask a, if you just put like a network or a security engineer and hook them up with a programmer, they tend to do whatever the network engineer does, which is basically say, I want to do what I do faster. They don't know how to reimagine the world. They sit there and say, how do I do it differently? Right. And so that looks like is, you know what I need? Basically it looks like it's just like a PuTTY session with a thousand tabs on it that could, you know, is convincing them to it, they don't sit there and take a step back because they're just taking direct requirements from the customer. You need someone who dabbles in both worlds and can translate that. In my humble opinion, again, very self-serving. –

Yeah, it's absolutely a collaborative effort. The initial building of the data model has to be a combination effort between the elements of the organization there where– And maybe there isn't one person who really understands all of that yet in a given organization. You need to groom or build that person or you need to work together to say, I have my application architect, I have my security architect, I'm locking them in a room for the next two weeks and they need to build some models for this environment, whatever that may look like. –

You need to have a system in place here because I think this model is going to evolve too as you discover new things, the organization changes, you bring a new application in-house that has different requirements. It's got to be… process whereby you're working with someone who can keep up with you over time. I don't know that that means a dedicated dev and certainly there is going to be a collaborative effort. You've got stakeholders, you've got people that are really good with databases, you've got people that are good at development, and you've got security engineers that understand what they need at the end, that build artifact that's got to get created and so on. So there's a lot to it.

I could imagine doing this without at least a dedicated person, probably multiple to manage and maintain this. But the reality is that I look at organizations and I talked to many, you know, customers or potential customers where they're saying how many rules a month. 300, 500, 600 rules a month are deploying. And it is, they have. teams, I mean, 20, 30, like, I'm 23 people out there. So The ROI there is clear and tangible.

If it's there's a couple of payoffs, it's you get. a predictable standardized security model that you can audit in a consistent way. You get reporting out of this. You have– a nice spread of ownership, you know, amongst appropriate groups of people, not like everything on the security person, the security engineer to make it go. It's now spread out amongst stakeholders across the organization. And you have a standard build artifact that you're creating that should look same, same as you're going through the policy and auditing.

quick get into it, I built something like this for a smaller, just like a single data center, but it needed to be ISO 27 that I can't remember what certification was required. And I actually assigned owners to every rule. The design wasn't great, but it worked. And I emailed people every three months and said, "Do you own this rule?" And they would promptly not respond back. And I would… You know, just say, okay, I'm going to disable the rule. And then they say, Oh, hold on. I moved groups and you know, we'd eventually get to it. But eventually the auditor came and they're like, they want to speak to you. Okay. No problem. And I gave them a spreadsheet of every rule and who owns it. And the conversation basically went down like. How'd you get this information? And so I– whipped out my Perl code that I developed eight, nine, 10 years ago, whatever it was. And so here's this and they're like. Wow. I'm like, wait, well understand how else was I supposed to do this? And they're like, we don't know. And then they're like, I'm like, well, will you ask other organizations? So like, they just tell us the process and we just trust that the process works. Now I'm like, that was an option. That was, it was mind boggling to me. That was an option. And I just spent like, you know, two months building out this entire thing just to manage this, this one data center. So many audits that you should have failed to just get swept under the rug because we just trust the process. It's what we trust the process, but, but actually be able to go and sit here and say, I mean, they, they were like, you know, And they literally got like three, four or five people on the phone just to hear me understand this because they had not seen anything like this apparently. And, but to go and sit there and say, you know, when you're all that becomes report, that's it. It's just like it's your audit is available at all times. And what that means for your organization can be invaluable as well. There's a lot of other, you know, less tangible ROI, but that that ability to do an actual audit is probably a man year or more to actually do it if you're required to. And once you have this system, which again, in fairness, it's a long journey, it's just a button away.

Yeah, well gentlemen, this has been a fantastic conversation. This is, I like conversations like this, it's mind expanding in that it challenges you to think about not just how to do something, but just how to do it. differently rethink your culture rethink how you interact with other people in the organization not just be the security engineer firewall jockey martyr that's like these rules are mine I live under their burden and I shall keep up with them because I'm gonna do the right thing no it's not actually how this is all supposed to work you may need to put other systems in place to make it work well but there are other ways to do it and we just talk through at some length a really good way to get it done not easy to make happen overnight it's not going to be It's a journey. We're always on a journey, guys. I don't know why we're always on a journey, but we're, it is a journey. It's going to take some time to get there. So again, fantastic conversation. Ken, starting with you, how do people follow you on the internet?

Primarily, on my Twitter, itdependsnet. My GitHub is itdependsnetworks and probably the place I'm most seen is on our Network to Code Slack, a large community, fellow automation. your types of engineers, et cetera, case lens on there.

Fantastic. Brett, same question to you.

Yeah, I'm on Twitter at lykinsbd and also on GitHub lykinsbd and then same as Ken. I'm pretty active on our Network to Code Slack instance app, Brett. And then blog off and on at the Network to Code blog. Same as same as Ken there. So it's a great place to check out. All kinds of things like this where we like to pontificate about the ways and means of doing things just a little bit differently.

Again, thank you for your time. Both of you, again, this was your fault really anyway. So it's, you know, but I do appreciate the time that both of you made to chat with us on Heavy Networking today and hope that you out there listening got a lot of value from this. It expanded your mind. No doubt you didn't get it all in one shot and you know, listen to it again and send it to your friends and send it to your boss and spread the love of packet pushers, Heavy Networking to all the network nerds.