Originally recorded in September of 2021...today’s guest is Justin Berman, the Vice President of Infrastructure and IT and the CISO at Thirty Madison. Thirty Madison is aiming to be a platform that everyone can use to deal with their chronic healthcare needs. Justin’s main focus is on building out the teams that enable scaling. With his development background, Justin has some unique ideas when it comes to cloud security, which makes for a fascinating interview. You’ll walk away from this episode with a new perspective on how to build security into products from the start and a better understanding of how to transition smoothly from on-prem to the cloud.
“I see security as an engineering problem. What I mean by that is not that there aren't things that you solve with process, or with policy, or training, but rather that in as many places as possible if you want to have a scaled effect within security, you need to write code to solve a problem.” — @justinmberman [0:06:03]
Justin Berman on LinkedIn
**NOTE: Generated via ML. Expect crazy stuff to be translated by an imperfect algorithm that may have never actually been said by the host or guest :-) ***
[00:00:25] MC: Amazingly, this is our 15th episode. When I started the podcast well over a year ago, I didn't know how long I would do this for. I can tell you that if you've ever thought about starting your own podcast, I would really encourage you to just do it. I knew nothing at all about podcasting. To be honest with you, it's not that hard. It is a time investment. If you've ever thought about it, go out and do it.
All right, so today's guest, I'm really excited. Special guest. To be honest, I think all our guests are special. It's Justin Berman. Justin is the Vice President of Infrastructure and IT and the Chief Information Security Officer at Thirty Madison. I wanted to have Justin on, because this is an interesting story here how I first got connected to Justin. I was actually applying for a role at a company he had just left. It was a Chief Information Security Officer Role.
I followed him after that, and I'm just intrigued by his career path. He has spent so much time around startups that with Justin, I didn't necessarily want to dig in specifically around cloud security, but rather, how you build security into platforms and products from day one.
The other secret that I'll let you in on is that, as you can imagine, I record a lot of these podcasts in advance. This episode is no different. I originally recorded this episode with Justin back in September of 21, so if there's any date references that seem off, you'll know why. I hope you enjoy the episode.
[00:01:57] MC: Today's guest on the program is Justin Berman. We are really excited to have him on. He's currently at Thirty Madison. Justin, thank you so much for joining us.
[00:02:08] JB: I'm delighted to be here, Matthew.
[00:02:10] MC: Tell us a little bit about your current role at Thirty Madison. First of all, what does Thirty Madison do? I know that you're hiring, so I guess, it's actually a couple of questions. What's your role? What you're focused on? What roles are you hiring for? Fourthly, what does Thirty Madison do?
[00:02:27] JB: I'll start with the last one first. Thirty Madison is trying to be the place in the world that everybody goes to deal with and manage their chronic care, or their chronic health care needs. We're not trying to compete as your primary care physician right now. We're really trying to help people that have lifetime health issues manage those health issues in a way that feels as good as it feels to buy or work with an e-commerce company.
E-commerce has done a lot of work to try and make their user experience absolutely seamless and beautiful. We want to hold that bar for user experience while providing authentic solutions to people's chronic health care issues. My role at Thirty Madison, so technically, my title is CISO, Chief Information Security Officer. I think the more accurate title would probably be VP of Security and Infrastructure. My actual role is I'm building out most of the teams that enable scaling. Infrastructure, DevOps, and SRE report to me, or what we call DevOps and SRE, call it cloud infrastructure reports to me. Security reports to me. IT reports to me. Privacy and compliance all report to me.
You asked what roles am I hiring for? The two that stick out the most in my mind right now is I'm looking for very senior product security engineers and I'm looking for staff level site reliability engineers for my infrastructure team. A whole hire. I mean, I'm happy to find a principal level engineer, but I think the bar we're looking for is staff. Does that answer all your questions? I think I answered all your questions.
[00:03:59] MC: You did. You went right through those pretty well. No, that's great. I love to ask the question about hiring, because I know that is something that seems to be perpetual in the security industry. Listening to the list of teams that report to you as the CISO, that's pretty different than many organizations. Give me a sense for how large is Thirty Madison? What are we looking at in terms of employees?
[00:04:23] JB: Oh, our current total staff is, I think, we just crossed the 290 mark. I would have to go look at our actual HR system to tell you the exact number. Then within technology, we're a bit over a 130. Then my current teams, hiring aggressively for them, but my current teams between all of them are in the 16, 17 range right now.
[00:04:51] MC: I think, I was looking at your background and I know that way back when, you were an intern at the University of Maryland and you were doing software development. Looking at your career, it looks like you've pretty consistently been at companies that have software as a product. I’m always intrigued when I see people who are in security who really grew up through a development background. That's really unique. I've said this in other podcasts, in other places before, but you typically see a lot of people in security who are from a network security background, audit compliance. How did that experience shape your then future career and security? I’m going to ask that, just because it's different than what you see in many that are at the CISO level.
[00:05:34] JB: Yeah, I appreciate that. I mean, in a very practical sense, it meant that I started my career more from a app sec red team side. The first real security job I had was consultative at Aspect Security, before they got bought by EY. I cut my teeth on finding bugs in custom code. If I reflect, I think it's caused me to have two philosophical stances that I know not all people in security share. One of which is that I think largely, I see security as an engineering problem. What I mean by that is not – that there aren't things that you solve with process, or with policy, or training, but rather that in as many places as possible if you want to have a scaled effect within security, you need to write code to solve a problem, such that it's impossible for someone to even make a mistake, rather than trying to focus on how do I teach people to make fewer mistakes, or how do I build processes that are more resilient to mistakes.
It also, I think, really benefits me, as you said, I've worked for a lot of companies whose products are software oriented. Those companies put a very high focus on engineering productivity. Coming from a software engineering background and software engineering work, I empathize deeply with that. Having been a software engineer, I know that that job is also challenging and also takes a lot of effort and energy. I think, I see app sec teams, or product security teams that too readily, will slow down the engineering community, not really realizing the impact that has.
I think I'm lucky to have the software engineering background, both because it gives me a detailed understanding, but it really gives me empathy for the experience that engineers have and why security can be a challenging thing for them to integrate into lifecycle.
[00:07:21] MC: Feel free not to answer this question if it's giving up too much, but I assume at Thirty Madison, you guys are a cloud company, right, that everything's in one of the big CSPs, or one or more of the big CSPs. Everything you're talking about, everything that you're doing is, I'm going to assume, for the most part, cloud-native. Is that right?
[00:07:40] JB: Yeah. I mean, I would say, I will gladly answer yes, because frankly, I think it is way easier for small companies to buy a bunch of OpSec, operational security, by using a cloud security company, rather than trying to run our own data centers. There's also all kinds of cool security benefits with software-defined infrastructure that I think because we're cloud-native, we didn't have the challenge of becoming a digital company. We were a digital company from the beginning. It means we can build practices in from the ground up that take advantage of the nature of rapidly creating and destroying infrastructure with repeatable configuration that just take other companies longer periods of time to get to the point where they're comfortable with. I actually think it offers us a ton of security benefits that far outstrip any security issues that are created, or different security issues that are created within the cloud side.
[00:08:32] MC: I'm curious, how much of that advantage – Personally, I spent most of my career in the Fortune 500 space. Really large companies. Tons of tech debt. Just decades of tech debt, which makes the transition to cloud difficult. Especially if you're looking at financial services. I mean, M&A after M&A after M&A. They're dealing with systems that sometimes haven't been updated in 30-plus years, sometimes. I'm curious from your perspective, obviously Thirty Madison’s on a different scope and scale of that. You said you've got about 300 employees. At the same time, you have still a very scaled, very large scaled platform. From your perspective, I think a lot of the listeners to this podcast are more than likely, I'm guessing, I don't know all the listeners, I wish it did, but I think most of them are probably from the more Fortune 500 space.
They're looking at and they're like, “Man, I would love to be where Thirty Madison is. I'd love to be all cloud-native. I'd love to have nothing in these seven colo facilities that still have a lot of mainframes that are, quite frankly, running things.” I guess, the question for you is, if you were to give some advice, like, “Here are one or two things that you can look at doing as you transition to the cloud,” that can maybe set them on a path of thinking about security in the way that you do, which is you build it, right? It’s policy, it’s code, it's security, it's code. Where might you recommend they start? This is going to take time, but where might be some areas you might recommend they start?
[00:10:07] JB: I mean, a generic, I think on purpose, very generic recommendation – Actually, before I say this, just like, I super empathize. When I was both at Bridgewater from a financial services perspective, I still had, we made, or rather, I was one of the security architects that designed their approach to digital back in 2013. AWS much less sophisticated as a platform at the time, but still really important.
Then at Dropbox, we had a hybrid of using both cloud and on prem. People used to ask me, “Dropbox was born in the cloud?” I'm like, “No, Dropbox created a cloud.” Storing that much user data at the time was not a cost-effective strategy in cloud. Obviously, Amazon continues to be brilliant at dropping the cost of things like S3, Glacier, etc. For Dropbox, creating their own data centers, creating their own systems. In fact, all the way down to thinking about custom firmware and whatnot is really critical. I empathize, is my point.
When I think about the transition process to cloud, I think the first piece of advice I usually think about is like, you are starting from a green field. Treat it that way. Do not simply port things. I mean, it feels generic advice to me, because it's not like, “Oh, make sure you do this one configuration thing, or set yourself up with this particular set of policies.” It's not like that, but it is instead of succumbing to the temptation of simply moving things in place over, which you will have to do sometimes. I think the reality for every company that does a digital transformation is they can't afford to re-architect and rebuild everything from the ground up.
I would start from a place of, we should set the right security posture in place, and then by exception, do something different. That whole attitude of approaching that project, which for most Fortune 500 companies is many, many, many years and hundreds, at least tens of, if not hundreds of millions of dollars a project. It’s like, that is a directive at the top of that project that happens, which is by policy, we are going to architect it from the ground up in a particular way. Security is a part of that. There are, by the way, reliable things that you can do that make it more effective.
There's all kinds of not security and privacy stuff you can do that make the cloud journey better, and knowing and intentionally making the choice to say, “It will take an exception to build things outside of this architecture,” puts a sometimes gentle, sometimes severe forcing function on how you move things over, so that you don't just slip into being like, “Oh, well. Let's just move it over and then we'll figure it out later.” Because that just persists the debt and can make it worse. Honestly, if I was going to give one piece of advice, it's that. It's starts with that directly. Get in agreement at that top side, that that is the thing.
Then as a follow on in short-form, if you can't do that, if that's not the thing, then the lot of the biggest mistakes that I think really bite companies as they move from an on-prem to a cloud is that people don't realize how swiftly you can create a world in which all the, let's say, layers of security you’re used to having in a data center, where you might have an external firewall routing layer below that, switching layers in between, or further routing layers in between and additional firewalls in between like, “Oh, yeah. We have a trust zone for web and a trust zone for apps and a trust zone for databases.”
That's like, you can set that up. You can mirror that. That thing, it's so – one small configuration change can also give your database a public IP with a /0 firewall rule. One mistake can take an S3 bucket and allow public lists and get on all files within that bucket. It's not hard to make those mistakes. Actually, luckily, I think it's not hard to catch them either.
I know lots of different companies have good cloud security posture management tools, so I won't try to bang through particular solutions, so much as just saying like, it's not so much that it's impossible to solve these problems, but that the assumptions we make no longer hold effectively. It is very easy and in fact, more default behavior that things are directly touching the Internet, or touchable by the Internet, as opposed to what we used to have which is you have to be explicit about creating a route that actually gets you all the way from Internet to a database. That requires changes in many layers that a lot of people could intercept and catch.
Yeah. Anyway, those two – and they're really, I guess, ways of thinking, to me. Although, the first one really is like, get in sync in your project, in your digital transformation projects or program, get in sync at the beginning. “We're going to do this. We're going to do this right. Then by exception, we're going to do this in a way where we have to create a bunch of compensating controls for a particular thing,” as opposed to, “Let's just move it over and figure out how to deal with it,” which I've also seen companies do.
[00:15:14] MC: I think the point you made there is very valid, and that I've seen threat research that came out, I think it was probably a year, a year and a half ago that said that 65% of cloud security incidents were the result of customer misconfigurations. Like you said, it's just, the cloud has all these great things that you can do with it. It's scalability, agility, etc., etc. That also, like you said, also goes the opposite way, which is you can also make poor misconfigurations, or you can make misconfigurations just as easily as well.
I guess, my question for you would be, there has been public examples of both really large companies that have made those kinds of mistakes and also, smaller companies that are, let's say, more cloud-native, is it just simply because of the nature of cloud? These things are API-driven. Because of the fact that like you mentioned, perhaps there's not all those layers of governance that used to exist, that is leading to that? Or do you think it's – I don't know. Do you think it's more of a skills mismatch, or is it a combination?
[00:16:14] JB: I like that question. I think there's a knowledge mismatch because I think that a lot of the sysadmins who did great work in traditional on-prem are actually totally capable of doing the skills that they have applied, but there's a whole bunch of new knowledge in order to understand how you implement, say, a similar control, or how you rethink about a control, given the different context the cloud has.
I think there is a knowledge mismatch where you want to upskill your existing people, but there is a period of time in which they don't yet have the cloud knowledge. Frankly though, unlike consultants in certain cases, rarely do I think you can have a consultant design a strategy for you and in a tech context and have that be super effective. Hot take, I guess.
I also think that part of that stems from the fact that the complexity of infrastructure is code, or rather, I should just say, the complexity of configuration-driven, software-defined infrastructure is not low. I think this is true even in platform as a service and software as a service things. Because I'll tell you, to pick on them not because I think they're doing a bad job, I think they're going out of their way to do a good job, but it's still hard. Salesforce is somewhere between that platform and software as a service offering, depending on the thing. There are so many ways just to configure identity and authorizations within Salesforce that have non-obvious, overlapping implications.
I think the same thing can be true within say, IAM, policy within AWS, or their competitors, Azure and GCP. Some of the problem is raw knowledge, just like, hey, understanding how cloud behaves, how you expect it to behave. Some of it is frankly that the level of choice that you have, the flexibility offered in order to make these things usable by everyone, even when you're not talking about infrastructure as a service, even when you're talking about these more restrictive systems, like platform and software as a service is just so high. The potential for mistakes is higher and the velocity with which a mistake propagates is very high.
That to me is reflective on why you can see mistakes that are avoidable in cloud create issues for a lot of companies. Especially if I'm thinking about the point you're making before in the digital transformation side. I think for cloud-native companies, there's a whole different set of challenges, which is they often are at the bleeding edge. They're trying to move with incredible velocity. Infrastructure is a supporting function that is likewise trying to move with great velocity to empower the engineering community at the company. Often, there is a bias in cloud native companies to allowing your, say, application engineers to also affect cloud, the infrastructure itself.
In that case, you have someone – I actually think that comes back to the knowledge thing, although it's from a different angle. When you have a software engineer who's spent their life building frontend code and you tell them like, “Okay, great. Go set up whatever EC2 instances you need for your frontend app.” That's not really fair. They might have that experience, but chances are, they don't have all the experiences they need. At Thirty Madison, I see that as a responsibility for a team to provide, or productize the infrastructure for the engineer, so that they don't have to think about it. It's not about, I don't want to allow them to ever care about it. It's about making their jobs easier.
When I come back to, where do cloud-native companies make mistakes? It often comes from the fact that I think they're expecting people who don't really have that much understanding or experience to make these decisions rapid-fire as they roll out products. That trickles back into, if you ask someone to make a decision with insufficient knowledge, they're less likely to make the right decision.
[00:19:55] MC: No, I love that.
[00:19:56] JB: I could belabor that point for probably a long time.
[00:19:59] MC: No. I think that's some really great insights that I've not heard before. You touch on another area, which is just from what I have seen, and I'd love to get your thoughts on this, DevOps teams, historically speaking, are really focused on moving fast, breaking things. Security traditionally has really felt the weight of trying to then manage that resulting risk.
I think you're in a unique position and it sounds like you've got both of those teams reporting to you. How do you bring those two together? What have you found that really works from a security and risk perspective, in terms of allowing them to move fast, hit their spring goals, but at the same time, not allowing the risk to be beyond what, I guess, is acceptable to the business?
[00:20:47] JB: One quick clarification, our application engineers do not report through me, but our infrastructure teams do. When I think about the full DevOps ideology as expressed in things like the Phoenix Project and whatnot, they're still – I don't think we have a fully DevOps model where every developer is a full-ops engineer and whatnot. We still are trying to offer infrastructure as a product internally within the organization.
Splitting the different parts of the problem, I actually think my infra teams want to move fast but are relatively security cautious in the way that they behave. They would rather take a little more time, get more advice from the security team, work in more partnership. Frankly, in terms of the culture at Thirty Madison, because we're dealing with health information, my engineers, in general, want to move with – they want security’s advice. I have more trouble keeping up with the requests than I do trying to get people to pay attention to us. That's a luxury that I know not every security team has.
When I think about enabling product development across the entire SDLC to move quickly and effectively, and in fact, security first philosophy and then specific ideas here about tactics. Security’s job is not primarily – If you measure security only on number of vulns shipped, or number of vulns that we find in prod, which I think is a useful metric, but if you measure it primarily there, then you miss half the equation. Security's job is to establish a bar for what's acceptable risk with – and it's my job to work with leadership to help normalize that bar tolerance, all that stuff.
Then, security's job is to make it easy to hit that bar. Not drop the bar, but make it easy to hit the bar we say is secure enough. If we reframe security’s job that way, then we start to think about, “Well, what can I do at scale across engineering?” Instead of asking the question like, “Okay. Well, how do I find all the bugs and make sure that the engineers get all the bugs so that they go back and fix all these bugs and all the security debt?” Which is also a necessary set of vulnerability management practices. You're going like, “Hey, we have all these –” This is not an issue at Thirty Madison, but I'm going to pick this class because it's easy to talk about and everybody seems to know what SQL injection is.
If you have a pervasive SQLI, you can go fix all those bugs, right? You can go ad hoc, fix one by one, whatever. If the security team stops there, then they've failed the DevOps cycle. What they should be doing is going like, “Hey, we fixed all these bugs. But holy shit. Why do we introduce so many of these and do we need to overload the functions associated with our SQL? Do we need to set up linters?”
Well, actually, I'll say it. If you think about crawl, walk, run. Crawl is, I find the bugs maybe in prod. I hand them to developers, they fix them. Walk is I find the bugs during dev, during commits and pulls for code and I hand them to the developers there. Run is, I wrote a library, I overloaded a function, I built a service that prevents developers from ever having to worry about SQL injection. Again, because it makes it impossible, as long as they use the service library function, etc. Then by the way, you also make the rest of your job about finding the problems easier, because you can start asking the question, did they use the service we built? Instead of asking the question, is this particular chunk of SQL vulnerable?
When I think about tying back to the empathy for the developer, I want them to have to think less about the security world, writ-large, and more about the unique problems within their product. Where can security build things that abstract away problems is to me, the way you at-scale solve in a certain company size, right? If you're a 30-business unit, 10, 15, 17, or 10, 15, 30,000-person shop, that may not work. Although, I know there's some tech companies that really have done this. Good examples in my mind, both Uber and Slack have really invested very deeply into what I'll call security software engineering.
There's a lot of tech companies, or tech-forward companies that have made the decision to invest in software engineering as a part of the security team, or software engineering as a set of solutions that security, say, like tech product manages. That is how you actually shift from, “Oh, my God. The development teams make so much change and I can't keep up with it,” into “Yeah, eliminating whole classes of issues, because we're actually building a solution that prevents the possibility of introduction of those issues.” I can sort very easily on, when are you not using our solutions, when are you using our solutions?
Also, there's a lower false-positive rate to that answer, which has a massive impact on the work lives of your app sec and product security engineers. Because if all they're doing is spending all their time going like, “Oh, man. I got to look at another line of SQL and figure out this particular one.” That's a lot longer of a threshold than them being able – Because they don't want to ship a false positive to an engineer and have that engineer turn around and be like, “What is this bullshit?” Excuse my language, I guess for your podcast.
If you can authoritatively say, “Hey, you didn't use the function,” that's a 01. Either they did or they didn't. Then they might have a good why, but that's a very easy thing for you to sort through and push to engineering as being like, “Hey, we set the standard. We solved these problems for you. Why didn't you want to use our solution? We might need to evolve our solution. We might need to do something different.” But your job as an app sec engineer becomes, you get rid of the bottom classes of problems and get to focus on what's really interesting, I think, in product security, which is like, “Hey, where's their unique stuff in our products that doesn't exist everywhere else?”
I don't want my app sec engineer to focus on what are the XSS issues. Because frankly, that should be solvable. I'd much rather them go like, “Oh, what are the fraud issues? What are the unanticipated consequences of our users trying to abuse our products? What are we doing to prevent account takeover?” The stuff that is specific to your business in both risk and solution.
[00:26:45] MC: I love that.
[00:26:50] MC: Prisma Cloud secures infrastructure, applications, data and entitlements across the world's largest clouds all from a single, unified solution. With a combination of cloud service provider APIs and a unified agent framework, users gain unmatched visibility and protection. Prisma Cloud also integrates with any continuous integration and continuous delivery workflow to secure cloud infrastructure and applications early in development. You can scan infrastructure as code templates, container images, serverless functions and more while gaining powerful full-stack runtime protection. This is unified security for DevOps and security teams. To find out more, go to prismacloud.io.
[00:27:40] MC: You've touched on a couple of things there. I think, if we go back to when we were talking about many organizations not being born in the cloud are now in this process of digital transformation, where they're looking at their portfolio of applications, whether it's 10, or a 1,000, and they're saying, “Hey, some portion of these, they're going to go to the cloud and we're going to completely refactor these apps. They're going to be made cloud-native.”
I think that presents a unique opportunity for those, let's say, larger, more mature companies to really start to build security into the actual software that's being built, the app that's being built. You've done this, I think, in a number of companies as they've built and scaled their platforms. Certainly, you're doing it today at Thirty Madison. I think you've touched on some things that I think can really help to make sure that it's actually built in. Because what we've seen historically in the past was that organizations would have to take some kind of legacy application and they'd have to bolt on all these different security tools, like WAFs, etc., all over the place.
What have you seen, what have you done in the past that has worked well in terms of building security into the actual application? This is not how, in the traditional enterprise, they usually didn't think that way, right? You had the app and maybe you ran some static code analysis. If you got really crazy, you did dynamic analysis, right? That's the beginning and the end. What have you seen worked well in terms of like, if someone now has this opportunity, they're going to be building whatever N number of apps, they’re refactoring them to be cloud-native? What are some things that maybe they should be looking at to try to build it actually into the actual application?
[00:29:22] JB: Yeah. I think, if we're talking about actual refactors, and so we're scoping out legacy apps that are never going to get revisited from a software engineer perspective, because I think in those cases, you really are trying to ask a question like, “How can I put the right walls around this.” If I need to bring this into cloud and expose it in new ways and I'm not going to really do much dev, then you really do have to think about like, “Okay, if I can't touch the thing, what do I touch around the thing?”
I think there, you're thinking about, I think about things, like identity-aware proxying, or zero trust stuff. I do think about labs. It's like, limit who can touch a thing more. I think incident response becomes a critical part of your security posture around applications that are like that legacy side. Not that it isn't for the rest, just that you start to rely on it more, because you believe that you can't make as many changes, so you at least want to be able to respond rapidly when bad things start to happen.
If I'm moving an application, I guess it depends on the degree of debt and the degree of re-architecture the company has an appetite for. Because if it's like a, “Hey, we're going to move it to the cloud, but we're really going to try and keep app code the same, or as much as possible, keep it the same,” then I think you're looking for the gates. I don't mean security gates. I mean, access gates. Are the assumptions you were previously able to make about authentication a thing that's going to hold? Are the assumptions about who can touch this thing – I don't just mean who can post-auth touch it, but pre-auth, does it require some network restriction to actually access the application, if you think about internal applications like, I have seen companies move an admin app to public web, sometimes for good customer service oriented reasons, where they want to have more flexibility about where their CX agents live.
God knows in pandemic world, we have had a whole different lift with regards to suddenly, companies have tens of thousands of people on VPN, when it used to be hundreds of people a day. There's if low appetite, then it's like, I would focus on attack surface reduction. I don't mean that on just every vulnerability’s attack surface. I actually mean just like, how can you put barriers in between the adversary and the first touch of the application where they could even spend time doing vulnerability discovery work on that application to try and find a bug that would actually give them what they want, or the underlying infrastructure.
Wherever possible, don't put a server directly on net. Put it behind a load balancer, so that someone has to pass through these things. Unless there's additional gates. You can put your WAFs in application load balancers and what not to try and bolster defenses. I don't think WAF’s a panacea, but I think it is a logically useful part of strategy when you're trying to defend applications that you don't want to heavily invest into. If you're going to totally rewrite the book, then, what are the security and variance you need? You need to be able to prove identity. You need to be able to prove authorizations. You need to be able to audit.
Any one of those things is a library. If it's not a library, then a bunch of engineers are rebuilding a thing over and over again that really should be built once, audited heavily in that one build place, trustworthy for the organization and then used consistently. If you're going to convert, that's the place. Spend a bunch of time getting the controls that enforce the core security and variance right, and then that gives you a foundation that is very strong to work off of, as opposed to a world in which you're still playing whack-a-mole with bugs everywhere.
You may have to do that thing, by the way. Again, it's always a “yes and,” but it's how much energy do you put in the different places? I put more energy into fixing things at scale, than I put to getting a little bit better at finding bugs. If I've done my 80-20 on finding vulns, probably the majority of attackers aren't going to go past that 80-20, unless I'm, you know, certainly, your Fortune 500 companies, they're screwed. They're facing every adversary in the book. For a lot of the smaller companies, they can say like, “I am 80-20 good at finding vulnerabilities. If I can prevent these kinds of things, whether from being introduced in the first place, which is the ideal, or at least prevent their exploitation, then I have probably a reasonable risk posture.” Then do that prevention in a library form as opposed to doing that prevention in the form of just reminding engineers to get better at not introducing SQL injection, or XSS, or whatever.
Yeah. I think that there's a whole different thing, which we probably don't even have time to go into right now. It’s about another conversion that companies try to make at the same time, which is monolith to service to microservice transitions. Because I think the world of security, I mean, it is hard at best to reason about the security of a complex system. When you have microservices that each have their own assumptions, if there's not a framework that you're working in, that declaratively says what the assumptions about security all microservices must make and what the invariants they must enforce are, then it's very hard, if not impossible, to reason about a specific trace all the way from one side of an app – the frontend application through the end microservices it touches to the data layer.
It becomes like, any one of those could inaccurately enforce the invariants. Any one of those places could inaccurately enforce authorizations or authentication, could fail to audit, could return data that it shouldn't. I think that that thing is probably equally if not more challenging for organizations to go through than the conversion of, this is in an on-prem data center, to this is in AWS, GCP, Azure.
[00:35:05] MC: Recently, the US Cybersecurity and Infrastructure Security Agency or CISA, they started cataloging cybersecurity bad practices. I'm curious from your perspective, you've been doing this long enough now, when it comes to cybersecurity bad practices in, let's say, DevOps teams, what do you commonly see? I don't know if you can think of a top three, but what does that look like from your experience?
[00:35:29] JB: I don't know whether it's fair to call this a DevOps team thing. I think, I'm going to say, at a company level, I still see the same security gaps that I saw when I started my career back in the mid-2000s, which is inventorying is hard. People don't do it very well. Patch management and vulnerability management is hard. People don't do it very well. Basic controls for AAA, I don't actually think are as hard, but people still don't do them consistently. I wish we were in a world in which those things were just solved for at this point. It feels like they should be commoditized. It feels like they should be easy to solve for but the reality is it's still hard.
I actually think cloud makes it a little bit easier. At least on the inventorying side, you can get authoritative, accurate inventory from a cloud system about what you have in it. I think, until those problems are no longer things that are, let's say, immature within an organization, you're really aiming too high, if you can't solve those problems. I still see so many companies that can't solve those problems effectively, or consistently.
That, yeah, I can come up with some very cloud-specific, these kinds of misconfigurations mistakes get made, etc. Honestly, basic vuln management, basic patch management still bites so many companies, still cause so many big breaches that it's hard for me to want to recommend anything other than get this right. Then we can talk about all the other cooler stuff, or more interesting, or challenging to solve security problems that emerge beyond that.
By the way, getting it right in cloud I think is, in many ways for many people, more doable. The AAA side of things, whether at the infrastructure layer, or within the applications is a thing that you can enforce with consistency, right? For example, using AWS, because I know more about it than GCP, and so I can talk about specific things. you can turn on cloud trail. You can turn on guard duty on top of cloud trail and immediately, I mean, you can turn on a few other things, too, and then you immediately get a meaningful amount of logging of all the actions that are occurring.
Suddenly, you go from like, “I didn't have reasonable auditing,” to, “No, it's not perfectly complete. I don't have every piece of data that I need. Our team is certainly not going to be satisfied with just cloud trail and guard duty. But, wow, I have a meaningful start to that control, and then I can evolve from there.”
In cloud, I can literally call an Amazon API and spit out a list of everything within my organization. In terms of the vuln patch management thing, I think the opportunity here is you can get coverage that is hard to get otherwise. Traditionally, we have agent-based and network-based vuln scanners and they maybe catch everything and maybe don't, and maybe you can keep them up to date.
When they can just pull off an API and say like, “What are all the systems? I need to scale those systems,” and then there's another API that they can be granted access to that allows them to run commands on all of those, you see two machines, we’ll talk separately about the concern that might create, but you can get a level of coverage and information in a short period of time that, when I think back to trying to help secure, or build those kinds of controls in on-prem systems with more traditional IT, or satellite technology, it’s just leagues easier. This is like, do these things in cloud, too. You can do these things more right in cloud more easily. Do that first. Then we can talk about all the cool whiz-bang things that you might want to do beyond that.
[00:39:09] MC: It sounds like from what you're saying, and I love the knowledge that you’ve imparted to me and both the audience, but it sounds like you're saying that if someone's listening to this and they are in a Fortune 100, Fortune 500 type of company, that it really is about fundamentally changing the approach that your default way of getting security when we've got shouldn't be, “Hey, I need to go buy a security product.” But first, I need to fundamentally think, “How do I build security into this from the start?”
Maybe that is a security product, but that shouldn't be the end goal. Does that sum up what your –
[00:39:47] JB: I 1,000% agree with you. I think one of the biggest challenges I've noticed, just self-owning here, one of the biggest challenges I've noticed that people have when reporting to me is I put so much emphasis on, “Start with why.” A lot of people come to me with like, “I want to buy this tool.” I'm like, “Why?” They're used to just not having to answer that question, or it’s self-obvious to them. Now, I do it to the people that work for me, because in part, I'm trying to prepare them for the reality of talking to other people outside security, because I expect a partnership between security and the way they solve problems and engineering and infrastructure and IT and HR and finance and all these other teams.
It's also because, I think, we too often get focused on buying a tool, instead of asking ourselves like, “What are we trying to achieve? Why are we trying to achieve that thing?” Then how is a separate question from that what and why. Because what is engineering, really? Engineering is, measure what you want to get better. Change something that you believe is going to affect that. Measure again and repeat that cycle ad nauseum.
Now, I'm oversimplifying that greatly, to be clear. I think in security, we often talk about the whole OODA loop, Observe, Orient, Decide, Act. The idea is the same. It's like, the what and the why establish what are you actually trying to affect. When you know why you want to do something and what you're really trying to accomplish in that thing, there's so much opportunity for partnership.
I'll give you a very specific example of what I mean. I do not understand why security teams want to run separate log collection anymore. If you are in a company in which there is a monitoring team, or a logs team, or something like that, they probably run a service that is better than the security team will run. Also, as a CISO, I don't want my security engineers to be sysadmins for – I was going say for Splunk because I have a particular – It's so expensive to keep up and running. No offense to Splunk users, but I don't want to pay for my security engineers to be sysadmins.
I don't understand why the default posture is like, “Oh, we need our own stack. We need our own tools.” I can reason about that but rarely is it because someone's like, I have never yet without prompting heard someone come to me and say, “We need our own stack.” I say, “Why?” Then they say, “Because the monitoring team won't give us an SLA that we think is enough for the quality of their tools.” No one says that. They're like, “Because this is a security product. It's not a log thing.” I'm like, “It is. I'm not saying you don't need to build something on top of our monitoring tools to make it useful for you, but why would you want to deal with all the data lake issues that exist? Why would you want to deal with all that collection infrastructure? Doesn't make sense to me.”
Coming back to the, what is the capability you want to create? Why do you want to create it? What is the outcome that looks good enough for you? It gives freedom to security engineers more than anything else, to actually solve the problem rather than getting stuck in like, “I want to use this tool,” and then finding that a tool may not solve the problem for them.
[00:42:48] MC: I love that. One of the things I have found consistently with high-profile leaders like yourself is that leaders are readers. Unfortunately, I can't claim fame to that. Somebody else said that. Tell us a little bit about what you've been reading lately, and maybe how’ve you found it useful in your role at Thirty Madison.
[00:43:07] JB: A couple of things. When I think about at the whole book level, I've been rereading Phoenix Project, which, if anyone's not familiar, is all about the rise of DevOps and the ideal – At least, an ideal version of what DevOps means. I've been reading both the Google SRE book, as well as a bunch of other writing that exists on the web about SRE and how to think about different ways to build out reliability over time within an organization because that's in my infrastructure team responsibilities to lead that organization. I am thinking a lot about reliability.
Obviously, as a company that serves patients, we want our systems to be extremely reliable. For me, thinking about how do we want to start moving into higher levels of maturity there, how do we want to invest resources into moving the bar there etc. makes a lot of difference to me.
Then, the other one that sticks out to me as a book that I – well, actually I'm going to say as an author, because I like almost all of his work. When I think about the raw leadership side of things, this isn’t a security book, everything Simon Sinek writes, I like. Leaders Eat Last is probably my favorite book of his but I think that basically everything he writes about is fundamentally about what it means to be an empathetic leader that generates real outcomes with people at the center, as opposed to seeing people as just a tool. He's very motivational. He's very good at bringing people on a journey. I think that that's a thing that I'm always trying to get better at. I'm always trying to think about the way I can tell the right – not the right story from – I want to tell the facts to everyone, and I want to help them understand those facts. Because not everyone hears the same data and understands it the same way.
It’s like, I want to help my engineers understand the facts of the situation in a way that makes them effective. I want to help the managers that report to me, understand the facts in a way that makes them effective. I want to help the peers in the organization, the other directors and up and VPs and such, make the right decisions. I want to help our senior leadership team make the right decisions about a specific set of facts. I think a lot of Sinek’s work is really good about helping people go through journeys and helping them understand. His understanding of storytelling as a mechanism for communicating the right information is so important.
[00:45:34] MC: Justin, I've just loved this conversation. We've covered so much. If people want to connect with you and they want to learn more about Thirty Madison, the roles that you're hiring for, where should they go to do that?
[00:45:46] JB: You're welcome to DM me on Twitter. I'm @justinmberman. You're welcome to find me on LinkedIn. I'm Justin Berman on LinkedIn. You're welcome to – I'm not going to give my email out, because I know that's going to turn into a bunch of vendor spam. I'm not going to give my phone out. You are welcome to apply through our careers page thirtymadison.com/careers. I am happy to talk with anybody who wants to talk to me directly as well.
[00:46:12] MC: That's awesome. Well, thank you for joining us today, Justin. I know that a lot of people will walk away –
[00:46:15] JB: Thanks for having me.
[00:46:17] MC: - from the podcast just really, with a change in mindset, I think, around how to build security into products to start with. Thanks for joining us.
[00:46:25] JB: I appreciate that so much, Matthew. Thank you for having me.