Mackenzie Jackson: Hello everyone and welcome back to another episode of The Secure Disclosure, Guy. I'm so thrilled to be sitting here with you. Thanks for joining me here today.
Guy Barnhart-Magen: Thank you for having me. Yeah, it's a shame we couldn't have cooler weather, but we will survive somehow.
Mackenzie Jackson: We'll get through this. I want to kind of start off just a little bit about your background. We'll get into some pretty wild things. You've had a talk here that's kind of I don't think could have been better timed in terms of predicting the future. But you know, you're one of the co-founders of Profero and I wanted to ask what made you first of all, what are you guys doing and what made you want to kind of get into the space and start dealing with these challenges?
Guy Barnhart-Magen: Well, that's actually a funny story. Around the end of 2019, my partner and myself were raising money for a startup around AI security, which sounds pretty cool now because everybody's doing that. But seven years ago, it was kind of a niche.
Mackenzie Jackson: And the reason I was doing that because that was correlated to the work that I've been doing for Intel.
Guy Barnhart-Magen: So I led a group of researchers into AI vulnerabilities and did a lot of stuff around that area. So we did get our term sheets, everything else, and then we got a surprising call from the Ministry of Health, which they wanted us to help them with some issue that they had with their application. They prepared an application and the QA people refused to install it on their phones. So they brought us in and we redesigned the entire application. We made it privacy by default. We had secure architecture, open sourced it, which was amazing because it was a government bureau. So having them agree to open source something that was amazing, and that was the first contact tracing application for COVID. Right, wow. So that was around that time and when we did that, we sat with a team of epidemiologists that explained how the epidemic is going to behave, like there are going to be infections, infections, and then there will be a quarantine, infections go down, and then there will be infections again and again and another cycle again and again until we have some sort of vaccine around. So my partner and I looked at each other, said we are opening a startup into a niche product in a world that's going to change dramatically in a couple of months. We are going to return the term sheets and find some other piece of business. And that was January, January 2020. And by March of 2020, which was the first quarantine in Israel, at least was the first quarantine, first widespread epidemic, we already shifted gears into something else. We understood that the world is changing. People are going to work from home, people are not going to be coming into the office. Security systems in offices were not built for this. Everybody was relying on the office as a perimeter, pretty old concept now, but it was the norm back in those olden days of seven years ago. So we made a decision to go into incident response. We understood that incident response used to be the realm of the enterprises, of the people with loads of money, and everyone everywhere is going to need something right now. And after the work personally that I've done on cutting edge technology like Cartoon and Meltdown, I really wanted to go into something that really moved the needle, really had an impact on people, and incident response is exactly that. You get to help people where they are in dire need, and this was not common for the SMBs of the world.
Mackenzie Jackson: And I think that kind of perfectly leads us into probably the direction of the conversation, which is going to be about the next generation of incident responses, because things are changing dramatically and the talk that you have at the moment is about what happens when the attackers don't have a face, don't have, you know, something's not behind it. And then like magic, you've predicted a couple of things by the sounds of things. And then just before that, we had the OpenAI Hugging Face drama, which is kind of exactly that when a company was attacked by a system that not only doesn't have a face but it wasn't even kind of acting in a malicious way but did real damage. So I guess let's start with how did you see this coming? What were the triggers for you that you saw okay, this is going to be the next wave of attacks?
Guy Barnhart-Magen: So one of the things that we are tracking from all of those incidents that we take part in is how the attack groups change their threats, tactics, the way that they perform. And when we look at this, we looked at this data for the past couple of months. We got to see more and more instances where things started off as an incident, a cyber incident per se, where we were called in because it was a suspected ransomware or suspected data theft or anything of the sort. But once we start investigation, we start to see that this does not have the DNA of a cyber attack. And further investigation led us to show and prove that this was actually AI based. Someone gave misguided instructions to an LLM or the LLM did exactly what it was supposed to do in the wrong environment, and something happens. And this really shifts the way that incident response is working because incident response is really geared up to focus on cybersecurity. But there are other types of incidents which have different ramifications. So if you are looking at an IT incident, something that's an insider issue inside the company, that means you don't have to disclose anything to the regulator. You don't have to notify your customers. You don't have any kind of legal responsibilities. A lot of things happen differently in a company when we are not talking about a cyber incident. From a damage point of view, the damage might be exactly the same. We've seen companies losing their production databases. We've seen companies that they had huge data leaks and everything else. But when you attribute it to some IT factor versus a cyber factor, the whole behavior changes. And this is something that's not happening enough today. People don't have the playbook, the viewpoint to understand this is an incident for cyber or this is an incident for some IT related issue.
Mackenzie Jackson: What I find so interesting about this whole system is it's how we've been able to kind of tell we've been attacked in the past is just from external IPs doing weird things. There's all these triggers in there. There's so much weird stuff happening on environments now. How do we even detect when a real incident is happening? Because we live in a world too where it could be a rogue agent that has interpreted instructions wrong or is doing something, or it could be a genuine malicious agent or perhaps someone's agent that's been given malicious instructions, or it could be an attacker, but the behavior could all be similar. How do you even tell?
Guy Barnhart-Magen: So that's a great question, and the problem is you really can't because the biggest differentiator for us as incident responders or anyone doing forensic work is to have some sort of evidence or logs or anything about that. And the way that AI or LLMs are deployed inside the customers' environments today is without any kind of logging, no centralized control. There's a lot of shadow AI going on. People paying for various tools out of their credit cards, out of pocket, not in an organized way. There's no centralized control.
Mackenzie Jackson: Guilty.
Guy Barnhart-Magen: It happens. People want to solve problems. There's no malice in it. But the bottom line is, just like we had shadow IT, now we have shadow AI. People use AI for many different things or people might be using things that even are not even aware that AI is running them. And then you go into forensic work and you start collecting that and you're really starving for information. There's a lack of visibility inside the organization. So, we usually recommend to customers, if you're already deploying, make sure that you collect all of those logs, because once you do have those logs, searching the logs for indications that an LLM did it are much easier than trying to reconstruct the timeline backwards.
Mackenzie Jackson: It is a terrifying world that we get into. I have a general question for you about this. Is where is your optimism at in terms of the AI usage? Because there's a lot of, especially in our industry, I think we really feel in our bones the doom and gloom sometimes. But sometimes, there is a lot of positive that comes from this as well. But where do you kind of position yourself not in kind of like, is AI good or bad because it's too. But just from security, do you feel optimistic that we can use AI as a defensive tool that will combat the offensive tool? Are you optimistic that we will get on top of that or is this going to be a very rough decade?
Guy Barnhart-Magen: Wow, it caught me into a corner and I'm pessimistic by nature and by trade.
Mackenzie Jackson: Look, it's kind of like the general theme of everyone on the podcast to be honest.
Guy Barnhart-Magen: Well, I'm not a big believer in FUD, in fear, uncertainty and doubt. I'm a very pragmatic, very practical person. So I think a lot of organizations are going to find themselves using LLMs for security for defensive purposes, in one way or another. So they might be using some tools or they might be using that to build their own. I think the gap between buying something and building something is getting narrower and narrower. And for example, we had a customer that decided to build an MCP in order to converse with their Elk stack, with their Elastic Logstash Kibana stack. And once they had that MCP, they were able to query their logs through an LLM interface and to get a lot more information out of it than they were capable of doing before. The same information existed. They did nothing to add more information into the system. But their ability to give a tool to the SOC people, to the security people, to actually do an investigation improved so much because now they are leveraging the LLM capabilities to go and search and sift through the data for whatever they were interested in. So I think from an optimistic point of view, the capability of people to reach data and reach decisions is going to improve dramatically. I think that the prevalence of LLM systems is going to be much higher than we see today. We're going to see that used for good or bad in many, many cases where we don't think maybe today that that's a good use case to do that. I'm part of a lot of different discussions of how much coding should LLMs be in charge of? Should code review be something that LLMs are in charge of?
Mackenzie Jackson: Yeah.
Guy Barnhart-Magen: But the bottom line here is not is this good or bad for the process. The bottom line here is that LLMs are able to ship out code much faster than we do. So if you can control the direction where this is working and the way that this is building upon, and we could fix bugs faster or we can respond to new features or design new features in a better way, we can do better security code reviews in areas that we don't have the capacity to do so today. This could optimistically put us in a much better place.
Mackenzie Jackson: Yeah. Yeah, I think there is the case for both, right? I think if you're willing to be on the cutting edge, if you're willing to push forward, adopt new processes and tools, and also drop them when they don't work, then you will be in a great shape, and I think everyone else is going to have a very tough time going through it. I think the ones that will really embrace it can do it. But I want this to kind of lead into a question. And if we go back to the OpenAI Hugging Face scenario, one thing that really interests me about it was how Hugging Face stopped it. They needed to use AI to be able to prevent the AI attack. And this is something that you have special skills and special expertise in is kind of, okay, we talked a little bit about discovering when we have an attack and the different areas and the challenges that we face in that. What do you do now in these, I guess, when an attack is happening but also when you have these AI systems going rogue, how do you manage that from an incident response point of view?
Guy Barnhart-Magen: So I think one of the underlying concepts that emerged from that incident or whatever was published on that incident, I think that the challenge here is real. When you're relying on cloud-based vendors, it doesn't really matter who, they usually come with their own set of guardrails. So, it doesn't matter if you use, I don't know, OpenAI or use something from Amazon Bedrock, it will still have some layer of guardrails in place. And those guardrails can be hindering whatever you want to do. And they're in place in order to hinder bad people from doing bad stuff, but they might also hinder you from doing good stuff. So I think that any organization that really wants to understand and deploy and build properly on LLMs can't just leave through a harness and some vendor API in the cloud. They need to build something on their own. They need to deploy their own models, even if it's in a lab scenario, just to understand how things actually work. And once you have that capability, growing from that capability into something that can be even a backup or just another tool set that you can use to reduce token usage, to extend capabilities, to have another build process, to just to be the main process is doing is something that's probably going to be very healthy. And people are learning that as a side note from the incident that you should probably be able to run your own model at your own cost, at your own design in order to fully understand how these things work.
Mackenzie Jackson: Yeah, and I think that it's an interesting world that we live in is that the company that I work for, we're building hardware for the first time ever because of this desire, as you speak, for people to do AI pen testing as an example. That's all well and good and that we want to have it in house, and I think this is changing in perception. You talked a lot about the guardrails that kind of in there. It's an impossible question to have like with these guardrails because in this incident too is that they needed to use an open source model because they needed to get rid of the guardrails to be able to stop it. This question can kind of come into a dangerous area of people's political inclinations, but do you think the guardrails are a good thing in general for these frontier models? Or because the bad guys can get rid of them anyway, is it a losing battle? Should we get rid of the guardrails as they as they stand? Is it I guess the ultimate question that I'm trying to answer is, is their kind of direction and pursuit of safety from the frontier models hindering our ability to defend?
Guy Barnhart-Magen: That is a deep question.
Mackenzie Jackson: I'm sorry, it's a terrible question to ask someone. I might not have an answer, but maybe I have a direction. I think that the cat-and-mouse game that we are playing in cybersecurity against malicious actors, I don't know, I've been in it for 30 years. The same question keeps repeating. So can we build our defenses strong enough? What happens if our attackers are going to use the tools that we've built? So let's put more defenses into the tools that we've built now we can't use them but the attackers are not using them anyway. So the same question kind of hops around from technology to technology. It's not specific to AI in general. And I think that the answer here is that you probably do want guardrails and you do want safety for the general use case because security is a use case right now for AI, is probably the highest for development is the highest use case. But that's not the vision. The vision is this is going to be useful in any and all areas of life. And if you have guardrails, it might be better or less friction for the general population to use it. Having said that, having models or having the same models with a different flag for security purposes might be reasonable because for specific people that you vetted or specific companies that you vetted or you charge more or whatever, and you provide them with the ability to run a frontier model without the guardrails where they take liability or they take some agreement or whatever, that's probably going to be the solution moving forward because the guardrails have their uses, but they're also uses without the guardrails.
Mackenzie Jackson: Yeah.
Guy Barnhart-Magen: So, we'll probably need to find a way to answer both.
Mackenzie Jackson: Yeah, I can kind of I I agree with the direction that you head, but is this an impossible question to ask because you can see it going terribly wrong in every direction of removing guardrails or increasing guardrails or kind of like this is there's no easy solution in this security.
Guy Barnhart-Magen: There's a parable that the attacker only needs to be right once.
Mackenzie Jackson: Yeah.
Guy Barnhart-Magen: So, the attack only needs to succeed once. So even if we find a thousand different ways to defend, they only need to be correct once. And we can't build our defenses on that they will be correct once. We need to be able to build defenses that will stop most of everything else. So we can't aim for the 99.9999% for security, but if we aim for 90% for security, that's still a pretty high bar, right?
Mackenzie Jackson: And what do you feel like if you were kind of to instruct clients of yours or people that you've worked with in when dealing with cybersecurity is, okay, we're going to put up walls. We're going to have endpoint detection. We're going to have these things to attach it, but we can't stop every breach, and we can't stop the risk of being breached, right? We can't, as you said, get there. What is what is something that people can do to be prepared in the event that something does happen, right? Like what does that look like and what should companies do beyond just trying to build, you know, walls and more vendors?
Guy Barnhart-Magen: Honestly, I don't think that walls are the first answer. I think the first answer before anything else is visibility.
Mackenzie Jackson: Mhm.
Guy Barnhart-Magen: So if you don't have visibility, meaning that you collect logs and the proper logs, for example, the session logs, the tool invocation logs, these are usually the two blind spots that we see. If you collect those and put them in whatever data lake or SIM, whatever you have, you're already halfway there because now you have visibility which you did not have before. It's not that obvious how to do that because that changes between if you're using something on the endpoint itself communicating with a cloud directly or you're moving through an internal proxy or gateway. Different architectures, different solutions. You might have both at the same time. So people running cloud directly in the cloud and someone else running cloud through a proxy inside organization. So both can happen. But if you have visibility or you build or architect around the problem so you will have visibility, you will at least be able to see when something happened and respond faster, or if something happened and you know that it happened and the reason was an LLM agent did it, at least then you won't be in a cyber incident, you'll be in an IT incident. Once you have those logs in place and you collect that information, then you can start thinking about what can I do to reduce risk and there are different types of risks that you might think of here. The main risk that I think most organizations care about is data leak prevention. What happens if one of my people are using the tool to analyze customer data and now that LLM decided to publish it on some GitHub or something might happen? This is a big concern and those agents don't have any kind of DLP built into them. So this is something that you need to consider if this is a risk that you're concerned about, build around it. Another risk which is I'll have to say it's pretty common is the tool is running as the same identity in a sense of the user who ran the tool. So it has access to all of their keys, all of their bookmarks. They can browse the web with their credentials. They can access any file on the file system. That means that if you have a DevOps engineer running a tool, now the tool has the entire profile and sets of credentials that the engineer has. The risk of them doing something to production or even more let's minimize just to the PC running for that DevOps engineer is pretty large, they can accidentally delete the entire laptop or accidentally run a Terraform command to delete production.
Mackenzie Jackson: Yeah.
Guy Barnhart-Magen: So you have to build around that risk as well. Segregate environment, segment the network, everything that we know how to do. And the most crucial part of that is to make sure that when an agent is running, only give it access to the things that it needs access to, least privilege principle for agents. It's working for other things, it's going to work for agents as well. It's not that easy to deploy, but it's going to solve the problem in a different direction.
Mackenzie Jackson: It is really interesting talking about this because something that freaks me out about agents is agents are authenticated in either two ways. They're either authenticated in the way that we authenticate systems and microservices like through API keys and all that or we give them human access. Now both of those things are inadequate for an agent because when we go down the API path, when we give someone, when we give a system, a microservice, an API, it's a kind of binary microservice typically, right? It does one thing, whereas an AI agent is non-deterministic, so it could do anything and you give it the same access, right? Now it can work in a very weird way to do something else. And the same with human access, so it's this least privilege thing is kind of like do we treat AI agents as systems and give them least privilege for systems or do we treat them as humans and give them least privilege for humans? And there's like trade-offs for both of those, I guess.
Guy Barnhart-Magen: So, in a seminar I did a couple of about two months ago, I did a seminar for a client, this exact question came up. And the way that I try to give a mental model for this is try to differentiate between running a harness and running an agent.
Mackenzie Jackson: Mhm.
Guy Barnhart-Magen: The difference here is both a harness and an agent can be CLI tools or some automation or something doing the agentic loop. The big difference here is that an agent is doing this autonomously from a set of instructions and the harness has a human driving behind the wheel. So why am I differentiating between these two cases? When we're building an agent, we can do a least privilege. We know exactly the type of job that this agent is designed to do. So we know the capability that it needs in order to perform that task. When we're running a harness and a human is driving it, then we don't know what task it's going to do beforehand and it needs to have access to some credentials to perform its job. And if we are not going to provide those credentials, the user is going to provide them themselves. So I think that solving the problem for harnesses in general is going to be a big difficult problem. Solving the problem for agents is things that we can do right now. And the problem is, do you know when you're running an agent versus running a harness? Because if you already have agentic workflows, focus on these, minimize their credentials that they need. Give them proper RBAC. Make sure that they're running with whatever they need in order to accomplish the task, but no more. You have two different agents, give them two different sets of credentials. But this is for agents where are autonomous and the task is pretty much deterministic. If you're looking on harnesses for people, I don't have a good solution for that now. So right now when we are looking at it, we don't differentiate between the human and the human running the harness. Same set of credentials, same kind of identity, doing same kind of tasks. That's why logging is so important.
Mackenzie Jackson: Yeah. And at least understanding what's happening in a full circle kind of takes us back to kind of that threat detection of like knowing what these, the getting visibility into what you said is as well. What do you kind of see as the biggest risks that we're going to be facing kind of in the in the short term? Like we've there's been a lot of headlines in the last year. We've seen supply chain attacks go kind of crazy. We've seen this new AI agent thing. It kind of feels like, you know, it's it's pretty wild out there. And I think a lot of it is because some of it's kind of unknown, which makes it feel scarier. What do you think is kind of like the biggest threats that we're going to be looking towards in the next five, 10 years, maybe? I think 10 years is too long to take out, they say.
Guy Barnhart-Magen: I have difficulty with five months. Honestly.
Mackenzie Jackson: Yeah.
Guy Barnhart-Magen: I'll try to answer it this way. When we started out in IR 2020, the biggest risks were ransomware and remote access. So it started with people gaining remote access to networks, then ransomware and ransom kept getting larger and larger and larger and larger as an industry problem up around to 2023, more or less 2022, right around the war with Ukraine and Russia. So about that time we saw a huge drop in the number of ransomware incidents. We can talk about the political context of that, but bottom line was we saw less ransomware. And after that, we started seeing a new type of ransomware where the goal was not to encrypt files, the goal was to exfiltrate files, to siphon files out. So doing a data leak and data leak ransomers. So I have your data, I don't need to encrypt it. Just pay me. So I won't publish that. So that has been the case for the past, I don't know, a year and a half, more or less, until 2025. And up until then, the cadence was more or less deterministic. We could more or less linear. We could tell that on average a company would have an incident every couple of months. Let's say twice a year, three times a year, more or less. We've kept seeing this cadence narrowing and narrowing and narrowing. And when I'm saying a cyber incident, I don't mean just somebody entering the network and ransomwareing all your files. It could be somebody entering the network and stealing files and extorting you. It could be somebody stealing user credentials by installing info stealers on endpoints. It could be someone running, I don't know, thinking about recent Monero crypto mining on virtual machines in the labs. It might be any one of those, but the amount of incidents kept increasing and increasing. So if I'm trying to predict the future, extrapolating the future, in the past we saw that incidents were months apart, then weeks apart, I think that we're going to see not days apart, we're going to see multiple incidents per day.
Mackenzie Jackson: And in incidents here, I also include someone published a CVE and now I need to find out am I impacted by this CVE or not immediately. Yeah.
Guy Barnhart-Magen: So you need playbooks for that. You need playbooks for my rogue AI did something, is it AI or not. I need something for somebody stole my credentials and now I have a rogue API token for something spilled out on the web. Some of my employees had info stealer stealing their credentials. Am I exposed? All of those things that used to take days or weeks to develop, getting a foothold in the network, lateral movements, all of that is going to happen much, much faster. That means you're going to have many more incidents. And that means that you have to be able to respond to those incidents faster. That means better IR playbooks. That means to have good people on retainer when you need them. That means that you need to be prepared ahead of time for the incident that's going to happen. You can't wait for the incident to happen.
Mackenzie Jackson: I totally I totally agree. I want to kind of we've got to start wrapping it up now, but I want to ask you just like a simple question then we're going to go into our final game. But generally speaking, are you optimistic into the like into the future from like security? Like are you optimistic about where we're heading?
Guy Barnhart-Magen: I'm not a very optimistic person and that serves me well in my profession. I'll try to be optimistic, but I will highlight one of the points that I see as problematic. Earlier today I had a conversation with two different people, juniors in the industry, who came to the Black Hat Las Vegas conference and we had like a heart-to-heart conversations of what can they do as juniors going into a world where they don't need juniors anymore.
Mackenzie Jackson: Yeah. Okay.
Guy Barnhart-Magen: So as I'm thinking about the future right now, and I had this discussion internally, other people had that discussion in their companies, but the bottom line is the way that we at least the people I talked to see LLMs right now really objectively is that when you give it to a senior, it doesn't matter if it's a senior incident responder, a senior developer, it's a force multiplier, they can do so much more when you give them an LLM to help them. And if you take a junior, you only multiply the risk but not the value.
Mackenzie Jackson: Right.
Guy Barnhart-Magen: And that means that there's less incentive to build out the junior's capabilities because if I define what do I want from a junior, I want to give them low-risk tasks. I want to give them things that are not very don't need a very deep technical background and I want to give them things that can be done on a relatively short life cycle because they can't it's not going to be a three-month project when I give it to a junior. So looking at all of those characteristics for tasks, I'm looking at task for LLMs, not task for juniors. And if they are interchangeable, that means I can do more with LLMs and not hire those juniors. Looking forward, this is going to be a huge problem for us down the line. I don't think that AI is going to replace humans in the loop here. I think that AI is going to be very helpful to humans in the loop here. I don't have a solution to how do we get from the junior to the senior stage without actually gaining that experience.
Mackenzie Jackson: Yeah. So, we have a big gap here. I don't have a solution for that. It is a challenge and I feel it. Yeah, very much. I could talk all day, but we got to have to wrap it up, but I'd like to finish off with a game.
Guy Barnhart-Magen: Okay.
Mackenzie Jackson: The game is called Would You Rather? And I'm going to give you two very crappy options, and you have to pick one. So the first one, I kind of I try and make these up based on the conversation, but I think what I'll do is you'll talk about open source at the start. So let's say that you're running a big application out there. Would you rather have that completely open sourced and entirely, you know, and in this world of AI scanning, understand maybe multiple zero days or run your application with absolutely no firewall? Which would you rather?
Guy Barnhart-Magen: Honestly, that's an easy question for me. Many, many years ago when I started out in security, there used to be the rule of thumb of security by obscurity. Don't tell anyone what you're doing. That'll make your things safer. And then I got into reverse engineering. So I don't tend to look at this as like as obscure as you might think it is.
Mackenzie Jackson: Yeah.
Guy Barnhart-Magen: So I think open source is great. I'm even neglecting to consider different licenses, just exposing your code because I think the bottom line here, if there is a vulnerability there, people will be able to reach that vulnerability with or without your source code. And if the source code is out there, the chances of someone reporting that vulnerability to you in a responsible way that allows you to fix it are higher.
Mackenzie Jackson: Yeah, that's a very good answer. I have another one. I'll try and you mentioned CVE before, so we'll do one here. Would you rather pin all of your dependencies to the latest version, but it stays there forever? So, new CVEs come out, you cannot update your dependencies. They're stuck on versions whatever forever or you automatically pull in the latest version which comes with the risk of malware as we've seen out there, but also the latest fixes.
Guy Barnhart-Magen: The latest fixes. Yes. Yes. We actually had the same discussion about two years ago. And the way that we look at it is that I can't trust the latest deployment or the latest release, but usually we try to do N minus one or N minus two. So try to keep one version, two versions back and hopefully in the gap between the versions, somebody would be able to catch any kind of malicious intent by the deployments. It's a risky behavior. I don't have a solution for that. But I do want the security fixes more than I want the risk of staying behind.
Mackenzie Jackson: Okay. Well, that's a very good answer. Well, Guy, thanks so much for joining me here today. My pleasure. It's been a fascinating conversation. I really appreciate you coming in here and I hope to follow your journey and maybe have you back on the podcast one day.
Guy Barnhart-Magen: My pleasure. Thank you very much.
