Ammar Alim: [00:00:00] In security, the biggest currency is speed. If you can't secure quick, might as well you just don't secure. He told his AI system, "Find a way to give me root access to this device, shell access." And the AI was able to stitch multiple vulnerabilities together and have root access in the Wi-Fi system.

Ashish Rajan: Vulnerability being announced, uh, in publicly and exploit found in the wild, the window was less than twenty-four hours.

Ammar Alim: Urgent health issue, going to the gym is not the answer. The answer is calling nine one one. I didn't go to school for it or anything, um, because- No academic,

Ashish Rajan: none of that stuff,

Ammar Alim: right? No, I didn't build a model, uh, for-

Ashish Rajan: In a given day, there's ton of vulnerabilities coming every day.

Ammar Alim: Things will be discovered very fast and taken advantage of very fast. I'm talking about we're going from month and weeks to hours.

Ashish Rajan: Try using AI with a web application firewall. Yes, the AWS WAF or whichever third-party WAF you end up using in your organization can actually be [00:01:00] amplified into being automated for any CVE that comes out.

Ashish Rajan: If that sounds surprising, let me introduce you to Ammar Alim. He works in the product security team at Adobe, and we spoke about how do you even start preparing AWS WAF or any third-party WAF for being agentic, and I mean agentic in the sense of having an agentic pipeline. So, if you have been consuming a huge amount of threat intel on the latest CVEs across multiple sources, trying to see what would that look like if you were to apply AI to it to scale it out, especially in a world where vulnerabilities are almost being exploited within a twenty-four-hour window, this is the conversation for you.

Ashish Rajan: As always, if you have been enjoying the episodes of the podcast for a while and are here for a second or third time, I really appreciate the support, and I would really appreciate it if you take a quick second to hit the follow/subscribe button, whichever podcast, audio/video platform you may be listening to this on.

Ashish Rajan: We are on Apple, Spotify, YouTube and LinkedIn, and wherever you consume your podcast from. I really appreciate the support, [00:02:00] and I hope you enjoy this episode with Ammar, and I'll talk to you soon. Hello, and welcome to another episode of Cloud Security Podcast. I've got a return guest today with Ammar. Hey, man.

Ashish Rajan: Thanks for coming on the show.

Ammar Alim: Absolutely. Thanks for having me.

Ashish Rajan: Man, I'm super excited for this conversation, but for... It's been a while, so why don't we start with a little bit of introduction about yourself and your professional background and where you are these days, man?

Ammar Alim: Yeah, absolutely. So my name is Ammar.

Ammar Alim: I currently lead one of the product security engineering functions at Adobe, and I came to this kind of, uh, stage of my career through a long, career in technology, starting from being a, a person in a VMware data center to then not wanting to be in a cold data center and decided to do the same thing in the cloud.

Ammar Alim: So I transitioned to cloud engineering and as security was becoming more relevant in the cloud, uh, naturally and organically found [00:03:00] myself the more security-focused person on the team. This is a little over ten years ago, maybe ten years ago. Just, for reference, just because this the word, uh, or the phrase of, uh, cloud security wasn't a thing.

Ammar Alim: It was not even, there was not a role available with the cloud security engineer. So I kinda started early in the cloud security area, and from there I don't like having gaps in my understanding, so I ventured more into application security. And I think both cloud security and AppSec now are under the product security umbrella, so you're expected to know a little bit more about those in addition to AI, of course.

Ammar Alim: I mean, AI security, using AI for security and securing AI systems. So all of it is product security. And now I'm actually more interested in this, like, how can I use AI to do more of the things that were not possible in the past? I think there is a ton of opportunities, and I [00:04:00] would like to be, um, in these conversations.

Ashish Rajan: Yeah, and I think the unique background you've had, 'cause you and I have known each other since, to your point, when cloud security title was not even a real thing. We j- we used to call ourselves cloud security engineer in our private Slack groups with fwd:cloudsec I guess. Outside of that everyone was like, "You're just a regular security engineer," but we've come a long way.

Ashish Rajan: And to your point, AI has enabled a lot of things that used to be Oh, uh, it's my limitation because I'm more a software engineer, I'm more an appsec person. But AWS WAF I think specifically going into that because I've spent quite, quite a bit of time on it. We built a whole agentic pipeline for it.

Ashish Rajan: But before I go into the whole agentic pipeline and the AWS WAF conversation, what was the starting point which-- or the frustration that you had with AWS WAF, uh, that triggered for you to go down this path of building like an agentic pipeline and using agents to, uh, manage and create WAF rules and stuff?

Ashish Rajan: So I will definitely tap into that, but I'm just [00:05:00] curious as to what was the, uh, the problem or the instigator to this journey that you walked into.

Ammar Alim: Yeah, absolutely. Great question. So just a disclaimer, it's not just AWS WAF. For my AWS friends, I love you, but it's just a sheer scale of complexity of dealing with web application firewalls.

Ammar Alim: Just for reference, we do use about seven web application firewalls, including open source, so things like, uh, Wallarm. Uh, you know, this is... I-I'm name-dropping, but Akamai, Cloudflare, AWS WAF, Azure WAF. You name it, we have it. This is the consequences of us being a tech company enabling engineers, and we allow them to use whatever kinda fit their environments depending on the stack, depending, you know, if we acquire the company as well, they bring you their own kind of tech, uh, tech stack.

Ammar Alim: So when you have seven different commercial products, understanding all of those and scaling management of them is very [00:06:00]hard acro-across a large enterprise. WAF is a little complicated. It's not something that you see a lot of conference talks about. It's always gonna be a very niche BlackHat researchers kinda digging into the nitty-gritty of web application firewalls.

Ammar Alim: It's not very widely known, and you need WAF administrators very specialized in one product to run your web application firewall. And you can't have a lot of those, meaning you can't hire a WAF admin for each product team and for each WAF vendor, right? So- Yeah ... th-this doesn't scale. Then you need to throw a lot of people at it.

Ammar Alim: So we were keeping up with it barely or doing the bare minimum, meaning we set it, we try to do it to the best of our abilities and resources. When AI came about, we started thinking about, yeah, this is the golden opportunity now to really get the most out of these technologies and [00:07:00] harness their powers and take it all the way.

Ammar Alim: Like the dial was, if you wanna use it for like maybe three out of 10, we want it to go all the way to nine minimum. The frustration is, is just again, sheer scale and sheer complexity of the technology itself.

Ashish Rajan: And I, I think I love what you said as well, and I appreciate you kind of saying that this is not just AWS WAF, but there's a lot of WAF providers.

Ashish Rajan: And it's funny, the, the first time you and I had this conversation, I remembered the amount of time that I had a team member, uh, this is one of my previous jobs that I was still a CISO. Two of them were primarily just working on looking at AWS WAF output And making like, you know, and there used to be so many false negatives or false positives, sorry, and just spend hours trying to go through that.

Ashish Rajan: And by then, if there was a new vulnerability that came out... And to your point, we were not even using AWS WAF, we were just using a third-party one. I won't name the vendor, but the vendors back then would not have any API calls that you could use. I mean, [00:08:00] MCP is like obviously this is pre-AI, and I had given them a task to kinda can we automate this?

Ashish Rajan: How much of this, like, can we automate? And it di- every single time, the challenge used to be, well, they don't have an API or they don't have the capability for it. The reason why I bring that up, e-even though we did get to a point where the in-individual ended up spending a lot of time and energy into reducing the amount of false positive by marking certain ways, with AI, did this change?

Ashish Rajan: Because there's obviously, like the, the newer kinds of threats. I, I... Before we go into it, I wanna talk about the existing kind to what you've mentioned. The volume, you have multiple WAF providers. The volume hasn't stopped, and that still continues to grow in. Does AI help with that as well in terms of managing the volume of the issues and alerts and everything else that comes out of WAFs in general?

Ammar Alim: Yeah. I, and I think it would be beneficial for the audience to understand what, in the sense of WAF and web application security, what a false [00:09:00] po-positive, I think, Gee, this is a good point, and what's a false negative quickly before I answer that question. Yeah. Because I think it's a great question. So a false positive is technically where, you know, the WAF is blocking traffic that it shouldn't be blocking.

Ammar Alim: And the other way, false negative is when the WAF is allowing things that it shouldn't allow, it should be blocking. And it's a very nuanced balance to achieve. It's not easy. So if you have a, a vulnerability such as Log4j, if your regex in the rule is so specific to that vulnerability, you may end up with traffic not being blocked.

Ammar Alim: Some of it, some of it might not be blocked. If it's, uh, generic, you might end up catching things that you shouldn't be catching. Traffic that it's in, the pattern of that traffic is very similar to legitimate customer traffic. So false po-positives have not gone away, and actually they increase just because AI makes, makes it easy for [00:10:00] anyone to start, mimicking real traffic trying to exploit your defenses.

Ammar Alim: With that being said, AI does Tremendous job as far as understanding your application and trying to help you reduce the amount of false positive. So you can use AI to do the tedious things that are really hard to do, which is reading logs of the WAF, right? Reading source file. W- one thing AI is really good at is reading code.

Ammar Alim: So you can actually parse the entire source code of your application and understand it very well while trying to develop a rule. You can feed it your architecture diagrams, you can feed it your threat model output. You can feed it as much as you want to be able to craft a rule that is custom-made For that vulnerability is trying to pa- temporarily patch until, like, a permanent patch is available.

Ammar Alim: So it does tremendously, uh, there.

Ashish Rajan: But do you find [00:11:00] that the lack of an API from the third party, the... Is that a barrier in this case? To your point,

Ammar Alim: I can manually do it- Absolutely. Absolutely. Yeah, absolutely. If you can't automate, you can't scale, and that's the frustration in the first place, right? AI cannot help you in...

Ammar Alim: at all. You need a automated way of pulling the logs. You need an automated way of testing the rules. You need an automated way of pushing the rules out. So, in the absence of automation, you can still do it, but it's, uh, it's an uphill battle. '

Ashish Rajan: Cause I think, uh, I've ex- tried experimenting with the whole agentic browser things as well.

Ashish Rajan: Just not really comfortable with that kind of going around, going around changing your production WAF, and I'm sure you're not either as I see you smiling. I was gonna say, so if I double... I mean, I-- and keeping the conversation aside, so we've set the foundation, okay. First of all, people, before they even go down this path, AI is not just for future AI specific vulnerabilities, but also for the existing lot they may be looking at.

Ashish Rajan: So that kind [00:12:00] of at least, in my mind, it addresses the thing that people have, "Hey, it's only for new kinds, not for the existing kind." So I love that. The thing that I wanted to also unpack here is why should they even do this? 'Cause there's obviously people have dedicated time and effort in customizing it, changing it, and even if it's not man- even, even if it's not automated, it's manual, why was there this drive from you and your team to, "Hey, we need to use AI to do this"?

Ashish Rajan: Was there, like, a series of events that, or at least what you're seeing in the industry that's making you go, "Oh, actually, you know what, guys? If you're not doing this, you seriously need to start thinking about this."

Ammar Alim: Yeah. I think in security the biggest currency is speed. If you can't secure quick, might as well you just don't secure, right?

Ammar Alim: And one of the trends, like this is not, it's not like a hidden knowledge, you know, secret knowledge. Everyone knows that today AI makes, uh, vulnerability discovery so much easier. It lowered the barrier to exploitation as [00:13:00] well, um, meaning anyone who is moderately trained is able to use AI to Chain multiple vulnerabilities together.

Ammar Alim: So I, I had a conversation with my VP recently about his thoughts about these frontier models in, uh, security. Because there are two camps, right? There's a camp number one, ah, this is all hype. Yeah. Camp number two, this is Mythos is gonna kill all of us and eat all of us alive, right? Yeah, yeah. Um, the reality is in the middle, which is he gave me a story.

Ammar Alim: One of his friends, I'm not gonna name the name because he's some- Of course ... top executives at Anthropics. Um, but he used it at his own home network, so he wasn't able to log into his own Wi-Fi hardware. And he told the Wi-- his AI system, which is we all use, Claude and, and whatnot, "Find a way to give me root access to this device, shell [00:14:00] access."

Ammar Alim: And the AI was able to stitch multiple vulnerabilities together and have root access in the Wi-Fi system. So-- And this is not something specific to that person. Right. We all see this every day, right? Like we c-- you can do this today. Like there's a ton of research already. Yeah. So compared to a year ago, we have this now, meaning things will be discovered very fast and taken advantage of very fast.

Ammar Alim: I'm talking about we're going from months and weeks to hours. So an, an excuse to not mitigate in such vulnerability would be a patch does not exist, or a patch need to be tested in lower environments, in staging for a few days, in, uh, in maybe pre-prod for a few days before we can release it to production.

Ammar Alim: Why? Because there are so many dependencies, whether you're using NPM or Python or any any modern software, you're using a ton of open source [00:15:00] libraries that you didn't write. You don't know how would they behave. They may end up taking down the entire system. So you need a quick emergency system, like to mobilize, to be able to mitigate a vulnerability when it exists until you sort out all that patching stuff.

Ammar Alim: Patching is still necessary. It's like going to the gym. It's like a long-term thing, right? Yeah. But you have an, like very urgent health issue. Going to the gym is not the answer. The answer is calling nine one one, right? Yeah. And the WAF for us is nine one one. Someone-- There is an emergency, we need to quickly deploy a virtual patch that stops that attacker before they knock on the door-

Ashish Rajan: Yeah

Ammar Alim: until we sort out, more layer of defense. So it's just we're reacting to what's coming, and we want to be ready.

Ashish Rajan: But is it, to your point, I think it's interesting, right? Because that's like an operational change as well. It's not just that I have... You know how people have higher [00:16:00] A... If I have a high vulnerability, high SEV one, I need to patch that within twenty-four hours.

Ashish Rajan: SEV two can be a two hou-- forty-eight hours to a week. SEV three, I don't know, maybe if you feel like it next, next sprint. That usually the behavior. Does this require On buy-in from the wider organization as well, 'cause d- and I'm with you, right? I'm, I've done-- seen the research. Uh, there is a good friend S- Sergei, he basically started s- started a website where you can actually see the trajectory of the exploit being announced to the point of actually being exploited in the wild.

Ashish Rajan: So sorry, vulnerability being announced, uh, in publicly and exploit found in the wild. The window was less than 24 hours, and it's very important for you to quickly know as well that, hey, am I vulnerable? Like if Heartbleed happened, Heartbleed happened again today, I wanna be able to quickly tell that, hey, I, am I impacted?

Ashish Rajan: Yay or nay. I think that's 100% the great love [00:17:00] that messaging. Is there an organization change required? And w- what was your approach that others can learn from on how to get the wider people into buying, buying into this? 'Cause the organization challenge, organizational challenge is real.

Ammar Alim: Yeah, absolutely.

Ammar Alim: Uh, you have to understand the ecosystem for all companies. All companies are here existing to Sell software- Yeah ... like all tech companies. That's the goal.

Ashish Rajan: Yeah.

Ammar Alim: And as long as you follow that rhythm and enable that, you will be able to get by. If you cut against the grain, you're not gonna get by. So simply put, this system allowed me to go to product leadership and say, "When there is a vulnerability, I will mitigate the emergency so you do not have to go in emergency mode.

Ammar Alim: You don't have to worry about it. You don't have to drop what you already planned. You don't have to drop features, customer commitments." There's so many things that they [00:18:00] are working on- Yeah ... that I would like it to be intact. All they need to know is that the vulnerability today is not exploitable.

Ammar Alim: They don't even need to know how. But the next patches, like rollout, we can manage and carefully plan the patch and roll it out. We still patch as quickly as possible, but it's not as quick as deploying a WAF rule. WAF rules take minutes. I can't say this about patches.

Ashish Rajan: Yeah. That is true.

Ashish Rajan: Wait, so maybe it's a good time to introduce this. How are you guys doing this? Let me, uh, let's get into the, the details of this as well. What was the approach? Maybe if you start at the ten thousand feet view, what does this look like- Yeah ... from a component perspective, and then we can double down.

Ammar Alim: Absolutely. So this is an agentic pipeline that reacts and gets started as a consequences of the presence of a new CVE. So you can listen into CVEs, you can, you know, see what Rami [00:19:00] McCarthy had wrote about, or you can log into a faster means like the GitHub database advisory, right? So GitHub can tell you that there's a new CVE that just came out, and they publish it via API.

Ammar Alim: So all you need to do is listen into that via a-another automated system and get notified. For example, just start by get notified in your Slack channel when something like that happens. So we take it further because not all CVEs are relevant to us. Uh- That's right. Yep ... maybe it's, it's not-- it's something that is not even patchable, maybe because it's a desktop component.

Ashish Rajan: Yep.

Ammar Alim: O-once we get notified about it, we actually scan our environment to understand if it's presence in our software components or not. Is pa-- Is it part of the, you know- Our ecosystem or not. If it's part of the ecosystem, we move on to the following step, which is what is the proof of concept that researchers [00:20:00] have made available, if any?

Ammar Alim: We collect as much data as possible. We simply use a deep research agent. That's all it is. We collect as much information about this vulnerability. POCs, did it happen before? We find it that most vulnerabilities are typically related to previous vulnerabilities. Simply put, there's nothing new under the sun.

Ammar Alim: It's a different vari-variation of something that happened in the past, um, in authentication, in, you know, any, any component of your software. Once we understand this is a vulnerability that impacting us, this is a vulnerability we have a lot of information about, then that in-information get fed to an AI model that need to generate WAF rules for all our WAFs.

Ammar Alim: So we have, as this, this AI model must generate rules for all the WAFs. We start with something simple like open source that runs locally in your machine [00:21:00] like ModSecurity WAF, right? ModSecurity can run in a container on your laptop. We can start there, or you can run it in GitHub actions, or you can run it in container, what-whatever.

Ammar Alim: Um, start there and test a rule by generating benign traffic similar to the application and generating false positive. Meaning, will you catch the vulnerability without blocking legit traffic and allowing bypasses? Uh, WAF bypasses as are a thing, so we test for that So once the testing is done, um, locally, we deploy to produ-- like to a testing environment in the cloud, um- Mm-hmm

Ammar Alim: like AWS and Cloudflare and those systems. And we actually deploy the vulnerability in a private environment. The same vulnerability that is available, we deploy an application with that vulnerability, put the WAF rule in front of it, and test again. So this is our last step. The, the step before the last step.

Ashish Rajan: Okay.

Ammar Alim: Once this, once we satisfied with the results-- and we have a scoring [00:22:00] system. We see how many times it was bypassed, and we see how many times it actually blocked, and we try again. We iterate to make sure it's actually working as expected. We re-review the logs and see what happens and so on.

Ammar Alim: So with all of that, our confidence is now around eighty percent. We're still not confident. How can we get one hundred percent confidence? We actually move the rule to production, but not in blocking mode. All rules allow you to shadow traffic- Yep ... to see what the rule would do when you start blocking.

Ammar Alim: And we monitor that for minutes, maybe half an hour, maybe maximum forty minutes, to see if the rule is working as expected. If we find any issues, we correct-- quickly use AI to correct that problem- Yeah ... and correct it, then switch it to blocking. The good news is, if you have a vulnerability, it's a bad news, but it's, it's good for me as someone working on WAF.

Ammar Alim: When there is a vulnerability [00:23:00] One, there is opportunistic people with scanners who try to spray all applications. So that's good testing for me- Yeah ... because a lot of people are trying to see if-- They know you have it, right? One, this is a usual vulnerability on open source systems. So the software is available to all attackers.

Ammar Alim: They know about this CVE. It's not hidden knowledge. Yeah. They know about the thing. They know what type of traffic they can craft to exploit it, so they will try it against you, mass scanning. So you will definitely see that there are attempts to take advantage of that. So the WAF-- if the WAF rule is working correctly, you will get to know that.

Ammar Alim: Second, when there is an event like that, those opportunist-- opportunistic a-attackers are not trying to work hard. They are trying to take advantage of that vulnerability by doing the least amount of work because it's, uh, economics come into to the picture here. [00:24:00] They see a, an easily exploitable opportunity like a zero, zero day.

Ammar Alim: They create the exact traffic that will exploit that vulnerability, and they just try every company, every application on the internet. This is how they do it. That helps because you don't really ne-need to worry about false negatives. You need to worry more about false positive. I mean, you don't, you don't w-- need to worry about false positives because the, the traffic they will generate will be very specific, meaning if you create a very specific WAF rule, you are very unlikely to block legit traffic So worry more about, think more about, about it from a strategic stan point, which is these people are opportunistic. They're gonna generate just the traffic that exploit that vulnerability. You just go and generate that rule that just protect that vulnerability. Don't do too much. Uh, don't make the, the WAF rule so wide that it blocks [00:25:00] legit traffic.

Ammar Alim: Just make it very specific and narrow- Yep ... and you'll end up succeeding.

Ashish Rajan: Do you find that and I guess to your point, so the AI component, I love that you're actually user-- using WAF for what it is supposed to be, using the read-only capability, then building on top of it to go, "Hey, is this actually going to work or not?"

Ashish Rajan: And then moving forward ahead with it. How does this work like a pipeline? I'm just curious 'cause obviously we spoke about one, and any given day, to your point, there's, for lack of a better... I'm sorry to say, there's shit ton of vulnerabilities coming every day. If you look at any threat intel, it's like there's one coming from Palo Alto, CrowdStrike.

Ashish Rajan: The, the list just goes on. Sometimes there's a bit of an overlap, but you as an organization, or all of us as an organization, have to ongoingly be on top of all of this. So I'm curious, how does this work at scale, and is that where the pipeline fits that mold?

Ammar Alim: Yeah. So of course you have an agentic you have a harness, right?

Ammar Alim: Like a, you know-

Ashish Rajan: We should explain what harness is. You should probably explain- [00:26:00] Yeah ... what harness is and at the different stages you guys went through, 'cause you've done this clearly, but a lot of people... Wait, actually, maybe if we take a step back, how do you even approach this? Let's start with the AI agent and then the whole journey, if you like.

Ammar Alim: Yeah. How about we start one component of the pipeline, so we organize it one step at a time. Then when we get to the agentic part, we explain- Perfect. ... what the harness is. So again, I'm gonna repeat some of what I already said, which is good, right? If you're new to this. So the first step is to gather information about what you're trying to do.

Ammar Alim: I am trying to generate a WAF rule for a spe- to mitigate a specific CVE. So I don't really need AI here. Like- Mm ... one of the things that you have to think about is when you're trying to use AI is to strategically think about where AI fits. So this is a deterministic step. The CVE was published. I need to ingest that data, and I need to then do something with it.

Ammar Alim: So input, process, output. This is still [00:27:00] relevant in, in AI. So then the moment the CVE drops, I have a Lambda function. You know, that's the cheapest, most reliable way of Uh, ra-- like every fifteen minutes, I kinda ha-- I can have a cron job just fetching, scanning a database, right? That's, that's, you know, nothing new.

Ammar Alim: I fetch the CVE. This is step number one. This becomes an output for the second Lambda function that will scan my environment. This Lambda function, all it does, it can just scan the Wiz API or CrowdScri-- uh, whatever, whatever security tool that has all you of-- all of the CVEs, uh, and scan your environment already.

Ammar Alim: I don't have to scan manually. I can just take it back on an existing system. So far, there's no AI. This component als- uh, also it took input from the first Lambda and, and this doesn't have to be Lambda. Like, it-- you can do this on, you know, on [00:28:00] ECS. You can even do this using GitHub actions. Like anything that allows you to run Automatic jobs cron jobs can do this.

Ammar Alim: But Lambda allows you to, to do multiple things. There's flexibility. You can orchestrate with step functions. You can orchestrate, you can use serverless for the most parts because this doesn't need to run continuously. It can be something that runs as needed. There is a CVE, then I do X. So step number two is I gathered the information I need about this thing.

Ammar Alim: I mean, I, I-- This, this CVE actually exists envi- in my environment, but I only-- I just know the ID. I know it's impacting this product. I actually don't know more. I need to understand more. I know the product, and I know the ID. So then you just get an off-the-shelf deep resource agent. You don't have to build it yourself.

Ammar Alim: You can maybe-- The instructions can be, custom-made, which is the context [00:29:00] for that agent. Get this, get-- Take the ID and look up the POC and look up how it will be exploited, and actually look up how a given software would use this vulnerable component. So build as much understanding of that.

Ammar Alim: This is the first step where AI comes in, which is off-the-shelf deep resource agent with your specific context. Then once you have that, you break it down, so in, in very like a deterministic way to the following, because the following step, because AI now is coming. Like we-- This is where the heavy lifting of AI happens, and this is where you have the agentic system, and this is where we define what an agent is.

Ammar Alim: Agent is simply a model, an AI model, like any model, local model, frontier model, any model, plus a harness. You would-- The, the question is, what is a harness? Harness is simply the [00:30:00] operating environment for the agent to be able to get its job done. The harness includes the APIs that you need to make and connect with.

Ammar Alim: So where the secrets for those WAF APIs exist, right? That's part of the harness. Second The, the shell environment and the commands that you need to execute there. Third, things like the memory, because there are a few things that you want to remember. You don't want the agent to lose memory of. You need the agent to remember where to find secrets.

Ammar Alim: You need to remember to tell the agent what to not to do, because, there's a lot of that. There is also instructions for the agent, which is people call it context engineering. There's so many names for it. Maybe you have loops as well. Do not stop until a given condition is achieved, right? You have constraints, you have things that you know are no-nos.

Ammar Alim: Only create WAF rules for AWS, but not [00:31:00] for Akamai in this step. When we move forward, you then do this. So the agent needs to have all the access it needs, just enough access it needs to be able to get the job done and the APIs. So this is pretty much what a harness is. If you're using Claude Code, it's a harness.

Ashish Rajan: Yeah.

Ammar Alim: Right? It's, uh, it's going to execute some, some shell commands. Maybe it will research the internet. So if you have internet web capabilities within your, um, agentic environment, that's part of the harness. So just exclude when you're using something like Claude, exclude the AI model. Anything else is part of the harness And the harness can-- You, you can study each component in isolation, like how the lubing work, how the context engineering work, how the memory works, and collectively now you have a very stronger understanding of how you build a harness.

Ammar Alim: And when we s-- we, we're building this, we would-- we did not even know it was a harness because the term was not out yet. [00:32:00] Yeah. But eventually, the term came out, and one of the engineers on our team said, "Hey, we built a harness." Um, this is technically how it happened. So once you're done here with, uh, we got to this agentic step.

Ammar Alim: The agent and the harness environment is supposed to take all the instructions from the previous step and do some rules generation. And you need a different model that does, you know, act as a judge. Because I don't-- don't use one model when you do these kind of things. You need multiple different models.

Ammar Alim: They are trained differently. They bring a different perspective. S-simply put, every model wears a different lens, and it sees its environment from a different lens. And you need those kind of to complement and support each other. So you can have two judges, maybe the, the model the primary model that generates the rules is, [00:33:00] is Anthropic's model and then OpenAI kinda the, be a judge.

Ammar Alim: And you can, include another one. It doesn't matter. Like y- the more... take it as much as, as far as possible and then reduce to find that just balance. You, you don't need too much. You have to try it for the solution you're building. It's not, it's not something... A-and as models progress and improve, you might not need multiple models.

Ammar Alim: But for now, I think I highly suggest having a judge that test what's the primary model did the work. Like it scores the work. You have a scoring rubric. You, uh, what's the criterias? What's the requirements? The judge need to know as much information as possible to be able to score that model, and then the judge must produce feedback That feedback goes back to the model again if there's anything that needs to be corrected.

Ashish Rajan: Yeah.

Ammar Alim: If any- if, if, if it's perfect, we need to save that in the memory. We need to say, "Well done. [00:34:00] Good job. You did this really well." Like reinforcement actually works here as well.

Ashish Rajan: Yeah.

Ammar Alim: We save all the good things that the model did in the past, so it keeps doing more of the good stuff. We also save the bad things, so it ke- it avoid that.

Ammar Alim: So this system, the more we generate, the more it gets better. That's why one of the things we decided, we will generate OWAF rule even if the CVE doesn't impact our environment. Oh. Because we would like the lessons. Yes. It's cheap.

Ashish Rajan: Oh, actually, you're right, because to your point, if for whatever reason in the future, if you did have an application which was impacted by the vulnerability, there's already an awareness of it in the system, so you're almost preempting it as well.

Ashish Rajan: So it's not just the fact that to your point it's cheap, but also the fact that, uh, you're almost preempting, hey, if for whatever reason a new application comes out tomorrow by-- that some new business unit releases, and somehow this CVE that we're today we're thinking is not relevant may be relevant in the future, that can get picked up now and to your point, because the, the, at the [00:35:00] moment, the overall compute is a bit more cheaper to do this.

Ashish Rajan: Uh, that may not be a bad idea. So don't just ignore all CVEs, but be able to, if you have that model for One model to create the rule, another to judge whether this is a good one or not, and having that reinforcement, uh, loop, you're able to preparing not for just for today, but for the future ones as well.

Ashish Rajan: Did, did I get that right?

Ammar Alim: Abs- absolutely. Absolutely. And, uh, the more, uh, s- rules you generate, the more opportunity for the system to fail and succeed, and that's training. Not training in the classic ML, um, way, but you train-- you actually can train your agent by repetition. The agent creates something bad, you save that information, but you need more repetition.

Ammar Alim: And more repetition means don't ignore a CVE, even if it's not relevant to you. Do it anyway, learn from it, store it, and it will come handy in the future, uh, because you, you know [00:36:00] your agent is getting better.

Ashish Rajan: Yeah. I think you and I were talking about this before we started the recording, but we can definitely have like a very small MVP version of this harness on the AI Security Lab thing.

Ashish Rajan: And we'll probably, I'll probably put those links in the show notes as well for you to come and share that, 'cause it's very hard to-- To your point, I almost feel I'm imagining, as you were sharing it, I'm imagining in my mind there's memories, there's model. You're almost like an architecture diagram being built in my mind as, uh, as you were explaining it.

Ashish Rajan: So it'll be pretty cool to kind of visually see it for people who, whoever joined that. I'm curious for one more thing here. In terms of, we spoke about the existing vulnerabilities, we spoke about the newer vulnerabilities how-- and we also spoke about the fact that when you deploy the, the staged releases that you guys are doing to have these across from, hey, dev environment, test environment, production, and then the li- loop continues, the reinforcement learning happens.

Ashish Rajan: Do I need to be technical for this? 'Cause I, I do wanna, I'm, I'm trying to simplify this for a lot of people, and I think my hope is if we say it enough [00:37:00] times, it would resonate with people. Were you like a AI programmer, ML experienced person before you started working on this?

Ammar Alim: No, I was not an ML you know, the, the entire ML education I have is self-knowledge, that books that I've read, like nothing, nothing-- I didn't go to school for it or anything.

Ammar Alim: Um, I

Ashish Rajan: just- No academic, none of that, like, none of that

Ammar Alim: stuff, right? No, I didn't, I didn't build a model, um- ... for

Ashish Rajan: Yeah.

Ammar Alim: Um, I, I know how they, they work and how they get built. That's pretty much it. Yeah. Um, there was no AI books in the... I know you've released one in the- I, yeah ... uh, security, but There's no AI books when I s- we started looking into this.

Ammar Alim: We haven't read one, to be honest. Yeah. We just read the vendor documentation- Mm-hmm ... how these things work. People put stuff on X and LinkedIn. We started exploring. So you have to, you have to start. This is the only way I think you can learn about this.

Ashish Rajan: Yeah. And I, I guess I-- [00:38:00] obviously, AI secure engineering was written because to exactly what you said, there was not many people writing about books apart from...

Ashish Rajan: There was no official resource, let's just say this. OWASP Top 10. Actually, yeah. Does the OWASP Top 10 kind of become automated here? I know AWS WAF had a package. All the other third-party WAFs have a package as well, but they then, they ended up having like an agentic AI top 10 as well. Uh, is that relevant or is that...

Ashish Rajan: 'Cause, uh, obviously, there's a whole class of vulnerability that AI has opened up. With this ecosystem that you guys created with the harness and the j- LLM as a judge and reinforcement learning that you guys are doing is that smart enough to have, hey, we have a, I don't know, CSA comes up with the top 10.

Ashish Rajan: You just give it that top 10, and it understands it, builds it, and not just a CVE part, because CVE is like a collection that just keeps coming. But there's also this industry standards that people like to [00:39:00] see that when I'm going for my audit, say SOC 2 Type 2 or whatever other audit that I may, I have to do.

Ashish Rajan: Uh, one of the s-selling points for AWS WAF used to be that, hey, we have a package, which if you have that package, you can tick the box saying that the AWS WAF looks at O-OWASP top 10. Is it possible to, and I-- my assumption his answer is yes, but I'm curious to know from you, the whenever a new industry standard comes up as well for the AI ecosystem, you are able to easily add that in, and the pipeline still caters for not just CVEs, but any form because it's just English.

Ashish Rajan: It's not really like, hey, SQL injection being typed into it as a regex.

Ammar Alim: Yeah. Yeah, let me, let me simplify this for s- your, you know, n- audience who's maybe new to, to appsec. So there is those OWASP top 10 is the top 10 vulnerabilities that, uh, the industry has deemed problematic for the last maybe ten years.

Ammar Alim: They get updated frequently, but are the [00:40:00] top 10 problems that an application can s- uh, be vulnerable to. So those are known things. We all need to defend against them. Some of them are in the authentication s- area, some of them are like SQL injection, XSS, cross, cross-site scripting. Mm-hmm. Like it, it depends.

Ammar Alim: But those things, you ideally like to handle them by following secure code and best practices. But of course, the entire security industry exists- Mm-hmm ... because we don't do that.

Ashish Rajan: Yeah.

Ammar Alim: Um, in the absence of that, you need a mitigating control, which is using WAF rules to mitigate those concerns.

Ammar Alim: And yes, to answer Ashish's question, yes, you can use this system to, to do that, and you can actually make it more relevant and specific. The good news is, um, the only time I'm gonna give shout out to our vendors is here. I'm not being paid by them or anything, but [00:41:00] they do provide you with this because this is a known thing, it's applicable to, to the entire industry.

Ammar Alim: Yeah. And they also have a ton of visibility into internet traffic, so they reliably provide this for you. You can still use it, use your, uh, AI pipeline to Further customize it to make it more relevant to you. But I would say it's a point of diminishing returns, meaning I may not suggest you spend a ton of time here.

Ammar Alim: I suggest you spend a ton of time understanding how CVEs come about and what you can quickly do to mitigate and sup- reduce that risk for your business. This is the first step. Then you move on to the nice drafts, which is, how can I make OWASP Top 10 rules for my system that are very customized specific to myself?

Ashish Rajan: I love how you are sharing this as well because I think you hit the nail on the head. I made a very broad assumption that cloud security people understand appsec, and I'm sure some people do. It's not everyone [00:42:00]understands it. Because you've had that, uh, experience of co- transitioning from cloudsec to appsec to now prodsec.

Ashish Rajan: So you, you definitely had the unique experience for that. I'm curious, in terms of we've spoken about the applying, applying this on a scale, the skill set of it as well. We've also mentioned, uh, the fact that this is very much possible for AWS WAF. But do you see these kind of systems being extended to, I don't know, S3 bucket or like, you know, the wider AWS ecosystem, which may not, may or may not have these kind of ruling system, but we have the network firewalls that are be-- used by people.

Ashish Rajan: 'Cause I imagine what I'm hearing so far is, uh, is the fact that you've almost created a framework for what could be more than just a WAF, and which is why you mentioned, hey, it's not just AWS WAF, any vendor WAF. And I'm going maybe actually to what you're also saying, hey, this is just one framework for one category of security product, but you can build this for other categories of security in your organization as well.

Ammar Alim: Yeah. [00:43:00] This is relevant to any part of security, especially the tedious parts.

Ashish Rajan: Mm-hmm.

Ammar Alim: Which is there is no like if you are a SOC analyst in detection engineering or in these areas or in appsec, there is an opportunity, or cloudsec, there is an opportunity because there is a ton of the ability to s- synthesize multiple knowledge across information across from different platforms is something that it overwhelms us.

Ammar Alim: We cannot do it quickly. Again, the, the currency is speed, and if you have an incident you are investigating and you typically have a routine. You have tools you use, they probably very custom-made for your team. You guys have a playbook, maybe it's automated in Splunk. Maybe you have an orchestration framework to respond to incidents.

Ammar Alim: Maybe you have a Slack channel where you kinda create that Slack channel, and [00:44:00] you invite the right people. And you need to go to GitHub to maybe read the playbook, and you need to go here to understand more about the vulnerability. You need to go to your threat intel your LVD for the information about the database.

Ammar Alim: There is a ton of things you have to do, and AI loves a ton of things. It loves text, right? Yeah. It will read all of that. It will, with a little bit of API access, it may create the Slack channel. It may suggest who should be invited to that Slack channel, right? Because you have memory, and you tell it, "For this product, the product owner is this person."

Ammar Alim: Maybe you have a s-- mini memory file. It doesn't have to be fancy. Uh, you don't have to use memlang or anything specific. Just a text file will, will be enough. So I highly suggest finding Tedious work that takes a lot of time, and see what, which part you can automate it with AI. Just start small. Just do 10% automation.

Ammar Alim: Don't try [00:45:00] to think deeply about, "Hey, how does it apply?" Start by finding the tedious work that you do every day, and start there. That's it. It doesn't take a geni- Like, every day there's a things that you do or things that you do when there is an incident-

Ashish Rajan: Yeah ...

Ammar Alim: some of them you do not wanna do.

Ammar Alim: Some of them are just tedious. Some of them are just, you're not gonna be good at them. So this is where you introduce AI. And I guarantee you, if you inventory your day, your week, there's a ton of opportunities.

Ashish Rajan: I love this, man. And, uh, there's so much to unpack here and continue the conversation as well, but I think we'll leave that for the AI Security Lab, uh, live session we do for people to maybe ask some live questions too.

Ashish Rajan: Where can people learn more about the work you are doing? I think I'm gonna link the Forward CloudSec talk that you did as well, which I think had some really good elements in there too. Uh, where can people connect with you, learn more about the work you're doing, man?

Ammar Alim: Yeah. The only social media I am actively using is LinkedIn, [00:46:00] so Ammar Alim, uh, is me, and if, uh, you know, not a lot of people have this name, and it's easy to see that I work in security.

Ammar Alim: I also write sometimes on Medium. I have a personal blog. But LinkedIn is probably the place to, to go to.

Ashish Rajan: Awesome. And I'll, I'll put your, uh, LinkedIn link in there as well. But I appreciate you spending the time with us, man. Thank you so much for doing this.

Ammar Alim: Yeah. Absolutely. Thanks, uh, for having me, Ashish.

Ashish Rajan: Thanks, everyone. See you next one. Peace. Thank you for listening or watching this episode of Cloud Security Podcast. This was brought to you by techriot.io. If you are enjoying episodes on cloud security, you can find more episodes like these on cloudsecuritypodcast.tv, our website, or on social media platforms like YouTube, LinkedIn, and Apple, Spotify.

Ashish Rajan: In case you are interested in learning about AI security as well, do check out our sister podcast called AI Security Podcast, which is available on YouTube, LinkedIn, Spotify, Apple as well, where we talk to other CISOs and practitioners about what's the latest in the world of AI security. Finally, if you are after newsletter, it just gives you top [00:47:00] news and insight from all the experts we talk to at Cloud Security Podcast.

Ashish Rajan: You can check that out on cloudsecuritynewsletter.com. I'll see you next episode.

Peace.

‍