Derek Abdine : [00:00:00] The way the stack was designed for the last thirty years won't work the same way for the next thirty. It can't, because the problem has fundamentally changed, so the solution has to fundamentally change. It's downloaded a hundred million times a month or s- And the developer had their publishing token to the NPM repository stolen.

Derek Abdine : They took the library, and they inject a remote access toolkit in it. Long, long story short, you get compromised if you pull this library. And you feel shame every time you see the reports coming out that you're not making enough progress. Scan the network in a power plant, guess what happens? You knock over all the PLCs and valves in the compression of time.

Derek Abdine : But I think it was like two hundred and seventy days in twenty eighteen, and now it's like hours or minutes. Yeah. Model that is really good at looking at source code, analyzing it, and identifying holes. And when paired with tools, can obviously construct a path to actually utilize those tools to break into networks.

Ashish Rajan: Today, I wanna talk about remediation. Yes. Before you roll your eyes, I know a lot of people have been skeptical about remediation, but I had a great conversation with Derick from Furl.ai, and we have been talking about how remediation can actually be challenged with AI. And perhaps we [00:01:00] have a better approach for vulnerability management with AI than we had before.

Ashish Rajan: Whether it's vulnerabilities that you may have considered as unpatched or vulnerabilities that have been sitting in your backlog for a long time, there is a way that AI could actually be a force driver to reduce that backlog and address vulnerability management at scale. If vulnerability management has been top of mind for you, this is a great episode for you to tune into.

Ashish Rajan: And as always, if you are enjoying the episodes of podcast and have been here for a second or third time and should continue to enjoy the episodes, I would really appreciate if you hit the follow, subscribe button, whichever podcast platform you listen to on. We are on YouTube, LinkedIn, Spotify, and Apple, or wherever you consume your podcast from.

Ashish Rajan: I hope you enjoy this episode with Derick, and I'll talk to you soon. Hello, and welcome to another episode of the podcast. Today I've got Derek with me. Hey man, thanks for coming on the show.

Derek Abdine: Yeah, thanks for having me.

Ashish Rajan: Uh, maybe to kick things off, if you could share a bit about yourself, your professional background as well, man.

Derek Abdine: Yeah. So I have over 20 years professionally, and then if you look back even, you know, kind of [00:02:00] semi-professionally, that's about 30 years going back to my me- my teens, just kind of like in a dark room messing with stuff. Um, but I've been at Rapid7 doing, uh, kind of vulnerability management through threat intelligence and data collection.

Derek Abdine: A CTO at Census for a year, and then decided I just want to really get into remediation, started Furl.

Ashish Rajan: Interes- Well, and and you did pick the hardest of them all.

Derek Abdine: Yeah. If you're gonna go big... If you're gonna go, go big or go home kind of deal.

Ashish Rajan: Yeah, fair. May- so maybe let me start there then. Obviously people feel that they're very good at finding, which is like the quote, unquote "detection piece."

Ashish Rajan: Yep. And fixing of it is usually a bit dicey because there are non-technical elements, there are technical elements, depe- there's just so much complexity around it. What has changed now that may have made you go down, obviously this, down, down the path of going into remediation?

Derek Abdine: Yeah. So I think, you know, if you look at technology in the last 10, 20 years, it's boast- been mostly like if/then, you know-

Derek Abdine: type statements, logical based approaches to everything. [00:03:00] And the fact is that remediation just isn't that. Every time you go to fix something in an environment, there's nuances. Every endpoint's different. Um, they have different disk space, they have different services running, different configuration state.

Derek Abdine: And so reasoning around those environments is not something that's been scalable. So we've needed to really figure out a solution that can have that, and now models have come along that allow us to do that.

Ashish Rajan: Mm. And so does it a... Does that mean the existing model that we've had is it broken with AI as well?

Derek Abdine: Uh, what do you mean model? What do you mean?

Ashish Rajan: Uh, so like, so to your point about that the model is allo- that having an AI model has allo- allowed us to be more, responsive and re- uh, and remediation is actually more possible. W- with the advent or with the introduction of AI, the existing approach we've had with remediation, detection, does that- Not apply anymore?

Derek Abdine: It still applies in some ways, but, you know, if you're gonna go apply a fix to a system and you run a script it's gonna fix something, or it's not gonna fix something. And if it doesn't fix something, how do you know [00:04:00] when the failure mode happens? How do you know if it actually applied successfully?

Derek Abdine: Uh, and so, you know, if you really wanna fix it, you gotta really step back, and before you actually go apply the fix, reason about what the environment looks like. And then once you have that reasoning and really understand the context around it, then go apply the fix, and then look at the opposite end of that.

Derek Abdine: Now that the fix has been applied, what does the system look like? What's the state? And, um, does that fix actually stick, or is it something I need to go back up and retake another look at? So-

Ashish Rajan: Almost like a lifecycle management of remediation.

Derek Abdine: Yeah, pretty much. Yeah, actually, as you said it, you just de- debugged my mind, but yes, exactly right.

Ashish Rajan: Yeah. That's what it is. So how would you define remediation lifecycle? 'Cause I don't even think people know. I mean, apart from me just coming it out of, coming out of that word out of my ass. Yeah. What, what, how would you define remediation lifecycle management? Like, as in, what does it look like in your mind for this particular space?

Derek Abdine: Specifically talking about vulnerability management. Yeah. Let's, let's take a look at that industry for, you know, what it's existed for, you know, over 20-ish years plus, if you look at IBM ISS back in the mid-'90s.

Ashish Rajan: [00:05:00] Mm-hmm.

Derek Abdine: A long time. It's been around for a long time. And if you look at the state-of-the-art, what those, those products perform is, you know, you do a scan, you create a remediation plan report, and then what you do is you get a bunch of different items in that plan.

Derek Abdine: There are, s- 100 solutions that fix 1,000 different vulnerabilities, and you're like, "Okay, well, all these 100 solutions don't tell me anything about operationalizing these fixes." And so there's a lot of manual effort that is involved in addressing all those problems. And so now You have to figure out, is there a script that needs to be fixed and applied to a system?

Derek Abdine: Do I have to go configure my patch management tool? Is this a vulnerability, or is this a configuration weakness on the endpoint? And now you have to go have people go prioritize, ticket, and understand all that stuff, which just becomes a, a nightmare. So as a result, what we do is it takes 30, 60, 90 days to actually fix anything, if we can.

Derek Abdine: Um, what more or le- more than likely happens is usually we just kind of, like, write exceptions. We can fix what we can by prioritizing.

Ashish Rajan: And to your point, the [00:06:00] 30, 60, 90 day becomes just a, uh, uh, I guess for lack of a better word becomes a backlog at the end. Oh, yeah. Yes. And then now you just have backlogs of just- It-

Ashish Rajan: unremediated issues that you hold onto.

Derek Abdine: Yeah, and everybody has a backlog, and it's, it existed for, even before models came along. Yeah. I mean, we haven't even talked about those yet, but they're compressing time and adding more to the backlog. But mo- backlogs were already very large before.

Derek Abdine: They, they haven't really... They're not reducing. They're, they're, they're, they at the best case have gotten stagnant. So it's really about how do you start to reduce that? 'Cause at, at the end of the day, you're, you know, CISO's gonna be responsible for reporting down what the, uh, burndown rate on risk looks like.

Derek Abdine: And so if you don't have the... If you have solutions that are basically flatlining that reduction, then the question is, like, what do we have to do to actually reduce that more? Mm-hmm. And again, what we turn to is just kind of, like, ignoring the risk more than often is, like, just accepting it.

Ashish Rajan: Yeah, and I guess you bring it back to vulnerability management as well.

Ashish Rajan: Obviously most CISOs out there who have looked at as vulnerability management, they're being sold the idea of AI and AI [00:07:00] security. Yep. Obviously people have to prioritize depending on the budget and where the attention is at the moment. With remediation, uh, now that it's a possibility to actually do that, almost like if I were to use a, the cloud vulnerability management example or a data secure- data center vulnerability management scenario The one thing you mentioned which, uh, which stuck with me just now is the difference between a configuration versus a vulnerability.

Derek Abdine: Yeah.

Ashish Rajan: That has not been... Like, usually, a lot of people just go down, "Oh, it's just vulnerability management, so I look at just pa- apply the Tuesday patch, and we're good to go- Yeah ... until you get the blue screen of death, but hey, that's okay. Mm-hmm. That's another problem. But in t- is there an easier one to start with in remediation in terms of between the two, or do you believe it's possible to start with both at the same time?

Derek Abdine: Applications certainly, and that's where we've seen patch management already be successful. Um, but it's not the end-all be-all, and applications are also them hard- themselves hard to fix 'cause you might have an application that has data, like a MySQL server, that you can't just [00:08:00] upgrade it and hope that these, the server upgrades.

Derek Abdine: So it's not as simple as just fixing that. And if I could actually back up for a second, one of the things around configuration and vulnerability is that a lot of the vendors that have existed in the market, uh, they come from a a spa- a time and space where CVE was in its infancy. CWE didn't exist yet, and so vulnerability management to a lot of these products means that configuration issues, um, s- like things like world-readable files on a file system, certificate problems, and CVEs- Mm-hmm

Derek Abdine: are all kinda mixed up into this idea of what vulnerability is, and they all report vulnerabilities being, a, a kind of, like, super set of all those items. But the reality is the, the industry's evolved from that, and so we... it makes sense to start to separate a lot of those items out and deal with them as they are.

Derek Abdine: So deal with CVEs as CVEs. Deal with configuration problems as configuration problems. And, like, create maybe, like, secure baselines that you wanna deploy to your environment that, that define [00:09:00] what the, end goal of your configuration state should be when you have those vulnerabilities coming in.

Ashish Rajan: Oh, interesting, 'cause... So, uh, so to your point- A lot of people already have vulnerability management software today.

Ashish Rajan: I'm sure the vulnerability management provider are also telling them that, "Hey, we've got AI capability."

Ashish Rajan: Whatever that capability is, I'll let them be the judge of that. For people who are leading vulnerability management in their, in their organization one of the things obviously they're trying to do is that many are at Black Hat at, at the moment trying to figure out, "Hey, how do I uplift this?

Ashish Rajan: How do I make sure that I'm doing the best for my vulnerability management vertical," if I wanna use that example. What are you seeing as, uh, the right way to approach... is, is there a foundational thing people should already have before they even consider the, "Hey, I'm good to now start talking about remediation," or especially auto remediation?

Ashish Rajan: 'Cause I, I feel to what you said, there's, there's already a heaps of backlog.

Derek Abdine: Yeah.

Ashish Rajan: Right? And I'm sure it's a mix of configuration vulnerability and something else that is just not even defined yet, and now we've added AI on top [00:10:00] of it as well, which I don't know how many people actually throw AI vulnerability if they find one to vulnerability management in the first place.

Ashish Rajan: Yeah. But the point being, um, now that we understand, okay, there's a mixed bag of what you have in terms of vulnerability what do you think, uh, if people who are with a vulnerability management software today should be the first few steps for them to start, maybe start building that capability within the organization for remediation?

Ashish Rajan: Whether is this like a, "Hey, do I need a process change or do I need to enable more AI into remediation?" And obviously not everyone would be a full AI customer- Yeah ... but for people who are looking at this from a, "I need to get to that point where I can be a ful- full AI customer," what, what would that baseline be?

Derek Abdine: I think people that have the problem really really know that they have the problem already. Mm-hmm. So they've already used remediation products. They've probably used some type of patch management tool. They're probably doing a lot of scripts that

Derek Abdine: are deploying, or they're using exceptions, and they're finding that that process isn't working anymore.

Derek Abdine: Mm-hmm. They're finding that [00:11:00] they're being asked to remove those exceptions, and they have no clear answer on what they do when those exceptions get removed. And so now there's this pressure point building where they have to figure out a solution for it. And so our customers that end up being kind of like really ready for that are experiencing that visceral pain and need some help understanding how to navigate through it.

Derek Abdine: And so that's where we've had the most successful conversations is, is starting with that. Just do you have the basic need of remediation? Are you feeling it? 'Cause some environments we talk to too, it's we don't have remediation, like, 'cause we don't have vulnerability management. And what we do is we focus on basically inbound events.

Derek Abdine: I don't... Like, that's an opinion. Like, that's a great perspective. You can keep your own perspective if you want. I think I think that organizations do need to be proactive about removing risk in their environment because ultimately, like, they are holes that you have to plug, and you don't just wait until post-exploitation events happen and then responding to those.

Derek Abdine: You gotta be proactive. You gotta bo- have both angles.

Ashish Rajan: Yeah, 'cause I think it kinda ties back to the whole Mythos conversation, and I didn't even think we would [00:12:00] have gone down that path, but I think it's like we might because to what you said, what the vulnerability management piece, a lot of this is what being, at least being claimed to be exploitable as long as it is exposed on the internet.

Ashish Rajan: And to what you said, if you just don't have a vulnerability management program to get rid of all that backlog of potential vulnerabilities- Yeah ... some of which, which we own the internet, it does not, at least in my mind, doesn't make sense. But I guess, are you finding people they are preparing themselves, at least the ones who are ready with remediation, are they the ones who are preparing themselves for that?

Ashish Rajan: What is that world of Mythos? And I say that-

Derek Abdine: Definitely. Yeah. It- So, so a lot of the folks that we've worked with are They've they've found, they've read the M-Mythos, Mythos articles. Yeah. Um, everyone has a different pronunciation. The ones with tomato.

Ashish Rajan: Yeah.

Derek Abdine: Um, and, you know, there's a lot of...

Derek Abdine: Look, it's an-- Models are, are really good now because if I can just segway for a little bit, they've been trained on a lot of engineering expertise. And I, myself, I started off just as a nerd, like wanting to [00:13:00] engineer stuff, and it happens to be, and Anthropic even wrote about this, is that if you're really good at engineering, it, it happens that you actually end up generalizing to be really good at security.

Ashish Rajan: Yeah.

Derek Abdine: And so the natural question people have when they see Mythos is like, how much of this problem do I have now? And so Mythos by itself is a model that is really good at looking at source code, analyzing it, and identifying holes, and when paired with tools, can obviously, you know, construct a path to actually utilize those tools to break into networks.

Ashish Rajan: Yeah.

Derek Abdine: We've seen both those cases that aren't surprising. There's two different places that we tend to see Mythos come up, though. The first one is source code-based vulnerabilities. So can you actually find novel vulnerabilities in latent source code that's already existing out in, libraries that are published into corporate environments?

Derek Abdine: And the second one, um, that I think is not necessarily as well understood is, that Mythos itself is not a solution, and the model itself is not a solution to, uh, what we already have in our environments. [00:14:00] Because what we have in our environments are things that are compiled, packaged, and deployed by, by code, and then now they have to be maintained.

Derek Abdine: So Mythos will tell us that those, those things have holes, but now we have to go clean it up.

Ashish Rajan: Yeah.

Derek Abdine: And so a lot of the misunderstanding about where that model is, is on the, you know, it is equating the first half, which is the source code side, to the second half, which is deployment side.

Ashish Rajan: Which is the missing gap at the moment that you still have to remediate

Ashish Rajan: it's a missing

Derek Abdine: gap. Exactly right.

Ashish Rajan: Yeah. Right, so it helps you find the vulnerabilities, but not maybe patch them?

Derek Abdine: Not patch them. I mean, it can, it, uh, can help patch the source code- Yeah ... but it can't take that source code that's compiled and now deploy it all to your systems. Mm. So it's like you've been shotgunning all that code across all your systems.

Derek Abdine: Yeah. It's not code at that point, it's compiled artifacts most likely. Yeah. And you now have to go back and figure out how do you change all those systems to in- in, you know, have the, the plugged holes in those applications that were found by those models.

Ashish Rajan: I think it's, uh, probably a good segway into the Axios, uh, incident as well.

Ashish Rajan: Um, maybe- Mm-hmm ... A, if you could tell about, talk about that, and how does remediation kind [00:15:00] of play a role there as well?

Derek Abdine: Yeah, so the Axios issue is, you know, w- the developer, this is a library that's downloaded 100 million times a month or something like that, and the developer had their publishing token to the NPM repository stolen.

Derek Abdine: And what the, uh, threat actor did with that publishing token is they took the library, and they injected a remote access toolkit in it, and they essentially allowed the remote access toolkit to be installed with the setup scripts. Long story short, you get compromised if you pull this library.

Derek Abdine: And any developer who's working on web front-end based apps or back-end apps that are based on Node.js would have this library on their system. And so, you know, the big thing is, like, I'm a, I'm, I have developers in my company. How much of this problem do I have, and what do I do about it? Um, so essentially, like, what's the blast radius?

Ashish Rajan: Yeah.

Derek Abdine: And so, you know, when it comes to remediation- I think the way we've had to deal with this issue in the past is we've had to go and get a team together, it's a little war room team, and then get them in a Teams group or a Slack channel or in person, and then we have to [00:16:00] go look at, like, 50 tools to go figure out what's impacted, and then after the 50 tools, we have to create a bunch of tickets and assign them a bunch of people, and it's a big mess.

Derek Abdine: Everyone's stressed out. They're working long weekends and late hours, and I think we can do better. Um, I think we can go faster.

Derek Abdine: And so, you know, where remediation can help out is, one, the, kind of autonomous response aspect. Yeah. So identifying where the problem exists by forensically analyzing those systems, and the second one is creating the remediation artifacts to go clean it up.

Ashish Rajan: Interesting, and do you... Almost, I, I love how you staged them as, as well. Uh, the reason I wanted to talk about this is also because a lot of people with AI capability, especially engineering capability, um, would think that, "Hey, maybe I can use an LLM to do this myself. I, I can have a script library, I can build a harness."

Ashish Rajan: And how realistic is that, and how unrealistic is that as well? Uh, and clearly you're building a company around it as well, so I'm just curious as to what do you see as the difference between, like, say, can you... Can I do this with Claude Code in my laptop, [00:17:00] in my whatever the team that I'm part of, or do I...

Ashish Rajan: I don't have to. Do I need the full-fledged the, the full shebang? Yeah.

Derek Abdine: I think in the age of AI products, anyone can build anything, and that's the great thing about AI, vibe coding. The, now, now let's take that from that statement and decompose it. Okay, so you built the thing. Now you have to maintain it.

Derek Abdine: This is a classic SaaS problem. You've, you built a piece of software, and now it goes down. There's a new feature that needs to be added, and this is, this is also a commodity, right? Yeah. Because you can get other folks that can find and build those features and maintain the software application stack.

Derek Abdine: The third one, though, is that there are lots of edge cases in remediation. Like understanding, uh, if you go and fix something, how do you actually understand that the, the fix you're going to deploy will actually deploy successfully? Well, you need to check the ban- disk space and things like that before you deploy it.

Derek Abdine: You need to determine if you're on an Ubuntu system and the, the package is pinned, which is called being held on, if you're using APT- Things like that are domain expertise- Yeah ... is kind of what I'm saying is [00:18:00] not, is something that is useful to, to pool the knowledge of into a central place.

Ashish Rajan: Yeah.

Derek Abdine: And so if you're, if you're re-creating this application yourself, you're gonna start to have to reestablish that domain expertise, and you're gonna have a narrow view on it than the kind of combination of a view of multiple different, um, sources of information that can be consolidated into a single platform.

Derek Abdine: So that's the next step. And the third is the learning that you get by doing the iterations across many environments that are like yours and not like yours, and then using that learning to self-improve the system. And so, you can get that within your organization, but you're gonna run into things eventually that you've never actually encountered before, and that's where having a partner that understands and has visibility into that and can generalize into a model that can help you before you even run into those situations in the future is important.

Ashish Rajan: I think, as you said, the, the first thing I came to, uh, was as we... uh, at least people who tried doing remediation, one of the reasons why a lot of us would hit the wall was the whole ownership. Who do I attribute this quote unquote- ... thing to [00:19:00] solve? Is that not the case anymore? 'Cause that used to be like one of the...

Ashish Rajan: Like, how, how do I find out who's this gonna-- who made the change or push in GitHub and whatever other solution you're looking for. Is that easier now with- that we have AI capabilities?

Derek Abdine: I think the theoretical, um, versus there's a theoretical and practical. Right. I think the theoretical is self-healing systems is the future.

Derek Abdine: Um, like less pointing blame, um, at people and more like-- or like accountability towards people and more like get systems to fix themselves, right? Like that, that's a utopian vision. I'm not saying that anyone can even do that today- Yeah ... nor would I believe that anyone should- Mm-hmm ... even say anything remotely similar to that.

Derek Abdine: But if you bring that down as the, as the kinda theoretical approach that you wanna get to, the step back from that is, that ownership is still important today-

...

Derek Abdine: For many different aspects and it is an interplay between technology and culture. So what I mean by that is in some organizations some of the clients we work with, for example, uh, there are developers in those companies, and the centralized IT [00:20:00] and management, like, team might have, you know, some sets of systems they're accountable for, and they can run automated remediation on them.

Derek Abdine: But they defer the decision because of, many other reasons, I won't get into those, to other people, meaning the developers themselves, to fix system, uh, issues on their systems. That doesn't mean that auto-remediation can't occur-

Ashish Rajan: Yeah ...

Derek Abdine: but they wanna defer the decision to what gets auto-remediated on those systems to other owners, and that's fine, too.

Ashish Rajan: Yeah.

Derek Abdine: I think if I was to wrap up that answer, I would say, like, there's a old pi- paper, I think from IBM, saying a computer, something along the lines of computer can never be held accountable should the re- making a management decision. I'm probably butchering that statement. Yeah. But, um, but it still, that still applies today.

Derek Abdine: We don't, we don't have, we haven't built enough trust in these models yet to let them just go full tilt into our systems, so I think human accountability is still important.

Ashish Rajan: Oh, yeah. Um, another thing that I came across in all the convers- conversations that I've had is that usually there's the ownership side, and then there is the, to what you said about the vulnerability patching.

Derek Abdine: Yep.

Ashish Rajan: A lot of times, there is no patch, [00:21:00] and let's just say, call it classified as "unpatched," quote-unquote. How do you see the customers or people you end up working with, how are they managing that part of remediation? 'Cause the, it's not... If the goal is to reduce the backlog to remove and do auto-remediation, what do you see as a approach that some of your clients or customers are taking that maybe others can learn from, or if you have recommendations there as well?

Derek Abdine: Yeah there's a couple answers to that. I think the first one are OT and healthcare-like environments where you have a packaged unit of an application, like in an OT environment, a power plant. I was at a power plant in 2008 tuning a network-based scanner.

Ashish Rajan: All right.

Derek Abdine: And because, like, if you scan the network in a power plant, guess what happens?

Derek Abdine: You knock over all the PLCs and valves and stuff that-

Ashish Rajan: Oh, really?

Derek Abdine: Yeah, yeah. It's, it's, it's because- Oh, because

Ashish Rajan: that old.

Derek Abdine: The- they're old. They, they, you know, these large vendors, Siemens, Honeywell, every- everybody else, they come in and they'll develop a package, like, which is, like, designed for the plant of all these little controllers, data [00:22:00] loggers, historians, things like that, that all communicate to each other to build the plant, right?

Derek Abdine: Yeah. So it's one application built of many different software stacks and hardware pieces, like that. So in those types of environments you can't really just remediate stuff. And, and in case in point, you, when we did the scan, this was in 2008 uh, when this happened, it was running a tw- a Windows 2000 machine with plenty of vulnerabilities on it.

Derek Abdine: Yeah,

Ashish Rajan: yeah.

Derek Abdine: But, um, you know, the, the idea there isn't that you're just gonna go patch, because what you need to wait, and those vendors need to do, is they need to wait until Siemens or Honeywell comes in to go patch that as an ecosystem, because you can't just upgrade the operating system. That might break the PLC anyways.

Derek Abdine: And so the, the idea there is in those environments, and healthcare is very similar, too, with medical devices, you need to do your, like, more containment and mitigation. So, like, how do you make sure that you are maybe creating a walled garden around that environment? I read some recent notes actually this week around, I think it was the FBI recommending that people don't put things on the internet for control systems for some of these environments, and I hope no one's doing that.

Derek Abdine: Because, um, like having been in those environments before- ... and done analysis to try to [00:23:00] help those f-folks out as well, it's, it is important to do, to use mitigation as a strategy as well. So that's the first one. Right. And then there's a second class, which is like zero-days and other supply chain attacks where it's very late-breaking and there's really no, no, no patch available or no fix.

Derek Abdine: Um, so again, in that case, mitigation is key. And, um, essentially, you know, isolating the device into, so it can't have an impact on the network. So- Yeah ... there are ways you can do that on a network level. There's ways you can do that on the endpoint level. There's many different strategies, so it really depends on the specific type of situation, but there's two kinda classic areas that we tend to see.

Ashish Rajan: And do you find that obviously big-people are still, still trying to build trust in how AI is used for most things- Mm-hmm ... including remediation because of hallucination being actually a thing still.

Ashish Rajan: How do you find people able to build that trust for remediation? Like, what if that unpatch is actually a, a hallucination, whether the whether it's genuinely a thing or not?

Ashish Rajan: What have you found is a good way to start building that trust with your customers for them to start trusting the output of AI as [00:24:00] well?

Derek Abdine: Yeah. I think I've u- I've been using models since 2022 when GPT came out, GPT-3. I think it was out a little bit before I started using it, and then ChatGPT came out in December of 2022, and then so on.

Derek Abdine: Today, you know, we have mul- all sorts of models that we have access to, opens, open-weight and not. And I think along that line, I've seen the kind of like, over-the-long-horizon play of where models came from, where they're going. In the early days, there was a lot of hallucination, and it was really bad, and they were really bad at answering things.

Derek Abdine: They'd give you an answer that was just confidently false, right? Yeah, yeah. And so I'd say today there's much less of that, primarily because it's not just about the model, it's about the context that you provide to the model.

Ashish Rajan: Yeah.

Derek Abdine: And so, I would say it's still-- I would not trust a model just to have full unfettered access to an environment, and I don't think anybody should be asking, uh, anyone else to fully trust a model.

Derek Abdine: Even, even if you're looking at output from Claude, right? You wanna fact check what it does. Yeah. Um, and I think it's important, but it's still automating a lot of the work that you would normally do. And a [00:25:00] lot of the time it'll get the answer right, but one or, you know, a third of it might be wrong or something, okay, go work with it.

Derek Abdine: But my-- I guess that's a long way of answering that. Models have cl- come a long way. They do really well when you give them a lot of good context. The best way to manage that is to add the right guardrails and humans in the loop to make sure the information coming out of these models is correct. Um, and, and people still know that.

Derek Abdine: People are still at the center of that.

Ashish Rajan: Yeah.

Derek Abdine: Where they really help is accelerating all the stuff in between.

Ashish Rajan: 'Cause, uh, 'cause the other leg to this also is that how should someone identify red flags when they get pitched about AI security? 'Cause especially in the remediation space, right? Yeah.

Ashish Rajan: And it, it's coming from a fact that a lot of people talk about the fact, and to what you said and what we started with in the beginning about if you walk on, if you walk on the BlackHat floor, you suddenly see there's a lot more new vendors in the space. And every- I'm sure everyone's taught solving different unique problems in their own way.

Ashish Rajan: I'm just curious in terms of how you find that, um, what's a good way to separate signal [00:26:00] from the noise? Especially, you've had this practitioner background in the past as well, and now you're trying to s- or, look at remediation, which the more people men-mention remediation, a lot of people are like, "Oh, I've tried that one."

Ashish Rajan: Yeah. "That didn't go well." Yeah. So what's a good way to identify the... Red flag's probably an extreme word, but just more separating the signal from the noise?

Derek Abdine: Yeah, again, it's that "Portlandia" episode where it's slap a bird on it. And you see this in RSA. W- I've seen it when I was coming into the event today, too, when I was traveling.

Derek Abdine: Um, like everything says AI, so how do you differentiate between what is actually utilizing AI in a, in a good way versus what is bolting it on?

Ashish Rajan: Yeah.

Derek Abdine: Good products being developed right now are looking at hard business workflows, and it could be in legal space, it could be in any space, right? I'm happen to be doing in IT and cybersecurity.

Derek Abdine: And what they do is they weave, they interweave those, that, those models into natural workflows that already existed within the business, and they're not just chat [00:27:00] interfaces. Mm. They, they are achieving some outcome that, uh, and they're doing it in a way that's woven into the way that we would normally handle that and normally handle that activity.

Derek Abdine: And so I would say, you know, look out for vendors who are just saying and throwing around the word AI. I would say, you know, ask them how that works into the outcomes that they're trying, they're pitching that you're trying to achieve. Eventually, I would just say, actually, like, step back and say, "What problem are they solving?"

Derek Abdine: And if their solution is AI, right, that's not the right solution. Our s- our solution is, like, continuous autonomous remediation. Here's how we use the models. This is where they come in. That's the way we talk about it because, that's the way that you have to really frame it towards, like, what problem are you solving and, like, what are you gonna do?

Derek Abdine: A lot, again, a lot of the vendors that we tend to see d- or other folks in the space are just, you know, they'll just talk about it in the abstract. They'll just say, "We have AI." And you're like, "Okay, great." Yeah. Like, that's cool.

Ashish Rajan: I have two of those. Yeah.

Derek Abdine: Exactly.

Ashish Rajan: Yeah, yeah. I

Derek Abdine: collect them like Pokémon.

Ashish Rajan: Yeah. Do... So I... And I think, uh, I appreciate you saying that also because a lot of people are trying to separate their signal from the [00:28:00] noise, and I think it's harder. Do, do you feel like the vulnerability management teams also need to change the way maybe they operate or they exist in organizations?

Ashish Rajan: 'Cause obviously people who are looking into maturing a program, now that you have AI capability and you can auto-remediation, the whole point of a vulnerability management team was to find, "Oh, how many patches are w- are we deploying this month?" What... And you track that ROI from that perspective.

Ashish Rajan: Yep. If you start doing remediation, what, what is the updated new ROI for this team, and what does that team look like?

Derek Abdine: Yeah. So I would say that we've done something the last 30 years the same way, and we got really, really used to doing it that way. We got really good at it. We created all these little industries and roles and everything around it And I think now we're in a different era.

Derek Abdine: And we need to rethink from first principles how we solve those problems. And so what I mean by that is the way the stack was designed for the last 30 years won't n- won't work the same way for the next 30. It can't, because the, the, the, the problem [00:29:00] has fundamentally changed, so the solution has to fundamentally change.

Derek Abdine: So that's, like, that's, like, a very, like, um, nebulous statement, so let me, let me describe a little bit what I mean, specifically on vulnerability management. Vulnerability management has historically been, and I know this 'cause I built a lot of it at previous companies I was at, has historically been you scan a network, you find vulnerabilities, you create a report you know, and that's basically what you do.

Derek Abdine: You don't-

Ashish Rajan: Literally the job.

Derek Abdine: That's all the job is. Yeah. And there's different roles that are defined within those jobs, and then eventually those turn into tickets that get into an IT team's plate, and now the security team is throwing things to IT's, land, and then a lot of things are missed in between.

Derek Abdine: It just, it's not fast enough. Yeah. To, to be fast in this age, w- you, you know, kind of looking at zero dayclock.com is a good site that has been put together to, to track the compression of time, but I think it was like 20- 270 days in 2018, and now it's, like, hours or minutes- Yeah ... that's being tracked. Yeah.

Derek Abdine: And it's like you don't have that luxury anymore to create a ticket and then have somebody in IT eventually figure it out, and they're looking at it and like, "Oh, it doesn't say anything. How am I supposed to do this?" And they talk to the security team, and now it's been a [00:30:00] week before that ticket was filed.

Derek Abdine: Like, we have to rethink how, um, how we can align ourselves better, and I think that comes down to culturally-

...

Derek Abdine: How security, the future team of security and IT look is gonna look way more blended, and I think we're already seeing that within some of our clients.

Ashish Rajan: All right.

Derek Abdine: Um, you know, larger clients, I think they are still there's ones that are kinda more future-facing, let's just put it that way to be nice about it, and then ones that are kinda more nimble.

Ashish Rajan: Okay.

Derek Abdine: And then there's some clients, I think, that are still in the old cultural way of having IT separate from security and so on, and some newer cl- some other clients that are now starting to blend those two i- items together. And I think ultimately, the clients that are starting to create those blended organizations where security and IT are kind of almost one function-

Ashish Rajan: Yeah

Derek Abdine: and they're able to seamlessly communicate information in real time to each other, are the ones that can stay on top of the brunt of what's coming up, um, not, not only in the response side, but also in the kind of vulnerability and proactive remediation side.

Ashish Rajan: So the approach moving forward is just to identify where these roles are blending in between the remediation, detection- Yeah

Ashish Rajan: then kind of find [00:31:00] that, "Ah, right. Okay." 'Cause to your point, at the end of the day, at the moment, vulnerability management is just a starting point. It's just gonna evolve into something m- a b- bit more, more blended and across the board.

Derek Abdine: Yeah, I think a lot of the, a lot of the stuff that we tend to see, and this is a good thing I think models are gonna allow us to spend more time on value-added opportunities within our organizations.

Ashish Rajan: Yeah.

Derek Abdine: And a lot of the kind of, like, day-to-day grindy work in security and IT are gonna be accelerated through some of these models, and we're gonna allow ourselves now to have teams that are very tight on both aspects of those problem domains, as opposed-- because we can defer more of that information and knowledge to the models now.

Ashish Rajan: Yeah.

Derek Abdine: We're still gonna need really smart humans to manage those, right? So that's not going away. But you know, I think we're gonna start to see those teams, and I think 'cause we have to, 'cause there's no alternative. That's what I mean by, like, we have to rethink how the last 30 years have worked.

Ashish Rajan: Love that. That was the last technical question I had. Uh, we're doing this series called, uh, You Laugh You Lose.

Derek Abdine: Oh, boy.

Ashish Rajan: Uh, I know. So I mean, I'm happy to go first, uh, so that just [00:32:00] to save you the embarrassment. And so far I've only lost. Okay. Uh, so I mean, it could not be as worse, uh, as one may imagine.

Ashish Rajan: But hey maybe people outside could gi-give us a better, uh, judgment of who is actually really good at this. The way it's gonna work is I'll say a joke. You have five seconds to hopefully laugh- Okay ... in my case. Uh, or uh, but if you laugh, you lose. And I'll have the same. No pressure. Yeah. All right, no pressure.

Ashish Rajan: All right, I'll, I'll go first. All right. Um, knock, knock. Who's there? Remediation.

Derek Abdine: Remediation who?

Ashish Rajan: That's okay. Everyone forgets about me until after the incident.

Derek Abdine: Ouch.

Ashish Rajan: Yeah. I'm like, "That shit took me two hours, by the way." Don't judge this by the joke, but all right, you're up.

Derek Abdine: There's two options that I have. How about why did the vulnerability management product go to therapy?

Ashish Rajan: Why?

Derek Abdine: Had too many unresolved issues.

Ashish Rajan: Okay, fair. You're making me cry a little bit. [00:33:00] Okay, fair. That was good. That was good. Okay, what was the second one? I'm curious.

Derek Abdine: All right

Derek Abdine: How do you know whether a vulnerability management program is mature?

Ashish Rajan: How?

Derek Abdine: The findings are old enough to have opinions.

Ashish Rajan: You can hear the laugh outside as

Derek Abdine: well. Pretty sure that's Austin.

Ashish Rajan: Yeah.

Derek Abdine: I know his laugh anywhere.

Ashish Rajan: Yeah. That's, that's a good one. I mean, okay, dude, both of them were really good. I'm gonna, I'm gonna use that now.

Ashish Rajan: Yes. I'm gonna, I'm gonna steal some of that and, uh, use that as well. But dude, uh, this was awesome. Thank you so much for the conversation. Appreciate it.

Derek Abdine: Appreciate it.

Ashish Rajan: Uh, where can people get in touch with you to know more about remediation and the work you guys are doing at Furl AI as well?

Derek Abdine: Yeah, at our website at furl.ai, and, um, you know, yeah, just reach out to us anytime.

Ashish Rajan: Yeah. And, uh, do you have LinkedIn? So I can probably put that link in there-

Derek Abdine: Yeah ... as well somewhere. My LinkedIn is very... It's, like, /derekabdeen. It's very bland and easy to remember, so yeah. I,

Ashish Rajan: I will put that link in the show notes as well. But, uh, but thank you so much again for the show.

Derek Abdine: Yeah, thank you for having me.

Derek Abdine: Appreciate

Ashish Rajan: it. Thanks, everyone. Thanks, everyone. Peace. Thank you for listening or watching this episode of Cloud Security Podcast. This was brought to you by techriot.io. If you are enjoying episodes on [00:34:00] cloud security, you can find more episodes like these on cloudsecuritypodcast.tv, our website, or on social media platforms like YouTube, LinkedIn, and Apple, Spotify.

Ashish Rajan: In case you are interested in learning about AI security as well, do check out our sister podcast called AI Security Podcast, which is available on YouTube, LinkedIn, Spotify, Apple as well, where we talk to other CISOs and practitioners about what's the latest in the world of AI security. Finally, if you are after a newsletter, it just gives you top news and insight from all the experts we talk to at Cloud Security Podcast.

Ashish Rajan: You can check that out on cloudsecuritynewsletter.com. I'll see you next episode. Peace.

‍