Ashish Rajan: [00:00:00] A good place to start could be could you tell us a bit about yourself?

John McBride: Yeah, sure. So my name is John McBride. I'm a senior software engineer at OpenSauce. I've spent time at AWS and was at VMware and Pivotal before the acquisition.

Worked on a bunch of cloud technologies. But I also maintain SPF 13 Cobra, which is a Go command line interface library in the Kubernetes and CNCF ecosystem. I've been all over the place, but yeah, do a lot of code work in the cloud space and yeah.

Ashish Rajan: And you have a talk as well at KubeCon.

Yeah, because maybe before we go into the talk, what are some different kinds of threats that you can think about or they should consider for a kubernetes cluster?

John McBride: Sure. Oh, goodness. Where to start? One of the things that you end up thinking about when you're like deploying apps to kubernetes is, you aren't just putting stuff onto a server.

You're putting stuff onto like a server onto a whole stack of software that then, gives you orchestration and availability and all the stuff we love about Kubernetes. Your threat models end up being like, tenfold at that point. There's container escapes, which are probably the [00:01:00] scariest but also The least likely because those are like the zero day vulnerabilities that exist within like Linux and the nodes that your Kubernetes is running on top of those are the most scary, but probably the least likely to happen maybe the not as sexy but just as scary and happens more frequently, it's just misconfiguration like, Oh, I accidentally gave this role to, the wrong user or the wrong service, and now it has an escalated role and some pod that I'm running, somebody's experiment off of GitHub or something that I'm running on my Kubernetes cluster now has elevated access or elevated service roles essentially in there. So misconfiguration is not as sexy to talk about, but that's probably the one that should be talked about more.

And what's your talk about?

So my talk is about essentially using social engineering and zip domain that I bought. It's actually kubernetes. zip that I got and using that to hack the kubernetes software supply chain. Yeah, a lot of social engineering involved in that and a lot of kind of perfect elements that came together to Put this proof of concept [00:02:00] together, basically.

Ashish Rajan: Wait, so tell that to me again because now technically zip domains are available for people to buy. It's pretty fairly cheap as well. How did you, like what was the the flow of how this kind of attack of sequence happened?

John McBride: That, yeah, that's a good question. And it's weird 'cause the lay person probably doesn't know a lot about oh, Google started offering this top level domain.

But yeah, in May of this year Google Domains started offering .Zip top level domains and, I could go and buy like John's cool website, zip, or pod security podcast. Zip. Yeah. And go to that as a website just like you would like my website.com or something. The really kind of thing that I think a lot of people raised a red flag about was this is just confusing.

It just makes the internet less of a safe place to be because there's already file extensions, dot zip file extensions. You might download some photos as a zip or some code as a zip. And even off of GitHub, you can get zip bundles of code just off of GitHub. And that's something that the Kubernetes ecosystem does.

A lot of [00:03:00] releases have zip files as sort of code artifacts. And so my idea was like I'm going to buy kubernetes. zip for $12 and start using that to initially I I told people about it cause I was like, Hey, I bought this domain. We probably want to do something safe with it. So initially I just redirected it to kubernetes. io cause I wasn't going to do anything with it. But I realized there was a proof of concept, a social engineering hack involved in this where. We're like, oh, somebody could use kubernetes. zip to say hey, there's a big vulnerability. We have to release code right now. Here's the zip file, kubernetes.

zip. Click this now and do the release. You would click it, get the kubernetes bundle of code, build it, release it, but it was not actually from GitHub. It was maybe something that I had that I, injected little bits of code to, it could have been anything.

I could have, injected code into kubelet, or kubedm, or kubectl, and all of a sudden all your clusters belong to me because you use some code that, that I socially engineered using kubernetes. zip to, to get you to download and use.

Ashish Rajan: Okay, so is there a [00:04:00] way to detect this, or?

John McBride: Yeah you're doing even the most basic, of checks doing shasum checks against the zip file, or any of the code from kubernetes, Say okay, Kubernetes will give you a shawsum check Shaw 256 or something.

You might check that and say, okay, does this shashum check that Kubernetes is saying is legit match what I get from this zip bundle of all this code? Yeah. If they match, then you're assured that they're the same. If they don't match, then you know that there's some code in there. That's not there's some diff.

There's something that doesn't match up. If you're doing that, then you would catch that. Most modern day browsers will also tell you if you're like being misdirected in a weird way to like different websites. Okay. But that's where the big social engineering part of this comes in, if you get some email from your boss or from your principal engineer to say we got to do this right now.

Yeah. Like we got to get this release going. We gotta, ship some Kubernetes. Here's the link to the release. Let's ship it. You're just going to yes. You might not do the shawsum check. Yeah. For my talk, I think the big kind of poster [00:05:00] point about it is that software without software validation, it's just risky.

If you're going to use some software that you haven't validated in some automatic way. That's just a problem. And you could, this could go for, some Kubernetes bits that you're downloading. It could go for just, curling something into Bash to install.

Or even just doing oh, Docker runs somebody's random container. Yeah. That's just a, an open invitation for some, messy, vulnerable code to land on your system somewhere.

Ashish Rajan: I guess adding another layer to this as well then, so there is capability to detect it, but software supply chain in itself is like a big topic as well.

What are your thoughts on that? And obviously, kubernetes.zip was a great example of how it could be misused in the wider context of Kubernetes software supply chain. How is that being addressed in your opinion at the moment? Maybe what's missing as well ?

John McBride: Yeah, that's a good question.

The supply chain at large is more or less this bit maybe consumes that bit, and then it, it creates this like long pipeline, this long chain, that might get you an actual like vendor service that you can use. That's inevitably where you get EKS. But in that long pipeline [00:06:00] of stuff, there's a lot of people involved.

Yeah. There's a lot of projects involved. And anytime you involve people, Very intimately, like you're just going to have those kinds of problems, those kinds of trust issues. So in my opinion, I think probably one of the biggest vulnerabilities within just the supply chain at large is just a lack of maintainers and a lack of people.

If we're inherently trusting this supply chain of, stuff going from A to Z and there's individuals involved in there who maybe have malicious intent or. Or their accounts get compromised or something. They could be that like single piece inside of the chain that ends up, just flowing all the way downstream to, some major service and those are very hard to detect.

So maybe I'll say that's twofold. It's just, there's a people problem. There's just not enough people involved. Yeah. And then there's also like a detection problem. It's really hard to introspect that long chain of like, where do things come from and how can I be assured what I'm getting is actually what I want and what I need

Ashish Rajan: Especially in an open source context as well. There's a lot of high trust being involved in it and maybe another one. And [00:07:00] when you and I connected, we spoke about how single contributor into a project as well. Could you share a bit about that story I think you spoke about it last year as well??

John McBride: I did. Yeah. So at KubeCon EU in 2022, my talk was about the risks of solo maintainer dependencies. And that's what I'm talking about there is those projects that maybe have just one or two or very small groups of people contributing into them where, they're the ones with the keys to the castle.

They're the ones with, the big key to the city and they can, have all sorts of ways to make things just terrible for people down the supply chain. The worst case for that would be where some malicious actor gets a hold of somebody's account. And I'll use myself as the example where I maintain SPF 13 Cobra.

Yeah. It's actually gotten much better these days cause there's a smaller cohort for a long time it was just me. And something that kept me up at night was like, okay, what happens If some nation state actor gets a hold of my account really goes after me, they're going to get my account.

They could inject some malicious commit deep into Cobra, [00:08:00] deep into the commit tree that like you wouldn't see it on the surface unless you were doing some like major Git surgery or something. Then they could cut a release that would be consumed directly and then that would get shipped as part of some Kubernetes service to Google, Amazon, Azure, et cetera.

And boom, the world is owned. And that's something that as a solo maintainer, it's like, Oh wow. I could be that single chain in the supply chain that breaks to cause a massive vulnerability. Now, I think that story is very dramatic and I'll play my own devil's advocate because.

There are people watching these things. I think if somebody saw that, Oh, John just force pushed to Cobra to, drop some weird commit in here, what's going on with that? Like it would get picked up eventually. I'm very unassured that it would get picked up immediately.

I guess that's what I'm saying.

Ashish Rajan: So to your point, the detection becomes even harder so where should people start?

This is my final question as well, cause I almost wonder that where do you even start building trust in open source as well. Like I think what would be one way?

Cause I obviously we are in KubeCon. We were talking about this earlier, how many people are here and a lot of them truly believe open source, but [00:09:00] and rightly so it's a community. So there's a lot of trust hanging in there as well. But for a lot of people who are probably enterprise, they want some assurance that I can call someone and go, Hey man did you work in that code or you need help with it?

Can I contribute into it? Like where do people start with that when it comes to open source? So I think you start building trust on it.

John McBride: Yeah. That's that's the million dollar question, I think, honestly I don't know if there's a great solid answer for it right now. I think there's some hard answers where, like companies could begin to invest more into projects that they intimately rely on, and I think.

Kubernetes services are a great example where, big clouds are shipping Kubernetes services that you could consume and, They're making X, whatever dollars on that. And I would love to see more of that invested back into the communities and back not only monetarily to help with literal infrastructure that, that the Kubernetes communities need, but also just engineering resources.

And engineering resources are expensive and, making those allocations is hard. So I won't deny that. That's a very hard thing to stomach. That's an unsolved [00:10:00] problem. But, to go back on my point about software, validating software you do have to also invest or you should start investing into solutions that can start to give you that kind of assurance.

The most basic thing, like just doing shop checks, there's places, there's whole Kubernetes shops who just aren't, or just consuming straight up upstream Kubernetes and it's just a riskier move to make. Yeah. Start doing maybe the most basic thing and then go from there.

Start consuming images, maybe that are built in house that you can have signed by something like Sigstore and SBOMs software build materials that s that flow downstream into your Kubernetes images and stuff. There's a lot that you could start doing with a lot of this stuff.

And then you can continue to take that to the nth degree and start using other services to, do monitoring and assurance of what's running on your Kubernetes and stuff.

Ashish Rajan: Would that be different between a managed Kubernetes versus self hosted one? Because I imagine a lot of people go , I use AWS, I use Azure, I use Google Cloud.

Would the open source conversation be a bit more different? This is assuming that Amazon, Microsoft and Google are all doing the same thing you mentioned, they are verifying where the thing is coming from.

John McBride: Yeah, [00:11:00] that's a good question. I'm not sure. I think that, and maybe this isn't the most ideal solution, but the first thing that comes to mind is that if you're building on top of platforms that exist in the open source and are really fed by, just free and open source contributions then, you should look deeply at your bottom line and how that flows into open source communities because say my terrible scenario goes as my nightmares presume where, Oh, John's account is hacked.

And, he just somehow got something terrible into Kubernetes via Cobra there's definitely a story where it's like, now our business is compromised, our bottom line is at risk. How could we have prevented that flowing into communities that need help and that need contributors.

Yeah. So maybe not the most ideal solution. And again that's a hard line to take. I do still think that, there's a lot of trust that you end up putting in cloud providers and doing your due diligence as well as important.

Ashish Rajan: So maybe a good takeaway for cloud providers is that if you, or maybe in general companies as well, just to look at the open source they use and if there's anything which has a single [00:12:00] contributor and potentially is of high value for the organization, that should be the first cab off the rank for. are we really using this, or do we really need this kind of thing, but this is good man, thanks so much for sharing. Where can people find you on the internet to connect with you and talk about the talk that you're giving at Kubecon as well?

John McBride: Yeah, for sure you can go to johncodes. com to get, all the social links and stuff.

I'm on X @johncodezzz, with a bunch of Zs, because John Codes normal was taken. You can find me on LinkedIn, John McBride yeah, I'm around, JohnCodes. com as well. I'll put that link in the show notes as well. Thanks so much for coming in.

Ashish Rajan: Hey, thank you so much, appreciate that. Thank you for everyone for watching, and I'll see you in the next episode.

Peace.

‍