SPEAKER_01 0:00

Hey everybody, welcome to the Ruby AI podcast. I am one of your hosts, Joe Leo. I am without my co-host, but only temporarily. So if you're listening to this recording, you're like, oh, I can't stand that guy. Joe, just skip ahead, like five minutes. Valentina will be here. But more importantly, I am joined by a friend and colleague and the newest minted member of the Rails Core team, Mike Deleccio. Welcome to the show. Thanks, Joe.

SPEAKER_02 0:28

It's really good to be here. I am a big fan of the podcast, and I appreciate you finally getting around to inviting me on.

SPEAKER_01 0:35

Well, thank you very much for coming on. You know, we haven't always had such royalty on the show. This is a big deal. Now, I was at Rails World where you were inducted, I guess, is the it's kind of like a almost like an Aaron Patterson dunk tank that you had to step into.

SPEAKER_02 0:51

Right. He embarrasses you on stage, and that's part of the hazing ritual. Yeah.

SPEAKER_01 0:55

I was really happy to hear that you were joining the core team because after two days of kind of like talking about the DHAH keynote, it was like, well, is there going to be a Rails? But then the question was answered, like, yeah, there'll be Rails for a while because Mike's on it. I guess that's true.

SPEAKER_02 1:11

Definitely as Rails has become a more mature framework, there is a lot of maintenance that needs to go on to keep it going, particularly around security. And so I think I certainly don't have as many commits as a lot of other folks who are on that committer's team. But I think uh being invited to join the team is an acknowledgement of a lot of the maintenance that I've been doing, uh, both behind the scenes and in security. Yeah. And I think that that is it is a signal that Rails is going to be a well-maintained framework for the uh at least the intermediate future. Who knows what we'll be doing in 50 years?

SPEAKER_01 1:44

But 50, I'm not worried about. I'm old enough now that I 20 is good. We could eek that out. Not everybody may know this, but you are also the longtime maintainer of Nokagiri, and you've done a lot of work there. When would you say, or has it been all along, that you started to get more of focus on security?

SPEAKER_02 2:02

So the the security focus has definitely been something more over the last few years because it's the needful thing. It's something that needs to be done. My day job at 37 Signals is at least partly focused on security. And so I had all these great agent tools I was using for my day job, and I was like, I'm just gonna bring those over to the Rails team and see if they tell me to stop. And I inflicted some help on them, and it's been working out great because we do get a lot of reports on Rails. A lot of them are hallucinations or duplicates or informative, like we know about it, but we're not gonna fix it. And so having a lot of agent tooling has really helped us handle the flood of security reports. And so, because I had those tools, I started spending more time on it. Like once you demonstrate competence in an area, everyone's like, Yeah, keep doing that. Here's the shovel, keep digging. And so it's really just been over the last few years, for Rails at least, that my focus has been on security. But having maintained Nokigiri for so long, it's a very widely deployed library. And certainly I had to come up with a lot of coping tactics and rituals over the years to deal with security reports on some of those libraries that I maintained too.

SPEAKER_01 3:10

Whenever I'm reminded of the XKCD cartoon with the one little library that is holding up the rest of the internet, I think of Mike Delecio maintaining Nokagiri. Although that's not such a small thing, it's a really big deal, but it's been working for so long and through kind of generations of software development.

SPEAKER_02 3:26

I guess that's kind that you identify me with the XKCD cartoon. Uh, you're not the only person who's mentioned that to me. That's just part of being an open source maintainer. Like it's not glamorous, and sometimes that's the position you're put in. I have other people who help too, so it's not just me. Of course.

SPEAKER_01 3:42

My honest opinion is that your talk at Railsworld was the best talk that I saw at Railsworld. Wow, that's very kind of you to say. But the fascinating part to me is one, what you built and how you built it, and I want to get into that, but also that you didn't actually plan to make that talk, right? You had a different plan for a different talk, and then something happened. So you want to take us through that?

SPEAKER_02 4:02

Sure. First the earth cooled and dinosaurs walked the earth. Okay. And then around November of 2025, the agents, kind of in in my perception and in maybe yours too, kind of turned the corner and became capable of a new level of task. And we started using agents everywhere inside 37 Signals for Basecamp. And one of the things that Rails is really missing is the ability to do markdown input and output. So I started to work on like making action text support, like generating markdowns so that agents wouldn't have to parse HTML. They could just get the markdown the way they wanted it. And I pitched the talk and it got accepted, and I was like, fantastic. So now I have six months. You know, this is in March, say. Right. I have six months to actually build the software into Rails and make it work and write the talk, like conference-driven development. Yep. And so I started working on it and I got a couple of pull requests in when in June we got the CVE that I talk about in the hot cell talk, which is active storage, image processing, vulnerability. And this was a big wake-up call where it's like, oh, being able to talk markdown to agents is like fun and interesting and might have made a good talk, but this is something that's serious and urgent and important.

SPEAKER_01 5:11

Yeah.

SPEAKER_02 5:12

So I basically cleared my open source schedule to work on improving active storage and writing hot cell. And then when it came time to give the talk, I was like, this seems way more relevant. And I pinged Amanda like a week before Rails World, and I was like, hey, hate to do this to you. You have to change all your marketing materials, but I'd really like to change the talk. And she was very, very supportive. Like she came back immediately and was like, Yes, I think this is the right decision. The Rails World website is already updated. Good luck.

SPEAKER_01 5:41

Yeah. From her standpoint, yes, that's more work, but there also is a good marketing opportunity. I mean, when I saw it, the first place that I saw it was through Amanda or through Railsworld saying, Hey, this is more urgent. So we're talking about this. To me, that's worth a click, right? That's got my attention right away. Yeah. And we talked about this after the fact. It does seem like the whole proposal to conferences thing, the happening as it does six months or so before a conference renders a lot of talks obsolete by the time you get there because of how quickly things are moving. Isn't that right?

SPEAKER_02 6:11

I agree. I think that as conferences are evolving for the new era we find ourselves in, this is something that is probably gonna have to change.

SPEAKER_01 6:20

Yeah.

SPEAKER_02 6:20

Like I get why you want to do that, right? You want to have your marketing materials set up. You also probably want to do like a maybe you're doing like a double blind right review of topics, but maybe that needs to change so that you actually do extend a little bit more trust to individuals and say, we want to invite you to talk. And we kind of trust that you'll come up with something that's gonna be interesting. And I realize that that is depending a lot on privilege and probably seeing the same speakers over and over again. But I think there has to be like a new sweet spot somewhere in there where conferences have to allow for talks to develop late.

SPEAKER_01 6:52

We could evaluate people as speakers, have them send a TikTok video in of you speaking about just uh anything, and then say, okay, you're good enough. Now let's have you talk about something more relevant when the time comes. But I I agree it's tough.

SPEAKER_02 7:02

You could watch my dance tape of me doing the loss or whatever it is.

SPEAKER_01 7:06

Yeah, yeah. I would definitely like to see that. I feel for conference organizers because this is a challenging time. We have both organized conferences in the past. I talked to a lot of conference organizers at Railsworld, interestingly enough, including the RailsWorld folks who did a phenomenal job, but also people that were doing local and regional conferences. And it's tough out there. We have Exo Ruby coming this weekend, and I know they're having a tough time. I mean, New York City's always hard, but it's it's a real challenge to get people interested, to get people here, to get people thinking long-term enough that, like, sure, you know, we had this joke about Rails being the future before we even started even talking today. And people have to think, well, it's at least the future for a couple of years in order to keep going to the conference. Sure. When you're trying to attract new people to the industry and to two conferences to learn and grow as professionals, you want them to feel like they have some stake in the game.

SPEAKER_02 7:52

Yeah, I sometimes wonder how new people to the industry or newer people are going to find their way to a community. I think you and I were lucky enough to join Ruby when it was still a relatively small community, and we've built those relationships over 20 years. And yeah, even if this ship is going down, like at least we're gonna like we're gonna have a good time with our friends.

SPEAKER_01 8:11

That's right, exactly.

SPEAKER_02 8:12

And I do wonder how people will find a community or what those communities will coalesce around. Like, is it shared values, is it shared business outcomes?

SPEAKER_01 8:20

Like it's gonna be interesting to watch for sure. Yeah, I agree with you. So getting back to your talk, you started the talk with what I thought was a very Mike Delestio thing to say. It was also very accurate.

SPEAKER_02 8:29

You said, Hey, I don't know what that means.

SPEAKER_01 8:32

I heard you say dinosaurs roamed the earth. And then man started building software up until let's say a couple of months ago, everybody was writing code by hand. And you said, by and large, we humans we weren't doing such a good job. And that I thought was telling because you could play it for laughs, but it also is revealing about how we are reacting right now to a world that has a gentic development and agents running rampant. So, what does that look like from your vantage point as a software maintainer, open source maintainer?

SPEAKER_02 9:02

Like, what does it look like now? Yeah, right. It's a good question. I think there's a spectrum where I think on one end you've got total vibe coding where maybe people who aren't even technical are asking their agent, build me an iOS app to track my weight and eating habits or something, right? And they can get customized software really quickly. And it's probably absolute garbage code that you would not be able to sell or give to someone else, probably insecure, probably has lots of bugs, but it doesn't matter. Right. It is bespoke software. So at one end of the spectrum, it's like it could be garbage code, but if it's useful to one person, that's super, super powerful. And I think about the early days of the internet when people were started posting videos on YouTube, and like some people were like, Scorsese is gonna have a heart attack. And but then I was like, but think about all the Scorsese's who are now gonna have an interesting opportunity to make and publish movies. On the other end of the spectrum, I think there has to be a world where professional software developers are still writing code but telling agents what to do and how to do it, and are using agents to review and harden the code, red team the code before it gets released. And all of this used to be unreasonably expensive to do. You would write the code, you'd have positive assumptions about the users and the use case in mind, and you would ship it, and people would find it useful. And if you were going to do a red team or have an outside reviewer on every commit, that would really slow down your delivery time. And so, like, it doesn't have to take forever, right? It's just tokens now. And for me, at least, this is the end of the spectrum that I'm trying to live in and encourage other people to be in is like, yes, agents will help you write the code fast, but then you need to slow down, tap the brakes, do adversarial code review, do an attack, ask codec if Claude wrote it, ask Codex to try to break it. I work with Jeremy Dare, and he has this line that he uses to motivate the agents to find bugs. He goes, Is that the best you can do? Right? Like as if they're like a preteen child trying to please a parent, and it works with the agents, and they will go back and do another pass and re-examine their assumptions. Like whatever the model has been trained to do, it responds really well to that phrase. So for me at least, like there's a whole spectrum. Most people I think are somewhere in the middle. Yeah. My job at least is to live at one end of the spectrum and try to pull people towards that. You know, what does professional software development look like? End of the spectrum.

SPEAKER_01 11:23

You've said something interesting, and you just mentioned it again with Jeremy Dare that when you came to the Rails core team with some security tools, they were tools that you built. And actually, through most of my professional career, the tools that we used were available to everybody, especially in Ruby, right? They were all open source. I didn't have some tool that you didn't have. That's actually different now. That is actually a different thing in 2026. You've got some tools that you built that may or may not be suitable for release, right? Because they could be proprietary or they could have personal identifying information, could it be any number of reasons? But you, Mike, are not actually just bringing your brain anymore. You're also bringing these tools with you, which I think is an interesting dynamic that's developed.

SPEAKER_02 12:04

Is that any different from hiring a really good cabinet maker to come into your house and install cabinets in your kitchen? Like that person is gonna come in with their own set of experiences, their own preferences for where to get wood from, and maybe they have their own toolkit, maybe they've got a lathe they've been using for 10 years or like I don't know.

SPEAKER_01 12:23

Yeah, maybe not, but I think it is different from hiring a software engineer maybe five years ago.

SPEAKER_02 12:28

You're right. It has changed over the last at least, you know, maybe two years. But I'm trying to think a little bit more in terms of well, let's let's let's unpack the assumption that's going into my statement. Okay. My assumption is that going forward, the industry is gonna get smaller in terms of the number of people who are going to be able to make a living doing software development. Seems that way. A lot of this may just be people retiring and saying this isn't how I want to work. Uh, some of this may be junior devs struggling to break in, but as a result, then the people are gonna be less fungible. You're gonna hire someone with a particular set of skills or experiences or tools. I don't know. I'm just thinking out loud.

SPEAKER_01 13:05

I want to thank you for telling me that I'm right. That's actually the thing, the quality I like the most to my guests. So let's dig in a little bit. How did the idea for hot cell come up? What was the security threat that you were dealing with, and how did you solve it?

SPEAKER_02 13:18

It's hard to even say where the vulnerability was. Because again, the exploit was I can get your Rails secret key out of a file in your application container, and I can use that to get remote code execution. That's like the outcome. But like to get there, you had to go through five kind of low severity vulnerabilities. So it's hard for me to even say like the vulnerability was in X because there was like some weird Rails behavior, there was some weird libvips behavior, libvips is an image processing library, there was some weird libmatio behavior, which is a library for MATLAB files. The end result is if you could manage to trick Rails, libvips, and libmatio all at the same time, you had this really bad vulnerability. The aha moment for me was realizing that there were a hundred image libraries that libvips pulls into your Rails application container. Right. And any one of them could have a vulnerability like this. And realizing that the actual issue here is architectural in nature. Like you can patch any one of these libraries, or you can teach Rails to like block one of these libraries, but you're always going to be in reactive mode chasing the next zero day. Like the next zero day comes out, you have to be reactive to it. Wouldn't it be nice if there was an architectural solution here where you could run those image processing libraries in a sandbox, a sidecar container of some kind where there's no application code and there are no secrets and there are no environment variables to steal, and maybe there's not even any network for lateral movement or for denial of service attacks to be launched. What would it look like to put all of that somewhere where it actually doesn't even matter if there's a vulnerability? And so that's where the idea for hot cell came from. A hot cell is a device that's used in national labs, like radiation labs, where you have something that's so radioactive you want to put it into a big lead box with a window. Right. If you're gonna deal with it, you have like robot arms that you use, and anything that goes in or out of the box has to be like tightly controlled. There's a really interesting video from Oak Ridge National Labs in Tennessee where you can actually watch how stuff gets posted into the box, and it's like through a pneumatic tube that's also not touched by humans. Anyway, so the idea for hot cell came from that physical analog. That's the basic idea for what hot cell is. And it actually turned out to be not that hard to do because Rails already abstracted all of Active Storage's image processing behind classes, right? And you can configure what classes you want to use. Like it was built to be extensible, and if it's extensible, it's also replaceable. And so Hot Cell essentially just replaces that whole chunk of active storage.

SPEAKER_01 15:54

Yeah, it's very cool. I want to take a moment to welcome to the show Valentino Stoll, my co-host. You know, V, we were totally lost without you, but luckily Mike has been steering us effectively.

SPEAKER_00 16:06

First of all, I just love Hot Cell having worked with so many vulnerabilities in the past related to PDFs. The need for this is very high. Yeah. You get it. People are so creative. There's so much you can do with uploaded files of varying kinds, and there's not enough ways to uh keep up with it all. So I'm curious. Can I tell you a quick story before you ask your question?

SPEAKER_02 16:28

Yeah, please. The first thing I did when I built Hot Cell was I rolled it out to all of our apps at 37 Signals, right? And we have one or two security researchers that we have a relationship with who have been probing a lot of the image processing that goes on in, in particular in Hay and in Basecamp. And he or she sent in a new one last week that would have been another pants on fire, like, oh my God, arbitrary file read in another image library. But it came in and I like I sat with it for a day, and then I went and I commented on it to my team, and I was like, I think that we can just close this as informative because I don't care. That's amazing. Yeah.

SPEAKER_01 17:05

I think it's all gonna be okay. Yeah.

SPEAKER_02 17:06

Yeah, the report demonstrated that yeah, there's an arbitrary file read in this library, and the only thing he could read was the hot cell file that was processing the thing. And I was like, mission accomplished. Yeah. That's operating as design. So for me at least, that was like another checkbox for like, yes, hot cell is the right architectural pattern here. Sorry, I didn't mean to interrupt you. Please ask your question.

SPEAKER_00 17:27

I love your analogy of the radioactive cell. What are the hot cell gloves, right? What's the pneumatic tube? Like, what are the things that we can like attach to that analogy from a Rails perspective, you know?

SPEAKER_02 17:39

Certainly your secrets are radioactive and you don't want them to get out.

SPEAKER_00 17:42

True. Where do like the incidents come in, play here once you have everything, like the boundaries defined? If you do find like these vulnerabilities and like bad actors, in your case where you're just checking the box, is that where the future of it goes? Is like, well, we've defined the boundaries and there's no leakage outside of the boundaries, and we can know that we can verify that there nothing leaked out, and that's it, and just call it a day. I think so.

SPEAKER_02 18:07

I want to acknowledge that certainly there's a bug here that needs to be fixed, right? So this report came in last week. It was for a new and different image library. Certainly that's something we want to close, but we can treat it as a bug, and we can do it thoughtfully, and we can do it on our normal release cycle. We don't need to interrupt everything that we're working on and treat it like pants on fire. And we don't need to worry about like, were we exploited? Was this secret exfiltrated? Do we need to rotate all of our keys and passwords now?

SPEAKER_00 18:37

So much stuff goes out the window, yeah.

SPEAKER_02 18:38

So from our perspective, it's a huge just change in approach and how you can respond to it. You don't need to be so reactive, and it's not such a severe reaction. And for me, that's a win too. Like maybe we consider those reports to be like a low severity vulnerability. Yep. Great. We have a lot of low severity vulnerabilities in our products. We as an industry, I mean, that we're fine just living with because maybe the fix is expensive, and maybe the fix breaks behavior for Rails in particular. I think that is true for a lot of the lowest severity vulnerabilities that are reported, but is fixing it would be a breaking change. It would break existing signed keys or something. And so we can live with it, and then that just lowers the temperature of everything in this security universe.

SPEAKER_01 19:23

For me, when you were giving this talk, you threw up a uh a slide with the LibVIPS dependencies, and that was the aha moment for me where I was like, oh, right, any one of these or any one of the libraries that they themselves are depending on can walk up this tree and perpetrate an attack. But then the mind starts to reel a little bit, right? Because this isn't the only place where we have system dependencies and live and dependencies in libraries. And so, and I think you made a really interesting point just now, Mike, about living the life of a person on a security team, you are always balancing trade-offs. How much work is it? How many breaking changes is it? We don't live in a world where there are no vulnerabilities. So we live with some, we decide some are not, we can't live with, and we have to do something about. And I wonder if you've thought about this. Are there other places in Rails in the ecosystem where a hot cell architecture or something like it is appropriate?

SPEAKER_02 20:18

Almost surely. I haven't put a lot of thought into it, but some of the typical places I think that would benefit from running in a container with control dependencies is like accepting archive files from users, right? I have a zip file. Great. There are like 13 ways for me to turn you unzipping a file into a denial of service attack. Right. Wouldn't it be great if that was running somewhere where you could control the resources used? That's one example for sure. And I'm sure that people will come up with creative uses for hot cell. I'm hoping they do, which is why the gems are structured so that there are kind of the base classes like hot cell client, hot cell server, that represent just sending a message and having work done in a container. It isolates that from the active storage gems, which are then built on top of that foundation. And I'm hoping that people will use that foundation for all of their security needs.

SPEAKER_00 21:09

I love that structure because I I immediately think clustering, maybe some uploads you want clustered if there is a particular uh sensitivity to some aspect of a file that people are uploading where it has like very secret information that maybe is grouped with stuff that is just like images for the site or something like that you want treated and handled differently. And the shape of this seems to align with that well. I'm looking forward to messing with that. Where do agents play into this? What's in your mind as far as interacting with untrusted parties like that within the Rails ecosystem?

SPEAKER_02 21:43

Again, I haven't given a lot of thought to it. I feel like I was heads down on the implementation and hadn't given a lot of thought to We're going to encourage you to think deeply on this show for a change. Thanks. Agents have made it way easier to attack applications, both for From a perspective of finding vulnerabilities and exploiting them, but also just from like the sheer number of ways that an agent could come at your application, especially if you have like a command line tool and an MCP. And so my personal paranoia has never been higher. You know, I'm on a security team, like I'm paranoid by nature, but now it's like I walk around all day being like, that's gonna break, that's gonna break, that's gonna break. One idea that people have talked about is potentially using Hot Cell to like if I want to add agent behavior to my application, it might be great if that could run in a sandbox. And there's no reason that Hot Cell wouldn't support that style of work too. So it might be interesting you have an active job that kicks off to ask an agent to run a sonnet model and quickly give you a result for something and do that in a way that is very limited in blast radius. Is insulated. So it couldn't be tricked into going and attacking your Redis instance, right? There are a lot of uses like that that I think that as time goes on, those needs will become a little more apparent. But certainly things like that are kind of exciting to think about in a positive way, too.

SPEAKER_01 23:02

Yeah, I agree with you. I'm gonna piggyback on V, who's brought this up on a couple of different episodes, and I know this is gonna be near and dear to your heart, Mike. How do you secure your own agents when you are developing? When you're just writing code, whether it be for Rails or for 37 signals or RubyGems or whatever. What do you do? Are you just firing this thing up in YOLO mode and having it run wherever it wants? I suspect not.

SPEAKER_02 23:24

You're asking me like what I do?

SPEAKER_01 23:25

Yeah, I am.

SPEAKER_02 23:26

YOLO mode is my best friend, yes. Okay. The reality is that most things that require any kind of elevated access privileges, everything is in one password for me. God, I'm gonna give away the roadmap to hacking, hacking my home system.

SPEAKER_01 23:41

Don't worry, we don't have that many listeners.

SPEAKER_02 23:43

I think the new like auto mode that Cloud Code has is interesting, but still a little bumpy, which is a little bit more buttoned down than YOLO mode.

SPEAKER_01 23:52

A little bit. I f I find it a little bit scary myself. I've got some great stories on auto mode if you want to hear that.

SPEAKER_02 23:57

So what I'm in the process of doing right now, actually, is moving into what we're calling like an agent shed. So I've worked on laptops, just laptops, for 10 years, because laptops have been powerful enough to do what I've needed to do. But now what I need is I need something that's gonna work overnight on the 50 code review requests I get a week. So I have a little B-link, like a little like Magmini-sized Linux machine sitting on my desktop right now. And what I want to do is I want to set that up so that it just has a GitHub access token. Like that's it. So it can't actually do anything else dangerous. It won't have access to one password, it'll just have that GitHub token. And I'll talk to it through Fizzy or Basecamp where again it'll have a token and not full access to everything that I do. And then that is going to be a little bit more of a secure environment than running it on my laptop and having it have access to everything, right? I have everything on my laptop. I have personal files, I've got photos, I've got everything that one password can give you access to. So I'm thinking about that as the next evolution for me personally. I should be moved into it by the end of the week, this agent shed. And then that will do all the work and we'll see how that goes. But that's how I'm thinking about it right now. What do you what are you two doing?

SPEAKER_00 25:07

Go ahead, V. I was introduced to paranoia of running agents early on. I maybe have some absurd setups in some cases. But I run this AI server next to me that I run some workloads on, and I want to just like let it YOLO. And so I have some like tail scale buffering to try and like help with like network, just network-related, like firewalling that I connect the agent to, but I still want to communicate with it. So I have like my own local SSH over to communicate it and uh have a kind of like personal protocol I built so that no outside actors can know about the protocol, right? And maybe I've gone a little excessive in relaying right from one machine to another to try and gate the accesses as much as possible. Similar to Hot Cell, right? Like it's all about like just cording off the specific things that you want to do that are more sensitive. And so I try and do that as best I can. I mean, it's just me and it's just for fun, so it's like probably not perfect, right? Yeah, but at least I could be a little bit more peace of mind uh to myself. Like, okay, I can at least reflect on all the logs and just check my sanity over time, and then maybe if there were any issues, I would notice them. It's kind of my hope. And with all the gates in place, I can get more assurance. But as soon as you connect it to the internet, like you know, that's the real problem.

SPEAKER_02 26:31

That's why tail scale can be so good, right? Because you can totally shape what network access looks like at that point. That's smart.

SPEAKER_00 26:37

Even that has its own drawbacks or holes, right? So tail scale is pretty great at like discovering a lot of that. They are also an attack vector, right? A lot of people use them, so like they are a target that I am now potentially vulnerable to. And it's just like any other target, an attack vector. So yeah, it's just really just trying to create the systems that keep you proactive, right? And trying to keep up with the things that are trying to come at you and juggling them, right?

SPEAKER_02 27:02

And you can always trick agents into telling you exactly what's insecure about the box you have them in.

SPEAKER_00 27:06

Right.

SPEAKER_01 27:06

That's right.

SPEAKER_00 27:07

You're right. I should have a loop to just ongoing trying to tell me for sure. Uh I don't have that.

SPEAKER_02 27:11

So set up a capture the flag, and if it ever actually captures the flag, unplug it.

SPEAKER_00 27:16

Unplug it, yeah. That's right. Immediately. Well, I have a UPS on it, so uh, there might be a series of unplugs.

SPEAKER_01 27:21

Uh so with the pace of your talk, was one of the bullet points that I've been amassing over the last few months in exploits, vulnerability exploits, ones that hit home to our language and framework specifically, right? We've had several in RubyGems, which you are also a maintainer. We've had, you know, many, of course, there are always many in a major framework like Rails, but certainly, you know, the pace has ticked up in large part because agents are running rampant. I, like you, don't see that as a bad thing. I think it's an opportunity to mature as an industry. But one thing that I thought was interesting from your talk is that you said that you deliberately kept parts of the exploit vague, as vague as you could, right while you were filing it. And I just found that really interesting because that's yet another thing that it was just a standard practice for as long as I've been writing code was that you got to see all the details because it wasn't so easy to recreate and exploit. And now, not so much. What does that mean for you as somebody who sometimes has the unenviable position of reporting the thing that you've found?

SPEAKER_02 28:22

All of the supporting infrastructure around reporting and responding to security vulnerabilities is really showing its age. It's really showing that it was designed for humans. And I give you some examples of things that are wrong with the system right now. Number one, if I want a CVE assigned to a security advisory on GitHub right now, I am waiting weeks because GitHub has manual review. And I think that NIST and some of the other government agencies require legally a manual review process, but that puts everybody behind the gun. It's like no matter where I want my CVE generated from, I have to wait literally weeks most of the time. That's very surprising. We've had an advisory written and ready to go for an active resource security patch since late August. And I haven't checked this week, but I'm pretty sure we still don't have a CVE assigned. Wow. I'm not trying to paint the security team at GitHub in a bad light. This is true for all of the CVE pass-through entities. I know. It's really showing its age. Like having to do a manual review there is something that needs to change. It's just like an outdated practice in policy again. That's one. Two, I think that all of the distros, the Linux distros, for example, that are backporting patches, this is also a heavily manual process today. I want to talk just for a minute as an example, the CVE that kicked off this hot cell work, right? This was a behavior in libmat.io, which is a matlab library, and behavior in libvips, which is the image processing library that delegates out to libmat.io, right? So these two libraries, number one, libmatio does not consider the behavior to be a bug. It's functioning as designed, it's going to trust any file you give it. So don't give me an untrusted file, is the maintainer's point of view. Okay, cool. So then the libvips maintainer, who's a very nice guy who I've started to build a relationship with as well, he was like, Well, you know, it's really just a bug in how we're parsing headers. Like, you have to combine it with libmat.io's behavior in order for it to be so he doesn't issue a CVE for it and he patches it. Now we have two problems, right? There's a public patch that anybody can go on GitHub and see in libvips. If I know there's a bug in Rails in image processing, I can go to Libvips and I can see this patch sitting there. But because there's no CVE signed, I don't even know if that's ever going to get backported in the Linux distros. Like I might have to wait for like the new version of Ubuntu or Debian to come out in order to get that patched. So in the meantime, now everybody's on an unpatched Linux system for a known bug. Like, what do you do with that? Then it's on you as a human to have to try to figure out how do we make Rails protect users, and then you have to upgrade your Rails instance. Right. And all of these things are manual and slow, and there's lag time built in, which is just perfect for agents to get in there and exploit those cracks to get at your system before it gets patched, to get at Rails before it gets patched, to get at your application before you've upgraded Rails. I'm sure there are companies out there who are still running a version of Rails that does not have this CVE patch. Oh, there's no question about it. And those companies are probably like, let's be honest, look each other in the eye. They're not going to be in business for very long unless they change their practices. Because these CVEs are gonna come fast and furious. It's gonna get worse before it gets better. Yeah, I love this. Have I scared you?

SPEAKER_00 31:38

Sorry, I didn't mean to like do the jump scare, but I wonder if maybe what happens is the compliance becomes so difficult to upkeep that the compliance rules change. Yeah. In in order to upkeep you, and then people are just vulnerable. Yeah. As a principal, which is also a scary thought.

SPEAKER_01 31:56

Yeah, I wonder how that changes. I understand that you need manual review right now. That's the the theory that we have right now with the information we have right now. How does that actually change? I mean, agents get a lot of stuff wrong. Moving it to a fully automated process carries with it its own risks, but I wonder if those risks outweigh the risks that we are facing right now. And I'd like you to answer that.

SPEAKER_00 32:18

Ubuntu's unintended upgrades, right? That's a great advancement. Sure. And that's one way it does work in a way. Not that it can't be detrimental.

SPEAKER_02 32:27

I I've been thinking a lot about JEV or JEV style tools lately. Yeah. Where they give you, for anyone who's not familiar, right, you it's a model where you give it input and you ask it a yes-no question. Or you give it a multiple choice question, multiple choice answers, right? And it gives you a confidence interval, and that's the best bit. Hypothetically, if I was in charge of one of these CVE pass-through uh entities, I would say a project has written a security advisory, feed that into the GEV model and say, is this genuine? And give me a confidence interval. Like, yes, no. And if it's yes and it's over 80%, why would you not auto-approve that? Like, what is a human going to add to that process? Good point. And you can even add more in. You could say, is this a project that has a history of high quality security advisories? Put them on the short list and just auto-approve everything. Like there's a bunch of things that you could use agents to speed this up, or present a decision to a human, and then you have like Homer Simpson just hitting Y over and over again, right? There's a human in the loop, but the human's not adding much. But part of the problem there is like then that's a very unsatisfying job for a human. Nobody actually wants that job.

SPEAKER_00 33:32

You make a great point though.

SPEAKER_02 33:33

I think that as the tools get better, that some of this will get automated or will minimize the amount of human in the loop that's needed. That confidence interval is like everything. Right. Because we can't trust agents. We can't trust them fully.

SPEAKER_01 33:45

V, you've been messing around with Jeb a bit, have you?

SPEAKER_00 33:48

Yeah, I made a uh port of Leia, which is a similar system one fully open weight model to Ruby, so you can run that. It's not as performant as Jeb, because Jeb has some secret sauce built into their system. It's a system. Leia's more of a the model approach, but you can fine-tune it. So you can fine-tune your own model from the the weights on hugging face a published. So like you mentioned, Mike, if you get a confidence interval of 80, even with Leia, pretty confident. And if you just skim the top, that reduces your volume of intake so much that it's worth just auto-applying and then having something watching with what gets it applied later and offsetting the cost. You know, there's a lot of like edge cases which are difficult. If you just ignore those, I feel like honestly, the Rails way of focusing on the 90% of your problems and simplifying and optimizing those, that gives you the flexibility to solve the real problems yourself. Right? Focusing humans where they are truly valuable.

SPEAKER_01 34:42

True. I think it's a good way to look at it. I mean, we're at the start of a point where there's going to be vastly more code than we've ever seen before. I think we're already there, right? GitHub would tell you we're already there for sure. And vastly more production code, which means vastly more places to pay attention to. We as humans cannot do it. And it's not like we're getting hired in droves to do it anyway. And so we do need to focus our attention or we need something to do it for us.

SPEAKER_00 35:06

Yeah, I've got a great analogy here on what you were saying, Mike. I've been reading this book, The Revenge of the Tipping Point. It's like a follow-up to the tipping point book, if you're familiar with it. Okay. Basically, it's like a huge influx in bank robberies in like the 70s and 80s. And it turned out to be like a regional thing that in LA there was like an unusually high volume of bank robberies compared to like other areas. Like on the East Coast, you would have thought, okay, well, people are catching on, it's in the news all the time. You would just like think there would be more bank robberies because there's bank robberies and people are seeing, oh, there might be some money here, right, to be made. And it wasn't, it was very regional, and they were trying to like investigate what it was. And what ended up happening is like people were doing like just cash graps over the counter in small volumes, and then knowing that it took time for people to come up and investigate and take questions, that they would then rob a bank down the block or a few blocks away, and then knowing that it would take time to get to that one and then stack them up, we're to the point where they got to like seven banks in one day, knowing that this would be a problem, and they got away with a bigger volume. And this is kind of like what we're seeing with agents, they're spreading it out in larger swaths, and we're seeing an influx in volume, and we're stuck there trying to investigate. To your point, Mike, that's like we're using these antiquated ways of intake and triage that we're not going to be able to keep up, and people are gonna rob us. Just overwhelm the system.

SPEAKER_02 36:29

It's uh tried and true tactic.

SPEAKER_01 36:31

Let's talk about RubyGems for a second, because you're also your on-call security team there. You're part of the RubyGems.org, or like which part of the team are you are you on?

SPEAKER_02 36:41

I am part of the RubyGems open source team, but I'm also on call for RubyGems.org, which is the Ruby Central funded and supported service.

SPEAKER_01 36:50

How has that evolved over the last, let's call it, a year to 18 months? Have there been big changes in process, in personnel, in approach? You know, what's happening there?

SPEAKER_02 37:01

I haven't been very close to that team over the last few months, but I just sent a message out saying that I want to make a point of getting more involved over the next few months. Now that I'm kind of past Rails World, I have some additional time I want to spend. Yeah. And one of the things I would really love is to bring some of the security triage tools that I've been using for 37 Signals and for Rails and bring those to the Bundler and RubyGems projects as well. The flow of security reports there is steady but not overwhelming. And there's a really great team who are already kind of handling that okay. But if we can automate some of that, I'm really interested in doing that. Ruby Central also recently hired a security expert on staff. People are gonna be mad at me. I cannot remember his name right now, uh, and I apologize for that. My interactions so far have been really great. He's got a lot of really directionally valuable ideas that I'm hoping we can execute on with a little bit more bandwidth. I wish I had more current and more up-to-date information for you on that team, but I have been not as involved as I wanted to over the last few months.

SPEAKER_01 38:05

I think there's an encouraging message here, which is that you're there.

SPEAKER_02 38:07

Lots of folks are. There are quite a few members of RubyCore team who are also on that um security team for Ruby Jumps and Bundler.

SPEAKER_01 38:14

Absolutely. From my perspective, just seeing how some of the larger incidents were handled, of course. I'm outside looking in, how the larger incidents that have made the headlines have been handled has made me feel good about the 180 or 190,000 or so gems that we're all depending on every day to do our jobs.

SPEAKER_02 38:30

One of the other things that that team has been doing is they partnered with Anthropic in the Glasswing project. Right. Which, you know, Glasswing basically gives certain organizations like Ruby Central a lot of token funding to go look for security problems. And there is a whole subsystem where they have been scanning popular Ruby gems with a lot of downloads and submitting security vulnerability reports in a very constructive way. Yeah, that's awesome. They wrote a blog post about this a little bit ago, which you can read, but they've been continuing to work on that behind the scenes. And I feel like that is a super valuable contribution to the Ruby community's security posture. I think that's really valuable and it's going on behind the scenes because everybody wants to submit these in a responsible way. Totally agree.

SPEAKER_00 39:13

I'm wondering, do you see as a maintainer, mine aren't as popular as maybe yours are my Ruby gems. As a maintainer, do you see this tooling set be distributable eventually where we get to a point where maintainers can use some tools and automate the same things and we get shared reporting in some capacity? Is that kind of like the eventual goal of all of this?

SPEAKER_02 39:33

Yes, I certainly have the intention of sharing my tools with the world. And I'm not the only one. There are a lot of people who are working in this space that are building and sharing tools. What I'm hoping is that once I take the tools that I have and apply them to RubyGems, then at that point I've done it three times and I can extract the common bits and I can ship something. Right. Even if it's just a repository that's like, use it, don't use it, I don't care, I'm not gonna version it. But like here are the skills, the agent skills you need, and here's like the command line tool to interact with hacker one and GitHub to try to make it easy for people. Again, it's not about like my process, it's about building the tools so that other people can build their process on top of it. I'm hoping this is all gonna get a lot easier for maintainers over the long run because if it gets any harder, people are gonna unplug their computers and go be sheep farmers.

SPEAKER_00 40:17

I know, I'm thinking about it, you know.

SPEAKER_01 40:19

Mike, we're gonna wrap up here. I really appreciate you coming on the show. Thanks for sharing with us all the stuff that you're working on. Thank you for all of the work that you do for the Ruby community. I'll see you around real soon.

SPEAKER_02 40:28

My pleasure. Thank you for listening to me prattle on.

SPEAKER_00 40:31

And real quick, if anybody wants to follow and await your tooling and all of the security work, where should they follow you?

SPEAKER_02 40:37

I'm generally on Flavor Jones, is my uh handle on GitHub and on most of the social media platforms. I guess that's probably the best place.

SPEAKER_00 40:46

Or the announcements. Thanks, Mike. Thanks, everybody.

SPEAKER_01 40:48

We'll see you again next week.