Valentino Stoll  00:00
Hey everybody, welcome back to another episode of the Ruby AI Podcast. We have a very special guest today, Kinsey Durham Grace, and obviously joined by Joe Leo. Joe, do you want to say a little word before we bring on Kinsey?

Joe Leo  00:15
I'm very happy to bring on Kinsey Anne Durham Grace. I like the four names, but welcome to the show.

Kinsey Durham Grace  00:23
Awesome, well thank you so much for having me. I'm really excited to be here and chat with you both.

Valentino Stoll  00:30
Well I'm excited too, but before we start I've got a bone to pick with you, Valentino.

Kinsey Durham Grace  00:35
I'm glad it's not me.

Valentino Stoll  00:37
Yeah, no, no, it's not you. Kinsey, you're innocent, although I've got to say, can I share the screen? Can I do this?

Joe Leo  00:44
I want to know what's going on with these images. If you're listening at home, I'm looking at the show image where Kinsey, appropriately, she looks great, she looks like a movie star in the center. Valentino looks like maybe present-day Luke Wilson, perfect three-day-old beard.

Joe Leo  01:03
Look at this hair, it's incredible. I look like I ate our producer and I'm about to go into shock. And I don't like it. And I just want to tell you, for the record, I am going to be looking into how these images are generated. We're going to get to the bottom of this.

Valentino Stoll  01:23
We use AI for almost everything with this show, and it has its drawbacks. Unfortunately for Joe, this is one of them. I basically have this standing thing, I have this Y, the lucky stiff clone, that I use as like our creative forte for all of our content.

Valentino Stoll  01:39
And one of its great things is like one-shotting these prompts for our image episode artwork. I actually stole some things from Sublayer's team image generator. They actually have a daily thing where, oh, based on this day in history, find the best, and that's still,right? Yeah, that's still going, I think.

Valentino Stoll  02:00
And so I use that, and then I combine it with, I say, here are the guest headshots, incorporate them into this like prompt, and you know, my Y, the lucky stiff goes, and it creates this like crazy fantastical description, which usually comes out pretty well. But unfortunately, OpenAI likes to embellish,

Valentino Stoll  02:20
I think, Joe a little more than everybody else.

Kinsey Durham Grace  02:24
Yeah, I like frame that it generated, so.

Joe Leo  02:27
Everybody seems pretty satisfied with it, except for one disgruntled host, but I digress. Kinsey, back to you, we're very happy to have you on this show. You are a big star in the Ruby community.

Kinsey Durham Grace  02:40
I wish. I am definitely not, but thank you so much for having me.

Joe Leo  02:44
You do lots of speaking engagements. We both are big fans of, it might not even be your last talk, but your last very high-profile talk at Rails World. And you are part of Ruby Central. You are on the GitHub team, one of the really most exciting teams that GitHub has. And so we're excited to have you,

Joe Leo  03:03
and we're kind of happy to dive in wherever you'd like. But let's start with, what's your day-to-day like at GitHub? Like, what's your day job?

Kinsey Durham Grace  03:13
Yeah, totally. So I specifically work on the Coding Agent Core team at GitHub. So I don't know if you've done this, but if you've assigned an issue to Copilot, that service. Most recently, I worked with a small group of other engineers to build the Agent HQ, is what we're calling it.

Kinsey Durham Grace  03:33
So now you can assign an issue to Claude specifically, or Codex. More to come on there, but that's the most recent thing that I've been working on day-to-day. So yeah, my day-to-day is actually a lot of writing code, and I really appreciate that it's not totally bogged down by meetings and things like that.

Valentino Stoll  03:52
Yeah, I'm sure. And so you just mentioned the Agent HQ, and there was a big announcement about this. This is now, what, seven days ago. So I can go and use Claude and Codex in GitHub. Why and how? Would I do such a thing?

Kinsey Durham Grace  04:08
Yeah, so I guess I'll start with the how. It's just like you're on your issues view page, you see an issue, you're like, oh, hey, that's kind of low-hanging fruit or something that's not super complicated, so I will assign it to Claude. You know, maybe there's other use cases. I feel like everyone had their own almost philosophy of which models they choose and why,

Kinsey Durham Grace  04:29
and like what works for them or what has worked for them in the past. So it's just giving you more opportunity. And I see it as why is GitHub really wants to become, you know, the platform for other models and other companies. GitHub, we're not going to compete in that space as far as building new models and things like that.

Kinsey Durham Grace  04:48
So let's get all of the models that folks are using into more of a platform on GitHub. So that was the reasoning there. And yeah, it was really cool to partner with these companies that essentially can be competitors with Claude code and different things like that, and to have that relationship where we were very collaborative with them.

Kinsey Durham Grace  05:08
You know, we worked closely with their teams, their engineering teams, their product teams, to get them on our platform.

Valentino Stoll  05:15
I was really happy to see this because this was like my dream come true of like, hey, here's an issue, just go do it. And so how close are we to like, okay, somebody coming onto your open-source project and paying for it? Let's say like they want a new feature, and I go,

Valentino Stoll  05:35
yeah, go ahead, approve, but I don't want to pay for this one. I spend too much already on all my agents. Is that something that's even thought about, or?

Kinsey Durham Grace  05:47
Well, like providing almost like free credits almost to like open-source type projects. I haven't heard that be discussed yet, but I think that's a great idea. And you know, GitHub very much invests in open-source and things like that, so I don't see that being too far off. And I could just see that making a huge difference on open-source projects.

Kinsey Durham Grace  06:06
But at the same time, sometimes I feel like those issues that maybe AI could handle are great for people who are wanting to contribute to open-source and don't know where to get started. So a little bit of me is like, oh, I don't want the, you know, AI to totally take over that because I think there's a lot of really great issues that if people are wanting to contribute to a project,

Kinsey Durham Grace  06:26
those would be the types of issues to pick up and kind of get your feet wet in the repository.

Valentino Stoll  06:30
Yeah, I get it. I'm kind of torn on. I originally had you on because I saw your Rails World talk, which is fantastic. Please go watch that. It outlines everything I think a lot of people should see for a holistic view of how to use AI, like responsibly, but also like for your different use cases,

Valentino Stoll  06:51
because there are so many, as you put, and you broke it down really great. So go watch that talk. It's really fantastic. You also kind of mentioned like these aren't like coming to like take our jobs,right? As kind of like your final address of the talk. And I feel like that's more and more what I see is like people trying to just say,

Valentino Stoll  07:13
I still like to code,right? Like I feel like this is DHH's like, he keeps harping on it, saying how much he doesn't like it, and now I see him writing more and more about how much he does like it. So I don't really know. I like the whole like Copilot aspect of things, of something working with me, doing the tasks that I don't want to do,

Valentino Stoll  07:33
but also like a sounding board, like so many of these things of like assistance versus ownership. How do you feel about that?

Kinsey Durham Grace  07:42
I totally agree. I think there are some things, it's kind of rare when I can literally just assign an issue and then it's like done, it's perfect, I ship it. It's like a lot of steering, a lot of guiding, like think about this, you know, it's really not that yet. And for me, it's been learning how to treat the tool as a learning tool,

Kinsey Durham Grace  08:02
instead of just, hey, give me the answer, or hey, just like do this. If I'm just using it almost like a pair programmer more, versus go do my job for me. But I won't lie, sometimes when I have to write tests, I'm like, please go write the tests, and I'll just go look at them. But like stuff like that that I don't necessarily love doing,

Kinsey Durham Grace  08:23
I like being able to have Copilot write those tests for me. That's like a big win. But yeah, most of the time when I assign an issue to it, it's not perfect, I can't just ship it. Some things, if they're very small, but you know, we work in huge, the GitHub monolith is insane. So it struggles a little more there, and you have to steer it,

Kinsey Durham Grace  08:44
you have to guide it, you have to know what's going on, to know like, okay, is thisright?

Valentino Stoll  08:48
Yeah, I mean, I don't know how many times I've, you know, had to course-correct constantly. And I feel like maybe people don't realize, you know, there's a lot of like chat out there of, oh, you can just like have all these autonomous agents go run and do the things, and then you just check in on it. And you can do that.

Valentino Stoll  09:05
You're going to waste a lot of resources as it tries and like thinks about and reasons about every single step of the way, and then kind of you have to review it anyway. So I don't know how much value folks are really getting out of it, other than maybe every once in a while it does exactly what they ask them.

Kinsey Durham Grace  09:25
Yeah, I think it really depends on the types of issues,right? Yeah, the types of tasks.

Joe Leo  09:31
I wanted to voice a gripe. I've been on Hacker News like every day the last 10 days, and every single time there is a person, and it's always a different person, who is about my age, who is writing like this soliloquy. They're writing like the, oh, this is the end of programming as I know it. And I used to type the words to the different code,

Joe Leo  09:51
and that used to make me happy, and now I am sad. And I really am tired of it. You know, it's making guys of our age and ilk, like me and Valentino, not you, Kinsey, you're out of this, but it's making us look bad. I think, you know, what you're describing, the two of you, is a much healthier take on, hey, how do we leverage these tools?

Joe Leo  10:10
Because they're really powerful and they're getting better, which is not, I'm going to have it do everything for me. And it's also not, well, I, you know, get out of here, I'm like the old man yelling at the rain. It helps us to be more productive, which is what we want,right? Don't we want to solve problems? Like that's why we got into this thing.

Kinsey Durham Grace  10:27
Right. And I still feel like I'm solving just as many problems, even though I'm using these AI tools. It's just learning how to use them and when they slow you down and when they make you faster, and you know, running with that. So I am more productive, and I do appreciate that, because maybe I can finish a task quicker.

Kinsey Durham Grace  10:45
And I know I touch on this a little bit in my talk, and you know, I have two young kids, so I can do that, and I can go pick my kids up from school, but I still can meet that deadline, and different stuff like that. So that's important to me.

Valentino Stoll  10:58
Yeah, so I'm curious, like getting back to, like getting back to your work. This is like super impressive, like being able to just assign Copilot to an issue and just like have it work and come back. Like there's got to be a lot of like behind-the-scenes aspects and challenges that you face through that. What are some like high levels,

Valentino Stoll  11:17
what are you still seeing as far as like challenges with that kind of thing,right? And like what has really worked well for you?

Kinsey Durham Grace  11:26
So the service that powers this is a service that's separate from the monolith, but we have to do a lot of work in the monolith. So when you ask me that question, the first things that come up to me are what other everyone else is facing, you know, maybe in big monolith apps is I have DX problems,right?

Kinsey Durham Grace  11:45
Running something locally, getting our local environment working between all these services and all the things has been a really big challenge for us and keeping us from moving as fast as we would like to. And then second, being just the pain points of deploying in the monolith.

Kinsey Durham Grace  12:02
You know, if there's like a rollback, or we have so many people committing to the monolith in GitHub that it could take a whole day to deploy, which for, you know, bigger companies, that's pretty normal,right? But I'm used to running like pretty lean and that sort of thing. Like, oh, just deploy, deploy, deploy. You know,

Kinsey Durham Grace  12:20
we do deploy multiple times a day and that sort of thing. So really just, I think problems that everyone else are their biggest barriers there is just the pain points of working in a big monolith. And it just feels a lot easier when I'm in the smaller service that's behind the coding agent. But yeah, when I have to go back to the monolith,

Kinsey Durham Grace  12:40
which a lot of work is talking back and forth, that sort of thing, you know, assigning to issues, that's all monolith, are all things that we all experience day-to-day working in big Rails apps.

Valentino Stoll  12:50
I really enjoy open-source because I get a break from that.

Kinsey Durham Grace  12:54
Yeah.

Valentino Stoll  12:55
And I can run tests in like under a minute,right?

Kinsey Durham Grace  12:58
Right. And it touches so many things. So I'll have a test fail over here, but it takes a half hour for my CI, you know, yeah, I get it.

Valentino Stoll  13:09
Yeah, so I'm curious, like what are your like agentic strategies? What is your day-to-day workflow as far as how involved you get your AI usage to build the thing that's making it easier for everyone else,right?

Kinsey Durham Grace  13:23
Yeah, I mean, I really try to use all the tools that I can, just to experiment with it and also to give feedback to other parts, you know, of GitHub, because, you know, different teams work on different things. Like we have the S Code Chat, all the things. And so I really do try to use those tools because, again,

Kinsey Durham Grace  13:42
I'm still trying to figure out, okay, where can I be faster and where does this slow me down? So for smaller issues that I know aren't too complicated, I do assign Claude or I assign Codex or Copilot, that sort of thing, to really use that tool and see how it does. And again, I usually have to steer it and maybe make some changes, that sort of thing.

Kinsey Durham Grace  14:02
But I also have been using the GitHub CLI a lot in the code space. We have custom agents that are built into those that I've been using. So it's really nice to, you know, use a custom agent that is specifically geared towards what I'm trying to do. So one of the services I work in is in Go.

Kinsey Durham Grace  14:23
So we have an awesome Go custom agent that my coworker, who's a Go expert, wrote, and it's been really improving my Go and writing it in the way that, you know, he's a Go expert, that he wants me to write it, that sort of thing.

Kinsey Durham Grace  14:36
So, and instead of having to bug him, you know, he's very busy, and I can leverage that agent to help me get better at writing Go and that sort of thing, because, you know, my background is Ruby.

Kinsey Durham Grace  14:47
So that's one of the major use cases that I've been doing besides like assigning the smaller issues, because if it's a bigger or complex thing, like a lot of the stuff we had to do for Agent HQ, I could not assign a new. I get one sentence, I'm going with it, I know what I need to do in my head, but I can't assign it. It's too big of a problem.

Valentino Stoll  15:05
So I'm interested in this like agent aspect that you're talking about. Where is like your coworker that's an expert in Go making this agent, and like where does it live and how do you make use of it? Is this all like on the GitHub project that you share?

Kinsey Durham Grace  15:19
Yeah, so basically you can add custom instructions and you can build custom agents within GitHub, have that in the GitHub CLI, in a code space is how I use it. It's really easy to use. The UI is pretty simple once you're in there. It's like, I can't remember exactly the command because I haven't do it automatically on my machine,

Kinsey Durham Grace  15:39
how I have it configured, but I think it's like slash GitHub CLI or anything if you have it installed. And then you can just say like custom agents and you click the one that you want to use or select the one that you want to use. And yeah, I've found that to be really helpful. And anyone, you know, if you have Copilot and things like that,

Kinsey Durham Grace  15:57
you can build and use these custom agents on your team and tune custom instructions to all that. And it's been pretty cool to use. And then sometimes I'll use the chat in VS Code as well too, but mostly I've been relying recently on the CLI. So they've been doing a ton of work on that and it's been getting a lot better and really helpful.

Valentino Stoll  16:19
Yeah, that's really awesome. Is that something that you can use outside of the code space or do you have to be within the code space?

Kinsey Durham Grace  16:25
No, you can use it outside of a code space. Yeah.

Valentino Stoll  16:28
I honestly haven't looked at this yet, so I'm pretty excited.

Kinsey Durham Grace  16:31
Yeah, I'll send you a link and we can post it. And yeah, it's a newer thing. And in the monolith, we have various ones that are for helping you with like bigger Rails applications or different things like that. I don't know, there's a few in there that folks are using too.

Kinsey Durham Grace  16:46
So yeah, seeing that catch on and just seeing how useful those specific ones can be has been helping my day-to-day. But yeah, but also just good old, you know, chat VS Code sometimes. And you know, now with like OS 4.6.

Valentino Stoll  17:02
Yeah, full disclosure, I am a heavy Claude code user.

Kinsey Durham Grace  17:06
Okay, no, that's totally.

Joe Leo  17:07
It's been disclosed more than once on this show.

Kinsey Durham Grace  17:10
Yeah, I think I watched an episode where he mentioned.

Valentino Stoll  17:13
I like to explore. I'm an explorer. So I'm going to give this a shot, because I do find my workflow tends to cater toward like very specific instruction sets, which then become like, hey, go review this PR in this way. And we're in a different repo, so maybe that review is different depending on which repo that it's in.

Valentino Stoll  17:35
And so I have these various commands that live within the repos themselves, and maybe an agent would be better use case for that.

Kinsey Durham Grace  17:43
Yeah, I think you should try it. You should try it and let me know what you think.

Valentino Stoll  17:46
It's sounding more of a biology. Yeah. That's actually one thing I dislike about it is I have to be like an active participant for those like kinds of things. I could send them to a background agent to work on, but I like to see what's happening. Is that something that I can get out of an agent as well as kind of peeking at their activity?

Kinsey Durham Grace  18:04
Yeah, so we have sessions. We have like a UI where you can literally read through everything that's syncing, everything that's doing. Part of, I think, responsible AI and that sort of thing is gating and also being transparent with the user of what's happening. And yeah, because I learn a lot going back through that stuff a lot of the time, you know, or just seeing how it's reasoning about something, I'll learn something new.

Joe Leo  18:23
I have a question about, and I think it relates to this. And Kinsey, in your talk at Rails World, you talked about how it's a good practice to make your agents modular and maintainable. And that stuck with me. I agree with you. And then I thought, okay, you know, we're talking about using agents while we're coding.

Joe Leo  18:43
So that, I think, maybe Agent HQ is a way to make that modular,right? Because I can switch out my agents,right? I can plug and play. When it comes to using an agent in or embedding it in my software, I think it gets a little bit more difficult, or at least it's harder for me to reason about. So I think about modular and maintainable,

Joe Leo  19:03
and I think, okay, I know how to make a library modular and maintainable. How do I make an agent modular and maintainable? And so what are the things that you try to do? What are the practices you put in place?

Kinsey Durham Grace  19:14
Yeah, I think that kind of following like similar rules and guidelines and things that we are used to. But a big thing for us with Agent HQ is it's pretty manualright now. So how we built that, it was very much like we had to do a ton of work,

Kinsey Durham Grace  19:35
a ton of configuration, and that's not going to scale,right? So as we have more third-party folks joining the platform at GitHub, you know, Jules or Gemini, who knows what will be there.

Kinsey Durham Grace  19:49
We're actuallyright now totally kind of rebuilding what we did for this MVP to make it so that it's modular and maintainable in more of an engine fashion. So it is plug and play. So even when I think of Rails engines and different things like that, it's very similar. So having it so we don't have to do all this manual configuration,

Kinsey Durham Grace  20:10
how can we put this on the user so that they can give us something, they can give us our SDK, and we just plug it in? So that's actually something that we're working onright now, and it's exciting to kind of pull apart, like, okay, where can we make this more modular so we don't depend on this configuration here,

Kinsey Durham Grace  20:31
we don't depend on this torp call to the monolith, blah, blah, blah, blah, blah. So, but I think following similar guidelines that we do when we're pulling things out, I don't know if either of you have done that, like pulling things out into like an engine in Rails or anything like that. It feels like I'm asking myself the same questions, I guess.

Valentino Stoll  20:49
Yeah, it's funny. This is a hot topic at Gusto is extracting from the monolith.

Kinsey Durham Grace  20:54
Okay, so yeah.

Valentino Stoll  20:56
Definitely, I feel like in the past five years has been a spreading phenomena of, I guess, distributing the monolith in a more modular way. I forget what they call it now. It changes.

Joe Leo  21:12
You're not calling it service-oriented anymore?

Valentino Stoll  21:15
No, I think modular monolith. I've heard.

Joe Leo  21:17
Modular monolith.

Kinsey Durham Grace  21:19
Oh, there you go. Yeah, because the custom monolith is pretty big,right? It's probably comparable to.

Joe Leo  21:25
Yeah, too, I would imagine close. Enormous monoliths, which is interesting. And of course, there's a lot of reasoning that goes into that,right? I mean, it's a choice. Like, you know, some companies end up with a monolith, others make a deliberate choice to.

Valentino Stoll  21:37
I've actually worked through many, I guess, breaking apart of monoliths, and they all end up in the same way. It's all like a data thing. Where does the data live and how do we share it? Then you just end up with like the monolith data. And so I feel like maybe there's something broken there. Maybe that should be like step one,

Valentino Stoll  21:57
figuring out how you're going to share data easy and then just make it easy to do that. Well, what does the Kent Beck make the change easier than make the easy change? Something like that.

Kinsey Durham Grace  22:09
I've heard this quote, but I don't remember the exact words.

Joe Leo  22:12
At the 10,000-foot view, I can't help but notice that with Agent HQ, there's this move, and I think from a business perspective, this makes a lot of sense. You know, everybody wants to have the platform, and you know, you've said that word yourself a couple of times here, and GitHub already is a platform. But everybody wants to have the platform for AI because they want stickiness for their users.

Joe Leo  22:33
And it's become pretty apparent that there is no stickiness for individual AIs,right? Individual LLMs. People change them as often as they change their shoes, except for Valentino, who's Claude code for life until the next episode when he tells me about a different one. But it sounds like the Agent HQ is a play to say,

Joe Leo  22:52
okay, well, yeah, bring whatever agent you want. We want this to be the playground, whether it's the orchestration layer or it's the HQ. We want this to be your headquarters for where you develop. I mean, you don't have to tell me what the business reasoning is, but does that apply to your building?

Kinsey Durham Grace  23:08
Yeah, I think so. You know, your code already lives there. Everything's there. Your issues, or if you use that, your pull requests, everything. So yeah, just have it all on one platform to use. And, but yeah, I know that Copilot has been a little more dominant in the enterprise space versus like individual users and things like that.

Kinsey Durham Grace  23:27
Like, I don't know if Gusto pay for Copilot. Oh, they do. Okay.

Valentino Stoll  23:32
Yeah, we have Copilot. Yes. Gusto probably pays for them all. We very much like cater toward exploration. We have Gemini access. Yeah, ChatGPT. These aren't agents, but we have all the agents.

Kinsey Durham Grace  23:45
Right.

Valentino Stoll  23:46
Cursor. I feel like some people might use AMP. I'm not sure.

Kinsey Durham Grace  23:51
Well, that's cool. They provide all those options. It's awesome.

Valentino Stoll  23:54
Yeah, it's more just to see what works better. I don't know if it'll stay like continued support that way forever. I feel like we're kind of honing in on what is working good for certain use cases now and leaning into that more, as I imagine, like a lot of other people are. It makes a lot of sense to have your agent where your code is.

Valentino Stoll  24:16
And so I think that there's a lot, and this is one thing I'd like to maybe hash out more with you on your thoughts here. It seems like there's a lot of like observational and reactive, I think you called it reflexive agentic use.

Valentino Stoll  24:32
Something is watching what's happening with your code base and kind of like preemptively performing actions. Do you see this evolving more? Is this challenge kind of just like too big for now?

Kinsey Durham Grace  24:47
Yeah. Do you mean like how Claude bot can like have all these PRs ready for you in the morning type thing?

Valentino Stoll  24:52
Right. If it notices a lot of churn on a particular file, maybe it like tries to find a better design because it sees a lot of changes, or it can reason about like you're trying to call services too much, and that's like you're trying to reach into the data in one object,right?

Valentino Stoll  25:12
And that's a no-no. Like kind of just observing like maybe software best practices as things are changing and like making suggestions, stuff like that.

Kinsey Durham Grace  25:24
I am really excited about that just because I feel like that's a really big gapright now. And a lot of times, even when I am working with AI and using that, it is I have to tell it to do theright, like it doesn't just write a test for me. I have to prompt it to write a test. It doesn't follow best practices unless you tell it to. And a lot of the times I'm like,

Kinsey Durham Grace  25:43
no, that's not good code. It's like, you can do better than that. So I'm excited for that. And I know there are certain things, like we had a lot of, you were talking about, you know, Redis falling over. We had, you know, a similar issue this week at GitHub, which I'm sure a lot of you felt. And yeah, you know,

Kinsey Durham Grace  26:02
help with that or agent with that, that could like alert us to that, like, hey, this is like going to be a problem really soon. You know, I think that's really exciting and something that we should be investing in, you know, agentic workflows, that sort of thing. But there's always that other side of the coin where it's like, oh, you don't want it doing too much. And kind of where that balancesright now.

Kinsey Durham Grace  26:23
And hopefully the models are good enough when it can do that kind of stuff. So.

Valentino Stoll  26:28
I'm just doing baby stepsright now where I have like pre-commit hooks that are like, hey, make sure that I'm not committing a thousand lines of change.

Valentino Stoll  26:40
I'm like trying to use it to like combat AI, which maybe isn't a great idea, but that's kind of me being a little bit proactive deterministically. Yeah, I'm more curious about the observational stuff. Trying to remember there. Mastra, Mastra AI, they just announced their observational memory.

Kinsey Durham Grace  27:01
Okay, I'm going to have to look in. Because yeah, I would love to wake up in the morning and have like an issue or instead of like a pull request or whatever, just like it created an issue for me that tells me all these like pain points and like a high level, like, hey, you should. Yeah, I think that would be really cool.

Valentino Stoll  27:16
And especially, you know, if you're working in the GitHub ecosystem, it has your activity. And so like it has the activity of the repos. Like every object has activity related to it.

Valentino Stoll  27:25
And it seems naturally that this would be a thing that just like falls in the place of, okay, here's the observation that we noticed and we're going to do these things because we know you're going to have to do them anyway tomorrow.

Kinsey Durham Grace  27:38
I love it. Hey, I'm going to go pitch thisright now.

Joe Leo  27:40
It's interesting that you bring this up because the things that I noticed from OpenClaw when I started, I set up an experiment with it is that they have two really easy concepts, the heartbeat, which is really just like a cron tab, and then polling. And it costs a little money to be polling and have this thing kind of constantly looking for changes.

Joe Leo  28:01
The heartbeat is a little bit less, I don't know, ever present. But those things are pretty easy to set up,right? And that can be, of course, you still have to do all the things that you both talked about. You have to tell it to do theright thing. You have to give it guidelines. You have to put in constraints. All of a sudden, that seems very doable.

Kinsey Durham Grace  28:18
Yeah. I'm excited for what that looks like in the future. And it's, again, like that is very collaborative. That's not it taking your job. It's like helping you, like that pair programmer, you know, like bounce ideas off of or like, how can we be better? How can we solve problems better? And that's exciting to me. But again, it's like not it totally being able to do that for me.

Joe Leo  28:40
Kinsey, I also like when you said, well, you know, you're kind of telling your LLM, like, you could do better. One of my favorite things to do is express my disappointment at this inanimate object.

Kinsey Durham Grace  28:50
How does that look like?

Joe Leo  28:52
I mean, usually I'll take a snapshot of it and paste it on LinkedIn or something because it's usually me just being like, come on, don't tell me this. You know, we've been through a lot together and it's always so apologetic. So it feels good.

Valentino Stoll  29:02
If only you could send it memes and it would know.

Joe Leo  29:07
Yeah, and it could interpret that and be like, yep, allright, my blood.

Valentino Stoll  29:10
Yeah, you're onto something there.

Joe Leo  29:12
Now you mentioned observability. I'm curious, Kinsey, because this came up in the talk also and I don't know of a third-party tool that's great at this yet. Do you use in-house observability tools at GitHub or are you using something off the shelf to see, to kind of monitor how your agents are performing?

Kinsey Durham Grace  29:30
Yeah. So we use a lot of smattering of things. You know, we have a lot of dashboards. We use Custo, Datadog, various things. So, but yeah,right now it's just like Kusto dashboards that we're looking at. So I guess they're in-house technically because they're Microsoft.

Valentino Stoll  29:45
Allright, that's good to know. Thanks.

Kinsey Durham Grace  29:47
Yeah, I'm excited for kind of advancements and, you know, using AI in that space too.

Joe Leo  29:54
Now there was an interesting headline that caught my attention yesterday that a former GitHub CEO launched a product called Entire and this new open source application. And it's called what? It's called, these names are too generic. It's called something like Collective.

Kinsey Durham Grace  30:15
What was it like? I don't know. I know, yeah, the company's Entire. I can't remember the name of like the platform.

Joe Leo  30:22
Yeah, I'll find it in a second. And so the interesting thing I thought is that they're trying to capture intent, which is actually similar to what friend of the show, Obi Fernandez, announced with his product just a couple of weeks ago. And it seems like there's this move to say,

Joe Leo  30:41
okay, we're all working with these agents and we do this individually, almost like a pair programmer, as you've brought up, Kinsey, which I agree with, except that pair programmer kind of keeps all this stuff to themselves. And so you, let's say each of us was on a team, we'd each individually interact with AI, but the context of that interaction gets lost.

Joe Leo  31:02
And the idea is like, well, what if we could be more collaborative across our agents? Now I'd be very surprised to hear that GitHub isn't working on something similar already. And I'm just curious to know, like what is your personal take on what else other than code can be persisted as an artifact of software development?

Kinsey Durham Grace  31:23
Honestly, a lot of times things that you get in like ADRs or just the back and forth, maybe like you were just saying that I have personally with Opus in my VS Code chat is like the trade-offs and those conversations that we're having there. And that's all lost. Like why did you make that decision in the code? Why did you do X?

Kinsey Durham Grace  31:42
Why did you do Y? And obviously you're not going to just like write that in a comment in the code base. So I feel like that very much gets lost. I don't know. I definitely think there's a better way to do that. And it's cool to see this kind of concept emerging because collaboration is, you know, very important in software development.

Kinsey Durham Grace  32:02
And again, like taking it back to learning instead of just treating this, I feel like a lot of people can just think of it as like, oh, it just does your work for you, boom. But no, it's really about enhancing collaboration and how we do that. So if we can, you know, improve our tooling and the tools we use and the systems that we're building around these models to do that,

Kinsey Durham Grace  32:23
I think that will be huge. And like I'm learning, I'm learning like how to reason about a problem or better architects. I don't know. I'm curious.

Valentino Stoll  32:31
I have a lot of questions on this topic.

Joe Leo  32:34
Let's go. Yeah, this is something I am actively like trying to solve personally. What I keep coming back to is like encapsulating all of this data in the repository itself. But then how do you deal with the cross-repo knowledge gap and sharing that? And where does that data live?

Joe Leo  32:54
And everybody doesn't need all of it,right? Also, and how does like search work? Like everybody searches different too. And so what it filters and what it's searching for, you might want to be the same for some things and not others.

Kinsey Durham Grace  33:10
To clarify, when you say data, do you mean like the decisions behind?

Valentino Stoll  33:14
The decisions.

Kinsey Durham Grace  33:15
Okay. Yeah.

Valentino Stoll  33:16
Yeah, exactly. Like the technical decisions, but also the business decisions. Like if the stuff technically in the repo is really supporting a business product or other product, what those decisions were, like there's a reason why the code exists to begin with. And it's usually not the code,right?

Valentino Stoll  33:34
And so it's easy for like an agent to get hung up on all of the decisions and existing code that's there or even where it wants to go, but it doesn't necessarily make sense from a product level,right? And so having those decisions is important under certain circumstances.

Valentino Stoll  33:50
So like where do you start to tie those things together in a cohesive way? I feel like isn't really the best for all use cases of the agent. And so I'm curious, where do you start to see those swim lanes for pulling together the kinds of information for theright kinds of tasks?

Kinsey Durham Grace  34:11
Yeah, I agree with you because I feel like that context is lost a lot of the time and our LLMs could be even more powerful if they had that. So yeah, I think we're also like focusing on memory and how we improve that across the board with these systems we build around the LLMs.

Kinsey Durham Grace  34:31
And then also like you were saying, I think that's really interesting to like have that, yeah, stored in a repo or being able to, you know, with like MSEBs and stuff like that, you can have it look at like a Google Doc that has like an ADR behind decisions and stuff like that.

Kinsey Durham Grace  34:47
But yeah, I'd be curious to hear if folks are like finding cool ways to do this and bring all of that knowledge in because yeah, how much more powerful would it be?

Valentino Stoll  34:57
Yeah, you know, my current exploration is like having a repo local fact storage. So likeright now it's tied into Claude code in a hook. So like every single change gets captured into and extracted as like a fact, an entity.

Valentino Stoll  35:16
And so I can search on it later. And then it stores also global stuff. So it'll notice as I'm working through that this isn't just repo specific and maybe store it in a global database, which is just in my home folder. And so I basically end up with these like two SQLite databases that uses Stephen Marham's great,

Valentino Stoll  35:38
was it ExtraLight database? So I can get like multiple database searches to like blend the answers together. So as it starts, anytime it like makes another request, it'll go and it'll search for any facts that are related and inject those into the context automatically as like a hook mechanism. It's fun, but I was like, you know, I made a benchmark.

Valentino Stoll  36:00
It underperforms like Toby Lucke's QMD, if you're not familiar, which is just like a really great project, but like has very opinionated ideas towards like what it's extracting and when and like all of the crazy vectorization stuff. It runs very slow on my MacBook. It underperforms in terms of speed.

Valentino Stoll  36:21
It overperforms in terms of quality of responses. So meaning it finds theright bits of knowledge to inject for the context. But personally, when I use both, I don't necessarily see a difference as far as like what value that the actual improvement in the knowledge made toward it because they.

Kinsey Durham Grace  36:40
Turns out you're getting back or the collaboration is improving.

Valentino Stoll  36:45
It's improving in that way, but the quality of the knowledge that gets injected into the context while you're using it, that doesn't necessarily have as big of an impact because the agent will still make use of other tools to read about the repository and do all these other little things that kind of fill the gaps anyway.

Valentino Stoll  37:04
And so what I'm finding is like more of like, as long as you can have a way to like surface these facts and entities and things that can like organizationally represent different things than what the agent itself can surface, I feel like that's where like the real value I've started seeing.

Valentino Stoll  37:24
And so it's like a challenge for me personally because I don't want to have to maintain and like think about these things. So like I'm hoping that somebody somewhere just makes something I could snap in and just suddenly, you know, my agent is working great. And I feel like it's not like an easy thing to solve.

Valentino Stoll  37:42
And so I've been doing a different mechanism now where I'm like taking an agent off the shelf and giving it a knowledge corpus that I've pre-trained it on other stuff using like curriculums and stuff to then use that to like try and be a better version of itself for,

Valentino Stoll  38:01
let's say, Ruby or Ruby on Rails or like Go, all these different domains that we have. And I found that like that aspect has like better performance for those specific tasks. So I'm wondering like, is that something that you want to pursue too? Like does it make sense to even do that or will these models just get better?

Valentino Stoll  38:22
And then like all of this time of mine is wasted.

Kinsey Durham Grace  38:25
This sounds like what we were talking about earlier with custom agents,right? Like I have my Go custom agent that I use to, you know, help me write Go better specifically. And it is specifically trained, has the context, has the knowledge built into that to guide me in that direction. And so that's why I use custom agents. So I think it's a similar,

Kinsey Durham Grace  38:45
correct me if I'm wrong, like a similar ask or need there. Just, you know, very.

Valentino Stoll  38:50
I think what you're saying is just go use GitHub agents already, Valentina.

Kinsey Durham Grace  38:53
Custom agents, yeah. Yeah, no, I've just personally been using them. I like that you can give them that specific information or tell them, you know, just like you're saying how you want them to be, how you want them to act and using those. And that's, you know, the beauty of agents,right? They can have that principle where they just focus on SRP,

Kinsey Durham Grace  39:14
single responsibility, and they just, that's their job. And, you know, then getting into like orchestrating all of those and just having all these different agents that are good at different things like teams, you know, to help you.

Valentino Stoll  39:24
So circling back to the knowledge aspect of things, is that something configurable? Is that something that you can add like a new kind of knowledge or memory like adapter to these agents?

Kinsey Durham Grace  39:36
Not necessarily like a specific memory adapter, but I know that you can build them using custom instructions and providing them with context and that sort of thing. So, but yeah, that's definitely something I think that's coming down the pipeline, at least at GitHub, is really just like this focus on memory and what does that look like?

Joe Leo  39:58
What I think about with the custom agents is that that's really helpful. And I want you to correct me if I'm wrong about this. This sounds really helpful for me personally as I start to work on a project. What I'm curious about, and I think with Valentino, I think your project is geared towards,

Joe Leo  40:19
although it's not there yet, would be, okay, Valentino has been building this project for a while. Now, Kinsey, you are a new developer on this project. Today is your day one. What will you do and how will you meaningfully contribute to the project? Am I getting thatright, Valentino, that that has? And then I would want to hear your from you, Kinsey.

Joe Leo  40:39
Okay, so what would you do today? What's your go-to?

Kinsey Durham Grace  40:43
Yeah, I think that custom Go agent that I was talking about in the Go space, just because it can help me write Go in the way that I want to, that matches how it's supposed to be in this project, the same philosophies.

Kinsey Durham Grace  40:57
But I also would use even just the CLI in general or VS Code chat just to ask it questions. I'm like, hey, can you explain to me what this does? Like, can you explain this part of the code? Can you give me like a high-level overview of like how this works?

Kinsey Durham Grace  41:15
And I've definitely asked questions like that to even in the Copilot chat, and it's helped me get up to speed quicker in code bases. But it's also good for me to kind of go in and double-check that it's telling me theright thing and it's not just making an assumption.

Kinsey Durham Grace  41:31
But yeah, I definitely would be curious if either of you have done anything like getting up to speed different besides just like asking it questions like that or using.

Joe Leo  41:41
I have not gone as in-depth as Valentino. I love that project, Vi. What we started with, and this is like almost a year ago, you know, we started by saying, okay, let's have this agent write up some ADRs. That's a good practice that we know. And yeah, it was cool. But now, you know, looking back, it's pretty naive because it's assuming that a human is going to read this.

Joe Leo  41:59
And no human ever read an ADR, and no human is going to read an ADR now that the LLM can read it for you. So we were making these succinct and concise and, you know, human-readable. And so you could look at bullet points. And, you know, I'm being a little facetious. Of course, people can look at it. But nowadays what I tend to do is I say, okay, write your reasoning, write what got us here to this point.

Joe Leo  42:19
Just write it down in a doc, put it in a markdown file, and put it under docs or whatever so that there is something for me or the future developer to go back to and say, okay, I was trying to do X or I'm doing X, and it's a little bit like what Joe did when he implemented Y. So let me have my LLM go and read this doc. Something simple like that.

Joe Leo  42:39
And that's imperfect, but it's also very easy. It's something you could do just as kind of like a cleanup task. You could even put in a hook if you wanted. That's as far as I've gotten.

Kinsey Durham Grace  42:48
Yeah, we're still very much, especially on my team, we've been trying to write more ADRs, that sort of thing. So yeah, just how we bring all that context in.

Valentino Stoll  42:58
I would say just as Joe alluded to, I have this no-code solution called the AI Software Architect. It basically does just like anytime you're working on something for your planning mode, it'll create an artifact. I call them ADRs. And so it'll create an artifact for what it plans to accomplish for that session.

Valentino Stoll  43:19
And so it's great for like feature development, but bugs also.

Kinsey Durham Grace  43:23
How does that show up? Does it like export a PDF or how does that work?

Valentino Stoll  43:26
It shows up, you know, as you're actively working in the agent. It's not the best UI, but it just like dumps the actual whole thing. You can open up in a file and make edits to it or ask it to make edits or explain things. It slows down the process so you're not just like jumping into AI slopright away. And so I like it personally because it just like slows down like what it's going to generate.

Valentino Stoll  43:48
And it helps me refine what I want it to work on better. I've gotten it to the point where these documents are not so like verbose. At first it was like, hey, here's like a three-page document you should read. I would just be like, no, that looks good. And then when it starts going wrong and implementing the wrong thing, it's like, well, it's obviously implementing what I told it to do.

Valentino Stoll  44:09
And so at least there's a document pointing to, oh, well, I can adjust the document, then delete everything that is started and go back to the beginning because like it's always easier to just start fresh. But what I have liked is that as you start to use and keep track of these artifacts, it starts to reference them. And so I set it up in my like agents file.

Valentino Stoll  44:30
Hey, if you're wondering about a technical decision or why we did something, just like look in these ADR folders and you might find what you're looking for. And obviously it's like, does it scale well? Because like if you get the hundreds of those things, like it's not going to search through files the best way that way. But what it is nice,

Valentino Stoll  44:50
when you have like these very broad like general things, you can keep these documents pretty small and it does help to just point to, hey, I'm working on this very particular thing. Like can you check the ADRs to see if we have anything? And it does work great that way. So it makes me wonder like,

Valentino Stoll  45:09
well, what kind of artifacts do you use and like lean into to make your like agentic experience better personally?

Kinsey Durham Grace  45:17
Yeah, I think sometimes it's a lot of learning from other folks on my team and also just having that knowledge of working and seeing things that have worked in this space and haven't. But yeah, I think it's still very much a manual thing where, you know,

Kinsey Durham Grace  45:36
we do have ADRs and that sort of thing, but I feel like I more rely on learning from other folks on my team who have a lot of experience doing that. Just like before, to be honest, they don't have like a cool fancy answer like you do.

Valentino Stoll  45:49
Yeah, I mean, I would love if you could like from your coding agent just be like, oh, I wonder what my coworker thinks and like sends it off to an agent and then the agent brokers that to your coworker in like a concise way. Hey, Kinsey's wondering about how this thing in Go works. Yeah, no user interaction.

Valentino Stoll  46:13
That makes me wonder too, like the human element of all of this. Do you worry about that? Do you see it like dissolving it in any way?

Kinsey Durham Grace  46:21
I still like having these like architectural discussions about, you know, how we're going to build this with my coworker over a Zoom call because I'm remote. So I also just don't want that to go away, to be honest. And I still enjoy that part of my job a lot. So.

Joe Leo  46:38
Yeah, I don't expect it to go away for what it's worth. I expect that the conversations over things that we currently spend a lot of time thinking about probably will go away and they'll be replaced by us thinking about higher-order problems. That's my guess. I'm a little bit of an optimist.

Joe Leo  46:54
I even see a lot of potential for this kind of system where context is persisted because it actually can make us more collaborative. Likeright now, the only thing I want to collaborate with is my LLM. But if we all have LLMs and they're all kind of doing some of the same things and sharing context, well then all of a sudden it brings me closer together to my teammates again.

Joe Leo  47:14
Maybe, maybe. Kinsey, I know we're running up on time. I had one question for you, which is a little different, but I was hoping you could answer for us. You know, there's a lot of engineers for good reason. Look up to you and look up to the work that you're doing. And, you know, I think you have one of the dream jobs. And I'm curious what skills today,

Joe Leo  47:35
engineering skills, give me one skill that matters more than it used to, let's say three years ago before this popularity in AI, gen generative tools exploded, and one skill that matters less.

Kinsey Durham Grace  47:48
Ooh, one skill that matters more. Honestly, with all of this, I think just in general, being a more well-rounded developer is going to be like, there's going to be a higher bar there because it's not just someone who can just sit there and like crank out code, like line for line, like horsepower. I know the syntax in and out.

Kinsey Durham Grace  48:08
I don't have to look anything up. That's going to be less of a skill that's needed. Versus now you need to be able to reason about the product. You need to think about the higher picture of like observability. How can this break things down the line? Like best practices. How can I communicate with stakeholders? How can I communicate better with other engineers?

Kinsey Durham Grace  48:26
Just being more well-rounded in that aspect and not just like kind of in the dark corner, just like cranking out code. And I just think the bar is going to be there that you need to also be able to reason about product and other things like that.

Joe Leo  48:42
You answered two questions with really the same answer, which I think makes you good for consulting. So if you ever want to jump out of GitHub, you give me a call. Valentino, any questions from you? Any final questions?

Valentino Stoll  48:54
One thing, it seems like GitHub has embraced Ruby and Rails within all of this AI stuff. And so we talked earlier this year about like wishes we had for Ruby and Rails ecosystems. Do you have any wishes of the Ruby and Rails ecosystems as far as what's missing from AI?

Kinsey Durham Grace  49:15
I definitely think it's not the first thing that people reach for when they're building things. You know, as I mentioned, like in the service that I'm working in, that back to coding agent, it's in Go. It's not in Rails at GitHub. So I think the tooling is a little bit of a step behind like the official like Ruby SDK for MCP or whatever.

Kinsey Durham Grace  49:35
That was like way after Python and TypeScript and things like that. So just making sure that we are thinking about this stuff and making sure that our open-source community follows that and that we're just remaining a key player in the space.

Kinsey Durham Grace  49:52
But I've heard that there's a big resurgence with AI and Ruby because it's really good at like syntactic sugar. I've heard all the different things. So just making sure that we're staying.

Valentino Stoll  50:03
You've heard is correct.

Kinsey Durham Grace  50:05
We don't want to fall behind. But I don't know exactly what that looks like, so.

Valentino Stoll  50:10
Nobody does. Just keep pushing.

Kinsey Durham Grace  50:13
Yeah, there you go.

Valentino Stoll  50:15
And please keep Ruby weird. I love that track on the new RubyConf coming.

Kinsey Durham Grace  50:20
Yeah, that's a track a lot. I feel like since the conference has gone, it's sad. So yes, I love that.

Valentino Stoll  50:27
Will you be in Las Vegas?

Kinsey Durham Grace  50:28
I don't know for sure yet. I have a lot of travel. So will you be there?

Joe Leo  50:35
I'll be there with the DevMet the team. Yeah.

Kinsey Durham Grace  50:38
Oh, cool. Awesome.

Joe Leo  50:40
I'm trying to get Valentino to come. If we can ask him, if I can get every guest to ask him from now until July, and maybe we can get him out there.

Kinsey Durham Grace  50:49
Awesome. Well, thank you both so much for having me. It was super fun to talk. I learned a lot. So thanks. It was great.

Joe Leo  50:55
Yeah, it was great having you on the show. Thanks so much, Kinsey. We'd love to have you on again to talk more. Yeah, tons of questions.

Valentino Stoll  51:02
Like me too. Everybody out there, go check out these GitHub agents, custom agents, custom agents. This sounds great. Exactly what I need. So.

Kinsey Durham Grace  51:13
Cool. I'll be curious.

Valentino Stoll  51:14
I'll send you questions.

Kinsey Durham Grace  51:16
Yeah, totally. Feedback.

Valentino Stoll  51:18
Allright. Thanks for coming on, Kinsey. This was great. Thanks very much. Bye, everybody.

Kinsey Durham Grace  51:22
Bye.