SPEAKER_01 0:00

Hey everybody, welcome to another episode of the Ruby AI Podcast. I'm your host, Valentino Stoll, and joined by co-host Joe Leo.

SPEAKER_00 0:08

Hi, I'm Joe Leo. I have one question for my co-host. Whose contract is larger? Juan Soto's contract with the New York Mets or your contract to be an AI engineer at Gusto?

SPEAKER_01 0:23

You know, I wish I could be categorized with all these AI engineers. Unfortunately, my frame of reference is not deep enough in the ML aspect of things to qualify, I think, for $100 million. Yeah.

SPEAKER_00 0:38

Yeah, that seems like it seems like a matter of positioning. They teach you this when you're going out to the market with a product. It's like, okay, well, if people view you as this, then maybe the product's only worth like $10. But if they view you as this, you get a $200 million signing bonus.

SPEAKER_01 0:54

And it's all marketing, right? That like it's all marketing. It should just be taught in uh all high schools, right? Part of the curriculum.

SPEAKER_00 1:02

Yeah, exactly.

SPEAKER_01 1:02

I've how to market yourself. Yeah. How to market. That's God, that's horrible.

SPEAKER_00 1:08

I am thinking, like, I came across this thing. It's called the Metis list. You know, it's the world's top AI researchers. I'll put it here in the show notes. And so it's just ranked list. And you know, you can see like Ilya there at the top, but then there's also all these names. So I put my email in. I don't know what they're going to send me, but I now can see the top 100 AI engineers ranked. And it's like trading cards. You can try and catch them all.

SPEAKER_01 1:38

Well, so what are you gathering up from all this? Just like how many hundreds of billions of dollars are in this list?

SPEAKER_00 1:45

Or yeah, you know, I think it's interesting to note the disparity. I'm not begrudging anybody on this list. I want to be clear about that. We have talked on the show before, I think, about how lucky we are, who are guys of people of a certain age who came to software development from a place of like, hey, this is interesting, and I could probably make a decent career. And then all of a sudden, a few years in, software development became the hottest thing. And all of the salaries increased and all of the demand for us increased. No real work on my own part. You know, the industry changed. Now, nobody ever offered me $100 million, but I was able to live a nice life. And so I think that's great. Now I'm seeing this kind of turning. And, you know, it's gotta be, there's obviously hype in this, right? Nobody's worth $100 million. It's a matter of scarcity and demand on the parts of these companies that have hundreds of billions of dollars at stake.

SPEAKER_01 2:48

When I first saw these deals come out, like Facebook or whatever, you gotta think all these deals have to be tied up in equity, just like they would be if you were doing a startup, right? Yeah. Are these deals really like that incredible from like that perspective, right? Or is it really just like the same deal with just more stock?

SPEAKER_00 3:09

Like that's a great question. And that also is a good alignment with sports because sports deals are always, it's all about the number, right? And then when we dig in, we're like, well, this is really just a $20 million deal with a bunch of avoidable years at the end of it. And yeah, I think the probably the same thing is true here. It's like, yeah, you get a couple hundred million dollars if Meta wins the race, which nobody actually knows what that looks like. Nobody I've talked to, maybe I should talk to these hundred people that are the smartest people ever. But nobody knows what it looks like to win this AI race or why nobody's been able to satisfactorily answer me why it matters that one company does win. Because so far, with the advent of LLMs, we've seen either incremental or tremendous improvements. But after each improvement, there's always a short period of time when the next model by the next company catches up. It doesn't seem like there is a huge gap or a huge way that somebody can build a moat and become the LLM that the human race uses.

SPEAKER_01 4:13

I feel like recently kind of proven with like the Deep Seek releases, right? Like everyone's just like, oh, well, like DeepSeek now is like just as good as OpenAI. Like everybody's gonna use it, and it's just like not the case. Yeah. Well, that's true. That's true. And not that DeepSync isn't useful. DeepSync has, you know, has great features and tasks that are really well performed by it. But yeah, I feel like it's all about the vendor lock, right? And like once you get people using it, it necessitates them continue using it.

SPEAKER_00 4:49

I think that's a good point. And we were talking about this right before the show started that what OpenAI built is a consumer tech product. And the efficacy of the LLM behind it really is secondary to the experience that people have using it as a chatbot. So that deep seek example is a perfect example. You and I might go out and use the best models for whatever job that is at hand, and we might evaluate the different models and we might even find a way to objectively determine which one is best. But most people are just reaching for open AI because that was the early leader, and everybody went out, myself included, and got themselves an open AI subscription so they could use Chat GPT.

SPEAKER_01 5:36

We've talked before about okay, well, how do we test out, you know, all of our processes on different kinds of models or like watch what might happen? Like, how are you doing that with Phoenix today?

SPEAKER_00 5:50

The first thing we did, and this is now going back at least eight or nine months, and you know, the whole history of Phoenix is just one year. It was called the September experiment when we started. So we're coming up on one year. We built a test harness that would run test generation against a number of different repositories. We had a couple of open source repos, we had a couple of deaf method repos that we threw in there. I think we had a customer repo that consented to have us do it. And then we could run it against all of these different models. That still exists, but we've found it lacking in the kind of efficacy that we needed because there is so much nuance to the test generation process. And so it's great to say, okay, this works. This one is the best model for generating specs end-to-end for a code base. But, you know, we just got off the call with a prospect and they've got 1.2 million lines of code. We're not going to just go and generate tests for the entire application. It would, you know, it would cost an absolute fortune. It would take, I don't know how long. And it's not necessary, right? We want to start with high flog scores, high complexity, high dependencies. And so then it becomes like, okay, how well does it do in these different circumstances? And then you throw in what dependencies are being used in that test, what needs to be mocked, what factories are available versus what is not available. And I find it, maybe my engineers don't find it like this, but I find it to be an overwhelming number of variables to accurately say, okay, this one is the best. And so then what happens is there's a lot of circumstantial evidence. Well, this one worked really well for me. So, like, this is my guy, ChatGPT5. That's my guy. But I talk to somebody else, and their guy is, you know, Gemini 4. So it's hard to know.

SPEAKER_01 7:43

You make a great point. Like it's funny, like, if you dive into the open source large language model world, the first thing you learn is that every model has its own unique way of crafting prompts. There are tricks that you can add to each different one that help signal the model in different ways. Right. Right. Right. And so, like, the same is true even of these bigger ones, right? Like your GPT-5 prompt probably might not perform as well as like a O3 Mini or something, just because it was structured in a way that reasoning models would benefit from it.

SPEAKER_00 8:24

Right.

SPEAKER_01 8:24

Right. And like maybe the GPT-5 model isn't as good at reasoning about the reasoning section of the prompt. And then like you're just scoffing it up as, oh, this model isn't as good, right? Right. But really, it's like, well, the prompt needs to be different in order to justify that. And then at that point, how are we like managing then? It goes back to like there was a talk at the AI engineer conference where somebody was talking about prompts as like a compiled language. And like basically people are thinking of them as like just kind of text that gets hydrated, but really it's like a compiled source code in a way with instructions for an LLM to complete. You know, once you make an update to that, it like changes and it like recompiles and it like needs to like work differently. And like you then can't use that compiled source code, plug it into Windows 95, right? Like, right, yeah. You know, like it's the same idea. I'm definitely getting behind that. And I don't know. I feel like a lot of the prompting strategies libraries are like maybe misaligned in this way, where like I was hopeful with like the DSPY stuff that we could see maybe some self-improving things, but it's still almost too complicated from an instruction set standpoint to just like blanket apply prompts across the board, right?

SPEAKER_00 9:53

Yeah, I think you raise a really interesting point. And I'm curious to know in your role now at Gusto, do you have any benchmarks? Do you have any hard and fast ways of saying, okay, new model came out? I'm gonna determine whether or not it's better or worse for the use case that we have at hand.

SPEAKER_01 10:12

We have many, many uh different kinds of evaluation test sets, right? So it's always good to have like your core data sets that you're like churning off of for efficacy of whatever you're trying to measure against. And so we do like build some golden data sets around what we're trying to test output for. And yeah, we could swap models in and out, but again, it like circles back to like, well, are we gonna spend the time to try and like reprompt all these prompts that we have specifically for the new model? Like, we don't really have that automated yet. Now that I'm thinking about it, I probably want to now. Yeah, see if see how possible that is, like get a prompt to reprompt it. But at the same time, yeah, it's all about measuring just like output. And so, like, how close can you get all your evaluations in line? And there's an easy way to just drop in a new model and see, okay, is this gonna really have a measurable impact? And like typically with these big ones, it's like negligible, right? Like you might see like 0.1% or like sometimes 5% for some tasks as they come up with new releases too, like that may get improved too, right? And so then you have to timestamp the fingerprint of the model and make test that because this whole juggling act, which I don't know, if you find the task that you want to solve, like is solved by a model, it's almost like, well, why change the model? Right. Create all this extra work.

SPEAKER_00 11:38

Yeah, that's a fair point. I do think that this is something that's missing. There's going to be a company that comes along and kind of takes this off our hands, this benchmarking and evaluation. Because what I'm hearing from you, you know, I hear from others as well. And it always feels like, okay, well, we have this hand-rolled thing. And that typically means that there's an opportunity there in the market that's not being addressed today.

SPEAKER_01 12:02

Yeah, you know, at Doximity too, you prefer like setting up your pipelining and evaluation mechanisms around a vendor, right? Like it's just so easy to just buy something and then plug in your data sets and then have it churn through the calculations. But really, that's all it's doing. Like all these services, they're all around like LLM as a judge. So you pay for like the baseline evaluators to get you up and running to calculate something, and then you realize what those somethings are, and you're like, well, we want to add these specific calculations to target something, and then you realize, well, I gotta build out this whole thing, and like you'd be surprised how hard it is to like create these things. It's really just an LLM call in the end, and so like, you know, and you can generate synthetic data even to like help insulate and cushion the data sets that you do have, so you have more data to test with that's you know similar and generated in similar ways that helps. But yeah, at the same time, you always circle back to okay, well, we have too much custom things that we want to measure and test, and so you do just end up running your own out of necessity.

SPEAKER_00 13:11

Uh yeah, no, I don't think it's an easy problem to solve.

SPEAKER_01 13:14

Uh at the same time, so we had Klaus on an episode or two ago. Yeah, and he presented the Leva Gem, like easy way to like just test stuff in Rails. Really awesome. And so I see kind of like more of that augmented service coming out just across the board. Like I know we've mentioned Langsmith before, and that's like I don't know if they're free anymore. They used to be free, yeah. They're not, but that was the de facto, like, okay, yeah, you could just watch your traces and then like measure the impact. Uh-huh. And you know, that works great for a while. And they've got some great tools. I'm not gonna knock Langsmith. If you're looking to set up your eval pipeline, it's all there. I was hopeful when OpenAI had their like they have their own evaluations, I forget what they call it. Yeah, I guess it's just evals.

SPEAKER_02 14:10

Okay.

unknown 14:12

Yeah.

SPEAKER_01 14:13

So they have their like own built-in version of this where you can like sample all of the requests that come in through their API and then drop them in the data sets and then run tests against them. And I think that they even have it set up so that you can take it a step further and like fine-tune based on what it finds out from some of that evaluation, which is kind of neat, but it's like I haven't had the desire to use it. And I don't know if that's just like just me, or maybe there's something like you're mentioning, where it's like, you know, we want to stay model agnostic. And so like we don't want necessarily tooling that's like tying us back into the thing that's locking us into the models.

SPEAKER_00 14:59

I can easily see that as well. That was but kind of the first thing that I thought of with eval. Because I don't want a self-reinforcing, like, okay, I've made the decision to use open AI. Let me use an open AI tool that convinces me that I gotta keep using it. I also think that Langsmith has been good enough for us. I think that you know, Langsmith has a real advantage in being a first mover, but there are other tools available. We've stayed on Langsmith, I think there's some inertia there. We haven't seen anything that is blowing us away in terms of being better. But it's also, well, you know, it's not Ruby. It's a SaaS product, but you know, it's a Python first solution. And that was fine when we first started out. But as more and more of our application is built in Ruby, and in our recent turn to integrating Ruby LLM into our application, we came up short in trying to do tracing and observability. And, you know, and so one of our engineers went out and had to kind of roll their own tracing and observability and integrate it between Ruby LLM and Langsmith. PR may be coming soon to Ruby LLM. But it surprised me that there's nothing out of the box here, right? Like because I think it is something that is just that new. And I think, and I know you've got some opinions on this. My take is that unlike evaluating in an objective manner what model is best for a particular task, that seems like a very big and onerous problem to solve, this doesn't seem like a big problem to solve. This seems like something that should be solved again and again and again because there's a straightforward way to do it and you know when you've done it correctly.

SPEAKER_01 16:40

I think it goes back to I feel like the open tracing days, right? Like I guess we're still in them. There was like, you know, this concept that like spread wide and is still kind of prevalent in ways of having like a distributed tracing mechanism that you just like hook into your application and your language, and it just works and attacks everything, and you can do spans if you want to get more focused on details.

SPEAKER_00 17:11

That is open telemetry to an extent, right?

SPEAKER_01 17:13

Yeah, open telemetry. I guess it's yeah, I forget the transitionary period of each of those. Yeah, open telemetry. To me, this seems like the same problem, except we have like the whole streaming thing is like busting everything. You know, there's so many failure modes now that we have to account for. And then like, how does the tracing tie into those failure modes?

SPEAKER_00 17:40

Yeah. And you don't want the act of tracing itself to start contributing to the failure modes, right? Right. Because with more more streaming and more ways to connect, there's more tracing.

SPEAKER_01 17:50

And then the data pipelining too, right? It's like it's not like you're just sifting through logs anymore. You're like calculating things based on those logs. Maybe post-processing is still thing. I feel like maybe the logging tooling had a miss here. That's interesting. In order to like just adapt to LLM usage. You know, hey, just use your existing open telemetry and then oh, you can also like sample your data and pipe this into some evals. Yeah. I'm honestly surprised that that doesn't exist.

SPEAKER_00 18:25

That's a fair point because rarely do we need this up to the minute. Most of the time, I don't need to know the how much it costs for me to run this, you know, use this amount of inference on a model in real time. But I may want to know at the end of the day or at the end of the week, the end of a month, you know, what are the hotspots. And another big part of this, I didn't mention this before, was that there's a real cost-benefit analysis. Whenever we're using the leading models, the big heavy hitters that are, you know, paying billions of dollars to make these incremental improvements. Well, they're passing some of that cost on to us. And whenever we can do something with one of the other models, whether it's open source or it's just a generation behind, we're going to save a whole lot of money. And, you know, in our case, we're talking about continuous running. You know, this thing is running all day long on your repository. So it adds up quickly.

SPEAKER_01 19:16

We're talking with Klaus in the in this previous episode, and like he gave like the most incredible talk at San Francisco Ruby AI meetup.

SPEAKER_02 19:25

Yeah.

SPEAKER_01 19:25

Where it was like, you know, he was showcasing the Leva Gem. He used it specifically for this purpose you're talking about to like compare some, you know, lesser model to see how it performed and also how much money he's saving, so that as a business owner, he could decide, okay, is the like difference in quality worth the money I would spend on this model, right? And he was like, I think his the example he showed was like uh GPT 4.0 mini or something versus like GPT, I think it was 4.0 at the time. The percentage in quality loss from his evaluations were like, I don't know, 1% or 2% or something like that. That's interesting. And so he was just like, but the cost savings was like outrageous. It was yeah, like 100% or 50%. Yeah. And so he was like, well, obviously I'm can make this decision to just use the cheaper model, right? Yeah. And so it's like interesting to see that exposure of like, okay, cost actually really does matter. And it's like easy if you're in a bigger organization to maybe like just overlook that idea. But at the same time, like it is needless spending.

SPEAKER_00 20:34

It's needless spending. And you know, when you're building products with AI, which I think we're gonna get into, you know, it used to be that their, you know, SaaS model was almost like zero marginal cost. You had another user, it costs you just a couple of bytes, a couple extra minutes of EC2 or compute, and you know, and you're off and running, right? It used to be like, well, we can cost us, I don't know, you know, half a million dollars a year in fixed costs to build this thing. And so we need to make that up. But then every time, every user we add that gets us over that is just profit. That's not the case for, I can tell you from firsthand experience, that is not the case for AI products that leverage existing LLMs, especially commercially available LLMs. The variable costs now are very high. And the famous ones in our industry, the cursors and clients of the world, they lose money at just as rapid a pace as open AI does. They just do it in smaller quantities, right? There's an order of magnitude difference there. But they're all losing money, right? So, what do you do? Not everybody wants to lose money. I can also tell you from personal experience. So, you know, so what does a small and medium sized business owner do when they've got a great idea, either internally or externally? They want to leverage the LLM. It's a continuous thing. But now you need to be able to make Take back that cost. And a lot of times you're doing that through some combination of price engineering, which is difficult because the loss leaders are charging just a few pennies and managing the costs on the other end, which I think Klaus had a really good idea there. And it's something that we're starting to incorporate as well.

SPEAKER_01 22:18

Yeah, it reminds me of whether you hire somebody right out of college or somebody that has 10 years of experience doing something. You like to think of that as well, this person that has more experience is going to accelerate development of my product by this multiplier. And the cost may be more upfront, but the value from that, like you get speed. And so it's like a similar idea for like these models where you pay for the more expensive model in hopes that it's just faster at getting to what you want out of it. Even though there's all these like free workers behind the scenes that you could be running, it'll take them a while, you know? Like you know, if you spin up Claude Code with like Quinn 3 or something, you know, on your MacBook, it may take a few days to run something, but like you're gonna get it, yeah. You're gonna get it, and it's free. So like I guess the time, it's funny because like all of this AI is accelerating the everything, right? And so like speed is almost like the most important part if you're like trying to innovate. But if you're like truly just solving somebody's problem as a business owner, do you need that acceleration, right? Like, are you gonna be dwarfed by some competitor because they're more innovative solving the same problem?

SPEAKER_00 23:42

I honestly don't know. Probably not. I mean, that's not how businesses worked to date. The first entrant into the market is not guaranteed to dominate that market forever. It happens sometimes, but that's usually a business that has really worked hard to shore up what it does best and its customer service and all the rest of it. And I still think that matters. I've yet to see a case where that doesn't matter. And so I think that's a I think that's a good point. And I think that with respect to the models themselves, the changes I think what we're discussing here today is that the changes have always been incremental. And now with Chat GPT five, we have this like mass realization or this mass kind of like understanding that, oh, hey, this was just a little incremental change. Like every time we got a new version, everybody was kind of losing their minds for either good or not good reasons. They were kind of elated by it. Now we have ChatGPT five, and it's like, oh, okay, this is an incremental improvement, which is seen widely regarded as a failure by many people that I think outside of the tech community. Whereas in the tech community, we're like, well, incremental's good. You know, we'll take a little bit of an improvement here. Let's see what it can do. But the thing is that we really haven't scratched the surface of what these models can do. And so getting something that's like an order of magnitude better, that could be great, but we're not even using what we have today.

SPEAKER_01 25:08

Right. Yeah. I mean, that's a great point. And it makes me think, like uh, I remember reading the release notes on OpenAI, and there's just like so many details of the API updates as they relate to the model specifically, that I think we're just so overlooked. And it makes me think that it's like the Apple, like you're holding it wrong, right? Like basically everybody has to like redo kind of some portions of what they've built in order to get the optimizations that are in play now, right? And like I have a feeling this is gonna be a consistent iterative cycle like this. The model may seem very underwhelming, but if you use all these other little features we've added, you're gonna be really wowed.

SPEAKER_00 25:54

Yeah. Do you have any examples of that?

SPEAKER_01 25:57

Of what you read? Oh, yeah. So one example is you can now constrain tool outputs to specific grammars. So like instead of just saying, oh, this structure of this tool is gonna return this specific structure from a parameter standpoint or attributes, and instead you can use an actual grammar like Ruby. Ruby doesn't have a grammar, unfortunately. But you know, some other languages have no programming languages have grammars, but you can make your own grammars. And they're basically just like language templates that let you like generically describe how something should be respond in a formatted way. And so like basically they made a way to make structured outputs very flexible and like more toward like language. So like if you wanted something to generate in a report in a specific way, like you could just generate a grammar that is a report formatted in that way, and then all the outputs for that tool would return in that way.

SPEAKER_00 26:58

Yeah, that is very cool.

SPEAKER_01 27:00

So that's that's that's one example of it. And another one is like they introduced this idea of preambles so you can ask the model to like think out loud before it makes a tool call, even. So you can capture like if GPT 5 is reasoning about whether it should like call a tool, you can have it basically supply you that information ahead of time. Yeah. So that when you get your tool responses, when it's like, oh, you should make all of these tool calls, and it's like this is the reason why now. Maybe you could be like, well, that reason does we don't really care about, right? Like that could be useful, passing it along to some tool that makes another L11 call.

SPEAKER_00 27:40

Um you need a human in the loop at that point, right? Right, exactly. Or another agent.

SPEAKER_01 27:45

You could, yeah. I mean, if you had an agent that you know you wanted to approve all the tool calls beforehand by somebody, even the front-end user, you could present that reasoning back to them and be like, this is why I think we need to do that, instead of you having to be like, Well, we think this is you know why we should call this tool.

SPEAKER_02 28:02

Yeah.

SPEAKER_01 28:02

You know, or just nothing, right? Uh we're making it. We just yeah, or we already did it. Right. We'll send you a bill. Another one is like oh yeah, so you you can now get like reasoning tokens back and feed them back into subsequent calls. I know they don't expose like the thinking steps you see in Chat GPT, but now you can basically get those for free if you do multi-turn responses. So if you get like a response and it has reasoning in it, you can pass that back in your next call and it will be so close in proximity in the vector space that it can reuse it optimally and it doesn't have to like re-reason about things again. Yeah. Yeah. I mean, these are just like small examples, I feel like, of what is to come in this kind of space. Because you think like this is open AI, right? Like Anthropic has like similar API with these other adjustments that you can use their API for, right? Right. And so it's like back to the compilation thing, like, well, okay, well, you can't just try an anthropic model up against GPT 5 and say, hey, can we just use anthropic, right?

SPEAKER_00 29:17

Like so this is some way of translating the way that I prompt OpenAI over to the way that I prompt Claude to try to achieve the same result.

SPEAKER_01 29:29

Right, exactly. And like even on the API levels, maybe you use the API features differently for reasoning and anthropic than you would open AI because the models are formatted and structured a different way and built differently, to be honest, right? So like it's funny people just being like, okay, like I entered this thing in the chat GPT, and look at this, what I entered it in the cloud, and they're completely different. And it's like, well, yeah, it's not really apples and apples, you know? Yeah. And I may from a product standpoint it is, but I feel like there's just like you mentioned, like way too many parameters.

SPEAKER_02 30:06

Yeah.

SPEAKER_01 30:07

And so I wanted to circle back to the Ruby world for a minute. Because you did mention Ruby LLM, like integrating that. Yeah. So like, how do you weigh that, right? As a product builder, because like there's all this like hype lately about is AI actually delivering anything? Right. So, like, as a business owner, like building AI products, how do you decide like, okay, well, we should switch to using this framework, even, right? Like, what is the value that that's going to even provide?

SPEAKER_00 30:41

I know what you mean. I mean, there's kind of two sides of it. I'm a business owner with engineers that use AI to build an AI platform, right? And so there's it's AI all the way down already. And soon they'll just they'll just replace me with AI and then they'll take all the money. But the decision for Ruby LLM was, I suppose it was twofold. One was we're doing a lot of different work around the test generation process of Phoenix. We are trying to improve the efficacy and efficiency of Phoenix. And so there's graders that we're building and trying to analyze, okay, how good are these tests? How do we put them back in the loop to make them even better? Now, we've always had evaluations for the tests that we generate. But in this case, we want to take, you know, we want to build our own just greater, our own assessment of how good these tests come out and try to make it as death method-esque as possible, right? So we have our own opinions on it. I don't think they're that controversial, but they're strong opinions loosely held about how tests should be formatted today. So there's that piece, and then there's also this piece of when we started this 12 whole months ago, there were not a lot of Ruby LLM integration options. And so we built a lot of it in Python. And so when, for example, ActiveAgent, we had Justin on the show, you know, that didn't exist when we started Phoenix. And so we have, we've always had a Ruby and a Rails application because there's a web app behind this. But the lower level, the test generation was done in Python. So that is still the case. But more and more of the application, as we're getting deeper into the end-to-end integration of both test generation and then generating new tests for new code that's added, creating Everflame reports where we're evaluating the tests and the code as it comes in, those kind of end-to-end features, more and more we're able to say, okay, well, we want to do this with Ruby. Or in some cases, it is advantageous for us to do this in Ruby. So we went out and we did an evaluation. We looked at sublayer, we looked at ActiveAgent, we looked at Roast. We started with Roast for that evaluation piece that I mentioned, which is not part of the web app, but we started Roast for the evaluation piece. It didn't hold up well, and I think that is probably because it's not being sunseted, but I think it is not being as actively developed as it was, which is funny. You know, we had Obi on here like six weeks ago, and it's like, well, you know, things change. Things change quickly. And I think Obi himself, I could be wrong about this, but I think he's also more in favor of putting more effort into Ruby LLM and seeing that become sort of the leader for AI integration. And then when we look at Ruby LLM compared to Sublayer and Activation, I love both of the maintainers on these projects. There's not as much support, there's not as much use. And so we decided we'd plunge in with Ruby LLM. And so far, so good. It's only been a couple of weeks. And as I've mentioned, you know, it doesn't have out-of-the-xport for tracing and observability. So we had to roll our own. But we have felt like it has really added a boost. It's definitely helped us shore up that end-to-end integration. So, you know, we're happy with it.

SPEAKER_01 34:05

Yeah, that's awesome to hear. I haven't had a chance personally to play with Ruby LLM. I think the structured outputs was maybe preventing me from messing with it, which is now in there. So like uh I'll probably circle back and look at it. Check it out. It's kind of funny, like the Ruby AI gem, or Ruby Open AI gem, rather, you know, that was our de facto like gold standard. And now it's well, how do we get all these other LLMs in there? And yeah, it's you know, Langchain RB seemed promising, and that works great. And then as all of these APIs evolve, like somebody's like, oh, Ruby LLM, like I'm gonna make all the things that I'm missing from all these other libraries and release this. And then at what point does like that just keep continuing, and like somebody else comes out and is just oh, I'm making the LLM gem, and it's just like this is what you use for it. Yeah, yeah.

SPEAKER_00 35:02

Yeah, I mean, I I wrote about this or I talked about this in a talk that is uh it was back in, I don't know, March. So it's probably everything I said back then was absolute, is obsolete now. Right. But one thing that I I stressed to the developers was being able to, you have to make things you know modular and you had to reduce the dependencies on your integration points because you're gonna switch them out. Right. And I think that goes for these gems as well. It goes for roast, it goes for Ruby LLM, whatever you're using right now, it might be the thing that you want to use a year from now, but the chances are not good. And so for that reason, you know, you cannot tightly couple. And you know, we've always been warned about this, right? Like tight coupling. But now it's like, hey, you know, the thing that you're using today, you may not want to switch, but you oh you want to test something new out, and you're gonna want to do that over and over and over again because the field is evolving so quickly.

SPEAKER_01 35:57

It does come down to just, I feel like standardization, which is so boring, especially for AI, which is like, oh, all this new, new, new, you know, and like when it comes down to it and you're maintaining this stuff, you're just like, Man, I wish like there was a standard. Yeah, no, I totally agree with you. Thank God for anthropic with MCP and like that actually taking hold, which is like a little weird because like there was like the open API standard, which is like essentially the same thing, yeah. But at the same time, you know, like okay, like everybody just get behind this one thing and then we're gonna push it hard. I feel like you know, Ruby is great at like polar opposing both forces here, right? Like being innovative and just like doing everything all the same, all different ways, and also like coming getting behind something eventually.

SPEAKER_00 36:49

Well, because that's what Rails is, right? We could have just kept creating web apps and deciding where to put our models every single time. And Rails said actually, you just put them here, they go here.

SPEAKER_01 36:59

Right. Yeah, exactly.

SPEAKER_00 37:00

And that was still there. And it's in an expected place. It's more important that there's a convention than that you can configure it. And I think there's a lot of that in AI and in this world. We may not know what which is which yet, like what the most important thing is to abstract away, and what's the most important thing to just let developers be creative and explore.

SPEAKER_01 37:32

Yeah, totally. And it seems at this point that speed is almost outtrumping long-term support in a way. I worry a little bit about it, but like I feel like Ruby's very you could just monkey patch anything, you know? Yeah. You know, you just go in and be like, okay, like this class is like, oh, we're gonna eval in real time, like to do something completely different than what it was like built for. And I feel like, well, it's not maybe the best thing to do. It's like something you can do, you know. Something you can do. Yeah, it's true.

SPEAKER_00 38:11

I wonder how different that is. Uh, you know, being in the world of software engineering services for as long as I have, you know, people have, you know, they've committed atrocities of programming so that they could get, they can go from zero to one and get their next round of funding. And the answer is always, well, we'll fix it with the next round of funding, right? And and some people actually do and commit to it, and some people don't, because it's just more important to chase the next thing and chase the next vertical curve in in user adoption. This definitely it accelerates that kind of problem, but I don't think it's a new problem.

SPEAKER_01 38:46

Right. Yeah, it's definitely not a new problem. This reminds me like there was this the hype cycle over Rails, which the hype dropped off and people are still using it. And now it's like it's not a hype, but it's like popular, right? Yeah, so like a hype versus popularity, right? And so, like, where are we leading in this AI space with Ruby of uh like Ruby LM is a great example of popularity, right? Like there was a hype around it, and now it's like really just becoming popular. Yes. Uh and so, like, great, use that tool. You know, what else is coming out, right? Like, what else is like are we facing from a hype cycle perspective? We don't have to go into the coding world, agent world yet, right? But uh that's feel like it's its own episode.

SPEAKER_00 39:38

It's coming, yeah, it's coming. Don't worry, tune in next week.

SPEAKER_01 39:41

Yeah. But like, where does Ruby fit in that AI building business space? Like from a, okay, like it's no longer a hype and it's really just popular. And like why is it a good use case, right? That we keep coming back to, right? Like it's more than just like Ruby is, it's like, okay, people are you know innovating on things still, and it's like becoming a useful tool to use it for that.

SPEAKER_00 40:07

I would say that we're engineers. If you're listening to this podcast, you're probably an engineer or you're my mom. So hi, mom. And I think it's actually it's incumbent on us to rise above the hype. That's our job because the hype has always been there. And the hype for Ruby and and even Ruby on Rails needed to dissipate. And it's okay to get caught up, but the fact is that at the end of the day, we're building useful tools that we want to stand the test of time. And that's always been, at least for me as a software engineer that takes himself seriously as a software engineer. I judge my own abilities and success on whether or not the applications I build are still around in a few years, right? They're useful, they serve a purpose. Most of the time, they either earn money or they are funded, right? If it's a nonprofit, but those tools that I build are still being used. And if they're not, I think, well, I have something to learn from that. And I think the same thing is true for us here. It is easy to get caught up in AI hype, especially when there's a new tool coming out every five minutes that we can go and check out. And I think it's okay to, and this is where the Ruby comes in, it's a great place to explore. You know, we had Chad Fowler on the show, and he was talking about some other languages that might be better suited for automation or code gen with AI. And that may be true, but Ruby still is the best for the user experience of the developer to play with things, to figure things out, to try things out. And I think it's by doing that, it's by doing that creative work, we learn what is actually just hype and it's not actually going to be lasting for the test of time. And what is, as you put it, popular and suitably popular and serving a good function.

SPEAKER_01 41:57

Yeah, hopefully more thanks. Yeah. Because you know, it is the Ruby nature to make things that are not useful at all and just fun.

SPEAKER_00 42:06

Yeah, I think that's good. That's how you learn, though, right?

SPEAKER_01 42:09

That is true. Yeah. Which is funny. I just found out about this thing called Y Day, where it's like a memorial event that happens every August 19th. To like inspire the Ruby community to see how far you can push the weird corner of Ruby.

SPEAKER_00 42:27

That's funny, but you know, uh the thing is that Y Day exists because everybody got something out of Why the Lucky Stiff, right? Right. You know, he was being playful and he did something that was great. Okay, I'm sorry. So continue, I'm gonna get off this.

SPEAKER_01 42:41

But as an example, I was like, well, like let me have a Claude, you know, take this description of Y Day and see like what it comes up with idea-wise. And it came up with these two incredible ideas. One was around quantum Ruby, so creating a version of Ruby that sometimes executed and sometimes didn't based on your observation of it. And so like basically there would be a quantum state, and like if you were observing the code executing, then it would execute. Otherwise, it would be in this weird holding pattern.

SPEAKER_02 43:16

Yeah.

SPEAKER_01 43:17

And then there was the Schrdinger state of like, you know, maybe it does, maybe it doesn't. Right. Right. Very much in line, right?

SPEAKER_02 43:25

Yeah.

SPEAKER_01 43:26

Yeah. And another one was just like, oh, what was it? It was an emoji rubber duck session. And so, like, basically, like opening up a Ruby console where like you can use emojis to define your code. Yeah. And depending on the kinds of emojis you used, you could like make it run faster or slower. Nice. You could like put it in angry mode, you know, and like have more errors raised. Yeah. Yeah, I like that. So I don't know. I feel like it's what helps the Ruby community be so popular, but also maybe how it prevents it from being taken seriously in a lot of ways.

SPEAKER_00 44:11

Yeah, I'm okay with that. I mean, let's put this another way. If you're doing that, and I've got another example as well, but if you're doing that, then maybe you're resisting the urge to do something like create the next billion dollar company with just one person, right? Like that's everybody taking themselves, I think, way too seriously. So we could learn a lesson from why, you know, the whys of the world. And from your example, V, where it's like, hey, you know what? I built this thing and it's, isn't it funny and isn't it silly? And we did this with AI. Right. And because I think that tinkering is actually really important. And I think it's a lot of fun.

SPEAKER_01 44:47

Yeah. The serious notes, I definitely align with the anti-pattern there. Maybe that's where Ruby really shines. It's just like that it grinds against the you know draconian seriousness of everything. Right.

SPEAKER_00 45:00

Look at how look how great this thing is. We all have to become rich. We all have to, you know, or it's got to take over the world. Like we don't have to be in this world of absolutes all the time. Right. My thing that I've been checking out, I think I've mentioned this before, but if you're not already reading Scott Werner's works on my machine Substack, he has been churning out, I've been following it for a while, but in the last few weeks and been churning out this outstanding content. I mean, I think that he is one of the most creative thinkers of the usage of AI, how it works in your code, how you can leverage it. So what he released recently were these two new gems, VSM and Airby, and his post about it and why he did it and how he feels like it can be a template for artificial intelligence systems that need to iterate and loop into all the things that we talked about on this show. Check that out. We'll put it in the show notes and give them both a try.

SPEAKER_01 45:58

We're gonna have to get him back on to talk about these because like uh this is definitely the future. And I don't know how he comes up with this content because I know he's he must have a lot of coffee, he says. I know I feel like he just like talks about this in his you know living room or kitchen. Yeah, yeah.

SPEAKER_02 46:17

Yeah, yeah.

SPEAKER_01 46:18

But yeah, totally like the self-building runtime style nature that he's gearing toward. I think it's just what so many people have been talking about, but haven't figured out how to do it. And it this is very promising.

SPEAKER_00 46:35

I would put money on him to figure it out or at least figure out some large piece of it. You know, I don't know if this is it. He doesn't know if this is it, but this is worth checking out.

SPEAKER_01 46:43

Yeah, totally. All right, well, we we've talked about a lot.

unknown 46:47

Yeah. Yeah.

SPEAKER_01 46:50

Uh is there anything else you want to dive into? I feel like we've hit all the corners. Yeah. Like we can, I think we can how do you measure AI in reasonable ways in Ruby space, tracing all of this fun stuff that really isn't solved. Yeah, yeah. Right. I don't know. Hopefully, somebody listening is like gonna come out and be like, yeah, you just use this thing now. You know? Oh, yeah. Yes.

SPEAKER_00 47:18

If you are that person and you're thinking, I can't believe these guys just talked about this for 45 minutes where there is an obvious answer. Don't be shy. You can shout that at me on social media or you know, my email. You know, just let me know.

SPEAKER_01 47:30

Otherwise, we'll just keep talking about how this, you know? Yeah, yeah.

SPEAKER_00 47:35

Otherwise, we'll drive you slowly insane with this mindless chatter. All right. Well, this was great. Very happy to be on here and talking about this stuff. We'll do it again soon. All right. Thanks for listening. Take care, everybody.