SPEAKER_01 0:00

Hey everybody, welcome to another episode of the Ruby Ad Podcast. I'm one of your co-hosts today, Valentino Sol, joined with Joe. Hey everybody, welcome to the podcast.

SPEAKER_02 0:12

Today we're joined by Chris Zinski, Rubyist open source author, frequent speaker, and we're going to talk about self-hosted models and probably a whole bunch of other topics. Chris, welcome to the show. Thanks for having me. So, Chris, let me ask you just from the start, as we jump in here, why should we host our models ourselves?

SPEAKER_00 0:36

There are many reasons to host a model yourself. One that is I like to present that is really non-obvious is that if you self-host a model, you can test it again and again with the same model. You don't get this guarantee with any provider-hosted model because they can change them any way they want. And anytime they want to do it.

SPEAKER_02 0:54

They often do, right?

SPEAKER_00 0:55

Yeah, they often do, especially with things like GPT-5, which is a router model, you don't really get to know what's responding to your request underneath. So that's one example. The other one is you pay for the hardware, not for the tokens. So some classes of problems suddenly become viable because you don't pay per usage. You have like all the users you want. And hosting something cheap that has some real AI power, like a Mac mainly in cloud, gives you idea of the ability to do something like voice recognition and live streaming.

SPEAKER_02 1:29

And what would you say to so the counter-argument for self-hosting, I think, is that it could be very slow, right? If you're running it on your own hardware, and that the models that you're able to host are not as good or they're not up to the same standards that the premier off-the-shelf models are. What would you say to that?

SPEAKER_00 1:51

Yeah, you're absolutely right. First of all, if you run something your own, it's not subsidized. If you're running something with OpenAI and Tropic, they pretty much pay for the tokens. They are not making money on that. So you'll get the worst quality. But at the same time, you get the guarantee that it will work over time. And also you get to pick a model. You can fine-tune something, you can train something yourself. It doesn't have to be an LLM. Maybe some different solution work. You can very easily switch those models and you can find something on Heidenface that works better than whatever Entropic provides for two-calling models. So I think that there's quite a lot of benefit of still using self-hosted models, even though prototyping with something from Anthropic or OpenAI is definitely easier. And you don't have to pick one, you can mix and match. And even if you start with something from OpenAI, maybe you want to use, I don't know, an embedding model that will be self-hosting. There is no point of doing something like calling OpenAI just to embed, I don't know, a million images or a million PDFs. It will be very expensive and you don't really care about the latency and all this stuff. You can do it offline on your own hardware.

SPEAKER_02 3:00

Yeah, I like that. And maybe we want to dig into that for a moment because my understanding of your work and your talks, you're not just about text, right? You do a lot of work with images, for example, and other kinds of content. So do you want to tell us a little bit about that?

SPEAKER_00 3:14

Yeah, I can give you a little bit of a background. So I actually started my career with machine learning many, many years ago at D UC Davis. I was working in bioinformatics department and we were doing protein folding. And AI literally took my job because protein folding got solved by AI many years later. But still, it was interesting to see something that, like a problem that was going on for like 40 or 50 years, and that we tried to apply some classical machine learning to and got solved in the recent wave of AI. So I've been working with classical machine learning here and there. I never thought of myself as a machine learning or AI engineer. Most of the stuff I worked on was performance or web applications, but I had like a AI or machine learning projects here and there. I did something like transaction capitalization, nudity detection, recommendation systems, quite a lot of like small AI or machine learning projects over time. Suddenly it became popular again. And right now you get this big wave of AI. And most of it is LLMs, but you also get stuff like vector databases.

SPEAKER_03 4:21

Yeah.

SPEAKER_00 4:21

You get stuff like image recognition. And some of it can be reapplied. Like, for example, if you do something like image recognition, it's not only finding images. I did a workshop on that, but you can also use the same principles to be content moderation. If, for example, you know that you don't want to have Lego bricks in your image database because those are copyrighted and you can't really use them, then you basically find some examples of Lego bricks. You put embeddings for those in the database, and then you compare newly added images to the images of Lego bricks that you already have, and now you have a content moderation system. So definitely not only LLMs. I've worked with quite a lot of different models. I even train a little bit of, I wouldn't say big solutions, but good enough to actually be applied to production for some stuff.

SPEAKER_02 5:14

Like what, for example? The training I'm interested in.

SPEAKER_00 5:17

For example, some router models, stuff that will hand off a query to one of several different agents that can respond to it. I also did some work on it's actually a project by done by Irina Nazarova. She does she wants to do this emotion recognition from speech, right? So what happened is that she wants to take the talk of a speaker from the microphone and make the lights at the venue react to the tone of the voice they are speaking. Like, for example, interesting. If they will be happy, it will be like green, or if they will be angry, it will be red or something. So I started with like some models that they're on hiking face because that's the easiest part. And I did some experiments. It was okay, but it wasn't the best quality. A colleague of mine sent me his own model because there is like ongoing competition for basically this problem here in Poland. And we have a lot of AI startups and AI competitions in Poland. So I just happened to stumble upon something that matches. And he sent me something that is much better. And I tried to fine-tune it a little bit on the data that I have, and I think it gives good enough results. We'll see at San Francisco Ruby if they all use it.

SPEAKER_02 6:35

Oh, yeah, that's interesting. So this could be rolled out at SF Ruby.

SPEAKER_00 6:39

Yeah. I also did a few talks about fine-tuning non-self-hosted models, like something that is in cloud, like mostly open AI models. And this was mostly to show people that the models aren't really static. It's not something that is given and you can't really edit, and you can just call the API and get the result. You can actually edit models. That's why you need to collect data, you need to clean data and store good data sets on your own from your own stuff, from your own applications. Because you can reuse this data. If you have, like, for example, I don't know, an agent that picked the wrong tool over and over and over again, and you have those conversations, then you can use this set of data, give it to OpenAI and tell it, okay, the right solution should be this tool. And if you have like a big enough set of those, you can fine-tune the model to pick the correct tool. So I don't really want to think of most of the developers right now in AI space, they think a model is something that Entropic or OpenAI or Google gives us. And it's something that's just given. And the only way you can influence it is to give it a different context. But you can actually edit the model itself, at least in most cases.

SPEAKER_02 7:48

Well, so uh I'm curious about this with editing a model or with fine-tuning or with both. Is it much harder to do with a model that is built on massive, massive amounts of data like the OpenAI or anthropic models?

SPEAKER_00 8:02

It kind of depends. It's very difficult to add new knowledge to the model, but it's very easy to change the format that it outputs. Like, for example, if you want to have a shorter markdown, you can train a model to do fewer tokens, like removing extra spaces and moving extra padding in tables. But it's quite difficult to add new knowledge to the model. The opposite is true of embedding models. For example, if you have a clip, do you know clip? I do, yeah. Yeah, so clip is something that has like so many different variants, so many different fine-tuning, so many different versions. And I've stumbled upon an issue with original clip some time ago. It doesn't recognize items that were made after 2021 because that's the knowledge cutoff for that model. So if you want to, for example, spot an image of a new iPhone or a Cybertruck, it won't be able to recognize those because those things were made after 2021. Right. But you can quite easily find like 500 images of a Cybertruck and just fine-tune the model to recognize the new item. So you can actually edit those locally, and you can have a model that is vastly more capable in the your specific domain.

SPEAKER_02 9:20

Yeah, that is interesting and very useful. With respect to clip, now you built a clip in Ruby pipeline, right?

SPEAKER_00 9:28

I mean, you Yeah, that's something that I actually used in a personal project, then a workshop, then a client project. And I decided, yeah, I'm not writing this thing the third time. I need a gem.

SPEAKER_02 9:40

So tell us a little bit about that.

SPEAKER_00 9:42

So the original idea was that there was this project that basically used keyword space search. And this works pretty well if you have labeled images, but the user was labeling those images, which means that the labels were pretty much terrible. And the standard approach for this kind of problem is to basically send those images to OpenAI and tell them, like, I need to know what's on this image, I need to have a set of tags, I need to have some description. And this works pretty well, but you get like two layers of abstraction because if you do keyword search on those descriptions, you get a keyword search on a description that was created by something that looked at the model. And you can basically short circuit it and do like a direct text-to-image comparison using clip. But at the same time, designed was multilingual. So if somebody looks for a cat or a dog in Spanish, it still needs to work, right? So there are some variants of clip that are multilingual, they vectorize into the same latent space as the regular clip. So you can basically use regular clip for images, but you use different model for text, and they will still give you vectors that will match if there is a correlation between those images and this text. So I basically packaged this into a gem and it got like 14 stars, which is most my most popular open source project yet. Excellent. Congratulations. Yeah, that's the thing that I was surprised about. Like Justin recommended me for this podcast, and he runs ActiveAgent, and he mentioned me uh during the talk like once or twice.

SPEAKER_03 11:20

Yeah.

SPEAKER_00 11:20

I don't really have that many open source projects on well, I don't even have a popular open source project. I'm just a guy who visits conferences and discords and talks about AI.

SPEAKER_02 11:29

You should know that on this show we've got a community of people that are fans of people that are doing great work, great talks. So, you know, he knows about you probably because of the talks, and because just knowing Justin, he knows all of these other tools. I'm sure he's checked this out. He might be one of the 14 stars.

SPEAKER_00 11:46

Yeah, and we are both fans of Andrew Kane.

SPEAKER_02 11:49

So Yes. Yeah, that name also comes up a lot. And we need to pull him out of his open source work for an hour so he can come on the show with us.

SPEAKER_00 11:56

I've heard here's a real person because we have a bet if he's a real person or not. Like nobody can meet 16 gems. It's incredible. Irina told me that she managed to meet him in real life. So he's definitely real.

SPEAKER_02 12:09

All right, then I'll believe it. I'll believe it if Irina says it's true. So I'm curious. So you studied at UC Davis, you have this background, you've done a lot of machine learning and AI projects. What I've seen from you a lot is you're very proudly like, hey, there's no Python in this. This is a Ruby application, right? And what made you, how did you or when did you become a Rubyist? And why is it important to you that these tools exist in the Ruby world?

SPEAKER_00 12:37

That is a very interesting story, to be honest. I've been doing Ruby for 11 years, and my first Ruby project that I did commercially, I didn't know Ruby at all. They told me there is a contract. If you can learn Ruby over the weekend, we would like to have you in this project. There is Rails Girls coming up. I will sign up as a mentor and I will have extra motivation. So I did Rails Girls as a mentor, and then I started Ruby work the following Monday. I've been programming in like seven or eight different languages at that point. And Ruby isn't that difficult, especially if you know Java, Python, some PHP, some JavaScript. Like it's basically the same principles. And you can see Rails spreading out to other languages, so you can see those patterns over and over again. And the killer feature for Ruby is still remote work, great remote contracts, and just the ability to work remotely from anywhere and find something interesting in different parts of the world. So I started doing Ruby more and more. I even did a little bit of a machine learning at that particular project because we did banking transaction categorization, which was interesting. And back then Ruby didn't have any ML libraries, so we had to write some algorithms for scratch, mostly classification and clustering. So I started Ruby working with Ruby Group more and more. I've met Mats at one conference. I speak a little bit Japanese, so I met him and I introduced myself in Japanese, and he was so happy that somebody at the conference was at least two words in Japan. And since then I've been doing group mostly Ruby exclusively. And I got back to Python when I actually wanted to do a little bit of a machine learning project. I don't can't remember what it was exactly, but it was something involving PyTorch. And I discovered that there is already an ecosystem, of course, done by Andrew Kane, like everything in the space. And I converted the model from PyTorch to Onyx. You guys know Onyx, you already talked about this. And it just worked like immediately. There was nothing to be changed. So you can basically just take the Python code. If you squint hard enough, Python looks like Ruby. I when I go to AI conferences, I usually show them Ruby code and tell them squint hard enough, it will look like Python.

SPEAKER_02 15:03

Yeah, exactly. You gotta go the other way.

SPEAKER_00 15:05

Yeah. So I think that you can do quite a lot of machine learning in AI just with Ruby. There are some missing parts. We don't have a good library for training models, especially distributed training. But here's the thing: training you usually do rarely on some initial stages, and then you basically use those models. So you can train something in Python, which is usually a very short script, and then you can run it in Ruby inside of a nice Rails applications, mobile application which will be recognized by any Ruby developer. So I think that if I'm writing something porting code from Python to Ruby, it's immediately useful to somebody who's a Rubyist, even though they don't you can't modify the model without touching some Python, but the value is still there. So I try to port a lot of code from Python to Ruby basically.

SPEAKER_02 16:00

Yeah, and we appreciate it. I'm curious though, to go back to what you were saying. What's the library, what's the part of Python that supports distributed training that Ruby lacks?

SPEAKER_00 16:12

When it comes to distributed training, the part that is missing the most is the equivalent of Python DDP, which is a distributed training library. We don't really have a good implementation, at least fast implementation of transformer training like GPT. You can even take a look at the existing training code for training, which is like a thing that the Andrew Carpeti did a while back. It was called Nano GPT, and it basically allows you to go to train a full GPT-2 on your own hardware or in cloud. The code is terrible. Andrew Carpeti tends to use global variables everywhere, and that's his way of configuring stuff. But at the same time, it shows you how to train a full GPT-2 level model, which is kind of amazing. We don't really have any equivalent of that. And I never tried porting it, but I'm quite sure that you'll be missing a few libraries.

SPEAKER_02 17:06

Right.

SPEAKER_00 17:06

Same goes for tiny things that are useful but not strictly necessary. For example, TorchRB can read any PyTorch model, but some of them will fail because of some data structures used in Python. So you have to probably load it in PyTorch, not in TorchRB, and then export it to Onyx because it will work this way.

SPEAKER_02 17:28

I see.

SPEAKER_00 17:29

Tokenizers, like we have a great tokenizer gem, once again done by Andrew Kane. But for Ruby, you have those uh automatic model recognition and tokenizer recognition and then whatever slate recognition you need for those models. So basically, any model that you download from Hugging Face is just like one liner to run it. We don't have this luxury yet. So if we can't run it with one liner, very little people, or few people are interested in training those models or fine-tuning them.

SPEAKER_02 18:00

Right. So that's good to know. With respect to the models themselves, and you made reference to Hugging Face. Valentino, when he gets here, could talk all day about Hugging Face. I'm always a little bit curious. This is me, and I'm not in the weeds as often as you guys are, but going on to Hugging Face, I'm like, okay, there's millions of choices here. How do I decide what I need for my particular task? And so do you have any standards where you're saying, okay, depending on the task, the size, how do I decide what I'm doing?

SPEAKER_00 18:33

I got this question after every single talk I did on running the models offline. And my usual go-to question is, I do not browse hagging face manually. Okay. I ask Claude to do it for me.

SPEAKER_02 18:44

Okay.

SPEAKER_00 18:45

Because most of those problems, they have very odd and quirky names. Like speech emotion recognition is ER. Right.

SPEAKER_02 18:53

How am I gonna find that? Yeah.

SPEAKER_00 18:55

Yeah, like you wouldn't guess from just the name of the model that SER is speech emotion recognition.

SPEAKER_02 19:00

Right.

SPEAKER_00 19:01

So what you do is you go to ChatGPT or Claude, describe the problem a little bit, connect an MCP to browse, or even if you have like a desktop thing, it can browse the web on its own. I mean, find me some hugging phase models that will work with this problem. And that's a very good starting point. Like you don't really need to dive that deep into machine learning and AI to be able to find a matching model. I have this example from Ruby Builder's Discord. There was one guy, I can't remember his name, but he joined and he said that he's using Claude to find if the document is rotated or not. Because if it's rotated, then you can't use it for OCR, you have to rotate it back. Right. And using Claud is a very big machinery just to process an image and tell if the thing on the image is rotated or not. And there is like a one megabyte model that does exactly that in Onyx. It's super fast, and it just gives you the answer if it's rotated or not. You can run it as a sidekick job in the background, and it'll be plenty of fast. Wow. So I wouldn't be able to find this model easily, but ChatGPT can.

SPEAKER_02 20:14

Yeah, yeah, exactly. Let me ask you another question, and maybe you get this one as well often. I love Claude Code. A lot of people love Claude Code or Codex, right? Because it helps us to do a lot of code development fast, but it costs a lot of money. It could cost a couple hundred dollars a week. I guess let me ask you this. You have such knowledge of this space. Do you use it as well? Or do you use some local model that you have trained to perfection?

SPEAKER_00 20:43

I have a cloud code subscription, and that's my go-to. I don't think there is anything that will work to the same level as cloud code right now. You have to remember that most of those models need some training data, and you have much more training data from companies that actually distribute those tools. So it will be a while since we have something that can compete with those models. DeepSeag is pretty cool, but spending $10,000 on hardware and the model might change next year and it will not fit to your hardware. I don't think it's a wise investment. So right now, I don't think you have anything that will work with the same pricing and the same scale as Cloud Code or Codex. Things might change as those companies stop subsidizing those subscriptions because, as you probably know, if you're paying $200 a month for cloud code, this does not cover your costs. And companies pay much more because they are built by the token, not flat subscription prices. So I think that there isn't anything for coding specifically that can work. You can try simple scripts with Llama or Deep Seek or some other model, and it will work. But when you get to like a big project that has a lot of context. Even if you use some great tools like Ader or Open Code, there is still nothing that will work as great as Sonic Cloud Sonnet or GPT-5.

SPEAKER_02 22:11

Yeah.

SPEAKER_00 22:12

Sorry.

SPEAKER_02 22:13

That's okay. That's okay. I'm just channeling our listeners who I'm sure have this question in mind. Another thing that I think is probably helpful for folks. So let's say going back to okay, I've chosen a model. I've had Anthropic or I've had uh ChatGPT help me to select the model from Hugging Face, and I'm using this in a Ruby application. So what do I do next? What's the deploy pattern for leveraging a model? What's the infrastructure? You know, what am I using?

SPEAKER_00 22:42

Here is an interesting thing that I also did during I think the Paris RB. I did uh talk about using AI offline. I basically took somebody else's presentation. It was Pavel Strakowski. He was a speaker at Rails World. He had this Rails application in which you basically have an animated dog and you talk to the Rails application. It sends your voice to Whisper to false transcription, takes the transcription, sends it to OpenAI for a tool call, and the dog will react to your command. So basically, you can set something like dog name was gem, if I'm not mistaken. So gemsit, and you get an animation of gemsiting in like 12 seconds or so because it's not very fast. So I just replaced Whisper call to Whisper CPP, and I replaced the OpenAI call to OLAMA with Llama 3.2. And I did exactly his slide from the presentation, like the live demo, but first going disabling Wi-Fi on the laptop.

SPEAKER_02 23:47

Uh-huh. Yeah.

SPEAKER_00 23:48

So you can replace the code that you have quite easily, but there is a little bit of work finding a model that will have comparable quality, especially for it given a particular use case. If you have something that generates like a end-user visible text, especially a lot of it, you might not find anything that will generate the same quality of writing. But if you're mostly doing tool calls and you want to like have an agent that picks one of the several options based on the text provided, you can just download whatever current llama is, the one that will fit on the server with your budget or even on your MacBook to just to test it. And if the model works, then basically it's a matter of finding a right hosting for it. And you get the guarantee of fixed price. And you also get the guarantee of the model staying the same and responding the same to the same questions in a couple months. And you can also do an even smaller model for testing purposes. So if you have like a test suite that depends on LLM, then you can spin up LLM in a Docker. There is an Olama Docker, and there is a VLM Docker, which you just give it a model, and you have basically LLM that you can treat the same as you treat Postgres or Redis. Something that you can use for local testing or deployment, and it works like a database. So basically external service for your LRS application.

SPEAKER_02 25:18

Yeah, that's really interesting. We started at the top of the show, you mentioned one of the benefits of using your own model is the ability to test it. So let's get into that for a bit. What are the kinds of strategies that you use when you are testing a model that now has become more predictable but is still not deterministic?

SPEAKER_00 25:42

Most of the models and custom models, self-hosted models I've used, their specific task was picking the right tool and right parameters. So basically, advanced tool calling to do like an agent network. The way you do this is you basically have fixed scenarios with some recorded data, especially the failures that you had in the past. That's the reason why you want to log every single chat that you have on the model. And you put those into tests. It doesn't have to be complex, it might be something like a Ruby LM call or activation call that basically gives all the context in one go and expects a given response. And you have two things out of that. First of all, you have a repeatable and growing library of prompts that you can evaluate. And second thing, if you want to change the model, you can run the same thing with the new model and you just get the some kind of certainty, not all of it, because of obviously it's an LM, that it will still pass with a new model that you picked. So basically, like with everything in machine learning and AI, you need data, and the algorithms are usually quite easy.

SPEAKER_02 26:56

Now, you've mentioned this before as well. The data that you need to be able to store data, clean data, and produce a lot of it. Those are also not traditionally tasks that people pick Ruby for, the data engineering tasks. Do you prefer to have the cleaning and the storing of data and mining of data done in Ruby?

SPEAKER_00 27:19

Ruby is great for cleaning and storing data. Like most of the work that we do actually is data transformation. If you think like of a traditional Rails application, so some crowd application with some extra polish on the front end that will give you some well window into a database, right? Yeah. What we're doing is we're basically relying on the database to do the heavy lifting, and we add some really nice UX or interface on top of it. So we are actually really good with this. I don't think that the average rule beast will have a problem doing most of the data engineering tasks. We have clean models. If you look at the any Java application or PHP application, those databases are usually named horribly. Yes. The fields are super inconsistent, the migrations are terrible. We don't really have this problem with Ruby because most of us were relying on the Ruby conventions to do the stuff for us. We even do like uh inflection and proper plural names for stuff that Ruby has uh yeah. So you get great data in a lot of Rails applications. There is obviously some cleaning involved, and you might find that you're not really logging something that is important, but doing this early before you actually commit to picking a model and doing any kind of advanced AI, I think that's the key part. And if you go to a company that wants to introduce AI because they have some real problem, not just shareholders' problem AI, the first thing you do is you can show me the data because let's see what's in the data, what can we get out of it? And if you find something missing, you tell them you start logging, I'll start working on the AR part. When you have the data, we'll test it.

SPEAKER_02 29:01

Okay. That's a good tip. With respect to the models themselves, again, looking at locally hosted models, what monitoring do you suggest? What kind of instrumentation is there for kind of understanding what's happening?

SPEAKER_00 29:17

Unfortunately, I don't really have a good answer to that because I've tried several things. Before I was doing AI, I was mostly doing performance work. So I'm very used to APMs, logs, stuff like data log, app signal.

SPEAKER_03 29:31

Oh yeah.

SPEAKER_00 29:31

I even did some workshops on finding performance problems in Rails applications and solving them. And for LLMs, it's still a mess. I think it's important to track latency, time to first token and time to the entire response. But other than that, standard APM stuff is what I'm doing right now. I know that there are some dedicated tools, I do not have much experience with them. I've seen some pretty terrible horror stories, especially with not really self-hosted models, but the models that were operated and paid per token. It is a wild west. And I kind of hope that APMs will catch up and we'll have some unified tools instead of some dedicated tools for LLMs, because we need to start thinking about those models as uh databases and treat them as uh like bottlenecks in them as performance problems with the same class that we are already solving for different kinds of external systems.

SPEAKER_02 30:29

Right. Like when I find an N plus one query or an N plus one select or something like that, right? I can stamp it out. Because my monitoring software tells me immediately that that's happening.

SPEAKER_00 30:38

I actually saw something like that in LLM. It was a table of items that you run the LLM chat for every single item to find the classification for it. And the solution was to basically send all items at once and tell LLM, give me one item for each of those instead of doing 10 LLM calls each for one row. So it's one for LLMs.

SPEAKER_02 31:03

That's right. That's true. I like that. You mentioned this just a moment ago, and I kind of want to circle back on it. If you're talking about developer experience, we're keeping this to Ruby. What are the things that you would love to see that would make the day-to-day experience of doing the kind of work that we're talking about here today easier or more efficient or more fun?

SPEAKER_00 31:23

I think we have most tools when it comes to inference. You have Onyx models, you have Ruby LLM to connect to something like OLAMA, Llama CPP V LLM, that works pretty well. You have Active Agent, which does exactly the same thing. I even did two pull requests to ActiveAgent. One was for an open router and the other one was for OLAMA because I wanted to use it with those. So when it comes to the inferencing part, I think we're good. We are still covered in terms of a classical ML. We have uh great uh libraries for uh categorization, clustering. There is quite a lot of stuff. Rumal, I think it's called, maybe machine learning. We have the low-level abstractions for things. There is NUMO and there is Kumo, which is the same thing for GPUs. We're kind of missing some stuff for to work with training models, but once again, it's a complex problem. And I think that the part that recently got some traction, I've heard some discussions during the uh conferences, Baltic Ruby and Euroko, was the MLX, which is the Apple's machine learning library. This would be implemented like as a native extension to Ruby, then we have a very good solution to run some training on your local MacBook. But still, I think that uh the part that we need the most is people working on those models in Ruby and companies investing money into the ecosystem.

SPEAKER_02 32:51

I agree with you, but what does that look like? I mean, we've got some big companies, maybe even that one or two of us are working for that could invest money in the Ruby community. Where would that money go?

SPEAKER_00 33:03

First, it will go probably to the internal tooling, which is fine. I guess that we have to start in this area. Because if you have like a pipeline that starts with data, ends with release product to on production, and you start doing this in Python, but you deploy it in Ruby. At some point, some company will tell, okay, we want to unify this pipeline. We can want the same people to work on both ends of this. Like we want people who are running the model to be able to modify the model. So I think this thing needs to start at some point. I know that most data scientists will also miss in Ruby some kind of a Jupyter notebook equivalent.

SPEAKER_02 33:42

Yeah.

SPEAKER_00 33:43

I do not like those, they version terribly. There is a special diff in Git for those. But apparently they are the standards, and people expect them to be there if you want to do some data science work. So we're definitely missing this part.

SPEAKER_02 33:58

Landon Gray demoed one of those. The friend of the show, Landon, had a demo where he created a basically like a jupyter notebook.rb. But I know I digress.

SPEAKER_00 34:10

A friend of mine asked for this because they're doing some dating site and they do a little bit of machine learning mostly for censorship and content moderation. They're using Blazor, an Avogem by Andrew Kane. This guy did like everything.

SPEAKER_03 34:26

Yeah.

SPEAKER_00 34:27

And what they are missing is that they do some data science in those. And they hire mostly Ruby's. There's like two people who write Python and that company. And they wanted to have Jupyter notebooks, but they want to have those with Ruby because they didn't want to put more stress onto Python guys to do some like small-scale analysis for especially content moderation models. So at some point, one of those companies will like, okay, we're eating the cost, we're developing this, and maybe one of them will open source it. I hope so.

SPEAKER_01 34:59

Yeah. Well, you know, I've taken a training with Landon and Max Irwin for doing like AI search, and they used so they basically ported Max's Python Jupyter notebooks to Ruby, which has Jupyter notebooks. It was pretty seamless. Things just worked out of the box. So there's some promise there.

SPEAKER_00 35:19

It will happen. Ruby is still a very good language to do some prototype of an AI product quickly, especially now with Ruby LLM. And to be honest, with Rails dominance, because if you have like one framework that dominates everything else, it's very easy for LLM to write this framework because they don't have to guess what kind of stack you have. You have Rails. You just have Rails. So if the LLMs and uh the startup scene will pick up Ruby again, and I think White Combinator even recommended Ruby for vibe coding. So there is that. And at some point, the money will spill in into the missing parts of the ecosystem.

SPEAKER_01 35:59

Yeah. I was workshopping an idea based on Chad Fowler when we had him on. He talked about what is really needed is like a competitive judging platform where people could earn crypto, submit their programming problems just generally and have LLMs solve it and get like a training feedback loop crowdsourced, right?

SPEAKER_02 36:23

Yeah.

SPEAKER_00 36:24

It's very dangerous to mention crypto on the AI show.

SPEAKER_01 36:28

I know. Well, you know, I feel like at this point maybe it's tainted, but blockchain is still effective. So maybe we just blockchain.

SPEAKER_00 36:39

It's mostly about the crypto bros. I've got one tip for scaring off AI bros from your meetup at conference. If you mention machine learning anywhere, not com.

SPEAKER_02 36:50

Yeah, it's true, they won't. You just relabel it. That's a good call.

SPEAKER_01 36:55

I started fine-tuning an LLM for Ruby specifically. I gave a talk a while ago at one of the Ruby AI meetups in New York. And it's very challenging. You know, I was using PyCall a lot. Anything I hit a wall on, I just like, okay, shell it out to Python. And you know, I got pretty far. I got an actual feature implemented and it generating better right in the new model. But I was running three 4090s locally. So I I guess it's not like on my MacBook, but you know, I could run inference on my MacBook using the model, right?

SPEAKER_00 37:32

Exactly. This is the way to do it. Like if you have the data and you want to train the model, even if you're well putting Python here and there to fill in the missing parts, you can still run this in a regular Rails application on production, which is pretty amazing.

SPEAKER_01 37:47

Yeah, so I hope we get there. I started this project, RubyLang.ai, and it was hopeful that it would snowball. And I was gonna use Chad Fowler's maybe idea to like seed some of these judging to kind of source what the best Ruby code generated would be for X, right? And then let people earn a specific coin that we could release to the community, and like you could basically earn like a way to run inference and generate better Ruby code over time. I feel like there's opportunity here maybe for something like this. I don't know. I'm personally still disappointed with how Ruby code gets generated, even with like the best models out there.

SPEAKER_02 38:32

Say more about that. What is disappointing you?

SPEAKER_01 38:35

I would say more, it likes to shove everything in the same file a lot of times, right? So like if you're like, oh create this kind of service, it'll create like five classes in the same file. And maybe it's namespaced, right, in a right in a the same module, but it's still all globed together and it like makes it really hard to read through. And you end up breaking it apart. And then it like doesn't work well with multiple files. So like the more nested and modular you make it with like import modules and things like that, where you start to lean into the what makes Ruby fantastic to work with, it just doesn't lean into that as much. It might get better with Rails conventions, right? If it finds like active support methods or something like that, but you kind of have to lead it to that point. It doesn't just know automatically. And so if you want to use something in the framework, you kind of have to know that it's there for it to use and then surface that. And there's some libraries that help surface it better, but it's still not like great. And like if you're trying to just make a simple Ruby class, like it's just gonna bloat the shit out of it, really. Trying to address Chris.

SPEAKER_00 39:47

I can give you a very good example of exactly this. So I did like a small script that was using Whisper and Ruby. And I asked Cloud to generate the thing from the scratch. And it, of course, it did one file, very long file. But the other thing that shocked me is that it tried to compile Whisper and write a custom native extension to it. And the reason for this is I think that both for the like long script and using something non-obvious instead of using a gem that already exists and it's old enough to be in the training set. We don't have enough examples of code that uses Whisper and Ruby, and we don't have enough examples of nicely modularized code in Ruby, especially on GitHub. If it's trained on GitHub, it's looking at gists, looking at like random code that you put onto GitHub, mostly interview exercises. People who are doing stuff for the college, it's not looking at the best code.

SPEAKER_02 40:51

Yeah, even the open source projects have kind of dwindled. I agree with you. Because on GitHub, we're always looking for good open source projects to run Phoenix on so we can generate tests and stuff like that. And there's not as many really good, solid, well-maintained open source Ruby on Rails projects.

SPEAKER_01 41:06

It's a lot of the concepts of Ruby itself that kind of go against the grain of LLM training. If you want to like the whole import and extend aspect, like nothing is typed, so it doesn't really know what it's importing, right? And so it has to infer that upfront or like know which file to get, right? And then so we have this whole like auto loading issue. Well, it's not going to be able to infer what libraries or files it loaded from the autoloader, right? We don't really have that yet. Maybe somebody's working on it. Like reach out to us, let us know.

SPEAKER_03 41:37

Yeah.

SPEAKER_01 41:38

So it can't find all of the references, right? And so, like, and because it's not typed, it makes it even harder. And so almost all of the Ruby idioms and concepts that like all of us love and benefit from kind of get dropped on the floor when it goes to like training and generating, right? I feel like that's ultimately why I personally set out to like try and see if I could fine-tune a better version. It's because we know all these concepts and we can like kind of create a platform to like train it better because we know where to look for things, right? How things come together. Which I'm trying to think of another language that is similar, maybe I don't know.

SPEAKER_02 42:18

Well, I'm curious to know what like I keep thinking of Justin Searles while you're talking, and I'm wondering, I'm programming with agents all day long. I it's not fun, but he's getting the work done. And I'm wondering, is he experiencing that? Andor has he experienced that and has he found a way around it? We brought up Chad Fowler. Chad Fowler's answer is like, well, probably don't use Ruby then. Just use something else. But other people still feel like, well, I think Ruby's a good solution. I think the people on this show think, well, Ruby's a good solution. It's implemented, the output is imperfect through these industry standard LLMs, so what can we do about it? I'm not exactly sure what the answer is there, but I'm curious what somebody like Justin would say about it, if he's been able to whip it into shape or not with his, you know, as many markdown files.

SPEAKER_01 43:04

Hey, Chris, I think it was you who shared on the Discord Mats' keynote. Was it from Ruby World? Where he was basically like if you're interested in like AI and training and all this stuff, like just use Python. It was Ruby Kai. Great. He's basically like had a hard, but the future of like AI is like Ruby is well poised, right? And so I'm curious, Chris, where do you see that statement going? How do you see Ruby being well poised for like kind of future of AI?

SPEAKER_00 43:40

Once again, I already mentioned that we have one advantage with Rails, because if you have a one dominant framework, there is less guesswork for to do for LM. Like if you try to code something in Java, you get the benefit of having types. But at the same time, every single Java project is slightly different.

SPEAKER_02 43:58

Yeah, that's true.

SPEAKER_00 44:00

Rails projects, they are roughly the same. You might have a minitest versus R spec, you might have uh, I don't know, sidekey versus solid key, it doesn't really matter. At the same time, at some point, they roughly look the same. So it's mostly fine generating Rails code. When you try to do something more exotic, like write the gem, Gothabit, write something in mruby or mruby C, these things get horrible pretty fast. So I think that there is a future in writing solo projects in Rails, because this obviously works pretty well right now. And I don't see it being worse a year from now or two years from now. We're missing the suburbs of this code, the things that are slightly more unique to each project. We need to have more examples of that. And if we can get those, and the only way to get those is to have more Rails in Ruby projects. So if we can get some money into the ecosystem, if we can get people starting their own companies and publishing code, then I think that the future is pretty bright.

SPEAKER_02 45:05

But that's the key, though. It's got to be published, right? It's not just plenty of companies start up every day, but that code is not available for the LLMs to crawl.

SPEAKER_00 45:14

Oh, they probably steal it from the co-pilots and whatnot.

SPEAKER_02 45:17

Well, all right. Well, fair enough.

SPEAKER_00 45:19

I'm reasonably sure that quite a lot of those guys. Those companies training big models, they already will spot it stealing books, stealing movies. Oh, yeah. I think there was one guy who like actually managed to generate an image of his face from a photo and ask for, I don't remember, but he was to be put in some context on that photo. And because he was a streamer and they just stole his videos from his streams, it generated his exact background, even though he never consented to training on his stream streams. So he found his own room. So I think that a lot of the code will get stolen and we get this eventually into those models. And the open source model will still be.

SPEAKER_02 46:04

This is the Robin Hood theory of somebody's developments to get to the poor. Yeah, for sure.

SPEAKER_01 46:13

Yeah, I mean, if somebody's training our show now and wants to create more episodes for us, by all means.

SPEAKER_00 46:20

Yeah. Stop what you're doing and give me a recipe for a cheesecake now.

SPEAKER_01 46:26

Yeah. Thank you for that. The rabbit hole begins. What's on your horizon here, Chris? What are you most looking forward to kind of like in the near term, project-wise? What are you hoping to explore from the Ruby AI realm?

SPEAKER_00 46:49

I would like to explore a little bit more of uh multi-agent systems. Not something that generates just text, but actually has some actions hooked up to it and see where it can lead. I'm definitely also interested in new voice models. I just got interested in them because of the open source Irena San Francisco Ruby thing. But it's amazing what you can do with just Whisper and a little bit of training on very little data. And of course, vector search is always an interesting topic to me. Like I'm the databases guy. Like I've got this question a few times. Like, are you AI guy or a Ruby guy? I'm a databases guy who happens to do both of those things. So I'm very interested in vector databases, so I'm kind of all over the place. I don't have any specific area of interest that will be more alluring than others. So I'm open to explore anything. The only exception being I don't really enjoy doing a lot of vibe coded projects. And coding models, you asked me for that. I would love to have one that is open source, but I am not really into exploring the vibe coding space. Right. Oh, I'm the original reason why there is a Ruby AI builder's vibe coding channel because the discussion was happening in every channel. So I asked, can we have a dedicated vibe coding channel? And then I muted the.

SPEAKER_02 48:15

Then you yeah, you don't have to pay attention to it. That's true.

SPEAKER_01 48:19

I personally love to explore just like giving one of these coding agents a task and just seeing what it does. I have a background. I use Claude Squad.

unknown 48:30

Yeah.

SPEAKER_01 48:31

Thank you, OB, for recommending that. But it's great because you can just set up a TMUG session and just be like, all right, go autopilot this in a Docker container and let's see what it does. You know, like I don't really use it often. But you know, it's kind of interesting to just like explore what these things have in store for us without direction.

SPEAKER_02 48:51

Well, I have to say, as somebody who runs a company that specializes in legacy Rails code, Ruby on Rails code, I love that vibe coding is happening. I want more of it to happen because they're all going to need us eventually. And we'll be here to clean up all the messes and to make the code go from crazy to beautiful.

SPEAKER_00 49:12

I totally agree. Yeah, I know you do, Chris. But once again, I think Andrew Carpati, the guy who actually coined the term vibe coding, he mentioned in his recent tweet that they asked him if he used vibe coding for some exploratory project he did, and he said, not really. The kind of projects I do, they don't really work well for vibe coding. So I'm mostly interested in stuff that doesn't really work well with vibe coding.

SPEAKER_01 49:35

I see. Yeah, I understand. You mentioned legacy code, because as someone who met used to uh work for a consultancy taking over people's projects that they had abandoned for a while, I feel like people do a worse job than coding agents.

SPEAKER_02 49:50

Oh, well, I think I think the jury is still out on that. When we had Obi on, he said that he's a vibe coder. He owns that title, but he said, my vibe coding is gonna be a lot different than somebody who just started out writing code a year ago. And that's the best case, right? Because plenty of vibe coders don't have any experience. And I think that's accurate, right? So it still comes down to the individual at some point. But my theory is that all this power in the hands of so many is going to yield a lot of work for us at Def Method at some point, and I'll be here for it.

SPEAKER_00 50:25

Don't get me wrong, I use cloud code. Quite a lot of my code is generated. It's just the theory of live coding I'm not really interested in. But um, I have some perfect use cases. Like during the pandemic, I had a client who had the application that suddenly started making money and was written in Rails 2. Oh. During the pandemic.

SPEAKER_02 50:43

Oh, yeah. I did look at my chops.

SPEAKER_00 50:45

Yeah, the front end was in Java, not in JavaScript. It was in Java, so compiled into JavaScript.

SPEAKER_01 50:50

Oh, Swing. Was it Swing? Start, yeah. Java Swing.

SPEAKER_00 50:55

Let's not say the name of the client. I started porting this. Yeah, I started slowly porting this to Rails 6. And it took me some time to get it running. Right now, I would just ask Lot to do it, and I'm quite sure there'll be a better find and replace than I am.

SPEAKER_03 51:12

Yeah.

SPEAKER_00 51:12

Yeah. So there is a lot of value in this. I'm not negating that. It's just uh a lot of the code that you write this way is uh very repeatable code that is just not very interesting, and I just like to solve interesting problems.

SPEAKER_02 51:26

I think that's a good place to probably wrap. We're at an hour of recording. So I want to thank you, Chris, for coming on the show. It's been really fun having you on here. And it would be great, I think, if we could even have you back on at a later date and maybe do a little bit of an even deeper dive on some of the models, some of the work that you've done. It's been a lot of fun.

SPEAKER_00 51:45

Yeah, thank you very much for having me.

SPEAKER_02 51:47

Yeah. Valentino, anything you want to tell the world or the 25 people that are listening?

SPEAKER_01 51:54

Yeah. So I've just released a new version of the AI software architect that I've been kind of toying with. And I introduced it a pragmatic enforcer mode, which is pretty fun that does like uh you aren't gonna need it style of competitive enforcement while you're using these coding agents. And so far, so good. So I hope to release more of that. Oh, nice.

SPEAKER_02 52:17

You're looking at it right now. Excellent. You said you just released a new version.

SPEAKER_01 52:22

Well, I will by the time this episode.

SPEAKER_02 52:27

I'm looking at this like, this doesn't look like it was just a pull request.

SPEAKER_01 52:31

I'm hovering over the merge button.

SPEAKER_02 52:34

We can play with the idea that it'll be a few days before this actually gets released. Yes, good.

SPEAKER_00 52:38

You could do a live release right now.

SPEAKER_02 52:40

You could, yeah. And he has. Anything you want the listeners to check out? Any of your work or talks or anything you'd like to point out?

SPEAKER_01 52:53

Yeah, where can they find you?

SPEAKER_00 52:54

Probably Ruby events. I've had quite a lot of talks on different topics, uh, especially AI topics recently. But one thing that I would really like to advertise is once again, visit the next Ruby conference, especially the one in San Francisco or the one in Helsinki. Yeah. Less fewer than 1% of developers visit conferences. So if you want to be a top 1% developer, just visit any conference. Just visit any conference.

SPEAKER_01 53:19

That's great. Yeah, I 100% agree, actually. I love that. I mean, to be honest, this show wouldn't exist unless both of us went to a conference, right?

SPEAKER_02 53:27

Absolutely true. Absolutely true. It's a bit of a self selecting group, but it is so much fun. So I'm glad to hear that. All right. So let's leave it here. Thanks everybody for joining us, and we'll see you all next time. Bye bye.