Valentino Stoll  00:00
Hey everybody, welcome back to another episode of the Ruby AI Podcast. I'm one of your hosts today, Valentino Stoll, and I'm joined by my co-host, Joe.

Joe Leo  00:09
I'm Joe Leo. I'm the other host. Hey, RubyConf is coming in July in Vegas. Valentino, Ben, will I see you there?

Ben Scofield  00:19
Probably not, sadly. As much as I love RubyConf, I'm not a huge fan of conferences in Vegas.

Valentino Stoll  00:24
That is sad. On that note, we're going to introduce Ben Schofield. Thank you very much for joining us on the show. And sorry you don't love Vegas. I haven't been since I was in my 20s, so I'm looking forward to another chance to take a crack at it.

Ben Scofield  00:37
When RailsConf was in Vegas, when O'Reilly was still one of the partners, it was an experience. I do not think that it lent itself to the sorts of conversations that I find valuable at conferences, having to walk through the casino floor to get to the everybody else. There are great stories from that conference, but I like things a little bit more low-key.

Ben Scofield  00:58
I think it's for a different sort of conference.

Valentino Stoll  01:01
Well, that is an interesting point. That's funny to me to see it in Vegas. I mean, I know it's not the first time, but engineers in general typically prefer a lower-key event. You get some good rates and good hotel in Vegas, and I guess it's easy to get somebody there.

Ben Scofield  01:15
It is so easy to get there. It is a central-ish location. I mean, it makes a lot of sense. Different strokes for different folks.

Speaker 2  01:22
Yeah. The last time I was there was probably when I was 16.

Valentino Stoll  01:25
16.

Speaker 2  01:26
I'm kind of glad that I couldn't touch any of the shiny objects, you know? It is definitely like the manufactured world. If AI had controls over the physical world, like I just imagine it would produce Vegas.

Ben Scofield  01:40
Yeah.

Speaker 2  01:41
Maybe minus all the gambling, you know, but yeah. Yes.

Ben Scofield  01:45
Gambling seems to be everywhere with prediction markets now, and I can't imagine that won't be making it into AI's training data.

Valentino Stoll  01:51
Well, so we're really happy to have you on the show, Ben. Author and open-source contributor maintainer and longtime Ruby enthusiast and contributor. We're going to give you the floor here. Can you tell us a little bit about what you're up to now?

Valentino Stoll  02:10
'Cause I know you've landed in a full-time role relatively recently, and the focus of your many various interests lies today.

Ben Scofield  02:19
Yeah. This is an aside that we'll absolutely want to cut out. Bento did layoffs at the beginning of the year and dropped two-thirds of the company, so Friday is actually my last day.

Valentino Stoll  02:30
I'm sorry to hear that.

Ben Scofield  02:31
It happens,right? Well, AWS is apparently laying off 16,000 people today, so.

Speaker 2  02:36
You know, I was reading Bento and I was like, oh, this is a really cool service.

Valentino Stoll  02:40
Yeah, I know. Me too.

Ben Scofield  02:42
Honestly, I have been so happy here. It is very hard to find a place where you feel like you're doing direct good in the world if you work in software. It was at GitHub, and we enabled a lot of people to do lots of good in the world, but we were still X steps removed. Here, like in November, as part of the government shutdown,

Ben Scofield  03:02
we partnered with a nonprofit in California and fed 5,000 people. With work that I was doing, I was writing code on Tuesday, and on Wednesday meals were going out. But for various reasons, they laid off two-thirds of the entire company, including all of engineering leadership and all of us who had been hired most recently, so.

Valentino Stoll  03:22
Aside.

Ben Scofield  03:22
Jumping around on that thing.

Valentino Stoll  03:23
Yeah, well, Joe.

Ben Scofield  03:26
I think what I would love to talk about and where I'm coming from as I think about AI in general and to some extent AI with Ruby, I've been really fascinated by expertise forever. What does it mean to be good at the thing that you want to be able to do?

Ben Scofield  03:40
What does it mean to be a world-class athlete or an international soloist in music or a chess grandmaster or a Nobel Prize winner? Like, what does that mean? How do you decide what to focus on? Because we are all finite beings and we have limited lifespans and we can't be the best at everything we possibly want to do.

Ben Scofield  03:59
How do you choose the level of achievement or performance you want to attain? And then how do you get there as efficiently as possible? I know. I've been speaking about this. I think the first talk I gave on this was at an Ignite presentation at RailsConf maybe 12 or 13 years ago called How to Be Great by Someone Who's Very Much Not Great.

Ben Scofield  04:21
And it sort of pulled in expertise research at the time. This was a little bit after Malcolm Gladwell's promotion of the 10,000-hour rule came out, which I think he talked about in, I think it was Outliers was the book.

Valentino Stoll  04:34
That was also in Refactoring Your Wetware. That's where I read it.

Ben Scofield  04:38
Oh, was that Andy's book? Yeah. So Andy is, I don't know that he's written much recently, but he's also thought a lot about this at the time. And so I got really excited by that. The idea that you can be as good as you want to be at whatever you want to be at, regardless of sort of your initial talent with heavy quotations.

Ben Scofield  05:00
And so I went and started diving into the primary research and reading Anders Eriksson and reading responses to him, some of which have come out more recently. And so my thinking on this has evolved.

Ben Scofield  05:11
But I think what would be interesting to talk about with you all, and maybe I can ask you two a question, is the guiding principle of any discussion of expertise is it's domain-specific. And the way I like to individuate domains is by identifying the goals and aims within that domain.

Ben Scofield  05:32
So if I'm a 100-meter sprinter, then my goal is to cover 100 meters under human power as quickly as possible from, while generally racing against other people,right, faster than my opponents. If my goal is chess, then my goal is to win games and their constraints and rules and things and measurements for that.

Ben Scofield  05:50
What do you two think that the goals and aims of software development are?

Valentino Stoll  05:55
Are we allowing our guests to ask us questions?

Valentino Stoll  06:00
That's a great question. I feel like this is a "it depends" answer.

Valentino Stoll  06:08
Software is constrained really by hardware in my eyes. We have like very hard constraints physically. And so like really software is whatever you want it to be on that hardware. So like if you're trying to build something, you maybe constrain it for your end audience,

Valentino Stoll  06:28
the people building it, and the performance or utility of it will reflect the decisions that you're talking about. And so we almost have these kind of like similar domain constraints that are coupled to the hardware, but also coupled to the people that are building the actual things and maintaining them.

Valentino Stoll  06:48
And I'm interested in, okay, well what happens if we take out that maintenance burden and it is literally more of a passive thing?

Valentino Stoll  06:59
'Cause we're getting to this point where agents are evolving to a place and all of this AI is doing the software work and we are communicating the ideas and basically being in that communication layer as far as translating the domains to it and it can

Valentino Stoll  07:18
decide based on its constraints and the hardware it's using and the things that you're saying that you want to do with it. Hey, go build this thing and maintain it and service the end user, which is ultimately where we have been and we build teams around. I don't know what your thoughts are on that.

Valentino Stoll  07:37
That's where I see the industry moving. Well, so is your point, Ben, in asking us this question that is that software engineering is not nearly so constrained as a 100-meter dash or a chess game and we don't have a clear definition of what it means? My own thought, I mean,

Valentino Stoll  07:56
what I communicate to my engineers is that our goal with software engineering is to build sustainable systems, similar to what V is talking about, and to build something that is useful to humans at the end of the day. But of course there's a very wide range, certainly wider range of outcomes than in a chess game.

Ben Scofield  08:16
I think those are both interesting takes,right? So for Valentino, I would back up a step and say, before we start worrying about the hardware, like why are we writing the software? And I think that Joe answered that for one use case. That's not the only reason we write software,right? I remember the first computer I ever programmed on was an Apple IIe, I think.

Ben Scofield  08:36
And maybe I checked out from the library, but I had a book of Apple basic code that I was typing in there. I wasn't trying to build a sustainable system for users,right? I was, to some extent, entertaining myself and making this complex machine do something that it couldn't already do and exploring and doing it for the joy of it,right?

Ben Scofield  08:56
So I think, Valentino, you originally said it's kind of an "it depends" case. I think that's absolutelyright. There is not a single set of goals and aims for software development. We do it for different reasons. And I think that that should inform how we think about where and when and why we bring in agentic coding or AI more broadly.

Ben Scofield  09:17
Are either of you familiar with the philosopher C.T. Nguyen? He got some public acclaim a couple years ago. He's my favorite currently working living philosopher. He's a professor at the University of Utah. He was on the Ezra Klein Show and he's been in various public spaces for a couple different reasons.

Ben Scofield  09:37
But a few years ago he wrote a book called Games: Agency as Art, which is a book about the philosophy of games. And he identified two ways that people go about playing games. There are achievement players and there are striving players. Achievement players are in it to win and striving players are in it to actually play the game.

Ben Scofield  09:57
They get the value out of actually playing the game, not just from the outcome. And his example is he plays board games with his wife. If he were a purely achievement-driven player, then when he was not with his wife, he would research strategies on the internet to be able to beat her. But he doesn't do that because the value he's getting out of playing games with his wife is playing games with his wife, not out of beating her.

Ben Scofield  10:17
And I think that that is a useful lens through which to view when we bring in agents to code,right? If my goal is the outcome, then I am perfectly happy to hand it off to a swarm of Claude code agents and have them build whatever and engage in the feedback loop and work on how well I specify the problems and get the outcome,right?

Ben Scofield  10:37
'Cause I want the outcome. But that's not the only reason we code. Sometimes we code because we don't fully understand the problem and we work it out in our head and on the keyboard and we set up test cases for ourself. In the actual doing of it, we see how we want it to work.

Ben Scofield  10:54
That's something that I think we'd miss if we just hand that piece off to the edge of 'cause we'll be telling it to implement our mental model, but our mental model is incomplete and so it's going to fill in the gaps with its training data or history or something else.

Valentino Stoll  11:06
Yeah, I think that's absolutelyright. I was doing this just yesterday when I was doing some upgrade. It was a simple upgrade of Ruby 3 to Ruby 4 and things were failing and I was like, I could just, this is taking a long time, I can just give this to an agent and just have it fix everything. But I was curious about what was going wrong. Like why is it failing? What are the bindings that are failing?

Valentino Stoll  11:26
Like what's tripping? And that's an interesting thing that I think you'll probably take us into because we are all not young on this podcast and you know, we've got some experience. And so I think that maybe we can kind of use our inner or just our learned judgment to say,

Valentino Stoll  11:45
well, this is something that I think I need to learn or I just want to learn and it will benefit me versus this is not something I need to learnright now. I'm just going to move quickly. That the novice does not have.

Ben Scofield  11:58
Yeah, absolutely. And we've done this eternally,right? I do not know how DNS works. I have a high-level understanding of how DNS works. I have a friend who runs a DNS company and he knows at the very deep technical details how it works. I don't need to know that. I have chosen not to bring that to my attention.

Ben Scofield  12:18
I am focused on a different set of problems. I could have focused on that, but I know other people are working on it. So it's not something that's in my head. I think when we are talking to agents, we need to be very intentional about what we put in their head,right? Context management is absolutely key, even as context windows get bigger and bigger,right?

Ben Scofield  12:40
I don't want to shuffle the entire Encyclopedia Britannica into the context window 'cause there's going to be a lot of extraneous stuff that's just going to confuse it. So there's a certain level of attention and intention that we need to use our experience and histories with to help the agents do the things that we need them to do, if that makes sense.

Valentino Stoll  13:03
Yeah, it does. There was something that you said in your response just there that had me thinking.

Valentino Stoll  13:11
You are comfortable leaving the details of DNS to other experts and to potentially a machine because you have done this study and this kind of comes back to the 10,000 hours thing,right?

Valentino Stoll  13:25
If you had spent 500 of those hours learning DNS, it would actually be a distraction from your personal goal of learning or just your personal fulfillment as a software engineer. I think that that is, speaking from my own personal experience, a very difficult thing to do. It is very easy to say,

Valentino Stoll  13:45
but this could be important or, but I might find this really interesting or maybe this is the direction I need to go in,right? And so how do you limit those divergent paths? Because actually they come up all the time, every day.

Ben Scofield  13:58
Well, and they're infinite. There are always more levels you could dig into.

Valentino Stoll  14:02
Maybe I do want to be a sprinter.

Ben Scofield  14:04
Well, and I think there are actual clear benefits to having a broad base of knowledge,right? So another book is the science and sports journalist David Epstein wrote a book called Range, which is, I think the subtitle is How Generalists Triumph in an Increasingly Specialized World or something like that, where he contrasts the background of Roger Federer,

Ben Scofield  14:24
who did a lot of different sports as a kid and then became one of the greatest tennis players of all time, and Tiger Woods, who famously was golf as soon as he could stand up.

Ben Scofield  14:32
And some of the interesting research in talent development is that the highest performers as adults are generally not the people who specialized early and stayed specifically within their tiny rut all the way until they were 30 years old. There's a recent research paper by a collection of folks that is evaluating that and they found the same thing,right?

Ben Scofield  14:53
So there's actual scientific data as opposed to journalistic anecdata. Yeah. And so I think there are absolutely benefits to looking into the things that you don't know that you need. There's a metaphor I could go into that I will not, but I love it that is pirates and catamari de masi. Maybe we'll come back to it later.

Ben Scofield  15:12
I don't know. But the way I manage it is.

Valentino Stoll  15:15
This is going to keep everybody on the episode till the end. You know this.

Ben Scofield  15:18
Will we get king of the universe or king of all cosmos by the end? I don't know. So the way I manage it, manage that problem is suboptimal. I don't know that anybody has the perfect solution, but I am generally working towards a goal and I will keep an eye open to the things alongside that goal.

Ben Scofield  15:40
So if I were just now learning how to write web apps, I would not worry about hardware design. Who am I kidding? I don't worry about hardware design now. I know that.

Valentino Stoll  15:49
Me neither. I leave that to Valentino.

Ben Scofield  15:51
Yeah. Yeah. So GPU architecture, vitally important. I am so glad that people are fascinated by that, but that is five steps removed from the problem I'm trying to solveright now. And I will look at the things that are one step and maybe two steps.

Ben Scofield  16:06
And if I'm bored someday, maybe I'll be reading a history of Silicon Valley and I'll get really interested in the semiconductor industry or actually I read the history of Bell Labs.

Valentino Stoll  16:15
Yeah, that happened to me too.

Ben Scofield  16:17
Yeah. So I'll get a little bit about it there and if.

Valentino Stoll  16:19
But fortunately or unfortunately, I can only get so far before I'm just like, God, this I'm so over my head. I can't even.

Ben Scofield  16:24
Well, and I think that's okay. So I'm not a huge fan. I recognize that there are individual differences that are sort of innate to people, but I think that the capacity for improvement across any domain is so much greater than we think it is. I will never be a world-class sprinter. I'll never play in the NBA.

Ben Scofield  16:42
I could be a lot faster than I am now and a lot better at basketball than I am now if I chose to focus on that, but I don't have to choose to focus on that. I can't be the best I could possibly be at everything I could possibly do. So maybe I've just resigned myself to being mediocre and stuff. I think I wrote an article called Impreasive Mediocrity at one point. I think keep your eyes open for things.

Ben Scofield  17:02
Don't ignore them, but you don't have to seek them out. There is so much wonder and fascinating information in the world that you're not missing out on the most wonderful thing. Like you will find a most wonderful thing and be happy and be better at whatever you're trying to do.

Ben Scofield  17:22
You just dig into anything and it's more fascinating than you could ever imagine.

Valentino Stoll  17:26
You make a really interesting point here on domains and the importance of kind of cross-domain knowledge. And it makes me think of DHH's conceptual compression talk. And I mean, he's constantly talking about it. But it makes me think of two things.

Valentino Stoll  17:45
When we think about like designing systems for like stepping into the, okay, we're designing the build software, domains are great 'cause we make boundaries for them,right?

Valentino Stoll  17:56
Like that's the whole purpose for making great software systems that work together in quality ways is we can create the boundaries that make it easier to work within those specific cases. And it makes it easier to extract out and step back and make sense of those cross-domain things.

Valentino Stoll  18:15
And setting the boundaries is very important. And I feel like you're saying it's true of even like our own domains and our own domain knowledge. So I think about this idea of conceptual compression as like kind of making those boundaries and setting up the domains and making it easier to cross them.

Valentino Stoll  18:36
And what DHH is known for saying is how lossy that is and how that loss is okay. But what are your takeaways from a systems perspective with all of these agents? It's good to constrain them to domains and that's kind of like how we're doing it with our software is, okay,

Valentino Stoll  18:54
we're creating a better focal point and setting these boundaries up. You know, how does that loss affect things and how do you think about the loss in those ways?

Ben Scofield  19:03
No, I think that's a really good point. When we make decisions, we can make them thinkingly or unthinkingly. So we can choose to say this is within our domain or this is not. And when we compress, we are making decisions about what we think is important and what is not. So I could have a compression algorithm that eliminates all white space.

Ben Scofield  19:23
And that would be fine for JavaScript. It would be fine for Ruby. It would not be fine for languages that have significant white space. So the choice I'm making there is, so it defines the domain on which this will work for one. And it sets some things aside.

Ben Scofield  19:42
We talk about this in business with what is the core competency of your company,right? So if I work at, Amazon's a really bad example 'cause they've explained it so greatly,right? If I work at, I don't know, Cloudflare, then I am not going to set up warehouses and start shipping hardware around. That's not my core competency.

Ben Scofield  20:00
My core competency is in routing and caching and doing all the other like internet layer stuff. That's a choice. For Cloudflare, it's probably not an intentional choice to not have warehouses and ship a fleet of delivery vehicles. It's kind of an accidental one. But whenever we design our systems, choices are being made.

Ben Scofield  20:20
We are making some of them. Some of them are made by default. And I think this comes back to something we mentioned earlier,right? I would say to be the most successful user of agentic coding, you need to be as intentional about as many of the choices as you can, including the choices about what are intentional choices or not.

Ben Scofield  20:38
So I'm choosing to give this set of agent.md files or expose this set of MCP servers or whatever it is. I'm choosing to let my agent know about the GitHub MCP because I want to use Git as my source control and I want to use GitHub as my sharing platform.

Ben Scofield  20:58
I don't have to do that. It'll probably use that anyway. Maybe it's not a core concern of the system that it used in GitHub, but it's important for me for whatever reason. Maybe I work at Microsoft. So yeah, I think compression's really interesting as a tool for understanding because it forces you to decide what you care about.

Ben Scofield  21:18
Compression is getting rid of the stuff you don't care about.

Valentino Stoll  21:21
Valentino, this reminds me of our conversation from last week where you were talking about the GPT you created that was a software architect. The answers would change depending on what books you let it look at. I loved that example. And that sounds like a similar thing to what you're talking about here, Ben. We need to make an intentional choice about what we're supplying the agent with.

Ben Scofield  21:42
This makes me think about going back to the hardware. I don't want to harp on this too long. Basically the constraints of the system. I remember hearing a talk at honestly like a local Ruby meetup. Somebody came on and talked about building systems for like Mars rover as an example. Do you build that software much differently than you would say a web application,right?

Ben Scofield  22:04
Because you literally have constraints with hardware and resources that you need to account for. And so like you're more critical of the choices that you're making, maybe the language preferences even because it is so crucial to get like the most optimal use of those resources because you are constrained by like how much time you have to use them even.

Ben Scofield  22:24
And so like I think about other constraints like that. Most of everybody is building on the web, but you know that's not where everybody's building. And so I think about how those constraints maybe like affect software in general.

Ben Scofield  22:39
I'm curious, like where do you see like constraints servicing and how are you maybe approaching them differently depending on those cases? Yeah. That's an interesting question.

Ben Scofield  22:50
Before I answer it, and hopefully I won't completely forget it, is my favorite example of this is software for like satellites and probes that go out apparently has to be resilient to memory locations being affected by like cosmic radiation. Like a particle comes out that hits.

Valentino Stoll  23:09
I didn't think about that.

Ben Scofield  23:10
Flips. And that is a constraint I have never and will never have to worry about.

Valentino Stoll  23:14
Thank God.

Ben Scofield  23:14
Yeah. I was reading a terrible sci-fi thing and they were taking equipment into a zone where everything had to have like eight different levels of backup because the environment itself was gumming up the works so continuously. And that's a level of resiliency I don't need in anything I do. I'm not building software for heart monitors. I'm building software that makes money and that's important.

Ben Scofield  23:36
Like I'm not going to crash an economy. I'm not going to kill anybody. Fingers crossed. Yeah. So let me try to answer your question and you can tell me if I failed or not by shifting a little bit. And instead of constraints, I want to talk about defaults.

Ben Scofield  23:53
The favorite question I started asking when I was interviewing for StaffPlus engineers in the past is, what is your default? If I ask you to build a system and I give you some specifications for it, you are going to make it either more secure or more scalable or more performant or more modifiable or something else.

Ben Scofield  24:11
Your history has led you to a point where you know in your heart that people always underspecify how X it needs to be. And so you're going to make it more X just by default. I love that question 'cause the answers I got back were so different from different people and it informs whether you need to hire that person.

Ben Scofield  24:29
Because if I'm trying to fill a gap on my team on security, I will want someone more if their default is security, if making things more secure or less prone to abuse and harassment for our users or something like that,right? People have all kinds of different defaults.

Ben Scofield  24:44
And this ties back into Joe, what you were talking about, Valentino, with the software architecture GPT is being aware of your defaults is almost more important than like getting rid of them or changing them because you can shore yourself up with other people who have different defaults or maybe custom subagents who like security is not my thing.

Ben Scofield  25:05
I can do security, like all the basics and I know OAuth and all that and I know what to run, but my default is not making things more secure. So I can have a subagent going behind me and saying, hey, this might be vulnerable to some sort of maybe a prompt injection. I don't know. I don't know that anybody has a solution for prompt injectionsright now, but it'd be great if they did.

Ben Scofield  25:23
Does that sort of start to talk about what you wanted to talk about?

Valentino Stoll  25:26
I think so. I mean, I do see this personally too in my own development 'cause I have a wide range of interests and I am all over the place. And it's very helpful to be honest, all of this AI, because I can experiment in different ways and like you said, set up different defaults for those varied use cases.

Valentino Stoll  25:46
And I do. I have very wildly different Claude files or other agents that work better in very specific use cases, different domains. As an example, like doing hardware stuff, I like to tinker and like make new things and I just have components and it's more of like image-related activities where I'm taking pictures of components and saying,

Valentino Stoll  26:06
how can I use these things and connect them to make something,right? And so the overall experience with that is much different than it would be like poking around on a website or something. Yeah. It's interesting to think about the defaults in these different ways.

Valentino Stoll  26:22
And something I think about related to this is how the defaults might affect the power of these things or manipulate them in ways that prevent them from doing things without all of the defaults,right?

Valentino Stoll  26:40
And I'm curious what you think about this where let's say you just install Claude for the first time and you're just using it and on your own machine you have like all these defaults that make it use in specific ways. Are you missing out on things that come from the general purpose and how do we measure that? How do we know,

Valentino Stoll  27:00
you know, I worry about this, like maybe, you know, the side effects of the defaults negatively impacting us,right? How do we even know that that's happening?

Ben Scofield  27:09
I think this goes to Joe's question about how do you decide what to pay attention to outside of your, what you identify as your core domain. Maybe knowing about DNS would make me an infinitely better web programmer. I don't know. And this has always been true. Like AI doesn't make this worse. I have a .files repo that lets me set up a new computer.

Ben Scofield  27:27
There are so many different ways you could set up your .files that my defaults work for me. There might be something that works 10 or 50% better. I don't know. It's a weird example 'cause part of the value is actually the familiarity with it. And so it's hard for me to see what would actually be a 50% better .files setup than the one I have now. I could see incremental improvements, but like not a step change,right?

Ben Scofield  27:46
One of the things I'm excited about with generative AI and code is it lowers the barrier to experimenting to find out if there are other steps that would work better. So I haven't done this largely. I've done it in the small,right?

Ben Scofield  28:01
But I want to, or I would love to see people instead of generating one output from code, generate five. One's in a different language. One's using a different framework. One uses a completely, like these patterns are banned. Other, these patterns are required. And just see if we,

Ben Scofield  28:20
and I think you talked about this with Chad Fowler, the importance of sort of evals and specifications for judging the correctness of the generated code. And I know he's been writing about it on his leaflet publication with the Phoenix Architecture stuff. Some things we care about, there are lots of different ways to accomplish the things we care about.

Ben Scofield  28:39
Generated code makes it much cheaper to try to achieve those things in a lot of different ways and then pick between them. And that will then go in farmer spec. So I generate three different solutions for this website and I'm playing with them and performance was not one of my requirements, but one of them is dog slow and one of them is actually too fast.

Ben Scofield  29:01
Then I'm going to put the performance requirements of that middle Goldilocks one back into my spec and do it again. One of my favorite quotes of all time is, how do I know what I think until I see what I say?

Ben Scofield  29:12
And there's a, which I like it for philosophical reasons about the lack of internal privileged access to our own mental states, but I think it's relevant here because how do I know what I care about until I see it in the world?

Valentino Stoll  29:27
Schrödinger's thoughts,right?

Ben Scofield  29:29
A little bit, yeah.

Valentino Stoll  29:30
I want to take us from here to a category that I think is near and dear to your heart, which is now that we're thinking deeply about all of these ways of making ourselves and the software that we produce better, for some definition of better,

Valentino Stoll  29:47
how do we bring that to the training of experts or the training of expertise in the engineers that we manage or that we lead or that we just work alongside of? I think you've done some thinking about this, Ben, and I'd like to know what you think.

Ben Scofield  30:04
For a number of years, I have thought that the traditional breakdown of engineering career paths into manager and IC is insufficient. I think there are actually three paths. There's management, which is fairly well understood. There's super senior deep IC within a specific narrow domain working within that domain and they can have incredibly deep expertise.

Ben Scofield  30:25
I worked with like Git Core contributors who are amazing. And then there are also StaffPlus engineers, which I think everyone has recognized there is a problem of vagueness around for ages. I think Will Larson and others have done really good work around like these are different archetypes of Staff engineers or StaffPlus engineers.

Ben Scofield  30:46
There's some secret component that is different from a super senior. They are literally different jobs. You can excel at one and not excel at the other, either direction.

Valentino Stoll  30:56
Can you describe the basic tenets of what the differences are there?

Ben Scofield  31:00
The core for me is that a Staff engineer works broadly. So they're talking to people outside of their specific team. You're more likely to get them in, so in a big company, like a platform organization where they have to talk to all the people who are implementing their, or using their tools. Whereas a deep super senior IC,

Ben Scofield  31:20
they might not talk to other people outside of their team very often at all.

Valentino Stoll  31:24
Yeah. They are better that way.

Ben Scofield  31:26
I think.

Valentino Stoll  31:26
Super senior, yeah.

Ben Scofield  31:28
One is externally focused and one's internally focused. Again, I think it's very vague and squishy.

Valentino Stoll  31:33
I think that's true. I wanted to zero in on it because a Staff engineer is a title that everybody kind of recognizes and super senior is not, but I think people know what you're talking about and can think of an example of like, oh yeah, I knew this brilliant guy and he was just, if you put him on this one type of problem, he was better than anybody. It was not even close, but you didn't really move that guy or that girl outside of that domain.

Ben Scofield  31:53
And there's value in both of them. We need both of those people or all those people. So if we acknowledge that there is a, this third career path, how do we train people into that third career path? We sort of know how to train managers and maybe they start as engineers and we see they have more soft skills in quotes and then we lead them to the management books that we like and they become managers.

Ben Scofield  32:14
We know how to train deep technical experts. Not everybody wants to be there, but we give them harder and harder problems within their domain. We don't ask them to talk to other people that much maybe. I don't know. But how do we train staff? It feels like they spring fully formed from the head of the company somehow. And every company treats them a little bit differently and promotes them a little differently and identifies them a little differently.

Ben Scofield  32:34
So as long as I've had this three-part career path model, I have thought that the best training for being a Staff engineer is being an very early employee at a very small startup because you're forced to work across the teams.

Valentino Stoll  32:55
Well, I'd say just fire 'em all. Just get a bunch of experts and don't let any engineers talk to anybody. We don't want to talk to you anyway. Yeah. So what do you think? So you've got RubyConf. What else do you have your eye on this year?

Ben Scofield  33:09
I'm looking to join more of these Ruby AI meetups in New York.

Valentino Stoll  33:14
Oh, in New York? Yeah. Next one's in February.

Ben Scofield  33:16
I've been missing them for a bit and I've been experimenting just so much.

Ben Scofield  33:22
We could talk about this later, but I have this project I started called Chaos to the Rescue and it makes use of method missing to implement methods on the fly and redefine them at runtime.

Valentino Stoll  33:39
I remember there being an experiment like this in a book that I'd read like a million years ago. It was pre-AI. But okay. So continue.

Ben Scofield  33:46
So I, I made this special console and you boot it up and it loads and sets everything to auto by default. So it just automatically defines methods that are not there. And I start off with just like a class, it's just like a shell class called calculator. And so I say, okay, calculator.new add five and two.

Ben Scofield  34:06
And the method doesn't exist, but because this module's included, responds to method, it actually returns true to everything. So that's what I was curious about.

Valentino Stoll  34:17
So it does it go and then define the method and then.

Ben Scofield  34:19
Yeah. So it'll go and I'll, it'll intercept it in method missing and be like, you know, Ruby will say it doesn't exist and it'll pass that to the LLM and be like, okay, here's a class calculator and it's being asked to call this method with these arguments. What, you know, define it.

Ben Scofield  34:34
And so it goes and it generates the code for the method and then it uses define method and it literally adds the method to the class at runtime and executes it. And it works.

Valentino Stoll  34:47
That's great. You gotta present on that.

Ben Scofield  34:49
Oh yeah. I gotta do that. I even hooked it up to Rails 'cause they have a rescue from method. So I, I just have it in Rails and anytime, you know, an error happens, it rescues it in Rails and redefines what it should exist. And so I'm gonna experiment and see, you know,

Ben Scofield  35:08
how far could I go with a Rails app if nothing exists and it's rescuing everything and defining it at runtime, like how far will it get, yeah.

Valentino Stoll  35:18
Is the new experiment that Valentino's building?

Ben Scofield  35:21
I actually wrote a plugin way back in the day. The idea was if you ever got an unhandled exception in staging or development, it would automatically write a test for you with the path that you were on and the parameters and just drop that test in your test suite and then you could go and make it pass.

Valentino Stoll  35:39
I feel like I've used something like that. Maybe it was yours.

Ben Scofield  35:41
It was not mine.

Valentino Stoll  35:42
It was not yours? Oh, yeah.

Ben Scofield  35:44
Yeah. My open source success was very limited.

Valentino Stoll  35:48
We've still got a bit of time and you were on a tear and you were talking us through the Staff engineering model. There's no sort of model for, I think the last thing I heard you say was that the Staff engineers appear to just come like fully formed from the mind of the company.

Ben Scofield  36:03
Yeah. We don't have a way of training Staff engineers. And part of that is because we're really bad at defining what a Staff engineer is, as I think we just talked about. And it looks very different between different companies.

Ben Scofield  36:14
So as I've thought about this and the people I've seen sort of be most ready to be Staff engineers at big companies are those who were very early employees at very small startups that did not immediately die,right? So as a very early stage engineer at a very early stage company, you are forced to work across, not just deep.

Ben Scofield  36:35
You can go to work for IBMright outta college and go incredibly deep and never have to talk to another soul except for your manager ever again. You can't do that if you're a three-person company that survives. I should be very clear. That survives.

Valentino Stoll  36:47
That survives. Yeah. No, that's absolutely true. You know, I think Ben, the first time you and I crossed paths was when you were at Living Social, which was a startup that didn't die and you had to wear a lot of hats to get that thing off the ground and running.

Ben Scofield  37:00
I would say I didn't. So I joined Living Social early, but not incredibly early,right? So I like interviewed and still this crappy office building, but a month after I started, they had a building that was an old refurbished bank two blocks away from the White House or whatever.

Ben Scofield  37:16
Living Social was a crazy experience, but particularly the reason I don't think that this super applies to me is because when I got to Living Social on the second day maybe, they handed me the keys to email and said, here you own this. And I did not have the luxury of doing anything else because Living Social was an email business.

Ben Scofield  37:34
And I say that I've never written code where people died or you wrecked the economy. That was the scale at which I had never worked before and a lot of people hadn't worked before. We were sending more email than mail service providers on a monthly basis. Like it was ridiculous. So I was very focused there, but I did work with people who floated around more and got that sort of experience.

Ben Scofield  37:55
But again, at a very, very small startup, you don't have the luxury of focusing. And so if you are to succeed in that role, you are doing at least one of the skills that a really great Staffless engineer needs, which is communicating. Maybe another is figuring out with limited context,

Ben Scofield  38:16
and maybe that's a big C context. I don't know what you're trying to accomplish and how to make sure that everybody you're working with is on the same page for that.

Ben Scofield  38:27
And I think that that is a skill that is separable from being an early stage engineer at a company and is one that you might be able to train by working with agents. 'Cause agents are dumb. They only know what you tell them and what they trained on, which granted is the entire internet, but still they,

Ben Scofield  38:47
they only know what you tell 'em to focus on to some approximation. And if you can get good at managing the context for each of them and specifying what you're actually trying to accomplish, you're gonna get much better results than if you don't get good at that. Yeah.

Ben Scofield  39:03
I've mentored people and folks I've mentored to Staff roles have said that the most valuable question that I taught them to ask was, what are we actually trying to accomplish here? If you can answer that for the agent, you're gonna get a better result.

Valentino Stoll  39:20
So what about the leadership skills that are also required, that are similar but maybe slightly different than managerial skills that the Staff engineer inevitably needs to be successful?

Ben Scofield  39:31
I don't think you get those by talking to agents, sadly. I think not every training path will develop every skill that you need to be successful, but I think it's better to be able to identify certain skills that you'll need and be able to train those, and then you can look elsewhere for the other ones. So I think the management track has some decent leadership skill training.

Ben Scofield  39:52
I think talking to people with more power than you in your situation and figuring out how to get them to do the things that you think need to be done, that's the core leadership skill I think that Staff engineers need,right? 'Cause Staff engineers generally have very little actual power. It's all influence instead.

Ben Scofield  40:12
And for that, like reading things like Robert Cialdini's Persuasion and reading psychology, like this is a place where you should branch out. If you wanna be a Staffless engineer, you should be reading in psychology and in motivation and in leadership texts and biographies of great leaders. I think there's a lot to learn there. I don't think working with agents is gonna get you leadership skills. The specific.

Valentino Stoll  40:32
Learning, training, it makes me think a lot about what is missing from this whole industry really,right? And I feel like I keep seeing a lot of like, you're giving the knives, box full of knives basically with a whole bunch of other useful stuff in there.

Valentino Stoll  40:53
And you know, you're saying, okay, hey, go use this thing. And like they go to reach in and like obviously they're gonna get cut,right? Everybody else knows there's knives in there and they just like can move around them. And I feel like what is missing really is like layering,right?

Valentino Stoll  41:08
Like we're missing this, okay, you're gonna use this thing in this specific way because you don't know how to use it. And maybe this is a learning mode. Maybe there are modes or restricted knowledge or things like this. And the more that you use it, the more that you get exposed to because you know about that foundation.

Valentino Stoll  41:26
And I feel like this is what has been missed from software in general is like you have these bootcamps and they're like, allright, go build a web app because that's what everybody builds and you're, that's what you're getting trained for. And you go and you learn how to do it, but really you're missing all that foundational stuff that maybe more traditional like computer science programs set you up to maybe learn better.

Valentino Stoll  41:47
I don't know if that's better, but just differently all the same, you get like a base shared knowledge that can compound,right? And going back to like the folks at Every, they're doing this compounding engineering. It almost seems to like be a recurring theme too in the industry of as you start to compound your learning,

Valentino Stoll  42:07
the past learnings then start to build on those additional ones. And so how can we start to set these tracks up in different ways? If you're trying to get to a, a Staff level engineer and you're a senior, you're like, there are some clear paths there, but there are also a lot of non-ones.

Valentino Stoll  42:26
And really depending on your domain of expertise, like that could be different as well. What are your ideas in this kind of realm of like layered adoption? Like I think about like the Unix philosophy of like small tools that work together and can compound to create better tools.

Valentino Stoll  42:45
Are we following that path or is this completely new territory? What do you think about this kind of thing?

Ben Scofield  42:52
I think that is an active area of research in the skill acquisition and expertise development domains in academia. How much skill decomposition makes sense.

Ben Scofield  43:03
And there are people who say that none, like the absolute best way to learn soccer is to play small-sided games with manipulated rules that cause the player's attention to focus on different things or train a pitcher with a slightly heavier ball or a slightly lighter ball or different constraints that lead them to, I,

Ben Scofield  43:21
I don't wanna get too deep into the technical weeds, but they lead them to figure out how to solve unexpected problems on their own. So it's essentially teaching them how to learn how to solve motor problems as opposed to teaching them a specific solution. I think most domains, software development included, don't do a great job of this. I think computer science is great for learning computer science skills.

Ben Scofield  43:41
I don't actually know that it makes you that significantly better at being a software developer in today's world working on a web app. There are things that are absolutely useful, but there are things that are not useful for that too. It is an unsolved problem as to what the skills that you need are.

Ben Scofield  43:59
And partly because there's not a single answer to what are the goals of software development. There's not a single set of skills that's useful for it,right? You can be the best basketball player in the world and not be as good a center as Shaquille O'Neal or as good a point guard as Steve Kerr or there are lots of different ways to be good at software development, which means that there's not a single set of skills that'll get you there.

Ben Scofield  44:16
And so to some extent, your exploration and your development has to be guided by what you care about and what sorts of things you wanna do and what you want to accomplish with software.

Ben Scofield  44:25
But that said, I think one of the things, and maybe agentic AI helps with this, is lowering the bar to experimentation, increasing the feedback loop on seeing more fleshed out examples. So when I was coming up and when you guys were coming up,

Ben Scofield  44:45
we'd work at a couple of big companies and see a couple of ways of doing big company architecture or something. Now I can have Claude generate, again, like five different examples in an afternoon and I can look at them and see how they differ.

Ben Scofield  45:00
And to some extent, the most successful theory of learning across like these wicked domains where the feedback loops are not clear, like the inputs and outputs, the relationships are not very clear, is looking at examples and identifying not specific patterns,

Ben Scofield  45:15
but like patterns of patterns that these are some of the things that can vary and still get you to the goal. And getting more examples of that I think is gonna help train engineers faster than anything else that we have currently.

Valentino Stoll  45:31
I like that theory. The thing that sticks out to me though is that the difference between the architecture I was exposed to at my first or second job and the architecture that an AI generated example can expose me to is that the first one I needed, like there was immediacy to that. I needed to understand it to keep my job and to do well.

Valentino Stoll  45:50
And I had this drive to do well. And I think unless we bring that same sort of immediacy, I'm not saying it's gotta be like you're fired if you don't understand it, but unless we bring this stuff into something that is critical path for the organization that the person is a part of,

Valentino Stoll  46:09
the aspiring engineer is a part of, then it becomes, you know, like an exercise for the reader. And I think it loses that impact that it had by necessity.

Ben Scofield  46:19
Yeah. Absolutely. One of the problems I've always had with learning a new language or a new framework or whatever is I don't actually need anythingright now. And so if I start doing a toy version of it, it's a toy version of it and it doesn't tell me all the things I need to know. There was a talk I wanted to give for a few years and never got around to submitting anywhere called The Best Code I Ever Wrote,

Ben Scofield  46:39
where the twist was that code never went to production. It was when I worked at actually Chad Fowler. I worked with Chad at Zechs Lunderkinder and I wrote the message bus. I think he talked about the architecture like it was super, super microserviced.

Ben Scofield  46:53
So I wrote essentially the routing layer for those microservices and I wrote it in Ruby and it was beautiful and you could read it and you knew exactly what it was doing. You had to change it. You knew exactly where it went. I love that code so much. And I ran it and it was dog slow. So I reran Enclosure and it was amazing.

Ben Scofield  47:15
Still, it ran for like years and years after I left, even after the Microsoft acquisition. It was great. You're absolutelyright. Having an actual need is infinitely better at driving actual development of understanding and exploring all the things you need to find. I think it's a little bit of a double-edged sword in that sometimes you'll stop once you've solved that need and you won't go further.

Ben Scofield  47:33
And so you won't find the dusty little corners that have things that will bite you later. But yeah, finding an actual need is like pouring rocket fuel into your learning.

Valentino Stoll  47:41
Now that said, you know, there's the story of fighter pilot training,right? And they go into these simulators that everybody knows is not going to kill them. And they go in there for like two hours. They come out drenched in sweat. They're adrenaline like jacked up to here. So whatever they've been able to do,

Valentino Stoll  48:00
they've been able to put, the instructors have been able to put their trainees into a situation where, yes, it's not life or death, but they can suspend their disbelief enough to sort of act as if. And then I think the training is, I am assuming the training is a lot better because you feel like your life depends on it. Now going back to, I've never written any code where anybody's life depended on it either,

Valentino Stoll  48:22
but I certainly have written things where I felt like, man, I really better get thisright. And I think that that, I'm just using personal experience. That has helped me to feel like, you know, whether it is for the respect of my peers or the satisfaction of a job well done, that kind of a, an experience mattered to me in how much I learned.

Ben Scofield  48:41
Absolutely. I think this kind of gets back to the striving achievement play before,right? So if you're purely achievement motivated, you only care about the outcome. In those cases where you cared about the code, you were striving. You cared about the outcome, but there were also non-direct outcome things that you cared about that you were doing.

Ben Scofield  48:59
I think figuring out how to put yourself into a striving mindset for this learning session that you're gonna go on where maybe you prompt Claude to generate a bunch of different examples, maybe that's the key,right? Instead of having, I need a to-do app, maybe it's, I really need to understand how this ORM works.

Ben Scofield  49:16
And so I wanna see like five different examples of how it works because people have done forcing functions of submitting a talk on a topic and using that to force them to learn the thing. You can make up your own things that you have to accomplish out of learning. Maybe you wanna contribute to an open source library.

Ben Scofield  49:35
And so you force yourself to learn the things at the level you need to, to do that.

Valentino Stoll  49:39
It's funny 'cause I'll start vibe engineering something and it might ask, hey, X, Y, Z, what are you really trying to make out of this thing? I just don't have time for it. I just say, oh, whatever, just pick one of these things. And so, you know, how do we take a step back from that?

Valentino Stoll  49:57
Are there like actual channels we could start to build to help train people better? Is this worth effort to focus on, to force people to think about and react to the things rather than just push past it and get to an end result? 'Cause likeright now people just want to push a button and like keep hitting enter to see the result.

Valentino Stoll  50:18
And like, what's the force function to like skew that into a new shape,right?

Ben Scofield  50:24
The sad answer is there isn't one,right? The forcing function is for people to care about what they're doing for some reason. It's not like there is a single reason that would force everybody to care about it. I think it's perfectly fine to say, I don't actually care which of these three options you take. Claude Code, do one. As long as you are being honest about that and caring about some of the options.

Ben Scofield  50:45
I hate authentication. I hate it. It's terrible. I've written it a bunch of times. Now I am perfectly happy with, gimme the median devise implementation for my rails app. Don't care. Do that. Or something else. I don't care. As long as it works, I don't care. But there are aspects of the user experience that I care a lot about.

Ben Scofield  51:03
And so you either gimme options or I will then go in and hand edit it. I'm not Gastown level six or whatever where it's like never look at the code at all. The forcing function is deciding what you care about and then caring about that. There are lots of posts on doing the thing is doing the thing and not doing the thing is not doing the thing. They're going around various places. My overall advice is be intentional about what you're doing.

Ben Scofield  51:26
Say you're gonna be intentional about it and then actually be intentional about it. That's the forcing function to get you to care about the things that you wanna care about. There's no shortcut. There's no magic.

Valentino Stoll  51:35
Intention engineering.

Ben Scofield  51:37
Yeah. No.

Valentino Stoll  51:39
It makes me think like of the old, like a Microsoft DOS games where they're like, choose your own adventure, but like it's very specific. At this point, you know, you can use one of these things. Or you don't know how to use it,right? And so like you have to like pick theright thing in order to know that you can use it and then you can use it. And I wonder if like we're kind of experiencing the same thing,right?

Valentino Stoll  51:59
Like should we be prompting people if they're, I wanna make a to-do app and they're like, okay, what do you know about web apps? Right? And if their answer is forget about it or no or like, I don't know, it steps into a new layer of, okay, well what do you know about the routing layer? And like the more that you like enrich it, then it gives you the more permissible next step of,

Valentino Stoll  52:20
okay, yeah, we'll just build this thing. Maybe that's what we are missing is like intention gathering and skill acceptance. Okay, this person knows this skill. Like we're gonna skip all these other steps because they don't need to learn about it or understand,right?

Ben Scofield  52:36
150%. I wanna make a to-do app. Why? What's your goal? Is your goal to explore how this new frontend framework works? Cool. I will use bog standard to-do backend database model. I am unhappy with how Notion allows me to classify things and subclassify them. Cool. We'll talk about the database schema. I think making our agents aware of our intentions,

Ben Scofield  52:57
which requires us to be aware of our intentions, I guess that gets back to my like, I don't think I have privileged access to my own internal mental states thing from before. That's hard, but it is absolutely key.

Valentino Stoll  53:08
I hope somebody builds an intention-driven Claude that forces that. All kinds of stuff,right? I just wanna use the tools. I don't wanna build them in that way. I hope somebody's listening out there and like, you know, the next day I just like, oh, hey, look at this new thing. You know, like that sounds like what I was talking about. Yeah. You never know.

Valentino Stoll  53:26
But I personally, I work at a big organization at Gusto and we have like a lot of these similar challenges of like training and adoption of AI tooling and all these things and how do you get people to use them theright ways and, you know, varying, you know, levels of expertise. How do you get everybody onboarded in a similar but different way to make the most use of the tools?

Valentino Stoll  53:47
I mean, it's a challenge and not one that's solved easily. If you're out there and you have solutions, keep trying 'cause it isn't there yet or I haven't seen it.

Valentino Stoll  54:00
We're way over time. I don't know if we wanna wrap here. We'd love to bring you back, Ben, and continue to follow this thread.

Valentino Stoll  54:07
I think that people that are listening to the podcast, you know, aspiring engineers, people who have been in that staff role, people who wanna be in that staff role, I think that that, and people who are training others in any capacity or just want the people around them to be constantly improving. I think there's a lot more to be learned and I'd love to continue to keep talking about it with you.

Ben Scofield  54:28
Yep. Happy to chat whenever. Thank you guys for having me. This has been a great chat.

Valentino Stoll  54:32
Yeah. Yeah. Thanks, Fenes. Thanks, everybody.