How to Learn ML in 2026 interview

I had the pleasure of being interviewed by Yacine of Deep Learning with Yacine. He posed a lot of very thoughtful questions and I had a great time. Please check it out!

The lightly edited transcript follows, auto-transcribed with faster-whisper and then reviewed by me for correctness, adding links for context, and editing out a lot of “I think” and “um” and “like” selectively.

(Yacine) A lot of people asked me how to get into machine learning in 2026. Instead of giving you my usual advice, I figured it’d be way more interesting to sit down with expert contributors who have like a diverse background and ask them what do you think about that. So today we’re chatting with the legendary Vicki Boykis. She’s an absolute gem if you know her from Twitter and Bluesky at giving well-grounded advice that are full of pragmatism. She has a pretty impressive career so far. She built a recommendation system at Tumblr. She worked on the ML platform at Duo Security.

She worked on LLMs at Mozilla.ai and today she’s a founding ML engineer at Malachyte, a New York startup doing real-time search and personalization for e-commerce. The cool thing is that she’s actually self-taught and actually started with a degree in economics. Her first job was in trade consulting which she described as just doing a lot of Excel spreadsheet stuff. Then she noticed her co-worker automating their analysis with R and SAS which I haven’t heard in a long time.

And she asked herself how do I do that too. That question, how do I do that, pretty much drove our entire career. After first asking herself that, she then did an analytic outcome cast, then she went into data science and then machine learning engineering. Every time she saw someone one step further down this machine learning kind of career pipeline, she went and learned what they were doing on their own and then internalized that skill. So I asked her this simple question which is if you were starting learning this stuff in 2026, how would you actually do it?

It was a very insightful and pragmatic conversation and I hope you enjoy it.

(Vicki) I graduated with a B. S. in Economics. So nothing to do with machine learning or programming. I started working in analytics and international trade consulting, which is a fancy word for saying I did a lot of Excel spreadsheets. I saw my co-workers were using this thing called R. Oh God, that’s so long ago. R and SAS to automate some of the analysis we were doing. I’m like, how do I do more of that? I want to be able to control things. And then from there, I moved into data analytics and worked at a few places like Comcast, also working on some of their recommendations and then more into data science.

And when I was doing data science, I saw that people were actually getting data programmatically themselves. And I’m like, okay, I can analyze the data. How do I actually get that data? So that’s when I moved more into machine learning engineering, which is the process of synthesizing the data, being able to ship it. And then I saw that people were not only building models, but they were shipping it to production. I’m like, how do I do more of that?

So my whole career has been about moving towards doing more and more and more of that process myself. And I think that’s pretty reflective of where the industry has been headed in general, right? So we started out with data science or with analytics, with bespoke packages, bespoke models. And now we’re at the point where we’re running enormous clusters that are running LLMs. And now we have inference engineering as the hottest topic.

So I think my career fairly mirrors that. So like I said, I’ve worked at Comcast. I got into the recommendations and search space completely by accident, I think, as a lot of people do. I love it. I love working on search and recommendations.

I worked on recommendations for Tumblr, which is an old school social network. Then I worked on ML platforms at Duo. I worked on LLMs at Mozilla.ai. And right now I work at Malachyte, which is a startup based in New York doing search and recommendations and real time personalization for e-commerce. So, it kind of echoes of the same thing that I’ve been doing.

(Yacine) But for e-commerce merchants specifically and personalizing their stuff in real time. So cool. And I feel like every time I ask somebody, like how they got started with them, they never, it’s very rare that somebody went to, oh yeah, I did a bachelor in machine learning. Like they usually come from all over the place. In my case, I’m from neuroscience. So like I had to pick up like machine learning because I was like, how the hell I’m going to do like all these stats on like these data sets. There’s just no way. And then they were so like complex that I had to programmatically like interact with them in some shape or form. And I had to run them on the Slurm, with Slurm on the cluster. So I had no choice, right?

Otherwise, like I’m not going to get the work done. But then you start to, I don’t know, I feel like once you start to kind of realize what you can do with it, it’s kind of like something like a facet of programming that just appear, right? A bit like before you know programming, you’re like, you don’t know, like that you can automate stuff deterministically, then you learn maybe about like functional programming, like object oriented, there’s different ways of thinking about that. But then you learn about like machine learning, where you have like this black box, and then you can feed it data, and then the output that you want, and it will automate this without you having to like define it. So do you feel like machine learning is more of a set of tools and skills than like a career, per se?

(Vicki) It certainly seems like it’s becoming more of that. I think like we have a lot more blending, because machine learning used to be, like you said, very much a black box, right? It just kind of sits over there. And there’s even a paper about it, which is not machine learning is the high interest credit card of technical debt. But there’s another one that says like the model is the black box. And basically all the parts of the machine learning system are around it. And you don’t know what’s in it. But now what’s happened is like everybody has access to this model, right? Everybody has access to the API.

And so we don’t necessarily have access to what’s inside of it. But at least more of that has been exposed and is now being synthesized in products a lot, a lot more than it used to be. So it definitely feels like it’s become more of a tool that you need to learn. At the same time, like with my own biases of working with a lot of these larger systems, is that these systems also themselves have gotten bigger and have integrated more of like, allows in those traditional tools.

(Yacine) And so you still really need to know what you’re doing with machine learning to figure out where you want to incorporate that. This is good. And I think the big question I want to ask here, and I have a whole bunch of others, but like the big question is like, what do you think like, in 2026, someone that wants to learn like mastering or deep learning should be doing right? And let’s say it was you, like, back in the days, and now you realize that there’s this whole thing, what would you do?

(Vicki) Yeah, this is a big caveat. Because the thing with people who are kind of like in the industry is they don’t remember what was hard. So I’m thinking back to like myself, when I was starting, we might get into this a little bit later, using Git. It’s like, oh, yeah, just now you just use Git. But I remember the first time I used it, I wanted to cry. I’m like, Oh, my god, why do we need this? Like, it’s so hard.

Why do we do this stuff? And then just when I would ask somebody to like, for example, I needed to run the report, they would be like, Yeah, you do these three things. And then you start running the report. And you’re like, Oh, okay, you need to connect to the database. Oh, you need to shell into this thing. There’s all this other stuff that they forget, because it’s just so intuitive to them after doing it a while. And I feel like that’s true with, with starting out too. So that’s my caveat for what I would do, because I don’t know, but I was looking at, I put together a gist, it’s called normcore LLM reads.

And it’s called like the anti-hype LLM reading list. I put this together, I think, in 2023. It was just when ChatGPT came out, and everybody’s like, Whoa. And I think everybody in machine learning was also like, Whoa, even though we’ve been working on this stuff, this kind of came out of left field. So I took a look at it. And I’m like, Well, there’s a lot of building block components that go into this. It’s not like ChatGPT just emerged out of thin air. And so at that time, I put together a reading list that had things like word to back transformers and support vector machines, like deep learning systems, fundamental ML readings, linear algebra, all of this stuff. And I’m like, what would I update this list to now, it would still be the same.

Like, I think that if even if most of your work is with APIs, it is empowering to understand what’s going on behind the API. With local models, you can just take it all apart and look at every single thing, look at the stack trace, etc. But even if you’re working with these other models, and especially if you want to evaluate like stuff like Jev, or whatever that just came out, you need to have the context of like, this is where it fits into machine learning.

Like, it’s not whoa, Jev came out of nowhere. It’s like, okay, you know, we’ve been doing prediction and classification for a long time. Like, that’s what machine learning is, right? prediction, compression, all those things.

And I think I would learn the same things like do the books, do the old school readings, like build nano GPT, all that stuff, because what it does is it grounds you, it says, okay, this is not entirely new. We’ve done, you know, history doesn’t repeat, but it rhymes. We’ve done this before. I’ve seen this pattern before. key value, cash is just a hash map, or it’s just the dictionary, or it’s just like Redis, I actually gave a talk about this once.

And once you do that, you start to learn patterns, and you become a lot more powerful. So my TL; DR is I would still learn all the same stuff. And moreover, I would even start to learn from books, there’s something about like reading text for humans. I know a lot of people have a lot of different ways that they consume information. But there’s something that like when you highlight when you mark it up, when you do the equations yourself, you really start to soak it in a lot more than you would. The one thing that can help now is if you’re confused, you can ask, right? You can be you can ask, I’m confused about this, explain it to me like I’m fine. I still don’t understand it. Give me a concrete example. Outside of Perplexity, like within harnesses, we don’t utilize links to web search as much as we should. So we have this thing that can augment us.

But if you want to learn machine learning, like most of the pain should be on your side, because you should be learning, not the machine. And then you can power the machine.

(Yacine) I think that’s, that’s so true that like, you have like, especially if you’re starting out and you don’t understand the concepts. I don’t know. I feel like there’s, there’s no, there’s no shortcut. Like you have to struggle a bit with the material and figure out the concept for yourself and like internalize this link. Like, just like pretty much any, any field, like if you want it to become, I don’t know, like a, like a biologist, right? You can’t just speed run it. You have to understand the basic, even if some of the stuff are automated now, like being able to understand like, I don’t know how to microscope is actually working like how you’re, you, you, you should do like your Western block and stuff like that.

Even if you have a machine that is doing this, like it does help a lot in grounding your, your, your understanding of the component. And then like, when there’s something else that comes out, you can pattern match it a bit better, which is interesting because the, the job stuff, I find it very interesting because a lot of the people that knows what, how to build this stuff and things like that, they’re very not impressed. They’re like, yeah, it’s a, you can do it yourself and stuff. But that’s it. Like, I feel it’s the same thing that happened with, with ChatGPT, a bit like people that know, knew this stuff. They were not that impressive. Like, of course, like this is where it going. But, but I think there’s a point where the technology reached something that is directly useful for people to start your thinking without any understanding whatsoever. And like, don’t have to hear about any complicated word, right?

(Yacine) And they just start to go with it and then, then it take offs. How the same, I have the same feeling about this. What’s your thought? I know we’re a bit off track, might cut this up, but like, what’s your thought on, on the, on the Jev stuff?

(Vicki) Oh, I don’t, I actually haven’t read too much about it. And this kind of goes into my philosophy of like, I skim what’s on the feed. And then if people are still talking about it, two to three months later, I’ll be like, okay, this is, this is really something like I need to deep dive on it.

I’m not always great at this. Like sometimes I’m like, oh my God, I need to know about this right away. But usually I found, so for example, was, you know, like RAG used to be really popular in 2022 but now it’s dead. Who knows, I think it’s still useful. Like I use Perplexity all the time. And then all the links have been incorporated now, even into like Google fast search results, they synthesize and then all these services give you contextual links.

So it’s a question of like, there’s so much stuff. If you actually be interesting to do an experiment of what the top Hacker News headlines were in 2023, when this stuff came out. And then what has gone out of fashion or has not been used or been subsumed by the models, right? Because a lot of it is like, you need to be doing looks seem to be doing graphs. You need to be doing chain of thought, right? Which is now part of the model reasoning.

So it’s like, it’s good to kind of like find skim. And then if it is relevant, it will come back into your life somehow.

(Yacine) Yeah, yeah, definitely. I agree with this too. And because like there’s like, especially if it’s, it’s very useful. I feel for like economical valuable task, I feel like people will figure it out and like, we’ll test it out and then, oh, now it works super well. And then it like, it will, they will use it. I know that because like, I’m doing that type of work with the client. So like when Jeff came out, I was like, I’m going to just do use this for two of my clients and I’ll be done with this part of it. I don’t have to fine tune anything. I don’t have to like train from scratch. Like, that should be good enough for the, this like sort of classification that I have over there.

I don’t have to have an LLM that I’m trying to do stuff for speed and stuff. Yeah. I feel like there’s there’s a lot of this, but it’s very, very confusing for newcomers that don’t have the grounding. And they see like this, this fire hose of new stuff coming out every day, like every day, if you’re not careful, there’s journey, like some new things and you have the grounding, you don’t know, they don’t tell you everything also. So how do you, how would you deal with that? Like, like as a very early beginner, like, would you like be looking at these news even from that time or like try to do a deep dive there? Or would you just head down, grind on like the basic?

(Vicki) I think when I was a beginner, I for sure looked at everything cause I just didn’t know what to look at. I remember when I got started, I would just basically skim Hacker News headlines every day.

Cause I’m like, oh, this is where everything is going on. And it’s good to see that still is mostly the case, which is like one thing has taken us at least. But I would be like, oh, I need to know this and I need to do this. And I would be very swayed by this stuff because I didn’t have the grounding that I talked about earlier, where I knew like how to separate the wheat from the chat. And that only really took time.

Because then you can evaluate it very deeply and say, yes, okay, this is something. And I think even now, like I can’t, I can’t really tell because with a lot of stuff comes out, there’s like a lot of glaze over it. A lot of stuff doesn’t have papers attached to it. There’s a lot of people talking about how they got all these results. And even as like an experienced practitioner, it can be very, very hard to separate that out, but it’s much, much harder if you don’t have these fundamentals. If you’ve never worked in systems and production, if you’ve never kind of tinkered with them to be able to evaluate them for yourself. That’s why that’s so important.

(Yacine) Is there anything that you would skip in 2026 versus like when you got started or like you would still go through the same path?

(Vicki) That’s a good question. I don’t know if I would skip anything. I think that’s hard to say because a lot of the times the stuff that you learn that you don’t need can sometimes also lead you to stuff that you do need. And it’s like, I used to have a math teacher in high school, we would always be like, why are we learning this? Well, she’s like, you’re only going to use 10% of the math that you learn here in the real world. You just don’t know which 10% it’s going to be.

So I have to teach you all of it. And I think that’s true. Like we never know what we’ll use or what we won’t use. So we can’t just try to, you can try to curate your experience, but I think like failed experiences are helpful too.

(Yacine) I think that’s a great point. Like, if you actually learn a topic in depth and push it out to the maximum of like, okay, I understand now what it is. Even after that, you’re like, yeah, nobody’s using this anymore.

We’re using that sort of stuff. I never felt like that was wasted time, right? In the sense of like learning about whatever gizmo that this company did when the product is like random, like when it’s come to technical topic, there’s always something in there, right? That you’re practicing as you’re, you’re, you’re discovering. And, or like just the meta scale of there’s this unknown and I’m trying to now understand the unknown. And then you just iterate on it. That is massively useful.

I say that because there’s plenty of papers that I’m reading, like spending a week and a half on it. And after that, I’m like, wow, that was garbage. Like literally like, there’s no use for this at all. But the topic in the way the experiment was set up and then after that you realize, okay, like they tried to this and they put this duct tape. So like that shape is not that useful, but even when I look back at these articles, there’s like, there’s plenty of stuff that, that was to get out of like, there was one, I remember this article, because I spent like weeks on it, when I was doing my master and it was like a emotion recognition from like physical physiological single and the results were like crazy and I was like, wow, I need to understand what they’re doing. And then when I realized that like they did like just normal key, k-fold cross-validation, instead of like taking a person out and then testing on the person, I was like, Oh, okay. Because there’s no way you’re going to use the model after that.

Like if you’re using, like if you have to train on the people, it like to, to be able to understand the people, like you’re going to do what, like you get labeled from them and then now you retrain the model and then now that the person can use it. Like that’s not how I work in during the inference and for instance, like there’s another person that you don’t know about. And, this is where like it led me to like a physical single fingerprinting, which I didn’t use also in the neural experiment because the brain is even worse in the brain. Like if you could literally see my brain in a spectrogram and so like, yeah, this is the scene. This is Vicki, like this is this other person. So then I was very careful after that to not make the same mistake of this paper. The paper was garbage, but like the learning was, was still super useful.

How do I as a distributed system software engineer with years of experience begin to learn ML? Like, I think there’s a lot of people in this boat, where they already have a whole bunch of experience, like experience, that are, that is technical, right? It’s not like, I don’t know, like they, they come from absolutely nothing, right? They already have technical experience, but they kind of feel that they don’t know how to approach like, demo stuff. Would you still go through the same route that, you’re outlining in your, normcore LLM gist?

(Vicki) I think if you already have distributed systems engineering, you’re at a huge advantage. Or any software engineering, really, I think this is going to be a hot take, but it’s a little bit easier to pick up industrial applied ML. If you have software experience than software, if you have ML, like it’s going to be harder. It’s harder to go to one way than the other. And the simple reason is that ML is written in software.

And I think like the biggest thing that you have to understand is a foundational thing, which a lot of us have learned over the last three, four years is that it’s software, but it’s not deterministic. And if you understand this one basic thing, it makes everything a lot less frustrating for you because even especially distributed systems are not always deterministic, but you accept, you put in an API call and you expect a result out and you know that you’re always supposed to get the same thing, or you write data to the database and you know that it’s supposed to write to the database, or you mark the button as blue and you know it’s supposed to be blue. Machine learning systems are all about trying to create a representative model of the world. That’s not certain, like we can never make it exactly like the real world.

And so a lot of the answers we get and a lot of the engineering work that the labs have been doing is trying to move it towards certainty, right? Like even with Jev, for example, people just want exact outputs, right? And I think like the biggest underlying understanding is that it’s not a deterministic system. And once you have that, I think a good thing to do, maybe this is still a good thing is to try doing a Kaggle project. Kaggle projects have historically been really good for learning that stuff. And there’s a lot of alpha, as the kids say, in all the Kaggle comments and all the notebooks and all of that stuff. And I think just literally pick any project on Kaggle and start to try to reproduce it. And again, like, do it yourself first. Once you get stuck, start asking questions. Why is this happening?

What is this? What’s a p-value? How do I set up this experiment? Like what are these fields? I think it really helps to have like some sort of template for what to do. And Kaggle is probably still the best place that provides that. If there are blog posts that go through it, by all means, you probably could get a model to generate your project.

Although I would still prefer to use like the human generated stuff, especially with the data, because it’s just like, it’s so fine grained and it’s so nitpicky and it’s so specific that I would, I would look at some of those well-vetted notebooks and go through cell by cell and see what each thing is doing.

And then once you do two or three different projects, you know, Titanic, iris, whatever, I don’t know. I don’t know what the cool thing is anymore. I think it might be like penguin species or something, but you do a couple of those and you start to get a feel for the machinery of how a project should run. Because in software development, it’s like you set up your system, you set up your local testing, you have your application, you run it, and then you check the results for machine learning.

It’s like, okay, I need to build a data set. I need to really understand the data that I’m building. I need to do an analysis first. Then I need to select a model. How do I select the model? It’s an art because you don’t know which model makes sense for which task you have. What’s a machine learning task? Okay. Now I’m learning about the different kinds of tasks, right? Prediction and completion. Okay. Now I pick a model.

How do I get that data into the model? How do I train the model? How do I look at the loss? How do I interpret the loss? Which optimizer should I use? It’s very, very different than just like, I’m running code and building it. It works or it doesn’t work. So I think it takes like a lot of trying those different projects to get to the muscle memory of like what that kind of flow looks like and what it means to move from deterministic code to non-deterministic where you start with data and it’s very squishy.

(Yacine) Yeah. I agree here. I think like a project is good. I also agree that like you shouldn’t ask, I don’t know, like an LLM to generate you. I feel like the stuff that the LLM generate to, like to teach is sometimes a bit too over-engineered for absolutely nothing, right?

And then you’re going to be there like still wondering like why you’re using this and that. But to learn when you already have a material, right? That like it’s kind of proven to be good. Using the LLM to explain and walk you through stuff and give you like extra example. That this is actually pretty good. Even with research paper, I found like, like when I’m struggling with like some of the part, I just ask it, like, can you just give me an example of what the hell they mean here? And then sometimes it also, I like that it’s just not well explained, but this comes from like these three papers that they’re mentioning, but you haven’t read.

And this is exactly the same example here, but like they messed it up. I feel like this is something that like, I don’t know, like in 2018, you wouldn’t have access to this stuff, but now you can leverage it to be able to kind of walk through problem without having to kind of wait, right? You could just, there’s uncertainty. You think about it. You ask the agent, can you explain this? You explain it. You go back like, yeah, that makes sense.

Talking about the past, like you said in the 2022, most ML work is software engineering grunt work, like get SQL command line. But like models are pretty good at that now. Do you think that it’s still about this, this type of grunt work is just shifted in different area? Or like the ML engineering stuff is still, is different, but you still have grunt work to do? What do you think?

(Vicki) This is a really good question. I was thinking about this once you asked it. And I think that yes, it still matters to learn the surprise. Yes, it still matters to learn this stuff. Because when I think about how I use agents on a day to day basis, I’m still doing git all the time. Like all the time, I’m asking them to commit stuff. Or I’m personally committing stuff. Maybe this is like a good segue. But the way I use it is more like, I know some people use agents to to create loops, and they just let that loop go for a bunch of hours, and then they come back and check them. For me, particularly with machine learning work, where you have to deal again with data, nitpicky interactions in data between different services, I can’t do that. I have to be close to what the agent is doing.

And so a lot of the times, I’ll prefer to actually even do the git commits myself, I have to be looking at the diffs, I use hunk for that hunk is really great.

And so in order to do that, you need to go get like not amazing, which is the great part is you don’t need to be an amazing person and get I know a couple of people for masters at it. And they were like gods. I think it’s still a great skill to have because you can move fast. I’m a fairly confident Git user, but I don’t like I’m not going to get into the top 10 to 5% of use cases. And that’s amazing [to have agents handle that].

But at the same time, if I didn’t have that muscle memory of git fetch, git rebase, git commit, git push, git pull, it just speeds you up. And if you have that muscle memory initially, whatever you do with the LLM is just going to make you go five times faster. So I think having that baseline is important.

SQL, perhaps, like, yes, it’s not as important to you can generate, you can have a generate the queries, but you still have to make sense of it. And so a lot of the time, like, when you’re looking at data, I personally still like to call touch the data, it’s not really touching it.

But, I want to run the SQL query, I want to look at the results of the columns, I want to get a better feel for what’s in it. And the more I personally delegate that to the machine, the worst results I get, because the further away I get from where I need to go. And so yeah, and command line, obviously, like the agentic stuff, like, you have to know it.

Perhaps you don’t need to do all the intermediate stops, because it uses graph, bash, etc, as a tool, but you still have to be able to navigate it. And like, the more you get these, these are also fundamentals, but just in grunt work software, the more you get these down, the more you can command the agent instead of the agent commanding you. So it’s important for that.

(Yacine) Yeah, this is this is very, very pragmatic. I love it. I also like to be in the loop, especially when you’re doing something that has high context load, in the sense of like, you’re not sure if everything is there, right? Because if you’re for example, the data, maybe there’s something missing [in the data], or there’s something that was said, and there’s some behavior, they’re trying to like, predict and whatever it is, but like, you, you’re not sure how 100% you have enough, even if there’s a lot of data, there’s maybe something that is that is missing in there. If you just let the thing go, it doesn’t know, like the world is this for it, right? So it’s not going to be asking and stopping. And so like, hey, can we get that? Or like, how can we infer this thing and then make the problem simpler?

Usually, it just goes and they take like, whatever you’re giving it as like, this is the world and like, there’s nothing else. And like, we’re just going to like, make our the very best solution for this, which end up very convoluted and sometimes like a bit flawed. There’s there’s a very, very high amount of value in understanding what the data is saying. Because then you understand, like how we connect to the context, the business case, or like, whatever the outcome you want to, to do, and then you can make something that will make the whole problem much simpler for the whole thing. And then it’s that then it’s like, okay, you can take it and you can like, have the LLM like, think of different ways of like, improving like the system or whatever it is, juicing it up a bit. But like, that insight, I’ve never seen a proposal like, hey, maybe you should ask them about that thing, right? You don’t know, like, it’s very hard for them to, to know that stuff.

(Vicki) Yeah, the way I usually think about it is like, are you steering the model? Or is the model steering you? And I’ve definitely had cases when if it’s perhaps a software package that I wasn’t as familiar with, or even a repo that I was working with that I just didn’t know.

It’s like, okay, go, go explore this, because I don’t know anything. And the model is like, would you like me to do this? I’m like, yeah, sure. I guess that makes sense. And at that point, it’s like, you need to stop and read more about what’s going on. Because then I have cases where it’s like a repo I’m very familiar with. And it’s like, do you want me to do this?

No, this is not what I wanted. I have an opinion. I wanted to do X, Y, and Z. Yeah.

(Yacine) And this is where you see that they have a tendency to do a lot, right? Just do too much sometimes. And they’re like, did you want me to like add these three things and like do it like that? And like, I mean, we’re in and out here. I want to spend two years in this stuff. Let’s just just do this simple stuff. And let’s get out of here. But a caveat is I find very useful when I don’t care too much about I know where the output is. I don’t care too much about the middle because anyway, I just want to see if like there’s a feasibility of this. I’m going to throw away the whole thing anyway, but I’m using this to see if there’s a path. And then I’m just going to ask it like, what’s up? What do you what do you do here? If I see that like they did some some insane things. And like, I guess maybe not like this. But sometimes it’s just fun. Show me like, yeah, there’s a path over there. It’s totally feasible.

And like, I’m like, okay, cool. Like, if this is serious, I can just take it back and then just like build it step by step in the simplest, cleanest way. And it will be much faster because I already know that there’s there’s a there’s a way out of this. It’s a good way for like this other blog post that you do that I really like about building yourself flour. I think it’s like, like mastery in general, right? Like, what’s your opinion of mastery? Like, we’ve talked about like this get guy that is master and get and stuff, right. But just generally speaking, like the different bits of ML, like, how do you feel about mastery in the age where you have these agents that are kind of good in a lot of different fields and can do a lot of stuff?

(Vicki) Yeah, I think I think it’s super important. Just like I said, like it’s important to master your tools. It’s important to master the fundamentals of ML. I think because mastery is one of the things that gives you taste, which is like the moat now, I don’t even know, like I saw some posts where it’s like taste is no longer the moat, like something else. I guess taste is like a statistical distributed probability over all the possible outcomes. So we’re gonna automate taste soon.

So I don’t know, but I really think that like, getting good at something does teach you what is good, what to throw away, what to keep, particularly in complicated software systems. Like, ML has interaction between both software and ML, you really need to think about how to make the system as easy as transparent as fast and as accurate as possible. And you need to think about when it makes sense to do trade offs between those two, right?

Because sometimes you want the system to be fast. Sometimes you want it to be accurate. Sometimes you want certain things. And really, the only way you can get to that is by building a lot of these systems. And that really is mastery. Because at that point, you have opinions on what makes good, clean systems. So I think it’s still, it’s perhaps even more important because it’s now harder to get over the hump of the middle, where there’s like just so much stuff in the middle, like so many ideas, so many projects, etc. So maybe that part is harder.

But I think it’s even more important now than it used to be. Yeah. But do you think generally speaking, like now, is it easier? Because like you could use the LLM to train you a bit.

(Yacine) Do you think it’s like easier or harder to reach a level where you have actual opinion about things and like people are listening and like the system is actually good, right than before? I think it’s harder because LLMs are like the compressed average of what we think good software should be. And don’t forget that it’s only public facing code, right? So we don’t know, like, in theory, I guess they might do some post training on some internal code, probably not. But we don’t know what good software, we’re basically missing 80% of the software that the world has produced, simply because it’s in private repos, right? And that’s some of the best software, like the big companies that have run distributed workloads, etc. So we’re missing those, we’re missing those opinions.

And then because [the opinions are] the compressed average, it’s harder to break out of what the middle is and think about like, what a good design might be for your use case. And the example that I have is I watched this video from a while ago, there’s a guy like ranting about LLMs. At that point, they weren’t very good yet at all.

But basically, he said, like, if you use this to produce a flappy bird game, everybody’s going to produce the same flappy bird game, right? Because it has some ideas about what a flappy bird game should be. But you personally have to know when to differ from that and produce, I don’t know, like something higher than that. So in order to get, if that’s something that you want, for most engineering, you might not need that. And it’s just fine to build like the average CRUD app that has like the average security spec, the average routes, whatever. But if you want to get to something where you want it to be efficient, or it has a new idea, you need to go higher than that. And I think that part is a little bit harder.

(Yacine) It’s very interesting. Yeah, it’s a very interesting insight. And because now, like, I don’t know, like GitLab was down. GitLab also is like struggling for its life. Do you think that there’s more new ideas in software that is being like created than before? Or like it’s just more of the same average stuff that is being attributed?

(Vicki) I think any new idea is a combination of a bunch of good old ideas that work really well, just kind of in a new way, kind of like, so there’s no really new novels, right? We have a core set of fundamental stories as humans, that we understand a paradigm like this. So there’s like the hero’s journey. If you think about like, what’s in the Odyssey, right? Like, I was on my mind, because I’ve been talking about the Odyssey. But like, the hero’s journey, you have the person going to war, you have, I don’t know, like the sirens seduction by somebody else.

So we have all these like paradigms that are these ideas that have been circling around civilization for a while. So we just take pieces of those ideas, and then we remix them, right? So from The Odyssey, you get this movie, Oh Brother, Where Art Thou, which is like The Odyssey, but it’s in the south, right? And you get The Odyssey movie, but it’s different. So I think like a lot of good things, it’s fine to remix and so like great artists steal, right? And what does that mean?

You take pieces of certain ideas and you put them together. I think there are new ideas in that you need to be able to synthesize old ideas, which again goes back to like, knowing the fundamentals and knowing how to stick those pieces together in an elegant way.

(Yacine) Yeah. Do you think like LLM right now are generating like, it is in different ways, or like it’s more of the same generally here?

I think it depends on what you bring into it, right? It’s kind of like in Star Wars, when in The Empire Strikes Back, where Yoda is training him and he goes into the cave to fight. And Luke is like, what should I bring? Should I bring my lightsaber? Yoda is like, no, there’s nothing in the cave, only what you take with you. So the thing that you take into the prompt is the combined synthesis of your life experience, of your opinions, of your taste, whatever that means now. And so like the ideas that can be generated, maybe you’ll get some new ideas, but really that’s like the genesis of it is what you bring to it, I think.

I agree. I agree here. And I especially see it also from a kind of a research perspective. Because like a lot of the research paper, like it doesn’t look like it when you just read one, like that it just came out because it’s like, it’s a lot of stuff that was from the past, but then they did a twist on it. And then like, now they get this novel thing that they can use and stuff. And then they get these results and they try this other thing here and there.

And the whole thing looks novel, but it’s really like a very like a continuation of six, seven different things. So this is pretty good. Well, while we’re on the subject of book, do you have any non-tech book recommendation for like the upcoming era? Like this is Nisha Rand on Twitter that is asking this.

Oh, I’m going to have to go back and see what I read last year. So I have like, I keep track of all my favorite books of last year. I might have to put that into a link. A really good one was Chess Story. Super short. It’s about this guy who plays chess and goes insane during World War II.

So if you’re into that, you can do that one. I think a lot of times I like to read a fiction that’s unrelated to tech. And then it like comes back into my ideas about tech. So I’ll have to, I’ll probably like post the link of the favorite things that I read recently, but that’s a good one that I read recently. What’s the best way to loop development and learning to the same cycle if you’re early career? I think like building project versus just like reading from books and stuff.

I think the best part is if you are, are we talking about if you’re doing your own project or if you’re working at work, they’re not typically different, but like, I’m just curious. Yeah, I think they’re talking, it says sometime I find that traditional junior engineering workload doesn’t really allow for learning. Because even though, even though the AI can just one-shot what used to be a junior dev work, we just get more.

Yeah, I feel that. I think, but it’s always been true that like software development work wants you to go faster. Like that’s the goal of the company. And I mean, I know now it’s super paced, but that’s always been the case that you want to be finished as fast as possible. So if you’re a junior engineer, it’s kind of your responsibility to push back on that a little bit and give yourself time. So for example, if you’re implementing a method and you need to do it quickly, stop.

What is in this map? Like, let’s say there’s four lines and you don’t understand one of them, take the time to understand that one line. Take the time to understand that method, especially now it’s really easy. And one of the ways that I use LLMs at work is to navigate our code base.

And to understand parts that like I have implemented and forgotten six months ago, parts that other people have implemented, but I haven’t reviewed. And it only takes the thing about like stopping to understand something is we think, Oh, it’s going to slow us down. We’re going to be behind. But it only takes wild, wild, but like 10% of your time up front, but it’s going to give you 70 to 80% of the benefit later on, because you’ll really know what’s going on.

And then you’ll be able to move faster later. Yes, it adds friction. But at points in your day where if you’re implementing something, you would usually gloss over a method, like ask what is this method? What is this library? What is it doing? Give me an example. And it’s like, I’m mixed on the future software development with LLMs. But this part where you can understand things, you can link to things, you can synthesize them. I think it’s a godsend for any code base, really, but particularly larger ones, and particularly when you don’t entirely understand. So you can quiz it.

And I think a lot of times we forget that because we just use it to like generate or do work. So we don’t use it as much to synthesize. And Anthropic actually had a super interesting paper where they did in collaboration with another institute, they did a study of who was more likely to retain information when they reviewed a new code base.

And they had one that was just like just generating code, one that was done generating and quizzing the code and those people learn the most like they retain the most information. So I think we’re not using these things as much as we could be. And a lot of that is because of the UX, which is just like would you like me to generate more? Can I generate more?

So I think we should cause ourselves a little bit of friction and get use it for understanding rather than generation.

(Yacine) I agree. And the thing is, generally speaking, it has been true, and it’s still very, very much true. I think that is a summarization task, right, that you can shape it like this. And I’m just super great at that. They could be very poor at like doing that task, but just summarizing it and like just explaining like the link between the stuff. They’re pretty good, especially if you know that the thing is good. If you don’t sure if the thing is good, it will try to kind of like explain away some of the stuff because it will assume that this is a good thing that you’re giving it as the main input. But if you have a good thing here and you’re not understanding it, elements are fantastic. And they can just go ahead and read a bunch of paper and explain to you in the language that you need, because it’s still a summarization style task. Even if they go and search, like you go back and then you want the output to be a certain formatting. It’s a summarization task plus like a formatting task, and it will tell you exactly what you need. And it’s much faster now, before you had to watch a bunch of videos and then like, that’s it, like ask a bunch of questions and wait, right?

Put that in the backlog of like, I don’t understand this bit, right? And I’m not going to understand for like next six months. But you can shorten this out and like, that’s it, like not have to go into too much rabbit holes without like I really needed to. A recent graduate university student, AVMS, that says, you’re currently looking for employment considering grad school, how to integrate AI or focus on AI research. I have a data science degree, but it seemed like I a barely scratched the surface. I’m worried if I push the academia, I will be left behind in AI acceleration. Should I be learning a LAM or a CHF more in depth while staying on the top of new release? So this is what I was talking about, like, especially people that are graduating right now, they are so lost. Like people that were contemplating academia, even back in the day, they were kind of a bit lost. I was like that, right? But now it’s kind of worse because like, why do I even, why even, right? Like AI is moving so fast. Why even study AI during a PhD? So like, I think the question is more in the sense of existential question in the term of like, what should a new grad like do here? Like, do you think they should go into grad school? Do you think that it should be like, you’re looking for employment, like working on stuff? What’s your general thought process here?

(Vicki) Oh, I don’t have any big life advice, but my small life advice would be to do projects, do small projects, like stand up Hermes agent or open, no, don’t stand up open clients can expect everything. But stand up. Yeah, stand up an agent. find out what it takes to sandbox it correctly. And have it, I don’t know, give you a calendar readout every day. That’s a small project, create an LLM summarization task that summarizes papers every day. Find stuff in your life that you could design a small project around. If you find a pain point, you could work with the LLM to create a project plan for you to implement it. I think the best way to like, understand where you want to go, is to do small projects. And that’s going to directionally, at least tell you if you like doing this kind of stuff, or if you would prefer to more, you know, study it in a theoretical way, or if you would prefer to do x and y.

I don’t think like I’ve ever seen that someone that did a small side project, I was like, No, I shouldn’t have done this, this is terrible. Even if they didn’t finish it, because good projects will always, even if they fail, they’ll teach you something. And they’ll it gives you like more of a sense of ownership of this thing that is just like everybody saying this API use this blah, blah, blah. Oh, I’m using this and it’s good. I’m using this and it’s bad.

You will have opinions on that now. I know how to build evaluations, or I don’t know how to build evaluations. I like this API. I don’t like this API. I like using local models. I don’t like using local models. It turns out I like using Hermes agent, I don’t like using Hermes agent. So all of that will get you directionally more to the point where you have opinions and you can decide more what you want.

(Yacine) Yeah, this I like this, this idea of like, looking for small project and like, taking them on. But what’s your opinion on big project? Because like, if you think about it, when you’re going to a company and you’re working on some piece of the of the software stack, whatever it is, that’s a big project. If you’re doing like a graduate thing, you have to like find the lab, the thesis and stuff. And then you go on for a big project. You can chuck it like you can have like different like a small project along the way to be able to understand the stuff.

But like, how would you advise somebody to like find that big project that they can take on for for a long while?

(Vicki) I think that only comes from doing smaller things that cascades in the stuff. So for example, a bit back, I wrote a paper on embeddings. Where did that come from? It’s because I was working with them at work. And I was trying to explain them to business people. And I couldn’t really explain them really well. So I’m like, Okay, I’m just gonna write a document for the business people that tells them like what these are. I started to write it and then it started being like, Oh, okay, well, I don’t understand this about it. So I’m going to write more. And then I’ll turn into like a whole 80 page thing, where I’m trying to explain like what this underlying fundamental thing.

A recent project that I’m working on, just like as a side project is seeing if I can have a small model generate commit messages. That’s it. Why? Because I’m really interested in small models. I’m interested in like what the smallest model you can do to generate commit messages. LLMs do them, they will do them for you. But like, I just want to do this that brings up all sorts of interesting questions. How does local small inference work? Like, how do we have small file artifacts? So if you pick the right project, that’ll lead into a lot of other different questions that will become a whole big thing, which is why I haven’t finished it yet. Because I’m like digging into this stuff.

(Yacine) How do you get to that right question? Or like, how do you pick that?

(Vicki) It has to be something that’s interesting to you. Otherwise, you’re going to be bored. Yeah, I think I think you you you you got the the essence of it is like, I think now more than ever, following your interests deeply has so much value, like, because like all of the little uninteresting bits and stuff like that, or like the clerical stuff or things that you don’t understand the confusion, they can get smoothed over with this beautiful search summarization thing that we have going right.

But like, even if you go and you go and do like a PhD in academia, and like in a in a subject that she literally cannot care less about just not be left behind. I mean, at the end of the day, you’re going to get so much less than if you’re doing like the thing that you want. And then like along the way, pick up like the pieces that you need to do even more of this stuff.

Yeah, I don’t know, like, I’ve talked to a whole bunch of people and like the difference between the people that are very interested in their thing, and the people that are just want not be left behind, is night and day and like the some of the projects that the interested like group are doing are just boring to me, like they’re they’re so not interesting for my like, perspective, but just seeing them going at it with such passion and then explaining it and like, actually, no, this is this has something right. And you understand why the person is is so interested. But in the person of doing this, they’re doing so much pretty built a benchmark over there, they had to fine tune this over this thing over here, they develop a new methodology, they tested this that nobody tested before. And then you get that kind of whole interesting flywheel going. It’s hard to learn, like, from, from my perspective, it’s very hard to learn when you’re deeply uninterested about a topic. I don’t know if it was like your experience, but like, I always found that like, when you had this kind of curiosity about something to catch it, while you have it, and then to go deep on it, you learn so much more because the drive to go deeper is kind of blitzing you through the topic, the same topic a year later, or like a year before, you would have slugged to it, and it would take in like three months, right. But this, when you catch it, like this, this moment in time where like, it just drives you. I don’t know, I feel like it’s, it’s, it’s wasted time to not take it.

If you pick something that is interesting to you, you will always, it will never feel like work, which is super cliché, but that’s literally always been true for me. Like building soft that is interesting. It doesn’t feel like work. It feels like you’re playing with your mind. So that’s it. Try to find that thing. And in my view, like this is what leads to mastery, right? Like when you’re playing with the stuff, and then you’re just getting better and better.

Somebody might ask you like, why are you making this thing faster? I just want to make this thing faster, right? Like any, it just need to, like, if I can make it faster, that would be fantastic. But then you master the topic to such an extent that like this is reusable in a lot of the part in your life. Again, it’s with year two, like how you use this stuff at work. You’re saying, I’ve seen some from some of the blog, you’re using the agent at work to build like an old system. How do you go about it to like, leverage them like without offloading, like, I would say like the ownership of the other thing that you’re building?

I think the first thing I do is if I’m writing a new feature, I try to write some of it by hand, because like I said, I need to touch the code, I need to touch the data. I’ll at least start, well, first what I’ll do is if it’s a complicated feature, with interaction with a lot of services, I’ll like, I’ll literally physically sketch it out on a whiteboard or a piece of paper. I don’t even use the computer at that point, because I just need to think.

And that’s the only way I can think. So I do that first. Then I will try to start writing some code by hand, or at least like the entry points where I know I need to change this, I need to change this, I need to change that. Then at that point, I’ll be like, okay, I’ve added these things to the agent, help me create a plan for what needs to happen next.

And it’s always like, I always try to ask it to plan, like never just blindly do stop. I’m like, show me the code that you’re going to implement. Do not push this code, just show it to me. And then I’ll go through it and be like, okay, yes, this seems reasonable.

And at this point, it’s like easy to be like, yes, implement. Yes, yes, yes, yes. So I try, I try to basically, it’s very hard, but to deliberately create friction for myself. And usually I will do it like piece by piece, and then check those pieces, and then commit those pieces myself, which is another reason why it is important for me.

Because I like to manually, like, sometimes the agent will do it. And I’m like, do not do that. I like to do that myself. And I like to push the code up myself. And then I like to look at the changes. And then I like to physically test. Well, it’s not just like you need to test it to make sure it doesn’t break anything.

But like to test it to start up the app, etc. I know for a lot of people, like they’ve moved to like, “I don’t read the code.” I don’t do this. I think in some cases, maybe it does even make sense. For me, I can’t because we’re building complicated interactions for recommendations for search for real time stuff. So I really need to be checking how this stuff happens. And the best way is just like to slow down a little bit and use the like, again, is are you driving or is the agent driving?

And you should be driving and directing the agent to like, do x and y.

(Yacine) Yeah, yeah. I never understood like, I mean, yes, I could understand when like, you don’t care about like, you just care about the output input. And like, you have some clearance.

But like, whenever you dive into that sort of code base, it’s like looking at the alien mind, like it’s, it’s, it’s not really, I don’t know how to explain that when you when you look at a code base made by like an organic human, right, you can start to understand. And you can tell how they were thinking about stuff, you can tell all the their, the party are still kind of in progress, like how they’re envisioning this. And when you have a discussion, you’re in the shared latent space, right, where you’re talking. But I feel like the code bases are just like, surely, like, let’s true, even though they work, sometimes you’re like, what’s the thought process here, right? And the loop that we’re doing, it’s a bit harder.

And also, like, every time that I hear somebody saying this, like they never follow up by what they really like, like some stuff. Yes, like, I don’t care what the thing is. Because, for example, like I’m redoing my website as an experiment where it’s just my office, the website is the office, and it’s like my toddler is entering the office, and then just like barging in and like looking at I don’t care how it’s generating like the 3d mesh of my my printer, it doesn’t matter to me, right? But like, that’s the type of stuff I’m doing. I’m not it’s not something mission critical, like that is like pushed to to my clients or whatnot.

(Vicki) The one thing like that I think about a lot with that is, Karpathy had this talk a while ago, or maybe he wrote it, he said that like, agents generate code from the top down, and humans generate code from the inside out, we start with like, one method, or we’ll start with one class, or we’ll start with one module and be like, okay, this is how this should be. This is how connects to here. And they write it just from the top down. So I think that’s a lot of the difference is like, in the way we cognitively build things that just makes it be like, yeah, this is a wall. And the other thing for me personally, also, I totally agree with that is like, the variable names are hard to read. Yes, what I like to do is I like to like, describe something and then have it generate a variable name for the thing that I’m describing, it’s really good at that. And then like the methods for some reason, they like make sense, but they’re hard to read. They’re like, oh, yes, addressing cyclomatic complexity. I’m like, okay, it just has too many loops. Like just describe it normal.

Talk like a human.

(Yacine) I’ve been using them recently to it’s like not even well, it’s kind of related, because I have to get a client. So I’m doing like, there’s a bunch of RFP I need to answer to.

And then I use it to kind of help me like write the code, because like, I look at like the requirements stuff like that. I do a bunch of search and then summarize stuff. But I don’t know, like the way it’s written the stuff, it’s clearly written it in a way that it’s talking to itself most of the time. I don’t know if you have this feeling, because like, sometimes we will say something and like, why are we talking like this? Why are you saying this? But it’s like, it is to prime it to not talk about something I’ve said before, right? Or like when you like, you’re six version down, and it’s still talking about version 0.2, and you’re gonna send it to dude, like, he’s never gonna see this. He doesn’t know about that. He’s just here, man. But like, the truth is, it is receiving some of the stuff from prior in like, the context. So it has, it has to kind of account for this. And like, this is the way it is doing it. So but like, not understanding how these things work, you get so frustrated, even myself, I understand, like, sometimes just like, I just yell at a bit. How on earth to get interest engineering practice without owning, renting expensive GPU clusters? But what do you think about that?

(Vicki) I think, well, I guess is the goal, like you want to get a job at a top AI lab doing inference engineering, or you want to build intuition for how to serve machine learning model artifacts? Because the second one is much easier. It’s possible to get intuition by serving local models and looking at how like, well, LM Studio or llama.cpp do inference. And then you’ll understand like what inference means, what all the different parts are, you can build your own inference engine for GGUF model, for example, and start to understand some limitations. You can also use on an x and just serve that and like, see what the limitations of that are for inference engineering, where it’s like, you need Infiniband, and you need distributed GPU stuff.

There’s not a good way other than just working with these, it’s like, it’s kind of like, how do you get experiences distributed systems, you’re working distributed system, how do you get that experience, you’re on call. And so for a lot of that stuff, it’s harder. But there are ways you can asymptotically approach that, by just understanding that inference is the ability to serve a model artifact to a lot of people. And you can simulate at least some of that with local models or just smaller models.

(Yacine) The other way also is just to figure a way to get access to these things, because like, you don’t have to necessarily to go and like, because I say that because at my university, you have access to like a super cluster of like, GPUs, and you could use it, right? Maybe like, if you’re really into this, and then you want to get experience on that, there’s ways of looking at the different, like organization, or like, even like academic, I said, like path where you have access to these inference. And then you can just play with it, right? What are the extremely hard challenge in the ML infra that you can think of?

(Vicki) I think just managing state, I think being good at Kubernetes is a big challenge. That is always important. And like, if you know Kubernetes well, that’ll serve you well in almost anything you work with. I think latency is a big concern, especially for things where you have to do next token prediction quickly, or just inference quickly. Those are the top ones that come to mind.

Really, if you want to get super normcore about it, what are the big challenges? Security, dependency management, making sure that the version of what you have deployed is correct. And I would say like an even bigger one is when you redeploy or change the model, are the results still good. And that’s always been the case for machine learning and production for almost all time. And you can even see it, like when OpenAI are in traffic-releasing models, people are like, this one’s much worse than the last one. And for sure, they’ve had a million test suites, right, that measure what X and Y is. But the interesting thing is, like now, machine learning models have a wide range of customers with a wide range of use cases, ranging from writing code, writing different kinds of code in different languages, to people just using them for like calendar apps, to people integrating them in applications.

And so now, like the hardest thing is if you’re serving in LLM, does this new model still meet your use cases, which I think is a really interesting challenge. In one of your latest blogs, you said that like LLM that are local are good now.

(Yacine) And so the open weights, what things have you learned that are very useful for your work or just your general knowledge, that you learned from actually running them yourself?

(Vicki) They make really good classifiers for small things, especially for small jobs that you want to use. They’re very good for privacy. I use them a lot of times when, for example, there’s just some stuff you don’t want to paste into API calls. Or I actually use them for personal stuff a lot. Like I know people will sometimes use LLMs to ask questions about like what’s going on in their life or whatever. If I have stuff like that, I’ll just ask a local model because they’re good enough directionally now to do that kind of stuff.

Or if I have private data that I just don’t want to go into these APIs. They’re great for that. They’re very, like I said in the post, they’re very good now. I’m using the latest Gemma. And it’s like, I don’t know, maybe 20, 30% worse, but it’s good enough for, oh, the thing I’ve learned is like not all tasks require like Mythos 5 billion, right? Sometimes I use local models for like fancy autocomplete, right? Sometimes you have a method and you just want to fill in the method. That’s actually another way at work that I use to give myself friction is like I won’t have it implement the whole thing. I’ll just paste in a specific method and like be like, fill in this method.

And that’s a way to get more granular control over your application. So I think a lot of people have shifted it over to agents. I still use agents and chatbots interchangeably and chatbots specifically when I want to use more on complete tasks.

And then you can introspect them a lot more. I have like a million projects that I want to do with local LLMs. One thing I wanted to understand was like how KV cache works on a Mac, what happens there? And so they’re really good for like, you can poke every single thing and then you can understand how it works. And that’s amazing too.

(Yacine) Yeah. And to be, I don’t know if you’ve seen this shift or not, but like, there’s a bunch of my enterprise clients that just don’t want to have a closed model in their, like their product flow. Their team when they’re using it to code and stuff like for them, it’s like, it’s not that bad. But as soon as it goes into their product, it’s like it happened three times. And before that, it didn’t happen. But now it’s starting to happen where they’re like, can we, any model that we can use that is not these guys, please, right? I don’t know if it’s the price that started to creep up and they’re like, yeah, on top of that, we hate them. I don’t know if it’s this or it was just like, no, we have one privacy, we want to have control, we want to be able to one of them asked me like, like, I want to have an open model because I can actually fine tune and have more room to play around with it. Have you started to see like this shift or or is it like still localized?

**(Vicki)**I think from people that I’ve talked to, I think concern, the cost is now a huge concern for almost everybody. And it went to like, oh my god, this is magic. Let’s just use the magic all the time to, oh, we’re now monitoring tokens on a per day basis for this project. We attach a billing code to this thing. So I have definitely seen that shift in my circles. And also interestingly enough, like away from the top models, like how can we use this and maybe less so but like, yeah, maybe routing for different tasks. That’s more complicated because it’s like, how do you know which task is what? But yes, routing definitely for sure. It definitely does feel like we’re starting to see a focus on perhaps smaller models, but at least different alternatives to the labs. Yes.

(Yacine) So you’re actually largely set up like that you started in econ and like you went and learned all this cool stuff. Like from that self thought perspective, like what is I would say like the anatomy of a good project that like actually because you assume you did a whole bunch of them, what is the idea of a good project? What are they like the characteristic that make it very useful for you?

**(Vicki)**Yeah, so there’s a couple things. One is, as I said before, it has to be something that is a personal problem for you that you want to solve. Otherwise, you’re not going to want to work on it at all. And the second part of this, which I think is critical, but unfortunately, not a lot of people get access to it is you have to have someone critique this project.

A lot of my personal learning and growth as a developer came from having people consistently review my code, people better than me consistently review my code. And I read a Farnham Street blog about like how basically pro athletes continue to be good and pro musicians. And it’s they have a coach or a mentor, someone who’s consistently telling them, okay, do this movement, do it differently, do it this way.

And I think in software, that’s become harder to find. I think it’s important that if you’re senior, you contribute to that as much as possible so we can get good seniors. But I was the beneficiary of a lot of that of just people consistently reviewing my code. And there’s a great post about this by Ethan Rosenthal that says [data scientists work alone, and that’s bad from five years ago. And I think that was the case, like everybody was kind of siloed when they didn’t get exposed to best practices, they didn’t get a chance to learn it. So I think that piece is really important.

Perhaps the best way to find that for a person is maybe through Discords or communities, etc. If your work, if you’re at work, seek out those kinds of people. If you’re in university, also seek out people to review. So you basically need external review from unbiased sources.

(Yacine) That’s beautiful. It’s a beautiful way to end it. Thank you very much for taking the time of your day to do this. I hope you enjoyed this discussion as much as I did.

If I had to boil down the essence of what she said is that the fundamentals still matter, maybe more than ever. Not because like you’ll do everything by hand, but because that’s what you let you steer the agent instead of the agent just tearing you into a direction that you don’t even want to. So if you want a good next step here, I recommend you to check our anti-hype LLM reading list on GitHub, which still stand at test at times and start your journey to study topic in order. Have a great week everyone and see you in the next one.