Robotics & Semis

Sergey Levine is a pioneer at the intersection of robotics, reinforcement learning, and machine learning. He is recognized for shifting robotics away from hard-coded systems toward neural networks that map visual data directly to motor controls. He is a co-founder of robotics company Physical Intelligence (pi), and he detailed on the Ryan Peterman podcast the current state of the robotics field:

“Initially when people started working on models for language, the dominant design was LSTMs. Some people remember what those are. They were kind of okay; they were a lot better than what came before that, but they didn’t really scale as well. And then the big thing with transformers was not that transformers were somehow particularly mathematically elegant or anything like that. It’s just that they scaled better. So they were easier to train on very large amounts of data with lots of parameters. The technology proceeds in phases. First you figure out what you can scale, and then you pour on a lot industrial-scale effort and adding lots of data, adding to model size. And that’s when the magic happens.What I would say about robotics is that it’s not in the GPT-4 to GPT-5 stage, where it’s an industrial-scale effort to make the model bigger and get more capability out of it. It’s in that stage where we’re establishing the fundamental technologies. And because of that, what one should expect to see right now is not necessarily that each month the model gets bigger and more powerful by some predictable kind of scaling curve. It’s that the scaling properties themselves are evolving as we develop the right technologies. So, where we’re at now is that we’re actually getting all those puzzle pieces in place. And I think it’s actually very close. I think a lot of the puzzle pieces are falling in place, but it’s not yet at that predictable scaling stage. Certainly we’ve seen a lot more happening as we progress, but they were kind of magical because in robotics, basically prior to 2024, the stuff never happened. So the little things that happen, and this was maybe at this point about two years back, we would see things like, okay, we train our policy for folding laundry and it takes individual shirts out of the hamper and tries to fold them. And then one very vivid memory I have in late 2024 is we were watching one of the evals, and it takes out two shirts at the same time. And I’m watching this, I’m like, okay, it’s done for. There’s no way you can possibly do this. And then it puts the two shirts on the table, disentangles them, puts one of them back, and then starts folding the other one. It’s like, wow, okay. In retrospect, you can do some detective work and figure out where it got that from, some piece of training data. It’s exhibiting the common sense you expect people to have.What would have the biggest impact on the advancement of robotics. Certainly, availability of reliable, low-cost hardware is a big deal. Right now, hardware used for robotics research, a lot of that does come from China. And it’s good, it’s relatively inexpensive, it’s of high quality, and meets the standards that people generally need. But it would be awfully nice to be able to source all that domestically as well. Having an effective positive feedback loop where more deployed robots translates to more model capability, that’s kind of the key. The hard thing in robotics was always generalization. What is very easy to miss in the hype cycle and the demos that people put out, is that when somebody shows a demonstration of their system, the demonstration alone usually doesn’t make it clear what level of generalization is being shown.The highly acrobatic robot demos, for example, are really exciting to look at. But typically, it’s a show that is obviously rehearsed. It’s not the same as doing a task every time reliably in any home. So you might see the robot doing something fairly mundane and unimpressive, but what’s exciting about it is that it’s doing it with an object that it’s never seen before in an environment that has never been tested before. And that is actually harder than doing an acrobatic backflip that it’s practiced millions of times. One of the steps on the roadmap is to have a robotic system that gets better with autonomous experience in a setting that it wasn’t originally trained for. I get my model, I get my adaptation algorithm, and I put it in a new setting. Maybe it’s a home, maybe it’s a factory, something where it’s doing something real, and it does okay. But then over time, it gets better and better. That would be a major milestone. Live demos with really long time lapses, I think it’s a great idea. We had our robot assembling boxes at Dandelion Chocolate Factory. So we ran it for several days. But we had this coffee task, which was the robot using an espresso machine to make espresso. We ran it for 13 hours making espresso drinks. It screwed up a few times. It’ll spill all the coffee grounds, and then it needs to go and get a cloth and wipe it down. The way that experiment was done is that there is a high-level prompting. So the prompt is updated roughly every five minutes or so in between semantically coherent tasks. So you tell it: make espresso, clean up the machine, etc. So those steps, the actual clean up the machine, was commanded by a person. In principle, we could automate that. In fact, one of the things we’re spending a lot of effort now on is improving our high-level policy that does those commands. But for that experiment, every five minutes, somebody basically updates what it’s being asked to do, the way we intended. It was like the commands would be like, if you go to an actual coffee shop, you say, “Oh, I want a latte, I want an espresso.” That was supposed to be the prompt. Except then you also have to tell it, “I want you to clean it up before you do the next one.”One thing that I’ve learned over the last few years is that addressing robotics effectively requires using very broad prior knowledge. For example, I was very skeptical initially about the utility of language. And I think scientifically this is defensible, which is that animals can do some pretty impressive things, but they can’t speak, at least not very eloquently. So maybe our robots should also be able to do stuff, and they don’t necessarily need to understand language. But what’s important is prior knowledge that you can put it in as a scaffold on your learning process. And you can pull in that knowledge in all sorts of ways.”

If we look at the progress in AI—or machine learning, as it used to be called a decade ago—we’re probably in a similar phase now in robotics as we were in digital neural nets one decade ago, around 2016 or so. This was a time when AI had a good understanding of what is in an image, what’s the meaning of a text, and reinforcement learning had just been introduced to give AI superhuman performance. Currently, while robots start being able to do basic tasks, we’re not in a phase yet where they have as good an understanding of the surrounding environment as humans do.

A few months ago, on the Invest with the Best podcast, Levine gave more background on his company and the robotics field:

“Part of the thesis of this company is that we believe doing it at the full level of generality might actually in the long run be easier than trying to special-case very specific narrow application domains. Again, in much the same way that for language models, it turned out to be easier in some ways to solve natural language tasks in their full generality than to narrowly target machine translation or sentiment analysis. In the world of natural language, we saw that there were a lot of efforts to develop domain-specific solutions that tackled specific problems. For example, somebody would spend a lot of time thinking about how English differs from French and then build a machine translation system. The reason that language models actually took over for all of those different application domains is because they can leverage much broader sources of data. When you can leverage weakly labeled data, like data that you mine from the web in the case of language models, you actually learn more about the world. You establish a foundation of world understanding, and on top of that foundation, it turns out to be much more effective to build out different applications.To bring this into robotics, the calculus does not look quite the same because in robotics we do not have an internet-sized dataset that we can just draw on. But this notion of understanding the world is actually more important in robotics. If you have many different tasks and many different physical systems, then you can go from training individual dishwashing specialists or laundry-folding specialists to instead training a model that actually understands physical interaction. People can master new skills very rapidly because we understand physical interaction. We can intuitively grasp what is going to happen in a new unfamiliar situation, which lets us bootstrap things really quickly. If we can draw on data from many sources, many applications, and many robots, then we can have a model that has physical understanding, making it much easier to put new applications on top of that platform.The point of generalization is that it does something relatively mundane that any human could do, but it does it in any situation. We released some demos where we showed our robot cleaning kitchens. If you watch an individual video out of context, it just looks like it is picking up plates, which anybody can do. But the point was that we put the robot into that home just for that demo, and it had never seen training data from that setting. You have to understand what is going on behind the scenes to appreciate why that is pushing the frontier. In the future, we will have a robotic foundation model that can be adapted to all sorts of applications, running the gamut from bulldozers to humanoids to robotic arms. The fundamentals of how you interact with objects, how things move in the world, and how causality works are conserved across all of these different systems. Machine learning works best when there is a lot of data. If you naively approach a problem like washing dishes by collecting an enormous amount of dishwashing data, it is not cost-effective because you have to repeat that entire process for the next task. Training general-purpose models reduces the data needed per task. What has changed most recently is the ability to handle unusual scenarios. In edge cases where you lack prior experience, you must rely on knowledge acquired from other sources and ground it in the new situation. Multimodal language models are adept at pulling in broad web knowledge. While they are not inherently grounded in physical situations, they provide a path to import common sense into robotic control. The first deep reinforcement learning systems in the early 2010s were a major milestone because deep RL gives us a way to surpass human-level performance. More recently, the advent of multimodal LLMs adapted for robotic control to supply common sense is a critical advance, and we will likely see several more major breakthroughs over the next few years. Over the past few years, we started by developing Vision-Language-Action (VLA) models. A VLA model can be understood as an LLM adapted for robotic control. It is pre-trained on text, adapted with extensive web image data to understand visual scenes, and then fine-tuned on diverse robotic data. That serves as the foundation for bringing web-scale knowledge into robot control. On top of that, we explore two primary capabilities: handling unusual situations with common sense, and improving through reinforcement learning. To achieve common sense, we utilize chain-of-thought reasoning. When the robot encounters a scene, instead of immediately executing motor commands, it reasons through the task. If asked to clean a kitchen, it analyzes the visual input, generates an intermediate semantic thought like picking up a specific plate, and then acts. These intermediate inferences benefit from web-scale pre-training. Reinforcement learning is applied when practicing tasks repeatedly to improve robustness, speed, and precision. In our espresso-making demonstration, the system practiced repeatedly to optimize throughput and reliability. We are continuing to build upon this dual foundation. Nobody knows exactly how much robot data is needed to achieve universal generalization, but we may not need to know in advance. The objective is to deploy systems capable of performing diverse tasks and establishing a data flywheel. Humans intuitively assume that tasks difficult for us, like advanced calculus, are hard for machines, while tasks easy for us, like grasping a cup, should be simple. In reality, human brains have evolved massive specialized machinery for physical interaction and visual perception. Machine learning shifts this dynamic. Where data collection is straightforward, tasks become tractable even if they are physically intricate. While learning is now widely accepted, there is still debate over end-to-end learning versus modular systems incorporating hand-coded physics. The Bitter Lesson posits that leveraging general methods with data and compute outperforms hand-engineered domain knowledge over time. While incorporating explicit physical models offers short-term structure, end-to-end learning provides the scalability and self-improvement necessary for true open-world generality.Former Everyday Robots researcher Benji Holson proposed a practical Robot Olympics consisting of everyday manipulation tasks that humans find trivial but robots struggle with—such as opening doors, scrubbing greasy pans, or using a bag to pick up waste. We used this list to test our general onboarding pipeline. Without developing specialized algorithms for individual tasks, our foundation model successfully solved nearly all of them. The only exceptions were turning a dress shirt inside out, which was physically constrained by gripper size, and peeling an orange purely with fingers, which required a small tool due to grip strength limits. This demonstrated the power of a general model to onboard diverse tasks rapidly. Tasks expected to be among the last to be automated by robots involve direct, nuanced physical interaction with humans such as changing a child’s diaper, eldercare, or helping an injured person out of bed. These tasks require precise force modulation, high safety margins, and deep social-physical common sense.”

While it’s an interesting field to follow, it will probably be another 5-plus years before robots really start to scale massively. So, investing in this space is akin to investing in AI in 2015-2016 or so, with the big J-curve in terms of demand only coming 7 years later. We think this is a good framework of how to think about the space as it likely will still take a while before the really big returns come for the winners. Next, we will review a number of the best positioned names in humanoid robotics and their supply chains, and we’ll also overview our semis outlook.