Latent Space: The AI Engineer Podcast

This was recorded on the sidelines of the new Compute Trading track at AIE NYC. Catch us next at AIE CODE, our last flagship for the top agentic engineers in the world.
In the early days of cloud, former guest Martin Casado made something so disruptive that Cisco was threatened enough to pay $600m for it, and VMware outbid for $1.3B:
That concept was software-defined networking, which virtualized the routing network by separating the control plane from the data plane. That revolution is now coming for GPUs - where previously running your own supercompute cluster required $50-100m upfront to get commitments and do the entire stack from hardware to firmware to kernels, SF Compute is now enabling you to not only buy or sell GPU hours now and in the future, but also to programmatically control everything from your network to memory.
We last spoke with Evan about commoditizing compute. At AI Engineer New York, we went deeper: what does it take to physically deliver a GPU contract, and what has to be true before a market in compute can become liquid?
From selling a room to running the hotel
Evan describes SF Compute’s expansion with a hotel analogy. A customer wants a room, then asks for cleaning, room service, and security. Eventually you discover that delivering the room means operating the hotel. SF Compute began by trying to sublease a cluster; the company has since built much of the operating capacity of a cloud to deliver the compute traded on its market.
That work is physical settlement: the buyer gets usable compute, rather than simply a cash payment against an index. It brings the physical constraints of a cluster into the market itself. Can a workload be interrupted? How quickly can it recover from storage? Which GPU generation, networking and software configuration does it need? Those details affect both the contract and the price.
Making compute tradeable
Rather than waiting for every GPU cluster to become interchangeable, SF Compute matches orders with different constraints. Evan argues that this makes trading possible while the technology changes, at the cost of spreading liquidity across more variants. We also discuss spot and forward pricing, batch versus live inference, and why an index can become disconnected from the cost of actually buying compute.
Independence is central to Evan’s case for a market operator. He argues that companies with large hardware or inference businesses have incentives that make it harder to run a neutral exchange. We also get into the unglamorous engineering: visiting sites, designing clusters, running the backplane, and making infrastructure programmable. Near the end, he describes the ability to subdivide InfiniBand fabric on the fly as one example.
Guest: Evan Conrad, CEO of SF Compute.
Show notes
* Evan’s AI Engineer NYC talk: How Compute Markets Work
* Our previous SF Compute conversation
* Why oil futures went negative in 2020 — EIA
* How SF Compute customers resell unused GPU capacity
* SF Compute’s $245M compute sale
* Abel: SF Compute’s settlement ledger
* CFTC: Barclays and LIBOR manipulation
* Elastic InfiniBand partitions
Timestamps
* 0:00 — Why can’t NVIDIA build this?
* 0:25 — Welcome back to Latent Space
* 2:23 — From selling a room to running the hotel
* 3:47 — Physical settlement: actually delivering compute
* 6:00 — Liquidity and the GPU order book
* 8:42 — Trading nonstandard compute
* 10:00 — Interrupting and moving workloads
* 12:20 — Why hardware cohorts matter
* 14:32 — How far out do compute contracts go?
* 16:19 — Batch versus live inference leases
* 17:58 — The shape of compute prices
* 20:33 — GPU generations, specifications and SKUs
* 23:30 — Physical versus cash settlement
* 27:00 — The emerging compute market ecosystem
* 29:03 — What it takes to deliver a GPU cluster
* 32:24 — Sovereign compute
* 33:26 — Why NVIDIA can’t build this market
* 34:48 — A programmable data center
Transcript
Automated transcript, lightly formatted; speaker labels have not been verified.
00:00 NVIDIA is obviously like the 800-pound gorilla in the room. Why didn’t they make SF Compute? So our impression is that NVIDIA can’t make it, but benefits tremendously from it existing. The reason why they can’t make it is because they would then not be independent enough for people to trade on it.
00:25 Anyway, Evan, welcome back to Latent Space. Thank you so Welcome back to AIE. Actually, first time at AIE. I think so, yeah. This is a very special one because I feel like you have been working on compute trading, compute markets for a while. Now it’s like feeling of the moment. Yeah. Jump basically announced that they are getting into compute markets, which is why I’m excited to sort of cluster people. You met Oliver from SemiAnalysis, who is basically initiating coverage on compute markets is my read on this. So people are going to watch your talk.
01:00 I think it’s a very good explanation of both your origin story and why you need to exist and all the hard problems you work on. Yeah. Done in a very simple, engaging manner. I’m very jealous of your writing skills because you showed the flyer that you showed at New Reps. Yeah. I was like, it’s pretty straightforward and simple and it’s good. Yeah. So anyway, congrats. Thank you. I appreciate that. And also congrats on AIE. It seems to, y’all are rolling. We’re rolling. I just came back from the site tour for next year’s AIE. then for You have like a hint? At Javits Center. It’s the same place they do Comic Con. Okay. Wow.
01:35 Wow. Wow. Damn. But I think the finance focus is really hitting for folks. Yeah. So we’re going to try to split out. Like this is, here we just lump everything in finance. Like your finance, their finance, like insurance is finance, but like they’re all super different. Yeah. So we’ll probably break out things more minutely in future because people want to go deep on hedge fund stuff or banking or personal wealth. Yeah. Especially in New York. Yeah. Anthropic was like, yeah, we’re all about RIAs right now. Yeah. RIAs like registered investment advisors. Yes. Like so different from you.
02:10 No, no, no. I’m aware of this. Yeah. Yeah. Okay. So for listeners who don’t know, we did a podcast two years ago where you talked about your, you know, where SF Compute was. Um, today you’re, you’re much more in a sort of physical settlement. You’re talking, you’re talking about your, your back fabric or what do call it? Uh, the backplane. Backplane. Yeah. Which obviously that takes a lot of hardcore engineering stuff. Yeah. Um, can you give a status update on like, where is SF Compute versus listeners who last heard you on a podcast two years ago? So, uh, when we started, we were this company called Junark and then we were just trying to sublease our cluster.
02:45 Um, and we started there in the technical stack. Um, and a way to think about it is it’s like you’re in a hotel and you’re trying to sell that hotel room. And then somebody asks you, Hey, like, are you going to clean my room? Um, and then you say, Oh, I guess I probably also need the hotel room as well. Or like I need the, um, the cleaning closet. And so you go get access to the cleaning closet and then they want room service. And so you get access to the kitchen and then, you know, they want, um, security. And so you get access to the security closet and you get this and you do that and you do this. And at a certain point you’re just running the whole hotel. Um, and the moment you’re running the whole hotel, you say, Oh, wait a second.
03:21 Um, instead my supplier on this market, um, is somebody that looks closer to a, fund, um, or a cloud that has access in our case, a cloud that has access to the data center. And we’re going to level up. And so what we’ve spent the last several years doing is building the entire operating capacity of a cloud, um, in order to offer, in order to actually deliver, uh, the compute at a reliable level. Um, we call that physical settlement. We’re taking a term from the futures market, from futures markets, from other markets, um, where in, you know,
03:57 oil or soybeans or something, physical settlement means you get delivered the oil and the soybeans. Yes. Um, but, um, in some markets, um, like in oil and also in power, um, there’s like a delivery network in oil. You have to have pipelines. If you would like the oil to be delivered to you, there has to be a way for it to get to you. Yes. Um, and finance, and for people interested in financial history during COVID oil went negative because, uh, so like I will pay you to take oil from me because it is sitting there in the port. Uh, and like, I can’t pay for this.
04:32 Yes. And so I will pay you to take my oil. Yes. That is like the weirdness of physical settlement. Yes. There’s a bunch of weirdnesses like this in other markets as well. So in the power markets and the grid, um, there’s parts of the orders that you fill out, the buy orders and sellers that, um, are there because the boiler takes a while to cool off. Um, and you can, you know, stop the boiler and then it will, you can start it back up again within some period of time. And so that has to get factored into the order book. Um, this genre thing is the same thing that happens in compute markets. So we co-designed physical settlement and the order book at the
05:10 same time. And that meant that we get to actually handle the real constraints of compute, which might include something like, oh, I’m okay with being interrupted, but only if there’s like storage on the thing that I can recover from really quickly. Um, very similar types of problems. Yeah. So, uh, people can refer to your talk and you’ve done a couple other podcasts already. Um, so I want to be like the sort of 201 level instead of the 101 level. Sure. Uh, and also because I, I am, I have been a derivative trader, so I can give people a bit more detail that you normally don’t get into. Um, so for example, well, how, how mature or deep or liquid is the compute
05:45 market today? Uh, not very mature or deep or liquid. Okay. so what we’ve been doing is first we had to, we had to go get a bunch of supply online in the first place, which just takes a while. And then we had to build the entire platform in order to do this. Um, and we just now got that working. And so now we move, um, capacity over onto the order book, um, which makes it liquid. Um, and so 5% of our total capacity is on the order book and we’re consistently moving more and more. And then we’re bringing on more capacity. That’s all on the order book. Um, the nice part is that, um, maybe I should have said the, the volume on the actual order book is small, but the,
06:21 um, the liquidity on the order book is high. So for that small amount of capacity that you have, um, you have, um, like very concentrated liquidity. So, um, a sort of standard metric, I would say for someone who is using this is a customer who buys capacity from us and they spend a hundred dollars to buy that capacity. We’ll make back $25 on average from selling it back to somebody else. Um, and that’s the number that we, or this is one of the metrics that we try to track because, we want the liquidity to be useful to the commercial participant and you can measure liquidity in a bunch of different ways. You can say like, I just want tons of makers or something.
06:56 Um, and you can measure it in like high-minded ways. We try to measure it by like, is this solving the problem, um, for the customer? Yeah. Got it. And so, uh, last time we talked as well, there was a lot of contract, uh, conversation about standardizing the contracts. Yes. talking with like, uh, the CFTC or whatever, what’s the status of those things? Um, so I think standardizing the contracts in general is still pretty far away. Um, and the, what we were trying to do at the time was, um, or basically what we learned over the last several years was we just needed to
07:31 operate the whole thing. Otherwise you cannot standardize the thing. And so to get closer to something that can be standardized, you first need to make all the operations of the thing the same. Right. To give an example of what can be different is a single operator can offer two different types of virtual machines. One type of virtual machine, um, might have a 40% performance, um, degradation versus another types of virtual machine. And 40% is a lot. When you say performance degradation, are you talking like MFU type things or like amount of burn in that you’ve experienced or what? It will affect. So in this particular case, I’m talking about GPU kernel launches and how long it takes for that to happen.
08:07 Um, which is a process that happens quite frequently when you’re doing sort of either training or inference. Um, the, um, uh, that will basically, the end result is that you get a worse MFU, but, um, like is the intermediate level. the intermediate level basically. Yeah. Yeah. So, so what I’m trying to, so for anyone who’s traded commodities before, there’s different grades of commodity. Yes. Right. There’s like, it’s not oil. It’s light sweet crude versus like whatever the opposite of light sweet crude is, or it’s like, you know, uh, corn level a or corn B or whatever. Uh, and so like people are trying to, well, you have to establish the equivalents of those things. Correct. Yes. I’m wondering if they have been established.
08:42 So what we did instead, um, was we created an order book that had multiple different constraints and you can say, I want to have, um, you know, something that’s between this and that. Um, it runs from this time to this time. Um, it’s like, uh, X to Y, it’s a bunch of different properties that you can attach to your order. And then we match that against sell orders that have those properties, um, in a big matching engine. Um, and the reason why we did it this way is it allows us to generate liquidity without having to standardize before, know, what the technology can do. Um, and because the technology was shifting so quickly, um, we wanted to basically be able to do that. The trade off of this is that you have less concentrated liquidity.
09:18 And so the prices are going to have, there’s going to be more spreads between the, the buy and the sell. Yeah. Um, it sucks. Yeah. But on the other hand, it means there’s lots of arbitrage to make, um, if you can do that correctly. So if you have a product that can use lots of different types of capacity, um, then you might really like SF compute because you will be able to use lots of different types of capacity and you’ll make more money. Um, you can take the spread. Okay. What dimensions matter in this grid of things? Um, so a bunch of different things. Um, one is dimensions categories that I will say is like, is there a
09:53 workload running on it right now or is it capacity in the future? Sure. is it capacity right now? Um, capacity right now in order for you to sell that, um, you have to interrupt your current workload. So you have to think, do I care if my current workload is interrupted or not? Capacity that’s sold that’s starting a month from now, um, you can buy and sell that all the time. And so you get, um, liquidity on that really easily. Um, there is some concern like, oh no, well I have capacity next month, but it’s very different than if you’re interrupting a workload right now. Um, the way that the interruption happens matters. So an example is if you’re on a cluster and the cluster doesn’t have as amazing storage as you want it to have. Um, so an example is like, do you have a Weco or Vast system under the hood?
10:31 Um, and do you offer that? And like, what’s the size of that? Um, this changes how you respond to, um, being preempted or removed off the cluster. meanwhile, the location of that storage matters. So, um, if you’re able to move that data from one cluster to another, um, quickly, then maybe you might be able to spread across multiple different clusters. Um, and you can be interrupted over here and then pop back up over there. Um, to make that work, you also need, like a, a fast transit path between one cluster and another. Typically this is implemented with a WAN, but most of these clusters don’t
11:06 necessarily have, um, like a wide area network. Yeah. A wide area network. So, basically cables that connect the clusters together. A private internet is another way to think about it. uh, a lot of clusters that are being spun up today are like, they’re not attached to, um, traditional CPU clusters were, next to the customer for latency reasons. In GPUs, they’re next to the power, um, wherever the, like, you know, you put the data center next to the power rather than next to the customer. Um, this means that a lot of stuff is getting put in places that previously they weren’t expecting data centers there.
11:41 And so you may not have, um, as, um, excellent of a transit path, um, from one to the other. So this, this starts to matter. Um, other examples are, um, weird s**t where we theoretically support, but don’t actually, um, uh, like this is a thing that we’re thinking about, but haven’t done basically. Um, the early GB 200s broke. The early H 100s broke. Um, and so if you were buying H 100s or you’re buying GB 200s, you might really want the ones that were deployed after a certain time because you’re not getting the first shitty ones off the rack.
12:16 And so if you’re a super sophisticated customer, you may be willing to pay more, um, because for that. Oh, so it’s, it’s about like what cohort they were, they were produced and they get better over time. So like, uh, the first ones off the rack maybe had some faults or some problem. Right. And so if you were the ones who got the first ones, um, or, you first ones from, um, the, the first shipment of them, um, then yeah, they would, they would be broken or, um, in some way. I assume the sort of process is like, look, there’s defects everywhere. Like it’s, it just happens. Right. Totally. And so like you get it, you get a new shipment offer from TSMC or whatever, and you just kind of run your tests and like you, you figure out
12:53 which ones, which ones are bad and you throw them out. And isn’t that how it works? And I, I didn’t know about this element of cohorting where like, let’s say Q4, 2026, uh, B300s are way worse than 2027 B300s. Yeah. So, um, I’ll give an example. Um, you’re describing burn-in. Yes. so when you get a cluster, um, a good cloud, what it does. Is it tries to kill all the GPUs, um, before they ever get in the hands of the customer. Um, and then, um, the GPUs that are broken or the nodes that are broken, um, you get RMA’d. You basically go and request it. Yeah. You return it and you get a new one and there’s a lead time to that, but
13:30 whatever, you’ll get, you’ll get some back. Um, and then the customer never has to experience this. Um, the, um, thing I’m referring to is that, um, like in early H100s, there were like firmware bugs that may not even be related to the specific, like they may not be NVIDIA. They might be like Super Micro has a firmware bug in the, um, in the box that you bought. Um, and it applies to all of them and it doesn’t matter. Like, um, the solution to that was you need to get them to update their firmware. Um, or they need to ship a new thing, um, in every new box.
14:07 Yeah. had this for like the, the B300s that, that we got. Um, and the IQ had to use Devin to go fix it. Yes. Great. So like different types of problems like this can happen. Interesting. Okay. So I’m like, okay, look, that seems small compared to the overall. Like, I just want to, like, I want people to understand that the shape of this market, um, you mentioned like preemption, you know, and, you we talked about spot instances and how, how that, uh, the analogy is. Yeah. And then there’s also, you know, booking a month out. How far out do this, do these markets go? And is, is real time or sort of preemption more interesting to you versus,
14:42 far being further, further out? Um, to answer your first question, three to five years. Um, wow. We’re starting to see. month by month or quarter by quarter or how’s this? Are you asking how the cashflow works? Like the, the contracts, like the, the futures contracts, right? Like again, like I’m a former contracts, uh, derivatives trader. In the, in the market itself, um, like is in the broader market. Yeah. clusters are financed three to five years. Okay. Um, so the, the, the payments are totally over that period of time. Yeah. there is also like blocks of capacity in our order book, um, which can extend any arbitrary time. Yeah. there’s some stuff that we have to do, um, for, um, the cashflow on that
15:18 right now. What we do is we just ask people to pay entirely upfront for a block. Um, we don’t ask them to do that for a lease for a longer term contract. This gets into something that I definitely need a whiteboard to explain. Um, but, um, you had another question I was about to answer. I forgot. do you want real time or do you want, like, do you want more, uh, what’s more interesting to you? Real time or long, longer out? So we care more about longer outs because we’re trying to build a forwards curve that has a lot of liquidity on it. Um, the, um, these are kind of the same thing and kind of not. Um, so what we do is we, since
15:54 a, you can buy an hour out, you can buy right now to the next hour or you can buy right now to the next three years and anywhere in between. That’s wild. I know. Um, and so the, um, you know, or if I, you’re technically, um, the, there’s a certain difference that happens that’s dependent for each customer. Basically like if I have a workload that is, um, uh, that’s batch inference basically where I can be interrupted at any point. I don’t really care. Um, then my ability to be
16:29 cut off, um, is probably like, I need an hour of time to know, um, that I’m going to be cut off and then I’m good. Or I can even be notified in, you know, a minute or two. Um, but if I have, um, live inference customers who are expecting me to be online, um, then you can interrupt me, but in a month, maybe two months, maybe three months. Um, and then I can adapt to that capacity over that period of time. Um, but you cannot interrupt me in a minute. Um, and so those two things are priced differently. And the reason why they’re priced differently is because what is the price
17:05 you would pay in order to interrupt your batch job? It’s very different than the price you would pay to interrupt your, um, live job. If you have a live job, you’re, you are probably paying an SLA when you break for your customer. Um, does that make sense? Yeah, I see. So this is actually the market for breaking leases. These are the market for breaking leases. Oh my God. And then there’s also the market for future leases or like for leases that are, um. Oh, I thought this was like a subletting situation. This is like a lease break. So one way to think about it, um, is in power markets, there’s actually two markets. Um, there’s like the day ahead market and
17:40 there’s the live market. Compute kind of has the same thing, um, where it has capacity that is coming online, um, or capacity that’s about to, um, arrive to you. Um, and then capacity that is currently in use. And these have different price dynamics. Okay. Interesting. backwardation or contango? Backwardation or contango? Yeah. so do, uh, is compute further out price higher than spot or lower than spot? Um, is compute further out price? Does it, does it go like this? Does it go like that over, uh, you know, over like one year, one year out, two year out?
18:16 Do. Oh, this is an, uh, both at different times. I find it fascinating that you don’t know. Yeah. I mean, both. It depends on who you are. Um, okay. So, uh, I’ll make the case. Um, wait, so you’re asking is compute sold right now? same unit of compute today. Yes. Uh, maybe not today, maybe a month out versus one year out versus two year out. Is that, is that curve going up or going down? We see, like a hump. Oh my God. okay. So explain, um, like, I think this will make intuitive sense is basically like if I have milk and it is about to expire, I will lower the price on it. Yes. Um, and so that’s basically what we see.
18:52 It’s like, it’s the Nvidia release cycle basically, right? Uh, well, we’re more saying like if there’s a block of compute and it’s approaching now, um, it can’t be sold in the past. And so it has to drop its price in order to clear at some price. Yeah. And so initially you see this like drop where, um, stuff that’s sold right now, anything that’s on the market right now, um, is something that hasn’t been sold. And so therefore the price will drop. Um, and then it goes up, um, to where it wants to be. Um, and then maybe compute in the future is worth less, um, because people don’t want, like people would like the compute now. And so they’re sort of paying for the shorter lead time. Um, and they pay less for the compute that’s off into the future.
19:28 This is what I think a normal healthy curve looks like. Yeah. The place where I’m like, oh man, no s**t. Um, is that, um, right now if you took a B300 and you trying to run inference on it, um, you can make $40 an hour or so, um, selling that as inference. Any model basically. Um, and so that’s if you use it at a hundred percent and maybe if you only have like 50% utilization or something, so like $25 an hour. Yeah. So $25 an hour. Meanwhile, you can buy B300s at like five to $6 an hour. One of these numbers has to change. Yeah. It’s a huge arbitrage. It’s huge arbitrage. this in your talks a lot.
20:04 Yeah. So, um, a bunch of people are basically looking at this and thinking, um, if I think prices are going to go up, I might be willing to pay more than what I previously thought in the future. Um, because I might think, okay, I can buy a five right now, but if I can’t buy anymore, then maybe I will be willing to buy even more in the future just to lock it in if it’s going to come online later. So we see the other side as well. Um, kind of depends on who you are. Cool. So, is it okay. And, and I just, I just really want to make super clear on this. Of course. Do you care about the different generations of chips and, or do you just care about overall flops and you spread it among whatever chips you want?
20:40 Uh, we care about different generations of chips because. you care. I think like, I only, I guess do your customers care? Yes. Right. Oh, wait. When you say a different generation chips, do you mean like. So like B300s, B300s. exactly. Yeah. Or like even AMD or whatever. Right. Uh, yes. I think people totally care about the generation of chips. Um, uh, I think part of this is because of things like VRAM. Um, and part of this is because of like, um, larger NBLink domains. Yeah. And things like that. So there’s just no way to abstract all this to like some generalized unit of compute that you can sort of mush around and whatever you like. I think like when you were saying oil, even oil isn’t oil. Oil is actually just like a bunch of different types of oil.
21:16 Yeah. Like jet fuel or car fuel, whatever. I think it’s the same thing. Um, you’re going to get a bunch of different types. It is, but you have oil refineries. Yeah. can effectively have black boxes. You can put oil in and get like whatever out. Um, if compute, it is so heterogeneous, I guess is the word. It’s that you just have to expose the underlying complexity to your end customer, which is bad for you as a market maker. Maybe. Like you want to identify the one like choke point or like, I don’t know what the better word for choke point is like a shelling point where everyone agrees like, okay, like this is kind of the standard and we’ll, we’ll all agree to this standard and, uh, you know, it’ll
21:53 be somewhat inefficient. Yes. Uh, so we’ve been able to reduce the, um, the, the problems to a certain level, but yeah. Like your job is like dimensionality reduction. Yes. We reduce a lot of the dimensions, but nonetheless, there are still dimensions. Yeah. It’s still going to exist. Do you, do you know how many SKUs you have? So, um, like five times a hundred times like 5,000 equals, I don’t know. Like what’s, we actually made SKUs like a concept, um, that, and the goal is to make it so that somebody else can make their own SKU. So if I want to buy Blackwells, I can say I’m willing to
22:31 buy Blackwells at $6 an hour or less. And you can match me against GB300s or you can match me against B300s and I don’t care which Blackwell as long as it’s less than $6 an hour. but you just said you care about memory and other stuff. Totally. But not every customer does. So some customers like, um, just can work on Blackwells, whatever it is. Some customers are super hyper specific and they need this exact thing and they’re going to pay more for that because the market’s going to charge them more. Okay. Um, and that’s what we do. So, and so people use your APIs to programmatically define what they want and then you match that with whatever composition of sellers, uh, they might, they might, they might be able to fulfill.
23:08 Yes. Um, with a caveat that having a named query is a new thing we’re doing. Um, but the query system as a concept is how the order book works right now. Okay. Interesting. Um, all right. So, uh, you know, I, I think that’s like a lot of like, I guess, nitty gritty. Are there anything about like, I guess that you want to educate people on whether it’s financial people or engineers about the state of the compute trading market as, as you’ve developed it? Because effectively you guys are defining it. Um, mainly just the requirement for physical settlement. Yeah. Um, we think that if you, we, we spent three years learning why you need
23:44 physical settlement. Um, and, um, our entire conversation today is, um, that it is, these things are not as fungible as you think we’re going to try and make them fungible. That’s the industry’s job to do that. Um, but without that, you run a lot of risks. Um, if you try to build, um, too much financialization before you’ve, the industry has had time to build out the, um, the physical settlement. Um, I think what you have is you have, um, a market that’s very easy to manipulate and a market that has a lot of basis risk where, um, the price you may be hedging against actually has nothing to
24:22 do with the price that, um, the market, you can actually buy the capacity for. Yeah. So for, listeners, uh, we, we talked about this briefly last time, but, um, people are often shocked to find out that let’s say the market for credit default swaps is 12 times the, the, the size of the bonds that they insure, uh, something like that, or like, uh, the, the markets for mortgage, um, sort of reinsurance or, uh, all that is, is larger than the underlying physical settlement. So that’s why physical versus cash settled. Yeah. And, and like basically cash settlements are like exactly the kind of, uh, the side bet that you see in the big short, um, where, uh, they,
24:59 they may, uh, may or may not have anything to do with the underlying thing itself. And it may, may be used as a proxy for overall other stuff. Yes. That, and you don’t want to, that to go out of control. Yeah. So to be clear, we think, um, cash settlement as a future concept is a great idea. Yeah. Um, we just think it should be derived from physical settlement. Yeah. because as you said, it’s a side bet. Um, and it would be kind of like if you, I mean, be kind of like if you made a cash of the future before you made the physical settlement, um, which I think people are kind of doing already. Yeah. Totally. People are trying to make cash on it before you make the physical settlement. And people do that for all sorts of things, including
25:34 startup stock. Yes. Right. Like you can buy and sell a made up share of open AI, but it’s nothing to do with the actual share. Yes. Um, and lots of those go very poorly. Um, like it would be kind of bad if you made a, um, it would be bad if you started selling, um, the, the derivative on a stock that doesn’t exist yet. Um, yeah. Yeah. Well, so I mean, one thing that we do, so I used to be a currency trader, right. Yeah. And, interest rates as well. And what, one thing we do is you have a board that gets together and says like, this is the London interbank offered rate LIBOR. Yes. And we all agree like this is the, the, the monthly fixing and we all
26:12 trade to that fixing. Yes. Uh, is that even meaningful at some point in three years from now? So critically LIBOR was manipulated. Yes. our concern at the moment. there’s 12 banks? Yeah. Yes. Um, well, our concern is there’s a very similar problem, um, where you can manipulate an index. Um, our impression is why you can manipulate that index is because a lot of the indexes are based on a very small amount of data that has, um, uh, not a lot of correlation with the actual price of buying a new reserve contract. Um, that’s what we’re concerned about. Okay.
26:47 Interesting. Um, all right. I think, I think that’s the, the, the rough state of things. Uh, um, I think the other part of just, just today is, uh, you, you, you, you talked about an event with jump and you’re, you’re friends with Silicon data. Uh, what are you learning from all these other players? Like who are the major players that you would want to call out as like doing interesting things? Ooh. Um, so we’re big fans of Silicon data and compute exchange. Um, I think they were also very early to this. Um, and I think they are doing like a reasonable shot at the, a path that we think, um, that we’ll likely end up being partners with.
27:23 Um, can you distinguish like your, your different paths? So, um, the general just is they’re doing cash settlement. We’re doing physical settlement. Um, we think both can exist and should exist. Um, and that they’re sort of mutually beneficial to each other. Um, and so totally fine. Um, so I think they are one of the earliest in the space and I think they are, very clearly took the bet, um, well before anybody else did. And I think that’s like very honorable to do. Um, um, you mentioned, uh, jump. Semi-analysis, jump. Like these are the players emerging. Like as far, like you were my first introduction to this. Yeah. And then, and I see all these other guys coming in and I’m like, okay, like it’s something’s happening here.
27:58 I don’t know the landscape as clearly as you do. Yeah. So, um, the categories of folks that we’ve seen is there’s folks that are trying to set better standards. So semi-analysis I think is working to try and create better standards. Um, there’s folks that are trying to do, um, residual value insurance and so on. Um, there’s a number of different companies in this space. I don’t know any of them super well, but, um, I know, know this is a concept and I think that’s important. Um, there are folks that are trying to do, um, uh, deposit, um, lending where like they’ll front your credit risk for the cloud. There’s an ecosystem of companies that are forming around how do we resolve the financial bottlenecks of compute.
28:34 Um, and it is cool to see. I think effectively what you want in this ecosystem is buyers and sellers. You want an analyst. You want, uh, we said insurance or, uh, what’s the, uh, residual value. Yeah. Yeah. and you, we want this like healthy ecosystem out there. Um, but obviously you guys are owning the problem of physical settlement, which, which is really interesting. Um, okay. So, I mean, we can go into like the, the hard tech side because obviously you are also hiring, uh, for extremely cracked engineers. Yeah. describe your stack. Um, we do all the things a cloud does without being the
29:10 cloud. Um, as in we don’t own the hardware. the reason why we don’t own the hardware is because if we did, we would be able to manipulate the forwards market. So we have to be Switzerland. We have to be independent. Um, which creates a lot of sort of strange things about the company. Um, but in order to do physical settlement, you have to do site visits. have to visit the data center. Um, you have to, um, you have to design the bill of materials and ensure that, um, you have to basically do like supply chain consulting, um, where you have relationships with different OEMs that, um, ensure that things get delivered
29:46 on time. Um, then you have to, um, do cluster architecture. Um, uh, so you’re sort of designing the cluster itself. Um, then you’re, um, rolling out racks or you’re working with partners to roll out racks and supervising that. Um, then you’re doing, uh, backplane management. So you’re taking over the, um, uh, PDUs and the switches and the DPUs and, um, UFM and BMC and you’re operating, um, the whole, um, harness around the nodes that you’re selling. When you say you as an SF compute employees. Yeah. Wow. Um, so, uh, when you buy a node from a cloud, um, there’s the bare metal, but there’s all this s**t around the bare
30:23 metal. Yeah. Um, and in order to make that bare metal node work, you have to operate everything else. Then we. Because I just like traditionally the job of the data center is to extract some of this away from you. Yeah. We’re the ones that do that. Okay. Right. Um, but you’re not, you’re not in the data center business. You’re not the sort of equinixes of the world. We’re, we’re not in the data center business sort of, um, what we do work with is, so we offer supercomputing as a service. Um, and what this means is folks come to us and say, you know,
30:58 I’ve got X where X is, I’ve got, you know, X megawatts here. And maybe sometimes they mean I’ve got a data center. Um, that’s like, um, would like to turn into a GPU cluster. Like my options are, I could sell the GPU cluster or I could sell the data center to somebody or I could become the cloud. And then we go to them and say, great. Um, we will turn you into the cloud. We’re the supercomputing part. You’re the data center. Let’s partner together. And then we will sell and I’ll make money. Um, sometimes it’s people who have something that could turn into a data center, in which case we’re going to basically project manage that up to the layer.
31:33 Um, and then we will work with those far, we’ll work with those folks to turn their powered land, um, or whatever they have, um, into a data center by working with partners. Um, and then we take over at the supercomputing part. Okay. This requires us to have some sort of data center expertise in house, but we don’t self-perform builds. Um, so yeah. Yeah. So it’s, uh, somewhat modular because you have different scenarios you need to play under. Yeah. Uh, someone once described us as like, um, uh, like Palantir for compute, um, because it’s like these, each time it’s this new rodeo.
32:10 Um, uh, but, and we like, we, we deeply partner with folks. On both sides. Yeah. Uh, and primarily the U S primarily the U S, but we are operate around the world. Yeah. At some point does SF compute become a misnomer? Uh, so when, so we’re deploying clusters all around the world. Yeah. And when we do that, we’re working with people who themselves are trying to be clouds all around the world. Um, they look like sovereign compute players. Um, and our pitch to sovereign compute is amazing. Uh, let’s help you set up a cloud in your own country. Um, and we will bring the San Francisco offtake. We’ll bring the San Francisco expertise.
32:45 We’ll set up the clusters. Um, that’s our whole business. Ah, um, we’re going to make it. Who are your main, uh, your biggest announced partners that you can name? Um, nobody I can name. Really? Wow. Uh, we will have some announcements, um, in a week or two, um, that we expect to come on. Yeah. All things sort of. Sure. Sure. Good luck. Yeah. No, I mean, uh, I, I think this is really interesting for every country to like, who’s, who’s like investing in sovereign AI to figure this out. Um, and it’s interesting that like SF is still sort of like that brand name and, uh, that, that is, that is very marketable.
33:20 So I think one of the interesting things is just thinking through this market. NVIDIA is obviously like the 800-pound gorilla in the room. Why didn’t they make SF Compute? And again, the answer is going to be market manipulation. But let’s just really get into the incentives of the multiple players. Yeah. So our impression is that NVIDIA can’t make it, but benefits tremendously from it existing. The reason why they can’t make it is because they would then not be independent enough for people to trade on it. If you don’t trade on it, then you don’t have enough liquidity for lenders to finance against the forwards market.
33:55 This is also why Crusoe and CoreWeave and Lambda, why the hyperscalers can’t make it, why Base 10 and Fireworks and Modal can’t make it because they have too much demand on that side. This is a thing that benefits the entire industry because it makes the financing costs go down. But nobody can make it unless they give up the ability to do all the big inference money and all the big cloud deals. You have to kill your business in order to make this. That’s why nobody does it. like structurally, they’re not set up to do that. And you’re basically reducing friction and that’s all your revenue.
34:30 You’ve announced some milestones. Actually, I forgot to say that up front. Can you repeat what they are and just take a victory lap a little bit? We’ve closed several hundred million dollars of volume. We have a significant number of new clusters on the way. And then the things that I think I’m most excited about is we’ve shipped a whole bunch of new features that make SF Compute very viable for platform-as-a-service companies. An example is we allow customers to subdivide
35:06 their InfiniBand fabric without having to ask us. So many clouds, when you go to buy them, they will give you one InfiniBand fabric. And then if you want to redraw the lines of that InfiniBand fabric, you need to contact them and they will then do something for you. We now can do that on the fly for customers, which I think is really great because it interfaces with the market really well where you want to be able to redraw things in order to scale up and down your clusters. Would you describe this as like you’re virtualizing the data center and making it programmable? Yeah, we’re virtualizing the data center and making it programmable. So all the stuff that you would have gotten if you had owned the
35:43 hardware, we’re trying to virtualize that, which I think is exciting to us. It’s funny, like the last time this happened was like the cloud, which is really cool. And then finally, you know, I think what is your call to action for people? You know, is it engineers? Is it like, give me your supply? Like, what is it? If you would like to own a GPU cluster and you want the residual value from it, you want to have it for your portfolio, we would love to work with you in order to build that GPU cluster. We will take over and we will do the supercomputing portions of that and
36:18 ensure the things go right. We own the SLA. And then if it turns out that that was a bad idea that you need out of that cluster, we can immediately sell that cluster to somebody else so it doesn’t hit your balance sheet. Very simple pitch. Well, thank you so much for your time. You’re doing a great service to the industry and I just am happy to be your friend. I appreciate it. All right. Thanks. That’s it.


This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.latent.space/subscribe

A few years ago I stumbled upon Leash Bio’s Substack, and was hooked. Be it their Kaggle competition that showed that no, we really haven’t solved protein-ligand binding, or their crazy discovery that knowing a molecule’s author is perhaps all you need to know to predict how good it is. Their substack — or at least Owl Posting’s deep dive on them — should be required reading for any scientist trying to do chemistry with AI.
So I was super excited to finally be able to sit down with Ian Quigley from Leash Bio, and pick his brain. Leash specializes in DNA Encoded Libraries (aka “DELs”), a way of collecting ridiculously high-throughput interaction data between proteins and small molecules. Before I discovered Leash, I had heard nothing but horror stories from DEL users. And yet, Ian and friends somehow found a way to tame the beast, extracting clean and reproducible data that is just perfect for large scale modeling. Without teasing too much, the answer comes down to epistemic rigor, and an incredible conviction that you just have to Do The Thing.
It gets even better! Near the end of this episode, Ian teases something his team was just playing with when we recorded: an AI playground for agents to do verifiable scientific reasoning. They discovered the data they had spent the last few years collecting could provide tens of thousands of “lab-in-the-loop” simulations, just what an agent needs for scientific RL. Two months later, it’s matured into Minos, and today they’re releasing a full preprint, blog, and open dataset. We’ll be releasing a lightning podcast in a few days with an update. Stay tuned!
We had a blast with Ian, probably one of our most fun episodes to date. For both complete chemistry novices and expert AI x Bio practitioners, we think you’ll get something out of this.
Enjoy!
We covered:
* Why DELs traumatize everyone who touches them, and how Leash puts dozens of barcodes on each molecule to catch proteins that bind the DNA instead of the drug.
* Ian’s rule that biology only scales two ways: taking pictures and sequencing DNA.
* The Kaggle competition: about 4B measurements from a basement and about 2,000 teams, with nobody beating chance on an unseen chemical library. Also why Leash quietly feared co-folding would make them obsolete.
* Clever Hans in ChEMBL: models can guess a molecule’s chemist from its structure, and “who made it” predicts activity about as well as the structure does.
* Hermes, trained only on Leash’s own data, keeping pace with Boltz on public benchmarks. Plus the BRD4 molecule Boltz had already seen in the PDB.
* Breaking the pocket: engineering protein mutants at scale after catching Hermes memorizing “nicknames” for proteins instead of binding sites.
* Lumacaftor-like hits as an accidental detector for misfolded proteins.
* The agent sandbox: LLMs that reason about SAR in plain English, hunting for the best binders among 6.5M molecules on a budget of 100.
* The Recursion mafia, and why its alumni companies all rhyme.
* The Kosuri paradox: if you can collect the data, why do you need the model?
* Going from a DEL hit to good mouse PKPD in six months on an inflammation program, and the case for pills over antibodies on cytokine targets.
* What stops a better-funded lab from outscaling them, and Ian’s by-fiat wish for massively parallel purification.


This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.latent.space/subscribe

Watch along AIE NYC and join us at AIE CODE in SF next month!
This episode includes a live demo of the Taku robot that is best enjoyed on YouTube.
Last month, Dyna made waves by announcing Dyna-2, a world-action model pre-trained on one million hours of human video, claiming several new scaling laws on human data across four orders of magnitude. They have now followed up the model with a physical robot agent, Dyna-2.1, that has to be seen to be believed:
In his talk at AIE this year, Jason talked about how the real challenge for robotics in the real world is reliable, long-horizon autonomy, and how Dyna got its models to a 99.4% success rate folding restaurant napkins for 24 hours straight, including recovering after it pulled over the whole stack.
From training a world-action model on over one million hours of human video to deploying robots that fold thousands of napkins daily at Din Tai Fung, Dyna Robotics is pushing physical AI beyond impressive demos and into real-world production. In this episode, Dyna Robotics co-founder and Chief Scientist Jason Ma joins swyx on-site to unpack how robots can learn new tasks from as little as 45 minutes of demonstrations, why predicting the future of the physical world may be the key to general-purpose robotics, and what it actually takes to build robots that work reliably for real customers.
We go deep on Dyna-2, Dyna’s million-hour world-action model that combines future video prediction with robot actions, the scaling laws behind human-to-robot transfer, diffusion transformers and flow matching, and why synthetic environments and self-generated training data could unlock the next wave of robotics progress. Jason also explains why Dyna builds its own hardware instead of relying on off-the-shelf humanoids, how agentic harnesses enable robots to execute hour-long workflows, why dexterous hands and missing sensory data remain major bottlenecks, and what separates real commercial deployments from the graveyard of robotics startups. We also cover world-action models versus vision-language-action models, GPT-6 and coding agents in robotics, and Jason’s journey from AlphaGo and Penn to NVIDIA, Google DeepMind, and founding Dyna.
We discuss:
* Why Dyna-2 was trained on over one million hours of human video
* World-action models: predicting future video to guide physical robot actions
* Diffusion transformers, flow matching, and real-time robotic control
* How robots learn new tasks with as little as 45 minutes of demonstrations
* Why robotics has no equivalent of the internet and data remains its biggest scaling challenge
* Human egocentric video, robot demonstrations, and cross-embodiment transfer
* Synthetic environments, simulation, and models that generate their own training data
* Why robots need on-device inference, model distillation, and custom hardware
* Why Dyna chose wheeled semi-humanoids instead of traditional bipedal robots
* The hardware bottleneck: why dexterous robot hands still overheat
* Best Western, Din Tai Fung, and the difference between robotics pilots and scaled deployments
* The Bitter Lesson of Robotics: scaling data instead of engineering specialized solutions
* Agentic harnesses and robots executing multi-step, hour-long physical workflows
* GPT-6, coding agents, and AI-generated simulation environments for robotics research
* World-action models vs. VLAs, and why vision alone may not be enough
* Jason’s journey through AlphaGo, Penn, NVIDIA, Google DeepMind, and the Eureka project
* Dyna’s research culture, deployment-first philosophy, and vision for millions of robots
Jason Ma
* LinkedIn: https://www.linkedin.com/in/jason-ma-742224a2/
* X: https://x.com/JasonMa2020
Dyna Robotics
* Website: https://www.dyna.co/
* X: https://x.com/DynaRobotics
Timestamps
00:00:00 Introduction and Dyna-2’s World-Action Model
00:03:08 Diffusion Transformers, Flow Matching, and Generalist Robots
00:08:19 The Robotics Data Bottleneck
00:13:50 Synthetic Data and Self-Improving World Models
00:16:51 Real-Time Inference and Model Distillation
00:19:45 Full-Stack Robotics: Hardware and Supply Chains
00:25:02 Why Dyna Built a Wheeled Semi-Humanoid
00:29:35 Teaching Robots New Tasks and Evaluating Performance
00:31:59 Autonomous Workflows and Hardware Limitations
00:36:06 Real-World Deployments at Best Western and Din Tai Fung
00:41:07 The Bitter Lesson of Robotics
00:46:13 Agentic Harnesses for Physical AI
00:49:07 GPT-6, Coding Agents, and AI-Native Robotics
00:52:02 World-Action Models vs. VLAs
00:59:17 From AlphaGo to Founding Dyna Robotics
01:04:20 Dyna’s Research Culture and the Future of Robotics
Transcript
Introduction: Jason Ma and Dyna Robotics
Swyx [00:00:00]: All right, we’re here on site with Jason Ma, the co-founder, chief scientist of Dyna Robotics. Thanks for having us in your office.
Jason Ma [00:00:06]: Of course. Thank you for coming.
Vibhu [00:00:07]: It’s been a blast. We got to see—you’ll see plenty of B-roll—drinks being pulled, a lot of evals, robots with knives for the first time. So under the hood of all this you have a world action model. What is that?
Dyna-2: A World Action Model That Predicts the Future
Jason Ma [00:00:22]: So last month, we released our world action model called Dyna-2. So it’s the first world action model trained on more than one million hours of data. So going back to your question first, world action model is a model that’s able to predict the future video, the future state of the world, so hence the name world model. But in addition, you can also use that future generation to also produce actions robots can execute in the physical world to control the robot. So why do we do that? It’s a very powerful architecture, because if you think about how humans act in the physical world, in our brain, we are implicitly thinking about how the world will evolve based on our actions. So now we have a model architecture that’s able to do that to guide robot actions. So what we find is that by doing so, the model is much more dexterous, much more robust, and much more capable of generalizing to environments, to scenarios it’s never seen before.
World Action Models: Inputs, Outputs, and Visual Reasoning
Vibhu [00:01:17]: I wanna prompt this question a bit. I think a lot of people hear world model, and they think you’re outputting video or world or 3D state. You’re also doing action. I think most people don’t understand what models are under the hood of robotics. So what are the inputs and outputs? What’s the training data? What is the architecture? Give us all the details.
Jason Ma [00:01:39]: So, for inputs, the most important input is video, right? You can think about videos that the robot is capturing with its cameras, right? So video or images of the past many seconds, that’s the input. But for robots, what’s really interesting is that robot also has a lot of actuators and joint encoders on itself. So the robot also knows, the angles of its joints. So those can also be inputs to this world model. That’s something that’s very different from just a pure video model that’s only looking at videos, right? So that’s the input. And additionally, for robots, you want to be able to talk to it, to instruct it to do different tasks, right? So you can also feed it language instruction. So those are what I consider the basic set of inputs that go into any robotics models, including world action models. And once we have that the output, at the basic level, it needs to output actions the robots can execute on the hardware. But in the case of world action model, what’s really interesting is that in addition to the actions, it can also output future video, right? So basically, you can almost output the future video or some representation of it first, and then use that to guide robot actions. So the way I think about it is this is kind of analogous to chain of thought, you know. And basically, the models are reasoning in the visual, the physics space, and then outputting actions. So that’s the key difference between world action models outputs versus other kind of models, robot models outputs.
Diffusion Transformers, Flow Matching, and Smooth Actions
Vibhu [00:03:08]: You put out a good blog post around your model. I think a lot of people understand language models or transformers, next-token predictors. In this case, you shared quite a bit. There’s diffusion, flow matching, you’re predicting next world state plus action. One thing you say is a million hours. How do you determine the architecture? Like, what are the choices there? What is the architecture? I feel like in language models, there’s a lot of take something off the shelf- Take trillions of tokens, do prediction, SFT, RL, and you have a model. What about this? How do you go from nothing to your model? Is there vanilla architecture off the shelf? Do you determine all this and what’s, what is the current state?
Jason Ma [00:03:49]: Yeah, I think, yeah. So the mental framework to think about how do we build models for robotics is that there are essentially a couple things to look at. One is data, and the second is how do we make a model learn as much as possible from that data? So let’s first talk about the model actually, right? So the blog post we released is our Dyna-2 world action model. So under the hood is a diffusion transformer. The model uses an encoder-only transformer architecture.
Vibhu [00:04:20]: Right, yeah.
Jason Ma [00:04:20]: But what’s really interesting is that in visual domains, right, if you think about generating images, generating videos, and in this case, robots looking at images to generate actions, people have found the diffusion kind of architecture to work really well. So what is a diffusion model? A diffusion model is a model that takes some random noise and through iterative computation can go from noise to something that’s really high-dimensional, that’s something really highly defined and have very fine-grained feature. If you think about generating images, right, you want sharp edges, sharp corners, color contrast. And for robotics, even though you’re not generating image, you’re generating actions that needs to be very precise, very smooth, right? So it has kind of the same kind of consideration and desiderata as generating image. So that’s why from first principles, the whole field, not just us, have been using this kind of diffusion transformer architecture to train robotics foundation models. You need the transformer aspect because transformer is the thing that can just ingest the most amount of data, can learn. Then diffusion helps you to refine the outcome to get to the smooth, precise, fine-grained motion you need to actually execute on the robot. So that’s the architecture that goes behind a lot of our recent breakthroughs.
Swyx [00:05:37]: And we’ve covered flow matching before on the podcast. It is mostly an efficiency speed-up. Like, let’s say you normally would take forty to a hundred diffusion steps. You can reduce it to four.
Jason Ma [00:05:48]: Yeah, that’s right. So with the diffusion model, one thing that’s very hard in practice is that you need to tune the scheduler, like by how much you’re tuning or converting the initial noise input into the final output. So flow matching is kind of like a special case of diffusion that makes the transformation much simpler. In practice, you don’t have to tune as much, and it works just as well. That is why I think about two to three years ago, most generative modeling for image and video has shifted from a diffusion model to a flow matching model. But the concept is the same, and flow matching is just a special, simpler to tune, variant of diffusion. Yeah.
Swyx [00:06:26]: It actually makes me think that. So flow matching obviously has trade-offs in quality. But actually, it’s very suited for action diffusion and robotics because you get to reevaluate in real time. Whereas in images, you are— it’s basically one shot.
Jason Ma [00:06:40]: That’s right.
Pretraining a Generalist Robot and Adapting to New Tasks
Vibhu [00:06:41]: Which is pretty fun. We got to mess with the laundry folding robot … throw a random towel in. The models adapt, right? I think one key thing about your models is that they’re tr— they’re made to be generalists, so one foundation model and you adapt them a lot. How does that work? Basically, in 45 minutes of mapped data, you can generalize towards. We saw an example of pulling drinks out of a fridge. What’s the what’s the mindset there of one general model, and then how do you adapt to different tasks?
Jason Ma [00:07:14]: So the way we think about it is that for robotics, we need to build a very general, strong base model, right? And then— Because if you think about how a human interacts and behaves in the physical world, right, we have very strong base knowledge. We can just readily do a lot of tasks. But if you want to become an expert at any given task, there is some amount of training that needs to happen. And I think to train a general-purpose robot like the one we’re building here, the same principle has to apply. So our approach is to first train a general model that’s training on as much data as we can find. So that is why in our Dyna-2 report, our model is trained over a million hours of human data as a starting point. And then given that strong foundation, what we found is we can quickly make the robot or make the model do many different tasks with a very minimal amount of fine-tuning to just tell the model, “Here’s the task I want it to do, and here’s very few amount of demonstrations,” and the model can readily do bunch of the tasks that you have seen in office. Yeah.
A Million Hours of Data: Egocentric Video and Robot Demonstrations
Swyx [00:08:19]: We actually have quite a few people who deal in data, who sell data. What is good data? What is bad data? What do you want to buy? I think that these are all open questions. Like, obviously, I think the top, important thing is egocentric data of humans doing tasks. What else would you modify on that basic statement?
Jason Ma [00:08:42]: So I think data is the most important thing in robotics today, because unlike language model, there is no internet, right? So every company that’s building model, building hardware has to think about data. So there’s, roughly couple of different ways of operating here. One is that you can collect a lot of data yourself, which is what we are doing as well.
Swyx [00:09:02]: You have employees here doing that.
Jason Ma [00:09:04]: There are robots everywhere in the office, right? But if you think about how much data you need for robotic foundation model, I think the amount of data any individual company can collect by themselves is just nowhere near sufficient, right? It’s kind of like, you’re trying to build an internet by just having Reddit or just having Facebook. They’re great websites, a lot of users, but it’s not enough, right? So when we think about data, there has to be some sort of data acquisition happening, right? You buy data, you look at data that’s already out there. And then there’s many different tiers, right? You mentioned data quality. I think data quality in robotics is a non-stationary problem in the sense that what is high-quality data for your model changes over time. Like, when your model doesn’t know anything, like when it’s zero, any data is useful, right? You can use a lot of video data. In the case of egocentric data, I think it’s actually a very good way of pre-training robots because human data is so diverse. Like, it covers every single physical activity you can possibly imagine. And that gives robot, the model such a strong prior on how things, how the world can evolve around a embodied actor, human. But the issue with human data is that until your robot looks exactly like a human, there’s gonna be some gap between human actions and the robot actions, right? So just by training on human data, today, it’s not very likely you can just execute robot actions directly. So you still need some amount of adaptation using the robot’s own data, right? But the robot’s own data collected on a robot is gonna be much more expensive and much harder to collect. So the kind of recipe we’re thinking about is like, how do we take the largest source of data we have today, like egocentric data, to give the model a strong prior over everything physical manipulation can cover, and then combine that with a small amount of robotics data to make the robot master whatever task we’re interested in. So data quality and getting data in robotics is really, I think it’s a fascinating, multifaceted problem. But the reality is that there’s not a single winning strategy like, “Oh, here’s one kind of data. I’m only gonna collect on that,”
Jason Ma [00:11:25]: Which will give you a strong foundation model. You really need to think about, “This human data covers this part of robot cognition I need. Robot data covers low-level reflex dexterity the robot also needs,” and put them into one model. And I think that’s what gives a really adaptable, really strong model at the end. Yeah.
Data Partnerships and Specialized Training Examples
Swyx [00:11:44]: I’ll prompt you, just for again, for the people listening who want to work with you. What, d— what data vendors do you want. This is your call for data. What-
Jason Ma [00:11:55]: Yeah, no, that’s a great question. And yeah, so in our Dyna-2 blog, at the end, we actually have an email address called datapartnerships@dyna.co. So if you’re a data vendor that has a lot of high-quality, let it be egocentric data, robot data, video data, or anything you think could help robots, send it to that email. Our team will take a look, and we’ll get back to you.
Swyx [00:12:16]: But at your stage, are you. You’re probably not buying any more egocentric data off the shelf, right? That you probably want to be like, “Here’s what I need. You go-”
Vibhu [00:12:26]: Is there a niche that’s lacking?
Swyx [00:12:28]: You need a Scale AI, you need a Mercor for doing this. I see. The custom data-
Vibhu [00:12:32]: The folks that don’t work in robotics are probably thinking like, “Okay, some niche factory.” Like … you showed putting tubes in. There’s only so much data you’ll find about that but like would a lot of that help you?
Jason Ma [00:12:43]: Yeah, no. If you really want to nail a task, having data from that task will certainly help you a lot, right? So for us, it’s like we still think, just a million hours of data is not enough. It’s just the beginning. Like, we showed the first model on a million-hour scale, but we need more than that right? So we are still looking at a lot of just raw, diverse data, that we can get. But we’re also thinking about how do we get more custom data either collected in-house or by our partner for really, specific niche tasks? Because I think you still need that to get to excellence that any model needs to actually be useful in the physical world, right? It’s kind of like humans, you know. If, Sean or me, we go work at Rosewood Hotel tomorrow, we are not gonna be able to make the bed the way they want, right? We need a lot of training. So that kind of data, like special data, that’s kind of like what Mercor is doing today for language model, I think will be very valuable for robotics as well.
Vibhu [00:13:42]: So hotels, strap a GoPro on your employees’ heads and.
Jason Ma [00:13:48]: And then there’s probably more of that.
Vibhu [00:13:49]: There’s the
Synthetic Data, Simulation, and the Eureka Project
Swyx [00:13:50]: Put themselves in their way. Yeah. So one thing that we discussed is the closing this interview, and what’s very interesting is now you can basically fabricate environments on the fly. Have you tried that with the frontier models? What’s your experience? Are they good enough?
Vibhu [00:14:06]: Are you using any world model robotic. Like, world model companies to generate these and train on them?
Jason Ma [00:14:13]: So, I think generating synthetic environments, synthetic data, simulation data for robotics, it is going to be one of the next frontiers, right? ‘Cause today people have looked at egocentric data, robot data. Those are scaling, but I think synthetic data will be a new dimension of scaling. So going back to your question yeah, we have been exploring using frontier models to generate synthetic, environments for us on our data, on our environments that we care about. And surprisingly, I think now it’s at a point where this technology is actually very much worth going really deep into. So for some context, so, in my PhD, I primarily focused on simulation-based training for robotics. Back then, simulation wasn’t ready, but we had this paper called Eureka, which was the first paper taking a large language model. At the time, it’s GPT-4, which is how to self-generate environment, generate reward functions or reinforcement learning, and just do things in simulation. And at the end, I think, we had this really nice demo of the robot spinning pencils, right? That was really cool at the time. But that kind of thing was still very limited at the time, because back then the model didn’t really have good vision capability, didn’t have a innate 3D understanding, right? So it was basically just generating code and using very limited feedback from text to try to come up with new simulation environments. But today the vision capability of these frontier models are crazy. They can do real to sim, do CADing, there’s 3D reconstruction and all of that. So I think now it’s at a tipping point where
Jason Ma [00:15:52]: This technology could really accelerate robotics. And for us, right, we think about that as a way to accelerate our internal progress, right? Because we’re building models, we’re building hardware. So if a frontier model can give us more data that way, like generate really tailored data for ourselves, that’s extremely useful for us. And another thing we’ve been thinking about is instead of using these models to generate 3D environments, we have also used our own world model, right, the world action model I was telling you about, because the model not only generates action, it can also generate videos. We— The model can also generate its own training data. So that becomes a self-fulfilling flywheel that I think will also be powerful. So overall, I’m just really excited about this direction because I think even though we have scaled to a million hours of data, we are deploying a bunch of robots. The robots are act— already starting to do very cool things. We’re just at the beginning, just the tip of iceberg for what is there to come. Yeah.
On-Device Inference, Latency, and Model Distillation
Vibhu [00:16:51]: For those on the research side, we recently had Runway on the podcast. One of their things is robotic world models. They’re, shifting a little towards there, so … nudge people to look at it. One of the questions I have about you making your own models is what is the constraints you operate in? So if you draw a parallel to language models, you see GPT-OSS 20B is very fit for your average MacBook. Then there’s a larger one that’s fit for a single GPU. Do you work in the constraints of robot, of model on on device, in cloud, a hybrid? And then how does that affect your scaling law of data? A million hours is great, but like do we just scale up parameters, data?
Jason Ma [00:17:38]: No, that’s a great question. So I think what’s really different about robots versus language model and other applications is that for coding agents, language model, it’s okay if it’s— it has latency, right? Well today, whenever I ask ChatGPT, it takes like couple of seconds before it spits out anything. But for robots the inference latency has to be very short, right? The robot needs to act. Like, it can’t just hold a knife and not do anything for three seconds. That’s gonna freak people out, right?
Vibhu [00:18:07]: Yeah, we saw someone put their real hand under a knife.
Jason Ma [00:18:10]: Our model’s, pretty smart, so that’s okay. But in general if you think about very dynamic tasks, like for example, you want your robot to hold a I don’t know, a tennis racket and juggle the ball, right? It’s just very dynamic, real reactive. You can’t just like. The physics, gravity doesn’t wait for you, right? So for that reason, robotics model needs to be able to inference very fast. And the fastest way to do that is by serving on-prem, like on device, right? Because if you call a model-
Swyx [00:18:39]: Network latency.
Jason Ma [00:18:40]: There’s no guarantee-
Vibhu [00:18:41]: Self-drive a car.
Jason Ma [00:18:42]: There’s no guarantee, right? Like, so for that reason, the model that gets deployed eventually has to be of reasonable size that can be served locally. But that doesn’t mean that the initial model you train also has to be small, right? Like in language model-
Vibhu [00:18:57]: You can distill.
Jason Ma [00:18:58]: People do distillation, right? Like, the best way to make a small model very smart is by doing distillation, right? You train the biggest model, then you distill, right? So I think in robotics, we’re getting there, but I think that’s what people are going to converge on. Yeah.
Vibhu [00:19:14]: How about inference? You have your own inference stack or. A lot of this is architecture. You’ve pulled diffusion and whatnot. I’m guessing you can pull off of something. Anything you wanna share?
Jason Ma [00:19:26]: So we build our own inference stack for whatever model we deploy, because that’s— like I was saying latency is so important. So owning the whole stack is extremely important for robotics, right? You want to optimize the inference speed for the kind of model you’re running, and there’s a lot of work in there. So that’s what we do.
Building the Full Robotics Stack: Sensors and Actuators
Swyx [00:19:45]: You’re really full stack, like- Everything from the actuators all the way to in the industrial design ID, which I- Didn’t really know as a acronym. Obviously to the models and data as well.
Vibhu [00:19:57]: I’m curious are these your own actuators, or are they. How much of this is your own? Like, of course, the design, it’s not like humanoid- Actuators, like how far down the robotics stack?
Jason Ma [00:20:08]: So the way we think about, what we need to own versus what we can buy is like if the Input-output and of a component is very clear, and like there’s already extremely good mature solution, then there’s no point of building in-house. For example, on this robot, you see the LiDARs, right? So those, black circle, those are LiDARs to help the robot detect obstacle, navigate surroundings. So that’s a very mature technology that people have been using for now decades. It’s on self-driving car. So we don’t need to build LiDAR from scratch, right? Same goes with some cameras. But for anything that we think is extremely crucial for the whole system’s performance at the end, our vision is to be able to own that and build that ourselves. So one thing is, you mentioned actuator. So that’s something we’re working on and we’re going to own ourselves. The reason is that like, the quality and how you design the actuator really matters for how well you can model the robot, right? Actuator is what drives the robot. To be able to model your robot very well in simulation or to do any kind of modeling, you need to select actuators very good that can actually be easily modelable. So that’s what we care about eventually. So we’re gonna own that. Same thing with the robot’s proportion. This robot’s proportion is we measure it against real people. Why do we do that? ‘Cause we’re pre-training on human data.
Swyx [00:21:36]: Oh, yeah.
Jason Ma [00:21:36]: Best transfer, right? So there’s a lot of these kind of decision that goes into hardware that’s very much tied to like. Because eventually, it’s hardware that carries intelligence. Intelligence is the extremely important thing. We want to make sure the hardware serves for that.
Swyx [00:21:53]: I would just highlight, I don’t know if this is relevant to you, but one of the most interesting robotics discussions I’ve had is someone who was very stressed out about actuators and specifically magnets, ‘cause apparently we don’t make magnets in the US.
Manufacturing, Rare-Earth Magnets, and Global Supply Chains
Jason Ma [00:22:06]: No, we don’t.
Swyx [00:22:07]: And there’s only two countries that make magnets.
Jason Ma [00:22:09]: Rare earth material.
Swyx [00:22:10]: Rare earth, right? I don’t know what the. Wh-what’s the chemi— what’s the material? I don’t know what the thing is, but- so yeah, if anyone’s interested in that’s, apparently a bottleneck that we have.
Jason Ma [00:22:20]: No, not bottleneck.
Swyx [00:22:21]: It’s a bottleneck. So what do we do?
Jason Ma [00:22:23]: It’s a good question, right? I think, there’s a lot of exciting discussions on bringing manufacturing back to the US, but for robots especially, that’s actually much harder than what it seems. And I would say even now, there isn’t a company in the US that’s truly building everything from scratch in the US, right? Like, even when they say they assemble robot, they build whatever they’re actually doing in the US, they’re still sourcing their raw materials from other places in the world. And I think that’s something to really be aware of as a robot builder, right? Like, I think, like as a full-stack robotics company, we realized just how long the supply chain is and how long the whole problem is. It’s actually extremely hard. And I think these days, a lot of the discussion centers around a robot foundation model data, but I think hardware and just how hard it is to build good hardware is very under-emphasized. But I think that’s gonna be extremely important.
Swyx [00:23:27]: This is a platform, right?
Why Dyna Co-Designs Its Hardware and Models
Jason Ma [00:23:29]: Platform, yeah.
Swyx [00:23:30]: There are some companies that choose to be, let’s call it platform-less. They only do the model, and they will say, “We deploy on any platform.” We deploy on Optimus.
Vibhu [00:23:37]: Off-the-shelf robotics.
Swyx [00:23:38]: And so, it’s a choice, right? Like, obviously, you own everything, so you can do that. Do you see yourself building more in the US, or do you see building your own global supply chain? What’s the direction that is likely?
Jason Ma [00:23:52]: So we are currently exploring many different options, right? Because with global supply chain, then there is some, geopolitical things, right?
Swyx [00:24:02]: Shipping.
Jason Ma [00:24:02]: Shipping. Also shipping delays and all that stuff.
Vibhu [00:24:04]: Literal ships for robots.
Jason Ma [00:24:06]: And in the US, it’s just that going from zero to one will take some time, right? So we’re trying to think about a solution that can both allow us to build more in the US while leveraging global supply chain, right? But I do think, going back to your previous question on some companies choose to just build model, we build everything, I think it’s very important to get this right. I feel that for robotics, the hard thing is. Like building the whole system. What are the necessary things to build a whole system? Like, previously, I was at DeepMind, I was at NVIDIA, primarily just building models, and I saw the upper bound of the system when you don’t control the hardware, when you don’t get to optimize that right? Because if you think about the value creation for robots, eventually robots are here to deliver a service to solve some physical tasks. So if you wanna solve physical tasks, you wanna be able to optimize everything, and that’s important.
The Semi-Humanoid Robot: Wheels, Torso Movement, and Dexterity
Vibhu [00:25:02]: We have the robot. You gotta tell us about the design. Your own custom platform. What do you call this?
Jason Ma [00:25:11]: What do we call it?
Vibhu [00:25:12]: Yeah, so it’s not humanoid. It’s very smooth, but what’s the what’s the rotating arm?
Jason Ma [00:25:16]: Yeah, so we think of this as generally like a semi-humanoid. So basically, the upper body is, like a humanoid. It’s got, of course, shoulder, got arms. It also got four degrees of freedom on the waist. So you can basically do all the upper body motion that a human can do. But it’s a semi-humanoid because the lower body is wheels. All right, so why do we choose wheels? Because I think there is a difference. There’s, optimal practicality versus performance trade-off. I think with legs, the energy con— the energy consumption would be a lot higher. It would be a lot more unstable. Safety will be extremely hard to guarantee. So right now, for legged robots, there’s actually no guarantee that the robot will stay upright, balanced the entire time. I’m sure you guys have seen a lot of those, robot dancing videos. Like the robot dancing thing.
Swyx [00:26:08]: Seeing dancing, yeah. They all, they fall over very easily.
Vibhu [00:26:10]: No, but you also see the demos that you can tell are not production, where you can box the robot and you can push it over and it’s very good. But that robot is just good at balance, right? Like- there’s a.
Swyx [00:26:21]: I think the more interesting thing that you guys have innovated. So we have, one of the prototypes, which is the Mewtwo back here.
Jason Ma [00:26:29]: You can call it that.
Swyx [00:26:30]: This is just like an up-down, axis, and you’ve chosen to then move towards something like this which, has the ability to lean. And we’re gonna show the laundry demo where you have to lean into machines. Right? That’s another degree of freedom that I don’t even have the words for it. It’s not pitch, it’s not yaw. It’s something else.
Jason Ma [00:26:47]: So yeah, how did we think about that right? Like,. So actually, it’s very used for exercise. Like, I think sometimes, a couple years ago when I was exercising, I accidentally like, pulled a muscle. Right? I’m sure you guys have that experience too, right? Then the next few days you just couldn’t move, or like you pull it where you slept in the wrong position, so your neck is very stiff. We have all experienced that. And then whenever you experience that you kind of realize, wow, so many muscles, so many degrees of freedom that we use in our daily life that we take for granted. But when it’s disabled it’s very awkward for you to do some tasks that you feel are very natural, right? So that’s also how we think about when we design robots. When you have a— this kind of design, which is this purely elevator, could only go up and down, we realize a lot of the human torso movement, like we use our waist, back a lot for day-to-day physical activity, right? Even if I wanna get this bottle I’m not just moving my arms. It’s actually very strange for me if I have to intentionally think about fixing my upper body. I would always do this right? So there’s just a lot of subconscious way that humans move their body that I think will be equally useful if robot has a degree of freedom. And if you further think about, okay, we want to learn from human data. Like, humans are moving their torso all the time. You wanna be able to ingest that into the model to control the robot. But if you have a form factor that doesn’t allow for that then the transfer becomes less and the robot becomes less dexterous and it’s just more awkward.
Transferring Human Pretraining to New Robot Hardware
Vibhu [00:28:22]: What’s the data labeling side of this? Like, I— you’ve seen people,. Is this the last post-training fine-tuning side of actually the hour of using the robot tele-operated or remote-controlled mapping motion? Or does some of it generalize over?
Jason Ma [00:28:41]: When you say labeling, you mean?
Vibhu [00:28:43]: Like, taking the model you have that predicts state to mapping it to your specific hardware.
Jason Ma [00:28:50]: Oh, yeah. It generalizes very well. So for all the demos that I have shown you on the new robot, ‘cause the new robot, we, it got ready not so long ago. So we quite literally do not have a lot of data compared to the stationary arms we showed you, which had much more data, right? But a new robot, the model has not seen this in pre-training, in some of our pre-trained model. It’s able to learn the task that you saw very quickly. And I credit a lot of that to like, even before the hardware is ready, we already committed to pre-training on humans because we knew at some point our robots will be like human. You can get ahead on the training side while hardware is getting ready. And I think now we’re seeing that the transfer is actually there and is actually very helpful.
Evaluating New Skills and Learning from Real-World Feedback
Vibhu [00:29:35]: Talk about the eval side of this. So you can have predictions, you can have it do stuff, and every now and then we see it miss stuff. You test on very niche things as well. It’s like filling a pill bottle. How long does that process take? How do you think about the aspect of actually deploying on a new task, evaling it, iterating? What does that look like?
Jason Ma [00:29:56]: Yeah, so if we wanna learn a new task, we basically do a little bit of robot data collection. Then the model, these-
Swyx [00:30:04]: What is a little bit like?
Jason Ma [00:30:05]: Yeah, like couple hours. Yeah, we’re— In some of the tasks, like just tens of minutes, right? Yeah. And then-
Swyx [00:30:11]: The chopping one was 45 minutes.
Vibhu [00:30:13]: The open the fridge was on-
Swyx [00:30:14]: And that’s an interview— That’s like an onboarding task for one of your engineers.
Jason Ma [00:30:18]: So,.
Vibhu [00:30:20]: Fun thing if you wanna work here, you know. Find the task, have the robot do it.
Jason Ma [00:30:24]: Pitch me a new task and I’ll come to the office, collect some data, train a model and see what happens.
Vibhu [00:30:29]: Tie your shoes.
Jason Ma [00:30:31]: Truly, it’s tying. Yeah, that would be cool. But yeah. So, you know. A little, a small amount of data collected, train a model, then we just deploy the model in the real world. And then we look at the performance, then-
Vibhu [00:30:41]: To fill in collected means you are— you have a person actually operating-
Jason Ma [00:30:46]: Operating.
Vibhu [00:30:48]: Doing the task for a bit. And then you train? And it does okay, you improve. How do you measure?
Jason Ma [00:30:56]: So when you’re training the model, there’s, certain metrics you can look at right? Just like training a model, it converges nicely. Now, after that for robotics today, we just run the model in the real world, right, on the same setup that the data is collected to see how the model does. Sometimes the model just works great out of the box, right? You don’t really need to do much more. But when the model’s maybe like seventy percent, eighty percent there, then you see how the model behaves. You figure out what it’s not good at. Then you can collect a bit more data, then iterate, right? I think of this as the robotics version of reinforcement learning from human feedback, right? The robot rolls out. You can score it good, bad S- somewhat good. And then that data, the robot’s own experience can be used to train the model. And you can also just collect more data yourself to train a model, right? Eventually, our goal is like, this robot is gonna be our platform. Anyone should be able to train tasks on it.
Long-Horizon Tasks and the Bottlenecks to General Robotics
Vibhu [00:31:59]: I wanna get a bit of a prediction and timeline to where that goes. Right now, the tasks are, somewhat simple, like- Take laundry out. It’s a towel. You can’t crush it. Chop some food. There’s also the side of like, okay, set up this filming studio, right? Move stuff around, bring a table, set up lights. I think-
Swyx [00:32:21]: Manage my data center. And also, ref— like, recharge another robot.
Vibhu [00:32:29]: When do we see the factory or what do we need to scale up? Is it model? Is it everything? It seems like your hardware is set for that but where do we see it go to a point where I can go from instead of just cut one thing to, “Okay, here’s a recipe.”
Jason Ma [00:32:49]: I would say to get to truly general-purpose robots like basically a robot or a model that does anything, like-
Vibhu [00:32:58]: Not that. Like, okay, so to keep it somewhat grounded on a technical level, like- when do we start to see us pushing that feedback, right? So-
Jason Ma [00:33:06]: I see.
Vibhu [00:33:06]: Instead of very simple chopping- Maybe a five to ten-minute sequence, right? Where now the eval step, it can break at many more points. Instead of just fill one pill bottle- Ten different pills all in a sequence, right? What’s limiting us? What are we pushing on?
Jason Ma [00:33:20]: Yeah, I think. Yeah, I think a couple of things. One is that the laundry demo that you have seen, yeah, it’s gonna be one hour to finish a whole loop, right? So the longest duration the robot can do a full job is already in the hour range. Right? But to be able to do that for any task, I think the bottleneck is coming back to data and hardware, right? So for some of the tasks, for example, setting up the space, you might need more than a the parallel jaw gripper, right? You might need something more dexterous, like a hand, like a human hand, right? But then if you wanna train a model on the hand, then you’re bottlenecked by the hardware stability. Like, hand— Today, the best hands, dexterous hands are still not, durable enough. They overheat pretty quickly, right? So that becomes a bottleneck, right? Like, you— We— you quite literally have to put a ice bag-
Swyx [00:34:15]: Ice. To cool.
Jason Ma [00:34:16]: Human.
Vibhu [00:34:16]: We’re not just auto-researching, make better model we have. We have hardware limitations.
Jason Ma [00:34:20]: Yeah, so hardware limitation is a big thing. But then if hardware is limited, then that will slow down everything, right? Because to know that you’re making progress on any task, you eventually have to eval on the hardware, right? So then without the eval being very spontaneous, very fast, that also limits your research progress, right? Because then how do we collect the right data, right? I think, how do we collect the right data to do extremely complex, like setting up this studio, right? Because you can imagine, like today, most of the data is vision-based, right? It’s like video streams and some, encoding of your hand motion where the robot’s, joints. But Humans have a lot more sensors. We hear things. We feel things.
Swyx [00:35:05]: Smell.
Jason Ma [00:35:06]: Yeah, you smell things, right? So we subconsciously use all of those to help us be so good at manipulating the physical world. But today if we’re not capturing that at all, then even when we scale to millions of hours, you might just be missing critical dimensions of the task, so you’ll never be able to solve the task, right? So I think those are what I consider what’s dragging on the timeline. But for tasks or for workflows that you can collect data today to high fidelity, then I don’t ne— I don’t necessarily think there is a fundamental bottleneck, right? Just like what we are demonstrating, to the public this week. We have a physical agent that can do the entire laundry workflow, right, as a human would, right? Because for that task, we don’t really have like, oh, you actually need more than a parallel jaw gripper. You actually need some sensing modality that’s just not available today, right? So I do think the biggest bottleneck To have a much more general thing is still hardware and data.
Real-World Deployments: Best Western and Din Tai Fung
Swyx [00:36:06]: I want to take this chance to talk— let you brag about your customers and partners. You have some announced, relationships. We can start with Best Western and then maybe some of the others that you’re, that you’re proud of. Cool stories, anything like that.
Jason Ma [00:36:20]: So we have a deployment blog post, on our website that goes into much more detail. I really recommend watching the video because I think the video is-
Swyx [00:36:29]: Which one? The Best Western one?
Jason Ma [00:36:30]: No, we have one video for all the customers we’re working with. It’s a really nice video. It shows what we’re doing. And yeah, I think it’s great. But yeah, we have customers, Best Western. That’s in the hotel space. We fold a lot of laundry towels for them. Every hotel needs it, every gym needs it, every fitness center needs it. -
Swyx [00:36:51]: What’s the deployment so far? How— Like, where can I go to— Which Best Western do I go to see it?
Jason Ma [00:36:56]: So if you wanna go somewhere, I actually recommend going to our other deployment, Din Tai Fung. Have you guys been to-
Vibhu [00:37:04]: Very thin wrapper.
Swyx [00:37:05]: We should have filmed there.
Jason Ma [00:37:09]: Just filmed there.
Vibhu [00:37:10]: The B-roll.
Jason Ma [00:37:11]: But in Din Tai Fung if you go to Din Tai Fung or other somewhat upscale restaurant, there’s napkins on your table, right? So our robots today fold thousands of napkins for many different locations, many different Din Tai Fung locations in the United States. So that’s actually our biggest scaled customer, right? So I wanna make a distinction here. Today, there’s a lot of robotics companies that’s talking about deployments, talking about pilots. But what’s really hard is not putting your robot in one location or doing a pilot. What’s really hard is that the customers believe or validate that your robot creates enough value that they wanna deploy your robots across many of their sites. So that’s what’s happened for us with Din Tai Fung, which, to the best of my knowledge, hasn’t really happened for other companies doing similar general purpose robots. So at Din Tai Fung, our robots fold thousands of napkins per day. It’s — The quality standard is extremely high, ‘cause they want like, okay, the second fold or the seam of the napkin is only like a couple centimeters below the I don’t even know how you call it, the pocket, so you can stick the chopsticks in. So it’s actually very fine-grained, high-precision task. I can show you.
Vibhu [00:38:26]: At some level, that’s what robotics are good for robots are good for too, right? Humans are not the most consistent- Especially in small details, whereas a robot can somewhat do it perfectly. It’s interesting that’s the example you bring for general purpose robotics versus, at some level, I think people would assume- you can have a specialized robot that will fold napkins perfectly to the exact, millimeter as opposed to something like this but.
Jason Ma [00:38:53]: Yeah, I’m sure, maybe you can, right? But the hard thing about manipulating deformable object, right, like napkin, towels, everything’s- Deformable. There’s just infinite- infinitely many possible configuration. So that’s really hard for a specialized robot, right? And what we find is that by developing a general purpose model in a robot and adapting it to these use cases, you can actually get to a production quality faster. And what’s really nice is that now that Din Tai Fung has scaled, has signed a much bigger contract with us, they not only want our robots to fold napkins. Now they want our robots cut. That’s why we had the- Example, to cut food to the-
Swyx [00:39:31]: Is that actually the second use case?
Jason Ma [00:39:33]: That’s one of the use case they are thinking. Din Tai Fung needs to cut so many vegetables per day.
Swyx [00:39:38]: They need to do the dumplings.
Jason Ma [00:39:39]: Exactly. To the fold. That’s the final boss. That’s-
Vibhu [00:39:41]: That’s the final one.
Swyx [00:39:43]: I think you can. I think you have the dexterity and the precision to do this.
Vibhu [00:39:46]: You have a little robot.
Swyx [00:39:47]: But no, don’t break the dumplings.
Vibhu [00:39:50]: That’s the obvious benefit is you can adapt them anymore.
Generalist Robotics, Speed, and Consistency at Scale
Vibhu [00:39:54]: I think the other thing you really touch on is the speed, right? To make a custom napkin folding with the exact ply, whatever- you have to make the machine. You can’t just Amazon this. But this can do it pretty fast, right?
Jason Ma [00:40:07]: Yeah, this can do it ex— very fast. And now it’s actually al— it’s at human parity. And this robot doesn’t get tired, doesn’t need to you know. You can. And I think I really like the fact that you mentioned consistency, right? Like, one issue with Din Tai Fung is like, their LA location with New York location, Seattle location, the they fold napkins slightly differently, but from their perspective, they want consistency. But that’s exactly what these models are good at, ‘cause models are, just, it’s a set of weights. You can just put the weights in the different robots across different-
Vibhu [00:40:43]: You can OTA update them as well.
Jason Ma [00:40:45]: Yeah, you can just update all of them at once. It just. Once it works, I think it’s truly, like- Incredible.
Vibhu [00:40:52]: Final boss is Cheesecake Factory. If the story there, they’re from scratch. Everything is very consistent, down to the measured gram. Every sauce is made. Like, anywhere you go in the world, you’ll have the exact same thing. If you don’t know, you don’t know. But that’s the that’s the final.
The Bitter Lesson of Robotics: Scale the Right Data
Swyx [00:41:07]: The term that comes to mind is kind of like bitter lesson for robotics, right? You, instead of the specialized robots, which is a lot of industrial engineering, you build a generalized platform that can- Do a last-mile fine-tuning and-
Vibhu [00:41:18]: That’s right.
Swyx [00:41:19]: And do all those things.
Vibhu [00:41:20]: Give the expert add-on outside of your. What is more bitter lesson robotics?
Jason Ma [00:41:25]: Yeah, I think there’s several bitter lessons for robotics. One is that robotics is going to scale just like any other domains that machine learning has solved, right? So what does that mean? That means having as much data as possible. But in robotics, the one thing that’s a lot harder is what is the right data is much harder to figure out than other domains, right? Language is text. In video games, just video game data. But for robotics what sensors you choose determine what kind of data you have, right? But if you have the right set of sensors and just scale that data, I think you’ll get predictable return on model performance. So in that way, robotics isn’t so different. So that’s the bitter lesson. As a corollary of that is that like there’s a lot of research in robotics that’s coming out with very specific priors. Like, for example, we need to make the robot see objects as they are, draw some boxes. That will give robot better perception, right? But I think the bitter lesson here is that just keep the algorithm simpler and scale the right kind of data. Then generalization, better behavior will emerge. So that’s not so different. And the second. I think that’s the main bitter lesson, which is at the end of the day, scaling will work, but in robotics, to scale and how to scale is just much harder ‘cause there is no internet. You have to come up with your hardware, come up with the sensors, get all of those right, and then scale. Then I think,.
Humanoids, Practical Iteration, and the Path to Generality
Vibhu [00:42:53]: There’s the argument that the world is very made for humans. It should be a human, right? As hard of a hardware problem that is, think ten, twenty, a hundred, two hundred years down the line, can we make a really efficient human? But you have to be practical with what we have now, right?
Jason Ma [00:43:12]: So the way I think about it is that yes, eventually the robot should. If we wanna build a general-purpose robot, right, like a machine that can be adapted to as many use cases as possible, I think human form factor will be the best one, just because that’s what we have been doing, for ages, right? But I think the best way to actually build a humanoid robot is to take an iterative approach. Because robotic is so hard, the whole problem, the whole stack is so long, it’s really hard to know a priori what’s the right decision to make, right? And because hardware decision is much more expensive and much more costly, just ‘cause hardware iteration is a lot slower, you actually want to take an incremental iterative approach, not going all the way to the final form factor. Because the chances are the thing you build actually doesn’t last. It’s not good enough. So you just end up scratching it anyway, you know. Like, it’s public news Optimus, already gone through many iterations, and they had to scratch their designs, right? And I think by iterating with a semi-humanoid form factor and actually putting them out to real customers or in getting models really deployed, you get that feedback loop that you wouldn’t get if you’re just trying to build like, warehouse humanoid and you spend like five, ten years building it. I think the iterative approach will actually get us closer to the final robot faster.
Vibhu [00:44:33]: And I will say, for those listening and not watching, it’s probably the smoothest robot we’ve seen. Like-
Continuous Action Spaces and the Importance of Context
Vibhu [00:44:38]: … every humanoid you’ve seen is jerky and even humans don’t walk as smooth as this thing.
Swyx [00:44:43]: Because, yeah, the model is. Can move every single motor at once.
Vibhu [00:44:48]: In more than two cohort, yeah.
Jason Ma [00:44:50]: Yeah, and that’s actually a very hard problem, because if you just think about the model, it’s outputting action. There’s the action, dimension, right? The bigger the dimension is, the harder to fit, the model to the data. But here, the action space is quite large, but we did very good job to make sure the model fits well and gives you a smooth-
Swyx [00:45:10]: Can you bound the order of magnitude of the action space, or is that-
Jason Ma [00:45:14]: Yeah, so it’s on the order of, about a hundred action dimensions.
Swyx [00:45:19]: Yeah, and these are all continuou- continuous, right?
Jason Ma [00:45:21]: Continuous, right. So the model is just basically predicting at very high frequency, like tens, like at least fifty-dimensional vector, like ten, twenty times a second to get that continuous motion.
Swyx [00:45:34]: And potentially predicting one to two seconds out, so that is another hundred frames or whatever.
Jason Ma [00:45:39]: Of course.
Vibhu [00:45:40]: There’s a bit of a history, similar to context of previous state, I guess.
Jason Ma [00:45:45]: So context is a very important thing in robotics, right? Because otherwise, if you’re just making decision based on the current observation, sometimes it’s really hard to tell what to do next, right? For example if you think about opening a microwave, like in the middle part, it’s unclear whether or not you’re-
Vibhu [00:46:01]: Closing
Jason Ma [00:46:01]: … closing or trying to open it, right? So context is also really important.
Swyx [00:46:05]: And I have a screenshot of the fridge demo where it is, there’s agentic planning and tool holding and all those things. So we can show that for listeners.
Vibhu [00:46:13]: I think that’s an interesting side. After the model-
Agentic Harnesses and Planning Physical Workflows
Vibhu [00:46:16]: … how do you wrap it around into agentic harness? Like, what is-
Swyx [00:46:22]: Claude Code
Vibhu [00:46:22]: … Claude Code deployment? Claude Code UI. You have a lot of cool UIs, but what goes into wrapping model output towards final thing? Is this custom per use case? Is it you have your own SDK?
Jason Ma [00:46:37]: So, eventually we will have our own SDK, but the way we think about is we want a general harness layer for robotics foundation model. Today, that is our own model, right? The way I think about is for any Real task, long task in the physical world, there is a lot of a high-level decision-making you have to do, right? It’s not just like robot, only reasoning about the physical world in the action, continuous trajectories, right? Because that’s a very low level, the thing you’re actually executing in the physical world, but that’s not the intent. But like if. When we are doing decision-making, we think about steps of a recipe. We think about, oh I finished this I need to do that. While I’m waiting for this part to finish, I need to go somewhere else to do something else. There’s a lot of high-level cognition, decision-making that’s happening. And I think for robots to handle even, the laundry task we are showing you, it has to have that capability. Because, it needs to reason that okay, the washer is running, so I should go do something else while the washer’s finishing. But now the washer is done, I should take, I should take the towels from the washer and put it in dryer immediately, right? Because if not, then that becomes a bottleneck in the whole workflow, right? So there is a lot of reasoning that needs to happen for robotics. And I think in theory, you could just, try to learn that with a low-level model that’s outputting actions, but it’ll be extremely inefficient. So that is why having a hierarchical system, like putting a harness, like an agentic system around our world action model is what allow the model to compose different capabilities, different skills to solve a much longer horizon task. It’s very much like in language model, right? Like two years ago, three years ago, the model was only good at autocomplete. Like, humans have to tap to get the model to keep spitting out code. But today, just an agentic system just carries the end-to-end task. And that’s. I, and I think that’s important because I think today in robotics, people are still thinking about autonomy at the very task level like one minute, thirty-second task. But really, the value is about tasks or workflows that go on unsupervised for hours. And that’s what we’re building towards. And that’s why I think thinking about. The robotics problem, not just as a modeling problem, but rather as a system problem, you need good agent, good harness engineer, good model, good hardware, everything. It’s like how we think general-purpose robots ought to be solved.
How Coding Agents Accelerate Robotics Research
Vibhu [00:49:07]: Do you guys see benefits from language model coding tools getting better? So new Opus 5.5, new Astra. Do you have language models in your agent harness? How does AGI being developed in language model space affect you guys?
Jason Ma [00:49:24]: No. Yeah, no, absolutely. Like, it affects us very deeply in many ways. One is that we are extremely AI-native company, right? Like, coding agents, all of that is a very fundamental part of our work, and we’re very welcoming of that. Like, whenever there’s a new model, we’re the first one to test it, right? Like, we being. Like, when GPT-6 came out, we had a company-wide hackathon the same week, and the thing that our engineers built was incredible. And many of them are— Many of the features we built is already in main, in production, right? So we really like using coding agents to accelerate our work.
Swyx [00:50:00]: Just for the people interested, anything surprising in GPT-6 that you found?
Jason Ma [00:50:05]: … it’s about the ability for it to construct virtual environments, right? Turns dissemination. We talked that twenty minutes ago. I think that’s really exciting.
Vibhu [00:50:17]: It’s interesting. There were so many CAD, Zillow to 3D-ish.
Swyx [00:50:21]: He lost the hackathon to the CAD, -
Vibhu [00:50:24]: I won the. I did the CAD b******t. Other way.
Swyx [00:50:27]: No. Oh.
Vibhu [00:50:28]: That was. I won a GPT hackathon just for fun. I was like, “I know nothing CAD.” Might as well play around with it ‘cause every demo is doing it. I’m like, “It’s gonna be garbage.” And I’m like, “Well, it’s kinda grounded, it’s kinda good.” But it’s just hackathon stuff.
Deployment Milestones and Competing Robotics Strategies
Vibhu [00:50:45]:.
Swyx [00:50:45]: Cool. I wa— I was gonna round out, some of the use cases and then go into a little bit of company culture and your personal background.
Vibhu [00:50:52]: I have two technical questions. One is, are there any goals you wanna set for yourself, like six months, one year, two year timeline that you see robots should be able to do, fill in the blank, that you’re kinda working towards?
Jason Ma [00:51:09]: Yeah, in the six months to one year frame, our goal is definitely scaling up our robots and our model and truly getting them to deploy-
Vibhu [00:51:18]: Like a task.
Jason Ma [00:51:18]: … completely different.
Vibhu [00:51:19]: Like, right now it’s cutting or opening a fridge. Is there. Like, do you set yourself to by this timeline, we should, to stay on track, be able to do whatever?
Jason Ma [00:51:29]: I think, in terms of deployments, we’re exploring many different industries. There isn’t a particular task we think we have to nail, right? Because our goal is to build a general model, a general platform, and we think of each different task as validation of our platform as a way, as a wedge for us to figure out how to build everything better. So today, we are doing a lot of that in laundry, in restaurants. We have already deployed tens of robots, but our goal next year is overall just deploying hundreds of robots.
Vibhu [00:52:02]: My part two on the last technical question is, how do you see the rest of the industry? So fill in for people, what other approaches are people taking to world models? Why are they wrong? What is different but good about their approaches as well? Just what is the landscape of robotic world models in this space and why— how have you come to your current?
Jason Ma [00:52:25]: No, that’s a great question. So I think robotics is the most exciting space today, and there are just so many, amazing companies that I truly respect. They’ve been doing great frontier work like us, and it’s very exciting. And I think what we are seeing today is that there’s still many different approaches to robotics. Like, we believe that hardware needs to be built in-house. We need to own the whole, chain. We believe that scaling human data is extremely important. But there are companies that’s betting on just scaling robots’ data or scaling some sort of intermediate data collection devices, right? All of them are in play, and I think it’s different approaches. We’ll just see what happens. I think right now, it’s still a bit early to tell, but we feel pretty good about the way we’re building things just because we as a company, culture-wise, mission-wise, we’re not here to just build a model. We’re here to truly get robots deployed. So we deploy as much as possible, work on the hard problems. So we identify problems much earlier on. So then I think that really helps us identify what the true bottlenecks are and then solve them. And I think for folks, for people who are only working on one part of the problem, it’s kind of like fog of war. You don’t really see the whole picture, and you might be building something that’s extremely cool technologically, but may not be what people need or want at the end, right? I think there’s a whole graveyard of robotics company, and I think all of them had very shiny, very cool robots, but just figuring out what people actually want is actually. It’s very easy to say “Build something people want.” everyone says that but it’s actually really hard to execute a robotics company.
World Action Models vs. Vision-Language-Action Models
Vibhu [00:54:07]: On the technical side, you have your world action model. Is that standard across industry? Is there other directions people are taking?
Jason Ma [00:54:15]: Yeah, I would say that’s also not standard. So we’re building world action model on top of world model. So there is many companies building world model. You mentioned Runway. They’re also getting to adapt-
Vibhu [00:54:26]: Video world models.
Jason Ma [00:54:27]: There are also companies building a vision language action model.
Vibhu [00:54:31]: VLA.
Jason Ma [00:54:31]: So that’s like taking vision language models and adapting them to robotics. So I think there’s many different ways to go about it. But I think at the end of the day what we call the models is less important than what we believe the model needs to be able to do at a first principle level. So I wanna explain that a bit. I think for robotics, the most important thing is that you need to be able to predict the future. Right. Like robotics, any physical system is fundamentally about based on the information you have at the present moment and before, what should you do next? So it’s really about predicting what comes next. So predicting robot actions is like predicting what the robot should do next. But if you think about it, robot is just a part of the world, right? Robot influences the world, but the world also influences how the robot should behave in that environment. So if you think about it this way, then the maximal, the most first principle thing you can do is just predict as much of the world that you can possibly predict, right? So that’s why we believe predicting the future is extremely important. And today, the instantiation is world action model. And VLA only predicts robot actions or some more abstract representation of the world. But whenever you’re predicting abstract representation of the world, you’re encoding your own bias into what’s important to predict. And the bigger lesson tells you, let’s remove human biases. Let’s just take the most raw representation, the camera inputs, and just predict all of that right? So that’s how I think about, modeling for robotics today.
Swyx [00:56:12]: So just to press you a little bit on this you’re, basically you’re saying your approach is better than the VLA, right? Because you’re emphasizing prediction. I fully buy that. I— Like, you’ve sold me on it. If you were wrong, what is the most likely number two-
Beyond Vision: Tactile, Audio, and Infrared Sensing
Swyx [00:56:26]: Someone else is doing?
Vibhu [00:56:27]: Interesting question.
Jason Ma [00:56:28]: So yeah, there’s a caveat in what I’ve said before, and I think that’s where I could be wrong, right? I said that you need to predict the world as much as possible, right? But what that doesn’t say is that does your sensor have everything about the world that you need to predict? I think I touched on this a little bit, right? Like, if you’re training a world model that’s purely based on videos, I think it will be a very powerful world model, but it may not be a sufficiently rich enough world model because you’re missing critical aspects of the world, like sound, like touch, right? So I think where we could be wrong is like, okay, I have the best world model on vision-based world model, but tomorrow someone else comes out with a vision plus tactile world model that’s just like a hundred X more data efficient just because, vision is a very high information or a highly redundant, input because, every other pixel, they don’t change that much. You can see— you can understand the world without perfect vision. But tactile, other modality might be much denser in information, right? So maybe having that predicting that would just allow you to use much less vision. So you can get to the same performance with much less compute, with much less data, right? But-
Swyx [00:57:51]: I think you will be able to adapt very easily.
Jason Ma [00:57:53]: No, exactly.
Vibhu [00:57:53]: There’s also a level of how much data.
Jason Ma [00:57:56]: Right. So the reason why I bring this up is because I think if a vision-tactile world model or vision-tactile-audio, whatever world model, is the final thing that solves robotics, then I think robotics companies will be able to do that better than video generation companies that’s trying to pivot to robotics. The reason is to understand sensing is actually very complicated. Like, we’ve been building that since the company started, so we know the bottleneck. We know how to combine them. But for— if you’re just trying to adapt a video model to robotics action, then if we were wrong, if, video model is not enough, then you don’t really have the know-how to get tactile to work very well. But we have that.
Swyx [00:58:39]: This is a side question, but I think it’s a quick one. Heat seems very important. Is infrared just another color to you, or is that a.
Jason Ma [00:58:47]: Yeah, it could be another-
Swyx [00:58:48]: Just another color.
Jason Ma [00:58:48]: It could be another color, another input to the model. Could be used in cooking scenario, actually. Like, I know companies that’s using infrared to know how cooked food is. Because that’s really hard to tell from RGB camera, right?
Vibhu [00:59:01]: My very naive pushback on your last point is maybe we don’t need to understand hardware, cloud code, or Codex can just, vibe code it in. There’s a lot you can do without understanding, but of course-
Swyx [00:59:15]: It’s just not in the distribution.
Jason Ma’s Journey: AlphaGo, Penn, NVIDIA, DeepMind, and Dyna
Swyx [00:59:16]: So I wanted to bring it a little bit to your personal story or your background. We both went to Penn.
Jason Ma [00:59:24]: I just found out today. Crazy.
Swyx [00:59:26]: Yes. I, my big regret is I didn’t join the self-drive, the Docker challenge. Which, and then I could have been early Waymo or something. You were at Grasp. And then you’ve also worked at Google and NVIDIA. What are the different focuses of let’s say, academia versus Google versus NVIDIA? You’ve seen all these places, right? And then that obviously caused you to say “Screw all that. I’m gonna do my own thing,” right? You could have joined OpenAI’s robotics lab. So may-just take us through chronologically, but also one thing you learned from each.
Jason Ma [00:59:58]: So chronologically, yeah, maybe it’s useful to talk about how I got into AI in the beginning, right? So, as a kid, so I spent time in Japan and China. I grew up playing Go, the Go in AlphaGo. So when AlphaGo came out, I was a senior in high school, and I was mind-blown because I’ve been playing this game for many years. It’s so complicated. Like, I think before AlphaGo came out, people were like, “Okay, Go is one of the game that AI will never solve.” The search space is larger than the total number of atoms in the world. How could you possibly search for solutions in that? Then RL, reinforcement learning just cracked it, right? And I think that was how I got into AI initially. Like, I was just really interested in models or systems, agents that can self-improve, right? So when I got to Penn as a PhD student, I just really wanted to work on reinforcement learning. So I’m co-advised. Both my advisors were interested in RL at the time. One of my advisor was more in robotics. He did a postdoc at Berkeley Robotics Lab. So that’s how I got into robots. At Penn, we have GRASP Lab, I think one of the best places to do robotics. And so that’s how I got into doing RL for robotics in my PhD. And then, yeah, I did a bunch of research thinking about how to connect reinforcement learning, with human data, how to do reinforcement learning, theory in a very rigorous way. And then I went to NVIDIA. That’s when GPT-4 came out. At the time, we were very interested in just solving a pen spinning task. I don’t know why. Like, -
Swyx [01:01:35]: It’s the dexter- it’s the dexterity limit, right? That you can solve Rubik’s cubes, and then you-
Jason Ma [01:01:39]: At the time, I was working with, -
Swyx [01:01:41]: Jinpeng
Jason Ma [01:01:41]: … Jinpeng at UCO, which you guys probably also know. Like, yeah, we were just like, “Okay, let’s try to do dexterous pen spinning.” The initial idea was learning it from human videos, right? We were trying to watch humans do it. But then I realized, okay, today we don’t have a dexterous hand that can spin pencil at all. So how would I even be able to evaluate the algorithm, even if learning from human worked at all? So I thought, “Okay, we need to first try this in simulation.” Then as soon as I tried to set up a simulated environment for a robot hand with a pen, I immediately saw what the issue is. The issue is like, if I had to come up with this simulation environment, the reward function for reinforcement learning to solve pen spinning, that’s impossible. Like, I wouldn’t be able to come up with all the terms that you need to get RL to learn the right behavior. And then I was just thinking, “What if a model that can write code can do that?” That’s what became the Eureka project, right? We basically. Like, in my opinion, we had the first recursive self-improving robotic agent in simulation, ‘cause we really used that to solve a new task that no one has demonstrated before. And that allowed me to see the potential of using language model, using coding models to really accelerate robotics. So then I went to Google DeepMind, ‘cause at the time, even today, they have one of the frontier models in Gemini, right? So I was looking at how to use Gemini for robotics. But eventually I realized like, I had a lot of fun doing academic research, but there was always a question in the back of my mind “Does any of this matter?” like, when you’re working on this kind of thing in a lab you ask me, “How do we come up with so many interesting, fun tasks?” You’re basically doing your own eval. You’re grading your own homework. Like, I wanted something more real. I wanted something that could actually be used by millions of people in the world. And I felt twenty-twenty-four was when we started Dyna, like end of twenty-twenty-four. That was the right time that we can take some of these technology to actually putting them out in the real world. So that’s why
Jason Ma [01:03:45]: I started Dyna with Lindon Gao and York Yang. And fast-forward two years, we built our platform, our model. Our robots are in Din Tai Fung. Go check it out and, -
Vibhu [01:03:55]: Go eat.
Jason Ma [01:03:56]: Yeah, go eat. It’s been a journey. And,. But what still gets me really excited is, like some of— a lot of the ideas that I temporarily abandoned when we started Dyna, because, to build a company, you have to be very practical. Like, I think simulation wasn’t ready in twenty-twenty-four. But now with like the new frontier models and what we’re seeing internally this might change. And that’s very exciting.
Product-Led Research, Experimental Evals, and Company Culture
Swyx [01:04:20]: And so then also let’s talk about company culture. What principles you’re building the company with. Obviously the— you’re responsible for the science and the research, agenda. And it just seems like when we meet the engineers, they are, they’re very engaged and then they have a lot of ownership. They have a lot of motivations.
Vibhu [01:04:38]: I’ll say it’s interesting to see how many side projects people just work on. So some of the demos we’ve seen, it’s just the engineers that work on it were like “Yeah, this seems fun. Might as well do it.” To your point on fun evals, right here, like two feet behind the camera-
Swyx [01:04:55]: Can’t talk about that.
Vibhu [01:04:57]: … just the e— the eval is just there’s highlighters, four highlighters being rubber banded together, right? It’s like, who comes up with, “ yes, we need to test putting rubber bands on high.” It’s just-
Swyx [01:05:07]: Fissionary-ish
Vibhu [01:05:08]: … fun stuff. People are very … creative here. It’s a beautiful space.
Jason Ma [01:05:12]: Yeah, I think culture. Yeah, so let’s first talk about, how we run a company, right? Like, we think of ourself as like a like a eventually like a research plus product company. It’s kind of like OpenAI where you do frontier research, but eventually all of them make their way into Codex through ChatGPT. That’s how we think about it too. We’re not here to just do open-ended whatever thing that interests us at the time. We’re always product-driven. Like, we wanna deploy our robots, our platform to a new customer. What are the technical problems we have to solve, right? We have to solve generalization. We have to solve whole body control. We have to solve how to adapt quickly. We have to solve how to make the robots follow language. We have to solve dexterity. So once all those problems are very clear, then, like. Our engineers are very capable. They’re very smart. So they just create instances of the task. They create tasks that will stress test those capabilities, right? Like the j— the grabbing a drink task is just how we test language following, right? You can just specify what you want. If the model follows intent, follows instruction, it should be able to do it. So that’s how we think about, like. At the top level, we identify what are the things we have to solve. Then we have a lot of bottom sub-project to actually drive execution to solve them. And you hire the right people. And everyone here is super passionate about robots. They’ve been doing this for a long time. They love it. And we just create a culture where it just naturally happens. And it’s a really fun place to work.
The Team: Hardware, Software, Research, and Learning
Vibhu [01:06:44]: What’s the split between technical research software versus hardware side of the company?
Jason Ma [01:06:51]: We recently grew so much, so I don’t have the perfect number. But I would say on the hardware side, we have, I would say probably like thirty percent, forty percent of hardware, just because hardware you need a lot more people. And then we have around ten people working on models. And then we have software engineers, robotics engineer. We have a lot of very talented people on the business side. The most fun thing about building a robotics company is you see everything. Everything under the sun that could possibly be helpful will be helpful, right? And so you just get so much exposure working with so many different people from many different backgrounds who are all very passionate about robots. And I think it’s been really eye-opening for me as a founder. ‘Cause when I was a researcher at a lab or a PhD student, I was talking to researchers every day, and that was great. That was really good. I still get to do that every day, but now I also get to learn just from so many different people from different walks of life. Like, some of our people who collect data, they had no robotics experience. They were working in retail, working in some other jobs, and they came here, you know. They also became very well-trained in robotics, and I think that’s just pretty amazing to see. And I think of myself as I’m still learning, you know. I think of myself as just always learning in the robotics space, and I haven’t learned so much as I am now. Even though, you think about me now, I’m not a student anymore. I’m here running a company, managing a lot of things, but I still feel like the day-to-day learning is just so much.
Dyna’s Vision and the Origin of Its Name
Swyx [01:08:29]: And then just to close off, where’s Dyna going? Two years, five years, what’s, the vision?
Jason Ma [01:08:37]: Yeah, I think our vision is to truly— The high-level mission is truly bring robots and intelligence, right, into the physical world. Like, actua— That means actually seeing our robots deployed thousands of customers, even more, millions of robots. That’s where we wanna get to.
Vibhu [01:08:58]: What’s the what’s the name from Dyna? Dynamic? Is there a is there a story?
Jason Ma [01:09:03]: Yeah, there’s many stories. So I na— I named Dyna. There were a lot of internal options.
Vibhu [01:09:09]: Take the credit.
Jason Ma [01:09:11]: … internal, different options. But how the name came to be was, there are two things. One is so, Turing Award winner, Richard Sutton, who was the pioneer of reinforcement learning, wrote a paper called Dyna many years ago. It’s about basically an agent that self-improves, world model. And I felt even when we started the company we weren’t doing world model yet, but I felt that’s how robotics will get solved. And now our technology direction is converging to that. So I felt that was a really neat inspiration, also because I worked in reinforcement learning, I have a lot of respect for Richard. The second thing is that it’s a bit more personal. Like, I grew up in Japan, and the year I was born, there was just a cartoon show. I don’t know if you guys have seen Ultraman.
Swyx [01:10:00]: Yeah, of course.
Jason Ma [01:10:01]: The yeah, the Ultraman that year was called Dyna. So, yeah, it’s a very subtle thing not people know, but I guess now everyone would-
Vibhu [01:10:10]: The team is finding out too.
Swyx [01:10:13]: But I think Richard would appreciate it. I don’t know if you ever talked to him, but-
Jason Ma [01:10:16]: No, I’ve seen him at RL conferences.
Swyx [01:10:18]: You should just be like, “Hey, this company-”
Jason Ma [01:10:20]: Yeah, I’ll invite him, come-
Swyx [01:10:21]: Inspired by you.
Vibhu [01:10:22]: Everyone knows Richard for a better lesson, but you know.
Jason Ma [01:10:24]: Yeah, also better lesson.
Swyx [01:10:27]: Okay, any parting thoughts or calls to action?
Closing: Deploying Real Robots and Building the Future
Jason Ma [01:10:31]: Calls to action. Yeah, I think I do feel like as an industry, robotics is still very demo-oriented. Like, I have done a lot of demos in the past in the lab, even here we had to show some demos. But I think we need more people that’s truly, mission-focused on doing whatever it takes to get robots deployed, get, robotic foundation models truly integrated with hardware and seeing them at real customer sites. I think we’re a pioneer on this regard, and I wanna see more in this field. And for the audience that’s watching, if this mission inspires you, if, solving the hardest problems today is what makes you wake up every day and you wanna contribute, we’re always hiring and yeah, looking forward to that.
Swyx [01:11:14]: Bay Area? Anywhere else?
Jason Ma [01:11:16]: Yes, just Bay Area today.
Swyx [01:11:18]: Bay Area.
Jason Ma [01:11:18]: But as company grows, never say never to other locations. Yeah, but that’s what I wanna say, and I wanted to see more robots in the world.
Swyx [01:11:27]: Cool. Congrats on everything. This is incredible progress for two years. Yeah. Amazing.
Jason Ma [01:11:34]: Yeah, I really appreciate it. Thank you for visiting us. I hope it’s been fun.
Swyx [01:11:40]: I don’t regret doing this on site. We do a lot of interviews in our studio, but this was worth it.
Jason Ma [01:11:48]: I think for robotics, gotta do it on site.
Vibhu [01:11:51]: For people listening in, watch the video version. We have fun B-roll.


This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.latent.space/subscribe

From the Bitter Lesson of AI scaling to the unsolved mysteries of protein folding, Google DeepMind’s Pushmeet Kohli and Biohub’s Sal Candido are rethinking what it takes to build AI that truly understands biology. In this special panel moderated by Brandon Anderson, they explore why AlphaFold’s breakthrough was only the beginning, why scaling compute and data alone won’t solve biology, and how the next generation of AI models could transform our understanding of proteins, cells, and human disease.
We go deep on the future of AI-driven biology: finding scaling laws in biological data, the tradeoffs between scientific intuition and general-purpose architectures, why protein structure prediction is far from solved, and what it would take to build predictive models of living systems. Pushmeet reflects on the lessons behind AlphaFold, the limits of human interpretability, and whether future frontier models could understand other AI systems better than we can. Sal explains why protein language models may already contain scientific knowledge we haven’t unlocked, how biological modeling must move beyond individual proteins, and why achieving Biohub’s mission to cure all disease requires thinking in terms of 10x breakthroughs rather than incremental improvements.
We discuss:
* The Bitter Lesson for biology: why scaling compute and data isn’t enough
* Why finding the right scaling law matters more than blindly increasing model size
* How low-quality metagenomic data can improve protein language models
* Why AI researchers optimize for available data instead of the most important scientific problems
* Lessons from DeepMind on balancing modeling, data generation, and scientific expertise
* Why building a virtual cell requires fundamentally different datasets
* AlphaFold’s handcrafted architecture and the role of scientific intuition
* Why good data matters more than simply having more data
* Inductive biases, scaling laws, and the future of specialized AI architectures
* Why we aren’t in a post-Transformer world, but architectures are evolving
* Why AlphaFold didn’t actually solve all of protein folding
* Protein dynamics, disorder, and the limitations of static structure prediction
* How cryo-EM micrographs could unlock richer biological representations
* Moving from models of individual proteins to whole biological systems
* Feynman’s famous principle and why AI can now create things we don’t understand
* The hidden biological knowledge inside protein language models
* Why trustworthiness and uncertainty calibration matter more than full interpretability
* Whether frontier AI models could interpret other AI systems better than humans
* When AI could deliver 10x–100x acceleration in drug discovery
* Why curing all disease requires thinking about 10x breakthroughs instead of 10% improvements
Pushmeet Kohli — Google DeepMind
* X: https://x.com/pushmeet
* LinkedIn: https://www.linkedin.com/in/pushmeet-kohli-4838994/
Sal Candido — Biohub
* Biohub: https://biohub.org/team/salvatore-candido/
* X: https://x.com/salcandido
* LinkedIn: https://www.linkedin.com/in/salcandido/
Brandon Anderson — Moderator
* LinkedIn: https://www.linkedin.com/in/brandon--anderson
Timestamps
00:00:00 Introduction: The Bitter Lesson for Biological Data
00:01:00 Finding Scaling Laws and the Right Data for Biology
00:04:54 DeepMind’s Bitter Lesson: Solving Problems vs. Scaling Models
00:06:50 AlphaFold, Data Limitations, and Building the Virtual Cell
00:09:52 Handcrafted Architectures vs. Scaling Compute
00:11:53 Good Data, Inductive Bias, and Model Design
00:13:39 Beyond Transformers: The Future of AI Architectures
00:14:38 Why AlphaFold Hasn’t Solved Protein Folding
00:17:39 Protein Dynamics, Design, and Cryo-EM
00:19:09 From Individual Proteins to Whole Biological Systems
00:20:43 Feynman’s Principle: Creating Without Understanding
00:22:18 The Hidden Knowledge Inside Protein Language Models
00:23:48 AlphaFold, Trustworthiness, and Interpretability
00:25:56 Could AI Understand Other AI Models Better Than Humans?
00:27:43 When Will AI Revolutionize Drug Discovery?
00:29:47 Why Curing All Disease Requires 10x Breakthroughs
Transcript
Introduction: Is There a Bitter Lesson for Data?
Brandon Anderson [00:00:04]: Great to be here. What an exciting morning. So many cool announcements. I think the future of bioscience is being announced right here. This is the modeling session. We’re all modelers, so of course it’s natural for us to talk about data.
One of the things I like to think about when it comes to data is how to scale it properly. This has brought me to the question of the bitter lesson, but recast in the frame of data. The bitter lesson, for those of you who are not AI people, is the statement that methods that scale win eventually. If you can scale enough, it wins. So my question, starting with Sal, is: Is there a bitter lesson for data?
Sal Candido [00:01:00]: For sure. You obviously need the right data in order for it to work. One misconception of scaling laws is that scaling laws are everywhere and they always exist. A lot of the work is actually finding that scaling law. A lot of what we do is trying to figure out: What’s a situation where, if you put more compute into it, if you put more data into it, you get a better result out?
That’s a great situation because, once that happens, you can turn the crank. It becomes an engineering problem, which is something I like. That has to do with architecture, but it also very much has to do with data. If you don’t have data with the right information and statistics to solve the problem you want, you’re not going to get a model that has the capabilities and understanding you want. You can only really pull information from the data you have and use that to generalize beyond it.
Choosing the Right Data: Availability vs. Scientific Impact
Brandon Anderson [00:02:09]: When it comes to data collection, how do you think about which modalities are best? A related question: Do modelers tend to work on problems where the data is available, rather than the problems that best serve their goals or have the greatest impact on translational medicine?
Sal Candido [00:02:32]: For sure. I won’t speak for all modelers, but I’m lazy, so I’m going to work with what’s available. There’s a good side and a bad side to this.
The good side is that, when you’re doing conventional machine learning, you’re often looking for the most pristine, high-quality data examples you can find. But if you’re like me, you’re rooting around in the back room, looking through people’s junk to see what’s there. A concrete example is training a protein language model on metagenomic sequences, which are not the highest-quality data. In fact, much of that data, I can guarantee you, isn’t even a real, whole protein. And yet it makes the performance of the model go up for designing real proteins that work and understanding proteins that we know exist.
That’s the positive side. But it can lead you to a negative place where you say, “Let’s just scale up the data that we can generate easily.” I don’t think that’s necessarily the way to do it. That’s one of the things that’s exciting to me about what we’re talking about here today with the BBI. It’s going out and asking, “What is the data that we need to solve the problem?”
It’s about the right resources, but also the right community. One thing we try to do at Biohub is work in the open, work with the community, and move the whole community forward. That’s critical because, if you just have people building models, we’re going to lean toward the data that exists. If you just have people generating data, they’re going to lean toward the things that can be generated. But if you can work together as a community, every step of the way, in an open fashion, you can figure out what data you actually need. Then you can find that scaling law.
The Problem Comes First: Modeling, Data, and Expertise
Brandon Anderson [00:04:50]: Pushmeet, what do you think about the bitter lesson for data?
Pushmeet Kohli [00:04:54]: I was at DeepMind when Rich was with us and thinking about this idea of the bitter lesson. What I took from Rich’s original lecture at DeepMind wasn’t about the specific notion of whether data is useful or how we should think about machine learning. My take was that he was talking about something more conceptual.
Sometimes when we’re looking at problems, we think about solutions in a very religious way: “I’m a modeler,” or “I’m a data generation person.” I think that is the bitter lesson. If you approach a problem with that mindset, you might not succeed. The problem comes first, and you should be flexible in your solution space. You should try both things. You should understand the problem you’re trying to solve. If it requires modeling effort, do the modeling effort. If it requires collecting more data, then collect data.
What happened in machine learning at the time was people saying, “We’re machine learning researchers. The dataset is there. Here’s some training data, here’s some test data, and we’ll just optimize the model.” That is broken in the sense that, if your eventual goal is to solve the problem, you have to look at both aspects of what goes into the process.
And what goes into the process is not just data or modeling. It’s also expertise. With AlphaFold, for example, we looked at what was possible with existing datasets because we didn’t have the core expertise, or even the resources, to say, “Let’s augment the PDB by a significant order of magnitude.” The investment that organizations and scientists across the world had put into constructing that data was invaluable. So there, you had to focus on getting the biggest bang for your buck by investing in modeling.
But in other areas, say cell genomics, we took the same approach and asked, “What can you do with cell-by-gene data?” After a lot of work, it was very clear that the data was not there yet to pursue that grand ambition of building the virtual cell. This brings people together to focus on the actual challenges of advancing science rather than religiously following advances in data generation or modeling.
Brandon Anderson [00:08:23]: I really like that answer. What’s the actionable takeaway? Always define the problem first and then figure out which solution space to search over. But more broadly, how should the community think about this as we move toward the next generation of translational medicine?
Pushmeet Kohli [00:08:47]: The advice I give to anyone starting in this area is to think of yourself as a multidisciplinary person. Understand the problem first. Why are you working on it? What are you trying to achieve? Then think about what’s needed, whether that’s modeling, compute scaling, or data generation. Understanding the problem is extremely important, and of course you need to build your expertise.
There are constraints, too. Maybe there’s only a certain amount of data you can generate, or a certain model size you can afford to train. Understand those constraints, try to fail fast, and look at which approaches will be feasible in the long term to get you to the level of impact you’re aiming for.
Scientific Inductive Bias vs. Scaling: The Craft of Building Models
Brandon Anderson [00:09:52]: This leads right into my next question. When I look at the evolution of modeling, AlphaFold 2 was essentially a work of art, with a lot of carefully handcrafted features. There was very intentional thought put into every part of the solution. Some of Google’s or Alphabet’s work still stays in that space, but the general consensus seems to be moving toward more scalable, general strategies.
Do we still need artisanal, craft solutions for certain problems? Or are resources better spent focusing on scale first? With fixed resources and money, you can invest in compute, talent, or data. How should we think about that trade-off?
Pushmeet Kohli [00:10:48]: Let’s approach it from first principles. The art and craft of constructing a better model wasn’t accidental. We did a lot of experimentation, but there was a vision behind it. Scientific intuition from biophysics and biochemistry told us that amino acid residues are not just doing their own thing. They’re influenced by other residues. So let’s bake that in. If you’ve learned something from the scientific community, use that information. Give the model that unfair advantage.
That makes the model much more data-efficient because it doesn’t have to relearn everything. I also have to mention that curating good data is an art in itself. It’s not as if you can just say, “Put in more data.” If you replicate the same data, you’re not going anywhere. It’s not just about big data; it’s about good data and understanding the coverage of the data necessary to make progress. That’s a much more interesting and challenging problem in itself.
Brandon Anderson [00:12:20]: It looks like you have a thought, Sal. What do you think?
Sal Candido [00:12:24]: I very much agree. It depends on the problem to be solved. If you have a smaller amount of data, having more inductive bias in the model is going to help you. You actually need that to get results at smaller data scales. As you get more and more data, sometimes the model finds things you didn’t necessarily know about, and sometimes that inductive bias, if it wasn’t exactly correct, can hold you back. There’s a tipping point as things get better.
But I also object to your question a little because there’s a lot of craft in the scaling part of things as well. There’s a lot of algorithmic work that goes into taking models, training them on more data, putting more compute into them, and making them bigger. And that isn’t only from an infrastructure perspective or about making the models run inference faster. We’re seeing things go beyond standard transformers to more bespoke architectures. I don’t think we’re in a post-transformer world in any way, shape, or form, but we’re modifying those architectures to make them more fit for purpose and work better, even with internet-scale data.
As more data comes in, the challenge isn’t only curating that data or selecting the next batch of information the models need. It’s also asking, at every step and at every scale, what the right architecture is to get the most out of that information.
Why AlphaFold Didn’t Solve All of Protein Biology
Brandon Anderson [00:14:36]: If you think about the state of protein structure prediction right now, the news will say that protein structure prediction has been solved. But if you talk to my friends, they’ll say, “We have so much left to do here.” Function, dynamics, and design are still wide-open problems. It’s been about five years since AlphaFold 2 was announced, and progress has been made.
Pushmeet, do you think there’s another big leap coming? Are there blockers to solving these problems? Do we not have the right data, algorithms, or ideas? Or is it just a matter of time?
Pushmeet Kohli [00:15:28]: Science operates by isolating something and then making progress step by step. When people say the protein-folding problem has been solved, at a conceptual level there have been advances. But think about the narrative of proteins being the building blocks. Proteins aren’t blocks, and they don’t act as blocks. I say proteins are the building blocks all the time, but I don’t actually believe it.
Proteins are extremely complex. They’re disordered. Their shape might change depending on the context. John Jumper and I used to discuss this: What are we trying to solve? We don’t know the actual true ground state that proteins take, or the actual distribution of structures that proteins take. What we were trying to do was replicate a structure that somebody had obtained and deposited in the PDB. That’s what we did. And it just so happens that it’s useful.
But that doesn’t mean we’ve understood all of protein dynamics. At the top level, it’s easier to communicate that we’ve made progress, but the scientists in this crowd know how much remains to be done. There’s a lot to celebrate, but let’s not stop funding protein structure prediction and protein dynamics, because we’re just getting started.
Cryo-EM, Molecular Dynamics, and the Missing Information
Brandon Anderson [00:17:39]: What’s the biggest blocker? If you could wave a magic wand and say, “We have more of this,” and that would accelerate function, dynamics, or design, what would you bring into existence?
Pushmeet Kohli [00:17:54]: My background is quite eclectic. I started as a security researcher, then went into computer vision, Bayesian theory, discriminative machine learning, deep learning, AI for coding, and finally science. So when you ask me that question, the computer vision researcher in me gets very excited about cryo-EM micrographs.
I thought, “What is this PDB data? I should be working at the source. I should be looking at cryo-EM micrographs. I don’t want those structures. They must be missing out on all the data. I should just operate directly on cryo-EM micrographs.”
Getting models that can scale at that level, with the right amount of data, and extract the dynamics and distributional information captured there would be amazing. I tried it, but it requires more work.
Brandon Anderson [00:18:57]: There’s still work to do, but you believe that’s a route that could give you that information?
Pushmeet Kohli [00:19:00]: Yeah. I think at some point maybe some people better than me will take a stab at it, and we’ll get somewhere.
From Protein Structures to Larger Biological Systems
Brandon Anderson [00:19:07]: What do you think, Sal?
Sal Candido [00:19:09]: It’s interesting because these models are quite useful, but they’re not exactly the problem that most people want to solve. They solve a very specific purpose, and you can also use them to do other things. For example, you can use them to design new proteins, which isn’t necessarily what you would start with.
I think of the models we’re building now as someone who’s trying to understand how a bicycle works, but is modeling a spoke on it. Those models can get better and better over time, but what you really need to do is move from models of spokes to wheels to whole bicycles, because that’s what people want to understand. To continue the analogy, it seems like people want to use a model of the bicycle to design a part for a pickup truck.
As we put these models into the particular biological context in which they’re operating, we’ll be able to learn more about these interactions on a broader scale. That’s where I think things are going. But I agree that we should keep working on folding models, because they’re going to keep getting better.
Protein Design vs. Scientific Understanding
Brandon Anderson [00:20:43]: With regard to design, there’s a famous Feynman quote: “That which I cannot create, I do not understand.” Now we’re in a world where it’s really easy to create things without understanding them at all. How important is it to have models that help humans understand things, versus magical black boxes that can effectively one-shot a picomolar binder or something like that?
Sal Candido [00:21:18]: We were designing things with magical black boxes long before AI came around. In some sense that’s still useful. But understanding is really important, and it’s one of the big things I think about with AI.
There’s so much for these models to learn. As intelligence gets cheaper and more available, you can deploy it to learn more and more about what’s going on in the world. But how do you pull that knowledge out of the machine so that I can understand it? Maybe that’s just my esoteric curiosity. I think there are so many things to learn.
Brandon Anderson [00:22:05]: A lot of scientists really want to understand things, and the endpoints may not be as important to them. But we’re here to solve translational medicine as a problem, right?
Sal Candido [00:22:18]: I think the more you dig into things, the more you find the right way to keep pushing them forward. One thing that’s really salient to me is that these models have a lot more information in them than we know.
We’ve worked a lot on interpretability for our models, for example, and you find a lot of information there. People know that protein language models learn some notion of structure within their representations, but we find information about functions and motions as well. There’s a lot still to be unlocked, even from the models we have now. That’s important for us to understand as we raise their capabilities.
At the end of the day, you expect a world model to emerge from compressing all this information into a model. How does it do its job? How does it design a protein? It has compressed information from evolution into that model. In addition to being able to produce something useful to us, there’s certainly something to learn just by looking at what’s inside.
AlphaFold Confidence, Calibration, and Interpretability
Brandon Anderson [00:23:48]: What do you think, Pushmeet? Design versus understanding?
Pushmeet Kohli [00:23:53]: I have a different take in the sense that some level of understanding is necessary. Let me explain what level I mean.
AlphaFold isn’t perfect. AlphaFold 2 had a GDT score of about 90 on that set at the time. But even if it had a GDT score of 95, if its pLDDT score were completely uncalibrated, who would trust it? Imagine that it magically gave good answers but told you it was very confident, and then you worked on it for the next year only to find out it was completely wrong. Calibration of the uncertainty measure was extremely important.
In that sense, we do understand AlphaFold 2, and we made a lot of progress in understanding how it behaves. That’s different from understanding how it worked internally to find the solution. There I agree with Sal that interpretability asks: Why did it work? Why did it give this answer?
At the highest level, AlphaFold 2 was interpretable in terms of its behavior and ability to generalize, and we didn’t discover everything about that before launching. When we launched AlphaFold 2 and made the weights available, people found that it was a great disordered-protein predictor. It could figure out which elements of the protein are disordered. That shows it generalizes.
But interpretability asks another question: Interpretable by whom? If you’re saying interpretable by a human rational system, with the cognitive and computational limitations of the human brain, then no, AlphaFold 2 is not interpretable. But if you’re asking whether AlphaFold 2 is interpretable to a much larger, more sophisticated model in terms of how it works, maybe it is. We just don’t get it.
As users of AlphaFold 2, we do need to understand what it can and cannot do. Understanding its behavioral characteristics, strengths, and limitations is extremely important. We can’t just use these models without that characterization, because otherwise, rather than being helpful, they can harm us.
Brandon Anderson [00:26:53]: So your take is that interpretability, strictly speaking, isn’t necessary, but ensuring that a model is trustworthy so humans can make actionable decisions is what people should focus on.
Pushmeet Kohli [00:27:04]: Exactly. Interpretability is also in the eye of the beholder. Who is interpreting it? If it’s a human scientist trying to interpret how the model makes a prediction, that’s a different question from giving another, much larger LLM access to the activation layers and saying, “Can you predict what AlphaFold will do?” Maybe those frontier models of the future will be able to predict that and come up with a theory of how AlphaFold 2 was interpreting and producing its results.
When Will AI Transform Drug Discovery and Clinical Medicine?
Brandon Anderson [00:27:43]: We have a two-minute warning, so one last question for both of you. There’s a real chance that AI will make dramatic improvements in human health in the immediate future. I like quantitative predictions. Best guess: How long until we start seeing AI results in the clinic? Pushmeet, do you want to start?
Pushmeet Kohli [00:28:08]: I think “AI results in the clinic” is an ill-posed question. AI is already being used today in every part of the drug discovery process. From that perspective, it’s already there.
But if you’re asking when we’ll see a 10x or 100x acceleration in timelines, then the next question is: What are we accelerating? Is it lead optimization? Target discovery? Preclinical work or toxicology?
Over the next few years, and this is why the effort announced today is extremely important, we need to tackle some of the hard challenges of biology. Only then will we get the true unlocks of acceleration that dramatically transform drug discovery. AI will continue to be used in things that go into the clinic all the time, but larger acceleration will only be unlocked with a better understanding of the biological models this effort is trying to create.
Brandon Anderson [00:29:43]: All right, thanks. Sal, a quick answer, if you can. We’re almost out of time.
Sal Candido [00:29:47]: I’m tempted to literally put on my biohacker hat to answer this question. It’s important to think about what it means to push the field forward. What we really want to see is outcomes being affected. When will a drug be made entirely by AI? I don’t know. That’s hard to predict. But I do think we’re going to see rapid progress very quickly, because all these tools are already being used.
Going back to your point about basic research and basic understanding, one thing I learned a long time ago in my career, back at Google, is that sometimes it’s easier to approach a problem by asking what it would take to make a 10x improvement rather than a 10% improvement. That’s not because the 10x path is necessarily easier. It’s because it allows you to take a broader view and see solutions you haven’t been approaching. You go back to first principles and ask, “How are we going to do this? How are we going to really push this?”
Both approaches, the 10% and the 10x, are valuable, and we should do both. I’m very happy that at Biohub we have a beautiful and lofty mission statement: to cure all disease. If you want to do that, you really need to figure out what the 10x approach is. It’s a great opportunity to be able to go do that.
Brandon Anderson [00:31:46]: Awesome. Thank you both. Thank you for being in the literal hot seat. Very interesting.


This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.latent.space/subscribe

It’s hard to believe that Periodic was only launched last September:
One year later, it is considered one of the pre-eminent AI scientist labs, with dizzying talent density and astonishing progress in the autonomous lab buildout:
Most people are familiar with the standard credentials of Liam and Dogus, but we found an incredible “talent slope” while learning more about Periodic, where each successive employee seems more impressive than the last:
From building AI systems that reason over noisy physical experiments to creating laboratories where every instrument can become intelligent, Periodic Labs is betting that the next frontier of AI won’t come from simply training on more internet data, it will come from letting models experiment with the real world. In this episode, Periodic Labs’ Liam Fedus and Ekin Dogus Cubuk join swyx and Brandon to explain why scientific discovery is fundamentally different from math and coding, and what it takes to build AI scientists that can actually discover new materials.
We go deep on Periodic’s vision for “synthesis superintelligence”: reinforcement learning grounded in physical experiments, AI-powered materials characterization, simulations and density functional theory, high-throughput labs, and systems that learn from the entire process of doing science rather than only its published results. Liam and Dogus also explain why frontier models will still need experiments, why failed experiments may be some of the most valuable training data, what it means to give every piece of lab equipment “140 IQ,” and how autonomous experimentation could compress decades of scientific trial-and-error into months.
We discuss:
* Why intelligence alone isn’t enough for scientific discovery
* How reinforcement learning changes when the environment is the physical world
* Why science requires reasoning under uncertainty, noise, and missing information
* Prediction, synthesis, and characterization in the materials discovery loop
* Why physics and materials science are still far from “solved”
* The “matter compiler” and Periodic’s goal of synthesis superintelligence
* Phase transitions, X-ray diffraction, and AI-powered materials characterization
* DFT, simulations, and why experiments remain the ultimate ground truth
* Room-temperature superconductors, new magnets, batteries, and more efficient compute
* Why quantum computing may not automatically solve materials discovery
* What it means for every piece of lab equipment to have “140 IQ”
* Why data quality and negative results matter more than simply throwing more compute at science
* Training models on the process of doing science rather than the final answer
* Why even future frontier models will still need to physically experiment
* Scaling autonomous labs across AI, chemistry, physics, and custom hardware
* How automated experimentation could massively increase the “surface area for luck” in discovering new materials
Liam Fedus
* LinkedIn: https://www.linkedin.com/in/liam-fedus-26547811/
* X: https://x.com/LiamFedus
Ekin Dogus Cubuk
* LinkedIn: https://www.linkedin.com/in/ekin-dogus-cubuk-9148b8114/
* X: https://x.com/ekindogus
Timestamps
00:00:00 Introduction
00:02:49 AI and Reinforcement Learning in the Physical World
00:09:17 The End-to-End Materials Discovery Loop
00:13:28 Why Physics Isn’t “Solved”
00:19:04 The Matter Compiler and AI Characterization
00:30:22 DFT, Simulation, and Experimental Ground Truth
00:42:27 Dream Materials, Compute, and Quantum Computing
00:45:57 Giving Every Lab Instrument “140 IQ”
00:50:02 Automating the Lab
00:53:16 Data, Models, and Negative Results
00:58:13 Training AI on the Process of Science
01:00:20 Why Frontier AI Still Needs Experiments
01:06:06 Scaling Autonomous Labs
01:10:45 Building the Team and Deploying to Industry
01:17:01 From AI Copilots to Scientific Outcomes
01:20:57 Automating the Search for Superconductors
Transcript
Introduction: Intelligence Must Meet Reality
Swyx [00:00:00]: Okay. We’re here at Periodic. We’re here today with Liam and Doğuş from Periodic. Welcome, and thanks for having us at yours.
Liam Fedus [00:00:11]: Yeah, great to be here, and yeah, thanks for coming in.
Swyx [00:00:13]: I want to start off with one of these quotes that I love from your website. It says, “Intelligence is necessary but not sufficient. New knowledge is created when ideas are found to be consistent with reality.” It seems so straightforward, but why is it non-obvious?
Liam Fedus [00:00:29]: I think that’s sort of the thesis and premise behind Periodic that Doğuş and I thought when we were building this, which is you can’t just think your way to a solution. The universe is so complicated that in order to actually push the frontier of knowledge and to make progress, you need to create these conjectures and then actually see whether or not it holds. No amount of rereading that textbook or paper, thinking, or disappearing into a room is going to allow you to think through all possible experimental outcomes, expected and unexpected, and you really need this iterative process. And that’s our core belief for putting this together, and that’s why from the very beginning, we’re like, “We need to have the AI systems, the simulations of the physical world, but then also to build up the physical high-throughput experiment.” And we think a different type of intelligence emerges from that.
Swyx [00:01:21]: I think it’s also notable, like with OpenAI and Google, you don’t have those resources in those big labs. You had to come out and do your own thing, because you’re kind of inventing your own playbook as you go along.
Building an Interdisciplinary Lab for AI Science
Ekin Doğuş Çubuk [00:01:34]: Yeah, and a team like this has never existed, I think. We try to bring together solid-state chemists, solid-state physicists, experimentalists, theorists, hardware engineers, LLM experts, computer scientists, and some of these technologies are very recent. The high-throughput experimentation, the robotic arms have only been tried in the last three years or so. The force field expertise, some of these force fields are also very recent. But yeah, we felt like having a lab is very important. Having this focus and bringing all these people together to work together is very important. And I guess the closest examples from history are places like Bell Labs, where incredible theorists and experimentalists and chemists worked together and achieved incredible things. So we are trying to do the same here, yeah.
Physical-World Reinforcement Learning and Uncertainty
Brandon [00:02:17]: I want to talk about force fields in a bit, but
Swyx [00:02:19]: Yes
Brandon [00:02:19]: Before we get to that. Yeah what are they? But before we get to that, For context, some of our audience are AI engineers, and some are scientists. For the engineers: what’s the difference between working at a big lab like OpenAI and what you’re doing here? I think you’ve already hinted at this but maybe explicitly, how do you have to reframe your thought process?
Liam Fedus [00:02:49]: Well, I think one interesting thing is now our reinforcement learning environments literally derive from the environment, from our physical labs. Our data comes from our physical labs, and this is sort of our ultimate truth. It’s not enough just to do optimization against, some answers that were known in some papers or textbooks because we’re going beyond that, and I think that’s one of the biggest differences. But along with that, there’s issues of say, decision-making under uncertainty. So when you’re doing optimization against math, there’s a high precision to it. You’re not really dealing with kind of like variance or uncertainties or aberrant measurements, whereas that’s very key to our process. For example, when we are doing a materials discovery loop, things don’t come out of the furnace labeled, right? Even the labeling process can be stochastic and noisy. And sometimes you’ll have aberrations between different machines. Maybe we’ll have insufficient telemetry. So we thought we ran it at this temperature, but really the temperature was, some delta away. And an intelligence that can take in this noisy data and make intelligent decisions like a scientist would, is a sort of a different set of reasoning strategies. So I think there’s a huge amount of commonalities, so the standard mid-training, reinforcement learning, the construction of tool-using agents, variance reduction, making sure infra doesn’t have mismatches between training and inference. But we have to go beyond that and really think about how do you do accurate, work when there’s a high amount of uncertainty? How do you make really incredibly efficient use of a limited set of data? So we’re scaling things up significantly, but still, it’s very different than a digital environment where you can arbitrarily add more environments or more rollouts. We don’t have that same capacity. So sample efficiency is another key thing.
Why Experiments Are Noisy and Incomplete
Brandon [00:04:48]: This seems a little bit abstract to me, and it is a theme which has been on the science pods for a bit, but I think it might be helpful when you talk about experimental uncertainty, to explain what would a specific process that you were doing experimentally look like, and how would a human approach this problem, before we get into the AI side? Do you have a specific example of a type of material you make and what sort of uncertainty in the measurements you would experience in the process?
Ekin Doğuş Çubuk [00:05:17]: I think there are many dimensions to this. So one of them is I think every scientific experiment has to do some dimensionality reduction, and this is very different than coding or math. So, when you’re doing math or coding, all the context could be available to the human or the LLM. It’s a bunch of axioms, some corollaries people have derived from them. It’s a bunch of functions and API calls. So everything you need to reason through is available to you, and then you just have to be very smart and figure it out. In physics, as you know, we start with more atoms than we could ever store on a computer. So clearly, we’ll have to go from the original number of dimensions and bits that represent a system to the number of bits we can fit in the computer. And this was, I guess, the original premise of thermodynamics. They were originally really confused about how steam engines worked, but then turns out you can find five or six thermodynamic variables that explain what’s going on pretty well, which is incredible. That’s the beauty of physics.
Brandon [00:06:13]: You have a room full of atoms, and there are 10²³ or 10²⁷
Ekin Doğuş Çubuk [00:06:19]: Exactly
Brandon [00:06:19]: Or something atoms in this room, and yet you need temperature and pressure and simple variables, and that tells you everything. No, it tells you mostly what you need
Ekin Doğuş Çubuk [00:06:28]: A lot
Brandon [00:06:29]: To know for most cases, right?
Ekin Doğuş Çubuk [00:06:30]: Yeah.
Brandon [00:06:30]: So let’s say you make a material, and you could do something like look at an X-ray structure, and this thing is a very lossy, projection, right? It’s not actually a structure, or it doesn’t tell you the true structure. It tells you some piece of principal components of that. So, how would a human take these simple principal components and then, kind of reason about them? And then how do you extend that to an AI?
Ekin Doğuş Çubuk [00:06:58]: So on the theoretical physics side, right, we’ve decided that energy, the average value of energy and the fluctuations of energy is very important. So a lot of thermodynamics has been derived from this. And then we realized reaction barriers are also very important. The kinetics is very important. So a human can try to reason through this extremely complex system by coming up with some, reduced dimensionality descriptors. People also really like atomistic picture. You think about what the local neighborhoods look like, whether they’re from octahedra, tetrahedra, and that especially chemists really benefit from that. Okay, so that’s on the theoretical side. And on the experimental side, you would try to collect as much data as possible. So you’d have a lab notebook, and you’d write everything you see, et cetera, but with the understanding, of course, that you’ll miss a lot. So like as Liam mentioned, there are all these issues that come up. For example, your furnace will degrade over time because as you’re using the furnace, some of the things you’re baking will evaporate and cover the heating element. So every time you use the furnace, it gets worse and worse. Another issue is you have a furnace, but the temperature isn’t perfectly uniform, so where you put in the furnace will affect the result. We don’t have this issue yet, but we will maybe someday. The optical instruments, suffer a lot from vibrations. So every time somebody walks. I remember when I was doing my PhD, one lab had this big issue because sometimes their results would be very different than other times, and they brought it down to at a certain time in the night upstairs, somebody would be walking and that vibration affected the laser setup. So yeah, science is very hard, but that’s what makes it special, right? The other thing Liam and I have been talking about is a lot of the current improvements focus on math and coding and theoretical computer science because it’s easier for LLMs. But in real life, most things that require intelligence are actually more like science. There’s a lot of uncertainty, there’s a lot of noise, there’s a lot of missing context, but you have to be the intelligent being and figure out what to do next.
Liam Fedus [00:08:53]: I think a really interesting point to build on that too is the optimal reasoning strategies to do math, to do theoretical computer science, to do these types of things may not be the optimal reasoning strategies to do science.
Ekin Doğuş Çubuk [00:09:04]: Yeah.
The Materials Discovery Loop: Synthesis and Characterization
Brandon [00:09:05]: How do you define an RL reward function when your output is a noisy crystal structure or some number of I don’t know collective variables or something? Like.
Liam Fedus [00:09:17]: Maybe, I’ll give, one part of the loop, and I’ll just kind of dig into that. So, as part of the materials discovery loop, we have to first figure out, what to make. So that’s the stability of the atoms together, as well as do we expect it to have the properties we want. So we don’t just want, a novel configuration of the atoms. We want them, when put together, to have the property of interest. Next is how do you synthesize it? How do you actually make the thing? That’s also non-trivial. So even if you know something will hold together, the process of actually, what are the processing conditions to actually bring that forth is highly non-trivial. But again the thing. Once you’ve done those two steps and you’ve made something, it doesn’t come off labeled, so you actually have to characterize it and figure out what you made. So in that case rather than thinking about the RL environment of we’re just going to kick off an experiment, wait a couple days, and try to do a update on did you find a room-temperature superconductor or not, that’s completely infeasible.
Brandon [00:10:15]: Yeah. I was going to ask that. Yeah you can’t leave your GPU sitting for a week or something.
Liam Fedus [00:10:19]: It’s. Yeah, it’s completely infeasible
Brandon [00:10:20]: Yeah
Liam Fedus [00:10:20]: Because you don’t have enough agents to reduce the variance sufficiently. The rollout time is going between the different instruments. You’re waiting. There’s just some physics limits of the experiments, for the actual synthesis or furnace time. So it’s just too slow, too noisy. So the way we think about the AI program is basically constructing agents around this set of data that we’ve produced. And so in the case of characterization, you’re looking for RL environments that are rewarding the identification of the phases actually present. Are you able to given the raw experimental data, fit that effectively? So the way we do this is we’ll shoot X-rays, at the material. X-rays have a frequency or a wavelength roughly measurable or commensurate to the spacing between atoms, so you get nice diffraction patterns that kind of give a fingerprint of the crystal structure. And doing this, identification of what actually is present is highly non-trivial. And so some of our early AI work has been on basically producing systems to do this. And this allows us to move much more quickly too, because if you’re running so many experiments, you really quickly become bottlenecked on your ability to understand what was produced. But again, now the reinforcement learning environment is fairly straightforward, which is you’re going to reward the identification of the phases present. You’re going to penalize spurious phases or things that don’t have any real chemical plausibility. So that’s like a little succinct thing where you can kind of do this for the different pieces. And when you stitch this together, that’s the end-to-end discovery loop.
Training on Novel Experimental Data, Not Memorized Answers
Brandon [00:12:02]: Okay. You
Liam Fedus [00:12:02]: Maybe another
Brandon [00:12:03]: Yeah. Oh, sorry
Liam Fedus [00:12:03]: Another piece would be as we build up, a large basket of experimental data, you can think of another thing of basically timestamping the state of the world. So you could say, “At this date, this was our experimental evidence up until that point.” And you can construct reinforcement learning environments where you say, “Okay, given that body of experimental evidence and given, the choices, what was the next choice that the scientist made, or what was an outcome of that experiment?” And. This is really interesting too, because this allows us to do types of programs that would be harder externally. Because if the pre-trained model has already memorized some of this data and then you try to do reinforcement learning on that, if it already knows the answer and you construct reinforcement learning tasks that depend on that answer, it can fake work. So, and then, because it’s like, oh, well, it already knows the answer, so it doesn’t actually have to do the hard physics reasoning. It doesn’t have to actually do the calculation simulations. And it gets to the right answer, and then you reinforce that policy and you up-weight those reasoning strategies. Those reasoning strategies will not generalize to novel systems, so it’s not useful to us. But as we build up baskets of experimental data and stitching things together, having the lineage through the whole process, that gives us the ability to construct new types of environments that are just not plausible elsewhere.
Why Known Physics Does Not Mean Perfect Simulation
Swyx [00:13:27]: I want to ask my question
Brandon [00:13:28]: Seriously
Swyx [00:13:28]: But I’m worried that I’m taking it off the rails. I’m just going to go for it because I, again, I come from the non-scientist, but engineering sort of. I’m familiar with the ML side, but not the not the physics side. Okay, there’s a few versions of this, but I think the basic question is, don’t we have most of the laws of physics worked out? And how come we don’t have perfect simulators already?
Brandon [00:13:50]: Oh, that’s a great question.
Swyx [00:13:52]: Like, this is very basic, like. And he was like, “Ha, that’s cute.” But like
Brandon [00:13:56]: No, it’s a great question
Swyx [00:13:57]: Dude like I have all these physics books. What do you mean, what do you mean we’re not done?
Ekin Doğuş Çubuk [00:14:03]: Yeah, so there are a couple of things going on here, right? So
Swyx [00:14:07]: Yeah
Ekin Doğuş Çubuk [00:14:07]: I think first of all, we’re definitely not done with any of the laws of physics. Even the
Swyx [00:14:12]: At quantum scale, okay, but. And maybe very large scale, but at human scale, at material scale, we’re, we. What’s left?
Ekin Doğuş Çubuk [00:14:22]: We’re not. So it’s a it’s a really good question. And it’s very interesting why we’re not done yet, right? So there’s a very, common story people talk about. When Dirac was figuring out quantum mechanics, like late 1920s, he wrote a textbook about quantum mechanics, and he kind of framed it as it’s all done. And then there’s a phrase people like to quote. I’m not sure actually how accurate it is, but apparently Dirac said, “The rest is chemistry.” The idea being he could solve the hydrogen atom
Swyx [00:14:50]: Just compose everything.
Ekin Doğuş Çubuk [00:14:51]: Yeah. He could maybe solve a chain of 1D hydrogen atoms, but then he cannot currently solve, say, nitrogen interacting with oxygen, but that’s okay, that’s just chemistry.
Swyx [00:15:00]: Yeah.
Brandon [00:15:00]: Physics physicists really love single atom rep. Really simple systems.
Ekin Doğuş Çubuk [00:15:05]: Simple systems.
Brandon [00:15:05]: And that’s all you can solve.
Ekin Doğuş Çubuk [00:15:07]: Spherical cow.
Brandon [00:15:07]: Yeah, spherical cow.
Ekin Doğuş Çubuk [00:15:08]: But it turns out, I think what we learned in the last hundred years is, first of all, that’s not true. It’s not like a simple extension of what it was to do just hydrogen. And it’s not just. The rest is not just chemistry. There’s actually a lot of physics there. I guess one of the things we still haven’t figured out is high-temperature superconductivity. But there are a lot of other
Swyx [00:15:25]: And high for you is 175, 200?
Ekin Doğuş Çubuk [00:15:28]: Oh, when people say high-temperature superconductivity, what they mean there is unconventional superconductivity. So there’s the conventional superconductivity that’s just mainly driven by electron-phonon coupling. And in that case, you can kind of see the isotope effect. If you take the same system but just different weight for one of the elements, you can see the superconducting temperature drops at the rate you expect. So that one is conventional. But then higher-temperature superconductors like cuprates, like the ones that are above 77 kelvin, like 93 kelvin, turns out don’t obey that physics. But we don’t know what physics they obey. They’re just incredible superconductors. But it’s not just that. So that definitely is one of these very, popular topics we don’t know the theory for yet. But there’s so many other things we don’t understand. For example, even strong correlation in simple quantum mechanical systems, we cannot simulate yet. Density functional theory is a very powerful tool. It’s incredibly accurate on some things, but as long as. As soon as there’s some strong electron correlation, it actually fails to capture some of the effects. There’s so much to figure out. That’s why I’m really excited for AI to be applied to this field, because theoretically, computationally, and experimentally, there’s so much to figure out. And every improvement we make here should improve human life, because, the better we can understand materials, solid-state physics, the better devices we can make for them.
Brandon [00:16:41]: There’s this famous quote by Phil Anderson, which is, “More is different,” which is basically you may understand all the basic laws of something, but when you add many things, they behave qualitatively distinct, differently from how the simple physics should tell you. So understanding how lots of things behave together, it seems like it should be simple. Beginning laws are simple, but the collective behavior is so complicated that it’s really hard to model.
Ekin Doğuş Çubuk [00:17:07]: It’s emergent and universal, which is crazy. And we’re also seeing this with deep learning models, right? We see power laws everywhere in deep learning. We don’t understand, but it is very reminiscent of physics where there is, emergence and universality.
Brandon [00:17:20]: There’s been many theory papers in the physics world about emergence in neural networks.
Ekin Doğuş Çubuk [00:17:24]: Yeah, absolutely, yes.
Brandon [00:17:25]: Yeah.
Synthesis Prediction and the Matter Compiler
Swyx [00:17:26]: And then my other follow-up question was also just on the process of let’s call it the yeah, process and the end result. Let’s call it like, you want to have these properties that you’re targeting for materials, and then you have to figure out the process to get there. Is it worth separating these things so that you can have. You just train a model that perfectly reverse engineers any process whatsoever given, whatever theoretical end state you want? Is that, meaningful?
Ekin Doğuş Çubuk [00:17:54]: Maybe going back to the conversation we had about how since there are more than Avogadro’s number of atoms, we’ll never know the exact context of what happened. I think that might be a reason why we’ll never be able to reverse engineer everything perfectly. But we’re just trying to reverse engineer sufficiently to be able to improve the performance of the materials, basically.
Swyx [00:18:13]: Yeah.
Brandon [00:18:13]: What you’re asking is there a forward model where you put in an input and you can predict the output, and you want to under
Swyx [00:18:19]: Yeah.
Brandon [00:18:20]: You want to create a like a surrogate model which can understand
Liam Fedus [00:18:22]: The input is the final state of that, this configuration of atoms, and then the prediction is
Swyx [00:18:27]: Because then you can, you can split up the work almost. You can have one team do the process side, and then the other team do the property side, and then
Liam Fedus [00:18:33]: Yep
Swyx [00:18:33]: You know, just race them.
Liam Fedus [00:18:36]: Yeah, I think that there’s, a lot of progress we can make in terms of these. Splitting up these agentic workflows to these different areas, I think there’s a lot of empirical work we can make, on synthesis prediction as well.
Swyx [00:18:49]: Yeah.
Liam Fedus [00:18:49]: Whereas taking in that configuration of atoms, what tools can you use? How, given this experimental evidence, given prior literature, prior papers, how do you actually get to these things? And you ultimately are testing, are you able to actually replicate it or not?
Swyx [00:19:04]: The reason I say this maybe also is just thinking about it in terms of a company. You know, in the way that TSMC is kind of like a fab for semis, but they don’t design the semis, you could be the TSMC of materials where people come to you with like, “Well, here’s what I want,” and then you can figure out the process and
Liam Fedus [00:19:21]: Yeah, it’s like a matter compiler.
Swyx [00:19:23]: Yeah.
Liam Fedus [00:19:24]: So I guess given these requirements
Swyx [00:19:26]: Very Star Trek.
Liam Fedus [00:19:27]: Yeah. But it’s like, given these requirements, is this actually a valid set of configurations? Can this, can this exist?
Swyx [00:19:33]: Is it a good objective function? I don’t know. It’s too big.
Liam Fedus [00:19:37]: Yeah. We kind of talk about one of our missions as synthesis superintelligence, so I think it’s in line with this.
Phase Transitions, X-ray Diffraction, and RL Rewards
Swyx [00:19:44]: Yeah.
Brandon [00:19:44]: I want to double-click on something that you said a little while back, which was you talked about phases. And I think this is, going to Doğuş’s comment about, what is emergence. The concept of a phase transition I think might be foreign to a lot of the audience. So can you explain what is a phase transition, and then why is this a helpful signal for something like an RL environment?
Ekin Doğuş Çubuk [00:20:08]: Phase transitions are fascinating. I think the maybe the one that people can most easily relate to in their life is, ice melting probably, or water boiling. You increase the temperature of ice, and it still looks like ice and it behaves like ice, but there’s a certain temperature at which it just stops raising its temperature and then turns into liquid. So it suddenly goes from this solid phase to liquid phase. Another one that’s very relevant to us, of course, is the Ising model. And I think computer science and mathematicians also study this maybe under different names, but basically, you can have spins up and down, and they have different interaction terms if they’re both up versus one is up, one is down. And at high enough temperature, they’re usually just randomly up or down. Entropy wins. You lower the temperature, it’s still looking the same. But then at some point, suddenly they perfectly start aligning. Yeah, the phase transition. Phase transitions are really nice because it makes physicists’ life easier for studying certain things. They tend to have certain, spatial correlations that really help us, as well. Okay, so in our lab, phase transitions come about usually because we mix the precursors that are different, crystals basically, but then we raise the temperature or do something else to encourage them to react, and then atoms start reacting and they form into this new crystal. And the this shows up in the XRD pattern because usually if the geometry of the atoms change drastically, the XRD pattern changes a lot. But what really happens in a typical practical materials discovery campaign is when you first try something, it doesn’t work, and the outcome isn’t just a clear yes or no, but it’s usually a very mixed phase. It will usually have some of the precursors. It might have some amorphous phase, and then it will have a bunch of phases that you maybe didn’t try to make. So that becomes a challenge, and that’s why we find it very difficult to bring together simulations and AI to do the characterization. So the simulations can always say, “Oh, this phase doesn’t look like what we predicted, but this is a slight variation of it. Let me now do force field calculation on it to see if it’s still stable.” Or the AI can say, “Based on the previous experiment and this one, this is probably not the phase we want to make, so let’s change the conditions.”
Liam Fedus [00:22:18]: Yeah, and in some cases
Swyx [00:22:19]: For listeners, XRD is X-ray diffraction, which you already described.
Liam Fedus [00:22:22]: Yeah. In some cases, there’s a phase that just hasn’t been recorded before. It doesn’t exist in any papers or database. And so the AI has to actually go and sort of use these tools to say, “Well, what configuration atoms would actually explain these phases?” But this becomes incredibly important for directing the scientific process because, let’s say we have a phase target in mind. We really need to sort of hill climb that to get better measurements on that. So if it’s, one percent and we’re like, actually, we want to increase this phase purity, we want to have a very good measurement on this experimental data to say, what is present and what’s not.
Brandon [00:22:57]: So one of the advantages is you have just a distinct variable that you can just observe. Going back to the statement about how do you deal with uncertainty, one of your, it seems like part of your answer is you make your observable signature somewhat unambiguous. There’s no interpretation there. Is that right? Is that the logic behind this? Or is this literally you just needed to find a new phase, and so this is the thing you care about?
Liam Fedus [00:23:20]: I think there’s still ambiguity in a lot of these cases. A very dumb thing is replicates.
Brandon [00:23:25]: Yeah.
Liam Fedus [00:23:26]: We run replicates, of course.
Brandon [00:23:27]: Yeah. Yeah.
Liam Fedus [00:23:28]: But no, I think there — You know, it’s a challenging task because it’s not just a fully deterministic, and there are well, there are ambiguities, and other things. Two different, phases can actually be consistent with the same pattern.
Brandon [00:23:42]: Yeah.
Liam Fedus [00:23:42]: And so I think that’s why it’s really important for the system to use some chemical intuition to say “Okay, well, this would be highly unlikely given the synthesis conditions, given this, the prior knowledge.” and so that can be used to help disambiguate.
Chemical Priors, Multimodal Measurement, and Ambiguity
Swyx [00:23:58]: So you’re, so you’re somewhat injecting priors there, based on what you expect.
Ekin Doğuş Çubuk [00:24:03]: Yeah. I think thermodynamics is the biggest prior, right?
Swyx [00:24:06]: It is.
Ekin Doğuş Çubuk [00:24:07]: And then physics is a big prior. Another one that we really benefit from here is multimodality or materials characterization. So we can do XRD, and yes, some phases might look similar on the XRD pattern, but then we can also measure their electrical properties, magnetic properties. We can measure their morphology, like some kind of electron microscopy. And then some of the phases that look similar on XRD will look different in some of these dimensions. And that kind of multimodality really helps. And again, AI is really helpful here because humans have limited context and computation power as well. So if you give humans 10 different modalities at the same time and say, “Analyze these all consistently,” it’s a bit difficult. But it’s actually not super intelligent for an AI to do it. It’s great, yeah.
Liam Fedus [00:24:46]: Yeah, it’s like the super intelligence is it can just do so much calculations. It can look at everything. And this is, kind of gets to that, phrase about better decision-making under uncertainty, where now it’s stitching together, the signal across many different instruments longitudinally, across different experiments, across the replicates, and then kind of getting to this underlying better model
Swyx [00:25:08]: Yeah
Liam Fedus [00:25:08]: Rather than just over-indexing on one measurement from one instrument.
Information Limits, Telemetry, and Maxwell’s Demon
Swyx [00:25:12]: And let me just, like. So you know, you started, I think a while ago also saying that there’s just more data than you can fit in any reasonable computer or process or anything. So at some point, you have to throw out data even though you have all these replicates, even though you have presumably ten different ways to measure the same thing. It’s. ‘Cause what if one of your temperature things is wrong? So when do you throw that out? How do you decide what to do? As a data-hungry ML guy, I just want everything, right? And then I just, you know
Ekin Doğuş Çubuk [00:25:37]: So it’s, yeah.
Swyx [00:25:37]: Waddle, learn everything. I don’t know what I don’t know, and it’s just throw it, throw it, the machine at it.
Ekin Doğuş Çubuk [00:25:42]: Yeah. So we don’t throw any data. I think maybe what I meant was potentially. If God was observing the experiment
Swyx [00:25:50]: Yeah.
Ekin Doğuş Çubuk [00:25:50]: There is more data in the experiment than humans can fit in a computer. Unfortunately, us mortals cannot observe all the atoms. So even though there’s more data than we could store, we actually can’t access that data. For example, I cannot trace all the atoms’ positions. So unfortunately, we’re still in a regime where any data we collect is very precious, and we don’t delete any of it. Yeah.
Brandon [00:26:11]: Maybe a way of thinking of it to connect to the AI side is think about probing, like linear probing of a of a foundation model, right? This model has all this data, so God has this entire picture of what the material is doing, and then the experiment is this little, tiny linear probe which has eight bits or something of output. And so you’re taking this humongous thing and coarse-graining it down to a little bit of information, and then it’s now, either traditionally a human’s job or now your agent’s job to reconstruct from those eight bits of information
Liam Fedus [00:26:43]: It’s an analog, yeah.
Brandon [00:26:44]: What is actually happening inside.
Liam Fedus [00:26:45]: It’s an analog where you can’t have full visibility over every parameter weight, over every activation for that input. Yeah, you’re getting some sort of subset on it or some coarse-grained features across the whole thing.
Ekin Doğuş Çubuk [00:26:58]: Yeah. But so you’ll probably find that. You probably, do find, Maxwell’s demon argument very interesting because that kind of
Swyx [00:27:05]: Can you, can you res-recite it? Because I don’t. He’s mentioned it before. I keep forgetting it.
Ekin Doğuş Çubuk [00:27:10]: So, in the 1800s, people realized that entropy of a system has to increase or a for a closed system
Swyx [00:27:18]: Yeah.
Ekin Doğuş Çubuk [00:27:18]: Or stay the same, but it can’t go down.
Swyx [00:27:20]: Yeah.
Ekin Doğuş Çubuk [00:27:21]: It’s the second law of thermodynamics. And there was a thought experiment that was asking, what would happen if there was a very tiny, all-knowing demon that could kind of look at atoms, and then whenever an atom had low energy, kind of open a door and let it into this other room and keep doing this to reduce entropy of the system, which would break second law of thermodynamics? And for a long time, I think there wasn’t a very satisfying answer for why that was the case. But then there was a more recent paper in the 1900s, more recently to us, Landauer came up with the argument that you’d have to spend energy to delete information, and Maxwell’s demon would basically have access to so much information by doing this thing he’s doing that he’d have to delete some. And for. When he’s deleting information, he’d have to spend energy, which would then make sure entropy increases. So even though we’re not at the level where we can be Maxwell’s demon, I think your, question is very visionary towards a future where we store so much data we have to delete some.
Swyx [00:28:17]: Yeah, no, I wasn’t, I wasn’t obviously going to that, to that extent. It does remind me of air conditioning. That’s kind of how air conditioning works. But also, I think it’s just a question of why don’t you just buy every sensor in the world and just over. Ridiculously over-instrument, over-replicate everything?
Liam Fedus [00:28:33]: Oh, we do that.
Swyx [00:28:34]: Right? Yeah, okay, yeah, okay. There we go.
Liam Fedus [00:28:35]: Yes, yeah.
Swyx [00:28:35]: Okay, it works out. Right. All right.
Liam Fedus [00:28:37]: Yeah, I think, I think it’s a matter of
Swyx [00:28:39]: Confirmed.
Liam Fedus [00:28:39]: Yeah, exactly. Are you getting visibility to individual atoms, which is infeasible? But no, we add a huge amount of telemetry, and that basically reduces the number of hidden variables, if you will.
Swyx [00:28:49]: Yeah. Yeah. And just out of curiosity, you guys do you worry that you’re just in California? Don’t you want to do this in Nepal and Australia?
Ekin Doğuş Çubuk [00:28:59]: Yeah, we could. We have plans to open more labs.
Swyx [00:29:02]: Because obviously gravity and where you face and where you. Where you are matters.
Ekin Doğuş Çubuk [00:29:07]: Also expertise can change, right? Here we have certain kinds of expertise. For example, Stanford Physics Department, Engineering Department has certain kinds of expertise that really helps us because we can hire from there. Our researchers can collaborate. As you said geography can matter, but then different cities can have some differences of expertise within physics, chemistry.
Swyx [00:29:27]: Yeah. Oh, we have Zoom
Ekin Doğuş Çubuk [00:29:28]: Yeah.
Swyx [00:29:28]: For that.
Ekin Doğuş Çubuk [00:29:29]: Yeah. But still
Swyx [00:29:31]: It’s not like your experiments are going to be affected by where you are.
Liam Fedus [00:29:34]: Yeah. Yeah, the future of Periodic is not a single lab.
Swyx [00:29:38]: Yeah. You gotta head up your private jet and go to space and you know.
Liam Fedus [00:29:42]: Right. Yeah, I think it’s necessary to have this sort of network of labs. And like, yeah, I think as Doğuş was just kind of pointing out there’s an optimal place for each of these based on expertise or materials constraint. There’s many different constraints, permitting.
Connecting Simulations With Physical Experiments
Brandon [00:29:56]: Okay, so one of them is. So one thing I-I’ve been curious about is how do you deal with computational tools? Or how do you integrate computational tools into a decision-making workflow, especially when, let’s say, computation and experiment don’t necessarily, meet? It’s also theory in terms of high-level understanding. And yeah, how does that sort of work together?
Ekin Doğuş Çubuk [00:30:22]: Yeah. So, one thing that’s really helpful here is there are certain things that are easier to do in simulation and more accurate in simulation. There are certain things that are easier to do in experiment and more accurate in experiment. In general, of course, experiments are more accurate, but certain measurements are very hard to make with experiment. So you make a bad version of it, so you get inaccurate data. Let me just give you some examples. So one of them is, we use density functional theory to estimate formation enthalpy of materials and
Brandon [00:30:51]: Sorry, quick pause. Can you explain densital density functional theory and formation entropy?
Density Functional Theory and Formation Enthalpy
Ekin Doğuş Çubuk [00:30:55]: Yeah, of course.
Brandon [00:30:56]: Sorry, enthalpy, yeah.
Ekin Doğuş Çubuk [00:30:57]: Yeah. So, density functional theory is probably the most commonly used simulation method for materials. It kind of comes from Hohenberg–Kohn theorem, and Kohn got a Nobel Prize for this. He. When he realized that, sure, quantum mechanics is very expensive because you basically have this exponential Hilbert space for every wave function, so you have to solve this very difficult problem, especially as the system size grows. Kohn and Hohenberg realized that turns out you don’t need to do it at exponential space. All you have to do is think about the charge density, at least for the ground state of quantum mechanical systems. So this it’s actually the theorem is pretty simple. So even I think an undergraduate in physics can understand. But the idea is all the properties of a quantum mechanical system ground state is a functional of the charge density, which is amazing because charge density is just this three-dimensional object, whereas the Hilbert space and wave functions are exponential. So that was a really interesting observation. And then Kohn published another paper, I think with his postdoc, Sham, Kohn–Sham, wave functions, kind of had a practical way to solving some approximation of quantum mechanical properties of materials. And today, we use it very much, both in our lab and externally. And then formation enthalpy is one of the measurements we can make to understand the energy of the material. Basically, we want to ask, what is the energy that this material has? Because if that energy is higher than other materials these atoms can come into, then it’s unlikely that this material will be able to be made. So stable just means like it’s on the convex hull of other materials, and their energies. Was that clear enough, or should we.
Brandon [00:32:42]: Maybe a simple high-level point. So density functional theory is a way of taking something which is exponentially hard, in number of electrons
Ekin Doğuş Çubuk [00:32:51]: Right
Brandon [00:32:52]: Or number, in your system size, and then reducing it approximately with some introduced error to maybe usually an n-cubed approximation. So now you can compute quantities which are, you know. You can now compute things which are just uncomputable, but at some cost, right? If you could do it perfectly, you would — we wouldn’t need a lab. But we can’t. This might be a good callback to our, actually our second episode ever. Heather Kulik had said, she famously said, “There is no AlphaFold for materials, not just computationally, but the ground truth just doesn’t exist.” Like, we don’t really know what material crystal structures look like, and DFT is generally our best route towards.
Ekin Doğuş Çubuk [00:33:34]: Yeah, I think so. Maybe two things I would add is DFT by definition doesn’t have to be an approximation. The Hohenberg–Kohn theorem shows it
Brandon [00:33:42]: It doesn’t
Ekin Doğuş Çubuk [00:33:42]: It can be exact. But I think, as you’re pointing out, the exchange correlation functional we don’t yet have access to. And then for the charge density-based models, we don’t know what the kinetic energy functional is. And then even if we had perfect DFT, I still think we’d need a lab because we. Even if we had perfect DFT, we cannot fit ten to the twenty-three atoms in a computer.
Brandon [00:34:03]: Yeah.
Swyx [00:34:04]: You’re very obsessed with that number.
Brandon [00:34:06]: But. No. Well, okay. Yeah. But with exponentially growing compute, you can argue that in n-cubed scaling size, eventually you could, in principle, calculate things. You just wait till the pause long enough.
Ekin Doğuş Çubuk [00:34:17]: Oh, yeah. We need massive computers, yeah.
Brandon [00:34:19]: But yeah. But as long
Ekin Doğuş Çubuk [00:34:20]: It’ll take a while.
Brandon [00:34:20]: Yeah. But as long as n-cubed is a good scaling, you can, you can.
Swyx [00:34:24]: As the non-scientist, I will make an observation, which I don’t know if you want to throw around or anything like that. So I came from finance. We had the Gaussian copula, which was a way of pricing credit default swaps, by doing correlations and reducing everything into a single kernel. And it’s very. It feels also spiritually similar to the VAE, in terms of like, condensing all these things into a single sort of parameter. I wonder if just there’s the same trick everywhere. Yours sounds, a bit more high-dimensional than mine, which VAE is just like, one sigma thing. But sounds the same?
Liam Fedus [00:35:01]: Yeah, the charge density file is still quite a big file.
Swyx [00:35:04]: Yeah. Right. You hand wave a bit, but it’s good enough. This is, this is the highest order bit. Yeah.
Ekin Doğuş Çubuk [00:35:10]: I personally don’t know of a better method to predict the stability of a new material. It’s definitely not perfect, but it’s definitely better than other methods I can think of. And yeah, so going back to your question, the reason we use them is for certain things it’s easier to simulate than experiment, and they kind of complement each other well. And then we can do this in a loop. We feel like simulations will never be enough by themselves, but in the loop of simulations, AI, and experiments, I think we can make progress much faster than before.
When Simulations Fail: Microstructure and Calibration
Brandon [00:35:41]: One thing that simulations let you do is scale up much more quickly. Because now compute is, much easier to bring online
Ekin Doğuş Çubuk [00:35:48]: Yeah
Brandon [00:35:48]: Than bringing in bringing on new labs. How do you avoid. Or let’s say you scale up a bunch of DFT, how do you avoid in having your models over-index on that and still be grounded that the real world is ultimately the truth you care about? If you. You know, how do you sort of balance that?
Ekin Doğuş Çubuk [00:36:05]: Well, we scale up the lab as well.
Brandon [00:36:06]: Yeah, you scale up the lab.
Ekin Doğuş Çubuk [00:36:07]: Right, yes.
Brandon [00:36:07]: But you could. I could imagine scaling up DFT to millions, hundreds of millions given modern compute, whereas your lab, I would imagine, your hundreds, thousands a day? I don’t know. Something still fairly. Well, even in a really high-throughput case you’re still fairly limited by comparison, right?
Ekin Doğuş Çubuk [00:36:27]: Yeah. One thing is humans, but also LLMs, are kind of pretty aware of the limitations of DFT. For example, even if we can do, as you said, hundred million trials with DFT
Brandon [00:36:39]: Yeah
Ekin Doğuş Çubuk [00:36:39]: We could never actually try their microstructure. And microstructure is this idea that DFT usually simulates the perfect crystal, but crystals usually actually aren’t perfect, and their microstructure, which is the structure at a medium order, can really affect the properties. Other issues, of course, like some properties like superconducting temperature cannot be easily simulated by DFT. Again even if you do a hundred million trials with DFT, it won’t be enough. So we have to rely on heuristics, experiments, et cetera.
Liam Fedus [00:37:08]: And so there’s a huge filter from all those calculations to actually what gets executed in the lab.
Swyx [00:37:13]: Is that human-mediated or LLM or a mix?
Ekin Doğuş Çubuk [00:37:17]: Can be a mix.
Swyx [00:37:17]: Yeah.
Ekin Doğuş Çubuk [00:37:18]: For example, for a long time, Materials Project, which was the best open source DFT database, Use experimental calibration for DFT. So they would actually use experimental data to calibrate DFT. We also do this internally here. Whenever there’s a certain chemical system we’re interested in you get experiments, you get simulations, and you can calibrate them.
Why Inorganic Materials Differ From Biology
Brandon [00:37:37]: This is another thing. I’ve talked to a lot of biologist friends, and they are always very confused when you say materials, you can’t just do an XRD structure and actually know what the structure is. Because if you’re in the biology world, you can look at a crystal structure of a protein and typically more or less with sub A well order angstrom accuracy reconstructed. What is the difference between the materials XRD, X-ray disfor — X-ray diffu- no
Ekin Doğuş Çubuk [00:38:05]: Diffraction.
Brandon [00:38:06]: Diffraction, sorry. Experimental XRD for materials versus, let’s say, the biology world. What information do you lose, and why is this sort of like a lossy projection?
Ekin Doğuş Çubuk [00:38:16]: It’s hard not to sound crazy when answering this question. But do you not feel like, organic chemistry and biology almost stem from a lower VC dimension, like lower Kolmogorov complexity model? Where like, you can almost represent as a one D sequence, and it can almost be compiled. So there’s a sense in which it’s, simpler in the Kolmogorov complexity sense.
Swyx [00:38:38]: One D?
Ekin Doğuş Çubuk [00:38:40]: Like just DNA is just
Brandon [00:38:43]: Yeah.
Ekin Doğuş Çubuk [00:38:44]: Or RNA.
Brandon [00:38:44]: I think most people would argue it’s more two D, but
Ekin Doğuş Çubuk [00:38:47]: Okay, well, yeah. I’m not a biologist, so
Brandon [00:38:49]: Yeah.
Ekin Doğuş Çubuk [00:38:50]: I’m sure they’re right. But with inorganic chemistry, virtually it’s crazy. It’s clearly not coming from such a low Kolmogorov complexity model because that three D inorganic crystals can host metals, insulators, superconductors, diamond. It’s really crazy different. People tried, but we could never find a one D representation. It’s very hard to find a SMILES string for three D inorganic crystals. And the way the atoms interact with each other is quite strange, like covalent bonds, ionic bonds, metallic bonds. But yeah, this is, it’s very interesting. I wonder sometimes have you ever looked into issues with solid electrolytes? Why people cannot replace liquid electrolytes in batteries with solids? There are all these dendrites that form. Basically the lithium will form these structures that go into a solid electrolyte and crack it. And you look at that, and you think this is so primitive compared to biology. In biology, these systems can do such complex nano technology- technological progress, whereas we can’t even make two interfaces work with each other because lithium will break it. Yeah, but then I guess inorganic stuff can be very resistant to high temperature. You can make space shuttles. You can make silicon, compute Moore’s law. So yeah, it’s a trade-off.
Brandon [00:40:05]: Yeah, we’ve talked about this with several other guests. I think one of the big differences in my mind is that biology, we have a toolkit that you can basically borrow from millions of years of evolution
Ekin Doğuş Çubuk [00:40:19]: Yeah
Brandon [00:40:19]: That has already given us all the tools, and we can just reproduce those. But also evolution also constrains the yeah, the VC dimension to be fairly low for proteins and so on. The emergent beyond- behavior of large-scale systems like cells can be much more complicated.
Ekin Doğuş Çubuk [00:40:36]: Yeah, and consciousness.
Brandon [00:40:37]: Yeah. Yeah, but maybe even just more specifically for. If you just look at an X-ray, you had a perfect crystal, and you shoot an XRD at it, and you get the you get the spectrum. That still doesn’t actually tell you uniquely what the crystal structure is, right?
Swyx [00:40:54]: No.
Brandon [00:40:55]: So, and that’s something that
Swyx [00:40:56]: What?
Ekin Doğuş Çubuk [00:40:57]: It’s just average.
Brandon [00:40:58]: Yeah.
Ekin Doğuş Çubuk [00:40:58]: It’s just average behavior. But like, for example, disproportionation is a very complicated thing, right, where it’s possible that you have the perfect crystal, and on average, it looks like this perfect XRD. But in reality, there is some other pattern going on where atoms slightly shift to the right or left, but on average, they’re in the middle. Yeah, it’s a difficult life, yeah.
Brandon [00:41:16]: Whereas with proteins, you basically know there’s some sort of underlying protein follows some rules, so you can, you can combine a forward model with that.
Liam Fedus [00:41:23]: Right. And then you can kind of move and move towards single crystal XRD, and that can help.
Swyx [00:41:29]: When we started the Science Pod, there were all these elements of AI in science, AI in math, AI in physics, and all these things. And we there was this kind of pissing contest which is harder. And I think you can make the scientific argument that materials science is the hardest.
Liam Fedus [00:41:43]: Well, we didn’t pick it necessarily for hardness. There’s actually a lot of areas where it is easier. We have these great simulators for large classes of materials, whereas in biology, it’s incredibly difficult to model a cell or an organ or a full organism. So that’s a huge advantage.
Swyx [00:42:01]: Yeah. That, isn’t that interesting that it, at the smallest level, it is one-dimensional or two-dimensional, but and then it just has many orders more scale and structure, and that actually introduces the complexity. It’s, that’s weird. Whereas maybe complexity in materials is more the microstructure.
Liam Fedus [00:42:20]: Yeah, I still — you still get different complexity at the different, length scales for materials as well. So yeah, microstructure and beyond.
Dream Materials: Superconductors, Magnets, and Batteries
Brandon [00:42:26]: Yeah.
Swyx [00:42:27]: Yeah. Just sort of a brief sort of palate cleanser. In terms of science fiction, let’s say you can invent whatever materials that you want. What is more valuable? You know, I have, I have a list here. There’s obviously room temperature super- superconductors. But also, you mentioned batteries. I’ve always thought batteries. Just do better than lithium-ion, you’re good. That’s stood around for a hundred years. And then the last one is carbon nanotubes, mostly for the space elevators. I don’t know if, like. Those are, those are the three that come to mind. Are there any that people talk about in your world that are the dream?
Ekin Doğuş Çubuk [00:43:03]: Definitely superconductors
Brandon [00:43:05]: Yes
Ekin Doğuş Çubuk [00:43:05]: Magnets.
Brandon [00:43:06]: Right.
Ekin Doğuş Çubuk [00:43:06]: Especially if we can lower the dependence on rare earth or transition metals that are hard to source, like cobalt, reducing cobalt in batteries. One of the things I think is very fundamentally important is, if we can get close to Landauer limit for compute energy efficiency.
The Landauer Limit, Reversible Computing, and Energy
Brandon [00:43:24]: Can you, explain Landauer limit?
Ekin Doğuş Çubuk [00:43:26]: Yeah. So if you look at how much energy we spend per, like FLOP or compute, it has followed the Moore’s Law type behavior, but it’s gotten exponentially more efficient. So we’ve been spending exponentially less energy per FLOP or compute. I haven’t checked recently. This was the case ten years ago when I last
Brandon [00:43:44]: Yeah, that’s great. I remember when I first learned about it, looked at that, and we were, what ten, twenty orders of magnitude away from it or something.
Liam Fedus [00:43:50]: Beyond the limit, yes, right.
Brandon [00:43:50]: And the last time I looked, it’s. We’re weirdly close. We’re within a factor of I don’t know three or four orders of magnitude away.
Ekin Doğuş Çubuk [00:43:59]: Yeah.
Brandon [00:43:59]: It’s actually like
Liam Fedus [00:44:00]: I just exponentially rolled out over many years.
Brandon [00:44:01]: Yeah, exponentially over many years are
Ekin Doğuş Çubuk [00:44:03]: Four or five, yeah.
Brandon [00:44:04]: Yeah, pretty impressive.
Swyx [00:44:05]: But in your lifetime, you
Brandon [00:44:06]: Yeah, no I. Yeah, since, just since I discovered this, what, ten or fifteen years ago? Yeah, no.
Ekin Doğuş Çubuk [00:44:11]: And one of the things we have to do is dissipate heat, because as these, computations are happening, it’s producing heat, which has to happen, but then we have to dissipate it. So if we can get close to the Landauer limit for how efficiently we do computation, that’s amazing for humanity, right? Because that means now we’re doing computation, which is probably one of the most fundamental things we do as humanity, but as efficiently as possible from energy perspective, which is a real occurrence in the universe.
Swyx [00:44:34]: That get affected by quantum computing? I’m just going to throw it out there. Theoretically, massively, embarrassingly parallel compute.
Ekin Doğuş Çubuk [00:44:41]: So it definitely gets affected by reversible computing. But honestly, I’m not an expert. I don’t understand. So if you can do reversible computing now, you’re not. You don’t actually have to spend energy to do computation because you’re not actually deleting any information. It’s reversible. But I don’t know if that works. I don’t know. Quantum computing, I assume, has to obey these laws somehow because they are still, like. They have to obey thermodynamic
Brandon [00:45:06]: It’s unitary, so
Ekin Doğuş Çubuk [00:45:07]: Yeah, exactly
Brandon [00:45:07]: It’s reversible, so for now.
Swyx [00:45:10]: One of my most memorable conversations with Elad Gil was he actually was a very skeptical person about quantum computing.
Ekin Doğuş Çubuk [00:45:15]: I see.
Swyx [00:45:15]: He’s like, he’s like, “Even if you had it today, there’s no applications.” I’m like, “Whoa.”
Ekin Doğuş Çubuk [00:45:19]: I
Swyx [00:45:19]: It breaks, it breaks the it breaks the security RSA. That’s it.
Ekin Doğuş Çubuk [00:45:22]: I agree with that. Like for them, people talk about how if you had a quantum computer today, you could simulate things so much better. And I asked them, “Okay, let’s accept that. What would you simulate?” Like, you’re still simulating perfect crystal. You still have the issues that DFT has if DFT was perfect in its prediction ability. So yeah, I do think people are kind of glossing over some things because quantum computing is so exciting. If we can compute in a quantum logic space instead of classical logic, it’s just so exciting that people are glossing over what it would actually do when it’s made. Maybe that’s fine because maybe once it’s made, it will do amazing things we can’t even imagine.
Putting Intelligence Into Every Laboratory Instrument
Swyx [00:45:57]: Good. Okay. I was going to move to the automating the lab side, where you’ve closed the loop on your experimentation. You can talk about robotic arms, talk about. But it’s basically just everything you’ve done here. My favorite quote from you was that every piece of equipment in your lab is going to have a hundred and forty IQ. Okay, what does that mean?
Liam Fedus [00:46:17]: In the early days, as we were scaling things up, we realized that we had huge bottlenecks imposed just by operating machinery. So, for one set of machinery, we would have technicians and scientists looking for particular morpho- morphology and trying to see, okay, what actually were we making, see if this is consistent with our intentions. And we really quickly, as we scaled up the lab, came into these bottlenecks where it just wasn’t keeping up. So we started doing some programmatic approaches to capturing the data on SEM. And the data that was being surfaced from this very simple program where it sort of takes a field of view, captures, zooms in captures, was just not all that useful. It was a little too dumb for what we really wanted to get. And so at that point, it was then, really pertinent to build AI systems directly onto the machines to start controlling these things. And now they have the full context as to what we were trying to achieve. So what was the intent of the experiment? What were we trying to synthesize? What were some other experimental evidence? And it’s actually looking, in the machine and capturing that data. And this is really valuable now because we’ve been talking about these hidden variables not being able to capture everything, but if you can do more intelligent data capture at the time of that experiment, your data for future AI systems and future computational predictions is that much better. You’ll have just a richer set of data. And so that’s sort of what we mean by just, everyth- everything on the lab has to be incredibly intelligent to just make the data as useful as possible. So that was, that was sort of the motivation and inspiration behind that.
Swyx [00:47:57]: What’s the state of the art there in terms of putting intelligence on every device, right?
Liam Fedus [00:48:02]: Well, I think one interesting aspect, is what’s the latency
Swyx [00:48:08]: Exactly
Liam Fedus [00:48:09]: Of controlling that instrument?
Swyx [00:48:10]: Because there’s cloud latency, but then there’s also device compute, which
Liam Fedus [00:48:13]: That’s right
Swyx [00:48:13]: Also.
Liam Fedus [00:48:14]: And then there’s sort of a time scale associated with different physical processes. And if calling out to an API or something is simply too slow, so the amount of reasoning or tokens or tool calls, is just not matched to the latency of that actual process, then that’s infeasible.
Swyx [00:48:33]: Okay. That is right.
Liam Fedus [00:48:34]: It’s almost like, self-driving cars, right? So.
Brandon [00:48:37]: I. Real quick. I just am actually surprised to hear that because the latency I’d imagine for reasoning seems small compared to a lot of these things take hours to run, right? Or maybe, you know. Or maybe I. Maybe your experiments are much faster or high-throughput or something. I’m just surprised to hear that, actually.
Liam Fedus [00:48:52]: So ultimately, the experiments do take, many hours, days, but there could be particular steps where latency would matter.
Ekin Doğuş Çubuk [00:49:00]: Yeah. You might want to have finer control. Another thing is, how expensive it is because the same reason some of these models can be really slow in analyzing data also makes them very expensive. And then finally, the human patience. If a human wants the analysis of a certain XRD pattern and they have to wait two hours before they get a good result, that’s very different, I think, than if they can get it in two minutes.
Brandon [00:49:25]: Oh, okay.
Liam Fedus [00:49:26]: Yeah. And then the reasoning, it’s. You know, you’re not doing a single call to that model, right? You’re doing, potentially many tokens, to kind of get to these patterns.
Ekin Doğuş Çubuk [00:49:35]: Run simulations in the loop.
Liam Fedus [00:49:37]: Exactly.
Ekin Doğuş Çubuk [00:49:38]: Deep research in the
Liam Fedus [00:49:39]: It’s not a single tool call. It’s not a single inference thread. So then the latency can blow up quite a bit.
Brandon [00:49:44]: I see what you mean.
Liam Fedus [00:49:45]: Yeah.
Brandon [00:49:46]: How much of your latency is tool calls? And I assume tool calls is largely DFT or computational- … and therefore. Yeah. So how much of your latency is derived from those versus actual reasoning?.
Liam Fedus [00:49:59]: It’s really process dependent.
Pragmatic Robotics, Automation, and Lab Reliability
Brandon [00:50:01]: Okay. Yeah.
Liam Fedus [00:50:01]: Yeah.
Swyx [00:50:02]: And then the other thing I think about is also, I guess, building up from small things to bigger things, where I assume that the general temptation or the typical development is incremental, where everything is human, operated, and then you find ways in which to automate it, and then you sort of build up from there. I worry that sometimes that is the way that people evolve things, but that’s a local minima, optima.
Ekin Doğuş Çubuk [00:50:25]: Like short-horizon optimization.
Liam Fedus [00:50:26]: Right.
Swyx [00:50:26]: Exactly. When actually you should get a humanoid and just put them in there.
Liam Fedus [00:50:31]: We. I think we’re of the opinion that solving humanoids would actually be slower to kind of getting to some of our goals.
Swyx [00:50:37]: Just checking. Again a lot of this is just like you do this every day. We, like
Liam Fedus [00:50:41]: Yes
Swyx [00:50:41]: See this, but we. I don’t know the reality of the situation. A lot of people having humanoids.
Liam Fedus [00:50:46]: Yeah. I think maybe one process is, by having a mix of humans and automation, you can identify really quickly what are some of the bottlenecks in the experimental process, and you start alleviating those bottlenecks one at a time. And so for example, if there’s some really tricky dexterity task that humans are excellent at but you’d have to spend months automating machine, maybe don’t spend a ton of time there. And maybe a more promising thing is something very routine that takes up a huge amount of scientists’ and technicians’ time, and it’s easy to automate. So I think it’there’s this pragmatism to it. But ultimately what we want from the lab is a huge quantity of data, high-quality data, diverse data, and those are our goals. And full autonomy is a non-goal. It’s sort of in the service that we use automation in the service of achieving the goals on the data.
Brandon [00:51:39]: Yeah.
Liam Fedus [00:51:40]: But also, I think another aspect to automation is, robots will just be. Make fewer mistakes potentially in the lab. And it’s been really helpful having AI systems with a full view over all of our data, because sometimes if there’s a permutation in data, then it can identify it. So for example, one of our steps at one point had, a cyclic error because one of the machines was loaded incorrectly, and the patterns were inconsistent. So the AI was reading through these things and says, “Well, given what was run, this is not expected.” And it’s looking at this basket of data longitudinally, and then it realized, “If I do this cyclic permutation and reverse it, everything is consistent.” And then we were able to go back to the physical infrastructure and understand that a mistake had been made in the loading. And I think managing data quality is so foundational to doing AI in the physical world. And these are the types of things that we’re building in. And so you can improve the operating process, but at that point you’re like, “Okay, this is another great opportunity for automation. How do we make this just so reliable, so durable that we never have those types of mistakes again?”
Ekin Doğuş Çubuk [00:52:51]: Yeah, the reducing the noise floor of experimental data I think is very important. Most experimental data in the literature have such a high noise floor that it’s actually usually worse than DFT accuracy. Because, when different people do experiments, different labs, different parts of the world, different times, it really introduces so much fluctuation to the result. So we are hoping that by standardizing these workflows, one of the biggest benefits will be that the noise floor will be lowered.
Open Models, Proprietary Data, and Compute Efficiency
Swyx [00:53:16]: You’ve been pretty public about how you use open source models and fine-tune them for stuff like this. Is it better. Is it basically. I can imagine a situation where, you have maybe the dumber models do one task, and then you optimize for that task, and then sort of like a generalist frontier model that supervises everything. Is that a good mental model to have? Are there more stages to this that I can think about?
Liam Fedus [00:53:37]: We basically use a mix of open source models and closed source models. You know, we’re huge beneficiaries of this from our simulation side, building up the ML stack, the simulation stack. We don’t need to push on that axis. But there’s a lot of areas where, the latency is too high, the cost is too high. But in some cases we can actually. You know, by having access to this data, we can actually push beyond the frontier of what these systems can do under even some the highest reasoning efforts. And so it’s not necessarily just about cost or Pareto efficiency, but in some cases you can push beyond it when you have access to data that no one else has. One, way to characterize this is in terms of compute efficiencies. So by having access to this data, you can be that much more compute efficient compared to some of the frontier models. And that’s been able. Been a really instrumental thing in our program.
Swyx [00:54:33]: Yeah. I’m so. Data quality or data access as a trade-off of compute I think there’s some amount of exchange rate of dollars for compute, for dollars for data that people. I feel like the pendulum might be swinging now towards data. Obviously you would, you would agree with that, but like.
Liam Fedus [00:54:50]: Yeah, especially like, talking about the noise floor of experimental data. We need to have high-quality data. If you throw a huge amount of compute against a bunch of noise, you’re not going to have a good thing emerge.
Swyx [00:55:00]: And it’s harder for you because you actively try to be sparse. You actually try to get null results and learn from that and
Liam Fedus [00:55:06]: That’s right.
Swyx [00:55:06]: Yeah.
Negative Results and Learning From Failed Experiments
Brandon [00:55:07]: Yeah, you’ve Yeah, you’ve talked about null results in several other venues. What does a null result look like for materials, and how do you use that effectively? Especially when I think your overall signal is probably quite sparse usually in terms of success.
Liam Fedus [00:55:19]: A null result could be we intended to produce some structure, and then all of the evidence points to us not producing that structure.
Brandon [00:55:28]: Do you ever intend to not produce a structure? Of course.
Liam Fedus [00:55:30]: Yeah.
Brandon [00:55:30]: Okay.
Liam Fedus [00:55:31]: Absolutely.
Brandon [00:55:31]: Okay. Cool. You do the negative controls
Liam Fedus [00:55:34]: Absolutely, yes.
Brandon [00:55:34]: Regularly. Okay.
Ekin Doğuş Çubuk [00:55:35]: There are impurity cases that will kill your property. Yeah.
Brandon [00:55:39]: Okay.
Ekin Doğuş Çubuk [00:55:40]: Can be toxic.
Liam Fedus [00:55:41]: Yeah, exactly. 100%.
Brandon [00:55:43]: Okay.
Liam Fedus [00:55:43]: And a negative result, in some cases is actually. You know, we intended to make this, the following thing, and then we’re actually able to identify a new structure that hadn’t previously been identified. So it’s. Negative in some respect. We intended to do something else, but something else emerged from the data.
Ekin Doğuş Çubuk [00:56:02]: Yeah. And also in general when you’re training a machine learning algorithm, especially like at a very basic level, if it’s a classification algorithm, if you don’t have negative samples, you can’t really train if everything is positive.
Brandon [00:56:13]: Yeah. Yeah.
Ekin Doğuş Çubuk [00:56:13]: And this is a particularly bad problem in material science because people usually publish crystals they could synthesize, but they usually don’t publish if they fail to synthesize a crystal. Sometimes they might. And also we never know really if a crystal is not synthesizable ever, right?
Brandon [00:56:30]: It could just be a skill issue.
Ekin Doğuş Çubuk [00:56:31]: Exactly.
Brandon [00:56:31]: Yeah. Yeah.
Ekin Doğuş Çubuk [00:56:31]: It could be a skill issue.
Brandon [00:56:32]: Especially with the literature.
Ekin Doğuş Çubuk [00:56:33]: Synthesis method issue, technology. Yeah. So, I think it really helps us when we do our own experiments and get negative results in the context of what we tried, so then we can even train a classification algorithm.
Liam Fedus [00:56:46]: I think there’s also another valid thing too
Brandon [00:56:48]: Yeah
Liam Fedus [00:56:49]: Of when you have this sort of string of negative results and then finally through process iteration, you’re able to get to that positive result, it’s a really interesting set of I’ll call it process engineering-type data. So it’s like through iteration, how did you actually get to that correct result? Because so much of material science has this ambiguity in actually how something was made. And so, yeah. And so there’s actually. Even if there’s a known material, it can be highly non-trivial to replicate that. Some things are, in like, a high school textbook, but other things are really at the frontier and we’re building up that know-how as well. And then building up a system that, given this string of negative results, how do you actually get to that
Brandon [00:57:35]: Yeah
Liam Fedus [00:57:35]: Positive case?
Brandon [00:57:36]: And going to the classifier result, I would almost assume that if you are doing everything in-house, most of your results will actually be negative rather than positive, which is kind of ironic because the literature only gives you positive results. So if you are sort of cold starting this, it seems like the problem is actually the reverse, that you
Ekin Doğuş Çubuk [00:57:52]: Yeah
Brandon [00:57:52]: Have an abundance of negative results and not enough positive.
Ekin Doğuş Çubuk [00:57:56]: And as you said if we were doing per experiment labels, it would be mostly negative.
Brandon [00:57:59]: Yeah.
Ekin Doğuş Çubuk [00:57:59]: But if we were doing per campaign and assuming campaigns end when we succeed, then it could be a bit more balanced.
Training on the Process of Science, Not Just Its Results
Brandon [00:58:08]: I see. So your reasoning traces could almost be over entire campaigns
Liam Fedus [00:58:12]: Absolutely.
Brandon [00:58:12]: And not. Okay.
Liam Fedus [00:58:13]: This is, this is the key thing. This type of data basically doesn’t exist anywhere else, and we spend so much of our time getting the full lineage of the scientific process into the model. And so tracking all this data, like the conversations, the intuitions, what was executed in the lab, where the computations run, where was the code written, stitching all this together is so valuable. And I think the overall goal is rather than training on the final output of science, you’re training on the process of doing science.
Brandon [00:58:44]: This feels very much like you. What you would try to do with RSI, where you are having a model trained to train better models. You’re now having a model trained to make better experiments. But
Liam Fedus [00:58:56]: Absolutely
Brandon [00:58:56]: The difference is, it’s not going back into the core model for. So it’s
Ekin Doğuş Çubuk [00:59:00]: Unless the physics models you’re developing are improving chips, which then improve AI. So this is the wider
Brandon [00:59:07]: The bigger. Yeah, the higher level RSI.
Liam Fedus [00:59:09]: That’s the big data flywheel. Yeah.
Brandon [00:59:10]: But does that mean that. Is it plausible that in Fable 6 or something, or GPT-8 could just have a. Which has been tuned on these sort of higher order, reasoning traces, could actually just do this without any
Liam Fedus [00:59:30]: Mm
Brandon [00:59:30]: Logic? Because it’s sort of the same thinking process, right?
Liam Fedus [00:59:33]: Yeah. I think there’s, decision-making under uncertainty. Obviously in machine learning, there’s noise, in running the AI loops. But we do think there’s a different set of challenges when you’re actually interfacing with the physical world. But also I think there’s another piece too, which is getting the compression of everything into weights is still really valuable. If inference time reasoning was sufficient, all of the frontier labs would have stopped training at GPT-4 and were like, “Okay, every. From now on out, we’re going to get really good at inference time improvements.” and so we think that by. You know, because of the differences between, physical sciences and machine learning, getting that compressed into our own weights will lead to different types of systems and different types of capabilities.
Ekin Doğuş Çubuk [01:00:20]: But also, we feel like even if Fable 7 gets really good at. Even better than what it is today, it will still have to run experiments to get results. And the reason for it is, right machine learning is really good at what it’s been trained on but scientific discovery is almost by definition what you haven’t been trained on. And that’s why we’re building these labs, so that, whether open models or closed models can use these labs to tinker with the universe, because we don’t feel like you can make a big discovery without trying things.
Liam Fedus [01:00:50]: Yeah, there’s
Brandon [01:00:51]: Yeah
Liam Fedus [01:00:51]: There’s not going to be like. No one’s going to zero shot the room-temperature superconductor.
Ekin Doğuş Çubuk [01:00:54]: Think theoretically
Brandon [01:00:55]: That would be pretty cool. Yeah. As someone who has worked closely with wet labs before, certainly I am more skeptical about zero shotting scientific results than some.
Liam Fedus [01:01:05]: Yeah.
Brandon [01:01:05]: But it is something that I think, some people might ask, so.
Liam Fedus [01:01:08]: Yeah. I think it’s really important to distinguish the results we see in math and theoretical physics from the physical world. Right? Like
Brandon [01:01:18]: Yeah
Liam Fedus [01:01:18]: These are two different things.
Brandon [01:01:19]: Two very different things.
Liam Fedus [01:01:20]: Yes.
Ekin Doğuş Çubuk [01:01:20]: I’m not even theoretical physics. So far it’s been theoretical computer science
Liam Fedus [01:01:22]: Yes
Ekin Doğuş Çubuk [01:01:22]: Coding and math.
Liam Fedus [01:01:23]: Right.
Brandon [01:01:24]: Yeah.
Ekin Doğuş Çubuk [01:01:25]: Maybe theoretical physics is next.
Liam Fedus [01:01:26]: Don’t come. Yeah.
Brandon [01:01:28]: Yeah.
Levels of Scientific Abstraction and the Human Role
Ekin Doğuş Çubuk [01:01:29]: Oh, yeah.
Swyx [01:01:29]: Can I get a sort of mental model of the layers, levels of extraction that you can go? So for example, in terms of automation, and I’m just on this theme again where we talked about the campaign, talked about individual essays and running tests and how they’re. Humans are bad at it, so we should stop humans from doing it. Are there others? So for example, one level higher than campaign could be, a physics theory that you’re testing? Or one level lower than campaign is what? And like, I just. I like to think about it from that point of view and Think about it from like, okay, well, this is an API call now. You never have to touch this again. And this one is. Yeah, no, this is still ninety percent human. And maybe we can sort of draw the map of the territory that way. Is there — are there other levels?
Ekin Doğuş Çubuk [01:02:17]: I can tell you some levels. I feel like you asked a good question. I don’t have a very systematic answer, but let’s talk about some levels. So one level is the atomistic structure. There’s an abstraction, right? There’s no perfect atomistic structure in anything we do, but it’s one approximation. Another one is the continuum model. So this one is not atomistic anymore, but it’s, a continuous mesh representing a material.
Swyx [01:02:39]: Okay.
Ekin Doğuş Çubuk [01:02:39]: And then in a different dimension, there’s the thermodynamics. Assuming, you do this experiment forever, what would be the final state? And then there’s another layer of abstraction, which is kinetics, acknowledging that we’re not doing this experiment forever, so time matters. So how quickly will a reaction happen or not, even if it’s lower energy or not? So thermodynamics, kinetics, atomistic, continuum. What else? Are there other levels of abstraction?
Liam Fedus [01:03:04]: Well, I think your point about okay, there could be an overarching new theoretical advance that informs multiple campaigns right?
Swyx [01:03:14]: It makes a difference. Kind of like as an investor, I want to go here’s the bottleneck, guys. And when we get this, we get everything else. I don’t know.
Ekin Doğuş Çubuk [01:03:22]: To me, from the beginning, our hypothesis has been that one of the big bottlenecks is automated characterization, because it’s not that hard to mix powders to get to try stuff. But if you can’t characterize and analyze it and then decide what the next step should be intelligently, you don’t really benefit much from mixing powders randomly. So we focused a lot on automated characterization, closing the loop with simulations, because that’s how you try a lot of things, make an informed decision, and then decide what to do the next day.
Brandon [01:03:51]: Maybe as a follow-up question, what is how do humans live in this loop? Where at what points do you pull drop the human in? I’m guessing you probably
Swyx [01:04:02]: You could, you could randomly drop in at any part
Ekin Doğuş Çubuk [01:04:04]: Of course.
Swyx [01:04:04]: And add value.
Liam Fedus [01:04:05]: Yes.
Swyx [01:04:05]: But where do you decide to spend your time?
Brandon [01:04:08]: It seems like it might be all levels. You have you have people in the lab who are physically moving materials, and then you also have scientists who are guiding decisions about what campaign, and then you have humans who are. What’s that? I’m just asking a question for you. Yeah.
Liam Fedus [01:04:21]: Yeah. One hundred percent. I think it’s always shifting, too.
Brandon [01:04:23]: Yeah.
Liam Fedus [01:04:24]: So, I think even just using the guidance of campaigns, in the early days, that was fully human-driven. Now, increasingly, it’s a mix of AI-driven and human-driven. And then Doğuş was saying early on we identified that characterization was a big bottleneck. In the early days, it was human-driven, and the scientists were very much overwhelmed, and the balance has shifted significantly towards AI-driven. And this has allowed the scientists who were previously spending all of their time doing these refinements to now elevate their work to something else. But I think we’re. It’s really sort of a function of time. There’it’s always changing at each of these levels.
Search, Optimization, and Design of Experiments
Swyx [01:05:04]: Yeah. In terms of just general search, are there things that perform well in let’s say, the physical world that don’t perform well in the other worlds or vice versa? Like so evolutionary search is pretty popular in LLMs. I don’t imagine it works well here.
Ekin Doğuş Çubuk [01:05:21]: So especially in simulations, people have been using evolutionary search, for a long time. It’s pretty good at structure prediction, for example, finding low energy structures.
Swyx [01:05:31]: Okay, so it works. Yeah.
Ekin Doğuş Çubuk [01:05:33]: And then process engineers in semiconductor industry will do DOE, design of experiments, and they’ll often use some kind of zeroth order whether it’s Bayesian optimization or evolutionary search or. I don’t think they use RL, but similar idea, yeah.
Swyx [01:05:47]: You have to have some kind of algorithm
Ekin Doğuş Çubuk [01:05:48]: Yeah
Swyx [01:05:49]: To just figure it out.
Ekin Doğuş Çubuk [01:05:50]: And all zeroth order optimization behaves the same anyway, right? So.
Scaling Labs and Reusing Data for Reinforcement Learning
Brandon [01:05:54]: Yeah. Okay, I want to go back to the question I had earlier, which was one of scaling. And so I-I’m curious, what is your vision for scaling the lab? So there’s different ways. Coming from the biology world there’s a lot of cool tricks you can do in biology. You can tag things, you can use DNA sequencing for all sorts of interesting readouts. And if you can map your complicated assay onto sequencing, you can now scale to millions or hundreds of millions or something. Can you do tricks like this? Ultimately, every assay has its own runtime. I’ll, I’I have a favorite blog post I would want to. I’ll quote in the show notes. But what does the runtimes look like for these labs? Is it. Is your scaling just for mater-materials just fundamentally linear in terms of you do you just need more resources to do more things, or can you paralyze things in clever ways and combine things, and are there tricks there?
Liam Fedus [01:06:52]: Well, I think one way to think about it is, the RL environment is not quite trivially run the experiment, wait for that agent to fully roll out, doing the calculations, going through all the instruments to the final outcome. We basically start producing data across all the different instruments and all these different campaigns. And one way you can begin to expand that for machine learning is now reinforcement learning environments can be constructed based on again dates for the experimental campaigns or computational campaigns at that point. Or you can also, start taking, subsets of instruments, and you’re like, “Okay, we’re going to create reinforcement learning environments on the basis of this instrument alone.” Or maybe it’s, across these different instruments to kind of get to some reward state. So that’s a way where. This finite basket of data, when viewed in different ways, can then be expanded, for training purposes. So it doesn’t get at the actual experiment run, but from an ML perspective.
Ekin Doğuş Çubuk [01:07:55]: Maybe I can give you some analogs for the bio examples you gave. So, one thing people have tried is combinatorial sputtering. I don’t know if you’ve ever heard of this. You have these sputtering targets, and you create this compositional gradient. So when you look at the final product, because there is gradient from each precursor, it creates different crystals at different spatial locations. So in one go, you maybe try hundreds, thousands of crystals. That’s an that’s similar, right, to your example.
Liam Fedus [01:08:23]: Oh, yeah, that makes sense.
Ekin Doğuş Çubuk [01:08:24]: And as far as I know, this hasn’t worked very well so far because it turns out there is diffusivity, so things move around. Another one that we thought about, but we haven’t done yet, but a lot of people talk about this is superconductivity measurements are a bottleneck because unlike XRD, there’s no high-throughput superconductivity measurement, and it can take one hour per measurement. So people have thought about taking the different candidates, mixing them all into one sample, and putting it in. And if, it has superconductivity, you know that one of the sources that. And then you can do a bit like how people are doing COVID testing to. So there are ideas like this exist. Some of them work well, some of them don’t. Yeah.
Custom Hardware, New Labs, and Scaling Ambition
Brandon [01:09:00]: So you, I think, are about to announce a big fundraise. How are you thinking about, scaling?
Liam Fedus [01:09:06]: We’ve been doing our design of our labs for a while, and so I think the resources allow us to continue to scale up the labs significantly, but also the compute, both from the AI side and the computational side. But I think really importantly too, it’s what are the new types of labs we’re able to build? So I think we’ve been proving out this loop, in our initial labs here in Menlo Park, but we’ll be continuing to expand to new types of labs and repeat the same process.
Ekin Doğuş Çubuk [01:09:33]: Yeah, like the kind of lab we’re trying to build, I think haven’t been built before at this scale or at this approach. So we have learned a lot from our own build-ups so far, and those lessons guide us for the next version and the next version. And then we can, as you said increase the scale, the ambition, the quality of instruments. Some instruments are very expensive. As we get a really good understand for what we get a lot of return from we can invest into those more.
Liam Fedus [01:09:58]: And it makes. Hardware engineering has become so core to it. So for example, in the early days, just for speed, we’d buy some off-the-shelf instruments. And as we kind of push the scale of the lab
Brandon [01:10:08]: You make your own.
Liam Fedus [01:10:09]: We have to make our own. So we realize, okay, for this instrument, it’s pretty fast. These components are actually pretty quick, but this weighing machine actually becomes the bottleneck. Or there can be things pertinent to data quality where, well, the resting position of this robotic arm actually is above the plate of previously mixed things, so there’s a contamination risk. So in our hardware design, we’re going to redesign it so that the resting position lies away from the-those things. These are the kind of subtle details that allow us to kind of push the noise floor down for our experimental campaign. So, basically take these learnings and scale it up.
Assembling the Team Behind Synthesis Superintelligence
Brandon [01:10:45]: You ship your org chart. You have many teams. I think the. We were surprised when we looked at your jobs page, and we were like, “Okay, we don’t think we fully map what Periodic does.”
Liam Fedus [01:10:56]: Maybe some organizing principles, so to kind of make sense of the job page. So we hire for AI research and infrastructure, computational, experimental roles, and then hardware engineering roles, and then product roles.
Ekin Doğuş Çubuk [01:11:10]: We want to achieve synthesis superintelligence, right? And we feel like for that, we need the right chemistry expertise, the right physics expertise, simulation theory, and thin films, powder. So hardware engineering these seem like very diverse roles, which they are, but they’re all actually, coherent in what they’re trying to do, which is a synthesis for intelligence. And this also applies to the LLM researchers, the infra, even the product engineers, because the product engineers kind of make sure all the research gets made into a product that the experimentals in the lab can use.
Liam Fedus [01:11:42]: Yeah. So in order to do the end-to-end loop with the physical world, it really requires your ability to have agency over the physical world to build these labs, to run the campaigns effectively, to create AI systems against that, to make them good users of the computational tools. That’s like the difficulty, but also the opportunity of Periodic. It’s like this group of people has just never been brought together before. It’s irreducibly a multidisciplinary problem.
Ekin Doğuş Çubuk [01:12:07]: Yeah. And one other guiding principle for us that is really important for us is we want the people who are very good at what they do and experienced to do it hands-on. You know the modern life has gotten us to this place where, especially in academia, when someone is really good at research in some area, we give them so much responsibility for grant writing, teaching, that they stop having the time to do research in that area hands-on. But if you look back at Bell Labs, IBM, institutions that made really good progress, it was really experienced people doing hands-on work. Like Bardeen was in the lab every day, even though he’s a theorist. Alex Müller was in the lab doing experiments, even though he was the lab lead. So we try to do that here too. So we have some of the world’s leaders in different fields, but they’re doing hands-on work.
Brandon [01:12:52]: Do you want to brag or just call them out? You know.
Ekin Doğuş Çubuk [01:12:54]: Oh, I would love to yeah. So, when I was doing my PhD, my favorite computational material scientist who’s kind of from my age group is Muratan Aykol. He did his PhD with one of our advisors, Christopher Wolverton. And he is kind of like our computational material science expert. He’s incredible. He’s sometimes, a really good experimentalist, even though he does simulations. He understands systems really well. Our lab lead is Joe Checkelsky, who is a professor at MIT, but he’s on leave to work with us full-time. And Joe is this incredible physicist, understands superconductivity really well, but he also was a professor in Japan for a bit, so learned synthesis and chemistry really well, especially for a physicist. We have Daniel Chica, who was a graduate student in Mercury Canasides’ group, which is maybe the best solid-state chemistry group in the world. And Daniel was one of his synthesis experts, so he’s like a magician with his fingers. There’s so many examples like this.
Liam Fedus [01:13:50]: No, I know. I think Dima Bahdanau, so he’s been leading a lot of our AI and LLM efforts. He was the inventor of neural network attention.
Brandon [01:13:59]: Oh, Bahdanau. The Bahdanau attention.
Liam Fedus [01:14:01]: Exactly. Yeah.
Brandon [01:14:03]: Yeah.
Liam Fedus [01:14:03]: Yeah. Yeah, so very hands-on in sort of getting into the traces, the training distribution, how to connect it to the lab.
Ekin Doğuş Çubuk [01:14:10]: The chemistry.
Brandon [01:14:11]: Chemistry.
Ekin Doğuş Çubuk [01:14:11]: He’ll go deep.
Liam Fedus [01:14:12]: He was very deep. He actually has solid-state synthesis books at his lab in Montreal. Ray Nakano, so he used to tech lead some of the operator work at OpenAI. He’s in the lab. So when we were doing some lab tours, we were initially, we’re like, “Oh, this new, scientist, this new technician really looks like Ray.” And no, it was literally this was Ray actually in the lab, with the scientists automating the machines. And it’s like this is sort of the DNA and the type of people we need to kind of pull this off.
Forward-Deployed Engineering for Semiconductor Partners
Swyx [01:14:43]: So, we also saw that you have forward-deployed engineering, roles. We happen to be spinning up a forward-deployed engineering podcast because there’s so many engineers that would do that role. They may not necessarily know that there even is a role for them at Periodic. What is it, and how do they work with customers?
Liam Fedus [01:15:01]: So we’ve been building our own products, our own tools, our own AI and computational
Swyx [01:15:06]: For yourself. Yeah.
Liam Fedus [01:15:06]: For ourselves. So we’ve been customer zero. Now what we’re doing is we’re taking these same tools to different industries, and we’we have a huge focus right now in the semiconductor industry. However, in order to do this these are very, private companies. We need to be able to operate within some of the most secure environments. And so our Periodic forward-deployed engineers and researchers will actually be on site working and actually integrating our AI systems, our computational systems to help our partners get to their goal states more quickly. So basically the tools, the know-how that we’ve built in doing our own materials discovery, materials engineering, we’re now bringing to industry. And we think this is the most effective way to get partners to their end states and to their goals, rather than just throwing some technology over the wall and telling them to figure it out. Then these forward-deployed engineers will also do the inference locally, but then also can, train on the data. So again, once we take our system that understands these different areas and we deploy, we can make it expert on the customers or the partners’ data so that they can own their own intelligence and have systems that understand this much more effectively than just hitting some API or some untrained model. And so this is a much expanded, forward-deployment engineering role than typical because this really requires some very deep, machine learning expertise. You have to be incredibly precise from a data training perspective, infrastructure perspective, but also a physics perspective. So we’re working with highly technical industries. They have to understand, different areas of chemistry, material science, some of the devices. And so that, those are the roles we’re building out right now.
Swyx [01:16:53]: Part of it is willingness to spend extended periods on site in Taiwan. It’s like, okay, well, I know that kind of customer.
Swyx [01:17:01]: It’s interesting that you’re monetizing. That is the primary way that you’re monetizing now, but it might change in the future. But like, that’s like a model that I think people don’t really get about APL Research Lab. It’s. And when you compare yourself to Bell Labs, that’s what people are thinking, right? Which is.
Commercializing Scientific Discovery
Liam Fedus [01:17:19]: I think a really great analog would be software engineering.
Swyx [01:17:22]: Right.
Liam Fedus [01:17:22]: So in the early days of software engineering we had GitHub Copilot, then we had early versions of ChatGPT. People were using these things as copilots to help them get to their solutions. And as the automation improved, now we have things like Codex, and very few of our engineers are writing code the way they used to. And we think a very similar thing could play out for Periodic as well, where you can have systems to accelerate the researchers, the material scientists, the materials engineers, the process engineers. But as the autonomy, intelligence, and capability grows, you can begin to price outcomes. So help me get to this kind of goal state, and I think that’s, a really interesting area for us.
Ekin Doğuş Çubuk [01:18:03]: Yeah. We kind of feel like our best contribution to solid-state physics and science could be if we made these tools and topics profitable, similar to how ChatGPT made CS and LLM majors way more popular in colleges before and after. We’d love to show that solid-state physics, material science research can make a big impact commercially, and then that will attract more attention. Young people will want to study physics, which will be, a dream. And Bell Labs did make a huge commercial impact, right? So they failed to commercialize some of their incredible advances. They, of course, did some research that just couldn’t be commercialized like the cosmic, background. But they did benefit a lot from the vacuum tube connecting, East Coast to West Coast by phone lines.
Liam Fedus [01:18:49]: Yeah. And I think it’s really also interesting to think that technology and capital are incredibly intertwined. So if you look at what was the progress on chatbots for many years imagine recounting, okay, what was the progress from chatbots from I’ll say 2010 to 2015? We’d be kind of at a loss to say over that five-year period, how much did they improve? Whereas if you look at the period from, 2021 to 2026, it’s night and day. And what happened was ChatGPT and these other systems were able to achieve a product market fit, and it changed the capital landscape entirely. This changes the landscape for hiring, compute, data, et cetera.
Swyx [01:19:31]: The virtuous cycle.
Liam Fedus [01:19:32]: Exactly. And so
Swyx [01:19:32]: Like success breeds success.
Liam Fedus [01:19:33]: Exactly. So technology, it’s incredibly coupled to the capital and those resources, and we want to achieve the same thing in the physical world.
Open-Source Contributions and Academic Research
Swyx [01:19:41]: Yeah.
Brandon [01:19:42]: So one thing I’ve been seeing is there’s this move from science being funded from government grants and so on to VCs and private funding. Do you all plan on making anything you’re doing, open source, like either releasing data sets or actual models? Is this something in the future even. You know, there are many models which may be not well, your state of the art, but which could still be you know, useful releases.
Ekin Doğuş Çubuk [01:20:07]: Absolutely. So, we have a couple of things. So we have been contributing to open source to kind of like PyTorch and the Materials Project, code base, Custodian, their DFT runner. TorchSim was actually created by one of the researchers here, Abhijit Ganggan, and he maintains it still. JAX-MD, Abhijit maintains it. So we do a lot of open source contributions, including LLMs.
Liam Fedus [01:20:31]: XLANG, Megatron.
Ekin Doğuş Çubuk [01:20:32]: Yeah.
Liam Fedus [01:20:32]: Yeah. So we’ve been very active contributors back into open source.
Ekin Doğuş Çubuk [01:20:36]: And we also have an academic grant program where we give academic gift grants to academic groups in universities that we feel like are really advancing this direction towards synthesis superintelligence. So that’s also been great to see. I think the first paper from this funding is about to come out, so it’ll be exciting to just get more papers come out like this. Yeah.
The Path to Better Superconductors
Brandon [01:20:57]: Okay. Let’s steelman this, say that you’ve achieved 100% lab automation, which does everything, characterizes everything correctly. You have all of the fun agents which can do everything. I still am somewhat unclear about how you actually get to superconductivity. There’s a lot of hard problems to solve. What is the path there when there is no sort of theory for model for most of the strongly correlated systems or high-temperature systems?
Ekin Doğuş Çubuk [01:21:23]: Yeah. So, our view is that it’s not hard to think of chemical spaces that would host superconductivity. What’s really hard is to synthesize them. That’s why we’re really emphasizing synthesis superintelligence. So maybe one example to give historically is Alex Müller, when he thought that transition metal oxides might be a good path to superconductivity, he was actually trying nickelates, the nickel, oxygen, and some, cations, and that didn’t work. And then he tried cuprates, which is copper, oxygen, and some cations, and that worked, and then he got a Nobel Prize. But then we know that later on turns out nickelates was a good idea. Nickel and copper are next to each other on the periodic table.
Brandon [01:21:59]: But only as two layers, right?
Ekin Doğuş Çubuk [01:22:00]: Yeah. Like there are only so many 3D transition metals.
Brandon [01:22:03]: Yeah.
Ekin Doğuş Çubuk [01:22:03]: And then, Harold Hwang, one of our, Stanford professors here who’s incredible, he realized he can make nickelates in thin film form and show the superconductivity. So, if we have a really good synthesis superintelligence in the lab and we can really scale up these things, I think we won’t run out of ideas, or the LLM won’t run out of ideas about what directions to try, like 3D transition metals, oxygen. There are very related ideas here. Many people have been trying cobaltates, which is like instead of copper or nickel, you try cobalt. So I think that’s super exciting. Maybe one historical example to also study is the Japanese group that discovered magnesium diboride. As you know, that’s the highest temperature, ambient pressure, conventional superconductor, and they discovered it by just trying a bunch of materials. I think they tried 30,000 different things. Thirty of them seemed to host interesting superconductivity, and MgB₂ was one of them. But magnesium diboride sat on people’s shelves as a precursor for decades before then, and BCS theory had been invented back in 1957. So the way people found these materials wasn’t projected from theory. It wasn’t because they couldn’t synthesize until then. It was just like they tried a bunch of things. So now imagine if we had this perfect automation that you’re describing, which sounds like a dream, and we tried the 30,000 the Taketo group tried in over his career in a month. We just really increased the surface area for luck.
Closing: Increasing the Surface Area for Discovery
Swyx [01:23:30]: No, but yeah, it’s very inspiring what you’ve done. Congrats on all your success. I feel, I feel like you’re creating, yeah, the modern playground. I imagine recruiting must be super easy for you, so I’m just a little bit jealous, but you’ve worked hard for it to get here, so yeah.
Ekin Doğuş Çubuk [01:23:44]: Yeah.
Swyx [01:23:44]: Yeah. Well, thank you so much. Yeah, it was great chatting with you both.
Ekin Doğuş Çubuk [01:23:46]: Yeah, super fun. Yeah.
Swyx [01:23:47]: Yeah. Thank you.


This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.latent.space/subscribe