The Information Bottleneck

John Langford, one of the heads of Microsoft's AI Labs, the creator of Vowpal Wabbit, and a co-inventor of CAPTCHA, joins us to talk about world models. Transformers need orders of magnitude more data than humans to learn the same thing, and John argues a compact, implicit world model is how you close that gap. He explains why he's skeptical of JEPA-style objectives, why a transformer's KV cache is the Ptolemaic epicycle model of belief states, and what his Next Latent work does differently.
We also get into whether research still matters in the age of scale; open versus closed models; agent-driven research after running 2,000 pre-training experiments in 90 days; the origin story of CAPTCHA; and why Muon and orthonormal optimizers actually work.

Topics:
Implicit vs. explicit world models, and the case against JEPA-style objectives
Compact belief states: why compression beats a growing KV cache
Does research still matter in the age of scale? The Kimi K3 argument
Agent-driven research: 2,000 pre-training experiments in 90 days
The invention of CAPTCHA
Optimizers from SGD and Vowpal Wabbit to Muon

Chapters
00:00 Why world models: the sample-complexity gap
09:48 The case against JEPA; a transformer-style implicit world model
15:52 Compact belief states: epicycles vs. heliocentrism
23:41 Does research still matter? The Kimi K3 argument
27:35 Open vs. closed models
35:57 Recursive self-improvement and agent-driven research
42:30 2,000 pre-training experiments in 90 days; weak baselines and reproducibility
54:54 The invention of CAPTCHA
1:00:53 Optimizers: from Vowpal Wabbit to Muon
Music

"Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0

Description
Tabular data is still where most of machine learning actually happens in industry, and the field has changed a lot in the last few years. In this episode we talk with David Holzmüller, a researcher at INRIA and one of the people behind TabArena, TabICL and RealMLP, about what the state of the art looks like right now and how to pick a model for your own data.
We cover the shift to TabPFN-style foundation models that learn to learn from whole tables, why TabArena was built and what earlier benchmarks got wrong, what Google's new TabFM means for the leaderboard, and when gradient boosted trees are still the right tool. David explains why LLMs struggle with tables, shares an early result comparing Claude Opus against TabICL on tiny datasets, and walks through how to embed text columns for tabular models. We also get into time series vs tabular data, the open research problems he thinks matter most, and why classical ML libraries are so bad out of the box.

Links:
TabArena: https://tabarena.ai

Topics
Tabular foundation models and in-context learning on tables
TabArena and Beyond Arena: building a benchmark that stays honest
TabFM, TabPFN, TabICL and the tradeoffs between them
When boosted trees and MLPs still win (large data, CPU, fast inference)
Why LLMs are inefficient on tabular data and where they might help
Embedding text columns with language models
Explainability, calibration and class imbalance
Time series vs tabular data
Open problems: invariances, synthetic data, uncertainty, scaling down
Where the field is heading in the next five years

Chapters
0:00 Intro
0:31 What changed in tabular ML: TabPFN-style foundation models
2:22 Which model to try first? TabArena and how it was built
5:14 What older benchmarks got wrong, and Beyond Arena
8:45 GPU AutoML vs foundation models
10:40 Reading the leaderboard: TabFM, TabPFN, TabICL and the tradeoffs
12:47 Calibration, class imbalance and small vs large data
19:45 Explainability for black-box tabular models
21:34 Why LLMs are bad at tabular data
25:39 Claude Opus 4.6 vs TabICL on tiny datasets
27:51 New classifiers, five-year outlook, real vs synthetic pretraining
33:13 Embedding text columns for tabular foundation models
36:13 Time series vs tabular data
39:59 When gradient boosted trees still win, and feature engineering
45:31 Open research problems and where the field is heading
52:54 Better MLPs and why classical defaults are bad out of the box
Music"Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0

Charles Frye (Modal, ex-Weights & Biases, Berkeley PhD) joins Ravid and Allen to explain why modern AI research is bottlenecked by compute, and why simply buying more GPUs doesn't solve it. We cover the three problems every lab hits (underutilization, saturation, resource sharing), when companies should actually train their own models, why inference is a "bad algorithm" for today's hardware, NVIDIA's monopoly, the OpenAI/Hugging Face hack and what it says about open models, and whether we're in a compute bubble.

Key topics
AI infrastructure challenges and when to train your own models
GPU resource management and virtualization
Inference optimization and speculative decoding
The economics and future of AI hardware
Agents, sandboxing, and open-model security

Chapters
00:00 Intro
01:03 Why AI needs special-purpose compute
03:22 Buying vs renting GPUs: the three problems
07:15 Modal's approach, and doing more with less compute
09:46 Do we actually need to spend more? The conflict-of-interest question
13:08 Should companies train their own models?
14:47 Efficient fine-tuning and prompts as fast weights
17:37 Are we in a compute bubble?
20:21 Why inference will dominate compute (the SQLite analogy)
22:42 Speculative decoding
26:44 Why scaling inference is hard, and neuromorphic hardware
28:36 Why NVIDIA's monopoly persists
33:09 Inference chip startups and the hardware lottery
35:24 How Modal stays hardware-agnostic (GPU snapshot restore)
38:45 Will agentic coding erode CUDA's moat?
41:18 Running one agent vs thousands: sandboxing at scale
46:27 The OpenAI/Hugging Face hack and open models as defenders
52:28 Rogue AI, self-replication, and fast takeoff
56:09 What's next: evals, embodiment, edge inference
1:00:27 Modal is hiring (modal.jobs)
Music"Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0

Stella Biderman, Executive Director of EleutherAI, joins us the week an OpenAI model autonomously broke out of its sandbox and hacked Hugging Face. Stella calls it what she thinks it is, an offensive cyber operation, and argues it's part of a pattern: this is not the first containment failure at a frontier lab, and sandboxes have failed basically every time they've been tested for real.
So we spend a good chunk of the episode on what actual containment would look like. Stella's argument is that the tools already exist, the labs just don't use them: run dangerous capability evals on air-gapped networks with no route to the public internet, put the most sensitive testing in SCIF-style secure facilities, and treat model evaluation the way the security world treats classified systems rather than the way startups treat staging environments.
And yet Stella remains one of the world's most prominent open-source advocates. From her perspective, the biggest risk isn't the technology; it's unchecked corporate power, and the only durable check on it is an independent scientific research establishment that doesn't depend on the AI industry for its funding or its facts.
From there the conversation spans the geopolitics of Chinese open models and whether governments can restrict them, sovereign AI and what it would actually take for other countries to train their own models, why harnesses and UX drive more of AI's perceived progress than raw intelligence, the AI-found counterexample to the Jacobian conjecture, and EleutherAI's "Deep Ignorance" approach to making open-weight models safe by filtering hazardous knowledge out of pretraining.

key topics
AI governance and regulation
Cybersecurity incidents involving AI models
Open source AI safety and security
The role of independent research in AI safety
Legal and ethical considerations in AI development

Timeline
00:13 — Intro: Stella Biderman and EleutherAI, a real non-profit in AI
02:05 — News of the week: Kimi K3, and OpenAI's model autonomously hacking Hugging Face
05:49 — "Frontier labs can't be trusted": repeated containment failures, air-gapped networks and SCIFs vs. sandboxes
22:45 — Can governments ban open or Chinese models? Import restrictions and the six-month open/closed gap
27:05 — Why Stella is still pro-open-source: unchecked corporate power as the real danger
31:11 — The opioid epidemic analogy: avoiding both regulatory failure and overcorrection
34:57 — Offense vs. defense: why open access to AI has empirically favored defenders
37:28 — Chinese labs, the CCP, and why safety and fine-tuning are low-prestige work in China
42:19 — Sovereign AI: does every country need its own foundation model?
49:29 — Sampling, harnesses, and why ChatGPT was really a UX breakthrough
54:09 — AI solves the Jacobian conjecture: domain data beats raw intelligence
58:02 — Safety is contextual, not a model property — and what HAL 9000 got right
1:01:42 — Is Stella optimistic about the future?
1:02:50 — Deep Ignorance, the science of AI training dynamics, and how to get involved with EleutherAI

Music"Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0.

In this episode, Frank Hutter joins us to talk about TabPFN and why tabular data is suddenly the hottest problem in deep learning. Frank is a professor at the University of Freiburg and spent 15 years building the AutoML field before founding Prior Labs, which SAP just acquired for over a billion dollars.
We get into why deep learning failed on tables for a decade and what in-context learning changed, how TabPFN is trained entirely on synthetic data, and why a model that never saw a real time series ended up beating specialized forecasting models. Frank also explains the architecture tricks behind scaling from 10,000 to a million rows, where LLMs fit into data science (and where they embarrassingly don't), and what happens to XGBoost from here.
Beyond the research, Frank talks about the jump from professor to co-CEO, why he refused to merge his 45-person team into SAP's 110,000 employees, the open-weights licensing debate, and the case for building a frontier lab in Freiburg rather than San Francisco.
key topics
The role of foundation models in tabular data
Impact of SAP acquisition on Pro Labs
The evolution of AutoML and hyperparameter optimization
Challenges and solutions for large context in models
Open source models and licensing strategies
The importance of independence for startup agility
Future directions in AI for science and medicine
00:00 Intro
00:34 The SAP acquisition and staying independent
07:39 Why tabular data is the next big thing in deep learning
14:19 What makes tabular data hard
19:14 AutoML, AutoGluon, and fifteen years of hyperparameter tuning
28:27 Scaling TabPFN: context limits and architectures
34:35 Agentic data science and LLMs
39:30 Online learning, time series, and Bayesian inference in a forward pass
47:05 Open weights and the license debate
54:51 Will LLMs and tabular models merge?
1:00:01 From academia to startup
1:09:42 Why build in Europe
1:12:53 Audience questions and hiring

Music"Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0.