Why we still don't have foundation models for physics

"We have foundation models for language, not for physics." Anima Anandkumar's line, a professor at Caltech, sums up a tension that rarely comes up in AI discussions centered on LLMs. On the Latent Space podcast episode, she explains why the formula that made GPT and its peers take off simply doesn't apply to continuous physical systems like climate, fusion plasma, and fluid flow, and why that demands a different kind of ML engineering than the one most devs practice today.

Scale doesn't solve everything

The dominant bet in AI in recent years is what the community calls the bitter lesson: throw more data and more compute at a transformer and the model improves. For language, it works, because text tokens exist by the trillions on the internet. For physics, the game is different, for two concrete reasons.

First, the data doesn't exist at the necessary scale. Open datasets in many of these domains have tens or hundreds of thousands of examples, far from the token hunger of a large transformer. Second, and more serious, the resolution physics demands blows past any viable context window. Anandkumar gives the number: "if each dimension has even just a few hundred grid points, which is where industrial scale begins... we're talking about a context of hundreds of billions up to a trillion. Forget having a transformer for anything at that scale, all the compute in the world wouldn't be enough."

In other words, it's not that there's a lack of GPUs. It's that the brute-force approach is mathematically unfeasible for the problem. The path forward, according to her, is to embed structure and inductive biases that physics already offers for free, instead of waiting for data that will never show up.

Neural Operators: learning functions, not grids

The central technique in Anandkumar's work is Neural Operators. The idea, elegant in its simplicity, is to stop modeling a discrete grid of values and instead model a function that evolves across multiple scales. Each layer of the network, instead of mapping vectors to vectors, maps functions to functions. This allows inputs and outputs at different resolutions and the incorporation of physical knowledge as a prior.

The most instructive example is global-scale weather forecasting. Earth is (almost) a sphere. Running a climate model on a regular Cartesian grid makes the error blow up fast. By working in the problem's natural basis, spherical harmonics, the model stays stable for much longer, long enough to project months ahead instead of days. The Fourier Neural Operator learns directly in the frequency domain, and its spherical variant powers FourCastNet 3, which models the climate of the entire planet and runs stably far into the future.

The practical point that matters to builders: FourCastNet was born precisely out of skepticism. When Anandkumar decided to create the first open source AI climate model, experts said it was unfeasible, climate is chaotic, physics-based simulations took decades to mature and require supercomputers. In about a year, the team delivered a model competitive with the best physics-based simulations, one that today allows short-term weather forecasting using consumer GPUs. That's the turn that matters: driving the computational cost of inference for physical systems down from supercomputers to accessible hardware.

The physical world is more forgiving than it looks

Another counterintuitive observation from the work: physical systems require less data than one would imagine, as long as the right structure is embedded. In the case of nuclear fusion, a few thousand samples are enough to predict plasma disruptions, and the model does so about a million times faster than traditional simulation.

This speed gain isn't an academic detail. Predicting plasma disruption in real time only makes sense if it fits inside a control loop, and a control loop doesn't wait for a supercomputer. This is where Neural Operators stop being a mathematical curiosity and become engineering infrastructure.

It's worth noting the honest caveat the source herself makes: none of this rejects scale, it's a different route to get there. Anandkumar still wants to reach a "foundation model for physics," capable of covering many phenomena and doing both simulation and the design of new systems. The difference is that you get there by building structure, not by stacking tokens. And it will take longer than the text-driven part of AI did, because "for the physical world, tokens were never the answer."

Formally verifying neural networks

A development that gets little attention, but is relevant to anyone thinking about putting neural networks into critical systems, is TorchLean. It's a framework that lets you write PyTorch-style networks inside the Lean proof assistant and formally verify them. In other words: instead of relying only on test metrics, you can prove bounds on the network's behavior.

The motivation is direct. If you're going to put a neural network inside the control loop of a fusion reactor, testing on a validation dataset isn't enough, you need mathematical guarantees that the network won't behave out of range. TorchLean is a step in that direction, and points to a future where formal verification and deep learning talk to each other, something that's almost nonexistent in industry practice today.

What this changes for the Brazilian developer

The message for anyone doing ML in Brazil is that there's an entire frontier outside the LLM, agent, and RAG bubble, and it's less saturated. Two concrete points:

• The data entry barrier is lower. If physical systems learn from thousands (not trillions) of examples when the right structure is embedded, academic groups and companies without access to giant data centers have a real chance to contribute. The bottleneck stops being "who has more data and GPU" and becomes "who understands the domain's math."

• Obvious local applications. Short-term weather forecasting on consumer GPUs has direct use in agribusiness, in energy (hydroelectric plants depend on water flow), and in disaster management. Fluid and heat modeling shows up in industry, oil and gas, and sanitation. These are problems where Brazil has its own data and real demand, and where a generic model trained elsewhere won't do.

The honest trade-off: this field demands a math prerequisite that the AI application stack doesn't. Fourier analysis, spherical harmonics, partial differential equations, and operator theory aren't optional here. Anyone who only knows how to call a model API and put together a RAG pipeline doesn't get into this game without studying. It's, in Latent Space's own words, a fine nerd snipe for anyone with a big block of free time, but that's precisely why there's still room.

The episode also notes that Anandkumar was appointed to the UN's Scientific Advisory Board, with the goal of bringing evidence-based views into the formulation of public policy for AI applied to science. A sign that modeling physical systems with AI is moving out of the lab and into the global infrastructure conversation, albeit at a slower, more structured pace than the LLM race.