Every model so far was trained from scratch on one problem's data. In language and vision, the dominant recipe is different: train one large model on a very broad dataset (pretraining), then adapt it to a specific task with a little data (fine-tuning). Physics is trying the same. A model that has seen many flows, plasmas and reaction–diffusion systems may have learned something general about transport, waves and boundary layers, and need fewer of your expensive runs.

What pretraining data looks like

The Well (Polymathic AI, 2024) collects about 15 TB of simulations across many physics in one format, with data loaders and baselines. Models pretrained on such collections usually work on regular grids and cut each field into patches, small square blocks of cells that each become one token, as images do in vision transformers. PDE-Transformer (Holzschuh and colleagues, 2025) is a transformer of this kind, built for physics on grids; pretrained once, it fine-tunes to new tasks better than the same model trained from scratch.

Fine-tuning, step by step

1. Load the pretrained weights.
2. Replace the output layer (the head) with one for your quantities.
3. Freeze the rest and train only the head, with a small learning rate.
4. If validation keeps improving, unfreeze everything and continue with an even smaller learning rate.

One contract to respect: feed inputs normalised with the pretrained model's own statistics and in its channel order. A new input channel needs a new or widened embedding layer, just as a new output needs a new head.

import torch
from torch import nn

class Model(nn.Module):
    def __init__(self, c_in=4, d=256, c_out=4):
        super().__init__()
        self.encoder = nn.Sequential(
            nn.Linear(c_in, d), nn.SiLU(),
            nn.Linear(d, d), nn.SiLU())
        self.head = nn.Linear(d, c_out)
    def forward(self, x):
        return self.head(self.encoder(x))

# Stands in for a released checkpoint.
torch.save(Model().state_dict(), "pretrained.pt")

# 1. Start from the pretrained weights.
model = Model()
model.load_state_dict(torch.load("pretrained.pt"))
# 2. A new head for your output (e.g. Cp).
model.head = nn.Linear(256, 1)
# 3. Freeze the encoder at first.
for p in model.encoder.parameters():
    p.requires_grad_(False)
trainable = [p for p in model.parameters() if p.requires_grad]
opt = torch.optim.AdamW(trainable, lr=1e-4)

# Your few runs, normalised with the pretrained model's
# statistics and in its channel order.
x, y = torch.randn(64, 4), torch.randn(64, 1)
for _ in range(100):       # train the head, watch validation
    loss = nn.functional.mse_loss(model(x), y)
    opt.zero_grad()
    loss.backward()
    opt.step()
# 4. Optionally unfreeze everything, smaller learning rate.
for p in model.encoder.parameters():
    p.requires_grad_(True)
opt = torch.optim.AdamW(model.parameters(), lr=1e-5)

Generative surrogates, in brief

Most surrogates predict one field per input. A generative surrogate learns a distribution of fields instead, and draws samples from it. Diffusion models learn to remove noise step by step, turning random noise into a plausible field; flow matching learns a velocity field that carries noise to data along smooth paths, with fewer steps (a velocity in the space of possible fields, not a flow velocity). For physics this gives two things: a spread of plausible answers where the flow is chaotic (turbulence, separation), and rollouts that often stay stable longer than those of one-step models (time stepping and rollouts). The price is several network evaluations per sample instead of one.

Judging a pretraining claim

• Same fine-tuning data. Compare with the same architecture trained from scratch on exactly the data used for fine-tuning, tuned with equal care.
• Held-out physics. Test on equations, geometries or regimes that were not in the pretraining set; otherwise the model may simply have seen your problem before.
• Compute. Count the pretraining cost. A gain that needs a thousand GPU-days of pretraining may not pay for one product line.

The idea is attracting large investments, under names such as "Large Physics Models": PhysicsX, Vinci for chip design, and BeyondMath for generative physics. Use the checklist above on their claims as on any paper.

Every model in this course started from a sampled stand-in for the geometry: a grid, a mesh or a point cloud. But a design begins as CAD, and making that stand-in is often slow, fragile and partly manual. It also cuts the link between the parameters an engineer edits and what the network sees. CAD-native models read the CAD itself.

What CAD stores

A CAD surface is assembled from patches, most often NURBS (non-uniform rational B-splines). A patch is defined by a grid of control points, which pull the surface towards them; a weight per control point, which sets how strongly; and two knot vectors, which say how the patch's two parameters u and v are divided into spans. The surface is exact at any resolution, and a handful of numbers describes a smooth piece of a nacelle or a car roof.

One token per patch

CANTO (NVIDIA, 2026) turns each patch into one token (tokens): the control points with their weights, and each knot vector, are embedded and pooled (averaged over their entries, so patches of any size give the same size of token), concatenated and passed through an MLP. The patch tokens together describe the whole part. A query decoder (decoders) then predicts surface pressure and shear, or volume velocity, at any requested point, without re-encoding the geometry.

import torch
from torch import nn

def mlp(i, o, h=128):
    return nn.Sequential(nn.Linear(i, h), nn.SiLU(),
                         nn.Linear(h, o))

class PatchTokenizer(nn.Module):
    def __init__(self, d=128):
        super().__init__()
        self.pw = mlp(4, d)    # control point + weight
        self.u, self.v = mlp(1, d), mlp(1, d)  # knots
        self.out = mlp(3 * d, d)

    def forward(self, ctrl, w, ku, kv):  # one patch, any size
        # Embed each control point with its weight, then pool.
        pw = torch.cat([ctrl, w[:, None]], -1)
        pw = self.pw(pw).mean(0)
        u = self.u(ku[:, None]).mean(0)  # U knot vector
        v = self.v(kv[:, None]).mean(0)  # V knot vector
        return self.out(torch.cat([pw, u, v]))  # one token

# Cubic knot vectors: one span, and two spans.
k4 = torch.tensor([0, 0, 0, 0, 1, 1, 1, 1.0])
k5 = torch.tensor([0, 0, 0, 0, 0.5, 1, 1, 1, 1.0])
# (control points, weights, U knots, V knots)
patches = [
    (torch.rand(16, 3, requires_grad=True),
     torch.ones(16), k4, k4),
    (torch.rand(20, 3, requires_grad=True),
     torch.ones(20), k5, k4),
]
tok = PatchTokenizer()
tokens = torch.stack([tok(*p) for p in patches])  # (P, d)
# The sum stands in for a predicted drag.
tokens.sum().backward()
# A gradient for every control point of the CAD.
print(tokens.shape, patches[0][0].grad.shape)

Gradients for the design

Remeshing is a discrete step: a small change of the shape changes which cells exist, so no gradient flows through it. Morphing a mesh of fixed topology is differentiable, and is how adjoint CFD optimisation already works, but it limits how far the shape can change. Without a mesh, the chain from design parameters to control points to tokens to predicted drag is differentiable end to end. The optimisation of the previous unit (optimising on a surrogate) then works on the CAD parameters themselves.

Further on this site: CANTO on Papers and CANTO in the news, with figures from the project page. Background: NURBS ↗.

You can now read the architecture section of almost any surrogate paper. This last lesson is about the rest: deciding what a result actually shows. Field surrogates are a young area with few standard benchmarks, and a 2024 review (McGreivy and Hakim, Nature Machine Intelligence ↗) found that weak baselines and selective reporting are common in machine learning for fluid PDEs. Eight questions catch most problems.

1. Task and data. What goes in, what comes out, how many runs? How was the test set chosen: random runs from the same design family (interpolation) or genuinely new shapes or conditions (extrapolation)? Split by design, never by point (train, validation and test), and check that near-duplicate geometries do not sit on both sides of the split.
2. Training data quality. Were the simulations checked: mesh convergence, converged residuals, failed or unsteady runs removed, one setup for all runs? How does the surrogate's error compare with the solver's own error against experiment (is the training data right?)? A 1 % field error means little next to a 5 % RANS drag error.
3. Metric. Relative L2 of which field, averaged how (per point, per run, then over runs)? Are integrated quantities such as lift and drag reported, and the worst case as well as the mean (measuring error)?
4. Baselines. Which models are compared, and were they tuned with the same effort as the new one? Is a simple baseline included (a GP, a neural field, a U-Net)? Are baseline numbers rerun or copied from other papers with different settings?
5. Ablations. When a part of the model is removed or replaced, how much does the result change? This tells you which idea does the work.
6. Compute and size. Parameters, training time, GPUs, inference time, and the cost of generating the data. A 2 % gain at ten times the compute is a different result from a 2 % gain at equal compute.
7. Physics. Are physical checks reported (conservation, forces, symmetry), or only field error?
8. Code and data. Are they released, so the numbers can be reproduced and the model tried on your problem?

Worked example: does Transolver need attention?

Take Does Transolver really need a Transformer? (Wen and Mishra, 2026), introduced in attention at scale. Through the checklist:

• Task and data: nine 3D fluid benchmarks, the kind of data Transolver itself is evaluated on, so the comparison is on Transolver's home ground.
• Ablations: the paper is built from them. Replace the attention among slices with a constant linear map: accuracy barely moves. Remove the slicing and deslicing: accuracy collapses. Only an ablation can separate the two parts this cleanly.
• Interpretation: the authors explain the result with the theory of averaging neural operators: the global averaging through slice tokens is what spreads information across the mesh. That predicts that simpler, cheaper models should do as well, which is a claim the next papers can test.
• What to check yourself: whether the result holds on your kind of data, with the metric you care about, and at your mesh size.

A second example to try alone: compute-optimal multi-fidelity training (from cheap and expensive data). Ask what the budget counts, which fidelities were mixed, and how the error curves were measured.

Where to go from here

The course ends here, and the site's two other sections take over. Papers explains the work that shaped the field, each with what it changed and the lesson that prepares you for it. News follows new models, releases and companies as they appear. To build your own surrogate, start with Get started in Reference, and the frameworks and datasets pages beside it.