A network cannot read a mesh. Each point (or patch, or cell) is turned into a token: a vector of numbers that describes where it is and what it carries.

You have already built tokens without the name. The neural field took one surface point, its position and the flow condition (α, Λ), turned the position into Fourier features, and fed the result to an MLP. A token is the same idea made explicit: one vector of d numbers per point (or per patch, or per cell), the unit that the models of this unit pass around.

From raw features to an embedding

The raw features of a point are the numbers you know: x, y, z, the surface normal, the distance to the wall, the flow condition, and Fourier features of the position (sines and cosines at several frequencies, as in the neural-field lesson). An embedding is a small MLP applied to every point separately that maps these raw features to d numbers, typically 64 to 512. Nothing is shared between points yet: the embedding only puts every point into the same learned d-dimensional space, where the next layers can compare and mix them.

import torch
from torch import nn

N, d = 8000, 128                 # surface points, token size
xyz = torch.rand(N, 3)           # positions, normalised
normal = nn.functional.normalize(torch.randn(N, 3), dim=-1)
# alpha = 4 deg and sweep = 28 deg, normalised as in the
# neural-field lesson, the same for every point.
cond = torch.tensor([(4.0 - 3) / 5, (28.0 - 28) / 8])
cond = cond.expand(N, 2)

# Fourier features with 12 random frequencies.
B = torch.randn(3, 12) * 4.0
fx = xyz @ B
raw = torch.cat([xyz, normal, cond, fx.sin(), fx.cos()],
                dim=-1)          # (N, 32)

# The embedding: one small MLP applied to every point.
embed = nn.Sequential(nn.Linear(raw.shape[-1], d),
                      nn.SiLU(), nn.Linear(d, d))
tokens = embed(raw)              # (N, d): one token per point
print(raw.shape, tokens.shape)   # (8000, 32) (8000, 128)

In practice, normalise the raw features first (normalising), as the snippet does for α and Λ, and choose the Fourier frequencies to match the smallest feature you need to resolve. The neural-field lesson used fixed frequencies doubling at each octave (as in NeRF); here they are drawn at random, the other option studied by Tancik and colleagues. Both work; the scale of the frequencies matters more than how they are chosen.

Where a token is: positional encodings

The attention layers of the next lessons mix tokens without knowing where they are, unless the tokens say so. Any way of putting position into the token is a positional encoding. Coordinates with Fourier features, as above, are the usual choice for meshes and point clouds. Language models and several recent surrogates use rotary position embeddings (RoPE) instead: rather than adding position to the token, they rotate each query and key by angles that depend on position, so the attention score between two tokens depends on their relative position. Grid models make one token per patch of cells; CAD-native models make one per surface patch (the CAD lesson).

Each token asks a question (a query), advertises what it contains (a key) and offers information (a value). It then gathers the values of the tokens whose keys best match its query.

A token on its own knows only its own point. To predict the pressure at the trailing edge, the model has to use what happens at the leading edge, the tip and the root. Self-attention lets every token gather information from every other token in one step, with weights it computes from the tokens themselves.

Queries, keys and values

Each token x is multiplied by three learned matrices, giving three new vectors: a query q = x·W_Q (what this token is looking for), a key k = x·W_K (what it offers to be found by) and a value v = x·W_V (what it passes on). They are three linear layers without bias, applied to every token. The match between token i and token j is the dot product of i's query with j's key, divided by √d to keep the numbers in a sensible range.

The scores are turned into weights with softmax: exponentiate each score and divide by the sum, so the weights are positive and add up to 1. Scores [2, 1, 0] become weights [0.67, 0.24, 0.09]. Each token's output is the weighted sum of all values.

import torch
import torch.nn.functional as F

N, d = 1000, 64
x = torch.randn(N, d)                      # N tokens
Wq, Wk, Wv = (torch.randn(d, d) / d**0.5 for _ in range(3))

q, k, v = x @ Wq, x @ Wk, x @ Wv           # three linear maps
w = torch.softmax(q @ k.T / d**0.5, dim=-1)  # (N, N)
out = w @ v                                # (N, d)

# The built-in does the same without storing w.
same = F.scaled_dot_product_attention(q[None], k[None],
                                      v[None])[0]
print(torch.allclose(out, same, atol=1e-5))  # True

The figure beside the text uses toy numbers: eight regions with hand-made two-dimensional embeddings, so you can see a weight matrix and how one row reads. In a trained model, d is 64 to 1,024 and the weights are learned. The race-car view in the industrial example is a qualitative illustration of the same idea, not a computed result.

Several heads

One set of weights can express one kind of relation. Multi-head attention runs h attention operations in parallel, each with its own smaller W_Q, W_K, W_V (size d/h), and concatenates their outputs. One head might follow the flow direction, another pair upper and lower surfaces. The cost is about the same as one head of full size.

The transformer block

A transformer is a stack of identical blocks. Each block does two things, each wrapped in a residual connection (add the result to the input, as in graph networks) and preceded by layer normalisation, which rescales each token's d numbers to zero mean and unit variance so that deep stacks train stably:

1. multi-head self-attention, where tokens exchange information;
2. an MLP applied to every token separately, where each token processes what it gathered.

import torch
from torch import nn

class Block(nn.Module):
    def __init__(self, d=128, heads=8):
        super().__init__()
        self.n1, self.n2 = nn.LayerNorm(d), nn.LayerNorm(d)
        self.attn = nn.MultiheadAttention(
            d, heads, batch_first=True)
        self.mlp = nn.Sequential(nn.Linear(d, 4 * d),
                                 nn.SiLU(), nn.Linear(4 * d, d))

    def forward(self, x):                  # x: (B, N, d)
        h = self.n1(x)
        # Tokens exchange information...
        a = self.attn(h, h, h, need_weights=False)[0]
        x = x + a
        # ...then each token is updated on its own.
        return x + self.mlp(self.n2(x))

print(Block()(torch.randn(2, 1000, 128)).shape)

The catch: N²

The weight matrix has N × N entries. For the aircraft's 8,000 points that is 64 million per head, manageable. For a car surface with 1,000,000 points it is 10¹², about 2 TB in 16-bit numbers for one head of one layer. Memory-efficient kernels such as the one behind scaled_dot_product_attention avoid storing the matrix, but the compute still grows as N². The rest of this unit is about getting around it.

In cross-attention, the queries come from one set of tokens and the keys and values from another. Surrogates use it twice: to read the geometry in, and to read the field out.

In self-attention the queries, keys and values all come from the same tokens. In cross-attention the queries come from one set and the keys and values from another. The formula is unchanged; only the inputs differ. If there are M queries and N keys, the weight matrix is M × N.

Reading in: a few latent queries

Take M = 256 learned vectors, the latent queries, and let them attend to all N mesh tokens. Each latent becomes a weighted summary of the points it chose to read. The cost is N × M instead of N²: for a million points, 2.6 × 10⁸ weights instead of 10¹². This is the Perceiver pattern, named after a general-purpose model (Jaegle and colleagues, 2021) that used it to read images, audio and point clouds into a small fixed set of latents.

import torch
from torch import nn

B, N, M, d = 2, 50_000, 256, 128
# N point tokens: the keys and values.
points = torch.randn(B, N, d)
# M learned queries, M much smaller than N.
latents = nn.Parameter(torch.randn(M, d) * 0.02)
xattn = nn.MultiheadAttention(d, num_heads=8,
                              batch_first=True)

q = latents.expand(B, M, d)     # same queries for each case
z, w = xattn(q, points, points)
# Weights w: (B, M, N), not (N, N).
print(z.shape, w.shape)         # (2, 256, 128) (2, 256, 50000)

Reading out: any point is a query

The same mechanism works in reverse. To know the field at a location, embed its coordinates into a token and use it as a query that attends to the latent tokens. The query can be any point: on the surface, in the volume, on a mesh the model never saw in training. This is how many current latent surrogates produce their output (Transolver, in a later lesson, decodes point by point instead), and it is the subject of decoders.

Because inputs and outputs are just sets of points, such a model can be trained on one mesh and evaluated on another. That is the same resolution independence the neural operators lesson discussed, with the same caveat: it runs on any mesh, but accuracy still depends on what the training data covered.

Industrial meshes have millions of cells. Modern surrogates encode them into a small, fixed set of latent tokens, do the heavy computation there, and decode back to any point.

You have met this idea before in linear form. POD writes a field as a mean plus a weighted sum of a few modes, so a whole Cp distribution is summarised by, say, ten coefficients. A surrogate then only has to predict ten numbers. A latent space is the learned, nonlinear version: an encoder compresses the input into M tokens of d numbers, the model works there, and a decoder reconstructs the field.

Encode, process, decode

1. Encode: embed the N points as tokens, then cross-attention from M learned latent queries (the previous lesson). Cost N × M.
2. Process: a stack of transformer blocks on the M latent tokens only. Cost M², independent of the mesh.
3. Decode: each output location becomes a query that cross-attends to the latent tokens. Cost Q × M for Q query points.

import torch
from torch import nn

class LatentSurrogate(nn.Module):
    def __init__(self, f_in, d=128, M=256, L=4, heads=8):
        super().__init__()
        self.embed = nn.Linear(f_in, d)   # points to tokens
        self.query = nn.Linear(3, d)      # coords to queries
        self.latents = nn.Parameter(torch.randn(M, d) * 0.02)
        mha = lambda: nn.MultiheadAttention(
            d, heads, batch_first=True)
        self.enc, self.dec = mha(), mha()
        layer = nn.TransformerEncoderLayer(
            d, heads, 4 * d, batch_first=True, norm_first=True)
        self.process = nn.TransformerEncoder(
            layer, L, enable_nested_tensor=False)
        self.head = nn.Linear(d, 1)

    def forward(self, feats, xyz_out):
        # feats: (B, N, f_in); xyz_out: (B, Q, 3)
        t = self.embed(feats)
        lat = self.latents.expand(len(t), -1, -1)
        z = self.enc(lat, t, t)[0]        # encode: (B, M, d)
        z = self.process(z)               # process M tokens
        q = self.query(xyz_out)
        # Decode at any points: (B, Q, 1)
        return self.head(self.dec(q, z, z)[0])

model = LatentSurrogate(f_in=8)
feats, xyz = torch.randn(2, 20_000, 8), torch.rand(2, 5_000, 3)
print(model(feats, xyz).shape)            # (2, 5000, 1)

Time in latent space

For transient problems, the processor can predict the next latent state from the current one, and the rollout (time stepping and rollouts) happens entirely in latent space. Decoding is needed only at the times you want to look at, which makes long rollouts much cheaper than stepping the full mesh.

Further on this site: UPT on Papers. Emmi AI, the company behind UPT and AB-UPT, is in the news: Mistral AI acquires Emmi AI.

Plain attention is quadratic. Recent architectures change what attends to what, so cost grows linearly with mesh size.

Plain attention costs N². Latent models (previous lesson) avoid it by compressing first. Transolver (Wu and colleagues, 2024) takes another route that has become a standard baseline for industrial meshes: group the points by physical state, and let the groups attend to each other.

Physics attention, step by step

1. Slice. A linear layer gives every point a score for each of M slices (M = 32 or 64); a softmax over the M scores turns them into weights that sum to 1. A point in the stagnation region might belong 80 % to one slice, a point in the wake mostly to another. Slices are not regions of space: points far apart in a similar flow state share a slice.
2. Slice tokens. Each slice token is the weighted mean of all point features, weighted by membership. N points become M tokens. Cost N × M.
3. Attend. Ordinary multi-head self-attention among the M slice tokens. Cost M², tiny.
4. Deslice. Each point receives the mix of slice tokens given by its own weights. Cost N × M again.

import torch
from torch import nn

class PhysicsAttention(nn.Module):
    def __init__(self, d=128, M=64, heads=8):
        super().__init__()
        # A score per point for each of the M slices.
        self.to_slices = nn.Linear(d, M)
        self.attn = nn.MultiheadAttention(
            d, heads, batch_first=True)

    def forward(self, x):                 # x: (B, N, d)
        # (B, N, M): each point's weights sum to 1.
        w = self.to_slices(x).softmax(dim=-1)
        # Slice tokens: weighted means of points, (B, M, d).
        s = w.transpose(1, 2) @ x
        s = s / (w.sum(dim=1)[..., None] + 1e-6)
        # Attention among the M slice tokens only.
        s = self.attn(s, s, s, need_weights=False)[0]
        # Deslice: back to every point, (B, N, d).
        return w @ s

x = torch.randn(2, 100_000, 128)
print(PhysicsAttention()(x).shape)        # (2, 100000, 128)

A Transolver model is a stack of transformer blocks in which this layer replaces ordinary self-attention. The real implementation adds multiple heads for the slicing and a learnable temperature in the softmax, which controls how sharply points are assigned.

Million-point meshes and what really matters

Transolver++ (Luo and colleagues, 2025) sharpens the slice assignment locally and splits the points of one mesh across several GPUs, which only needs the small slice tokens to be shared, reaching meshes of over a million points. Other linear-cost routes include linear attention (GNOT and others), which replaces the softmax with a normalisation applied to queries and keys separately, so the product (Q·Kᵀ)·V can be regrouped as Q·(Kᵀ·V), a small d × d matrix, and the N × N matrix is never formed; latent compression (UPT); and anchor points (AB-UPT, in decoders).

Two descendants appear in the news. GeoTransolver (NVIDIA) adds cross-attention from the slices to a shared description of the geometry and the operating condition, reused in every block. FLARE routes attention through a small set of latent tokens in each layer, with two standard attention calls, and scales to million-point meshes on one GPU. Both became stable models in PhysicsNeMo 26.08.

A 2026 study asked which part of Transolver does the work. Replacing the attention among slices with a fixed linear map kept the accuracy on nine 3D fluid benchmarks, while removing the slicing and deslicing made it collapse. The global averaging through slices, not the attention, carries most of the benefit. The authors connect this to averaging neural operators, a theory showing that mixing every point with a few learned averages over the whole domain is already enough for a neural operator to approximate a wide class of PDE solutions. That is the kind of result an ablation gives, and it is the worked example of the last lesson, reading a surrogate paper.

Further on this site: Transolver, Transolver++ and Does Transolver really need a Transformer? on Papers.

The decoder turns latent tokens into numbers engineers use: surface pressure, wall shear, velocity in the volume. Its design decides accuracy where it matters.

Whatever happens inside, a surrogate must end with numbers an engineer can use: pressure and shear on the surface, velocity in the volume. The decoder produces them. Most current models use a query decoder: give it a coordinate, and it returns the field there, reading from the model's internal state.

The first query decoder: DeepONet's trunk

DeepONet's trunk was already a query decoder: a neural field whose output the branch weights for each case. A cross-attention decoder (cross-attention) generalises it: each location reads different parts of the internal state instead of one fixed weighting.

Anchors: attention where it pays

AB-UPT (Emmi AI, 2025) splits the points in two. A few thousand anchor points, sampled from the mesh, carry the expensive attention among themselves. The remaining millions of query points attend only to the anchors, so their cost is linear in their number and they can be processed in chunks. Separate branches handle surface and volume, which need different resolutions, while sharing the geometry encoding.

import torch
from torch import nn

B, N, A, d = 1, 200_000, 4_000, 128
# All surface and volume points, embedded as tokens.
tokens = torch.randn(B, N, d)
idx = torch.randperm(N)[:A]       # A random anchor points
anchors = tokens[:, idx]          # (B, A, d)

mha = lambda: nn.MultiheadAttention(d, 8, batch_first=True)
self_attn, cross_attn = mha(), mha()
# Pressure and three velocity components.
head = nn.Linear(d, 4)

# A x A attention among the anchors only.
a = self_attn(anchors, anchors, anchors,
              need_weights=False)[0]
anchors = anchors + a
# Any points we want answers at read the anchors: Q x A.
queries = tokens[:, :50_000]
r = cross_attn(queries, anchors, anchors,
               need_weights=False)[0]
print(head(r).shape)              # (1, 50000, 4)

Physics in the output

The decoder is also where physical laws can be built in. Predicting a stream function or vector potential and differentiating it gives a velocity field with exactly zero divergence, so in incompressible flow mass is conserved by construction. AB-UPT offers such a divergence-free volume formulation. How to do it, and when it is worth it, is the subject of physics in the loss and in the model.

Further on this site: AB-UPT and DeepONet on Papers. CANTO uses a query decoder for both surface and volume fields: CANTO in the news.

Surface quantities (pressure, shear, heat flux) give the loads; volume quantities (velocity, pressure, temperature, vorticity) explain them: wakes, vortices, separation, mixing. Volume prediction is harder: a 3D simulation has 10–1000× more cells than surface points, the regions that matter are small, and the model must be queried anywhere in space.

• Query decoders (previous lesson): predict at any coordinate from a latent state, so surface and volume come from one model.
• Separate branches for surface and volume outputs, each at its own resolution, as in AB-UPT.
• Physics in the output, e.g. divergence-free velocity, to keep volume fields consistent (physics constraints, next unit).

You have now met six families of field surrogates. None is best for everything: the right one depends on how your data is stored, how much of it there is, how much the geometry changes and how large the meshes are. The table puts the questions side by side; the cards below recall how each family works and link back to its lesson.

A sensible first baseline

• A few scalars from a few parameters: no network at all. A Gaussian process or a polynomial, tested on held-out runs.
• A field, one shape family, tens of runs: POD plus a GP on the coefficients, and a neural field. Both train in minutes.
• Many shapes on a 2D grid: a U-Net. On a fixed domain with smooth fields, try an FNO next to it.
• Many 3D shapes on unstructured meshes: Transolver or a latent transformer, from a framework such as PhysicsNeMo or the Neural-Solver-Library rather than written from scratch. Add a MeshGraphNet when the problem is transient or local.

Whatever you choose, keep the simple baseline in the comparison. If a large model does not clearly beat it on held-out designs, the large model is not yet earning its cost.

The families, one by one