Mathematical Considerations for DeepSeek V4
Clamping, Routing Consistency, and the Geometry of Stable MoE Training
Chinese Edition: 关于 DeepSeek V4 的数学思考
In Single Token Geometry 02: Manifold Tearing, I argued that the loss spikes observed in MoE training are not merely optimization failures. They are signs of a topological failure occurring while the model’s internal geometry is still being formed.
The failure cascade was described as:
high local curvature → chart inconsistency → routing-boundary discontinuity → tear amplification → loss spike
DeepSeek V4’s three mitigations each interrupt this cascade at a different stage. SwiGLU Clamping acts before the tear forms by containing extreme activations. Anticipatory Routing intervenes when routing decisions become inconsistent with the evolving representation geometry. Manifold-Constrained Hyper-Connections act after a local tear has occurred by preventing it from being amplified across layers. Together, they form a layered geometric defense rather than three unrelated numerical techniques.
SwiGLU Clamping: Bounding Local Curvature
SwiGLU Clamping is the earliest of DeepSeek V4’s three interventions. It limits the activation outliers that can initiate the manifold-tearing cascade.
As illustrated above, when the linear component or gate becomes excessively large, the local representation geometry can develop a sharp, poorly conditioned spike. A small change in the token state may then produce a disproportionately large expert output.
DeepSeek V4 clips the linear term and caps the gate. In the illustration, the unclamped surface forms a narrow curvature spike, while the clamped surface remains smooth and bounded. Geometrically, clamping acts as a local boundary condition: it prevents singular local deformation while preserving the model’s global nonlinearity and expressivity.
Anticipatory Routing: Preventing Chart Inconsistency
Anticipatory Routing addresses the second stage of the manifold-tearing cascade: the moment when routing decisions become inconsistent with the evolving representation geometry. During training, the backbone and router are updated continuously. If routing is performed on a geometry that is already shifting, the token may be evaluated under one manifold but assigned according to another.
As the illustration shows, this creates a chart inconsistency. On the left, synchronous updating causes the router to use an outdated chart, so a token can be sent to the wrong expert. The problem is not merely a bad routing choice in isolation, but a mismatch between the token’s current geometric location and the routing partition used to interpret it.
Anticipatory Routing restores temporal consistency by routing from the earlier manifold snapshot that produced the token state in the first place. As the illustration shows on the right, routing indices are precomputed and cached so the token is transported using an aligned source chart. In geometric terms, this keeps the routing map synchronized with the manifold and helps prevent local mismatch from becoming instability.
Manifold-Constrained Hyper-Connections: Bounding Tear Propagation
Manifold-Constrained Hyper-Connections (mHC) address the third stage of the manifold-tearing cascade: what happens after a local tear has already formed. Their role is not mainly to prevent the initial discontinuity, but to stop that discontinuity from spreading across depth.
As the illustration shows, an unconstrained residual mapping can amplify a small local separation layer by layer. On the left, the tear grows as it moves through the network. In the center and on the right, the Birkhoff-constrained mHC imposes a doubly stochastic, non-expansive mapping, so the residual transport remains bounded rather than stretching the damage.
As the illustration shows, mHC therefore acts as a containment mechanism. A local tear may still occur, but it is prevented from turning into a large geometric failure across later layers. In this sense, mHC bounds propagation, contains local damage, and stabilizes the residual stream.
A Unified Geometric Defense
The three methods act at different points in one failure process.
SwiGLU Clamping limits the activation outlier that can initiate extreme local deformation.
Anticipatory Routing reduces inconsistency between the routing partition and the evolving representation geometry.
mHC prevents a local discontinuity from being amplified through the residual stream.
None is sufficient alone.
Clamping reduces the probability of extreme activations but cannot prevent every routing error. Anticipatory Routing improves temporal consistency but is activated only under particular training conditions. mHC bounds propagation but does not remove the original routing boundary.
Together, however, they form a cascade interruption:
clamping controls the precondition, routing consistency controls the event, and residual constraints control the aftermath.
This is why DeepSeek V4 can stabilize training without yet possessing a complete theory of the underlying instability. The three methods do not need to solve the entire geometric problem individually. Each blocks one stage through which a local failure would otherwise become global.
Extremely Sparse MoE: Stirring the Pot Without Tearing the Manifold
A extremely sparse MoE can be understood through the metaphor of a soup pot. At each step, only a small number of experts are activated, just as each spoonful contains only a small subset of the ingredients in the pot. This extreme sparsity reduces the number of incompatible transformations acting on a token at the same time. Instead of forcing many conflicting expert geometries into one local dimension, the model constructs a small, piecewise-smooth pathway through a limited expert set. In this sense, very sparse routing can reduce the immediate conditions that produce manifold tearing.
But sparsity alone is not enough; the pot must still be stirred. Routing, long-context continuity, activation constraints, and residual stability must coordinate the sparse expert regions so that they remain parts of one connected representation manifold. Without this coordination, extreme sparsity could produce isolated fragments rather than smooth specialization. When the components work together, however, very sparse MoE becomes a way to avoid manifold tearing: local computation remains selective, while global coordination keeps the learned geometry continuous, smooth, and coherent.
This article is now listed under Open Source Model Series and Mathematical Considerations Series in Deep Manifold, Two Years Later: 2024–2026.