Chinese Edition: 关于强化学习的数学思考
Deep Manifold: A Holistic View
Deep Manifold organizes neural learning through six connected ideas: learning as a forward–inverse process; computation on stacked piecewise manifolds; a residual-based neural-network equation; propertyless activations with sufficiently full-rank relational structure; training as the construction of fixed-point classes; and inference as boundary-conditioned traversal through the resulting neural fixed-point field. Together, these form one mathematical picture linking what is learned, where learning occurs, how convergence develops, and how learned structure is later traversed.
Within this view, architecture is important but not fundamental. Dense networks, MoE, attention mechanisms, residual systems, and other designs are different numerical realizations of learnable computation. The decisive question is whether the chosen architecture provides enough freedom, connectivity, and numerical stability for learning to converge. Architecture therefore acts as a set of constraints on the learning process rather than defining intelligence itself; different architectures can be valid if they support stable convergence toward useful fixed-point structure. Mathematical Considerations for Neural Network Architecture
MoE Instability: Manifold Tearing and Regional Reduce
Mixture-of-Experts architectures distribute computation across many local expert manifolds, gaining capacity and specialization but introducing instability by construction. Routing imbalance, uneven expert participation, and independently evolving local geometries can weaken compatibility between experts and shared pathways. In Deep Manifold, this appears as manifold tearing: local chart inconsistency can become routing discontinuity, then propagate across layers and amplify into training instability. MoE therefore makes coordination among local geometries a central architectural problem. Mathematical Considerations for Training Progression · Single Token Geometry 02: Manifold Tearing
DeepSeek V4 and Kimi K3 address this problem from complementary directions. DeepSeek V4 uses local stabilization mechanisms to limit extreme curvature, improve routing consistency, and contain disturbance propagation before tearing spreads. Kimi K3 highlights the complementary Reduce problem: Map, or local expert computation, is relatively easy; recombining distributed outputs while preserving manifold homology is much harder. Stable MoE training therefore depends not only on expert quality, but on preserving global compatibility across distributed local covers. Mathematical Considerations for DeepSeek V4 · Mathematical Considerations for Kimi K3
RL Progression: Shaping the Neural Fixed-Point Field
Training progression determines not only whether a model converges, but what geometry that convergence leaves behind. Poor progression can produce a jagged neural fixed-point field with steep peaks, deep valleys, narrow passages, fragmented pathways, and weakly connected fixed-point classes. Such geometry makes optimization sensitive to local conditions and can preserve numerical convergence while degrading relational continuity. Training must therefore manage when boundary conditions are introduced, how strongly they act, and how long the manifold evolves under them. Mathematical Considerations for Training Progression
The same geometry governs inference. Because inference follows an iterated-integral path through the neural fixed-point field, jagged geometry can force redirection, correction, backtracking, or switching between fixed-point classes. These costs become especially visible in agentic workloads requiring long chains of reasoning, tool use, and verification. Better RL can consolidate viable pathways, allowing the model to reach the same solution through shorter, more stable trajectories and often fewer reasoning tokens. Neural Network Fixed-Point Field
The goal of RL progression should therefore be to produce a rolling-hill-like neural fixed-point field: piecewise smooth, connected, and traversable, with stable pathways between fixed-point classes. Rather than creating steep peaks and deep valleys, RL should progressively smooth local geometry so inference can follow robust iterated-integral paths with less sensitivity and correction.
The Observable Geometry of Learning
Reinforcement learning is still tuned largely from the outside. We change rewards, data, KL constraints, curricula, learning rates, or training stages, then judge the result through loss, reward, benchmarks, and generated behavior. These signals tell us whether training improved, but not what geometric structure changed inside the model. Deep Manifold introduces the neural fixed-point field as an internal object through which the evolving geometry of learning can begin to become observable. Neural Network Fixed-Point Field
This is the critical step. Instead of seeing only outputs, we can begin to peek inside the model: inspect fixed-point classes, basins, saddles, barriers, connectivity, and the trajectories connecting them. We can ask whether RL is smoothing the field, consolidating useful pathways, creating new discontinuities, or pushing previously compatible regions apart. The fixed-point field therefore connects external training behavior with an internal geometric picture of how learning is actually progressing.
Once this geometry becomes observable, RL training can form a closed loop: set the learning problem → train → inspect the fixed-point field → adjust data, boundary conditions, progression, or architecture → train again. Human judgment can guide the loop initially, and eventually field-aware systems may automate parts of it. The important advance is broader than human-in-the-loop tuning: RL can move from black-box optimization toward training informed directly by the observable geometry of learning.
This article is now listed under Training and Mathematical Considerations Series in Deep Manifold, Two Years Later: 2024–2026.