From Sequence Shuffling to Geometric Reconstruction: Posing the Problem

The fundamental contradiction in current large model inference systems is the rigid coupling of computational logic to storage structure. These models forcibly map the attention relationships between tokens onto the contiguous address space of physical memory. This one-dimensional linear assumption, while seemingly efficient for autoregressive generation, fundamentally constrains the system's flexibility. When faced with concurrent multi-path processing, asynchronous data stream fusion, or dynamic context reorganization, every deviation from strictly sequential access triggers memory movement on the order of $O(L \cdot d)$, where $L$ is the sequence length and $d$ is the hidden dimension.

The crux of the problem is not a lack of engineering optimization but a paradigmatic limitation: compressing a high-dimensional coordinate transformation problem into a one-dimensional linked list. The key to breaking this impasse lies in acknowledging that attention computation is inherently dependent on relative positional relationships, not absolute physical addresses. The KV cache can be viewed as a static geometric basis, while the query vector acts as a dynamic viewpoint. By rotating the query to simulate different positional relationships, the computation graph can be dynamically reconstructed over immutable physical data.

The Group Structure of RoPE: Transformation, Not Labels

The power of Rotary Position Embedding (RoPE) lies in its additive and invertible matrix group properties¹. For a query $q$ at position $m$ and a key $k$ at position $n$, RoPE applies positional encoding via an orthogonal rotation matrix $\mathcal{R}(m)$:

The key properties of the rotation matrix satisfy the algebraic structure of a Lie group²:

This means that the attention score depends solely on the relative displacement $n-m$, irrespective of absolute coordinates¹. The position ID itself is a free parameter in the choice of coordinates and can be arbitrarily remapped without altering the computational result. Traditional systems hardcode physical memory addresses as the default coordinate system, which is equivalent to fixing a global rotational basis and thereby forfeiting the entire exploratory space of coordinate flexibility.

Static Basis + Dynamic Viewpoint: Rotating Queries, Not Moving Data

By treating the KV cache as a static geometric basis, the position encoding matrix $A(m) = mJ$ can be reinterpreted as a coordinate system shift operator. Attention computation is essentially the projection of a query vector onto this basis, and the group properties of RoPE allow us to simulate different positional relationships through query rotation transformations.

For a cache block $B$ at physical coordinate $p$, if computation is required at a logical coordinate $\ell$, no KV data needs to be moved. Instead, we simply apply a transformation to the query:

This operation achieves computational equivalence: the calculation $\mathcal{R}(-p)K$ in the physical basis, after transformation, produces an identical attention distribution to the calculation $\mathcal{R}(-\ell)K$ in the logical coordinate system. The complexity is reduced from $O(L \cdot d)$ for memory movement to $O(N_{\text{blocks}} \cdot d)$ for rotational computation, where $N_{\text{blocks}}$ is the number of cache blocks, and typically $N_{\text{blocks}} \ll L$. This complexity reduction has been validated in practice by systems like vLLM through its PagedAttention mechanism³.

The zero-curvature property, $\mathcal{F}_{\mu\nu} = 0$, guarantees the path-independence of parallel transport⁴⁵⁶. This ensures that inserting, deleting, or reordering logical views at any point during inference does not disrupt the geometric consistency of historical computations, providing an algebraic foundation for incremental topological editing. Empirical studies have also shown that the manifolds learned by deep generative models often exhibit near-zero curvature⁷, lending practical support to this assumption⁸.

A Unified Framework: Three Classes of Homomorphic Mapping Strategies

All computational modes share the same set of rotational operators, differing only in their coordinate mapping strategy, $\sigma$. This is the core idea of homomorphic transformation:

1. Identity Mapping (Standard Autoregression)

The boundary condition is $\sigma_{\text{causal}}(B_i) = p_i$, where logical coordinates coincide with physical coordinates, and the rotation degenerates to an identity transformation. This is the degenerate case of fixed coordinates, corresponding to the traditional inference model.

2. Permutation Homomorphism (Separating Concurrent Computations)

Let $N$ inference instances share a physical basis $\mathcal{B}$. Each instance $\alpha$ defines a permutation $\pi_\alpha \in \text{Sym}(\mathcal{B})$:

When a query $q_\alpha$ interacts with block $B_i$, a rotation $\mathcal{R}(\sigma_\alpha(B_i) - p_i)$ is applied. The computations for all $N$ instances proceed in parallel, with different rotations applied only on the query side. This strategy of achieving concurrent sharing through block-level address permutation is already employed in modern inference engines³⁹. "Conflicts" between instances are automatically resolved by the orthogonal decomposition of the representation space:

Concurrency control is thus transformed into a problem of group representation theory, eliminating the need for locking mechanisms—an algebraic implementation of the spectral isolation principle.

3. Spectral Shift Mapping (Asynchronous Stream Fusion)

We assign spectral parameters $\Phi: \mathbb{Z}_M \to \mathbb{Z}$ to $M$ asynchronous streams:

The idea of converting temporal disparities in asynchronous streams into controllable phase differences in a spectral domain is analogous to systems like FlowKV, which optimize KV cache transfer¹⁰. The attention operator splits into block-diagonal and off-diagonal components, a concept theoretically grounded in measure-theoretic interpretations of self-attention as an interacting particle system¹¹:

The constant offset difference $\Delta = \Phi_k - \Phi_{k'}$ acts as a learnable parameter encoding inter-stream dependencies, achieving spectral multiplexing: asynchronicity in the time domain is translated into a controllable phase difference in the frequency domain.

An Information Geometry Perspective: Attention as a Similarity Metric

Viewing token representations as points on a statistical manifold $\mathcal{S}$, the attention score can be reinterpreted as a probability decay based on distance⁸:

Here, $d(p_i, p_j)$ is the geodesic distance on the manifold. This perspective, which defines attention scores as an exponential function of geodesic distance, finds direct theoretical support in Transformer frameworks based on Riemannian and non-Euclidean geometries⁴¹²¹³. The invariance of coordinate transformations guarantees that $d(\sigma(p_i), \sigma(p_j)) = d(p_i, p_j)$, meaning that re-encoding does not affect performance because the geometric distance remains unchanged.

Zero sectional curvature implies that $\mathcal{S}$ is locally flat⁷, so all topological restructuring operations preserve information entropy. The optimal mapping $\sigma^*$ should minimize a cost function closely related to the variational principle for geodesics and optimal transport theory¹⁴¹⁵¹⁶:

The training process becomes a gradient flow on the moduli space, allowing the model to adaptively learn the optimal coordinate mapping.

Hardware Implementation: From Rotations to Complex Multiplication

The core operation is the matrix exponential $\exp(\theta J)$, which can be mapped to channel-wise complex multiplication on modern GPUs¹:

By leveraging complex matrix multiplication units, the query vector can be broadcast in parallel to all cache blocks while applying different unitary rotations, thereby localizing computation and delocalizing the perspective.

In a multi-GPU scenario, global coordinate transformations are synchronized via an algebraic protocol:

The central charge $K_c$ encodes the phase difference across devices. AllReduce only needs to synchronize $O(d)$ parameters, reducing communication from $O(N_{\text{tokens}} \cdot d)$ to $O(d)$.

Paradigm Shift: From Movement to Transformation

The essence of this revolution is to elevate inference from data movement to coordinate programming. Positional encodings are no longer static labels but dynamic coordinate transformations; attention propagation is a geometric instantiation of a group action; different computational modes correspond to different configurations of a spectral measure.

When the attention mechanism is promoted from a tensor contraction to a variational problem on a manifold, the model transforms from a passive sequence predictor into an active topological reconstruction engine. The future direction is to enable the model to autonomously learn the optimal mapping¹⁴¹⁵¹⁶:

When the model can evolve stable configurations that yield the smoothest coordinate transformations, inference becomes a constructive process of geometric optimization. This marks a leap from engineering heuristics to mathematical first principles: the core value of a language model is no longer predicting the next token, but acting as a programmable geometric engine that actively shapes the topology of information flow in digital space.

References