Research shelf / AI & machine learning / SGF · Algebraic Autopsy
Written Updated
Sixteen acceleration tricks that turn out to be one trick
Gradient checkpointing, online softmax, FlashAttention, Muon, K-FAC, WSD scheduling, speculative decoding, mixture-of-experts routing and compute-aware data filtering were each invented to solve a different problem. Surveyed together, they are the same algorithm sixteen times: replace a two-pass batch computation with a one-pass stream that carries a compact sufficient statistic, on a manifold whose metric is the Fisher information.
Experiments were run and the numbers are reported here.
Every efficient deep-learning technique reduced to one primitive — an online sufficient statistic on a curved manifold — plus a post-hoc diagnostic that reads a trained network’s implicit algebra off its weights.
The Streaming Geometry Framework paper surveys sixteen canonical techniques across memory, numerical stability, kernel fusion, optimisation geometry, loss-landscape dynamics, scheduling, inference and data curation, and extracts three meta-patterns: Streaming Sufficiency (any algorithm accumulating global batch state can be rewritten as a stream over a compact sufficient statistic), Spectral Geometry (the natural language for updates, normalisation, routing and attention is the singular-value spectrum), and Data–Compute Duality (data quality, compute and capacity sit on one joint Pareto frontier).
The three are shown to be projections of a single principle the paper calls Incremental Riemannian Estimation. From IRE it derives SGF, an architecture in which optimiser, normaliser, router, scheduler, memory manager and inference engine are all instances of the same primitive, with a C++17/CUDA reference library implementing five novel predictions — among them ZClip-triggered WSD cooldown and edge-of-stability-adaptive checkpointing.
The companion Algebraic Autopsy paper asks the inverse question: what algebra did a trained network actually learn to compute? Four instruments — singular-value power-law exponent, Marchenko–Pastur bulk deviation, ReLU dead-unit sparsity, and effective rank — are applied to an MLP trained on prime classification. The answer is that the network is mostly tropical routing and low-rank Grassmannian geometry, and that genuinely dense arithmetic accounts for only 11% of the computation it performs.
Every number, and what stands behind it
A claim is only worth the evidence attached to it. Each row below carries its basis: measured on the author’s own hardware, derived from the construction, measured on synthetic data, projected from literature, or simply cited.
| Claim | Figure | Basis | Context |
|---|---|---|---|
| Dense (ℝ, +, ×) content of the trained network | 11% | Measured | Prime-classification MLP, four-instrument autopsy |
| Dominant algebra identified | low-rank Grassmannian, score 0.758 | Measured | Instrument scoring across four candidate algebras |
| Spectral exponent of the trained network | α = 0.427 | Measured | Between the data prior (0.37) and a reference model (0.85) |
| Low-rank variant cost | 0.76× parameters, 0.75× FLOPs | Measured | Identical accuracy; rank chosen by the autopsy, not tuned |
| Composite cost of the native variant | 0.57× | Measured | Parameters × FLOPs against the dense baseline |
| Post-training exponent under rank constraint | α = 1.22 | Measured | Geometric constraint acting as a spectral regulariser |
| ReLU dead-unit fraction | 54% | Measured | The tropical-content signal in instrument 3 |
| Newton–Schulz spectral whitening overhead | <1% of training compute | Cited | 5 iterations in bfloat16, LLaMA-scale, from the Muon literature |
Measured — author-run experiment on the stated setup. Synthetic — measured, but on synthetic rather than real data. Derived — follows from the stated construction or proof. Projected — paper-stated projection, not an author-run benchmark. Cited — taken from external literature.
How it works
- Sufficient-statistic reduction. Welford’s (n, mean, M2) and the online-softmax (max, sum) pair are the canonical instances; the paper generalises the pattern to all sixteen techniques.
- Fisher metric as the parameter-space geometry. K-FAC’s Kronecker factorisation and Adam’s diagonal second moment are both read as approximations to the same natural gradient.
- Spectral transform family. Optimisers indexed by the exponent p in G → UΣᵖVᵀ — Shampoo at p = ½, Muon at p = 0.
- Four autopsy instruments. Power-law exponent, Marchenko–Pastur bulk deviation, dead-unit sparsity, effective rank — all computable post hoc from weights alone.
- Five algebraic moves. Semiring substitution, symmetry exploitation, basis factorisation, idempotent sparsification, sufficient-statistic compression — argued exhaustive over known efficiency gains.
What it does not do
Taken from the folder’s own README. Nothing here has been softened.
- The unification is a survey argument, not a theorem: no proof that the sixteen techniques are formally equivalent, only that they share a template.
- The SGF predictions about which technique combinations interact synergistically are stated but not experimentally validated.
- The autopsy substrate is a small MLP on prime classification. Whether the same instruments read the same way on a transformer at scale is untested.
- Five open problems are listed as unresolved; the framework is explicitly incomplete without them.
- The library is a reference implementation. No third-party benchmark against production training stacks.
Free under AGPL-3.0+ for almost everyone
Personal use, charities, education and organisations under AUD 50,000 a year pay nothing. A tiered commercial licence covers everyone else.