• Sparse attention 6 -- In-context Associative recall

  5 min read   ·   January 30, 2026

  2026   ·   Physics-of-AI   Sparse-attention     ·   AI  

• MLP 2 -- Effective linearity, Generalized SiLU

  4 min read   ·   January 29, 2026

  2026   ·   Physics-of-AI   MLP   Non-linearity     ·   AI  

• MLP 1 -- Gating is good for polynomials

  3 min read   ·   January 28, 2026

  2026   ·   Physics-of-AI   MLP   Non-linearity     ·   AI  

• Optimization 4 -- Loss Spikes

  6 min read   ·   January 27, 2026

  2026   ·   Physics-of-AI   Optimization     ·   AI  

• Optimization 3 / Depth 2 -- Adding Bias After ReLU

  4 min read   ·   January 25, 2026

  2026   ·   Physics-of-AI   Optimization   Depth     ·   AI