• MLP 2 -- Effective linearity, Generalized SiLU

  4 min read   ·   January 29, 2026

  2026   ·   Physics-of-AI   MLP   Non-linearity     ·   AI  

• MLP 1 -- Gating is good for polynomials

  3 min read   ·   January 28, 2026

  2026   ·   Physics-of-AI   MLP   Non-linearity     ·   AI  

• Optimization 4 -- Loss Spikes

  6 min read   ·   January 27, 2026

  2026   ·   Physics-of-AI   Optimization     ·   AI  

• Optimization 3 / Depth 2 -- Adding Bias After ReLU

  4 min read   ·   January 25, 2026

  2026   ·   Physics-of-AI   Optimization   Depth     ·   AI  

• Optimization 2 -- Elementwise Scale Reparametrization

  4 min read   ·   January 24, 2026

  2026   ·   Physics-of-AI   Optimization     ·   AI