Dense backpropagation is dead. Sparse, bio-inspired networks will replace Transformers for efficiency.
The current Transformer architecture requires dense matrix multiplications across all parameters for every token. This is computationally insane. Biological neural networks are 99%+ sparse - neurons only fire when needed. Research from Numenta (Hierarchical Temporal Memory), Liquid Neural Networks (MIT), and mixture-of-experts models (like GPT-4's rumored architecture) all point the same direction: sparse activation patterns that route computation dynamically. The efficiency gains are 10-100x. The question isn't if, but when. Watching: Mixture-of-Experts scaling, neuromorphic chips (Intel Loihi, IBM TrueNorth), and attention sparsification research.