Category: Pretraining

Dust: Pretraining Transformers Without Backpropagation

qlabs.sh

2026-10-05 by librarian

dust:-pretraining-transformers-without-backpropagation

Samip Dahal, Bishwas Mandal, Serdar Gülbahar, Akshay Vegesna October 2026Correspondence to [email protected]·Code·Cite Copy @misc{dahal2026backprop, title = {Dust: Pretraining Transformers Without Backpropagation}, author = {Dahal, Samip and Mandal, Bishwas and G{“u}lbahar, Serdar and Vegesna, Akshay}, year = {2026}, url = {https://qlabs.sh/research/dust} … Read more

Pretraining Language Models via Neural Cellular Automata

hanseungwook.github.io

2026-03-15 by librarian

Research Paper Blog What if the path to smarter language models doesn’t require more text — but synthetic data from abstract dynamical systems? 6% Perplexity gain 1.6× Faster convergence 164M NCA tokens used 01 — The Problem We’re running out … Read more