Dust: Pretraining Transformers Without Backpropagation

Researchers have introduced 'Dust,' a new method for pretraining transformer language models that avoids the traditional backpropagation algorithm. The study suggests that this zeroth-order method is computationally efficient and could potentially outperform backpropagation in compute-rich environments.