Training Text-to-Image Models 3.6× Faster

Beating latent diffusion models with pixel-space encoder-decoders

Road to Linum v3 · Issue 02 previously: data filtering TL;DR Linum v2 was bottlenecked by the enormous size of its attention context window. A 720p, 5 second clip cost a whopping 110K tokens. To put that in perspective, LLMs see samples with fewer than 8K tokens for 97% of their pretraining . Attention is quadratic in cost, so the biggest lever we have to accelerate model training is pruning the context window down.

Most generative image and video systems are Latent Diffusion Models (LDMs). They split compression and generation into independently trained modules: the Variational Autoencoder (VAE) and the DiT (Diffusion Transformer). Recently, pixel-space models like the JiT have shown to be a promising alternative. It reduces two models into one and allows the diffusion model to construct a latent space specifically for generation, rather than rely on one built for reconstruction.