AirLLM 70B inference with single 4GB GPU

AirLLM is a software tool that enables the execution of massive large language models on consumer-grade hardware with limited VRAM. By utilizing per-expert streaming for sparse models, it allows users to run models as large as 2.8 trillion parameters on a single GPU.