Article may be outdated

This article is 5 days old. Some details may have changed since publication.

Hacker News·5 min read·medium

I'm making LLMs work on small and old devices

M
mrashad74
I'm making LLMs work on small and old devices
✦AI Summary

A developer shares a project focused on running large language models (LLMs) on low-power, legacy hardware like old TV boxes and early-generation iPads. The project utilizes custom kernels and weight streaming to maximize the utility of idle computing resources.

Why it matters

This work pushes the boundaries of edge computing and AI accessibility, proving that sophisticated models can function on hardware long considered obsolete.

✦Dive DeeperCreate a free account to unlock

Every device around you is more capable than it looks. TV boxes, old phones, single-board computers and routers all ship with CPUs, RAM and flash that sit idle most of the day. smolmaxxing is about unlocking that capability: finding the biggest language model each device can really run, then pushing it further with custom kernels, smarter memory use and weight streaming. Every claim is measured on the hardware itself.

Device #1 is a Xiaomi TV Box S (3rd Gen) : Amlogic S7D, 4× Cortex-A55 @ 2.5 GHz, 1.9 GB of RAM (about 1 GB usable), eMMC flash, Android 14, no root. On it, this repo runs models from 260 K to 16 B parameters:

Continue reading on Headlinne

Create a free account to read the full article.

Read full article →
technologyai
✦

Get smarter about the news

Sign up free for a feed built around what you actually care about, Dive Deeper research on any story, and the full text of every article.

Create free account

Already have an account? Sign in