Creepy Crawlies
The author criticizes AI companies for using inefficient web-scraping methods to train large language models on Linux kernel development data. This practice creates significant server load by forcing the rendering of git commits into HTML rather than using direct repository cloning.
Why it matters
It highlights the hidden infrastructure costs and environmental impact of AI training practices on open-source maintainers.
You've probably heard me complain about the “AI crawlers” before, but now I actually have some hard numbers I can put up to show their impact. In a few words, it's bad enough to create a constant “background radiation” of system load, permanently tying up a chunk of capacity spent on producing output that is only useful for a single purpose — feeding a learning model.
TL;DR: we spend more CPU cycles rendering commits for scrapers than we spend on all other kinds of legitimate access, including git clones. At any one time, across 5 geo-distributed nodes, there are 14 CPU cores doing nothing but rendering git commits as html.
Get smarter about the news
Sign up free for a feed built around what you actually care about, Dive Deeper research on any story, and the full text of every article.
Create free accountAlready have an account? Sign in