Hacker News·3 min read·hard

Show HN: TrnRT – An open-source LLM inference engine for AWS Trainium

S
sanchitmonga22
Show HN: TrnRT – An open-source LLM inference engine for AWS Trainium
✦AI Summary

TrnRT is a new open-source inference engine designed to optimize large language model performance on AWS Trainium hardware. It features NKI megakernels and a prefix cache to improve efficiency for OpenAI-compatible API deployments.

Why it matters

This tool lowers the barrier for developers to utilize specialized cloud hardware for high-performance AI inference, potentially reducing costs and complexity.

✦Dive DeeperCreate a free account to unlock

Quickstart · Benchmarks · Docs · Kernels · Blog post · runanywhere.ai

TrnRT is an open inference engine for large language models on AWS Trainium. It serves an OpenAI-compatible API from one trn2.48xlarge , runs decode as NKI megakernels (about one kernel per layer half), and keeps a prefix cache across device and host memory. GLM-5.3-Flash is the first supported model; more open models are on the way.

How we built it, with the AgentX results: Serving GLM-5.3-Flash on one Trainium2 box with NKI layer megakernels .

Continue reading on Headlinne

Create a free account to read the full article.

Read full article →
technologybusiness
✦

Get smarter about the news

Sign up free for a feed built around what you actually care about, Dive Deeper research on any story, and the full text of every article.

Create free account

Already have an account? Sign in