Show HN: TrnRT – An open-source LLM inference engine for AWS Trainium
TrnRT is a new open-source inference engine designed to optimize large language model performance on AWS Trainium hardware. It features NKI megakernels and a prefix cache to improve efficiency for OpenAI-compatible API deployments.
Why it matters
This tool lowers the barrier for developers to utilize specialized cloud hardware for high-performance AI inference, potentially reducing costs and complexity.
Quickstart · Benchmarks · Docs · Kernels · Blog post · runanywhere.ai
TrnRT is an open inference engine for large language models on AWS Trainium. It serves an OpenAI-compatible API from one trn2.48xlarge , runs decode as NKI megakernels (about one kernel per layer half), and keeps a prefix cache across device and host memory. GLM-5.3-Flash is the first supported model; more open models are on the way.
How we built it, with the AgentX results: Serving GLM-5.3-Flash on one Trainium2 box with NKI layer megakernels .
Get smarter about the news
Sign up free for a feed built around what you actually care about, Dive Deeper research on any story, and the full text of every article.
Create free accountAlready have an account? Sign in