14× faster embeddings: how we rebuilt the ONNX path in Manticore
Manticore Search has updated its ONNX runtime backend, resulting in a 14x performance increase for text embedding tasks. By optimizing how the database handles model inference during data ingestion, the update significantly improves throughput for high-concurrency workloads.
Why it matters
Performance improvements in vector search and embedding generation are critical for the scalability of AI-driven database applications.
When we shipped Auto Embeddings — the feature that turns any text column into a vector automatically, with no separate model service to run — the most common piece of feedback was about speed. The previous path went through SentenceTransformers on top of Candle , Hugging Face's pure-Rust ML inference runtime, and it left a lot of CPU on the floor: most workloads sat in the low-double-digits of docs/sec no matter how we fed them, and concurrent calls serialised on a single model session.
The article is a technical engineering log focused on performance benchmarks and software architecture.
Get smarter about the news
Sign up free for a feed built around what you actually care about, Dive Deeper research on any story, and the full text of every article.
Create free accountAlready have an account? Sign in