Better Vector Search for Long Documents: Chunking Inside Manticore Search
Manticore Search has introduced a native chunking feature to improve vector search performance for long documents. This update allows users to automatically split documents into smaller segments during ingestion, bypassing token limit issues.
Why it matters
This technical improvement simplifies the development of RAG (Retrieval-Augmented Generation) pipelines by removing the need for external splitting libraries.
Say you are building search over your team's internal documentation — guides, runbooks, postmortems. You have a table with auto embeddings : you insert text, Manticore runs the model and fills the vector column for you. (If that is new to you, start with vector search in Manticore .) You load a 4,000-word document. The insert succeeds. The search works. Everything looks fine.
Except the model you picked has a 512-token input window, and that document is about 5,000 tokens long. The model read the first 380 words and threw away the other 3,600. Nothing in the document past that point can ever be retrieved, and nothing anywhere told you. The embedding may not represent the document as a whole either.
Get smarter about the news
Sign up free for a feed built around what you actually care about, Dive Deeper research on any story, and the full text of every article.
Create free accountAlready have an account? Sign in