⚙️ Recommendation Engine
Candidate Generation Explained
Before ranking anything, a recommender must retrieve a pool of candidates. Learn how Headlinne combines recency-based and semantic retrieval to build its candidate set.
By Headlinne Editorial Team · Updated on
The retrieval problem
Candidate generation is the first stage of a recommendation funnel. Its goal is recall: pull in every article that could plausibly be relevant, cheaply, so the ranking stage has good material to work with. If a great article never enters the candidate pool, no amount of clever ranking can surface it.
The challenge is doing this fast. Retrieval must run over the whole eligible corpus in milliseconds, which rules out expensive per-article computation at this stage.
Two retrieval strategies, combined
Headlinne uses two complementary retrieval paths. The first fetches recent, active, unexpired articles ordered by publish time—guaranteeing the feed reflects the current news cycle. The second is semantic: it queries an approximate-nearest-neighbor index with the user's taste vector to pull in articles that match their interests even if the recency pass missed them.
Merging the two gives both freshness and relevance. The recency path ensures timeliness; the semantic path ensures the pool includes articles aligned to what the user actually cares about.
Filtering at the source
Retrieval also enforces hard eligibility rules up front:
- Only active, non-flagged, non-sponsored articles
- Only articles within the freshness window (not expired)
- Excludes already-swiped articles
- Excludes sources the user has blocked
Sizing the pool
The candidate pool is deliberately larger than the number of slots in the feed—several times larger—so that ranking and diversity constraints have room to make choices. Fetching too few candidates starves the ranker; fetching too many wastes computation.
Getting this balance right is a practical tuning decision every recommender faces: enough candidates for quality, few enough for speed.
Key takeaways
- ✓Candidate generation optimizes for recall: retrieve everything plausibly relevant, cheaply.
- ✓Headlinne combines recency-based retrieval with semantic nearest-neighbor retrieval.
- ✓Hard filters (freshness, blocked sources, already-seen) apply during retrieval.
Frequently asked questions
Why retrieve more candidates than the feed needs?
A larger pool gives the ranking and diversity stages room to make good choices. Retrieving only as many candidates as slots would leave no flexibility for quality or variety.
What happens if semantic retrieval is unavailable?
The engine falls back to the recency-based candidate set. Semantic augmentation improves relevance but is not required for a working feed.
Related Headlinne features
Related reading
Continue learning
Embeddings and Vector Search
Embeddings turn articles and interests into vectors, and vector search finds the closest matches. Learn how Headlinne uses them to power semantic recommendations and search.
The Cold Start Problem
How does a recommender help a brand-new user with no history? Learn what the cold start problem is and how Headlinne handles new users with priors and more exploration.
Start reading personalized news with Headlinne
Create your free account and build a feed that learns what you care about.