The Perils of Greedy Search in Semantic Space

When we first start building systems that attempt to answer complex questions by retrieving information from a corpus, we often fall into a trap of simplicity. The standard retrieval pipeline—take a user question, embed it, find the nearest neighbor in the vector database, and stuff that context into a Large Language Model (LLM) prompt—works surprisingly well for factual lookups. It is the semantic equivalent of a breadth-first search that stops at the first solution. However, this approach crumbles spectacularly when faced with multi-hop reasoning. The “First Chunk Wins” heuristic, where the system grabs the most semantically similar document and hopes it contains the entire answer, is not just suboptimal; it is fundamentally incapable of handling questions that require traversing a knowledge graph rather than locating a single fact.

Consider a question like: “What is the GDP of the country where the inventor of the phonograph was born?” A naive vector search embedding this entire query will likely surface documents about Thomas Edison, the phonograph, or perhaps economic statistics of the United States. While the United States is the correct answer, the retrieval system hasn’t reasoned through the steps; it has merely matched high-level concepts. If the question were slightly more obscure—say, involving a less famous inventor or a geopolitical shift—the likelihood of the initial retrieval hitting the exact statistical document containing the GDP figure drops to near zero. The semantic gap between the complex question and the specific answer chunk is too wide to bridge in a single leap.

This limitation stems from the geometry of high-dimensional vector spaces. In a well-structured embedding space, concepts cluster together, but multi-hop questions require a path through this space. We need to move from “phonograph” to “Edison” to “United States” to “GDP.” A single vector query attempts to jump directly to the destination without traversing the intermediate nodes. To solve this, we must move from passive retrieval to active, guided retrieval architectures. This shift transforms the retrieval process from a simple lookup into a reasoning engine that constructs answers step-by-step.

Decomposing the Query: Path Building and Logical Dependency

The first step in guided retrieval is acknowledging that a user’s question is rarely a single semantic entity. It is usually a composite of several sub-questions linked by logical dependencies. Path-building strategies attempt to algorithmically decompose the input into a sequence of retrieval steps. This is essentially applying the principle of reflexion or tree-of-thought specifically to the retrieval layer.

Instead of embedding the full query, we employ a lightweight language model to break the question down. For the question regarding Edison’s birthplace and GDP, the system might generate two distinct sub-queries:

  1. Sub-query A: “Where was Thomas Edison born?”
  2. Sub-query B: “What is the GDP of [Result from A]?”

The system then executes these queries sequentially. The critical difference here is the dependency: Sub-query B is not static; it is dynamically generated based on the output of Sub-query A. This requires a retrieval architecture that supports state management. The “state” in this context is the accumulated context window of the reasoning trace.

Implementing this requires a shift in how we view the context window of an LLM. In a naive RAG (Retrieval-Augmented Generation) setup, the context window is a bucket for relevant documents. In a guided retrieval setup, the context window is a scratchpad for intermediate reasoning steps. The system prompts the LLM to act as a query decomposer, outputting a structured plan. This plan is then executed by the retrieval engine.

However, path building has its own complexities. Decomposition is not always straightforward. Some questions involve implicit hops where the connecting entity is not explicitly named but inferred. For example, “Who succeeded the CEO who founded the company that built the Apollo guidance computer?” requires identifying (1) the computer, (2) the company (MIT Instrumentation Lab), (3) the founder (Charles Stark Draper), and (4) the successor. The vector distance between “Apollo guidance computer” and “successor of Charles Stark Draper” is massive. A path-building approach bridges this gap by anchoring the search on the explicit entities first, then traversing the corporate hierarchy.

Leveraging Structure: Rules, Ontologies, and Knowledge Graphs

Vector embeddings are excellent at capturing semantic similarity, but they are notoriously bad at capturing structural constraints and logical rules. They operate on a continuum of similarity, whereas knowledge often exists in discrete, rigid relationships. This is where integrating ontologies and rule-based constraints becomes essential.

When we talk about “guided retrieval,” we often mean guiding the search with metadata or structural boundaries. Imagine a corpus of legal documents. A naive vector search for “contract termination clauses” might return clauses from employment contracts, rental agreements, and software licenses indiscriminately. A guided approach uses the document’s metadata—its ontology—to constrain the search space before the vector similarity calculation even occurs.

Technically, this is implemented through Metadata Filtering combined with vector search. In databases like Pinecone, Weaviate, or Milvus, you can perform a hybrid query: “Find vectors similar to X, but only where document_type = ‘statute’ AND jurisdiction = ‘EU’.” This reduces the search space from millions of documents to thousands, increasing the precision of the initial retrieval.

For more complex multi-hop reasoning, we can map our vector store to a Knowledge Graph (KG). A Knowledge Graph represents data as entities (nodes) and relationships (edges). While vector search finds “what is similar,” graph traversal finds “what is connected.”

Consider a scenario where we need to answer a question about drug interactions. A vector search might retrieve general articles about two specific drugs. A graph-based guided retrieval, however, traverses the path:

Drug A -> inhibits -> Enzyme CYP3A4 -> metabolizes -> Drug B

This allows the system to infer that Drug A increases the concentration of Drug B, even if no document explicitly states “Drug A interacts with Drug B.”

Implementing this hybrid approach requires a dual-index architecture. We maintain a vector index for semantic search and a graph database (like Neo4j) for relational queries. The retrieval logic becomes a router: if the query involves explicit relationships (“brother of,” “subsidiary of,” “metabolized by”), we prioritize the graph. If the query is conceptual (“impact of inflation on renewable energy”), we prioritize the vector store. The magic happens when we use the graph to disambiguate the vector search. If the vector search returns multiple candidates with similar embeddings, the one that has the strongest connectivity to previously retrieved entities in the graph is selected.

Iterative Query Refinement: The Recursive RAG Pattern

One of the most effective guided retrieval strategies is iterative refinement, often seen in architectures like Self-RAG or Iterative RAG. This approach treats retrieval not as a one-off event but as a loop. The core idea is that the initial query is almost always too broad or too narrow. We need to refine it based on what we find (or fail to find).

The process looks like this:

  1. Initial Retrieval: Execute a broad search based on the user query.
  2. Critique & Gap Analysis: The LLM analyzes the retrieved chunks. It asks itself: “Do these chunks answer the question? If not, what information is missing?”
  3. Query Reformulation: Based on the identified gap, the LLM generates a new, more specific query.
  4. Secondary Retrieval: Execute the new query to fill the gap.
  5. Synthesis: Combine the original context with the new context to generate the final answer.

This mimics the human research process. When writing a technical paper, we rarely find the perfect source immediately. We find a source, realize it lacks specific data, and then search for that specific data.

In code, this often looks like a state machine. The LLM outputs a “tool use” instruction. For example:

{
  "reasoning": "The retrieved chunks discuss Edison's inventions but do not mention his birthplace.",
  "next_action": "refine_search",
  "new_query": "Thomas Edison birthplace"
}

The system then executes the new query and injects the results back into the context. This prevents the “lost in the middle” phenomenon where relevant information is buried in the middle of a long context window. By iteratively focusing the retrieval, we ensure that each retrieved chunk is high-value.

Iterative refinement is particularly powerful for handling ambiguity. If a user asks “Who is the best programmer?”, the system can recognize the ambiguity (best by what metric? In what language? In what era?) and issue clarifying retrieval queries or sub-queries to narrow the scope before attempting a synthesis.

Graph Neighborhood Exploration: Beyond Immediate Neighbors

Vector similarity is local. It measures the cosine similarity between two points. However, knowledge is often global; the answer might be several “hops” away from the query vector. Graph neighborhood exploration addresses this by expanding the search radius dynamically.

Imagine a graph where every document is a node, and edges represent citations or shared entities. A naive vector search finds the node closest to the query. A graph exploration strategy finds the subgraph surrounding that node.

There are two primary ways to explore the neighborhood:

  1. Expansion (Breadth-First): Retrieve the top-K vector matches, then retrieve all documents linked to those matches (e.g., via citations or shared metadata). This creates a “cloud” of context.
  2. Traversal (Depth-First): Follow specific relationship types. For example, in a corporate database, starting at “Company A,” we traverse edges labeled “invested_in” to find “Startup B,” then traverse “acquired_by” to find “Company C.”

In practice, we often combine vector search with graph traversal using a technique called GraphRAG. This involves generating a summary of the local graph neighborhood for each retrieved entity and using that summary to enrich the context.

Let’s look at a practical implementation pattern. Suppose we are building a system to analyze software vulnerabilities. We have a vector store of CVE descriptions. A user asks, “What is the risk of the Log4j vulnerability spreading to our microservices?”

Naive retrieval finds the Log4j CVE description. Guided retrieval explores the neighborhood:

  1. Retrieve Log4j CVE (Vector search).
  2. Traverse graph for “dependencies” to find which libraries import Log4j.
  3. Traverse graph for “affected_versions” to see if the user’s microservices match.

This requires the retrieval system to understand the topology of the data. In a vector database, we can approximate this by using Hierarchical Navigable Small World (HNSW) graphs, which are the underlying data structure of many vector indexes. However, for true guided retrieval, we need to explicitly model the relationships beyond the vector embedding.

When implementing graph exploration, we must be mindful of the fan-out problem. Expanding too many nodes leads to exponential growth in context length. A common heuristic is to limit the expansion depth (usually 2 hops) and use a relevance score to prune low-value branches. For instance, if we are traversing from “Edison” to “Companies,” we might weight the edge to “General Electric” higher than the edge to “Edison Portland Cement Company,” effectively guiding the retrieval toward the most relevant neighborhood.

Technical Implementation: The Router and The Synthesizer

Building a guided retrieval system requires a robust architecture that separates the retrieval logic from the generation logic. A monolithic LLM call cannot handle the complexity of path planning, retrieval, and synthesis simultaneously without losing focus. We need a modular pipeline.

The core component is the Router. The Router is a classifier (often a smaller, faster model) that inspects the incoming query and decides the retrieval strategy.

Here is a simplified logic flow for the Router:

  1. Input: User Query.
  2. Analysis: Does the query contain explicit entities? Does it require numerical calculation? Is it a comparison?
  3. Decision:
    • If simple fact lookup -> Direct Vector Search.
      • Path Building (Decomposition).
    • If complex relationship traversal -> Graph Exploration.
    • If vague or broad -> Iterative Refinement.

Once the Router selects a strategy, the Retriever executes it. The Retriever is the interface to your data stores (Vector DB, SQL, Graph DB). It returns a set of documents or nodes.

The final component is the Synthesizer. This is where the “First Chunk Wins” fallacy is fully corrected. The Synthesizer receives the retrieved chunks—which are now a collection of facts, relationships, and intermediate steps—and constructs the final answer. Crucially, the Synthesizer must be prompted to cite its sources and acknowledge the chain of reasoning.

For example, a prompt for the Synthesizer might look like this:

“You are an expert researcher. You have been given a set of retrieved documents and intermediate reasoning steps. Your task is to answer the user’s question using only the provided information. If the information is insufficient, state that. Trace your answer back to the specific source chunks.”

When coding this, we use structured output formats (like JSON) to pass data between the Router, Retriever, and Synthesizer. This ensures that the system remains deterministic and debuggable. We avoid free-form text parsing between steps, which introduces fragility.

Consider the data structure for a guided retrieval step:

{
  "query": "GDP of Edison's birthplace",
  "strategy": "path_build",
  "steps": [
    {"sub_query": "Edison birthplace", "status": "completed", "result": "Milan, Ohio"},
    {"sub_query": "GDP of Milan, Ohio", "status": "pending"}
  ],
  "context_window": ["Chunk 1: Edison born in Milan...", "Chunk 2: Milan is a city in Ohio..."]
}

This stateful approach allows the system to handle failures gracefully. If the first step returns “Milan, Italy” (a common ambiguity), the subsequent GDP search will fail or return incorrect data, triggering a verification step.

Handling Ambiguity and Entity Resolution

One of the hardest problems in guided retrieval is entity resolution. The vector space is dense with homonyms. “Apple” could refer to the fruit or the technology company. “Washington” could be a state, a person, or a city.

In a naive retrieval system, the context window often fails to disambiguate these terms. The LLM might receive a chunk about the fruit when the question is about the company, leading to hallucination.

Guided retrieval solves this through Disambiguation Queries. Before committing to a retrieval path, the system can perform a lightweight check.

For example, if the query is “Apple revenue,” the system recognizes “Apple” is ambiguous. It can issue a disambiguation query: “What is the most likely entity type for ‘Apple’ in the context of ‘revenue’?” The retrieval system might return two candidates: “Apple Inc.” and “Malus domestica (Apple fruit).” The system can then check the vector similarity of “revenue” to both clusters. Since “revenue” is highly similar to corporate entities and dissimilar to fruit entities, the system locks onto “Apple Inc.”

Another technique is Entity Linking prior to retrieval. We can use a Named Entity Recognition (NER) model to extract entities from the query, then query a Knowledge Base (like Wikidata) to get a unique ID for each entity. Once we have the ID (e.g., Q312 for Apple Inc.), we use that ID to filter our vector search. This ensures we are searching within the semantic space of the specific entity, not the general term.

This adds a layer of precision. Instead of searching for “Apple revenue” in a space where “apple pie” and “Apple iPhone” are neighbors, we search for “revenue” in the subspace defined by the entity ID Q312. This drastically reduces noise and improves the quality of the first retrieval step.

Evaluating Guided Retrieval Systems

Measuring the performance of these systems requires more than just retrieval accuracy. Traditional metrics like Mean Reciprocal Rank (MRR) or Recall@K measure whether the relevant document is in the top K results. However, in a guided system, the “document” is often a composite of multiple sources.

We need to evaluate the reasoning trace. Did the system choose the correct decomposition path? Did the iterative refinement stop at the right time? Did the graph traversal avoid dead ends?

Common evaluation metrics for guided retrieval include:

  • Path Accuracy: Did the intermediate steps (e.g., identifying the birthplace) yield the correct entity?
  • Answer Completeness: Does the final synthesis contain all necessary facts?
  • Hallucination Rate: Did the system invent facts because the retrieval failed?

Benchmarking datasets like HotpotQA are specifically designed for multi-hop reasoning. They require systems to read multiple documents and reason over them. Naive RAG systems struggle on these benchmarks, often scoring below 40% accuracy. Guided retrieval systems, utilizing decomposition and graph traversal, can push these scores significantly higher, often exceeding 70-80% depending on the complexity of the implementation.

When building your own evaluation suite, it is vital to include “adversarial” examples—questions that look simple but require deep traversal or contain ambiguous entities. This stress-tests the router and the disambiguation logic.

Performance Considerations and Latency

It is important to acknowledge that guided retrieval introduces latency. A single vector search might take 50ms. A path-building strategy that requires three sequential LLM calls and three retrieval steps might take 3-5 seconds. This trade-off is acceptable for complex queries but wasteful for simple ones.

Optimization strategies include:

  1. Parallel Execution: If the decomposition yields independent sub-queries (e.g., “Compare the GDP of Country A and Country B”), execute the retrievals in parallel, not sequentially.
  2. Speculative Retrieval: While the LLM is generating the decomposition, start pre-fetching the likely documents based on the initial query embedding.
  3. Small-to-Large: Use smaller, faster models for the decomposition and routing steps, reserving the heavy LLM only for the final synthesis.

Memory management is also critical. In iterative refinement, the context window can bloat quickly. We must employ techniques like summarization of previous steps or selective pruning of irrelevant chunks to keep the context focused. We cannot simply append every retrieved document to the prompt; we need a dynamic context management strategy that prioritizes the most recent and most relevant information.

Conclusion (Implicit)

The shift from “First Chunk Wins” to guided retrieval represents a maturation of RAG architectures. It acknowledges that knowledge is not just a bag of words to be searched, but a structured, interconnected web to be navigated. By employing path building, leveraging ontologies, iterating on queries, and exploring graph neighborhoods, we transform the retrieval system from a passive database query into an active reasoning partner. This approach requires more engineering effort and careful orchestration of models and data stores, but it is the only way to reliably answer the complex, multi-hop questions that users actually care about.

Share This Story, Choose Your Platform!