The "too many RAG optimization techniques" dilemma
In April 2026, an article titled "Lost in RAG optimizations: mapping the landscape brought everything into focus" trended on Zenn, striking a chord with many developers. While the early days of RAG in 2023 were simply "vector search + LLM," by 2026 the landscape has swelled to over 10 optimization layers, including hybrid search, reranking, query rewriting, HyDE, Corrective RAG, Adaptive RAG, Parametric RAG, and Agentic Retrieval.
This article organizes these competing RAG optimization techniques across four layers — pre-retrieval, retrieval, post-retrieval, and evaluation — serving as a structured catalog to guide selection based on project scale and accuracy requirements.
For foundational background on external data connectivity, reading the Complete MCP guide will also help clarify the broader picture.
Layer 1: Pre-retrieval processing
This layer governs how user queries are processed before retrieval occurs. It is no exaggeration to say that over half of the performance variance in RAG is determined at this stage.
Chunking strategies
Chunk splitting is both the most frequently overlooked aspect of RAG and the one with the greatest impact.
| Strategy | Description | Suitable content |
|---|---|---|
| Fixed-length chunking | Mechanically splits by fixed token counts | Homogeneous FAQs, manuals |
| Semantic chunking | Splits by semantic boundaries | Technical documentation, blog posts |
| Hierarchical chunking | Maintains multi-level hierarchy: chapter → section → paragraph | Long contracts, research papers |
| Sliding window with overlap | Overlaps to prevent information loss at boundaries | Source code |
As a rule of thumb, a chunk size of 200–800 tokens with 10–20% overlap serves as a practical starting point.
Query rewriting
Rather than embedding user questions directly, this method uses an LLM to rewrite queries into formats optimized for retrieval. Context-dependent questions like "What happened with that project last month?" are concretized into "March 2026 progress report and results for Project X."
HyDE（Hypothetical Document Embeddings）
A method where the query is passed to an LLM to generate a "hypothetical ideal answer," which is then embedded to perform retrieval. Because questions and answers frequently use different vocabularies, embedding the answer space increases the likelihood of matching relevant documents.
Query decomposition
Deconstructs complex questions ("Compare the differences between A and B, and recommend which one we should adopt") into multiple sub-queries, executes searches for each, and synthesizes the results. This also forms the foundational technology for Agentic RAG.
Layer 2: Retrieval processing
Hybrid search
The de facto standard in 2026 is parallel execution of keyword search (BM25, etc.) + semantic search (dense vector). Keyword search excels at exact matches for proper nouns and numbers, while semantic search excels at paraphrase matching; combining both typically yields a 10–30% accuracy improvement over either alone.
# pseudo: ハイブリッド検索の骨格
keyword_results = bm25_search(query, top_k=20)
vector_results = vector_search(query_embedding, top_k=20)
merged = rrf_merge(keyword_results, vector_results) # Reciprocal Rank FusionVector filtering
The standard pattern in 2026 applies metadata filtering (e.g., category="技術", date > 2026-01-01) upstream of the vector search. Pruning search targets beforehand reduces both noise and compute costs simultaneously.
Multi-Vector Retrieval
A technique maintaining multiple vectors per chunk — such as summary vectors, raw text vectors, and child chunk vectors — and querying them conditionally. This dramatically improves accuracy in projects handling complex documents.
Layer 3: Post-retrieval processing
Reranking
A technique that takes the top 20–50 retrieval results, rescores them using a dedicated reranker (such as Cohere Rerank or BGE Reranker), and narrows them to the top 5–10 items. Although it incurs compute overhead, it has become virtually essential in enterprise RAG because top-k precision improves dramatically.
Context compression
A method that strips irrelevant sentences from retrieved chunks before passing them to the LLM. This avoids LLM context length constraints and elevates response quality by removing noise.
Corrective RAG（CRAG）
An architecture that scores the quality of retrieved documents with a lightweight classifier and automatically triggers fallbacks, such as web searches, if quality is poor. Proposed in late 2025, production adoptions have grown substantially in 2026.
Adaptive RAG
A technique that dynamically switches the retrieval strategy itself based on query complexity. It operates agentically: "Simple FAQ → basic search," "Multi-part question → query decomposition → parallel search → synthesis."
Layer 4: Evaluation and monitoring
Separating retrieval evaluation from generation evaluation
RAG failures fall broadly into two categories: "The retrieval layer failed to fetch the correct chunk" and "The correct chunk was fetched, but the LLM ignored or misinterpreted it." Improvement cycles stall unless both stages are evaluated independently.
| Evaluation target | Representative metrics | Application |
|---|---|---|
| Retrieval | Recall@k, MRR, nDCG | Retrieval layer tuning |
| Generation | Faithfulness, Answer Relevance | Prompt engineering and LLM selection |
| Task | Business KPIs, user satisfaction | Strategic decision-making |
Continuous error analysis
An operational routine of sampling and reviewing 10–30 failure cases weekly is essential for long-term RAG refinement. Chunk boundary flaws, embedding quality degradation, and retrieval strategy limits become visible during this process. For evaluation philosophy, please also refer to AI benchmarks are broken.
New trends in 2026: Parametric RAG and Agentic Retrieval
Parametric RAG
A radical approach that embeds knowledge directly into model parameters instead of using external retrieval. Domain-specific knowledge is injected via LoRA adapters or fine-tuning, bringing inference-time retrieval overhead close to zero. While well suited for static knowledge bases (such as corporate policies), updating costs are high.
Agentic Retrieval
A retrieval pipeline where the LLM autonomously determines which data sources to query, how many searches to execute, and when to conclude. Moving beyond simple RAG, this embraces the paradigm of planning the retrieval process itself.
Implementation roadmap by project scale
Attempting to adopt all techniques at once ensures failure. Below is a staged implementation roadmap:
| Phase | Scale | Target optimizations |
|---|---|---|
| Phase 1（PoC） | Up to 10k documents | Semantic chunking + basic vector search + Top-5 |
| Phase 2 (Pilot) | Up to 100k documents | Hybrid search + metadata filtering + reranking |
| Phase 3 (Production) | Up to 1M documents | Query rewriting + multi-vector + context compression + evaluation infrastructure |
| Phase 4 (Optimization) | 1M+ documents | Corrective / Adaptive RAG + Agentic Retrieval + continuous monitoring |
Conclusion — 2026 RAG is an era of "composition"
RAG can no longer be defined simply as "vector search + LLM." The winning pattern in 2026 is selecting the appropriate combination across four layers: pre-retrieval, retrieval, post-retrieval, and evaluation. Crucially, the golden rule is that this combination must be dictated by your data volume, accuracy requirements, and update frequency.
Starting with Phase 1 minimal viable components and progressively hardening each layer using collected failure cases results in fewer disruptions and faster delivery of business value.
Related articles: Introduction to harness engineering / Complete MCP guide
References
- RAG systems: Best practices to master evaluation — Google Cloud Blog
- Retrieval Augmented Generation (RAG) for LLMs — Prompt Engineering Guide
- RAG Architecture: Retrieval-Augmented Generation Patterns for Enterprise AI — Calmops
- Retrieval-Augmented Generation: A Practical Guide — Comet
- Best Practices in Retrieval Augmented Generation — Gradient Flow
