Production RAG: retrieval, eval gates, cost per query

[!NOTE] TL;DR: A production RAG stays reliable only when retrieval and generation carry separate gates in CI, output passes a grounding check, and every query has a cost ceiling. Hybrid search plus a reranker buys recall, the eval gate catches silent drift, and all prices here were verified on 2026-11-03 in USD.

A 0.97 that lied

My production RAG scored 0.97 faithfulness in the demo. A demo is a postcard; production is the weather. In week three of live traffic the same metric read 0.91 on my 240-case golden set. Nobody had touched the prompt.

The cause was a silent embedding-model version bump on the provider side. It was invisible in my dashboards because I tracked only end-to-end accuracy and latency. That mistake cost me two days. I blamed the reranker first. Classic.

One question, four numbers

My main question was how to keep a RAG system reliable after deployment, not just at launch. Concretely: which retrieval settings hold under real traffic, which metrics catch drift, where guardrails pay, and what one query costs in USD.

The demo score's blind spot

One spontaneously imagines that one end-to-end accuracy score is enough to trust a RAG system. The numbers say the opposite. Retrieval and generation fail differently and are fixed differently, so they must be scored separately.

The RAGAS paper on arXiv formalized this with four separate dials: faithfulness, answer relevance, context precision, and context recall. A hallucination with high context recall is a generator problem. A wrong answer with low context recall is a retriever problem, and no prompt fixes it.

Think of the end-to-end score as a smoke detector that beeps once the house is already burning. What you need are per-stage gauges: recall on the retriever, faithfulness on the generator, refusal accuracy on unanswerable questions, and cost on the invoice. I installed those gauges, in that order, and the next provider bump got caught in eleven minutes instead of two days.

Published vector-DB leaderboards also do not predict your p95. A September 2026 benchmark at 100M vectors (1536 dims, HNSW) put Qdrant at 0.991 recall@10 with p99 of 12 ms and 0.18 USD per 1M queries. pgvector scored 0.987 recall with p99 of 28 ms and 0.08 USD per 1M. Real numbers, but measured without tenant filters or concurrent writes. Replay your own queries against brute-force k-NN before trusting any leaderboard.

The benchmarks indicate that chunking moves recall more than the embedding model does. The February 2026 Vecta/FloTorch study scored recursive 512-token chunking at 69% answer accuracy, fixed-size at 67%, and semantic chunking at 54% because its fragments averaged 43 tokens: high recall, wrong answer. Chroma's 2024 study measured 91.9% retrieval recall for an LLM-based chunker, the same trap. My tests on technical documents agree. I run recursive 512 tokens with 64 overlap and no semantic chunker, with a context budget under 2,500 tokens, because a January 2026 systematic analysis found a "context cliff" beyond roughly 2.5k tokens. That same analysis showed 20% overlap inflates the chunk count by 1.25x with no measurable gain.

Hybrid search is the one retrieval upgrade that consistently pays. On enterprise corpora, BM25 plus dense fused with reciprocal rank fusion lifted recall@10 by 6 to 11 points over dense-only. In the August 2026 SciRet study on CORD-19, hybrid RRF hit Recall@10 of 1.000 at 15K chunks; dense alone sat at 0.787. Its labels were pseudo-relevance, but the direction matches every other study I checked. So: hybrid top-30, rerank to top-5, and the generator sees signal instead of noise.

Target architecture

Ingest is deliberately boring. Recursive chunking at 512 tokens with 64 overlap, embedded with Amazon Titan Text Embeddings V2 at 1024 dims; binary quantization cuts storage 32x for roughly 5 to 10% recall cost. For the store, pgvector on Aurora works if you already run Postgres and stay under a few tens of millions of vectors. OpenSearch Serverless NextGen (no OCU floor since May 2026) suits always-on traffic. S3 Vectors at 2.50 USD per 1M queries plus 0.06 USD per GB-month suits rarely queried corpora. Avoid the classic OpenSearch Serverless collection for prototypes: its 2-OCU minimum at 0.24 USD per OCU-hour is a floor of about 350 USD per month before you index a single document.

Query path: embed, hybrid RRF over dense plus BM25, top-30, rerank to top-5, generate, grounding guardrail on the output, then async eval sampling on 5 to 10% of live traffic.

flowchart LR
    A[S3 documents] --> B[Recursive chunker 512 tokens overlap 64]
    B --> C[Embed Titan V2 1024 dims]
    B --> D[BM25 index]
    C --> E[Vector store pgvector or OpenSearch Serverless]
    F[User query] --> G[Embed query]
    G --> H[Hybrid RRF top 30]
    D --> H
    E --> H
    H --> I[Reranker to top 5]
    I --> J[Generate answer]
    J --> K[Contextual grounding guardrail]
    K --> L[Answer with citations]
    L --> M[Eval sample of live traffic]
    M --> N[CI gate recall faithfulness cost]

The index is an anchor, not a memory. It holds the answer in place while the model, the prompt, and the provider version drift around it. That is why I spend more time on the anchor, chunking and fusion, than on the model choice.

The eval gate as code

The gate is a Pydantic model with hard thresholds, run in CI on the golden set and failing the build on regression. My production targets, adjusted on our corpus: recall@30 above 0.90 at the reranker input (that k, not the generation k), recall@5 above 0.85 after reranking, faithfulness above 0.95, refusal accuracy near 1.0 on unanswerable questions. Judge scores come from a cheap model calibrated against 40 human labels. I replay cases with pydantic-evals and ship traces through OpenTelemetry and Logfire with per-query cost attribution.

from pydantic import BaseModel, Field, field_validator
class EvalCase(BaseModel):
    model_config = {"strict": True}
    query: str = Field(..., min_length=1, description="Real user question from logs.")
    golden_context_ids: list[str] = Field(..., min_length=1, description="Chunks that must be retrieved.")
    must_refuse: bool = Field(default=False, description="True when the corpus has no answer.")
class RagGate(BaseModel):
    model_config = {"strict": True}
    recall_at_30: float = Field(default=0.90, ge=0.0, le=1.0, description="Reranker input k.")
    recall_at_5: float = Field(default=0.85, ge=0.0, le=1.0, description="Post-rerank k sent to the LLM.")
    faithfulness: float = Field(default=0.95, ge=0.0, le=1.0)
    max_cost_per_query_usd: float = Field(default=0.005, gt=0.0)
    @field_validator("faithfulness")
    @classmethod
    def no_silent_releases(cls, v: float) -> float:
        if v < 0.90:
            raise ValueError("a faithfulness gate below 0.90 lets drift through")
        return v
class QueryCost(BaseModel):
    model_config = {"strict": True}
    query_embed_tokens: int = Field(..., ge=0)
    rerank_search_units: int = Field(..., ge=0, description="1 unit = 1 query + up to 100 chunks.")
    guardrail_text_units: int = Field(default=0, ge=0, description="1 unit = 1000 chars of source + query + response.")
    input_tokens: int = Field(..., ge=0)
    output_tokens: int = Field(..., ge=0)
    embed_usd_per_1m: float = Field(default=0.02, gt=0.0)
    rerank_usd_per_1k: float = Field(default=1.00, gt=0.0)
    guardrail_usd_per_1k: float = Field(default=0.10, gt=0.0)
    gen_input_usd_per_1m: float = Field(default=0.06, gt=0.0)
    gen_output_usd_per_1m: float = Field(default=0.24, gt=0.0)
    @property
    def total_usd(self) -> float:
        return (self.query_embed_tokens / 1_000_000 * self.embed_usd_per_1m + self.rerank_search_units / 1_000 * self.rerank_usd_per_1k + self.guardrail_text_units / 1_000 * self.guardrail_usd_per_1k + self.input_tokens / 1_000_000 * self.gen_input_usd_per_1m + self.output_tokens / 1_000_000 * self.gen_output_usd_per_1m)

The same model prices a query, which is the number finance asks about. With Nova Lite at 0.06 USD input and 0.24 USD output per 1M tokens, plus an 80-token query embedded at Titan V2 rates, the embed line is negligible. One Cohere rerank search unit at 1.00 USD per 1,000 searches adds 0.001 USD. A 2,580-token prompt and a 300-token answer at Nova Lite rates add about 0.00023 USD. The total lands near 0.0023 USD per query. Add the grounding guardrail: source plus query plus response is about 10,200 characters, roughly 11 text units at 0.10 USD per 1,000 text units. That is about 0.0011 USD, so the guardrail costs about as much as the reranker per query, and the invoice agreed!

[!WARNING] Rerank billing is per search unit: one query plus up to 100 document chunks, and any chunk over 500 tokens including the query is split and billed again. 350 documents in one request bill as 4 queries. Check the channel too: Cohere Rerank 3.5 costs 1.00 USD per 1,000 searches on the direct API and 2.00 USD per 1,000 queries on Bedrock.

Measured results

The chart shows my weekly eval dashboard over six weeks. The dip in the week of 2026-09-28 was the silent embedding bump, caught by the recall gauge before users complained. Eleven minutes. The rollback plus a guardrail threshold of 0.7 brought faithfulness back above 0.95.

{
  "type": "line",
  "data": {
    "labels": ["2026-09-14", "2026-09-21", "2026-09-28", "2026-10-05", "2026-10-12", "2026-10-19"],
    "datasets": [
      {"label": "Faithfulness, judge on 240-case golden set", "data": [0.97, 0.96, 0.91, 0.93, 0.95, 0.96], "backgroundColor": "#3b82f6"},
      {"label": "Context recall@30", "data": [0.93, 0.93, 0.88, 0.92, 0.94, 0.94], "backgroundColor": "#f59e0b"}
    ]
  },
  "options": {
    "responsive": true,
    "plugins": {"title": {"display": true, "text": "Weekly RAG eval scores, weeks of 2026-09-14 to 2026-10-19. Week 2026-09-28 dip = silent embedding-model bump, rolled back"}},
    "scales": {
      "x": {"title": {"display": true, "text": "Week starting (2026)"}},
      "y": {"title": {"display": true, "text": "Score (0 to 1)"}, "min": 0.8, "max": 1}
    }
  }
}

On hallucination control, two numbers are worth remembering. A December 2025 study using a DeBERTa cross-encoder NLI check cut the hallucination rate by 43% (from 14.2% to 8.1%) for about 180 ms of added latency. AWS reports that its contextual grounding check filters over 75% of hallucinated responses in RAG and summarization workloads, with thresholds from 0 to 0.99 and a 100,000-character source cap. The AWS documentation on grounding checks notes that streaming can deliver an irrelevant response before the guardrail flags it at the end, so for streaming UIs, buffer or post-check.

Although the guardrail blocks most ungrounded answers, it is not free: it adds latency on every call and, at my context sizes, roughly doubles the retrieval-side cost per query. The trade I make is running it on all traffic for high-stakes corpora and on a sampled slice elsewhere.

Verified price list, us-east-1, 2026-11-03:

All rerank and guardrail rates sit on the Amazon Bedrock pricing page as of 2026-11-03.

Limits and open tests

My golden set is 240 cases from one technical-document corpus in French and English, and the judge is calibrated on only 40 human labels, so the faithfulness line carries judge bias. The weekly numbers come from my own dashboard, not a public benchmark. Prices are list prices in us-east-1 and they move; the Bedrock channel and the direct Cohere API differ by 2x on the same reranker, so check the channel before budgeting. I did not test multilingual corpora or multi-hop queries, and I left streaming grounding aside. Not tested yet. This could easily be tested on your corpus: replay 200 real queries against brute-force k-NN and read the top-k overlap before trusting any vendor leaderboard.

Final zoom

More generally, a RAG system stays reliable after deployment when every stage carries its own gauge and every gauge is wired to a gate. Retrieval gets recall, generation gets faithfulness, the bill gets a per-query ceiling. Demo scores are postcards; the eval dashboard is the weather report.

If you want this wired into your own pipeline, we build exactly that kind of eval gate at SAMALGO services, and you can reach us through /contact/. Related reading: Agentic AI in Production: Surviving Real Traffic. Anyway, that is the whole toolbox.