Improve the RAG chatbot result

You can set a minimum threshold and short-circuit if all retrieved docs are below it, but that should just be your first gate, not the only one.

A better pattern in LangChain is to introduce an LLM-based grading step before you generate the final answer. After retrieving documents, send the user query plus the retrieved chunks to a small grading prompt. Ask the model to return structured output like “relevant: true/false” and maybe a confidence score. If the grader says the docs are not relevant enough, you either trigger your fallback (web search tool) or return a controlled “I don’t know” response.

You can also add a second validation step after generation. Generate the answer strictly from the provided context, then run another LLM pass that checks: “Is this answer fully supported by the provided documents?” If the validator detects unsupported claims, you reject the answer and either retry with different retrieval or return “I don’t know.” In LangChain or LangGraph this is usually implemented as a conditional branch in your chain rather than a simple linear pipeline.

1. I use an LLM to grade the retrieved documents (LLM reranking) to improve retrieval quality.

2. You can use an LLM to evaluate whether the retrieved documents are relevant to the query. If none of the documents meet a predefined relevance threshold, you can trigger a web search tool, rewrite the query and retrieve again, or return “I don't know” instead of generating a potentially hallucinated answer.

3. ChromaDB provides metadata filtering (https://docs.trychroma.com/docs/querying-collections/metadata-filtering), which can help improve retrieval precision. However, the filtering strategy depends on your specific use case and data structure.

P.S. There are many practical RAG techniques discussed in this article:
https://abdullin.com/ilya/how-to-build-best-rag/
You may find it helpful.

In my experience, the hard score threshold matters more than people think. If nothing clears that threshold, letting the LLM “try anyway” is where the bad answers start. I also found that a second relevance grader helps for borderline cases: sometimes the vector similarity is technically decent, but the chunks still are not sufficient to answer the actual question. So I treat the grader as a second gate, not as a polishing step.

On the Chroma side, the biggest retrieval improvement for me came from metadata filtering before similarity search, not after. If your collection mixes different users, document types, versions, languages, or topics, you should aggressively narrow the candidate set with metadata first. Chroma supports where filters for metadata and wheredocument for text-level constraints, and combining those two usually cuts out a lot of “semantically close but practically wrong” chunks.

• tenantid / userid

• doctype

• language

• version

• updated_at

• source

• section

• tags