Answer to: Improve the RAG chatbot result

In my experience, the hard score threshold matters more than people think. If nothing clears that threshold, letting the LLM “try anyway” is where the bad answers start. I also found that a second relevance grader helps for borderline cases: sometimes the vector similarity is technically decent, but the chunks still are not sufficient to answer the actual question. So I treat the grader as a second gate, not as a polishing step. On the Chroma side, the biggest retrieval improvement for me came from metadata filtering before similarity search, not after. If your collection mixes different users, document types, versions, languages, or topics, you should aggressively narrow the candidate set with metadata first. Chroma supports where filters for metadata and where_document for text-level constraints, and combining those two usually cuts out a lot of “semantically close but practically wrong” chunks. The most useful filters for me were things like: tenant_id / user_id doc_type language version updated_at source section tags