Ragas: You Cannot Improve a RAG You Do Not Measure
Faithfulness, relevance, context precision. Three numbers that turn "it feels better" into a decision.
"It feels better" is how most people evaluate a change to a retrieval system. It is also how you ship a regression, because feeling better on the three questions you happen to test is not the same as being better.
Ragas turns that into numbers.
The metrics that matter
Faithfulness — is the answer actually supported by the retrieved context, or did the model fill a gap with invention? This is your hallucination measurement, and for client work it is the one that matters most.
Answer relevance — does the response address the question that was asked?
Context precision — of the chunks retrieved, how many were useful? Low precision means you are spending tokens on noise and diluting the model's attention.
Context recall — did retrieval find the information needed to answer at all? When this is low, nothing downstream can save you.
That last pair is the important diagnostic. It separates "we retrieved the wrong thing" from "we retrieved the right thing and generated badly" — two problems with completely different fixes that look identical from the outside.
Building the test set
This is the work, and there is no way around it. You need real questions with known answers.
Twenty is enough to start. Take them from actual user queries if you have them, or from the questions the client says people ask. Write down what a correct answer contains. Save it as a file in the repo.
Now every change — a new chunking strategy, a different embedding model, a rewritten prompt — produces a comparable number instead of an impression.
Wire it into CI
Once the test set exists, run it on every change to the retrieval pipeline and fail the build if faithfulness drops below a threshold you set. This is the same discipline as any other regression test, applied to a system where regressions are otherwise invisible.
The honest caveat
These metrics are computed using a model as judge, which means they carry their own error. Treat them as a directional signal on a fixed test set, not as absolute truth.
That is still enormously more than "it feels better." A number that is imperfectly measured but consistently measured will tell you which direction you moved, and direction is what you need when iterating.
Resources
- Repo: vibrantlabsai/ragas
- Docs: docs.ragas.io
- Video walkthroughs: YouTube: RAG evaluation tutorial
- What you are measuring: LlamaIndex chunking
Need this built properly?
I build secure, fast, bilingual platforms for clients across Egypt, Saudi Arabia, the UAE and Kuwait.
