RAG Evaluation Lab
A workspace for building RAG pipelines side by side, running them on the same dataset, and judging which configuration actually answers best, before anything goes to production.
- Result
- 60%
less manual evaluation time when comparing RAG pipelines - Role
- Designed and built it end to end
- Domain
- Independent project, open source
- Period
- Oct 2025 – Nov 2025
Problem
Teams building RAG systems change one setting at a time and eyeball a few answers. Without a controlled comparison, nobody can say whether a new chunking strategy or embedding model actually made answers better, or just different and more expensive.
Approach
Make every stage of a RAG pipeline configurable, run several pipelines against the same dataset and questions, and judge the answers with an LLM using structured verdicts, so the comparison is fair and repeatable.
What I built
- Dataset management: drag-and-drop uploads, sample datasets, and CSV import of evaluation questions, with LlamaParse for consistent document parsing.
- Pipeline configuration for every stage: parsing, chunking (recursive, semantic, layout and table-aware), embeddings, retrieval, LLM and judge, with a baseline and one-click duplication for experiments.
- An experiment runner that embeds chunks into Qdrant per pipeline, retrieves and answers each question with citations, and logs latency, tokens and cost as it goes.
- An LLM-as-judge service that returns structured JSON verdicts (pass or review), dimension scores normalized to 0–1 and a rationale per question.
- Reports that compare pipelines with metric deltas (pass rate, groundedness, context precision, p50/p95/p99 latency, cost per query), recommend a configuration, and export to Markdown or PDF.
Outcome
- Comparing RAG configurations went from reading answers by hand to a scored report, cutting manual evaluation time by about 60%.
- Source: github.com/farhan1301/RAG-Evaluation-Lab
Capabilities
Stack
- TypeScript
- React
- Vite
- Tailwind CSS
- Express
- OpenAI
- Qdrant
- LlamaParse
- LLM-as-judge
Skills shown
- Agent evaluation and LLM-as-judge
- GraphRAG and RAG
- TypeScript and Node.js
From the log
- 20 Oct 2025
Started the RAG Evaluation LabAn LLM judge scores every answer for groundedness and context precision, so choosing a chunking strategy or embedding model becomes an experiment with a winner, not a guess.