# RAG Evaluation Lab

- Period: Oct 2025 – Nov 2025
- Domain: Independent project, open source
- Role: Designed and built it end to end
- Key result: 60%, less manual evaluation time when comparing RAG pipelines
- Capabilities: AI evaluation and quality; Knowledge, retrieval and context
- Stack: TypeScript, React, Vite, Tailwind CSS, Express, OpenAI, Qdrant, LlamaParse, LLM-as-judge
- URL: https://farhanbaluch.com/work/rag-evaluation-lab

A workspace for building RAG pipelines side by side, running them on the same dataset, and judging which configuration actually answers best, before anything goes to production.

**AI angle.** Most RAG decisions (chunk size, chunking strategy, embedding model, top-k, prompt) are made by feel. This lab turns them into experiments: every pipeline runs on the same questions, an LLM judge scores the answers for groundedness and context precision, and the report says which configuration to ship and why.

**What this shows:**
- I treat retrieval design as an evaluation problem, with baselines, metrics and cost, not guesswork.
- I can build a complete AI product on my own: backend, vector store, judge, UI and reports.

### Problem

Teams building RAG systems change one setting at a time and eyeball a few answers. Without a controlled comparison, nobody can say whether a new chunking strategy or embedding model actually made answers better, or just different and more expensive.

### Approach

Make every stage of a RAG pipeline configurable, run several pipelines against the same dataset and questions, and judge the answers with an LLM using structured verdicts, so the comparison is fair and repeatable.

### What I built

- Dataset management: drag-and-drop uploads, sample datasets, and CSV import of evaluation questions, with LlamaParse for consistent document parsing.
- Pipeline configuration for every stage: parsing, chunking (recursive, semantic, layout and table-aware), embeddings, retrieval, LLM and judge, with a baseline and one-click duplication for experiments.
- An experiment runner that embeds chunks into Qdrant per pipeline, retrieves and answers each question with citations, and logs latency, tokens and cost as it goes.
- An LLM-as-judge service that returns structured JSON verdicts (pass or review), dimension scores normalized to 0–1 and a rationale per question.
- Reports that compare pipelines with metric deltas (pass rate, groundedness, context precision, p50/p95/p99 latency, cost per query), recommend a configuration, and export to Markdown or PDF.

### Outcome

- Comparing RAG configurations went from reading answers by hand to a scored report, cutting manual evaluation time by about 60%.
- Source: [github.com/farhan1301/RAG-Evaluation-Lab](https://github.com/farhan1301/RAG-Evaluation-Lab)