All projects

RAG Evaluation Lab

A workspace for building RAG pipelines side by side, running them on the same dataset, and judging which configuration actually answers best, before anything goes to production.

Result
60%
less manual evaluation time when comparing RAG pipelines
Role
Designed and built it end to end
Domain
Independent project, open source
Period
Oct 2025 – Nov 2025

Problem

Teams building RAG systems change one setting at a time and eyeball a few answers. Without a controlled comparison, nobody can say whether a new chunking strategy or embedding model actually made answers better, or just different and more expensive.

Approach

Make every stage of a RAG pipeline configurable, run several pipelines against the same dataset and questions, and judge the answers with an LLM using structured verdicts, so the comparison is fair and repeatable.

What I built

  • Dataset management: drag-and-drop uploads, sample datasets, and CSV import of evaluation questions, with LlamaParse for consistent document parsing.
  • Pipeline configuration for every stage: parsing, chunking (recursive, semantic, layout and table-aware), embeddings, retrieval, LLM and judge, with a baseline and one-click duplication for experiments.
  • An experiment runner that embeds chunks into Qdrant per pipeline, retrieves and answers each question with citations, and logs latency, tokens and cost as it goes.
  • An LLM-as-judge service that returns structured JSON verdicts (pass or review), dimension scores normalized to 0–1 and a rationale per question.
  • Reports that compare pipelines with metric deltas (pass rate, groundedness, context precision, p50/p95/p99 latency, cost per query), recommend a configuration, and export to Markdown or PDF.

Outcome

Capabilities

Stack

  • TypeScript
  • React
  • Vite
  • Tailwind CSS
  • Express
  • OpenAI
  • Qdrant
  • LlamaParse
  • LLM-as-judge

Skills shown

  • Agent evaluation and LLM-as-judge
  • GraphRAG and RAG
  • TypeScript and Node.js

From the log

  • 20 Oct 2025
    Started the RAG Evaluation LabAn LLM judge scores every answer for groundedness and context precision, so choosing a chunking strategy or embedding model becomes an experiment with a winner, not a guess.

Everything on this site in one file. Copy it and paste it into your agent.

Loading…