# Agent evaluation pipeline

- Period: May 2026 – Jul 2026
- Domain: Internal platform for an enterprise client
- Role: Designed the evaluation framework and built the harness
- Key result: 10+, quality dimensions scored per agent, from traces as well as answers
- Stack: Python, DeepEval, Promptfoo, OpenTelemetry, Jaeger, LLM-as-judge, Copilot CLI, FastAPI, React
- URL: https://farhanbaluch.com/work/agent-evaluation

Runs an agent programmatically, captures OpenTelemetry traces, and scores both the answer and the trajectory with LLM-as-judge plus deterministic checks.

### Problem

The organization wanted an internal marketplace of agents, skills and MCP servers, but had no way to decide which agents were safe to publish. Demo-quality impressions weren't a vetting process.

### Approach

Evaluate agents the way production systems are evaluated: from traces. Classify each agent (advisory, generative, workflow), generate a dataset for its class, run it headlessly, and score what it did as well as what it said.

### What I built

- A harness that classifies agents, generates and validates datasets, runs the agent programmatically and exports OpenTelemetry traces.
- Contract-based test scenarios per agent class, scored on 10+ quality dimensions with LLM-as-judge, programmatic checks and trajectory metrics (tool-call correctness, retries, failures), rolled into a scorecard and dashboard.
- Trust tiers (approved, community-vetted, beta) and a marketplace prototype: FastAPI and Postgres backend, React frontend.
- A companion spike with release gates and seeded regressions on a 15-case dataset.

### Outcome

- The trace view caught what answer-only grading missed: a correct final answer hiding a failed tool call.
- The evaluation framework became the vetting standard proposed for the marketplace.