# AI evaluation and quality

> I prove an agent is safe to ship before anyone has to trust it.

LLM-as-judge with deterministic checks, trace-based scoring of the whole trajectory, release gates with seeded regressions, semantic scoring for conversations, and AI-driven test design measured against injected bugs.

- Core skills: Agent evaluation and LLM-as-judge, Conversational AI testing, Risk-based test strategy, Playwright
- Domains: Telecom, Banking and financial services, Quality engineering, Legacy modernization, Conversational and voice AI, Developer platforms and tooling, Research and public sector
- URL: https://farhanbaluch.com/capabilities/ai-evaluation

## Case studies

- [Agent evaluation pipeline](https://farhanbaluch.com/work/agent-evaluation.md) (May 2026 – Jul 2026, Internal platform for an enterprise client): 10+, quality dimensions scored per agent, from traces as well as answers. An internal marketplace of agents, skills and MCP servers is only safe if something decides what gets published. This harness is that gate. It scores what an agent did, from its traces, as well as what it said.
- [Autonomous testing for a conversational voice agent](https://farhanbaluch.com/work/voice-agent-testing.md) (Jun 2026 – Jul 2026, National telecom carrier): <10 min, to generate branch-covering test cases for a call flow. Agents testing an agent. Worker agents hold real multi-turn conversations with a production voice assistant, a design agent writes the test cases from call-flow designs, and an LLM judge scores meaning rather than wording.
- [Risk-based test design with agents](https://farhanbaluch.com/work/risk-based-test-design.md) (Sep 2025 – Oct 2026, Internal R&D and client pitch): 9 of 9, injected bugs caught, against 1 of 9 for the standard set. AI assistants write tests faster than anyone can review them, and more tests isn't better coverage. This experiment showed that structured, risk-based agent workflows find bugs the default approach misses, and it became the basis for an agentic quality-intelligence offering.
- [Multi-agent legacy documentation](https://farhanbaluch.com/work/legacy-docs-agents.md) (Aug 2026 – Oct 2026, Client demo, embedded systems): 16 sections, every claim verified against cited lines. Modernizing with AI starts with understanding the old system, and an LLM summary that hallucinates is worse than none. Deterministic analysis, explorer agents and an independent reviewer together produce documentation people can trust, and that becomes context for the agents that do the migration.
- [Market research agents for retail banking](https://farhanbaluch.com/work/market-research-agents.md) (2025, Enterprise bank): 99.3%, extraction accuracy across 20+ bank websites. An LLM reads pages the way an analyst does, so the crawler survives layout changes that break scrapers. A vision model then checks every value against the rendered page before an analyst sees it.
- [Applied AI prototypes, from crops to small models](https://farhanbaluch.com/work/applied-ai-prototypes.md) (2023 – present, Public sector, small businesses and independent work): 25% → 75%, exact-match extraction accuracy after QLoRA fine-tuning of Llama 3.2. The models underneath agents: vision, retrieval and fine-tuning. Knowing when a small fine-tuned model beats a large prompted one, and how to measure it, is what lets me choose the right tool on client work.

## Evidence from the log

- 2026-10-08: **Pitched an agentic quality-intelligence offering.** Turned the risk-based test design results into a service offering, with a demo, a pilot plan and the evidence from the bug-injection study. AI angle: Packages agent-driven test design, AI-generated tests and coverage data into something a client can buy, backed by measured results rather than claims about AI. [Solution architecture and delivery; AI evaluation and quality] (https://farhanbaluch.com/work/risk-based-test-design.md)
- 2026-10-07: **Demoed multi-agent documentation for legacy firmware.** An orchestrator splits the codebase across explorer agents, and an independent reviewer agent rejects any claim it can't trace to a cited line. AI angle: Shows an agent system whose output an engineer can trust, because a second agent verifies every claim against the source before it reaches the report. [Agent architecture and orchestration; AI evaluation and quality] (https://farhanbaluch.com/work/legacy-docs-agents.md)
- 2026-09-15: **Added a reviewer agent to the documentation pipeline.** Early drafts read well but overstated what the code did. A separate reviewer agent now checks each section against the source and sends unsupported claims back. AI angle: Generator-reviewer separation applied to hallucination. The same model is never trusted to check its own work. [AI evaluation and quality; Agent architecture and orchestration] (https://farhanbaluch.com/work/legacy-docs-agents.md)
- 2026-07-28: **Scored agents on 10+ quality dimensions.** Extended the evaluation harness with contract-based scenarios per agent class, scoring correctness, tool use, safety and more with LLM-as-judge plus deterministic checks. AI angle: Gives the agent marketplace an objective bar for publishing. Each class of agent is judged against the contract it promises. [AI evaluation and quality] (https://farhanbaluch.com/work/agent-evaluation.md)
- 2026-07-16: **Added semantic scoring to voice-agent tests.** Exact-match checks failed whenever the assistant rephrased a correct answer. Switched to semantic-equivalence scoring so tests judge meaning, not wording. AI angle: Generative systems don't repeat themselves word for word. An LLM judge for meaning makes conversational testing reliable instead of flaky. [AI evaluation and quality] (https://farhanbaluch.com/work/voice-agent-testing.md)
- 2026-07-07: **Generated test cases from call-flow designs in under 10 minutes.** A Claude tool-use agent reads a call-flow design and writes one test case per branch, checked against a 22-case gold standard. AI angle: An agent does the slow design work a tester used to do by hand, and a gold standard keeps it honest. [Agent architecture and orchestration; AI evaluation and quality] (https://farhanbaluch.com/work/voice-agent-testing.md)
- 2026-06-24: **Built release gates with seeded regressions.** An evaluation spike for an SDK agent with deliberately broken versions, to prove the gates catch a regression before it ships. AI angle: Tests the evaluation itself. If a gate can't catch a known-bad agent, its passing scores mean nothing. [AI evaluation and quality] (https://farhanbaluch.com/work/agent-evaluation.md)
- 2026-06-15: **Architected autonomous testing for a production voice assistant.** Manual testing meant calling the assistant again for every branch. Designed an orchestrator that dispatches parallel worker agents to hold multi-turn test conversations. AI angle: Agents testing an agent. Worker agents play the customer, so coverage scales with compute instead of tester hours. [Agent architecture and orchestration; AI evaluation and quality] (https://farhanbaluch.com/work/voice-agent-testing.md)
- 2026-06-03: **Captured agent runs as OpenTelemetry traces.** Answer-only grading missed failed tool calls hidden behind a correct reply. Started exporting every run as a trace so the whole trajectory can be scored. AI angle: Evaluates agents the way production systems are monitored, so a lucky right answer with a broken tool call no longer passes. [AI evaluation and quality] (https://farhanbaluch.com/work/agent-evaluation.md)
- 2026-05-29: **Prototyped an internal agent marketplace.** A place to publish agents, skills and MCP servers, with trust tiers (approved, community-vetted, beta) and trace-based vetting before anything is listed. AI angle: Governance for agent sprawl. Teams share agents and MCP servers through one marketplace, and evaluation decides what they can trust. [AI adoption at enterprise scale; AI evaluation and quality] (https://farhanbaluch.com/work/agent-evaluation.md)
- 2026-03-23: **Generated contract and API tests with agents.** Extended agent workflows beyond UI tests to Pact contract tests and API suites, so service changes are caught before end-to-end runs. AI angle: Moves AI test generation down the test pyramid, where tests are faster and cheaper, instead of only producing more slow UI tests. [Agent architecture and orchestration; AI evaluation and quality] (https://farhanbaluch.com/work/copilot-agent-program.md)
- 2026-03-02: **Began adapting an agentic test-architecture framework for an enterprise.** AI assistants were writing tests fast, but many were redundant or asserted nothing. Started extending a test-architecture framework with connectors for the client's stacks. AI angle: Puts structure around AI-generated tests (risk registers, traceability, quality gates), so agents produce tests that matter rather than just more tests. [Solution architecture and delivery; AI evaluation and quality] (https://farhanbaluch.com/work/risk-based-test-design.md)
- 2026-02-09: **Caught 9 of 9 injected bugs with risk-based test design.** Ran a standard test set and an agent-assisted, risk-based set against the same feature with seeded bugs. The standard set caught 1. AI angle: Hard evidence that how agents are directed matters more than how many tests they write. [AI evaluation and quality; Agent architecture and orchestration] (https://farhanbaluch.com/work/risk-based-test-design.md)
- 2025-09-22: **Set up the risk-based test design study.** More AI-generated tests isn't the same as better coverage. Designed an experiment to measure whether structured, risk-based agent workflows find more defects. AI angle: Treats an AI workflow as a hypothesis to test, with seeded bugs and a control group, rather than a claim to believe. [AI evaluation and quality] (https://farhanbaluch.com/work/risk-based-test-design.md)
- 2025-04-14: **Reached 99.3% accuracy on bank-site extraction.** Added a validation pass that screenshots each page and has a vision model confirm every extracted value. Two weeks of market analysis now runs in under an hour. AI angle: One model extracts and another, with a different input (the screenshot), verifies. Cross-modal checking is what moved accuracy into the range analysts trust. [AI evaluation and quality; Knowledge, retrieval and context] (https://farhanbaluch.com/work/market-research-agents.md)