AI evaluation and quality
I prove an agent is safe to ship before anyone has to trust it.
LLM-as-judge with deterministic checks, trace-based scoring of the whole trajectory, release gates with seeded regressions, semantic scoring for conversations, and AI-driven test design measured against injected bugs.
- Case studies
- 6
- Log entries
- 15
- Domains
- Telecom, Banking and financial services, Quality engineering, Legacy modernization, Conversational and voice AI, Developer platforms and tooling, Research and public sector
- Core skills
- Agent evaluation and LLM-as-judge, Conversational AI testing, Risk-based test strategy, Playwright
Case studies
Agent evaluation pipeline
Runs an agent programmatically, captures OpenTelemetry traces, and scores both the answer and the trajectory with LLM-as-judge plus deterministic checks.
Internal platform for an enterprise client, May 2026 – Jul 2026
Read the case studyAutonomous testing for a conversational voice agent
An orchestrator and parallel worker agents hold multi-turn conversations with a production voice assistant, while a design agent turns call-flow diagrams into test cases.
National telecom carrier, Jun 2026 – Jul 2026
Read the case studyRisk-based test design with agents
Compared a standard test set against agent-assisted, risk-based test design on the same feature, then turned the results into a quality-engineering offering.
Internal R&D and client pitch, Sep 2025 – Oct 2026
Read the case studyMulti-agent legacy documentation
An orchestrator, a deterministic analyzer, parallel explorer agents and an independent reviewer turn an undocumented codebase into a verified, 16-section technical report.
Client demo, embedded systems, Aug 2026 – Oct 2026
Read the case studyMarket research agents for retail banking
An LLM crawler reads product and rate pages across 20+ bank websites, extracts structured data, and checks its own work against screenshots before anything reaches an analyst.
Enterprise bank, 2025
Read the case studyApplied AI prototypes, from crops to small models
Computer vision for crop disease and a policy chatbot for a federal department, AI prototypes for small businesses at the Vector Institute, and fine-tuning and RAG evaluation on my own time.
Public sector, small businesses and independent work, 2023 – present
Read the case studyEvidence from the log
- 8 Oct 2026
Pitched an agentic quality-intelligence offering
Turned the risk-based test design results into a service offering, with a demo, a pilot plan and the evidence from the bug-injection study.
AI anglePackages agent-driven test design, AI-generated tests and coverage data into something a client can buy, backed by measured results rather than claims about AI.
- 7 Oct 2026
Demoed multi-agent documentation for legacy firmware
An orchestrator splits the codebase across explorer agents, and an independent reviewer agent rejects any claim it can't trace to a cited line.
AI angleShows an agent system whose output an engineer can trust, because a second agent verifies every claim against the source before it reaches the report.
- 15 Sep 2026
Added a reviewer agent to the documentation pipeline
Early drafts read well but overstated what the code did. A separate reviewer agent now checks each section against the source and sends unsupported claims back.
AI angleGenerator-reviewer separation applied to hallucination. The same model is never trusted to check its own work.
- 28 Jul 2026
Scored agents on 10+ quality dimensions
Extended the evaluation harness with contract-based scenarios per agent class, scoring correctness, tool use, safety and more with LLM-as-judge plus deterministic checks.
AI angleGives the agent marketplace an objective bar for publishing. Each class of agent is judged against the contract it promises.
- 16 Jul 2026
Added semantic scoring to voice-agent tests
Exact-match checks failed whenever the assistant rephrased a correct answer. Switched to semantic-equivalence scoring so tests judge meaning, not wording.
AI angleGenerative systems don't repeat themselves word for word. An LLM judge for meaning makes conversational testing reliable instead of flaky.
- 7 Jul 2026
Generated test cases from call-flow designs in under 10 minutes
A Claude tool-use agent reads a call-flow design and writes one test case per branch, checked against a 22-case gold standard.
AI angleAn agent does the slow design work a tester used to do by hand, and a gold standard keeps it honest.
- 24 Jun 2026
Built release gates with seeded regressions
An evaluation spike for an SDK agent with deliberately broken versions, to prove the gates catch a regression before it ships.
AI angleTests the evaluation itself. If a gate can't catch a known-bad agent, its passing scores mean nothing.
- 15 Jun 2026
Architected autonomous testing for a production voice assistant
Manual testing meant calling the assistant again for every branch. Designed an orchestrator that dispatches parallel worker agents to hold multi-turn test conversations.
AI angleAgents testing an agent. Worker agents play the customer, so coverage scales with compute instead of tester hours.
- 3 Jun 2026
Captured agent runs as OpenTelemetry traces
Answer-only grading missed failed tool calls hidden behind a correct reply. Started exporting every run as a trace so the whole trajectory can be scored.
AI angleEvaluates agents the way production systems are monitored, so a lucky right answer with a broken tool call no longer passes.
- 29 May 2026
Prototyped an internal agent marketplace
A place to publish agents, skills and MCP servers, with trust tiers (approved, community-vetted, beta) and trace-based vetting before anything is listed.
AI angleGovernance for agent sprawl. Teams share agents and MCP servers through one marketplace, and evaluation decides what they can trust.
- 23 Mar 2026
Generated contract and API tests with agents
Extended agent workflows beyond UI tests to Pact contract tests and API suites, so service changes are caught before end-to-end runs.
AI angleMoves AI test generation down the test pyramid, where tests are faster and cheaper, instead of only producing more slow UI tests.
- 2 Mar 2026
Began adapting an agentic test-architecture framework for an enterprise
AI assistants were writing tests fast, but many were redundant or asserted nothing. Started extending a test-architecture framework with connectors for the client's stacks.
AI anglePuts structure around AI-generated tests (risk registers, traceability, quality gates), so agents produce tests that matter rather than just more tests.
- 9 Feb 2026
Caught 9 of 9 injected bugs with risk-based test design
Ran a standard test set and an agent-assisted, risk-based set against the same feature with seeded bugs. The standard set caught 1.
AI angleHard evidence that how agents are directed matters more than how many tests they write.
- 22 Sep 2025
Set up the risk-based test design study
More AI-generated tests isn't the same as better coverage. Designed an experiment to measure whether structured, risk-based agent workflows find more defects.
AI angleTreats an AI workflow as a hypothesis to test, with seeded bugs and a control group, rather than a claim to believe.
- 14 Apr 2025
Reached 99.3% accuracy on bank-site extraction
Added a validation pass that screenshots each page and has a vision model confirm every extracted value. Two weeks of market analysis now runs in under an hour.
AI angleOne model extracts and another, with a different input (the screenshot), verifies. Cross-modal checking is what moved accuracy into the range analysts trust.