LLM Evaluation in Practice: How We Test Across 32 Models

Introduction

LLM evaluation is how you find out whether a model, a prompt or an agent still does the right thing after you change something. For most teams the answer is still "we tried a few examples and it looked fine", which works until a provider updates a model or a prompt fix quietly breaks five other cases.

This article is a practical guide to how we evaluate LLM output at Mobile Reality, built around a real system: the evaluation suites in the open-source monorepo of MDMA, our TypeScript library that extends Markdown with interactive components like forms, buttons, approval gates and webhooks. It is for engineers and engineering leads who need an LLM evaluation framework that catches regressions before users do.

You will see how we test prompts across 32 model variants, why deterministic assertions do most of the work, where we use an LLM as a judge, how we test AI agents rather than single prompts, and how the same approach gates our own fine-tuned model.

Why LLM Evaluation Matters When You Ship Prompts

Most LLM evaluation guides focus on product teams testing their own AI assistant. If you ship prompts that other people's applications depend on, the problem is harder, and the stakes are higher.

For MDMA, correctness is very specific. An output is correct only if it parses without errors, follows schema constraints, marks personal data fields as sensitive, and contains the right number and types of components. Generic metrics designed for prose, such as coherence or fluency, are almost irrelevant to us.

The Challenge of Multi-Model Prompt Packs

We ship system prompts. Our prompt-pack package contains prompt variants tuned to specific model families, because the same instructions behave differently on Claude, GPT and Gemini. Some models respond better to explicit examples; others truncate aggressively under default token limits, which breaks multi-component documents halfway through.

Our compatibility matrix currently covers 32 tested model variants across OpenAI, Anthropic, Google and xAI, from GPT-4.1-nano to Claude Opus 4.8 and Gemini 3.5 Flash, with each one scored on one-shot generation, custom prompts, conversations and specific conversation flows. Every prompt change risks regressions across all of them, and every model update from a provider can break a prompt that worked yesterday.

Running an evaluation once is a check. Running it on every prompt change, across every supported model, is a quality system.

What We Needed From an LLM Evaluation Framework

Before choosing any tool, we listed our requirements:

• Deterministic assertions for structured output: we need to know whether a form has exactly three fields, not whether the form is "good".
• Custom evaluation logic tightly coupled to our own MDMA validator.
• Multi-provider support with easy switching between OpenAI, Anthropic, Google and OpenRouter.
• YAML-based test cases that engineers who do not work on prompts can read and write.
• Fully open source and local, so contributors can run evals without a platform account.

Why We Chose promptfoo

We looked at hosted evaluation platforms, including Braintrust, before settling on our approach. Hosted platforms are strong at dataset management, run history and production monitoring, and for a team building a user-facing AI product they are a sensible choice.

For us, two things decided it. Our assertions depend on internal workspace packages, @mobile-reality/mdma-validator, @mobile-reality/mdma-prompt-pack and @mobile-reality/mdma-cli, which a hosted scorer would need published or re-implemented. And contributors to an open-source project need to run evals locally with nothing but an API key.

promptfoo is open source, runs entirely locally and is configured in YAML or JavaScript. The deciding feature was its javascript assertion type: any .mjs file that exports a default function receives the LLM output and its configuration, and returns a structured { pass, score, reason } object. That maps directly onto our validator, which returns the same shape of result.

Our LLM Evaluation Framework

Our eval suite lives in the evals/ directory of the MDMA monorepo, as a pnpm workspace package that pulls in the validator, prompt pack and CLI as workspace dependencies alongside promptfoo itself.

Eleven Evaluation Suites

We now run eleven suites, each targeting a distinct aspect of prompt behaviour, with 157 test cases between them:

Running all core suites is a single command: pnpm eval:all. A separate set of suites, covered later in this article, runs against our own fine-tuned model.

Provider Configuration

Every suite uses the same provider pattern:

providers:
  - id: "{{ env.EVAL_PROVIDER or 'openai:gpt-4.1-mini' }}"
    config:
      max_tokens: 8192
      max_completion_tokens: 8192

The EVAL_PROVIDER environment variable accepts any promptfoo-compatible provider string, so switching to Claude Sonnet through OpenRouter is one line:

EVAL_PROVIDER=openrouter:anthropic/claude-sonnet-4-6 pnpm eval

Both maxtokens and maxcompletiontokens are set on purpose. OpenAI's reasoning models use maxcompletiontokens, while Anthropic models through OpenRouter use maxtokens. Setting both makes sure no test case is truncated mid-document, whichever provider runs it.

The Author Prompt Evaluation Suite

The author suite is the most important one. It tests the core mdma-author prompt, the system prompt that turns any LLM into an MDMA document generator.

Test Cases and Assertions

Each test case follows the same pattern: a user request that includes an exact MDMA blueprint, plus assertions about the generated output. A simplified contact form test looks like this:

- description: Generates a contact form matching blueprint
  vars:
    request: |
      Create a contact form matching this exact structure:
      type: form
      id: contact-form
      fields:
        - name: full-name
          type: text
          label: "Full Name"
          required: true
        - name: email
          type: email
          label: "Email Address"
          required: true
          sensitive: true
        - name: message
          type: textarea
          label: "Message"
          required: true
      onSubmit: contact-submitted
  assert:
    - type: javascript
      value: file://assertions/only-components.mjs
      config:
        allowed: [form]
    - type: javascript
      value: file://assertions/exact-field-count.mjs
      config:
        expected: 3
    - type: javascript
      value: file://assertions/has-sensitive.mjs

Every test case also inherits a defaultTest assertion, so the MDMA validator runs on every output automatically:

defaultTest:
  assert:
    - type: javascript
      value: file://assertions/validate-mdma.mjs
      config:
        exclude: [flow-ordering]

No document that fails basic validation can pass any test. The validator check is the floor; custom assertions layer on top.

Measuring Structural Quality

The author suite covers document types such as contact forms, employee records with personal data, KYC review workflows, HR onboarding, approval gates, webhook triggers and multi-step task lists. For each test we check structure: the expected component types, the right number of fields, sensitive: true on personal data, kebab-case IDs, and YAML values rather than JSON.

Each check is a binary pass or fail. Aggregated across all test cases, they give us an overall pass rate per model and prompt version, which is the number we compare across every prompt change.

Conversation and Flow Evaluation

Some of the hardest evals cover multi-turn conversations. MDMA documents can span several messages, with later messages building on components from earlier ones through binding expressions like {{form-id.field-name}}.

Multi-Turn Conversation Tests

The conversation suite holds 30 multi-turn test cases. promptfoo's conversational test type sends the full message history on each turn, so the model sees everything it said before. A passing multi-turn test requires that:

• The first turn generates a valid initial document.
• Later turns add new components without regenerating earlier ones.
• Binding expressions reference real component IDs from earlier turns.
• No turn breaks the validator's flow-ordering rules.

Evaluating Binding and Flow Behaviour

The flows suite tests binding expressions specifically. An MDMA document can specify that a button click triggers an action that updates another component, so the assertions check that the generated YAML contains valid {{componentId.fieldName}} patterns and that the referenced components exist. The has-bindings.mjs assertion starts with a targeted regex:

const bindingPattern = /\{\{[a-z][a-zA-Z0-9_-]*\.[a-zA-Z0-9_.-]+\}\}/g;
const matches = output.match(bindingPattern) || [];

if (matches.length > 0) {
  return { pass: true, score: 1, reason: `Found ${matches.length} binding(s)` };
}
return { pass: false, score: 0, reason: 'No binding expressions found' };

Bindings are harder than they look. A model that hallucinates component IDs produces bindings that parse correctly as YAML but point to nothing. The validator catches these with static analysis, and the evals make sure the prompt teaches models to generate bindings that resolve.

The Fixer Evaluation Suite

One of the most interesting parts of the setup tests the mdma-fixer prompt, a separate system prompt that takes a broken MDMA document and repairs it. The fixer suite feeds intentionally broken documents to the model and asserts that the output:

1. Still contains MDMA blocks, so the model did not strip the document.
2. Has zero remaining validation errors after repair.
3. Stays within a warning budget when the test sets maxWarnings.

The fixer-resolves-errors.mjs assertion implements these checks:

const blockCount = (output.match(/```mdma/g) ?? []).length;
if (blockCount === 0) {
  return {
    pass: false,
    score: 0,
    reason: 'Fixer output contains no ```mdma blocks, the LLM may have stripped the document',
  };
}

const result = validate(output, { exclude, autoFix: false });
const unfixedErrors = result.issues.filter((i) => i.severity === 'error');

if (unfixedErrors.length > 0) {
  const details = unfixedErrors
    .map((i) => `[${i.ruleId}] ${i.componentId ?? '?'}: ${i.message}`)
    .join('\n');
  return { pass: false, score: 0, reason: `${unfixedErrors.length} error(s) remain:\n${details}` };
}

return { pass: true, score: 1, reason: 'Fixer resolved all errors' };

This is structural evaluation at its purest: no subjectivity and no judge model, only a deterministic path through our validator. Repairing documents that also contain flow bindings is the hardest category, and it is where model differences show most clearly: some models fix the errors but break the bindings, others fix both but introduce new schema violations.

Building Custom Assertion Modules

The 42 assertion modules in evals/assertions/ are the core of our evaluation system, and the reason we chose a tool that runs our own code.

Why Generic LLM Evaluation Metrics Fall Short

Generic LLM evaluation metrics, such as semantic similarity to a reference answer or a judge's rating for helpfulness, were designed for prose. They work for chatbots, summarisation and question answering. For structured output, they are not enough: a document that is semantically close to the expected output can still have the wrong field types, missing attributes or IDs in the wrong case, and a judge asked to rate it might score it highly while our validator reports five errors.

Unlike traditional software testing, where you assert against one known-correct return value, LLM output testing validates a space of valid outputs against invariants. We need to know whether exact-field-count passes, not whether the document "seems right".

Domain-Specific Assertions That Encode Real Requirements

Our assertions encode domain rules that would be hard to capture in a judge prompt:

Each assertion encodes a rule from our validator or authoring guidelines. Running them in evals tells us the prompt teaches models these rules, rather than producing documents that only look right.

LLM-as-a-Judge: Where We Use It and Where Assertions Beat It

LLM as a judge means asking a second model to grade an output against a rubric. It is the right tool when the property you care about is real but cannot be written as code. It is the wrong tool for anything a parser, schema or regex can check, because it adds cost, latency and its own errors to every run.

Where We Use a Judge

We use judges in two places. The prompt-builder suite has six llm-rubric assertions that check behaviour a regex cannot see. One test generates a system prompt for a KYC workflow, and the rubric reads: the generated prompt must instruct the AI to show the KYC form only when the user says "start KYC review" or "verify identity", and fail if it says to show the form immediately. Another checks that an insurance claim flow keeps two forms sequential rather than merging them.

The conversation-flow suite uses a judge prompt to review whole conversations. Even there, the judge's verdict is cross-checked against our deterministic conversation validator, so a confident but wrong judgement does not pass on its own.

Where Assertions Win

Everything structural stays deterministic: component types, field counts, sensitivity flags, IDs, bindings and repair results. Some checks that sound like they need judgement turn out not to; our check that a generated prompt contains its expected sections is a set of regular expressions, and it has never needed more.

Our rule is to replace a judge with code whenever we can. When we cannot, we use a fast, inexpensive judge model, never the model being evaluated.

Testing AI Agents, Not Only Prompts

Prompt evals check one call at a time. AI agent evaluation has to check what happens across a loop: which tool the agent picks, whether it stays within its guardrails, and whether a conversation drifts into the wrong action several turns in. We apply the same approach to the agents we run outside MDMA.

Tool Routing and Guardrails on Our Own Agents

The agents on our own website have promptfoo suites of their own. Before each run, the system prompt and tool definitions are generated from the live source code, so the evals cannot drift from production.

• Website assistant: 19 single-turn tests, each run three times, covering real bug reports, feature questions and control cases, with hard gates that fail the run if the agent opens the wrong flow.
• Product agent for our recognition SaaS: 18 tests covering conversation flow, product values, competitor questions, grounding and off-topic requests.
• Visualisation agent: 17 tool-routing tests, including regression cases such as "vague tweak must refine, not jump to generate".

Multi-Turn Simulation

Single-turn tests missed one of our real bugs: the website assistant answered every individual message correctly but flipped into the wrong flow partway through a conversation. We caught it with a multi-turn simulator that runs the real tool loop with stubbed tools across six scripted conversations. The single-turn suite passed on both the old and the fixed prompt; only the simulator reproduced the bug.

That is the core lesson of AI agent testing: some failures only exist across turns, so at least part of your suite has to run the loop.

Scoring a Data Agent Against Facts

For a client's accounting workflow, we built a data agent that answers questions over financial records, and evaluated it differently again. Its 37 test questions cover counts, sums, comparisons, lookups, reconciliation, vague questions and unanswerable ones. Correctness is scored against the facts each tool returns, never against the prose, and the prose is only checked for two things: does it cite those facts, and does it state its limitations.

A deterministic version replays a scripted tool plan through the real agent loop with no model and no network, so it runs in CI on every change. A live version lets the model plan, and tracks pass rate per category, mean internal tool calls, p50 and p95 response time, and how often the system discards the model's prose and rebuilds the answer from facts.

Evaluating Your Own Fine-Tuned Model

The same framework gates the model we fine-tuned ourselves: an open-weight Gemma 4 26B-A4B model trained to turn a compact interface description into a valid MDMA document. A fine-tuned model needs a stricter gate than a prompt, because you cannot fix it with a one-line edit. We describe the training itself in our write-up on fine-tuning an open-weight LLM for generative UI.

A Held-Out Gate

The gate holds 95 scenarios that were kept out of training. Every output must pass the MDMA validator, with no partial credit. The current model passes all 95, and 181 of 181 cases across six own-model suites that mirror the main ones: holdout, author, custom prompt, fixer, flows and tool-calling guidance.

Why the Gate Mattered

An earlier fine-tune on the much smaller Gemma 4 E4B model, served with a 2,048-token context, topped out at around 90.5% valid on the same holdout. It looked fine in a demo. The gate is what showed that roughly one request in ten would have failed in production, and what told us the larger base model fixed it. What it took to serve the model once it passed, including cold starts, is covered in our guide to running a self-hosted LLM for business.

Benchmarking Formats, Not Only Models

Most model benchmarks hold the task fixed and compare models. We found it equally useful to hold the models fixed and compare output formats. Our generative UI benchmark asked Claude Opus 5, GPT-5.6-terra and Gemma 4 26B-A4B to produce the same 18 interface scenarios in five formats, five repeats each, and logged 1,647 generations with zero API errors.

The headline metric was the "every time" rate: the share of scenarios where all five repeats rendered. On MDMA, the small open-weight model matched Opus 5 at 94.4%. On the other formats it dropped by 39 to 44 points. Same models, same scenarios; the format decided the result. We unpack what that means for model choice in our comparison of small language models vs large ones.

We explain the benefits of structured LLM output without JSON schemas in a separate article.

Running Evals Across Multiple LLMs

Because every suite reads EVAL_PROVIDER from the environment, switching from GPT-4.1-mini to Claude Sonnet or Gemini is a one-line change. The default is openai:gpt-4.1-mini, which is fast and inexpensive for daily iteration, and the compatibility matrix in our README records how each of the 32 model variants scores on the author prompt.

OpenRouter makes this practical. Instead of managing keys for every provider, we use one OpenRouter key and route to any model with the openrouter: prefix.

What Model Differences Reveal About Prompt Quality

Running the same test cases on different models shows which prompt problems are universal and which are model-specific. Early on, some models generated component IDs in camelCase despite explicit instructions; unique-kebab-ids.mjs caught it consistently, and a more explicit constraint with a concrete example fixed it across all models.

Other failures are model-specific. One model family reliably generated bindings with component IDs that were off by a character, which told us the prompt's explanation of binding syntax was unclear for that family. Without deterministic evals, we would have learned about it from a bug report.

LLM Evaluation Metrics That Matter for Structured Output

Our evaluation is assertion-based, but we track aggregate metrics across runs to see trends. For structured output, these metrics are closer to software test coverage than to human rating rubrics:

• Pass rate: the share of test cases where every assertion passes.
• Every-time rate: the share of cases that pass on every repeat, for anything that runs more than once.
• Validation error rate: the share of outputs with at least one validator error.
• Field and flag accuracy: how often field counts and sensitivity flags are exactly right.
• Binding resolution rate: how often generated bindings point to real components.

These are binary, derived from code, and stable between runs. They show clearly whether a prompt version moved things up or down.

Regression Testing Between Prompt Versions

Before a prompt change ships, the suites should pass at least as well as before. Today that is enforced by convention: evals run as developer-triggered commands rather than CI gates, because they make network calls to model providers and take several minutes. The validator, spec and other packages are fully in our Turborepo pipeline, and because the assertions import those packages directly from the workspace, a changed validator rule is picked up on the next eval run.

Offline First

All of these are offline evaluations against fixed test data, not scores on live traffic. For a developer framework, that is the natural start: we control the cases, the blueprints cover specific edge cases, and results reproduce on any machine. Online evaluation needs a user base and production traffic, and we will add it when the data justifies it.

What We Learned: Eval-Driven Development

After building and extending these suites, a few lessons hold across prompts, agents and models. Together they amount to eval driven development: write the test that defines "correct" first, then change the prompt or the model until it passes.

Start With Assertions, Not Metrics

The most common mistake in LLM evaluation is starting with metrics. "Track accuracy, relevance, fluency" sounds rigorous but is vague for engineering purposes. The right starting point is assertions: what specific, verifiable properties must the output have?

Most assertion modules in our library started as a bug or a review note. Someone noticed JSON values where YAML belonged, and we wrote yaml-not-json.mjs. Someone noticed inconsistent IDs across conversation turns, and we wrote unique-kebab-ids.mjs. Build your assertion library from real failures, not from theory.

Use Judges Sparingly and Check Them

A judge is a model too, and it can be wrong. Use it only where code cannot express the rule, keep the rubric concrete enough that a human would agree with the verdict, and cross-check it with deterministic checks where you can.

Test the Loop for Agents

Single-turn tests are necessary but not sufficient for agents. Add at least a small multi-turn simulation that runs the real tool loop, because the expensive failures happen between turns.

The Real Cost of Evaluation

Running evals across many models is not free. We iterate on an inexpensive default model and run the full model matrix before a significant prompt change ships, which keeps costs manageable while still giving multi-model confidence where it matters. promptfoo caches responses by default, so re-running unchanged cases does not hit the API again.

Because MDMA is open source, the eval setup must also be accessible to contributors: clone the repository, add an API key to .env, run pnpm eval. No platform accounts and no shared credentials.

Matt Sadowski

CEO of Mobile Reality

Own the Model Behind Your AI Feature

We help teams decide whether a smaller, fine-tuned or self-hosted model can take over a narrow, high-volume task, and prove it before anything changes in production.

• Fine-tuning open-weight models for narrow tasks with a fixed output format.

• A held-out evaluation gate on your own cases before any switch.

• Serving behind an OpenAI-compatible endpoint, quantised for your hardware.

• Data that stays on infrastructure you control.

• An honest answer when a hosted API is still the better choice.

Book a consultationEstimate project

Conclusion

LLM evaluation is what turns prompts, agents and fine-tuned models from demos into systems you can change safely. Our approach, promptfoo as the runner, 42 custom assertion modules tied to our own validator, eleven suites for prompts, agent suites that run the real tool loop, and a held-out gate for our own model, lets us iterate quickly without silent regressions.

• Build assertions from real failures; deterministic checks should do most of the work.
• Use LLM-as-a-judge only for behaviour code cannot check, and cross-check its verdicts.
• Test agents across turns and score them against tool facts, not prose.
• Gate fine-tuned models on held-out cases with a strict validator before they replace anything.
• Measure reliability on every repeat, across every model you support.

The right evaluation tools are the ones your team will run on every change. If you want help putting evals around your own prompts, agents or models, our custom AI model development team and our AI automation agency both start there. And do check out the MDMA docs and demo; stars are appreciated.

Frequently Asked Questions

What is LLM evaluation?

When should I use LLM-as-a-judge?

How do you evaluate AI agents rather than single prompts?

Which LLM evaluation framework do you use?