How We Built LLM Evaluation Into an Open-Source Monorepo

Introduction

Building a framework that asks LLMs to generate structured, interactive documents creates an unusual evaluation problem: how do you know the LLM is actually doing it correctly?

For MDMA — our open-source TypeScript library that extends Markdown with interactive components like forms, buttons, approval gates, and webhooks — correctness means something very specific. An LLM output is correct only if it parses without errors, follows schema constraints, handles PII sensitivity flags appropriately, and produces the right number and types of components. Generic evaluation metrics designed for prose — things like coherence or fluency — are almost irrelevant to us. We needed something custom.

This article is a practical step guide to how we built LLM evals into our monorepo, what we learned from evaluating platforms like Braintrust and promptfoo, and why the most valuable investment was writing our own assertion modules.

Why LLM Evaluation Matters for Framework Authors

Most LLM evaluation guides focus on product teams testing their own AI assistants. But if you are building a developer framework that powers other LLM applications, the evaluation problem is different.

We ship system prompts. Our prompt-pack package contains 30+ prompt variants tuned to specific models: Claude Opus, Claude Sonnet, GPT-5, Gemini 2.5, Grok 4, and more.

Every time we change a prompt, we risk regressions across all of those model families. Every time a model provider updates a model, we need to know immediately whether our prompts still produce valid documents.

Without a robust LLM evaluation system, we would be flying blind.

The Challenge of Multi-Model Prompt Packs

Our prompt variants are not cosmetic differences. The way Claude Opus processes instructions differs meaningfully from GPT-5 or Gemini Flash. Some LLMs respond better to explicit examples. Others truncate aggressively under default token limits, which breaks multi-component documents mid-way through generation.

Running evals manually across 10+ models is not realistic. We needed automated, repeatable evaluation that could tell us precisely which models were passing and failing which test cases.

Running an evaluation once is a check. Running it on every prompt change, across every supported model, is a quality system.

What We Needed From an LLM Evaluation System

Before choosing any tool, we listed our requirements:

• Deterministic assertions for structured output — we need to know whether a form has exactly 3 fields, not whether the form is "good"
• Custom evaluation logic tightly coupled to our own MDMA validator
• Multi-model provider support with easy switching between OpenAI, Anthropic, Google, and OpenRouter
• YAML-based test case definitions that non-LLM engineers on our team can write and read
• Fully open-source — our eval infrastructure should be inspectable and forkable by contributors

Choosing an Eval Platform

We evaluated several tools before settling on our approach. Two stood out: Braintrust and promptfoo.

Braintrust: The Enterprise-Grade Eval Platform

Braintrust is one of the most mature LLM evaluation platforms available today. It is built around the idea that evaluation is an ongoing, collaborative process — not a one-time gate. Braintrust provides a rich web UI for viewing eval results, tracking scores over time, and comparing model performance across datasets.

Braintrust supports multiple evaluation modes. Its offline evaluation runs against fixed datasets with expected outputs. Its online evaluation mode uses a tracing SDK that captures production traffic for continuous monitoring and scoring. For product teams who need both development-time evals and production monitoring, Braintrust packages that entire lifecycle.

Braintrust's LLM judge capabilities are sophisticated. You can define scoring functions in Python or JavaScript, configure LLM-as-a-judge evaluators with custom rubrics, and track evaluation metrics like semantic similarity across model versions. The dataset management UI is genuinely excellent — you can build test data incrementally, tag cases by category, and trace which eval runs used which data. For teams that need to connect offline and online evaluation seamlessly, Braintrust is a compelling choice.

What Braintrust Gets Right About LLM Evaluation

Braintrust's scoring model is worth studying even if you don't use the platform. Braintrust separates scores into expected versus actual, supports partial credit with floating-point scores between 0 and 1, and allows multiple independent evaluation metrics on a single test case.

Its approach to regression testing is clean: each eval run is versioned, and you can diff two runs to see which test cases changed status. For a team iterating on prompts frequently, that regression testing capability reduces manual review time significantly.

Braintrust's LLM judge quality is also high. Its built-in judges cover factual correctness, helpfulness, and instruction-following. Custom judges let you write rubrics tailored to your domain. The judge infrastructure is one of Braintrust's strongest differentiators from simpler eval tools.

The core challenge with Braintrust was that it is primarily a hosted platform. Our eval assertions depend on several internal monorepo packages — @mobile-reality/mdma-validator, @mobile-reality/mdma-prompt-pack, @mobile-reality/mdma-cli, and their transitive dependencies — wiring those into Braintrust would require, among other things, publishing them all or re-implementing the logic in a custom scorer. For an open-source project where contributors need to run evals locally with zero platform account required, the hosted model adds friction. Braintrust's evaluation model — designed around datasets with single expected outputs — also maps awkwardly to our use case, where correct outputs are a large space of valid documents satisfying structural invariants.

Why We Chose promptfoo

promptfoo is fully open-source, runs entirely locally, and has a plugin model that made our custom assertion modules straightforward to implement. Its configuration is YAML-first, which fits our monorepo's existing conventions.

The decisive factor was promptfoo's javascript assertion type: any .mjs file that exports a default function receives the LLM output and configuration, and returns a structured { pass, score, reason } object. That maps directly onto our validator, which returns the same shape of result.

When to Choose Braintrust Instead

We want to be direct: if you are building an AI product rather than a developer framework, Braintrust is likely the better choice. Braintrust's online evaluation, production monitoring, and LLM judge quality are mature.

Its evaluation metrics — latency, cost, quality scores, semantic similarity — are exactly what product engineers need when moving from development to production. The regression testing history it maintains makes it easy to see whether a model upgrade or prompt change caused a real quality shift.

For our specific constraints — open-source, fully offline, structurally complex outputs, tight coupling to an internal validator — promptfoo was the right fit. But Braintrust remains the platform we recommend to teams building user-facing AI applications.

Our Eval Architecture

Our eval suite lives in the evals/ directory of the MDMA monorepo. It is a private pnpm workspace package that pulls in @mobile-reality/mdma-validator, @mobile-reality/mdma-prompt-pack, and @mobile-reality/mdma-cli as workspace dependencies alongside promptfoo itself.

Eight Evaluation Suites

We ended up with eight evaluation suites, each targeting a distinct aspect of prompt behavior:

Running all suites is a single command: pnpm eval:all.

Provider Configuration

Every suite uses the same provider pattern:

providers:
  - id: "{{ env.EVAL_PROVIDER or 'openai:gpt-4.1-mini' }}"
    config:
      max_tokens: 8192
      max_completion_tokens: 8192

The EVAL_PROVIDER environment variable accepts any promptfoo-compatible provider string. Switching to Claude Sonnet via OpenRouter is one line:

EVAL_PROVIDER=openrouter:anthropic/claude-sonnet-4-6 pnpm eval

Both maxtokens and maxcompletiontokens are set simultaneously. OpenAI's reasoning models use maxcompletiontokens; Anthropic models via OpenRouter use maxtokens. promptfoo strips the parameter the model does not accept, so setting both is safe and ensures no test case gets truncated mid-document. When running against OpenAI providers directly, you can verify token consumption in the OpenAI dashboard under usage analytics.

The Author Prompt Evaluation Suite

The base suite is the most important. It tests the core mdma-author prompt — the system prompt that turns any LLM into an MDMA document generator.

Test Cases and Assertions

Each test case follows the same pattern: a user request that includes an exact MDMA blueprint, plus assertions about the generated output. A simplified contact form test looks like this:

- description: Generates a contact form matching blueprint
  vars:
    request: |
      Create a contact form matching this exact structure:
    type: form
      id: contact-form
      fields:

    name: full-name
    type: text
          label: "Full Name"
          required: true

    name: email
    type: email
          label: "Email Address"
          required: true
          sensitive: true

  assert:
    - type: javascript
      value: file://assertions/only-components.mjs
      config:
        allowed: [form]
    - type: javascript
      value: file://assertions/exact-field-count.mjs
      config:
        expected: 3
    - type: javascript
      value: file://assertions/has-sensitive.mjs

Every test case also inherits a defaultTest assertion — the MDMA validator runs on every output automatically:

defaultTest:
  assert:
    - type: javascript
      value: file://assertions/validate-mdma.mjs
      config:
        exclude: [flow-ordering]

No document that fails basic validation can pass any test. The validator check is the floor; custom assertions layer on top.

Measuring Structural Quality

The base suite covers 25 document types: contact forms, PII-sensitive employee records, KYC review workflows, HR onboarding documents, approval gates, webhook triggers, multi-step tasklists, and more.

For each test, we measure structural quality: does the output contain the expected component types? Does it have the right number of fields? Are PII fields marked with sensitive: true? Are IDs in kebab-case? Are all values YAML — not JSON?

These are binary pass/fail metrics per test case, but aggregated across all test cases they give us an overall quality score per model and prompt version — a snapshot of overall performance that we track across every prompt change.

Conversation and Flow Evaluation

Some of the most complex evals concern multi-turn conversations. MDMA documents can span multiple conversation messages, with later messages building on components from earlier ones using binding expressions like {{form-id.field-name}}.

Multi-Turn Conversation Tests

The conversation suite defines 11 conversations, each with multiple turns. promptfoo's conversational test type sends the full message history on each turn, so the LLM sees everything it has said before.

A passing multi-turn test requires that:

• The first turn generates a valid initial document
• Subsequent turns introduce new components without regenerating the previous ones
• Binding expressions reference actual component IDs from earlier turns
• No turn violates the validator's flow ordering constraints

Evaluating Binding and Flow Behavior

The flows suite specifically tests binding expressions. An MDMA document can specify that a button click triggers an action that updates another component. Testing these flows requires asserting that the generated YAML contains valid {{componentId.fieldName}} patterns and that referenced component IDs actually exist in the document.

The has-bindings.mjs assertion checks this with a targeted regex:

const bindingPattern = /\{\{[a-z][a-zA-Z0-9_-]*\.[a-zA-Z0-9_.-]+\}\}/g;
const matches = output.match(bindingPattern) || [];

if (matches.length > 0) {
  return { pass: true, score: 1, reason: `Found ${matches.length} binding(s)` };
}
return { pass: false, score: 0, reason: 'No binding expressions found' };

Bindings are harder than they look. An LLM that hallucinates component IDs will produce bindings that parse correctly as YAML but reference nothing. The validator catches these with static analysis. The evals ensure the prompt teaches LLMs to generate bindings that actually resolve.

The Fixer Evaluation Suite

One of the most interesting parts of our eval infrastructure tests the mdma-fixer prompt — a separate system prompt that takes a broken MDMA document and repairs it.

Testing Auto-Repair Capabilities

The fixer suite provides intentionally broken documents as input and asserts that the LLM output:

1. Still contains MDMA blocks (the model did not strip the document)
2. Has zero remaining validation errors after repair
3. Has a bounded number of warnings when the test specifies a maxWarnings config

The fixer-resolves-errors.mjs assertion implements all three checks:

const blockCount = (output.match(/```mdma/g) ?? []).length;
if (blockCount === 0) {
  return {
    pass: false,
    score: 0,
    reason: 'Fixer output contains no ```mdma blocks — the LLM may have stripped the document',
  };
}

const result = validate(output, { exclude, autoFix: false });
const unfixedErrors = result.issues.filter((i) => i.severity === 'error');

if (unfixedErrors.length > 0) {
  const details = unfixedErrors
    .map((i) => `[${i.ruleId}] ${i.componentId ?? '?'}: ${i.message}`)
    .join('\n');
  return { pass: false, score: 0, reason: `${unfixedErrors.length} error(s) remain:\n${details}` };
}

return { pass: true, score: 1, reason: 'Fixer resolved all errors' };

This is structural evaluation at its purest: no subjectivity, no LLM judge needed, just a deterministic code path through our validator.

The fixer-flow suite extends this further — repairing broken documents that also contain flow bindings. This is the hardest test category and where model differences are most visible. Some LLMs repair errors but break the bindings; others fix both but introduce new schema violations. Fourteen specific test cases for this scenario let us pinpoint exactly which failure modes appear per model.

Building Custom Assertion Modules

The 35 assertion modules in evals/assertions/ are the core of our evaluation system. They are the reason we chose promptfoo over platforms that rely primarily on LLM judges or generic evaluation metrics.

Why Generic Metrics Fall Short for Structured Output

Generic evaluation metrics — like semantic similarity between expected and actual output, or LLM judge rubrics for helpfulness and correctness — are designed for prose evaluation. They work well for chatbots, summarization, and Q&A systems. RAG evaluation has similar domain-specific needs: retrieval quality requires context precision metrics that generic LLM metrics simply do not capture.

For structured document generation, they are insufficient. A document that is semantically similar to the expected output might have the wrong field types, missing required attributes, or IDs that use PascalCase instead of kebab-case. A judge prompted to rate document quality might give it a high score while our validator reports 5 errors.

We need deterministic, code-level assertions. Unlike traditional software testing, where you assert against a single known-correct return value, LLM output testing must validate a space of valid outputs against invariants rather than exact matches. We need to know whether exact-field-count passes, not whether the document "seems right."

Domain-Specific Assertions That Encode Real Requirements

Our assertions encode domain knowledge that would be very difficult to capture in a judge prompt:

Each of these encodes a specific rule from our validator or authoring guidelines. Running them in evals gives us confidence that the prompt is actually teaching LLMs these rules, not just generating documents that look superficially correct.

Using an LLM Judge Where Deterministic Checks Fall Short

We use LLM judges in two specific places where deterministic checks are insufficient.

The prompt-has-sections.mjs assertion uses a judge to verify that a prompt-builder output contains the expected structural sections: an opening description, a component blueprint, and a closing usage note. Section presence is checkable with code, but the judge adds a quality layer that catches cases where the sections exist but are incoherent.

The guidance suite uses a judge to score how faithfully a model followed a custom user-provided instruction. Instruction-following quality is difficult to codify deterministically — a judge scores it more naturally.

We keep LLM judges to a minimum because they add latency, cost, and non-determinism to the evaluation pipeline. Every time we can replace a judge with a code assertion, we do. When we cannot, we choose a fast, cheap model for judging — not the same model being evaluated.

Offline Evaluation in Practice

All of our evals are offline evaluations. We run them against fixed test data, not live production traffic.

Why Offline Evals Come First

For a developer framework, offline evaluation is the natural starting point. We control the test data. The blueprints are hand-crafted to cover specific edge cases, and the assertions encode precise requirements. This gives us high confidence in results that are reproducible on any developer's machine.

Online evaluation — capturing and scoring real user traffic — makes more sense for product teams with actual production traffic. As an early-stage open-source project, our "users" are developers integrating MDMA into their own applications and sending wildly varied prompts we cannot predict in advance.

As MDMA matures and we gain insight into how developers use it in production, we will add online evaluation using a tracing layer. Braintrust's approach to online evaluation is worth studying here. Their production monitoring model — where LLM calls are proxied through a tracing endpoint and scored automatically — is a design pattern we plan to adapt. Braintrust's combination of offline LLM evaluations and online evaluation in a single platform is genuinely compelling for teams at that scale.

For now, offline evaluation gives us the coverage we need.

Running evals is only useful if you act on the results. Our workflow: modify a prompt variant in packages/prompt-pack/src/prompts/, run pnpm eval, review which test cases changed, iterate, run pnpm eval:all, and commit only when the targeted model variants pass. This loop takes 5–15 minutes per iteration. promptfoo caches responses by default, so re-running with the same inputs does not hit the API again — a meaningful cost saving during prompt iteration.

Running Evals Across Multiple LLMs

One of the most valuable aspects of our eval infrastructure is that multi-model testing is trivial. Because every eval suite accepts EVAL_PROVIDER as an environment variable, switching from GPT-4.1-mini to Claude Sonnet to Gemini Flash is a one-line change. We regularly run the full suite against:

• openai:gpt-4.1-mini — default, fast, cheap for daily iteration
• openrouter:anthropic/claude-sonnet-4-6 — our primary Anthropic model
• openrouter:google/gemini-2.5-flash — Google's fastest capable model
• openrouter:openai/gpt-4.1 — full GPT-4.1 for complex multi-component tests

OpenRouter is particularly valuable here. Instead of managing API keys for four different providers, we use a single OpenRouter API key and route to any model with the openrouter: prefix. One key, access to all the LLMs we need to test against.

What Model Differences Reveal About Prompt Quality

Running the same test cases against different LLMs reveals which prompt features are model-specific and which are universal.

We found early on that some LLMs generate component IDs in camelCase instead of kebab-case despite explicit instructions. The unique-kebab-ids.mjs assertion caught this consistently. We updated the prompt with a more explicit constraint and a concrete example, and the failure rate dropped across all models.

Other failures are model-specific. One model family reliably generates bindings that reference slightly wrong component IDs — off by one character. That tells us the prompt's explanation of binding syntax is unclear for that model's training distribution, and we need a different example or a different framing.

Without deterministic evals, we would not discover these failure modes until a developer reported a bug.

Evaluation Metrics That Matter

Our evaluation is heavily assertion-based, but we track aggregate metrics across test runs to understand trends. The LLM metrics that matter most for structured output are fundamentally different from prose quality metrics — they are closer to software test coverage than to human evaluation rubrics.

Specific Metrics for Structured Output

The primary metrics we track:

• Pass rate — percentage of test cases where all assertions pass
• Validation error rate — percentage of outputs with at least one validator error
• Field count accuracy — for exact-field-count tests, how often the count is exactly right
• Sensitive flag accuracy — for PII tests, how often all sensitive fields are correctly marked
• Binding resolution rate — for flow tests, how often generated bindings reference real component IDs

These are not soft quality scores. They are binary, derived from code assertions, and stable across runs. They give us a clean picture of where specific metrics are trending up or down across prompt versions.

Semantic Similarity as a Supplementary Signal

We use semantic similarity in one specific place: the guidance suite tests whether the LLM followed a custom user-provided instruction when generating a document.

Semantic similarity between the user instruction and the document's textual content — field labels, button text, section titles — gives us a signal for instruction-following that structural assertions do not capture. If the user asks for a form "with a friendly, conversational tone," semantic similarity between that request and the generated label text is a weak but useful supplementary measure.

It is not a primary metric. We do not rely on it to gate prompt changes. But it adds nuance to the quality picture that pure structural checks miss.

Regression Testing Between Prompt Versions

Regression testing is built into our workflow. Before any prompt change ships, the full eval suite must pass at least as well as it did before the change. We track pass rates per suite in the PR description, and a regression in any suite requires a written explanation before the PR can merge.

This is lightweight regression testing — manual enforcement through review convention rather than automated CI gates. We plan to add automated regression testing once the eval suite is stable enough to run on every pull request. Unlike LLM evaluation benchmarks that measure general model capability, our evals measure prompt-specific behavior — so they must run against our prompts, not against a static dataset.

Integrating Evals Into the Monorepo

Our eval package is a first-class member of the pnpm workspace. It lists @mobile-reality/mdma-validator, @mobile-reality/mdma-prompt-pack, and @mobile-reality/mdma-cli as workspace dependencies, which means pnpm install at the repo root makes all of them importable by the assertion modules and test scripts.

![pnpm workspace dependency graph showing evals/ importing packages/]

Turborepo and the Eval Pipeline

We use Turborepo to orchestrate all build and test tasks in the monorepo. Evals are not yet in the Turborepo task graph because they make network calls to LLM providers and take several minutes to complete. We keep them as manual, developer-triggered commands.

The validator and spec packages are fully in the Turborepo pipeline. When you change a validator rule, turbo build recompiles the package, and the eval assertions automatically pick up the new behavior on the next eval run — because they import directly from the workspace-linked package, not a cached build artifact.

What We Learned: A Guide for Engineering Teams

After several months of building and iterating on this evaluation infrastructure, here are the lessons worth sharing.

Start With Assertions, Not Metrics

Good evals are specific. The most common mistake we see in LLM evaluation guides is starting with metrics. "Track accuracy, relevance, fluency" — these sound important but are vague for engineering purposes. The right starting point is assertions: what specific, verifiable properties must the output have?

Every assertion module in our library started as a bug report or a manual review note. Someone noticed a generated document had JSON values instead of YAML strings - we wrote yaml-not-json.mjs. Someone noticed IDs were inconsistent across conversation turns - we wrote unique-kebab-ids.mjs. The test case library grew organically from real failure modes.

Build your assertion library from real failures. Do not theorize about what might go wrong.

We also invested zero effort in online evaluation in year one — offline LLM evaluations with precise assertions came first. That paid off: we run the full suite in under 10 minutes and get a reliable picture of prompt quality. Online evaluation requires a user base and production traffic. For developer tools especially, start offline and add online evaluation once you have the data. Braintrust's production monitoring model — where LLM calls are scored automatically against configured evaluation metrics — is the benchmark we will measure against when we add that capability.

The Real Cost of Evaluation

Running evals across multiple models is not free. Running 100+ test cases against GPT-4.1 at 8192 max tokens each adds up quickly.

Our approach: gpt-4.1-mini or gpt-5.4-mini for daily iteration, full cross-model coverage before any major prompt change ships. This keeps costs manageable while still giving us multi-model confidence where it matters most.

Braintrust has cost tracking built into its platform, which helps teams that need to budget evals efforts carefully. If you are running large-scale evaluation, tracking cost per test case and per model is worth instrumenting early.

Because MDMA is open-source, our eval infrastructure must also be accessible to contributors. promptfoo's local-first design serves this goal: clone the repo, add an API key to .env, run pnpm eval. No platform accounts, no shared credentials. Several test cases were contributed by developers who found gaps while integrating MDMA into their own projects — those assertion modules are now part of the core quality system.

Conclusion

Building LLM evals into an open-source monorepo is not just possible — it is necessary if you ship system prompts as a product. Without evaluation, you cannot know whether your prompts work across models, whether prompt changes cause regressions, or whether the failure modes you fixed yesterday stay fixed tomorrow.

Our approach — promptfoo as the runner, 35 custom assertion modules tightly coupled to our own validator, and eight evaluation suites covering the full range of document types and behaviors — gives us confidence to iterate quickly on prompts without fear of silent regressions.

We evaluated Braintrust seriously and came away with genuine respect for what it offers. For product teams building AI assistants, Braintrust is a strong choice. Its LLM judge infrastructure, online evaluation, regression testing history, and production monitoring capabilities are mature and well-designed. The evaluation metrics Braintrust tracks out of the box map directly to what AI engineers building user-facing applications need to measure.

For open-source library authors with deterministic output constraints and a need for fully local eval runs, promptfoo's plugin model and YAML-first configuration are a better fit — at least at our stage and scale.

The broader lesson is about evaluation philosophy. Generic LLM evaluation metrics are useful baselines, but the real leverage in any evaluation process comes from domain-specific assertions that encode what "correct" means for your specific output format. This applies whether you are evaluating AI systems for document generation, code synthesis, or data extraction. Build those assertions from real failure modes and let them drive your test case design.

The right evaluation tools are the ones your team will actually run on every prompt change.

Don't forget to check out our MDMA Docs and demo! Stars are appreciated!

Frequently Asked Questions

What is MVP in AI development?

An AI MVP represents the minimum set of features required to satisfy early adopters while requiring continuous model performance tracking and labeled data infrastructure from day one. According to MIT Sloan Management Review, an AI MVP must be monitorable for improvement from day one, as data serves as the most critical resource necessary even at the earliest stages. Unlike traditional software MVPs, you cannot simply add intelligence later; the AI component must constitute the core architecture from the initial release, focusing on validating one specific intelligent component rather than broad feature coverage.

What is the 30% rule in AI?

While specific 30% rules vary across different contexts, the article demonstrates significant efficiency gains through AI-powered development workflows rather than defining a single 30% rule. Teams using AI-powered tools report shipping 40-60% faster than those building manually, while Mobile Reality's shared component layer reduces frontend code effort by approximately 70%. AI MVP development typically spans six to twelve weeks compared to three to six months for conventional MVP development, representing substantial timeline compression.

How much does it cost to build a MVP app?

AI MVP strategies minimize financial exposure by reducing the capital required to test core hypotheses compared to traditional development cycles. The approach protects runway during critical early stages by enabling rapid validation with smaller investments and shorter timelines of six to twelve weeks versus three to six months for conventional development. By treating AI as infrastructure from day one and leveraging automation, startups avoid weeks of expensive stealth mode that increase burn rate while competitors test hypotheses publicly.

How to build an AI MVP?

The AI MVP development process follows six key stages: problem definition and AI feasibility assessment, data strategy and preparation infrastructure, model selection and iterative training, integration of AI components with core application code, continuous evaluation testing, and deployment with monitoring systems for model drift. Begin by defining a single high-value use case using the D1-D5 framework to evaluate Desirability, Data Readiness, Differentiation, Delivery Complexity, and Durability. You can expect ideation through prototype validation within four weeks using AI-powered code generation, with full deployment requiring additional weeks depending on integration complexity.