Test Your Agents Like Code: AI Evaluation on Databricks

In April 2025, users of the AI code editor Cursor kept getting logged out when they switched machines. Support emails came back from "Sam", who explained that this was expected behavior under a new login policy. There was no such policy. Sam was an AI bot, and reports of cancelled subscriptions spread before a cofounder apologized on Reddit for an "incorrect response from a front-line AI support bot."

Nothing crashed and no alert fired. The bot returned a fluent, confident, wrong answer, which is how agents usually fail. Most teams can watch that happen but can't catch it. In LangChain's survey of 1,340 practitioners, 89% had observability on their agents, but only 52% ran offline evaluations and 37% evaluated live traffic.

We solved this problem for regular code a long time ago with test-driven development (TDD). AI evaluation is the same discipline for agents, and Databricks now ships most of the pieces. This post walks through them with one running example and shows what each piece gives engineers and the business. It's also the first part of The Other 15%, a Rearc series on controlling the share of answers an agent gets wrong.

So You Built an Agent. Now What?

Our running example is a support agent for a software product. It classifies the question, searches policy documents, looks up the customer's account, and drafts a reply. It passed the demo. Now it's in front of customers.

The usual checks miss agent failures. Agents rarely throw exceptions. The same input can produce different outputs on different days, so exact-match assertions don't work. And each answer comes from hidden decisions about what to retrieve, which tool to call and how to phrase the result, while the logs show only the answer.

The business sees the same problems from another angle. Gartner predicts that over 40% of agentic AI projects will be cancelled by the end of 2027, and names three causes: escalating costs, unclear business value, and inadequate risk controls. Behind each cause sit concrete problems that engineers hit every day, and Databricks has a feature for each one.

A better prompt fixes none of these. Visibility, repeatable measurement and controls do.

Evaluation Is TDD for Agents

TDD works because it turns "I think it works" into a check that runs on every change. Agent evaluation on Databricks, built on the open-source MLflow 3, has the same shape.

Traces, datasets and prompts live in Unity Catalog (UC), Databricks' governance layer, with the same access control, lineage and SQL access as your tables.

See It First: Tracing

You can't test what you can't see, so start with MLflow Tracing. A trace records one agent run as a tree of spans. Each span is one step, such as a model call, a retrieval or a tool call, with its inputs, outputs, latency and token usage.

MLflow integrates with more than 30 frameworks, so instrumentation is usually one autologging call plus a decorator for your own functions. Autologging is off by default on serverless compute, so call it explicitly.

import mlflow
from mlflow.entities import SpanType

mlflow.set_experiment("/Shared/support-agent")
mlflow.langchain.autolog()  # covers LangChain and LangGraph

@mlflow.trace(span_type=SpanType.RETRIEVER)
def search_policies(query: str):
    ...  # return mlflow.entities.Document objects so retrieval judges can read them

Here's what the trace of the answer looks like once you can see inside it (illustrative):

support_agent                          3.1 s   4,212 tokens
├── classify_intent     CHAT_MODEL     0.4 s   "account_access"
├── search_policies     RETRIEVER      0.3 s   0 documents
├── lookup_account      TOOL           0.2 s   status: active
└── draft_reply         CHAT_MODEL     2.2 s   "...expected under our new login policy"

The model didn't misread a policy. Retrieval found nothing, and the model filled the gap with something plausible. That calls for a different fix: tell the agent to escalate when no policy matches, and add a test that checks it. Without the span tree, you'd be tuning the wrong thing.

Three details matter in practice:

• Store traces in Unity Catalog. Traces should be stored in ordinary Unity Catalog tables in OpenTelemetry format, which allow you to query them with SQL, join them with business tables, and control access with standard grants, lineage, and audit.
• Let Databricks read traces too. Issue detection (Beta) scans traces for recurring problems, and you can use the plain English filters in the traces tab.
• Track cost where finance tracks it. Route model calls through Unity Gateway and every request lands in the system.ai_gateway.usage table with requester, model, tokens and latency (usage tracking). Spend comes from billing records in system.billing.usage, broken down by service, model, user and tag (cost observability). Budgets send an alert, block a user, or both, once monthly spend crosses a threshold.

A trace tells you what happened, not whether it was good. Deciding that is the next step.

Turn the Failure Into a Test

In TDD you write the failing test before the fix. For agents, the test case is a record in an evaluation dataset: inputs plus optional expectations, such as an expected answer or facts the answer must contain. Datasets live in Unity Catalog, are versioned, and hold up to 2,000 rows.

The assertion is a scorer. It receives the inputs, outputs, expectations and full trace, and returns pass or fail, a number or a label, ideally with a rationale. Because it sees the trace, it can check how the agent reached the answer.

Build a suite from cheapest to most flexible:

1. Code-based scorers. Plain Python for deterministic checks such as required fields, tool-call counts, latency and tokens. They're fast, free and never flaky. Write these first.
2. Built-in LLM judges. Ready-made checks such as RetrievalGroundedness (is the answer supported by what was retrieved?), Correctness, Safety and ToolCallCorrectness, plus seven judges for whole conversations.
3. Guidelines judges. Plain-language rules that an LLM checks as pass or fail. Product owners and compliance teams can write these without code.
4. Custom judges. make_judge() for domain criteria. A judge whose instructions reference {{ trace }} explores the whole run, for example to check that the right tool got the right parameters.

import mlflow
from mlflow.entities import SpanType
from mlflow.genai.scorers import (
    Guidelines, RetrievalGroundedness, RetrievalSufficiency, scorer,
)

@scorer
def no_invented_policy(outputs: str, trace) -> bool:
    # Deterministic: never cite a policy when retrieval returned nothing
    retrieved = trace.search_spans(span_type=SpanType.RETRIEVER)
    found_docs = any(span.outputs for span in retrieved)
    return found_docs or "policy" not in outputs.lower()

support_rules = Guidelines(
    name="support_rules",
    guidelines=[
        "Never promise a refund, credit or discount.",
        "If no policy document supports the answer, offer to escalate to a human.",
    ],
)

SUITE = [no_invented_policy, support_rules, RetrievalGroundedness(), RetrievalSufficiency()]

results = mlflow.genai.evaluate(
    data=mlflow.genai.datasets.get_dataset(name="main.support.eval_cases"),
    predict_fn=support_agent,
    scorers=SUITE,
)

For the business, a scorer is a written, versioned definition of "good." Before it exists, quality is just the opinion of the last person who tried the demo.

The trade-off is that an LLM judge is a model grading a model, with its own error rate. Treat its scores as a strong signal and calibrate the judge against your experts, as covered below in “Close the loop”. Also choose the judge's model with the model argument, and pick a different model family from your agent's, because models tend to favor their own text.

Fix It and Version the Prompt

Prompts change more often than any other part of an agent, and they're usually the least governed.

The MLflow Prompt Registry (Beta) treats prompts the way Git treats code. Each prompt is a Unity Catalog object. Each version is an immutable snapshot with a commit message. Aliases such as production point to a version and can move, and the agent loads its prompt by alias instead of hard-coding it.

v = mlflow.genai.register_prompt(
    name="main.support.reply_prompt",
    template="You are a support agent... {{question}} ... {{context}}",
    commit_message="Escalate when no policy document matches",
)

# The agent loads whichever version the alias points to
prompt = mlflow.genai.load_prompt("prompts:/main.support.reply_prompt@production")

# Promotion changes production behavior without a deploy
mlflow.genai.set_prompt_alias("main.support.reply_prompt", alias="production", version=v.version)

That last line is both the feature and the risk. Moving an alias is a fast release and a fast rollback, and it skips your deploy pipeline. Our rule is that an alias moves only after the evaluation suite passes on the new version. Product owners can still propose prompt changes. The suite decides whether they ship.

A dataset with scorers is an objective function, so a machine can search for a better prompt. mlflow.genai.optimize_prompts() (Beta) runs the GEPA algorithm. It evaluates the current prompt, reflects on failures, proposes variants and registers the best one as a new version. Databricks reports that a GEPA-optimized gpt-oss-120b beat a baseline Claude Opus 4.1 by 2.2 points on their extraction benchmark, at 90x lower serving cost. The catch is that it tunes exactly what your scorers measure on exactly your examples. This is how you can end up with an over-fitted evaluation, which is covered below in Tips and Pitfalls.

Before and After Release

Foundational scorers (like safety and guidelines) run before and after release to keep the core definition of "good" consistent. Adapt the evaluation strategy to what data is available: pre-production relies more on reference-based checks (like Correctness against known datasets with ground truth), whereas post-production emphasizes reference-free operational checks (like live groundedness and relevance) for unlabelled live traffic.

Pre-Production: Make Evaluation a Release Gate

Open-source MLflow 3.14 added @mlflow.test, a pytest plugin that turns an evaluation into an ordinary test. CI runs it like any other test suite.

# conftest.py contains: pytest_plugins = ["mlflow.pytest.plugin"]
import mlflow

@mlflow.test
def test_no_regressions():
    results = mlflow.genai.evaluate(
        data=mlflow.genai.datasets.get_dataset(name="main.support.regression_cases"),
        predict_fn=support_agent,
        scorers=SUITE,
    )
    assert results.passed, results.reason  # every case, every scorer

@mlflow.test
def test_not_worse_than_production():
    VARIANCE_MARGIN = 0.02
    results = mlflow.genai.evaluate(data=CAPABILITY_CASES, predict_fn=support_agent, scorers=SUITE)
    for metric, baseline in PRODUCTION_BASELINE.items():
        allowed_floor = baseline - VARIANCE_MARGIN
        actual_value = results.metrics[metric]
        assert actual_value >= allowed_floor

Practices that pay off:

• Split regression and capability suites. Anthropic recommends regression evals with a nearly 100% pass rate and capability evals that start low and climb.
• Compare to the production baseline. "No worse than production on any scorer" catches regressions an absolute threshold hides.
• Re-run on model upgrades. A provider's new model version changes your system even if your code didn't.
• Run each case more than once. A case that passes one run in three is a flaky test, and customers will see the failing run. Adjust VARIANCE_MARGIN or run multiple trials to produce more consistent results.

Post-Production: Score Live Traffic

Production monitoring (Beta) runs registered scorers against the experiment that receives your production traces. A background job scores a sample and attaches the results to each trace.

from mlflow.genai.scorers import RetrievalGroundedness, Safety, ScorerSamplingConfig

# Critical checks on every trace
Safety().register(name="safety").start(
    sampling_config=ScorerSamplingConfig(sample_rate=1.0)
)
# Expensive judges on a sample
RetrievalGroundedness().register(name="groundedness").start(
    sampling_config=ScorerSamplingConfig(sample_rate=0.1)
)

Practices that pay off:

• Sample by risk. Databricks suggests 100% for safety checks and 5 to 20% for expensive LLM judges.
• Use reference-free judges. Production traffic has no expected answers, so use groundedness, relevance, safety and guidelines instead of Correctness.
• Alert on trends. Scores are stored with the traces, so Databricks SQL alerts and dashboards work on them.
• Backfill new scorers. Run a new scorer over past traces (the last 7 days by default) to answer the question every incident raises: how many answers like this already went out?

Two gotchas. Custom scorers run remotely, so they must be @scorer functions with imports inside the function body. And monitoring compute is real compute, so attach a serverless budget policy to the experiment to bill the right team.

Close the Loop

Evaluation pays off when every production failure becomes a test and every test makes the next release safer. Databricks describes this as a quality loop:

The shortest path from failure to test is in the trace UI. Open the bad trace, add an expectation with the correct answer, and add the trace to the evaluation dataset. That's the agent version of "write a failing test that reproduces the bug," and your dataset grows from real failures instead of imagined ones.

Human judgment feeds the same loop. Developers annotate traces while they build. Domain experts label traces in review queues (Beta). End users click thumbs-up or thumbs-down, and mlflow.log_feedback() attaches the rating to the trace it came from.

Experts and generic judges will disagree on domain nuance. With judge alignment, experts correct the judge's verdicts on real traces, and judge.align() builds a judge that agrees with them more often. In a Databricks case study, about 50 expert labels cut judge error by 74 to 89% on the three dimensions where the judge did worst. The docs ask for at least 10 labeled traces, ideally 50 to 100, with at least 30% negative examples.

The loop fixes more than prompts. Retrieval judges separate "the context was wrong" from "the model misused good context." Tool-call judges expose unclear tool descriptions and redundant calls. And a switch to a cheaper model becomes a measured decision.

For the business, this is where AI quality stops being an opinion and becomes a trend line. Each pass through the loop produces a reportable change: the failure, the fix, and the scores that moved.

Tips and Pitfalls

Evaluation almost never breaks loudly. Nothing throws an error. It just slowly drifts until your scores look amazing and mean nothing.

• Teaching to the test. Optimizing continuously against the same evaluation suite triggers Goodhart's law, especially as automated tools rewrite prompts. For example, Databricks' prompt optimization quickstart tunes prompts on six examples without reserving a test set. That's fine for learning the API, but production needs held-out data. Databricks' evaluation guide recommends allocating roughly 70% of data for iteration, 20% for testing candidates, and 10% for final validation. The optimizer only ever sees the first 70%.
• Writing metrics before reading traces. Off-the-shelf metrics measure what's easy to measure, not what's broken. Hamel Husain and Shreya Shankar call error analysis "the most important activity in evals." Read real traces, sort what went wrong into a few failure types, and only then write scorers for them.
• Only looking at quality. An optimizer can "win" by tripling your prompt length. Track cost and latency next to quality.
• Trusting the judge too much. A judge you aligned on your experts' labels will agree with those labels. That tells you very little about the next batch. Check how many real failures it catches and how many good answers it lets through, on labels it has never seen.
• Treating scores like exact numbers. Judges make mistakes and agents give different answers from run to run, so your assertions are really about probabilities. Compare rates against the production baseline instead of exact values, and if a score moves a little, assume it's noise until a few more runs back it up.
• Expecting "good" to stay put. You figure out what "good" means by grading outputs, and your idea of it shifts as you go. Researchers call this criteria drift. So expect your guidelines and scorers to change. Pin the dataset and scorer versions, and whenever either one changes, re-score the production baseline so you're still comparing apples to apples.
• Letting the dataset go stale. A dataset you put together six months ago describes the users you had six months ago. Keep feeding it fresh failures from production.
• Forgetting that tests cost money. Every judge call is an LLM call, and it runs on every case, every repeat and every sampled trace. Start with code-based scorers, run the expensive judges on a sample, and give evaluation its own budget like any other workload.

Your best overfitting detector is production monitoring. Offline scores predict how the agent will do. Reference-free scores on live traffic show how it did.

Put the Controls at the Gateway

Evaluation tells you whether the agent is good. It can't stop a bad request in flight, cap a runaway loop, or keep a credit card number out of a model provider's logs. That's the job of Unity Gateway, formerly Unity AI Gateway. It applies the same controls to Databricks-hosted models, external providers and MCP servers (Model Context Protocol, the standard way agents call tools).

Two of these connect directly to evaluation. A policy deny returns HTTP 200 with a databricks_service_policy object, so your agent and scorers must recognize it, or monitoring will grade a block as a bad answer. And most built-in guardrails are LLM judges with false positives. Start new policies in Log mode, test them against real attacks and look-alike legitimate requests, and only then enforce.

On the Unity Gateway CLI episode of Streams, Lakes, and Deltas: Not a Nature Podcast, Riley Nastase set the test that runs through this series. An agent that's right 85% of the time still needs a control for the other 15%. Evaluation shows you where that 15% is, and the gateway limits what the agent can reach and spend while you fix it. Next in The Other 15%: routing, so your most expensive models only handle the work that needs them

Confidence Comes From Evidence

Agents fail quietly, so confidence in them has to come from evidence. Databricks covers the whole path: MLflow Tracing shows what the agent did, scorers and LLM judges define what "good" means, evaluation datasets in CI catch regressions before release, the Prompt Registry tracks what changed, monitoring keeps scoring live traffic, and Unity Gateway controls access, spend and content on every request.

Together they answer Gartner's three reasons projects get cancelled: costs you can see and cap, value you can show as scores over time, and risks you've tested. That's what TDD gave regular code: you move faster because you find out sooner when something breaks.