LLM Evals: The Complete Systematic Setup Guide

The AI Project Failure Crisis

Industry reports indicate that 40% of agent AI projects are expected to be cancelled by 2027, and 95% of generative AI pilots are failing. These numbers reflect a real problem: most teams lack systematic evaluation frameworks to reliably ship and improve AI applications.

95%

of generative AI pilots failing

MIT research on AI pilot success rates

Core Challenges in LLM Development

LLMs are nondeterministic and context-dependent: responses can be factually correct but have wrong tone, multiple valid answers exist for one question, and failure modes are subtle and hard to predict. Evaluations help you systematically identify and fix these issues rather than discovering them in production.

Three Fundamental Problems Evaluations Solve

Data understanding: you don't know what users actually ask or how your system behaves at scale. Specification gaps: you know good output when you see it but can't translate that into clear instructions. Inconsistent behavior: small wording changes cause completely different outputs, making your system feel unreliable.

Three core challenges in LLM development

Success Depends on Three Components

Evaluation quality: systematic measurements through automated tests, human review, and success metrics. Debugging capability: tools and scripts to diagnose failures including trace logging and error analysis (Langfuse helps here). Change behavior: techniques to improve your system based on evaluation insights. Most teams only focus on the third component, preventing them from improving beyond the demo.

Three components of improvement cycle

The Analyze-Measure-Improve Lifecycle

Analyze: collect examples and categorize failure modes from real data or user complaints. Measure: translate insights into quantitative metrics (boolean pass/fail, scores 1-10, rankings). Improve: refine prompts, test different models, adjust architecture. This cycle addresses data understanding, specification gaps, and inconsistencies through systematic evaluation.

Evaluation-driven improvement cycle

What Makes an Evaluation

An evaluation (eval) is a systematic measurement of quality in an AI system. A single eval measures one specific aspect of performance (e.g., factual accuracy, tone, instruction-following). Your system can have multiple evals. Evals can run in the background for monitoring, be implemented as guardrails to block bad outputs in real-time, or be used as improvement tools for labeling data and creating test suites.

Level 1: Unit Tests (Fast & Cheap)

Simple automated assertions that check if code and prompt changes maintain core functionality. These are fast, cheap, and should run on every code or prompt change. Examples: verify output is in expected category list, check confidence score is between 0-1, confirm response length exceeds minimum. Store raw JSON events in your codebase and create Python assertions around them.

Level 1 unit tests characteristics

Level 2: Human & Model Evaluation (Weekly/Bi-weekly)

Systematic review combining human judgment with automated LLM critique. Start by having humans (engineers, domain experts) review real data examples to establish what good looks like. Only after humans set the standard should you use an LLM as a judge to automate evaluation. Use the most powerful model available (GPT-4, Claude Sonnet), generate detailed critiques not just scores, and continuously track human-model agreement. This alignment process is the most critical component for improving LLM applications.

Level 2 evaluation effort distribution

Level 3: A/B Testing (Major Releases)

Real users experiment to measure business impact of changes (different prompts, models, workflows). Compare metrics like user satisfaction score, task completion rate, time to resolution, engagement, or business outcomes. This is the most costly level because you need sufficient data to draw reliable conclusions. Only implement when your application reaches maturity and you have enough users.

Level 3 A/B testing characteristics

The Judge Alignment Process

Collect model predictions and generate model critiques. Get human evaluations on the same data. Compare model vs. human judgments. Iterate on the evaluator prompt. Repeat until sufficient agreement is reached. This is the most fundamental and critical component to improving LLM applications—almost everyone wants to skip it and use off-the-shelf tools, but that leads to meaningless metrics.

LLM judge alignment workflow

Practical Judge Alignment with Excel

Start with 10-100 real or synthetic examples. Run them through your system to get model outputs. Create a judge prompt (e.g., evaluate on accuracy, helpfulness, tone; provide detailed critique; rate 0 or 1). Loop the LLM over all examples to generate model critiques and scores. In parallel, have a human review the same input-output pairs and score them. Calculate agreement percentage. Use meta-prompting: export discrepancies and ask a powerful model to optimize your judge prompt to increase agreement. Iterate until alignment reaches acceptable level (ideally 100%).

Example alignment improvement through iteration

Judge Prompt Template

Start simple: 'Evaluate this customer service response on these criteria: accuracy (are facts correct), helpfulness (does it solve the problem), tone (is it professional and empathetic). Provide detailed critique explaining your reasoning. Rate 0 (bad) or 1 (good).' Begin with binary ratings and detailed critiques. Use the critiques to understand agreement patterns and refine the evaluator. Gradually make prompts more specific as you understand your domain better.

Reference-Based Metrics (You Know the Correct Answer)

Use when you have a golden answer or know what good looks like. Examples: exact string matching, semantic similarity, code execution results, SQL query correctness, structured data validation. These are easier to implement because comparison is straightforward. Many can be captured with Level 1 unit tests.

Reference-based metric types

Reference-Free Metrics (Multiple Valid Answers)

Use when there are multiple correct outputs (e.g., customer service responses). You don't know the single correct answer but can define quality dimensions. Examples: tone appropriateness, length constraints, no hallucinations, format compliance, safety and toxicity. These are harder to implement but necessary for most real-world LLM applications. Typically scored on scales like 1-5 or 0-1.

Reference-free metric types

Mistake 1: Tool-First Thinking

Jumping to solutions before understanding the problem. Examples: 'We have accuracy issues, let's use a better model' or 'We need metrics, let's buy an eval platform.' Fix: start simple and build custom solutions based on what you actually need. Understand your specific failure modes first.

Mistake 2: Generic Metrics Obsession

Drowning in meaningless scores like 'helpfulness 4.2, truthfulness 4.5' without understanding what they mean or how to act on them. Fix: focus on specific, actionable metrics. Start with booleans (pass/fail). Build metrics based on what you want to know from your system.

Mistake 3: Avoiding Your Data

Assuming tools or LLMs will catch issues without looking at real data yourself. This is a critical red flag. A principal engineer at Zapier spends 3 hours per day looking at traces and data. You cannot understand or improve AI systems without understanding what data flows through them. Fix: look at lots of real data constantly. This never stops.

3 hrs/day

Principal engineer at Zapier spends on data review

Why data inspection is non-negotiable

Mistake 4: Unaligned LLM Judges

Assuming an automated judge works without validation. You cannot optimize for what you don't understand. Fix: always validate judge alignment first by bringing humans in the loop. Never skip the human evaluation step before automating.

Unit Test Structure

Store raw JSON events in an /evals/events/ folder in your codebase. Create a simple Python script that loads each event, runs it through your workflow, and creates assertions around the output. For example: assert result.category in ['billing', 'technical', 'general'], assert isinstance(result.confidence, float), assert 0 <= result.confidence <= 1, assert result.category == 'billing'. Use try-except to catch assertion errors and report pass/fail.

Extending Unit Tests Across Workflow Steps

Don't just test the final output. Create assertions for every step in your workflow. Example: classification step should equal 'billing', router should direct to correct handler, task context should be populated, structured output should have reasoning field, final response should be over 50 characters, API call should return success=true. The more assertions you add, the stricter and more reliable your system becomes.

Scaling Unit Tests

Start with a simple file and folder structure. As your system grows, consider storing test cases in a database with metadata. Automate the process by capturing events directly from production or Langfuse. Use pytest library for more sophisticated testing setups. The key is having a repeatable way to test that every time you fix something, all previous test cases still pass.

Start Simple and Specific

Focus on your biggest problems first. Use existing tools before buying new ones. Build domain-specific evaluations—each project is different. Start with unit tests and manual review; this gets you far. Never pad with generic metrics.

Look at Lots of Data

Manual inspection has the highest value-to-effort ratio. Sample broadly, then focus on patterns. Never stop examining real examples. Remove all friction from viewing data—build custom tools for your domain if needed. This is the single most important practice separating top AI engineers from the rest.

Use LLMs to Scale, Not Replace

Generate test cases automatically. Create synthetic data at scale. Automate critique and labeling. But always validate with humans first. LLMs are tools to amplify your evaluation process, not to replace human judgment.

Know When You're Succeeding vs. Struggling

Succeeding: you deploy changes confidently, failures are caught before users see them, you understand your system's behavior, improvements compound over time. Struggling: everything breaks something else, you're surprised by user complaints, progress feels like trial and error, you can't measure if changes help.

Impact of systematic evaluation