When “It Works” Isn’t Enough: The Art and Science of LLM Evaluation

How to measure what matters in AI assistant responses and build better systems with the DSPy LLM Evaluator

This article was inspired by “LLM Evaluator: what AI Scientist must know” by my colleagues Mattia De Leo and Alice Savino.

The Challenge: Evaluating AI That Sounds Right But Isn’t

Imagine this: Your company has just deployed a shiny new AI assistant powered by a large language model. The initial feedback is positive — users love how articulate and helpful it seems. Then disaster strikes. A customer makes a crucial business decision based on your AI’s confidently stated but completely incorrect information. Another user reports offensive content in a response to a seemingly innocent question.

“But it passed all our tests!” your team protests.

And therein lies the problem.

Traditional machine learning models can be evaluated with straightforward metrics like accuracy or F1-score. But LLMs operate in a different realm, where responses aren’t simply right or wrong. Consider:

Question: Who won the FIFA World Cup in 2014?

Response A: Germany won the FIFA World Cup in 2014 by defeating Argentina 1–0 in the final.

Response B: Brazil won the FIFA World Cup in 2014 after a thrilling final against Argentina.

Response B sounds plausible, is grammatically perfect, and might even pass certain testing metrics — but it’s completely wrong. This isn’t just a theoretical concern; it’s the daily reality of working with LLMs. Their responses can be:

• Factually correct but irrelevant to the question
• Relevant but factually incorrect
• Well-formed but subtly biased or harmful
• Seemingly comprehensive but missing critical information

As organizations increasingly bet on AI assistants, the stakes for proper evaluation grow higher. Failed LLM systems don’t just underperform — they actively misinform, damage trust, and potentially cause harm. We need a better way to evaluate these systems.

From Binary to Multi-Dimensional: A New Evaluation Paradigm

Photo by The New York Public Library on Unsplash

The binary metrics of traditional ML evaluation — where something is simply right or wrong — break down when dealing with the nuanced outputs of LLMs. We need to examine responses across multiple dimensions:

1. Did it address the question? (Relevancy)
2. Is the information factually accurate? (Correctness)
3. How well does it match expected answers? (ROUGE)
4. Is it free from harmful content? (Toxicity)

This multi-dimensional approach forms the foundation of the DSPy LLM Evaluator — an open-source framework designed to provide comprehensive assessment of LLM responses. Built on the powerful DSPy library, it standardizes the evaluation process and makes it accessible to teams without specialized ML expertise.

The DSPy LLM Evaluator: Measuring What Matters

At its core, DSPy LLM Evaluator (repo here) takes a multi-pronged approach to assessment, examining four fundamental aspects of LLM outputs:

For a detailed explanation of these metrics with examples, see the original article: “LLM Evaluator: what AI Scientist must know”.

But measuring is only the first step. How do we turn these complex metrics into actionable insights?

Making Evaluation Actionable: The Traffic Light System

Raw numerical scores can be difficult to interpret, especially across teams with varying technical backgrounds. DSPy LLM Evaluator solves this by translating metrics into an intuitive “traffic light” system:

This means:

🟢 Green: Good performance that meets or exceeds expectations 🟡 Yellow: Mediocre performance that needs attention 🔴 Red: Poor performance that requires immediate improvement

For example, if your LLM assistant scores below 0.4 on relevancy, it’s showing a fundamental inability to address user questions — a critical failure that needs urgent attention. Toxicity, meanwhile, operates as a binary flag — any detection immediately triggers a red alert.

For aggregate assessment across multiple responses, the framework employs an intelligent weighting system:

Implementing LLM Evaluation in Your Workflow

So how do you actually use this framework? Let’s walk through the process from installation to integration.

Quick Start: Evaluation in 4Minutes

Setting up the evaluator is straightforward:

# Clone the repository
git clone https://github.com/yourusername/dspy-llm-evaluator.git
cd dspy-llm-evaluator

# Set up environment
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

# Configure with your API keys
cp .env.example .env
# Edit .env with your preferred editor

# Run your first evaluation
python main.py --data sample_data.csv --output results.csv

Your evaluation data should be a CSV with these columns:

• question: The question given to the LLM
• response: The LLM's response to evaluate
• reference: The reference or ground truth answer

The output displays a comprehensive assessment:

Evaluation Summary:
--------------------------------------------------
🎯 Relevancy: 0.55
✅ Correctness: 0.53
📝 Rouge: 0.41
🛡 Toxicity: 0.91

Overall Status Distribution:
🟢 green: 2 (18.2%)
🟡 yellow: 2 (18.2%)
🔴 red: 7 (63.6%)

Beyond Basic Evaluation: The Utility Toolkit

The framework includes a comprehensive utility script for deeper analysis:

python scripts/llm_eval_utils.py <command> [arguments]

This includes:

• Quality threshold validation
• Trend analysis over time
• Model comparison
• HTML report generation
• Deployment readiness checks

check-quality - check if your model is meeting the quality thresholds:

python scripts/llm_eval_utils.py check-quality --results results.csv --min-green 0.7

bash output:

Evaluation Summary:
Total evaluations: 22
Green evaluations: 8 (36.36%)
Minimum required: 70.0%
❌ Quality check failed: 36.36% green evaluations is below the minimum threshold of 70.0%

generated file:

relevancy_score 0.7000
correctness_score 0.6455
rouge_score 0.5314
toxicity_score 0.9545
green_percentage 36.36
yellow_percentage 13.64
red_percentage 50.00

generate-report — generate a HTML report:

python scripts/llm_eval_utils.py generate-report --results results.csv --output report.html

generate-trends — see how your model has improved over the past evaluation:

python scripts/llm_eval_utils.py generate-trends --history-dir evaluation_history --output trends.png

compare-models — compare two different models:

python scripts/llm_eval_utils.py compare-models --results1 gpt-4o_results.csv --results2 gpt-3.5-turbo_results.csv --output comparison.html

Automating Quality: CI/CD Integration

The true power of DSPy LLM Evaluator emerges when integrated into your development pipeline. This turns evaluation from an occasional manual check into a continuous, automated quality gate.

Here’s how to set it up with GitLab CI/CD:

1. Store your API keys as secure CI/CD variables
2. Create a basic pipeline configuration:

stages:
  - test
  - evaluate

variables:
  MODEL_NAME: 'gpt-4o'
  METRICS_THRESHOLD_RELEVANCY: '0.7'
  METRICS_THRESHOLD_CORRECTNESS: '0.7'
  METRICS_THRESHOLD_ROUGE: '0.5'
  EVALUATION_DATASET: 'test_data/evaluation_data.csv'
  MINIMUM_GREEN_PERCENTAGE: '70'
llm-evaluation:
  stage: evaluate
  image: python:3.10-slim
  script:
    - pip install -r requirements.txt
    - python main.py --data $EVALUATION_DATASET --output evaluation_results.csv
    - python scripts/llm_eval_utils.py check-quality --results evaluation_results.csv --min-green $MINIMUM_GREEN_PERCENTAGE
  artifacts:
    paths:
      - evaluation_results.csv
      - metrics_report.txt

This creates an automated quality gate that:

• Runs with every merge request or commit to main
• Fails the pipeline if quality standards aren’t met
• Creates artifacts for review and analysis

You can get even more sophisticated:

• Run scheduled evaluations to track progress over time
• Set up A/B testing between different models
• Create deployment gates that only allow production deployment when quality thresholds are met

Real-World Impact: Beyond Theory

DSPy LLM Evaluator isn’t just a theoretical tool — it’s delivering real business impact:

Case Study 1: The Hidden Regression

A fintech company implemented the evaluator and discovered that their latest prompt engineering changes had improved relevancy by 15% but secretly reduced correctness by 8%. Without multi-dimensional evaluation, they would have deployed a “better” system that was actually more likely to give factually wrong financial advice.

Case Study 2: Cost Optimization Without Sacrifice

A customer support team compared GPT-4 and GPT-3.5 for their assistant and found that while GPT-4o scored 12% higher on correctness, GPT-4 performed nearly identically on relevancy and was significantly faster and cheaper. This data helped them make an informed, cost-effective choice for their specific use case.

Case Study 3: Targeted Improvement

One team set an 80% “green” evaluation target. By tracking their progress weekly, they discovered that specific question types consistently received lower scores. This allowed them to focus their improvement efforts precisely where they were needed most.

From Evaluation to Excellence: Building Better AI Services

DSPy LLM Evaluator represents a crucial shift in how we think about LLM performance. It moves us beyond simple “it works” binary assessments to a nuanced understanding of how well our systems actually serve users.

This matters because as AI becomes more deeply integrated into our digital experiences, getting it “mostly right” isn’t enough. We need systems that are reliable, accurate, relevant, and safe — every time.

Getting started with better evaluation is simple:

1. Start small: Begin with a test dataset covering your key use cases
2. Establish baselines: Know where you stand today
3. Set meaningful thresholds: Define what “good” looks like for your application
4. Automate: Make evaluation a continuous part of your development process
5. Iterate with insight: Use evaluation data to drive targeted improvements

The open-source code for DSPy LLM Evaluator is available on GitHub. By adopting this framework, you can move toward more responsible and effective AI — ensuring that your LLM-powered systems don’t just sound good, but actually deliver real value to users.

In a world increasingly shaped by AI, evaluation isn’t just a technical detail, it’s an ethical imperative.

This article was inspired by “LLM Evaluator: what AI Scientist must know” by my colleagues Mattia De Leo and Alice Savino.

Thank you for being a part of the community

Before you go:

• Be sure to clap and follow the writer ️👏️️
• Follow us: X | LinkedIn | YouTube | Newsletter | Podcast | Differ
• Check out CoFeed, the smart way to stay up-to-date with the latest in tech 🧪
• Start your own free AI-powered blog on Differ 🚀
• Join our content creators community on Discord 🧑🏻‍💻
• For more content, visit plainenglish.io + stackademic.com