Project · LLM evaluation data pipeline

Customer Support Evaluation using an LLM

A framework for evaluating customer support conversations with an LLM. This project walks through building data pipelines that normalize the conversation data, an LLM that scores each conversation against a defined rubric, and validation of the results against ground truth. It also provides metrics to measure the model's accuracy and support business processes.

The data

10,042 real customer support chats

The conversations come from ABCD, the Action-Based Conversations Dataset, published by ASAPP Research in 2021. Trained agents chatted with customers of a fictional online clothing store, following the store's written policies.

What each chat contains

  • The conversation, turn by turn, between customer and agent.
  • Every button the agent clicked, such as verify-identity or offer-refund.
  • The real reason for the contact, one of 55 intents such as a refund or a password reset, with the policy steps the agent should follow.

Because the buttons and the policy are both recorded, the right answer for policy questions can be computed from the data. The LLM's scores are checked against it.

What this project added

The dataset has no dates or agents, so these were simulated:

  • · 30 days of daily batches
  • · 40 agents in 4 teams, and two chat channels
  • · Customer survey ratings (CSAT)
  • · The contact reason the agent logged, wrong in about 8% of chats
  • · Three deliberate incidents, to test the monitoring
View the dataset on GitHub ↗Read the paper ↗Chen et al., NAACL 2021 · MIT licence

Interactive demo

Follow one conversation through the pipeline

Pick a chat and click through the eight stages to see the output of each one. Some stages have a “Try this” box.

Pick a conversation

Every required step, in the right order. The judge and the facts agree on everything.

1Ingest

Loader script → cx_eval.redact_batch() → raw_conversations (jsonb)

The record as the support system delivers it, with personal data masked before it is stored.

Scroll here to replay the run.

Recorded run from Oct 22 · judge: gpt-oss-20b · rubric v1 (the version in use)

Metrics

Judge quality, pipeline health and support performance

Hover a chart for exact values.

Model performance

How good is the LLM judge?

The judge's answers are checked against facts computed from the agent's logged button clicks, on every chat it scores.

83.4%

Agreement with the facts

1,498 checkable verdicts

23% → 85%

Identity failures caught

after one rubric fix: the judge must point to the exact button before deciding

95.3%

Correct format, first try

Replies that passed the format check without a retry; 24 more passed after one

Accuracy against the facts, same 135 chats for every judge setup

  • Original setup: gpt-oss-20b with the original rubric. Stricter rubric: same model and rules, but the judge must point to where each required button appears in the chat before deciding. Bigger model: a larger model with the original rubric.
  • “Identity failures caught”: of the 13 chats where the agent really skipped or got wrong a required identity check, how many the judge flagged. Click a setup to hide or show it.

Pipeline operations

How the pipeline runs

Every day's chats are loaded, checked and scored on a schedule.

10,042

Chats processed

over 30 daily batches

2.3%

Held back by validation

230 records, all from one bad batch

14.3 s

Judge response time

median; slowest 5% 52.5 s

2,536

Tokens per chat

prompt plus answer

Chats loaded per day

  • Passed checks
  • Held back by validation

Support operations

What it shows about the support team

The same scores summarised for support managers: chat quality, identity-policy compliance and customer ratings.

82.0

Quality score

0–100, from the rubric

87.6%

Identity checks done right

when the policy requires one

3.97

CSAT (simulated)

1–5, 2,482 survey answers

Identity checks done right, by team

From Oct 14, team T4 received most of the chats where agents skip identity checks. T4 dropped while the company line barely moved. Click a team to hide or show it.

Data architecture

From support chats to scores and alerts

Once a day, a scheduled job loads a batch of chats, checks and cleans it in Postgres, and has an LLM score each chat. Checks on the results open GitHub issues when something looks wrong.

CX evaluation pipeline architectureGrouped into data source, data pipeline and serving. Once a day a GitHub Actions job loads a batch of support chats, masking personal data before it is stored. SQL in Supabase validates and transforms it, sending failed records to quarantine. An LLM judge (gpt-oss-20b on the NVIDIA API) scores each chat, and scores are stored append-only in Supabase with marts built on top. Checks on each run open or close GitHub issues. The dashboard on Vercel shows an exported snapshot of the marts.
  1. Data source
  2. Simulated daily batches

    Support chats

    ABCD dataset

  3. Data pipeline· daily at 06:00 UTC
  4. GitHub Actions

    Load

    Redact and land the day's batch

  5. Supabase · SQL

    Validate & transform

    Schema checks, enrichment

    Supabase · SQL

    Quarantine

    Records that fail checks

  6. NVIDIA API · gpt-oss-20b

    LLM judge

    Scores against the rubric

  7. Supabase · Postgres

    Scores & marts

    Append-only, versioned

  8. GitHub Actions

    Checks & alerts

    Drift, volume, freshness

    GitHub

    GitHub Issues

    One issue per incident

  9. Serving
  10. Vercel · Next.js

    Dashboard

    This page

Daily batch, not a streamThe “Try this” box in the demo calls the judge live from Vercel; those answers are not stored. An hourly pg_cron check in Supabase raises an alert if a day's batch never arrives.

Load and check

A GitHub Actions job lands one day of chats, with names, contact details and IDs masked in the same transaction so personal data is never stored. SQL in Postgres checks each record against the expected fields; records that fail are held in quarantine instead of reaching the judge. The rest are cleaned and enriched.

Score with an LLM

For each chat, the job builds a prompt from the rubric, sends it to gpt-oss-20b and checks the JSON reply. Failed calls are retried on the next run. Scores are only ever added, each tagged with its rubric and model.

Watch and serve

Checks look for judge drift, volume drops and quarantine spikes. Each new problem opens a GitHub issue that closes itself when the check clears. This page shows an exported snapshot of the marts.