01 · The problem

Score every customer call for satisfaction and sentiment, and know how far to trust each score.

For contact-centre operations and quality leads in lending and loan servicing. Today they hand-review a sliver of calls and hear back from the few customers who answer a survey, who skew unhappy. They need a score on every call, the transcript lines behind it, and an honest measure of how much it can be trusted.

Synthetic data 400 generated calls, two simulated annotators and simulated surveys. Every number on this page measures the system on that data, not on real callers.

02 · The approach

A model extracts facts. Code turns them into scores.

What it does

Each call gets a satisfaction score from 1 to 5 and a sentiment of happy, mild or not happy, each with a one-line explanation and the transcript lines that produced it. Calls too broken to score go to a human review queue instead. Scores roll up by agent, intent and day.

Why it is built this way

A model asked for “satisfaction, 1 to 5” gives an opinion nobody can audit or retune. Here the model only answers narrow, typed questions (was the request resolved, how many times did the caller repeat themselves, which turn shows it). Deterministic code does the scoring, so every score is reproducible, explainable and testable.

How it works

1Quality gateSpeaker labels, ASR confidence, length, dropped calls. Failures go to review, never scored.
2Split by requestOne call often carries several requests; each is judged on its own turns.
3ExtractTyped fields with evidence spans, from Jev or a rule-based backend.
4Score in codeSatisfaction from resolution and effort. Sentiment from what the caller said.
5CalibrateMap the score onto the survey scale, corrected for who answers surveys.

One pipeline, two interchangeable extractors

Only step 3 changes. Both backends fill the same typed schema, so the gate, scoring, calibration and review queue are shared, and the two can be compared side by side on the same labels. The rules backend is the baseline the model has to beat, and it runs with no network or key.

transcript→quality gate→split by request→extract typed facts→score in code→calibrate→score + reason + evidence

Backend A · JevNarrow typed questions answered with probabilities, so every field carries a confidence.

Satisfaction (holdout): 0.85 κw
Sentiment (holdout): 0.82 κ
Latency: 293 ms p50 per call
Cost: $4.14 per 20,000 calls

Backend B · Rules (offline)Lexicons and patterns; no network, no key; the baseline.

Satisfaction (holdout): 0.85 κw
Sentiment (holdout): 0.87 κ
Throughput: 6,331 calls/s, one core
Cost: $0

They tie on this synthetic data because the rules' lexicons were written against the same generator. On real calls, which vary far more, the gap between them is the first thing to measure.

Key decisions

From DECISIONS.md: what was chosen, what was rejected, and what would make us switch.

Decision	Chosen	Rejected	We would switch when
Who produces the score	The model extracts typed fields; code scores	Ask the model for a 1–5 score directly	A directly elicited score beat the coded one against human labels on the sealed set, and we could still explain it in one line
Satisfaction vs sentiment	Separate evidence for each	One blended score	The polite-but-unresolved gap stopped predicting churn or complaints on real data
Evidence	Every field carries validated transcript spans; unsupported counts are dropped	Trust extracted fields as given	Never for scored fields; spans are how a disputed score is settled
Bad transcripts	Quality gate routes them to human review	Score everything with lower confidence	ASR quality improved enough that the gate’s reasons rarely fire and reviewers agree the scores are usable
Multi-request calls	Score each request; worst-weighted roll-up (0.65 worst + 0.35 mean)	Score the call once	Surveys showed callers weigh requests evenly; the weights live in code and can be refit
Retrieval	No vector store, no retrieval	Embed transcripts and retrieve similar calls or policy text	Scoring needed outside facts, such as whether the agent quoted the right policy
Calibration	Weighted isotonic regression, error measured out of fold	Platt or ordinal logistic, linear rescaling	Real survey volume was large enough per band that a smoother parametric fit had lower out-of-fold error
Model backend	Jev: typed answers with probabilities, behind an interface with a rule-based backend	A general LLM with prompt-and-parse	Another backend matched agreement on the sealed set at similar cost and latency; the interface makes it a swap
Evaluation	200 dev / 100 sealed holdout, read once per run and ledgered; bootstrap and run-to-run noise floors	Iterate and report on the same labelled set	Never; the holdout is replaced with fresh labelled calls, not reused

03 · Results

At the human ceiling on satisfaction. Weak on finding the quiet churn risk.

Measured on the sealed holdout of 100 calls, evaluated once per system, unless marked otherwise. Intervals are 95% bootstrap over calls.

Agreement with humans, against the human ceiling

Two annotators labelled 300 calls. How often they agree with each other is the ceiling: a system that agrees with a human as well as humans agree with each other has nothing left to gain from more prompt tuning. 200 calls were used to iterate; 100 were sealed and evaluated once per system (the ledger refuses a second look).

Satisfaction 1–5 quadratic-weighted kappa, 95% CI

Sentiment Cohen’s kappa, 95% CI

Confusion matrices and the noise floor

Satisfied is not the same as polite

Satisfaction comes from resolution and effort; sentiment comes from what the caller expressed. Scoring them from separate evidence exposes the cell a single blended score hides: callers whose problem was not solved but who stayed polite. They do not complain on the call, and they are the churn risk.

Calls by satisfaction band and expressed sentiment

Churn-risk calls

Satisfaction 1–2 with sentiment happy or mild. Select one to open it in the explorer.

Calibration: a score is a ranking until it is anchored

The raw score is fit to post-call survey answers with a monotone (isotonic) map. Unhappy callers answer surveys more often, so each answer is weighted by the inverse response rate of its score band. Error is measured out of fold, never on the calls used to fit.

Reliability diagram mean predicted vs mean survey score, per band

04 · The product

Every score opens onto the lines that produced it.

Start with a guided example, or browse all 400 calls. Evidence is highlighted in the transcript, and each score carries the one-line explanation code generated for it.

Call explorer

Filter calls

Human review queue

Bad transcripts produce confident garbage, so they are never scored. Scored calls with low extraction confidence, or where the agent claimed a fix the caller disputed, are flagged for a look.

Roll-ups

By agent

By intent

05 · Limits and next steps

What this does not show yet

Synthetic data. Calls come from a closed set of templates with ASR noise on top, so agreement numbers are optimistic for any extractor. On real calls the human ceiling will be lower than 0.85 and must be measured.
The rules are not a result. The rule-based extractor matches Jev here because its lexicons were written against the same templates. The paired dev difference is within noise; on real paraphrase expect the rules to fall away.
The quiet churn risk is under-found. Polite, unresolved callers are flagged with no false alarms on the holdout, but most are missed.
Calibration rests on 83 surveys. It helps the Jev scores and slightly hurts the rule scores, whose raw scale was already close. Until calibrated error is clearly below raw, publish buckets, not numbers.
Adjudication is simulated as recovering the true label. The like-for-like comparison (system vs one annotator) does not depend on it.
One disclosed leak. An eight-call smoke test of the first question version included two holdout calls; the changes that followed were generic. Details in METHODOLOGY.md.

Now

Raise churn-risk recall: inspect missed dev calls, change one thing, keep it only if a paired comparison clears both noise floors.
Tighten Jev intent precision: require a caller turn the model cites before a model-only request is scored.

Score-distribution drift alert against a reference window.
A full night of about 20,000 calls in shadow, published as buckets until the reliability diagram sits near the diagonal.

Later

Replace synthetic data with a frozen sample of real calls and real annotators; measure the ceiling before any extraction work.
Fit calibration on real surveys and recalibrate on a schedule.