01 · The problem
Score every call for satisfaction and sentiment, and know how far to trust the score.
For contact-centre operations and quality leads in lending and loan servicing.
A sliver gets read
Quality teams hand-review a small sample. Most calls are never looked at.
Surveys are biased
Unhappy callers answer more often. In this data, 60% respond after the worst calls and 22–28% after good ones.
A bare score is not enough
When an agent disputes a score, the answer has to be transcript lines, not “the model said so”.
All data in this deck is synthetic: 400 generated calls, simulated annotators and surveys.
02 · The approach
A model extracts facts. Code turns them into scores.
Extract, don’t judge
The model answers narrow typed questions: was each request resolved, how many repeats, holds and transfers, which turn shows it. It never emits a score.
Two scores, separate evidence
Satisfaction comes from resolution and effort. Sentiment comes from what the caller said. The gap between them is the polite-but-unresolved churn risk.
Measure trust
Compare with two human annotators (the ceiling), seal a holdout, calibrate against surveys, and publish the error alongside the score.
03 · How it works
Five steps, each one testable on its own.
- 1Quality gateSpeaker labels, ASR confidence, length, dropped calls. Failures go to review.
- 2Split by requestEach request judged on its own turns.
- 3ExtractTyped fields with validated evidence spans.
- 4Score in codeWorst-weighted roll-up, one-line explanation.
- 5CalibrateIsotonic map to the survey scale, bias-weighted.
C0379 → scored 2: hardship / forbearance not resolved · sentiment mild
The explanation is generated by code from the extracted fields; the transcript lines behind it are stored with the score.
04 · Live demo
Every score opens onto the lines that produced it.
The report walks through three real calls from the data: a clearly satisfied caller, a polite caller who is a churn risk, and a call the quality gate sent to a person.
Open the demo →Call explorer
Transcript with evidence highlighted, both scores and their explanations, human labels where they exist.
Review queue
Every call that was not scored, or needs a look, with the reason.
Roll-ups
By agent and by intent: mean score, calibrated score, unresolved share, churn risk.
05 · Results
At the human ceiling on satisfaction. Weak on the quiet churn risk.
Main weakness
The quiet churn risk is under-found
Polite, unresolved callers on the sealed holdout: Jev finds 6 of 15, the rules 8 of 15, both with no false alarms.
And on this synthetic data the rule baseline is not worse than Jev: the paired dev difference is −0.016 [−0.046, +0.011] on satisfaction, within noise.
06 · How it was built and tested
The rules that matter are enforced by code, not good intentions.
Evaluation discipline
- 200 dev calls to iterate; 100 sealed, read through one function, ledgered by hash, a second run refused.
- Baseline run twice: 98.8% identical satisfaction, 99.4% identical sentiment.
- A change counts only if a paired bootstrap clears sampling and run-to-run noise. One did: sentiment +0.059 [+0.008, +0.110].
Engineering
- 56 pytest tests, no network: determinism, span integrity, gate, holdout guard, calibration, schema.
- Architecture tests: one module may call a model, one function may open the holdout, scoring never imports an extractor.
- Every model response cached; model id, question version and code version on every row.
- Coding-agent guardrails: an edit guard on data and acceptance tests, tests after every edit, a read-only reviewer.
07 · Limits
What this does not show yet.
- Synthetic data. Template-generated calls make agreement optimistic; on real calls the ceiling must be measured.
- The rules are not a result. They were written against the same templates; expect them to fall away on real paraphrase.
- Churn-risk recall is low: most polite, unresolved callers are missed.
- 83 surveys behind calibration. It helps Jev and slightly hurts the rules; until error is clearly below raw, publish buckets.
- Adjudication is simulated. The system-vs-one-annotator comparison does not depend on it.
- One disclosed leak: two holdout calls appeared in an early eight-call smoke test.
08 · Roadmap
Now, next, later.
Now
- Raise churn-risk recall on dev; keep a change only if it clears both noise floors.
- Tighten Jev intent precision: require a cited caller turn for model-only requests.
Next
- Score-distribution drift alert.
- A shadow night of about 20,000 calls, published as buckets until calibration holds.
Later
- Real calls and real annotators; measure the ceiling first.
- Calibration on real surveys, refit on a schedule.