Tech
How we verify every quote in a coaching report
An AI coaching tool that invents a quote is worse than no tool at all. Here is the verification pipeline every Close-Rate report runs through, and what happens when a claim fails it.
The fastest way to lose a sales rep is to show them a coaching report that quotes something they did not say. They will find it, they will tell the team, and every genuine finding in that report dies with the fake one.
So the question we had to answer early was not how to make the coaching insightful. It was how to make it checkable.
The problem with asking a model to be careful
Large language models produce fluent text about the source material they were given. Most of the time the quotes are real. Occasionally a model will smooth a paraphrase into quotation marks, or attribute the customer’s line to the rep, because both produce a more coherent paragraph.
Prompting helps and does not solve it. Any approach that relies on the model choosing to be accurate fails at some rate, and the rate is not zero. If the output is going in front of the person whose performance is being discussed, a non-zero rate is the whole problem.
What we do instead
Every report goes through the same pipeline before anyone sees it, and that pipeline is deterministic code, not another model.
- The model returns structured blocks through a tool call, not prose. A quote is a field in an object, so it can be isolated and checked on its own.
- Every quote block is matched back against the transcript of the call it cites. Matching ignores punctuation, capital letters and spacing, and nothing else: the same words must appear in the same order, so a reworded sentence does not pass.
- A quote that fails the match is not printed as a quote. It is downgraded to a paraphrase card, which is visibly a summary rather than something in quotation marks.
- Findings carry the number of call segments they are based on, and a finding resting on too little evidence is marked as lower confidence rather than stated flatly.
- A word match proves the words are in the call, not who said them. When the matched words sit inside one speaker’s turn, the speaker named on the card is checked against that turn, and a mismatch marks the quote unverified. Speaker roles themselves come from the transcript’s speaker labels, which can be wrong.
The part that matters: what happens on failure
Plenty of tools have a verification step. The question is what it does when verification fails, and there are only two honest options: drop the claim, or show it with the uncertainty visible.
- Matches the transcript — printed as a quote card, in the rep’s exact words
- Does not match — downgraded to a paraphrase card, visibly a summary
- Printed anyway — the path that does not exist
We do both, depending on severity. What we never do is print the unverified version and hope. The report is allowed to say less than the model produced. It is not allowed to say more than the recording supports.
Why we are writing this down
Every vendor in this category will tell you their AI is accurate. That claim is unfalsifiable as stated, so it is worth nothing to you when you are evaluating.
A better question to ask any of us, including us: when your model produces a quote that is not in the transcript, what specifically happens to it? If the answer is a description of a prompt, the answer is that nothing happens to it.
Check it yourself
Import a call you remember well and read the report against the recording. That is the only evaluation of this that means anything.