Tech
TypeSafe Jev at Close-Rate: results from 300 calls
On 300 calls, Jev retained more source evidence than the GPT-5.4-mini benchmark baseline. Results on retrieval, claim checking and scorecards, plus our production integration.
In our evaluation of 300 previously untested call records, TypeSafe’s newly released Jev model retained the original evidence for 135 coaching observations, compared with 113 for the GPT-5.4-mini baseline. It ranked each set of passages in a median 0.56 seconds versus 2.08 seconds, at approximately 81% lower model inference cost.
Our AI coach needs the right exchange before it can explain a missed question or recommend a better close. We tested Jev at that evidence-selection step, then ran separate experiments on claim checking and scorecards. Its strongest result was selecting the passages the coach reads.
Meet TypeSafe JevA decision model that returns scores and probabilities
Close-Rate now uses Jev 1.13.0 to rank retrieved call evidence for AI Insights and coaching reports, with GPT-5.6 Luna as the fallback. Luna also handles our former mini-model text tasks; Claude continues to write coaching advice and evaluate scorecards. The September 18 benchmark below compares Jev with GPT-5.4-mini, not Luna. We have not measured a Jev-versus-Luna accuracy advantage.
More original evidence reached the coach’s shortlist
For each coaching observation, we gave three ranking configurations the same 30 candidate passages from the same company. Each selected ten passages. We measured whether the original exchange behind the observation survived that selection.
Source-recovery hit rate: original exchange in the top ten
of 300 queries · higher is better
Among the 145 shortlists that contained the source, retention was 77.9% for the short-context baseline, 89.0% for full-context GPT-5.4-mini and 93.1% for Jev. Compared with the short-context baseline, Jev recovered 28 sources that GPT dropped and dropped six that GPT retained: a net gain of 22. The paired call-bootstrap 95% interval was +3.7 to +11.0 percentage points.
Longer context helped both providers. The baseline read only the first 400 characters of each passage. Giving GPT-5.4-mini the same longer passages as Jev raised its source count from 113 to 129. Jev’s six-source lead over that version had a 95% interval of −0.7 to +4.7 points, so the full-context quality comparison was inconclusive. Both production ranking paths now read up to 2,400 characters per passage.
Twelve authored controls isolated the context effect: when the answer appeared beyond character 400, every short-context configuration missed it at rank one. Full-context GPT and Jev both recovered all twelve.
3.7× faster ranking, 81% lower inference cost
Median time to rank 30 passages
seconds · lower is better
Across all 300 queries, estimated model inference cost was $1.00444 for short-context GPT-5.4-mini, $1.99139 for full-context GPT-5.4-mini and $0.19222 for Jev. The 95th-percentile ranking times were 2.73, 2.82 and 0.80 seconds respectively.
Jev’s Score primitive suits this workload. Instead of generating an explanation or a JSON score list token by token, it returns a score and a distribution over descriptive levels. Our code supplies the candidate passages and scoring criteria, then uses the returned scores to order the original passage identifiers. The generative coach receives the selected text unchanged. TypeSafe’s reranking cookbook describes the same retrieval-then-ranking pattern.
- 30 candidates retrieved within the authorized company
- 10 passages selected by the reranker
- Coaching still written by the generative model
All 300 source quotes matched their transcripts
Before ranking, we required each observation’s quote to match a unique word sequence in its source transcript. A second audit of the frozen sample confirmed that all 300 quotes matched after word normalization, and 243 matched verbatim, including case and punctuation. All 900 ranking results contained only their original candidate identifiers; neither model generated these quotes.
This establishes text provenance. It does not establish speaker identity, transcription fidelity to the recording, or whether the observation correctly interprets the exchange.
Claim checking caught more questionable observations, but rejected more valid ones
In a separate experiment on 82 saved observations from 47 calls, reference labels marked 49 as supported and 33 as needing review. We compared whether each model approved the supported observations and withheld those needing review.
- Supported observations retained: Jev 38/49; GPT-5.4-mini 41/49. Higher is better.
- Review-needed observations approved: Jev 17/33; GPT-5.4-mini 28/33. Lower is better.
- Separately authored overstatements rejected: Jev 40/40; GPT-5.4-mini 39/40. These were constructed errors, not errors found in customer reports.
Adding six checks reduced Jev’s review-needed approvals to 10/33, but retained only 29/49 supported observations. Most review cases lacked established speaker identity, rather than containing a demonstrably false statement. These assistant-authored reference labels were not independently adjudicated by sales experts. The observed trade-off does not support automatic factual approval.
Scorecard grading remained unreliable
We tested four scoring behaviors: cost quantification, pain summaries, objection resolution and accepted next steps. Jev met the frozen coverage-and-score checks on 51 of 72 controls; GPT met them on 52. These were authored test cases, with repeated templates, rather than independently graded customer calls.
Seller: Do you have any concerns or objections?
Authored test dialogue, not a customer call
Buyer: None, I have no concerns.
Seller: Great, goodbye.
Should this objection-handling example receive a grade?
Recorded Jev choice probabilities · expected answer: not applicable
Jev also assigned a behavior score of 1.42 out of 4 to that example, with confidence 0.0. A confidence cutoff would block this particular grade. Across the controls, however, both a 0.5 cutoff and a stricter 0.9 cutoff still accepted eight cases with inadequate roles or context; the stricter cutoff withheld more otherwise valid grades. TypeSafe’s confidence documentation distinguishes distribution concentration from correctness.
We kept Jev out of scorecard grading and automatic factual approval. In production it selects evidence; the generative coach interprets that evidence, and our existing quote-verification code checks the text.
Methods and scope
The ranking sample was selected from 590 development records after excluding previously studied calls and content duplicates. It spans four workspaces and September 2025–March 2026; 220 calls came from one workspace. Queries were saved model-written observations, with uniquely matched source quotes. BM25 produced same-workspace shortlists from 590 bounded passages, not live OpenSearch. The source reference identifies one original exchange, not every potentially relevant passage.
Models were Jev 1.13.0 and GPT-5.4-mini (gpt-5.4-mini-2026-03-17). We froze three sequential batches of ten, six concurrent ranking workflows, 20-second timeouts and no retries. The longer-context models received the same bounded passages but used different scoring interfaces and rubrics. Costs use reported tokens and public prices, excluding infrastructure and engineering.
The expanded ranking and scorecard experiments made 2,934 provider requests, costing approximately $3.24: 2,700 for ranking and 234 for scorecards. Scorecard requests covered 72 controls, 29 natural excerpts without reference grades and 16 repeats per model. The earlier claim-checking study used 609 requests, including controls and repeated configurations.
No provider HTTP failures occurred. Eight GPT-5.4-mini batches returned invalid score arrays: two short-context and six full-context. The research fallback restored shortlist order; replaying the old application’s neutral-batch merge changed full-context GPT’s source count from 129 to 130, leaving the other counts unchanged. Excluding the eight affected queries gave 110, 127 and 132 sources out of 292. One workspace favored the short-context baseline, 17 sources to Jev’s 16. Bootstrap intervals resample these calls, not new companies.
This was an AI-assisted engineering evaluation. End-to-end answer factuality, transcription accuracy, sales lift and production reliability were not measured. Production integration followed on September 20; its Luna fallback was not an arm in this benchmark.