New Study Shows Why Booking Assistants Need Tested Review Thresholds, Not Defaults

What the study actually measured

On October 20, 2026, Hamel Husain and Antaripa Saha published a practical study on how automated judges of customer conversations make their calls, and why the cutoff point they use matters more than most owners realize. The study evaluated six model configurations on apartment-leasing conversations, checking whether each could detect failed bookings and missed staff handoffs. Its most useful lesson for a business owner is not which model scored best. It is that a default decision threshold, left untested against your own examples, quietly decides how many failures you catch and how many false alarms your staff end up ignoring.

The published dataset is small and specific. The source collection contains 100 conversation traces, but only 98 production traces were scored: 61 with a known failure and 37 without a labeled failure, after two internal quality-assurance examples were excluded. The authors gave six configurations, including Jev, Perplexity Decider, Liquid's d1 and d1-ultra, Clef Flash, and OpenAI Decisions, the same evaluation criteria, then compared each model's default 0.50 threshold against thresholds tuned with repeated stratified five-fold cross-validation. Precision, recall, F1 and probability calibration were tracked, and the authors released chart data and a threshold-selection audit so readers can inspect the numbers behind the curves.

The number that should get your attention

Because the original conversations are private, nobody outside the authors can fully reproduce the experiment. The published artifacts are verifiable; the labels and runs are the authors' work, and their cost figures reflect author-recorded usage and prices rather than independently verified current pricing.

The study's own audit shows how fragile a tuned cutoff can be. The authors report that no feasible training threshold meeting their target existed in 36 of 100 cross-validation splits for one configuration and 29 of 100 for another, with fallback values retained in the averages. A threshold that looks adequate on the data used to choose it is not a guarantee of the same precision and recall on fresh conversations. Independent methodology documentation from scikit-learn makes the same general point: a default cutoff is not generally ideal, tuning should use separate data to avoid overfitting, and probability calibration needs its own assessment because it can also make results worse.

For a single small domain, that means a hiring decision between evaluator models based on headline curves can be misleading. The strongest supported conclusion is that each business has to measure on its own examples.

What this means for your business

If you use an AI assistant for appointments or customer intake, the same failure modes apply. An evaluator watching your conversations can miss a request that never became a confirmed booking, or it can flag so many suspicious conversations that staff stop trusting the alerts. An unchecked threshold can contribute to both kinds of error, but it is not the only cause: label quality, the evaluator itself, the prompts it runs on, and the mix of traffic you receive all shape the results, and tuning a threshold does not fix every failure.

Detailed In Design could build a staff review dashboard alongside your booking or intake workflow as custom work: store permitted conversation traces under privacy and retention controls, let staff label failed bookings and missed handoffs, compare evaluator scores on held-out examples from your own traffic, set the review threshold against the real cost of a missed booking versus a false alarm, route only flagged cases to a person, and monitor for drift as your messages change. Staff would confirm a finding before any message goes out or any schedule changes; no automatic refund, customer email, or calendar change would follow from a score alone. The benefit would be clearer review priorities, more consistent handoffs, and evidence for choosing an evaluator, not a promised accuracy or savings figure.

The dependencies are real. The system needs authorized access to your conversation records, integration permissions with your booking or CRM software, representative labels from your staff, and a separate evaluation set that was never used to pick the threshold. The study itself is one private, single-domain dataset of 98 scored traces, so it cannot rank vendors universally or predict your numbers.

A small next step

Before buying or expanding any AI assistant, ask the vendor one question: what threshold decides a failure, and on whose data was it tested? If the answer is a default value tested on nobody's traffic, treat every alert score as a hypothesis for a person to check. If you want help measuring that on your own conversations, Detailed In Design can walk through what a bounded evaluation would cover.