FDE-Gym benchmark shows why correct code is not enough for a working business workflow

A benchmark built around unclear requests

Yang et al. (2026) posted the paper on arXiv on October 15, 2026. Their starting point is a mismatch between what agent benchmarks measure and what customer work demands. Benchmarks usually hand an agent a tidy specification, such as an issue report or a task ticket. The authors describe forward-deployed engineering as the opposite situation: the customer's request arrives vague, the software in play is already running, and the rules that decide success are often never said aloud.

To measure that gap they assembled 95 delivery tasks by having industry experts adapt 20 customer scenarios drawn from 18 sectors. Nothing in the evaluation touches a real client. Simulated customers make the requests, scripted stakeholders answer questions, and the systems agents modify imitate production environments. Grading rewards how far a submission moves the customer's own success metric, measured between doing nothing and matching a reference solution, minus charges for the messages, trial runs and waiting time spent getting there.

Results, and what they do not show

The authors tested 13 model configurations from seven providers. In the setting that mirrors real delivery, where the hidden rules must be found rather than read off a page, the strongest configuration came to a mean score of 0.56 and finished 14 of the 95 tasks cleanly, with a clean finish defined as a score of at least 0.8 and no incident. Their reading of the pattern is blunt: pulling unstated constraints into the open is where agents struggle most, and scores fall off when nobody points them toward the missing rules.

A second result deserves attention from anyone buying automation. Finding the facts did not close the gap. On the 23 tasks where the best configuration collected every key fact from stakeholders, it still achieved business uplift of at least 0.8 on only 10. The authors treat this as evidence that a delivery can be well informed and still fail to become a working system.

How far the evidence goes

These are benchmark findings with real limits, and the inspected sources bound how far they reach. The reviewed version is an arXiv preprint; the sources examined for this article do not establish peer review, independent replication, or validation in production. The study evaluates a single minimal agent framework, uses scripted stakeholders and expert-defined interaction costs, and includes no human baseline. The scenario sample is limited and weighted toward regulated workflows, and the authors operate the associated practice platform. Only three complete example cases are openly available; the full scenarios are provided on request. The 0.56 average is a normalized benchmark score, not a percentage of business value, and the 14-of-95 figure is not evidence that 81 real customer projects failed.

What this means for your business

The practical lesson is not that AI agents fail at customer work. It is that the difference between a demonstration and a dependable workflow usually lives in the rules nobody wrote down: which jobs need a second opinion, which exceptions a person must approve, and which systems the software must never touch. If you automate the request exactly as worded, you can ship a tool that answers quickly while missing the operational constraint that decides whether the job succeeds.

A concrete example Detailed In Design could build as custom work is a quote-intake workflow. It would extract the details from an inbound request, check the request against your documented pricing rules and required evidence, flag what is missing, route exceptions to a designated approver, and record that approval before a quote leaves the business. The expected benefit is clearer handoffs and more consistent review, evaluated against your own acceptance criteria rather than a published score. No numeric savings are promised.

That proposal is Detailed In Design custom software. It is not part of FDE-Gym, not a packaged integration, and not a partnership with the benchmark's authors. It depends on your documented business rules, named reviewers, authorized connections to your existing systems, representative test cases, data access controls, and a human path for incomplete or consequential cases. Human review improves consistency but does not eliminate error, and the software should not change scope, excluded systems, or contractual rules without your authorization. If you are weighing automation for an intake or approval process, start by writing down the constraints you currently carry in people's heads, then test the workflow against those cases before it touches a live quote. Detailed In Design builds custom AI, software, and workflow systems with human review gates, described at detailedindesign.com.

Reference

Yang, D., Yan, H., Wang, R., Zhang, B., Zuo, H., Zhao, Y., Chen, Y., Liao, X., Gao, C., Jiang, T., Chen, Y., Wang, Q., Yu, Z., Li, X., Zhang, Y., Sun, J., Xu, R., Mi, D., Wu, T., . . . Wang, X. (2026). FDE-Gym: The last mile of AI delivery [Preprint]. arXiv. https://arxiv.org/abs/2610.22350