I like this framing a lot: agent improvement is harness improvement, not model improvement

The interesting interventions are often at the tool boundary:

• what context you pass
• when tools are provided
• how you recover from failure
• what gets measured afterward

That’s the loop we’re building around deepagents (the orchestration logic) + LangSmith (how you measure)

Arky Yang (@arkyyang)
Beyond Prompts: Measuring and Optimizing LLM Tool-Agent Harnesses — new paper (arXiv, 4 Sep 2026; accepted EMNLP 2026). arxiv.org/abs/2609.05736

Treats harness optimization around a fixed model as budgeted selection: edits are guarded intercepts at the tool boundary, not rewrites of execution logic. PRISM clusters failures, routes each repair to a prompt, middleware, or joint surface, and picks candidates by gate pass-rate AND reliability (RelLift95), not score alone.

3 distinct takeaways for product builders:

• Report selected-harness reliability, not just mean lift. BH found the best individual harnesses but had negative reliable lift on both tau-bench domains — a strong explorer, an unreliable selector.
• Constrain middleware to tool-boundary edit patterns (silent correction, error blocking, prerequisite blocking). Unconstrained middleware (BH-MW) fell below its own prompt-only variant on Retail; the constraint, not the surface, is what makes middleware safely searchable.
• Route each failure to the right surface: prompts for global policy, middleware for locally-checkable execution invariants. Removing routing collapsed lift first-order on all three tasks; prompts alone miss runtime failures, unconstrained middleware adds selection risk.

Experiment setup (from the paper): fixed inner execution model gpt-5.4-mini, outer proposer claude-opus-4.7. Benchmarks: BFCL multi-round (600-case scorecard) and tau2-Retail / tau2-Telecom (74-case scorecard each of 114 tasks). Baselines BH, GEPA, MIPROv2 + PRISM; disjoint repair/gate/scorecard splits, 4 seeds, up to 16 runs. Baseline = native Deep Agents (v0.5.6). Metrics: mean held-out lift (pp), WorstLift, repeatability, RelLift95(B) by 5,000 bootstrap. PRISM-MW mean lifts 14.2 / 14.9 / 10.1 pp and positive RelLift95 on all three.