Failure-Aware Contrastive Learning for Web Search Agents via Commitment-Aware Credit Assignment
DOI:
https://doi.org/10.66421/int.j.datasci.intell.syst77Keywords:
Web search agents, failure attribution, contrastive learning, credit assignment, self-play fine-tuning, robustnessAbstract
Self-play contrastive fine-tuning lets a web search agent learn from its own successful and failed trajectories, but its step-level signal is only as good as its guess of where a failed trajectory went wrong. We study this failure-attribution problem in a reproducible text-browsing environment built from real pages (490 Wikipedia articles and 652 gov.uk pages) in which exact shortest-path distances label the point of no return of every failure. Building on the TAPE framework (dual-stream experience pool, trajectory alignment module, trajectory contrastive loss and adaptive pool evolution), we propose Failure-Aware Contrastive Learning (FACL). FACL replaces TAPE's KL-divergence failure attribution with commitment-aware credit assignment: success statistics are pooled on a graph of browser states, so reversible reading actions are not blamed while irreversible submissions made from recoverable states are, and every attributed step is contrasted with an action that succeeded from the identical state under a bounded unlikelihood penalty. On 822 oracle-labelled failures, commitment-aware attribution locates the point of no return in 40.3% of cases versus 20.8% for KL-based attribution, although a last-step heuristic reaches 71.8% because premature submission dominates. Over 3 seeds on 2,000 held-out tasks from unseen articles, FACL raises task success from 46.9% (SFT) to 50.6%, significantly above TAPE (49.3%), while outcome-based PPO reaches 50.3%. FACL increases the verification rate from 0.0% to 21.8% with the largest gains on the harder L2 and L3 tasks; structural perturbations and an unseen website remain challenging for all methods.
References
S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao, “ReAct: Synergizing reasoning and acting in language models,” in Proc. Int. Conf. Learn. Represent. (ICLR), 2023.
R. Nakano et al., “WebGPT: Browser-assisted question-answering with human feedback,” arXiv:2112.09332, 2021.
S. Yao, H. Chen, J. Yang, and K. Narasimhan, “WebShop: Towards scalable real-world web interaction with grounded language agents,” in Proc. Adv. Neural Inf. Process. Syst. (NeurIPS), vol. 35, 2022, pp. 20744–20757.
X. Deng, Y. Gu, B. Zheng, S. Chen, S. Stevens, B. Wang, H. Sun, and Y. Su, “Mind2Web: Towards a generalist agent for the web,” in Proc. Adv. Neural Inf. Process. Syst. (NeurIPS), vol. 36, 2023, pp. 28091–28114.
S. Zhou et al., “WebArena: A realistic web environment for building autonomous agents,” in Proc. Int. Conf. Learn. Represent. (ICLR), 2024.
J. Chen, X. Han, X. Zhang, and Y. Yang, “Exact concept-level explanations for early detection of financial misstatements,” Eurasia Journal of Science and Technology, vol. 5, no. 2, pp. 34–48, 2023.
J. Wei et al., “BrowseComp: A simple yet challenging benchmark for browsing agents,” arXiv:2504.12516, 2025.
B. Jin et al., “Search-R1: Training LLMs to reason and leverage search engines with reinforcement learning,” in Proc. Conf. Lang. Model. (COLM), 2025.
D. Teng, “TRIAD: Deadline- and reliability-aware scheduling for post-earthquake structural assessment workflows,” World Journal of Engineering Research, vol. 3, no. 6, pp. 55–66, 2025.
S. Ross, G. Gordon, and D. Bagnell, “A reduction of imitation learning and structured prediction to no-regret online learning,” in Proc. Int. Conf. Artif. Intell. Statist. (AISTATS), 2011, pp. 627–635.
E. Zelikman, Y. Wu, J. Mu, and N. Goodman, “STaR: Bootstrapping reasoning with reasoning,” in Proc. Adv. Neural Inf. Process. Syst. (NeurIPS), vol. 35, 2022, pp. 15476–15488.
R. Wang, X. Han, Y. Zhang, T. Baldwin, and H. Li, “NAT: Enhancing agent tuning with negative samples,” in Proc. Conf. North Amer. Chapter Assoc. Comput. Linguistics (NAACL), 2025, pp. 7385–7398.
Y. Song, D. Yin, X. Yue, J. Huang, S. Li, and B. Y. Lin, “Trial and error: Exploration-based trajectory optimization of LLM agents,” in Proc. 62nd Annu. Meeting Assoc. Comput. Linguistics (ACL), 2024, doi: 10.18653/v1/2024.acl-long.409.
Z. Chen, Y. Deng, H. Yuan, K. Ji, and Q. Gu, “Self-play fine-tuning converts weak language models to strong language models,” in Proc. 41st Int. Conf. Mach. Learn. (ICML), 2024, pp. 6621–6642.
R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn, “Direct preference optimization: Your language model is secretly a reward model,” in Proc. Adv. Neural Inf. Process. Syst. (NeurIPS), vol. 36, 2023, pp. 53728–53741.
P. Putta, E. Mills, N. Garg, S. Motwani, C. Finn, D. Garg, and R. Rafailov, “Agent Q: Advanced reasoning and learning for autonomous AI agents,” arXiv:2408.07199, 2024.
M. Rhee, J. Zou, T. Mo, D. Teng, and J.-S. Yang, “Enhancing web search agents with self-play contrastive fine-tuning,” IEEE Access, vol. 14, pp. 115695–115709, 2026, doi: 10.1109/ACCESS.2026.3717594.
H. Lightman et al., “Let’s verify step by step,” in Proc. Int. Conf. Learn. Represent. (ICLR), 2024.
P. Wang et al., “Math-Shepherd: Verify and reinforce LLMs step-by-step without human annotations,” in Proc. 62nd Annu. Meeting Assoc. Comput. Linguistics (ACL), 2024, pp. 9426–9439, doi: 10.18653/v1/2024.acl-long.510.
W. Xiong et al., “Watch every step! LLM agent learning via iterative step-level process refinement,” in Proc. Conf. Empirical Methods Natural Lang. Process. (EMNLP), 2024, pp. 1556–1572, doi: 10.18653/v1/2024.emnlp-main.93.
A. Kazemnejad, M. Aghajohari, E. Portelance, A. Sordoni, S. Reddy, A. Courville, and N. Le Roux, “VinePPO: Refining credit assignment in RL training of LLMs,” in Proc. 42nd Int. Conf. Mach. Learn. (ICML), 2025, pp. 29557–29590.
X. Yue and R. Zhu, “An explainable hybrid machine learning framework for cash-flow-conditioned multi-horizon liquidity risk early warning: An SME-oriented benchmark study,” World Journal of Management Science, vol. 2, no. 1, pp. 63–72, 2024.
S. Welleck, I. Kulikov, S. Roller, E. Dinan, K. Cho, and J. Weston, “Neural text generation with unlikelihood training,” in Proc. Int. Conf. Learn. Represent. (ICLR), 2020.
J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” arXiv:1707.06347, 2017.
A. Ahmadian et al., “Back to basics: Revisiting REINFORCE-style optimization for learning from human feedback in LLMs,” in Proc. 62nd Annu. Meeting Assoc. Comput. Linguistics (ACL), 2024, pp. 12248–12267.
J. Chen, X. Zhang, and X. Han, “Explainable identification of tax-related financial reporting anomalies in U.S. public companies: Cross-fitted tax-consistency residuals and Owen-value Shapley propagation,” World Journal of Economics and Business Research, vol. 1, no. 1, pp. 33–45, 2023.
H. Sun, W. W. Cohen, and R. Salakhutdinov, “ConditionalQA: A complex reading comprehension dataset with conditional answers,” in Proc. 60th Annu. Meeting Assoc. Comput. Linguistics (ACL), 2022, pp. 3627–3637, doi: 10.18653/v1/2022.acl-long.253.
S. Robertson and H. Zaragoza, “The probabilistic relevance framework: BM25 and beyond,” Found. Trends Inf. Retr., vol. 3, no. 4, pp. 333–389, 2009, doi: 10.1561/1500000019.
A. van den Oord, Y. Li, and O. Vinyals, “Representation learning with contrastive predictive coding,” arXiv:1807.03748, 2018.
H. Sakoe and S. Chiba, “Dynamic programming algorithm optimization for spoken word recognition,” IEEE Trans. Acoust., Speech, Signal Process., vol. 26, no. 1, pp. 43–49, Feb. 1978, doi: 10.1109/TASSP.1978.1163055.
W. Shang, J.-S. Yang, G. Ding, and S. Yang, “ArchMigrate-Swift: A Compiler-in-the-Loop Retrieval-Augmented Multi-Agent Framework for Objective-C-to-Swift Migration,” WEP Transactions on Computer Science and Intelligent Systems Research, vol. 13, pp. 275–285, 2026.
T. Mo, C. Zhang, J. Zou, Z. Guo, and M. Rhee, “Self-Evolving AI Agents With Dual Memory for Automated Software Testing and Bug Localization,” IEEE Access, 2026, doi: 10.1109/ACCESS.2026.3713401.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Daniel Rahman,