Towards Autonomous Web Navigation with LLM-based Agents

Abstract

Autonomous web navigation remains a complex challenge, particularly due to the dynamic, diverse and unstructured nature of web environments. Traditional web scraping techniques, while effective, require rigid configurations tied to specific website structures, limiting their generalizability. To address these challenges, this work explores the usage of autonomous agents powered by Large Language Models for autonomous web navigation, focusing on the retrieval of academic publications from webs of preprint repositories. The proposed solution, based on hyperlink exploration, is designed as a component of a potentially broader system for AI-driven paper search assistance. It leverages a multi-agent architecture and a structured tree-traversal like approach to explore and extract relevant documents. Each agent is assigned a specific role, including relevant URL extraction, document collection, planning, presentation and quality control. The system is implemented using AutoGen, which enables flexible agent interactions and modular design. Unlike traditional web information extraction techniques, this approach generalizes navigation patterns across different websites without relying on predefined HTML selectors, allowing its usage on different websites. Experimental results are promising, demonstrating the system’s effectiveness in retrieving relevant academic content. However, challenges such as increased response times and occasional hallucinations indicate areas for refinement. Future work aims to enhance interactivity by integrating advanced form-based search capabilities, optimize retrieval efficiency, and implement more robust evaluation frameworks. These improvements could contribute to fully automated AI-driven web exploration, facilitating the development of more generalizable autonomous web navigation tools.