Skip to content

LiveWeb-IE: A Benchmark For Online Web Information Extraction

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Live web information extraction benchmark tests agent adaptation to dynamic real-world pages

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

LiveWeb-IE is a benchmark that evaluates web information extraction (WIE) systems against live, evolving websites rather than frozen HTML snapshots, exposing a >15% F1 degradation that static benchmarks mask. The paper also introduces Visual Grounding Scraper (VGS), a four-stage agentic framework that mimics human visual search by progressively narrowing page regions via a VLM before synthesizing reusable XPath wrappers.

Problem

All prior WIE benchmarks (SWDE, WEIR, PLAtE, etc.) evaluate on static HTML snapshots captured at a single point in time. Real web pages continuously change layout and structure, so wrapper-based or LLM-based extractors tuned on snapshots suffer systematic performance collapse on live pages—the paper measures an average F1 drop exceeding 15% when applying state-of-the-art LLM wrapper-generation methods to the structurally evolved versions of the same websites. No benchmark previously required systems to fetch the live URL at evaluation time or covered non-textual extraction targets (images, hyperlinks) with multi-dimensional complexity levels.

Method

LiveWeb-IE benchmark. Constructed from 15 permission-granted websites across 8 domains (academic, e-commerce, sports, etc.), clustered into 46 structurally distinct page groups, yielding 342 natural-language queries over 97 unique attributes. Queries span four complexity levels defined by two axes: number of attributes \(|A|\) and value cardinality (single item vs. list): Type I (\(|A|=1\), single), Type II (\(|A|>1\), single), Type III (\(|A|=1\), list), Type IV (\(|A|>1\), list). Ground-truth targets factually stable values (historical sports results, archived paper metadata) so annotation does not require continuous refresh even as layouts evolve. Data categories include text, images, and hyperlinks.

Visual Grounding Scraper (VGS). A four-stage agentic pipeline: 1. Attribute Identification — an LLM decomposes the free-form query \(Q\) into structured target attributes \(\hat{A} = \text{LLM}(I_a, Q)\). 2. Visual Grounding — the page is rendered and divided into vertical screenshot strips \(R = \{r_j\}\); a VLM selects the relevant strip \(r_i'\) per attribute: \(r_i' = \text{VLM}(I_g, R, \hat{a}_i)\). 3. Element Pinpointing — candidate bounding boxes are overlaid onto \(r_i'\) via Set-of-Mark prompting (JavaScript injection with numeric labels); the VLM selects the true-positive subset \(B_i^* = \text{VLM}(I_p, r_i^*, \hat{a}_i)\). 4. XPath Synthesis — for each selected bounding box the corresponding DOM element is located, a local HTML snippet \(h_b\) (element + neighbors within distance \(d\)) is extracted, and the VLM generates a generalizable XPath: \(x_i = \text{VLM}(I_x, H_i, \hat{r}_i, \hat{a}_i)\).

The final wrapper is the set \(X = \{x_i\}\) which can be reused across pages in the same group without re-invoking the LLM.

Key Contributions

  • LiveWeb-IE: first WIE benchmark mandating live-URL access at evaluation time, covering text/image/hyperlink targets and four task-complexity levels across 8 domains.
  • VGS: multi-stage agentic scraping framework that replaces full-HTML parsing with vision-guided spatial narrowing + Set-of-Mark bounding-box pinpointing + local-HTML XPath synthesis, achieving state-of-the-art on both LiveWeb-IE and existing offline WIE benchmarks.
  • Empirical analysis: quantifies the offline-to-online F1 gap (>15%) for existing methods, motivating live evaluation.
  • Human evaluation baseline and ablation study guiding future directions.

Results

  • VGS (Gemini-2.5-Flash) overall F1: 43.44 vs. CoT 20.53 / Reflexion 22.39 / AutoScraper 22.02 — roughly a 2× improvement over all baselines.
  • VGS (GPT-4o-mini) overall F1: 35.85 vs. CoT 20.63 / Reflexion 17.47 / AutoScraper 21.85.
  • On Type I (simplest), VGS (GPT-4o-mini) reaches F1 50.36; on Type IV (hardest), F1 drops to 29.68, showing complexity scaling is a hard unsolved problem.
  • Existing methods (CoT, Reflexion, AutoScraper) achieve F1 ≤22 overall on LiveWeb-IE despite being competitive on prior static benchmarks, confirming the benchmark's discriminative power.
  • LLM wrapper-generation methods show >15% average F1 degradation when evaluated on structurally evolved live pages vs. their static snapshot versions (Appendix F).

Limitations

  • Dataset covers only 15 websites; generalization to long-tail or adversarially structured sites is untested.
  • Ground-truth stability is achieved by restricting to historically fixed values, which may underrepresent genuinely dynamic extraction scenarios (e.g., live prices, real-time scores).
  • VGS requires both a capable VLM and an LLM in sequence, plus JavaScript injection capability, making it heavier than wrapper-only baselines; latency and cost are not fully analyzed in the visible text.
  • XPath wrappers are group-specific; when a site undergoes a complete redesign, wrappers must be regenerated.
  • The paper text is truncated, so results for GPT-4o and open-source models are not available for this summary.

Relevance to Agentic AI / LLM Agents

VGS is directly an agentic pipeline: it chains LLM reasoning (attribute decomposition), VLM perception (visual grounding, element pinpointing), and tool use (JavaScript DOM manipulation, XPath execution) in a multi-step loop, closely mirroring architectures seen in web-agent research (WebArena, Mind2Web). The benchmark fills a critical gap for agent evaluation by demanding that systems handle a live, non-deterministic environment rather than a frozen replay—a prerequisite for deploying agents in real scraping or data-collection workflows. The visual-grounding-first strategy (render → narrow → extract) is a generalizable design principle for any agent that must localize information in a cluttered observation space before acting, with implications beyond scraping (e.g., GUI agents, document agents). LiveWeb-IE's complexity taxonomy (Type I–IV) also provides a reusable evaluation scaffold for measuring agent capability scaling with task structure.