Skip to content

LiveWeb-IE: A Benchmark For Online Web Information Extraction

🕒 Published (v1): 2026-03-14 05:55 UTC · Source: Arxiv · Venue: ICLR 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

LiveWeb-IE is a benchmark that evaluates web information extraction (WIE) systems against live websites rather than frozen HTML snapshots, exposing a >15% F1 degradation in existing methods caused by web structural drift. Alongside the benchmark, the authors propose Visual Grounding Scraper (VGS), a four-stage agentic pipeline that generates reusable XPath wrappers by visually narrowing the observation space rather than parsing raw HTML.

Problem

Existing WIE benchmarks (SWDE, Expanded SWDE, PLAtE, etc.) use static HTML snapshots captured at a single point in time, so evaluation results do not generalize to real-world live websites whose DOM structures evolve. Prior wrapper-based methods are brittle to layout changes, and pure LLM-as-extractor approaches are cost-prohibitive at scale. Hybrid LLM-wrapper methods inherit the brittleness of HTML parsing, losing >15% F1 when evaluated on structurally evolved versions of the same sites.

Method

LiveWeb-IE benchmark: 342 natural-language queries across 15 permission-granted websites (8 domains, 46 structurally distinct page groups) requiring extraction of text, images, and hyperlinks. Tasks are organized into four complexity types by attribute count \(|A|\) and value cardinality: - Type I: \(|A|=1\), single value - Type II: \(|A|>1\), single values - Type III: \(|A|=1\), list value - Type IV: \(|A|>1\), list values

Evaluation is mandated against live URLs at test time; ground-truth values are restricted to factually stable content so annotations remain valid despite layout evolution.

Visual Grounding Scraper (VGS): A four-stage agentic framework: 1. Attribute Identification — an LLM decomposes query \(Q\) into \(\hat{A} = \{\hat{a}_i\}\) via instruction \(I_a\). 2. Visual Grounding — a VLM scans sequential vertical screenshot strips \(R = \{r_j\}\) of the rendered page and maps each attribute to a region: \(R' = \{(\hat{a}_i, r'_i)\}\) where \(r'_i = \text{VLM}(I_g, R, \hat{a}_i)\). 3. Element Pinpointing — Set-of-Mark prompting overlays numbered bounding boxes on candidate DOM elements; a VLM selects the subset \(B^*_i \subseteq B_i\) corresponding to actual attribute values. 4. XPath Synthesis — the VLM generates a generalizable XPath \(x_i\) from the localized HTML snippet \(H_i\) (target element plus \(d\)-hop neighbors) and the visually marked region: \(x_i = \text{VLM}(I_x, H_i, \hat{r}_i, \hat{a}_i)\).

This pipeline produces reusable XPath wrappers rather than per-page LLM calls.

Key Contributions

  • LiveWeb-IE benchmark: first WIE benchmark mandating evaluation against live websites, with multi-modal queries (text, image, hyperlink) and four-level complexity taxonomy.
  • VGS framework: vision-grounded, multi-stage agentic wrapper generator that replaces HTML-only preprocessing with screenshot-based spatial reasoning.
  • Empirical finding: quantifies the live-vs-static gap (>15% F1 drop) and shows all HTML-only baselines degrade sharply on complex tasks (Type III/IV), motivating vision-first approaches.

Results

  • VGS (Gemini-2.5-Flash) achieves Overall F1 = 43.44 vs. CoT 20.53, Reflexion 22.39, AutoScraper 22.02 on LiveWeb-IE — roughly 2× improvement.
  • VGS (GPT-4o-mini) achieves Overall F1 = 35.85 vs. baselines 17–22 range.
  • On complex tasks (Type IV), all baselines score F1 < 10 (CoT: 7.40, Reflexion: 8.67, AutoScraper: 9.66 with Gemini-2.5-Flash); VGS (Gemini-2.5-Flash) reaches 38.13 F1.
  • LLM wrapper methods show >15% average F1 degradation when applied to structurally evolved vs. snapshot versions of the same websites (Appendix F).
  • VGS achieves state-of-the-art on both LiveWeb-IE and existing static benchmarks (specific numbers for GPT-4o and open models are truncated in the provided text).

Limitations

  • Dataset covers only 15 websites across 8 domains; generalization to the long tail of web structures is unverified.
  • Ground-truth stability relies on manually filtering for factually invariant content; dynamically updating pages (prices, rankings) are excluded by design, limiting coverage of common scraping targets.
  • VGS incurs multi-step VLM inference per attribute, making it slower and costlier than single-pass methods — the paper does not report latency or cost comparisons.
  • XPath wrappers may still break on sufficiently drastic DOM restructuring; the live benchmark captures this at evaluation time but VGS does not include a self-healing mechanism.
  • Human evaluation details and GPT-4o full results are referenced but not fully reproduced in the provided text.

Relevance to Harnesses / Meta-Harnesses

VGS is a concrete example of a sequential multi-stage agentic harness: it chains LLM and VLM calls through fixed stages (decompose → ground → pinpoint → synthesize) with structured intermediate state, directly analogous to the harness patterns studied in meta-harness research. The benchmark itself functions as a live evaluation harness — it mandates runtime access to external live systems at test time rather than a static corpus, a design challenge directly relevant to anyone building harnesses that must remain valid as the environment drifts. The study's finding that pipeline stage decomposition (visual narrowing) outperforms end-to-end HTML parsing supports the broader thesis that structured intermediate representations inside a harness improve reliability. The XPath-as-reusable-artifact design also illustrates how a harness can externalize extracted knowledge into a persistent, executable form rather than re-running inference per query.