NEXT-EVAL: Next Evaluation of Traditional and LLM Web Data Record Extraction

• URL: http://arxiv.org/abs/2505.17125v1
• Date: Wed, 21 May 2025 21:03:37 GMT
• Title: NEXT-EVAL: Next Evaluation of Traditional and LLM Web Data Record Extraction
• Authors: Soyeon Kim, Namhee Kim, Yeonwoo Jeong,
• Abstract summary: We introduce a concrete evaluation framework for web data record extraction.<n>Our framework generates evaluation snapshots, annotates supervision labels, and employs structure-aware metrics for consistent scoring.<n>It also incorporates preprocessing to optimize input for Large Language Model (LLM)-based approaches.
• Score: 6.09502686736443
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Effective evaluation of web data record extraction methods is crucial, yet hampered by static, domain-specific benchmarks and opaque scoring practices. This makes fair comparison between traditional algorithmic techniques, which rely on structural heuristics, and Large Language Model (LLM)-based approaches, offering zero-shot extraction across diverse layouts, particularly challenging. To overcome these limitations, we introduce a concrete evaluation framework. Our framework systematically generates evaluation datasets from arbitrary MHTML snapshots, annotates XPath-based supervision labels, and employs structure-aware metrics for consistent scoring, specifically preventing text hallucination and allowing only for the assessment of positional hallucination. It also incorporates preprocessing strategies to optimize input for LLMs while preserving DOM semantics: HTML slimming, Hierarchical JSON, and Flat JSON. Additionally, we created a publicly available synthetic dataset by transforming DOM structures and modifying content. We benchmark deterministic heuristic algorithms and off-the-shelf LLMs across these multiple input formats. Our benchmarking shows that Flat JSON input enables LLMs to achieve superior extraction accuracy (F1 score of 0.9567) and minimal hallucination compared to other input formats like Slimmed HTML and Hierarchical JSON. We establish a standardized foundation for rigorous benchmarking, paving the way for the next principled advancements in web data record extraction.

Related papers

• Beyond a Single Extractor: Re-thinking HTML-to-Text Extraction for LLM Pretraining [78.36592534300839]
  We show that for structured content such as tables and code blocks, extractor choice can significantly impact downstream task performance.<n>This suggests a simple intervention: by taking a Union over different extractors, we can increase the token yield of DCLM-Baseline by up to 71%.
  arXiv  Detail & Related papers  (2026-02-23T06:41:57Z)
• Dripper: Token-Efficient Main HTML Extraction with a Lightweight LM [35.10225876152952]
  We present Dripper, an efficient HTML main content extraction framework powered by lightweight language models.<n>We introduce a controlled decoding mechanism that strictly constrains the output space through logits processors.<n>Dripper achieves state-of-the-art performance across all evaluation benchmarks and outperforms all baseline methods.
  arXiv  Detail & Related papers  (2025-11-28T12:04:46Z)
• DeepJSONEval: Benchmarking Complex Nested JSON Data Mining for Large Language Models [6.653834890554154]
  Multi-layer nested structures organize data into key-value pairs, arrays, and nested objects.<n>For instance, in news aggregation, a object can nest an article's metadata (title, author, date), content (text, multimedia), and multimedia information (multimedia, caption) hierarchically.<n>We introduce DeepJSONEval, a novel benchmark featuring 2100 multi-domain instances with deep nested structures, categorized by difficulty.
  arXiv  Detail & Related papers  (2025-09-30T08:18:20Z)
• Beyond Isolated Dots: Benchmarking Structured Table Construction as Deep Knowledge Extraction [80.88654868264645]
  Arranged and Organized Extraction Benchmark designed to evaluate ability of large language models to comprehend fragmented documents.<n>AOE includes 11 carefully crafted tasks across three diverse domains, requiring models to generate context-specific schema tailored to varied input queries.<n>Results show that even the most advanced models struggled significantly.
  arXiv  Detail & Related papers  (2025-07-22T06:37:51Z)
• Semantic Outlier Removal with Embedding Models and LLMs [0.45080838507508303]
  We introduce SORE (Semantic Outlier Removal), a cost-effective, transparent method to identify and excise unwanted text segments.<n>SORE achieves near-LLM extraction precision at a fraction of the cost.<n>Our system is currently deployed in production, processing millions of documents daily across multiple languages.
  arXiv  Detail & Related papers  (2025-06-19T23:06:12Z)
• System Log Parsing with Large Language Models: A Review [2.2779174914142346]
  Large language models (LLMs) have introduced the new research field of LLM-based log parsing.<n>Despite promising results, there is no structured overview of the approaches in this relatively new research field.<n>This work systematically reviews 29 LLM-based log parsing methods.
  arXiv  Detail & Related papers  (2025-04-07T09:41:04Z)
• Benchmark Inflation: Revealing LLM Performance Gaps Using Retro-Holdouts [0.6282171844772422]
  Training data for many Large Language Models (LLMs) is contaminated with test data. Public benchmark scores do not always accurately assess model properties.
  arXiv  Detail & Related papers  (2024-10-11T20:46:56Z)
• FineSurE: Fine-grained Summarization Evaluation using LLMs [22.62504593575933]
  FineSurE is a fine-grained evaluator specifically tailored for the summarization task using large language models (LLMs) It also employs completeness and conciseness criteria, in addition to faithfulness, enabling multi-dimensional assessment.
  arXiv  Detail & Related papers  (2024-07-01T02:20:28Z)
• Spread Preference Annotation: Direct Preference Judgment for Efficient LLM Alignment [72.99676237703099]
  We propose a new framework that boosts the alignment of large language models with human preferences.<n>Our key idea is leveraging the human prior knowledge within the small (seed) data.<n>We introduce a noise-aware preference learning algorithm to mitigate the risk of low quality within generated preference data.
  arXiv  Detail & Related papers  (2024-06-06T18:01:02Z)
• Aligning Language Models with Demonstrated Feedback [58.834937450242975]
  Demonstration ITerated Task Optimization (DITTO) directly aligns language model outputs to a user's demonstrated behaviors.<n>We evaluate DITTO's ability to learn fine-grained style and task alignment across domains such as news articles, emails, and blog posts.
  arXiv  Detail & Related papers  (2024-06-02T23:13:56Z)
• Revisiting Sparse Retrieval for Few-shot Entity Linking [33.15662306409253]
  We propose an ELECTRA-based keyword extractor to denoise the mention context and construct a better query expression. For training the extractor, we propose a distant supervision method to automatically generate training data based on overlapping tokens between mention contexts and entity descriptions. Experimental results on the ZESHEL dataset demonstrate that the proposed method outperforms state-of-the-art models by a significant margin across all test domains.
  arXiv  Detail & Related papers  (2023-10-19T03:51:10Z)
• Element-aware Summarization with Large Language Models: Expert-aligned Evaluation and Chain-of-Thought Method [35.181659789684545]
  Automatic summarization generates concise summaries that contain key ideas of source documents. References from CNN/DailyMail and BBC XSum are noisy, mainly in terms of factual hallucination and information redundancy. We propose a Summary Chain-of-Thought (SumCoT) technique to elicit LLMs to generate summaries step by step. Experimental results show our method outperforms state-of-the-art fine-tuned PLMs and zero-shot LLMs by +4.33/+4.77 in ROUGE-L.
  arXiv  Detail & Related papers  (2023-05-22T18:54:35Z)
• Understanding HTML with Large Language Models [73.92747433749271]
  Large language models (LLMs) have shown exceptional performance on a variety of natural language tasks. We contribute HTML understanding models (fine-tuned LLMs) and an in-depth analysis of their capabilities under three tasks. We show that LLMs pretrained on standard natural language corpora transfer remarkably well to HTML understanding tasks.
  arXiv  Detail & Related papers  (2022-10-08T07:27:17Z)
• Value Retrieval with Arbitrary Queries for Form-like Documents [50.5532781148902]
  We propose value retrieval with arbitrary queries for form-like documents. Our method predicts target value for an arbitrary query based on the understanding of layout and semantics of a form. We propose a simple document language modeling (simpleDLM) strategy to improve document understanding on large-scale model pre-training.
  arXiv  Detail & Related papers  (2021-12-15T01:12:02Z)
• Learning to Synthesize Data for Semantic Parsing [57.190817162674875]
  We propose a generative model which models the composition of programs and maps a program to an utterance. Due to the simplicity of PCFG and pre-trained BART, our generative model can be efficiently learned from existing data at hand. We evaluate our method in both in-domain and out-of-domain settings of text-to-Query parsing on the standard benchmarks of GeoQuery and Spider.
  arXiv  Detail & Related papers  (2021-04-12T21:24:02Z)
• Evaluating Models' Local Decision Boundaries via Contrast Sets [119.38387782979474]
  We propose a new annotation paradigm for NLP that helps to close systematic gaps in the test data. We demonstrate the efficacy of contrast sets by creating them for 10 diverse NLP datasets. Although our contrast sets are not explicitly adversarial, model performance is significantly lower on them than on the original test sets.
  arXiv  Detail & Related papers  (2020-04-06T14:47:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.