Distilling Examples into Task Instructions: Enhanced In-Context Learning for Real-World B2B Conversations

• URL: http://arxiv.org/abs/2606.15641v1
• Date: Sun, 14 Jun 2026 07:07:52 GMT
• Title: Distilling Examples into Task Instructions: Enhanced In-Context Learning for Real-World B2B Conversations
• Abstract summary: We introduce the textttCall Playbook dataset, featuring five classification tasks derived from real-world B2B conversations targeting core sales concepts.<n>Our approach achieves a 99% reduction in token usage and improves macro-averaged AUC by up to 7% over traditional ICL.<n> Importantly, our framework enables direct refinement of classification logic, addressing critical needs for transparency, efficiency, and user interaction in real-world NLP applications.
• Score: 0.41998444721319217
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In-context learning (ICL) is the standard method for low-resource classification, yet its efficacy in specialized domains remains largely unexplored. We address the challenge of classifying semantically complex, multi-party B2B conversations, where traditional ICL encounters significant limitations, especially as context length increases due to the concatenation of multiple few-shot examples. We introduce the \texttt{Call Playbook} dataset, featuring five classification tasks derived from real-world B2B conversations targeting core sales concepts. To bridge the gap between performance and practical utility, we propose novel knowledge extraction methods that distill verbose examples into compact, interpretable representations of structured classification criteria and precise task descriptions. Our approach achieves a 99\% reduction in token usage and improves macro-averaged AUC by up to 7\% over traditional ICL. Notably, it remains robust as context grows, unlike advanced token compression baselines which degrade by over 9 F1 points. Importantly, our framework enables direct refinement of classification logic, addressing critical needs for transparency, efficiency, and user interaction in real-world NLP applications.

Related papers

• From Recognition to Reasoning: Benchmarking and Enhancing MLLMs on Real-World Receipt Document Understanding [15.376371030121094]
  ReceiptBench is a large-scale, human-annotated benchmark consisting of 10k diverse receipts.<n>We introduce ReceiptBench, a large-scale, human-annotated benchmark consisting of 10k diverse receipts.
  arXiv  Detail & Related papers  (2026-05-21T12:37:03Z)
• Beyond Statistical Co-occurrence: Unlocking Intrinsic Semantics for Tabular Data Clustering [52.97486694817375]
  This paper proposes Tabular-Augmented Contrastive Clustering (TagCC) to bridge the gap between dataset-specific statistics and intrinsic semantic knowledge.<n>TagCC distills underlying data semantics into textual anchors via semantic-aware transformation.<n>It is jointly optimized with a clustering objective, ensuring that the learned representations are both semantically coherent and clustering-friendly.
  arXiv  Detail & Related papers  (2026-04-13T00:25:22Z)
• Private PoEtry: Private In-Context Learning via Product of Experts [58.496468062236225]
  In-context learning (ICL) enables Large Language Models to adapt to new tasks with only a small set of examples at inference time.<n>Existing differential privacy approaches to ICL are either computationally expensive or rely on oversampling, synthetic data generation, or unnecessary thresholding.<n>We reformulate private ICL through the lens of a Product-of-Experts model. This gives a theoretically grounded framework, and the algorithm can be trivially parallelized.<n>We find that our method improves accuracy by more than 30 percentage points on average compared to prior DP-ICL methods, while maintaining strong privacy guarantees.
  arXiv  Detail & Related papers  (2026-02-04T19:56:24Z)
• Beyond Manuals and Tasks: Instance-Level Context Learning for LLM Agents [22.620674535292068]
  Large language model (LLM) agents typically receive two kinds of context: environment-level manuals that define interaction interfaces and global rules, and task-level guidance or demonstrations tied to specific goals.<n>We argue that the absence of instance-level context is a common source of failure for LLM agents in complex tasks.<n>We formalize this problem as Instance-Level Context Learning (ILCL) and introduce our task-agnostic method to solve it.
  arXiv  Detail & Related papers  (2025-09-29T05:38:51Z)
• QUITO-X: A New Perspective on Context Compression from the Information Bottleneck Theory [75.81394991657545]
  We introduce information bottleneck theory (IB) to model the problem.<n>We propose a cross-attention-based approach to approximate mutual information in IB.<n>Our method achieves a 25% increase in compression rate compared to the state-of-the-art.
  arXiv  Detail & Related papers  (2024-08-20T02:44:45Z)
• Text-Video Retrieval with Global-Local Semantic Consistent Learning [122.15339128463715]
  We propose a simple yet effective method, Global-Local Semantic Consistent Learning (GLSCL) GLSCL capitalizes on latent shared semantics across modalities for text-video retrieval. Our method achieves comparable performance with SOTA as well as being nearly 220 times faster in terms of computational cost.
  arXiv  Detail & Related papers  (2024-05-21T11:59:36Z)
• Pay Attention to Your Neighbours: Training-Free Open-Vocabulary Semantic Segmentation [19.20874993309959]
  vision-language foundation models, such as CLIP, have showcased remarkable effectiveness in numerous zero-shot image-level tasks. We propose a baseline for training-free OVSS, termed Neighbour-Aware CLIP (NACLIP) Our method enforces localization of patches in the self-attention of CLIP's vision transformer which, despite being crucial for dense prediction tasks, has been overlooked in the OVSS literature.
  arXiv  Detail & Related papers  (2024-04-12T01:08:04Z)
• RAR: Retrieving And Ranking Augmented MLLMs for Visual Recognition [78.97487780589574]
  Multimodal Large Language Models (MLLMs) excel at classifying fine-grained categories. This paper introduces a Retrieving And Ranking augmented method for MLLMs. Our proposed approach not only addresses the inherent limitations in fine-grained recognition but also preserves the model's comprehensive knowledge base.
  arXiv  Detail & Related papers  (2024-03-20T17:59:55Z)
• Leveraging Code to Improve In-context Learning for Semantic Parsing [48.66031267718704]
  In-context learning (ICL) is an appealing approach for semantic parsing due to its few-shot nature and improved generalization. We improve the effectiveness of ICL for semantic parsing by (1) using general-purpose programming languages such as Python instead of DSLs, and (2) augmenting prompts with a structured domain description.
  arXiv  Detail & Related papers  (2023-11-16T02:50:06Z)
• Efficient IoT Inference via Context-Awareness [8.882680489254923]
  We propose a new paradigm, CACTUS, for scalable and efficient context-aware classification. We show that CACTUS achieves significant benefits in accuracy, latency, and compute budget across a range of datasets and IoT platforms.
  arXiv  Detail & Related papers  (2023-10-29T18:57:15Z)
• Label Anchored Contrastive Learning for Language Understanding [17.28721753405111]
  We propose a novel label anchored contrastive learning approach (denoted as LaCon) for language understanding. Our approach does not require any specialized network architecture or any extra data augmentation. LaCon obtains up to 4.1% improvement on the popular datasets of GLUE and CLUE benchmarks.
  arXiv  Detail & Related papers  (2022-04-26T15:33:01Z)
• Multitask Learning for Class-Imbalanced Discourse Classification [74.41900374452472]
  We show that a multitask approach can improve 7% Micro F1-score upon current state-of-the-art benchmarks. We also offer a comparative review of additional techniques proposed to address resource-poor problems in NLP.
  arXiv  Detail & Related papers  (2021-01-02T07:13:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.