Standardize: Aligning Language Models with Expert-Defined Standards for
Content Generation
- URL: http://arxiv.org/abs/2402.12593v1
- Date: Mon, 19 Feb 2024 23:18:18 GMT
- Title: Standardize: Aligning Language Models with Expert-Defined Standards for
Content Generation
- Authors: Joseph Marvin Imperial, Gail Forey, Harish Tayyar Madabushi
- Abstract summary: We introduce Standardize, a retrieval-style in-context learning-based framework to guide large language models to align with expert-defined standards.
Our findings show that models can gain 40% to 100% increase in precise accuracy for Llama2 and GPT-4, respectively.
- Score: 4.1205832766381985
- License: http://creativecommons.org/licenses/by-nc-sa/4.0/
- Abstract: Domain experts across engineering, healthcare, and education follow strict
standards for producing quality content such as technical manuals, medication
instructions, and children's reading materials. However, current works in
controllable text generation have yet to explore using these standards as
references for control. Towards this end, we introduce Standardize, a
retrieval-style in-context learning-based framework to guide large language
models to align with expert-defined standards. Focusing on English language
standards in the education domain as a use case, we consider the Common
European Framework of Reference for Languages (CEFR) and Common Core Standards
(CCS) for the task of open-ended content generation. Our findings show that
models can gain 40% to 100% increase in precise accuracy for Llama2 and GPT-4,
respectively, demonstrating that the use of knowledge artifacts extracted from
standards and integrating them in the generation process can effectively guide
models to produce better standard-aligned content.
Related papers
- An NLP Crosswalk Between the Common Core State Standards and NAEP Item Specifications [0.0]
I describe an NLP-based procedure that can be used to support subject matter experts in establishing a crosswalk between item specifications and content standards.
The procedure is used to evaluate the match of the Common Core State Standards for mathematics at grade 4 to the corresponding item specifications for the 2026 National Assessment of Educational Progress.
arXiv Detail & Related papers (2024-05-27T15:47:46Z) - Flesch or Fumble? Evaluating Readability Standard Alignment of
Instruction-Tuned Language Models [4.867923281108005]
We select a diverse set of open and closed-source instruction-tuned language models and investigate their performances in writing story completions and simplifying narratives.
Our findings provide empirical proof of how globally recognized models like ChatGPT may be considered less effective and may require more refined prompts for these generative tasks.
arXiv Detail & Related papers (2023-09-11T13:50:38Z) - Commonsense Knowledge Transfer for Pre-trained Language Models [83.01121484432801]
We introduce commonsense knowledge transfer, a framework to transfer the commonsense knowledge stored in a neural commonsense knowledge model to a general-purpose pre-trained language model.
It first exploits general texts to form queries for extracting commonsense knowledge from the neural commonsense knowledge model.
It then refines the language model with two self-supervised objectives: commonsense mask infilling and commonsense relation prediction.
arXiv Detail & Related papers (2023-06-04T15:44:51Z) - CPL-NoViD: Context-Aware Prompt-based Learning for Norm Violation Detection in Online Communities [28.576099654579437]
We introduce Context-aware Prompt-based Learning for Norm Violation Detection (CPL-NoViD)
CPL-NoViD outperforms the baseline by incorporating context through natural language prompts.
It establishes a new state-of-the-art in norm violation detection, surpassing existing benchmarks.
arXiv Detail & Related papers (2023-05-16T23:27:59Z) - Pre-Training to Learn in Context [138.0745138788142]
The ability of in-context learning is not fully exploited because language models are not explicitly trained to learn in context.
We propose PICL (Pre-training for In-Context Learning), a framework to enhance the language models' in-context learning ability.
Our experiments show that PICL is more effective and task-generalizable than a range of baselines, outperforming larger language models with nearly 4x parameters.
arXiv Detail & Related papers (2023-05-16T03:38:06Z) - Controllable Text Generation with Language Constraints [39.741059642044874]
We consider the task of text generation in language models with constraints specified in natural language.
Our benchmark contains knowledge-intensive constraints sourced from databases like Wordnet and Wikidata.
We propose a solution to leverage a language model's own internal knowledge to guide generation.
arXiv Detail & Related papers (2022-12-20T17:39:21Z) - Stylized Knowledge-Grounded Dialogue Generation via Disentangled
Template Rewriting [55.10977824136768]
We study a new problem: Stylized Knowledge-Grounded Dialogue Generation.
It presents two challenges: How to train a SKDG model where no context, knowledge, stylized response> triples are available.
We propose a novel disentangled template rewriting (DTR) method which generates responses via combing disentangled style templates and content templates.
arXiv Detail & Related papers (2022-04-12T08:17:21Z) - CUGE: A Chinese Language Understanding and Generation Evaluation
Benchmark [144.05723617401674]
General-purpose language intelligence evaluation has been a longstanding goal for natural language processing.
We argue that for general-purpose language intelligence evaluation, the benchmark itself needs to be comprehensive and systematic.
We propose CUGE, a Chinese Language Understanding and Generation Evaluation benchmark with the following features.
arXiv Detail & Related papers (2021-12-27T11:08:58Z) - Unveiling Relations in the Industry 4.0 Standards Landscape based on
Knowledge Graph Embeddings [10.098126048053384]
Industry4.0 (I4.0) standards and standardization frameworks have been proposed with the goal of emphempowering interoperability in smart factories.
We study the relatedness among standards and frameworks based on community analysis to discover knowledge that helps to cope with interoperability conflicts between standards.
arXiv Detail & Related papers (2020-06-03T17:37:08Z) - SPECTER: Document-level Representation Learning using Citation-informed
Transformers [51.048515757909215]
SPECTER generates document-level embedding of scientific documents based on pretraining a Transformer language model.
We introduce SciDocs, a new evaluation benchmark consisting of seven document-level tasks ranging from citation prediction to document classification and recommendation.
arXiv Detail & Related papers (2020-04-15T16:05:51Z) - Towards Making the Most of Context in Neural Machine Translation [112.9845226123306]
We argue that previous research did not make a clear use of the global context.
We propose a new document-level NMT framework that deliberately models the local context of each sentence.
arXiv Detail & Related papers (2020-02-19T03:30:00Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.