NeuroLogic Decoding: (Un)supervised Neural Text Generation with Predicate Logic Constraints

• URL: http://arxiv.org/abs/2010.12884v2
• Date: Tue, 20 Apr 2021 18:59:28 GMT
• Title: NeuroLogic Decoding: (Un)supervised Neural Text Generation with Predicate Logic Constraints
• Authors: Ximing Lu, Peter West, Rowan Zellers, Ronan Le Bras, Chandra Bhagavatula, Yejin Choi
• Abstract summary: Conditional text generation often requires lexical constraints, i.e., which words should or shouldn't be included in the output text. We propose NeuroLogic Decoding, a simple yet effective algorithm that enables neural language models -- supervised or not -- to generate fluent text. Our results suggest the limit of large-scale neural networks for fine-grained controllable generation and the promise of inference-time algorithms.
• Score: 75.66980495245926
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Conditional text generation often requires lexical constraints, i.e., which words should or shouldn't be included in the output text. While the dominant recipe for conditional text generation has been large-scale pretrained language models that are finetuned on the task-specific training data, such models do not learn to follow the underlying constraints reliably, even when supervised with large amounts of task-specific examples. We propose NeuroLogic Decoding, a simple yet effective algorithm that enables neural language models -- supervised or not -- to generate fluent text while satisfying complex lexical constraints. Our approach is powerful yet efficient. It handles any set of lexical constraints that is expressible under predicate logic, while its asymptotic runtime is equivalent to conventional beam search. Empirical results on four benchmarks show that NeuroLogic Decoding outperforms previous approaches, including algorithms that handle a subset of our constraints. Moreover, we find that unsupervised models with NeuroLogic Decoding often outperform supervised models with conventional decoding, even when the latter is based on considerably larger networks. Our results suggest the limit of large-scale neural networks for fine-grained controllable generation and the promise of inference-time algorithms.

Related papers

• A Pseudo-Semantic Loss for Autoregressive Models with Logical Constraints [87.08677547257733]
  Neuro-symbolic AI bridges the gap between purely symbolic and neural approaches to learning. We show how to maximize the likelihood of a symbolic constraint w.r.t the neural network's output distribution. We also evaluate our approach on Sudoku and shortest-path prediction cast as autoregressive generation.
  arXiv  Detail & Related papers  (2023-12-06T20:58:07Z)
• Constrained Sampling from Language Models via Langevin Dynamics in Embedding Spaces [34.375537557235724]
  We propose a sampling procedure that combines the log-likelihood of the language model with arbitrary differentiable constraints into a single energy function. We evaluate our approach on different text generation tasks with soft and hard constraints as well as their combinations with competitive results for toxicity avoidance, sentiment control, and keyword-guided generation.
  arXiv  Detail & Related papers  (2022-05-25T08:09:03Z)
• COLD Decoding: Energy-based Constrained Text Generation with Langevin Dynamics [69.8062252611486]
  Cold decoding is a flexible framework that can be applied directly to off-the-shelf left-to-right language models. Our experiments on constrained generation tasks point to the effectiveness of our approach, both in terms of automatic and human evaluation.
  arXiv  Detail & Related papers  (2022-02-23T18:59:27Z)
• NeuroLogic A*esque Decoding: Constrained Text Generation with Lookahead Heuristics [73.96837492216204]
  We propose NeuroLogic A*esque, a decoding algorithm that incorporates estimates of future cost. We develop efficient lookaheads that are efficient for large-scale language models. Our approach achieves competitive baselines on five generation tasks, and new state-of-the-art performance on table-to-text generation, constrained machine translation, and keyword-constrained generation.
  arXiv  Detail & Related papers  (2021-12-16T09:22:54Z)
• Neural Rule-Execution Tracking Machine For Transformer-Based Text Generation [43.71069101841354]
  Sequence-to-Sequence (S2S) neural text generation models have exhibited compelling performance on various natural language generation tasks. However, the black-box nature of these models limits their application in tasks where specific rules need to be executed. We propose a novel module named Neural Rule-Execution Tracking Machine that can be equipped into various transformer-based generators to leverage multiple rules simultaneously.
  arXiv  Detail & Related papers  (2021-07-27T20:41:05Z)
• Neuro-Symbolic Constraint Programming for Structured Prediction [32.427665902031436]
  We propose Nester, a method for injecting neural networks into constrained structured predictors. Nester takes advantage of the features of its two components: the neural network learns complex representations from low-level data. An empirical evaluation on handwritten equation recognition shows that Nester achieves better performance than both the neural network and the constrained structured predictor.
  arXiv  Detail & Related papers  (2021-03-31T17:31:33Z)
• Posterior Control of Blackbox Generation [126.33511630879713]
  We consider augmenting neural generation models with discrete control states learned through a structured latent-variable approach. We find that this method improves over standard benchmarks, while also providing fine-grained control.
  arXiv  Detail & Related papers  (2020-05-10T03:22:45Z)
• Logical Natural Language Generation from Open-Domain Tables [107.04385677577862]
  We propose a new task where a model is tasked with generating natural language statements that can be emphlogically entailed by the facts. To facilitate the study of the proposed logical NLG problem, we use the existing TabFact dataset citechen 2019tabfact featured with a wide range of logical/symbolic inferences. The new task poses challenges to the existing monotonic generation frameworks due to the mismatch between sequence order and logical order.
  arXiv  Detail & Related papers  (2020-04-22T06:03:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.