Random Forest-of-Thoughts: Uncertainty-aware Reasoning for Computational Social Science

• URL: http://arxiv.org/abs/2502.18729v1
• Date: Wed, 26 Feb 2025 00:52:44 GMT
• Title: Random Forest-of-Thoughts: Uncertainty-aware Reasoning for Computational Social Science
• Authors: Xiaohua Wu, Xiaohui Tao, Wenjie Wu, Yuefeng Li, Lin Li,
• Abstract summary: We propose a novel large language model prompting method called Random Forest of Thoughts (RFoT)<n>RFoT allows LLMs to perform deliberate decision-making by generating diverse thought space and randomly selecting the sub-thoughts to build the forest of thoughts.<n>Our experiments show that RFoT significantly enhances language models' abilities on two novel social survey analysis problems requiring non-trivial reasoning.
• Score: 9.870701840926923
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Social surveys in computational social science are well-designed by elaborate domain theories that can effectively reflect the interviewee's deep thoughts without concealing their true feelings. The candidate questionnaire options highly depend on the interviewee's previous answer, which results in the complexity of social survey analysis, the time, and the expertise required. The ability of large language models (LLMs) to perform complex reasoning is well-enhanced by prompting learning such as Chain-of-thought (CoT) but still confined to left-to-right decision-making processes or limited paths during inference. This means they can fall short in problems that require exploration and uncertainty searching. In response, a novel large language model prompting method, called Random Forest of Thoughts (RFoT), is proposed for generating uncertainty reasoning to fit the area of computational social science. The RFoT allows LLMs to perform deliberate decision-making by generating diverse thought space and randomly selecting the sub-thoughts to build the forest of thoughts. It can extend the exploration and prediction of overall performance, benefiting from the extensive research space of response. The method is applied to optimize computational social science analysis on two datasets covering a spectrum of social survey analysis problems. Our experiments show that RFoT significantly enhances language models' abilities on two novel social survey analysis problems requiring non-trivial reasoning.

Related papers

• Are Information Retrieval Approaches Good at Harmonising Longitudinal Survey Questions in Social Science? [2.769064123193329]
  We present a new information retrieval task to identify concept equivalence across question and response options. This paper investigates multiple unsupervised approaches on a survey dataset spanning 1946-2020. We show that IR-specialised neural models achieve the highest overall performance with other approaches performing comparably.
  arXiv  Detail & Related papers  (2025-04-29T12:00:33Z)
• Hypothesis-Driven Theory-of-Mind Reasoning for Large Language Models [76.6028674686018]
  We introduce thought-tracing, an inference-time reasoning algorithm to trace the mental states of agents.<n>Our algorithm is modeled after the Bayesian theory-of-mind framework.<n>We evaluate thought-tracing on diverse theory-of-mind benchmarks, demonstrating significant performance improvements.
  arXiv  Detail & Related papers  (2025-02-17T15:08:50Z)
• Thoughts Are All Over the Place: On the Underthinking of o1-Like LLMs [86.79757571440082]
  Large language models (LLMs) such as OpenAI's o1 have demonstrated remarkable abilities in complex reasoning tasks.<n>We identify a phenomenon we term underthinking, where o1-like LLMs frequently switch between different reasoning thoughts.<n>We propose a decoding strategy with thought switching penalty TIP that discourages premature transitions between thoughts.
  arXiv  Detail & Related papers  (2025-01-30T18:58:18Z)
• The Computational Complexity of Circuit Discovery for Inner Interpretability [0.30723404270319693]
  We study circuit discovery with classical and parameterized computational complexity theory. Our findings reveal a challenging complexity landscape. This framework allows us to understand the scope and limits of interpretability queries.
  arXiv  Detail & Related papers  (2024-10-10T15:22:48Z)
• Crafting Interpretable Embeddings by Asking LLMs Questions [89.49960984640363]
  Large language models (LLMs) have rapidly improved text embeddings for a growing array of natural-language processing tasks. We introduce question-answering embeddings (QA-Emb), embeddings where each feature represents an answer to a yes/no question asked to an LLM. We use QA-Emb to flexibly generate interpretable models for predicting fMRI voxel responses to language stimuli.
  arXiv  Detail & Related papers  (2024-05-26T22:30:29Z)
• Puzzle Solving using Reasoning of Large Language Models: A Survey [1.9939549451457024]
  This survey examines the capabilities of Large Language Models (LLMs) in puzzle solving. Our findings highlight the disparity between LLM capabilities and human-like reasoning. The survey underscores the necessity for novel strategies and richer datasets to advance LLMs' puzzle-solving proficiency.
  arXiv  Detail & Related papers  (2024-02-17T14:19:38Z)
• Retrieval Augmented Thought Process for Private Data Handling in Healthcare [53.89406286212502]
  We introduce the Retrieval-Augmented Thought Process (RATP) RATP formulates the thought generation of Large Language Models (LLMs) On a private dataset of electronic medical records, RATP achieves 35% additional accuracy compared to in-context retrieval-augmented generation for the question-answering task.
  arXiv  Detail & Related papers  (2024-02-12T17:17:50Z)
• From Heuristic to Analytic: Cognitively Motivated Strategies for Coherent Physical Commonsense Reasoning [66.98861219674039]
  Heuristic-Analytic Reasoning (HAR) strategies drastically improve the coherence of rationalizations for model decisions. Our findings suggest that human-like reasoning strategies can effectively improve the coherence and reliability of PLM reasoning.
  arXiv  Detail & Related papers  (2023-10-24T19:46:04Z)
• Machine-assisted quantitizing designs: augmenting humanities and social sciences with artificial intelligence [0.0]
  Large language models (LLMs) have been shown to present an unprecedented opportunity to scale up data analytics in the humanities and social sciences. We build on mixed methods quantitizing and converting design principles, and feature analysis from linguistics, to transparently integrate human expertise and machine scalability. The approach is discussed and demonstrated in over a dozen LLM-assisted case studies, covering 9 diverse languages, multiple disciplines and tasks.
  arXiv  Detail & Related papers  (2023-09-24T14:21:50Z)
• ConvFinQA: Exploring the Chain of Numerical Reasoning in Conversational Finance Question Answering [70.6359636116848]
  We propose a new large-scale dataset, ConvFinQA, to study the chain of numerical reasoning in conversational question answering. Our dataset poses great challenge in modeling long-range, complex numerical reasoning paths in real-world conversations.
  arXiv  Detail & Related papers  (2022-10-07T23:48:50Z)
• AR-LSAT: Investigating Analytical Reasoning of Text [57.1542673852013]
  We study the challenge of analytical reasoning of text and introduce a new dataset consisting of questions from the Law School Admission Test from 1991 to 2016. We analyze what knowledge understanding and reasoning abilities are required to do well on this task.
  arXiv  Detail & Related papers  (2021-04-14T02:53:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.