Share, Collaborate, Benchmark: Advancing Travel Demand Research through rigorous open-source collaboration

• URL: http://arxiv.org/abs/2306.06194v2
• Date: Sun, 14 Jul 2024 22:11:43 GMT
• Title: Share, Collaborate, Benchmark: Advancing Travel Demand Research through rigorous open-source collaboration
• Abstract summary: A critical barrier preventing travel demand literature from effectively informing policy is the volume of publications without clear, consolidated benchmarks. We present a collaborative infrastructure for transit demand prediction models, focusing on their performance during dynamic conditions like the COVID-19 pandemic. We found that the prediction error for the LSTM deep learning approach stabilized at a mean arctangent absolute percentage error (MAAPE) of about 0.12 within 1.5 months, whereas other models continued to exhibit higher error rates even a year into the pandemic.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This research foregrounds general practices in travel demand research, emphasizing the need to change our ways. A critical barrier preventing travel demand literature from effectively informing policy is the volume of publications without clear, consolidated benchmarks, making it difficult for researchers and policymakers to gather insights and use models to guide decision-making. By emphasizing reproducibility and open collaboration, we aim to enhance the reliability and policy relevance of travel demand research. We present a collaborative infrastructure for transit demand prediction models, focusing on their performance during highly dynamic conditions like the COVID-19 pandemic. Drawing from over 300 published papers, we develop an open-source infrastructure with five common methodologies and assess their performance under stable and dynamic conditions. We found that the prediction error for the LSTM deep learning approach stabilized at a mean arctangent absolute percentage error (MAAPE) of about 0.12 within 1.5 months, whereas other models continued to exhibit higher error rates even a year into the pandemic. If research practices had prioritized reproducibility before the COVID-19 pandemic, transit agencies would have had clearer guidance on the best forecasting methods and quickly identified those best suited for pandemic conditions to inform operations in response to changes in transit demand. The aim of this open-source codebase is to lower the barrier for other researchers to replicate, reproduce models and build upon findings. We encourage researchers to test their own modeling approaches on this benchmarking platform, challenge the analyses conducted in this paper, and develop model specifications that can outperform those evaluated here. Further, collaborative research approaches must be expanded across travel demand modeling if we wish to impact policy and planning.

Related papers

• How Do Agents Fail on AutoResearch: End-to-End Diagnostic Evaluation on 100 Real-World Frontier Research Tasks [6.001213285107465]
  AutoResearchEval features 100 tasks grounded in published frontier science across 7 scientific domains.<n> evaluating 8 harness-model combinations yields 800 autoresearch agent trajectories.<n>We leverage a human-calibrated agent-as-a-judge pipeline to inspect complete trajectories and intermediate artifacts.
  arXiv  Detail & Related papers  (2026-08-14T21:39:38Z)
• Position: Anthropomorphic Misalignment Research Needs Stronger Evidence [52.76826423649968]
  We show how conceptual ambiguity, non-robust datasets, experimental design, and insufficient causal interventions can lead to overinterpretation of model behaviors.<n>This position paper aims to offer guidance on evidentiary considerations that can help improve methodological rigor in AMR.
  arXiv  Detail & Related papers  (2026-05-29T16:38:53Z)
• Learning to Predict Future-Aligned Research Proposals with Language Models [59.79457676644722]
  We build a time-consistent dataset of 17,771 papers from targets and their pre-cutoff citations.<n>To train models, we build a time-consistent dataset of 17,771 papers from targets and their pre-cutoff citations.<n>Across Llama-3.1 and Qwen2.5 models, future-aligned tuning improves future alignment over unaligned baselines.
  arXiv  Detail & Related papers  (2026-03-28T05:41:15Z)
• Step-DeepResearch Technical Report [90.50586290399683]
  We introduce Step-DeepResearch, a cost-effective, end-to-end agent.<n>We propose a Data Synthesis Strategy Based on Atomic Capabilities to reinforce planning and report writing.<n>To bridge the evaluation gap in the Chinese domain, we establish ADR-Bench for realistic deep research scenarios.
  arXiv  Detail & Related papers  (2025-12-23T16:32:27Z)
• IterResearch: Rethinking Long-Horizon Agents via Markovian State Reconstruction [107.49922328855025]
  IterResearch is a novel iterative deep-research paradigm that reformulates long-horizon research as a Markov Decision Process.<n>It achieves substantial improvements over existing open-source agents with average +14.5pp across six benchmarks.<n>It serves as an effective prompting strategy, improving frontier models by up to 19.2pp over ReAct on long-horizon tasks.
  arXiv  Detail & Related papers  (2025-11-10T17:30:08Z)
• Synthesizing Attitudes, Predicting Actions (SAPA): Behavioral Theory-Guided LLMs for Ridesourcing Mode Choice Modeling [4.901179513819295]
  Existing models for predicting ridesourcing mode choices often suffer from limited predictive accuracy due to their inability to capture key psychological factors.<n>This paper introduces the Synthesizing Attitudes, Predicting Actions (SAPA) framework to synthesize theory-grounded latent attitudes.<n> Experiments on a large-scale, multi-year travel survey show that SAPA significantly outperforms state-of-the-art baselines.
  arXiv  Detail & Related papers  (2025-09-18T01:52:27Z)
• Beyond Memorization: Reasoning-Driven Synthesis as a Mitigation Strategy Against Benchmark Contamination [77.69093448529455]
  We present an empirical study using an infinitely scalable framework to synthesize research-level QA directly from arXiv papers.<n>We evaluate a lack of significant performance decay near knowledge cutoff dates for models of various sizes, developers, and release dates.<n>We hypothesize that the multi-step reasoning required by our synthesis pipeline offered additional complexity that goes deeper than shallow memorization.
  arXiv  Detail & Related papers  (2025-08-26T16:41:37Z)
• Anomaly Detection and Generation with Diffusion Models: A Survey [51.61574868316922]
  Anomaly detection (AD) plays a pivotal role across diverse domains, including cybersecurity, finance, healthcare, and industrial manufacturing.<n>Recent advancements in deep learning, specifically diffusion models (DMs), have sparked significant interest.<n>This survey aims to guide researchers and practitioners in leveraging DMs for innovative AD solutions across diverse applications.
  arXiv  Detail & Related papers  (2025-06-11T03:29:18Z)
• Preference Learning for AI Alignment: a Causal Perspective [55.2480439325792]
  We frame this problem in a causal paradigm, providing the rich toolbox of causality to identify persistent challenges.<n>Inheriting from the literature of causal inference, we identify key assumptions necessary for reliable generalisation.<n>We illustrate failure modes of naive reward models and demonstrate how causally-inspired approaches can improve model robustness.
  arXiv  Detail & Related papers  (2025-06-06T10:45:42Z)
• Reimagining Urban Science: Scaling Causal Inference with Large Language Models [39.231736674554995]
  Urban causal research is essential for understanding the complex, dynamic processes that shape cities.<n>Current practices are often constrained by inefficient and biased hypothesis formulation.<n>We propose UrbanCIA, a conceptual framework composed of four distinct modular agents responsible for hypothesis generation, data engineering, experiment design and execution, and results interpretation with policy insights.
  arXiv  Detail & Related papers  (2025-04-15T16:58:11Z)
• LlaMADRS: Prompting Large Language Models for Interview-Based Depression Assessment [75.44934940580112]
  This study introduces LlaMADRS, a novel framework leveraging open-source Large Language Models (LLMs) to automate depression severity assessment. We employ a zero-shot prompting strategy with carefully designed cues to guide the model in interpreting and scoring transcribed clinical interviews. Our approach, tested on 236 real-world interviews, demonstrates strong correlations with clinician assessments.
  arXiv  Detail & Related papers  (2025-01-07T08:49:04Z)
• A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applications [52.42860559005861]
  Direct Preference Optimization (DPO) has emerged as a promising approach for alignment. Despite DPO's various advancements and inherent limitations, an in-depth review of these aspects is currently lacking in the literature.
  arXiv  Detail & Related papers  (2024-10-21T02:27:24Z)
• Assessing Reusability of Deep Learning-Based Monotherapy Drug Response Prediction Models Trained with Omics Data [43.57729817547386]
  Cancer drug response prediction models present a promising approach towards precision oncology. Deep learning (DL) methods have shown great potential in this area. This highlights the need for reusable and adaptable models that can be improved and tested by the wider scientific community.
  arXiv  Detail & Related papers  (2024-09-18T16:08:28Z)
• A Multilateral Attention-enhanced Deep Neural Network for Disease Outbreak Forecasting: A Case Study on COVID-19 [0.6874745415692134]
  We propose a novel approach to address the challenges of infectious disease forecasting. We introduce a Multilateral Attention-enhanced GRU model that leverages information from multiple sources. By incorporating attention mechanisms within a GRU framework, our model can effectively capture complex relationships and temporal dependencies in the data.
  arXiv  Detail & Related papers  (2024-08-26T06:31:53Z)
• Source-Free Unsupervised Domain Adaptation with Hypothesis Consolidation of Prediction Rationale [53.152460508207184]
  Source-Free Unsupervised Domain Adaptation (SFUDA) is a challenging task where a model needs to be adapted to a new domain without access to target domain labels or source domain data. This paper proposes a novel approach that considers multiple prediction hypotheses for each sample and investigates the rationale behind each hypothesis. To achieve the optimal performance, we propose a three-step adaptation process: model pre-adaptation, hypothesis consolidation, and semi-supervised learning.
  arXiv  Detail & Related papers  (2024-02-02T05:53:22Z)
• Less is More: Mitigate Spurious Correlations for Open-Domain Dialogue Response Generation Models by Causal Discovery [52.95935278819512]
  We conduct the first study on spurious correlations for open-domain response generation models based on a corpus CGDIALOG curated in our work. Inspired by causal discovery algorithms, we propose a novel model-agnostic method for training and inference of response generation model.
  arXiv  Detail & Related papers  (2023-03-02T06:33:48Z)
• Assaying Out-Of-Distribution Generalization in Transfer Learning [103.57862972967273]
  We take a unified view of previous work, highlighting message discrepancies that we address empirically. We fine-tune over 31k networks, from nine different architectures in the many- and few-shot setting.
  arXiv  Detail & Related papers  (2022-07-19T12:52:33Z)
• SAIS: Supervising and Augmenting Intermediate Steps for Document-Level Relation Extraction [51.27558374091491]
  We propose to explicitly teach the model to capture relevant contexts and entity types by supervising and augmenting intermediate steps (SAIS) for relation extraction. Based on a broad spectrum of carefully designed tasks, our proposed SAIS method not only extracts relations of better quality due to more effective supervision, but also retrieves the corresponding supporting evidence more accurately.
  arXiv  Detail & Related papers  (2021-09-24T17:37:35Z)
• Towards Robust and Reliable Algorithmic Recourse [11.887537452826624]
  We propose a novel framework, RObust Algorithmic Recourse (ROAR), that leverages adversarial training for finding recourses that are robust to model shifts. We also carry out detailed theoretical analysis which underscores the importance of constructing recourses that are robust to model shifts.
  arXiv  Detail & Related papers  (2021-02-26T17:38:52Z)
• Combining Graph Neural Networks and Spatio-temporal Disease Models to Predict COVID-19 Cases in Germany [0.0]
  Several experts have called for the necessity to account for human mobility to explain the spread of COVID-19. Most statistical or epidemiological models cannot directly incorporate unstructured data sources, including data that may encode human mobility. We propose a trade-off between both research directions and present a novel learning approach that combines the advantages of statistical regression and machine learning models.
  arXiv  Detail & Related papers  (2021-01-03T16:39:00Z)
• A Self-supervised Approach for Semantic Indexing in the Context of COVID-19 Pandemic [1.8782750537161614]
  We present a novel semantic indexing approach based on the state-of-the-art self-supervised representation learning and transformer encoding. Our study sheds light on the main challenges confronting semantic indexing models during a pandemic, namely new domains and drastic changes of their distributions.
  arXiv  Detail & Related papers  (2020-10-07T17:43:55Z)
• On the model-based stochastic value gradient for continuous reinforcement learning [50.085645237597056]
  We show that simple model-based agents can outperform state-of-the-art model-free agents in terms of both sample-efficiency and final reward. Our findings suggest that model-based policy evaluation deserves closer attention.
  arXiv  Detail & Related papers  (2020-08-28T17:58:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.