ELSA: Evaluating Localization of Social Activities in Urban Streets using Open-Vocabulary Detection

• URL: http://arxiv.org/abs/2406.01551v2
• Date: Thu, 05 Dec 2024 14:25:56 GMT
• Title: ELSA: Evaluating Localization of Social Activities in Urban Streets using Open-Vocabulary Detection
• Authors: Maryam Hosseini, Marco Cipriano, Sedigheh Eslami, Daniel Hodczak, Liu Liu, Andres Sevtsuk, Gerard de Melo,
• Abstract summary: We present ELSA: evaluating localization of social activities.<n>We introduce a novel confidence score computation method NLSE and a novel Dynamic Box Aggregation (DBA) algorithm to assess semantic consistency in overlapping predictions.<n>We report our results on the widely-used SOTA models Grounding DINO, Detic, OWL, and MDETR.
• Score: 22.420962041487282
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Existing Open Vocabulary Detection (OVD) models exhibit a number of challenges. They often struggle with semantic consistency across diverse inputs, and are often sensitive to slight variations in input phrasing, leading to inconsistent performance. The calibration of their predictive confidence, especially in complex multi-label scenarios, remains suboptimal, frequently resulting in overconfident predictions that do not accurately reflect their context understanding. To understand these limitations, multi-label detection benchmarks are needed. A particularly challenging domain for such benchmarking is social activities. Due to the lack of multi-label benchmarks for social interactions, in this work we present ELSA: Evaluating Localization of Social Activities. ELSA draws on theoretical frameworks in urban sociology and design and uses in-the-wild street-level imagery, where the size of groups and the types of activities vary significantly. ELSA includes more than 900 manually annotated images with more than 4,300 multi-labeled bounding boxes for individual and group activities. We introduce a novel confidence score computation method NLSE and a novel Dynamic Box Aggregation (DBA) algorithm to assess semantic consistency in overlapping predictions. We report our results on the widely-used SOTA models Grounding DINO, Detic, OWL, and MDETR. Our evaluation protocol considers semantic stability and localization accuracy and further exposes the limitations of existing approaches.

Related papers

• SEOE: A Scalable and Reliable Semantic Evaluation Framework for Open Domain Event Detection [70.23196257213829]
  We propose a scalable and reliable Semantic-level Evaluation framework for Open domain Event detection. Our proposed framework first constructs a scalable evaluation benchmark that currently includes 564 event types covering 7 major domains. We then leverage large language models (LLMs) as automatic evaluation agents to compute a semantic F1-score, incorporating fine-grained definitions of semantically similar labels.
  arXiv  Detail & Related papers  (2025-03-05T09:37:05Z)
• MMIE: Massive Multimodal Interleaved Comprehension Benchmark for Large Vision-Language Models [71.36392373876505]
  We introduce MMIE, a large-scale benchmark for evaluating interleaved multimodal comprehension and generation in Large Vision-Language Models (LVLMs) MMIE comprises 20K meticulously curated multimodal queries, spanning 3 categories, 12 fields, and 102 subfields, including mathematics, coding, physics, literature, health, and arts. It supports both interleaved inputs and outputs, offering a mix of multiple-choice and open-ended question formats to evaluate diverse competencies.
  arXiv  Detail & Related papers  (2024-10-14T04:15:00Z)
• SKADA-Bench: Benchmarking Unsupervised Domain Adaptation Methods with Realistic Validation On Diverse Modalities [55.87169702896249]
  Unsupervised Domain Adaptation (DA) consists of adapting a model trained on a labeled source domain to perform well on an unlabeled target domain with some data distribution shift. We present a complete and fair evaluation of existing shallow algorithms, including reweighting, mapping, and subspace alignment. Our benchmark highlights the importance of realistic validation and provides practical guidance for real-life applications.
  arXiv  Detail & Related papers  (2024-07-16T12:52:29Z)
• SLIDE: A Framework Integrating Small and Large Language Models for Open-Domain Dialogues Evaluation [23.203761925540736]
  We propose a novel framework SLIDE (Small and Large Integrated for Dialogue Evaluation) Our approach achieves state-of-the-art performance in both the classification and evaluation tasks, and additionally the SLIDE exhibits better correlation with human evaluators.
  arXiv  Detail & Related papers  (2024-05-24T20:32:49Z)
• Detecting Multimodal Situations with Insufficient Context and Abstaining from Baseless Predictions [75.45274978665684]
  Vision-Language Understanding (VLU) benchmarks contain samples where answers rely on assumptions unsupported by the provided context. We collect contextual data for each sample whenever available and train a context selection module to facilitate evidence-based model predictions. We develop a general-purpose Context-AwaRe Abstention detector to identify samples lacking sufficient context and enhance model accuracy.
  arXiv  Detail & Related papers  (2024-05-18T02:21:32Z)
• Differences of communication activity and mobility patterns between urban and rural people [0.0]
  This study uses the call detail records (CDRs) to analyze the social communication and mobility patterns of people. Results show that in the urban areas people have high calling activity but low mobility, while in the rural areas they show the opposite behaviour. Age and gender of individuals have also been observed to play a role in the seasonal patterns differently in urban and rural areas.
  arXiv  Detail & Related papers  (2023-11-22T19:10:14Z)
• Time-space dynamics of income segregation: a case study of Milan's neighbourhoods [0.0]
  We propose a three-dimensional space to analyze social mixing, which is embedded in the temporal dynamics of urban activities. While residential areas fail to encourage social mixing in the nighttime, the working hours foster inclusion, with the city center showing a heightened level of interaction. leisure areas emerge as potential facilitators for social interactions, depending on urban features such as public transport and a variety of Points Of Interest.
  arXiv  Detail & Related papers  (2023-09-29T14:50:13Z)
• Spatiotemporal gender differences in urban vibrancy [0.0]
  We show that there are differences between males and females in terms of urban vibrancy. We also find that there are both positive and negative spatial spillovers existing across each city. Our results increase our understanding of inequality in cities and how we can make future cities fairer.
  arXiv  Detail & Related papers  (2023-04-25T14:12:58Z)
• Accounting for multiplicity in machine learning benchmark performance [0.0]
  Using the highest-ranked performance as an estimate for state-of-the-art (SOTA) performance is a biased estimator, giving overly optimistic results. In this article, we provide a probability distribution for the case of multiple classifiers so that known analyses methods can be engaged and a better SOTA estimate can be provided.
  arXiv  Detail & Related papers  (2023-03-10T10:32:18Z)
• Flexible social inference facilitates targeted social learning when rewards are not observable [58.762004496858836]
  Groups coordinate more effectively when individuals are able to learn from others' successes. We suggest that social inference capacities may help bridge this gap, allowing individuals to update their beliefs about others' underlying knowledge and success from observable trajectories of behavior.
  arXiv  Detail & Related papers  (2022-12-01T21:04:03Z)
• Active Learning-based Isolation Forest (ALIF): Enhancing Anomaly Detection in Decision Support Systems [2.922007656878633]
  ALIF is a lightweight modification of the popular Isolation Forest that proved superior performances with respect to other state-of-art algorithms. The proposed approach is particularly appealing in the presence of a Decision Support System (DSS), a case that is increasingly popular in real-world scenarios.
  arXiv  Detail & Related papers  (2022-07-08T14:36:38Z)
• This Must Be the Place: Predicting Engagement of Online Communities in a Large-scale Distributed Campaign [70.69387048368849]
  We study the behavior of communities with millions of active members. We develop a hybrid model, combining textual cues, community meta-data, and structural properties. We demonstrate the applicability of our model through Reddit's r/place a large-scale online experiment.
  arXiv  Detail & Related papers  (2022-01-14T08:23:16Z)
• SSAGCN: Social Soft Attention Graph Convolution Network for Pedestrian Trajectory Prediction [59.064925464991056]
  We propose one new prediction model named Social Soft Attention Graph Convolution Network (SSAGCN) SSAGCN aims to simultaneously handle social interactions among pedestrians and scene interactions between pedestrians and environments. Experiments on public available datasets prove the effectiveness of SSAGCN and have achieved state-of-the-art results.
  arXiv  Detail & Related papers  (2021-12-05T01:49:18Z)
• AES Systems Are Both Overstable And Oversensitive: Explaining Why And Proposing Defenses [66.49753193098356]
  We investigate the reason behind the surprising adversarial brittleness of scoring models. Our results indicate that autoscoring models, despite getting trained as "end-to-end" models, behave like bag-of-words models. We propose detection-based protection models that can detect oversensitivity and overstability causing samples with high accuracies.
  arXiv  Detail & Related papers  (2021-09-24T03:49:38Z)
• JRDB-Act: A Large-scale Multi-modal Dataset for Spatio-temporal Action, Social Group and Activity Detection [54.696819174421584]
  We introduce JRDB-Act, a multi-modal dataset that reflects a real distribution of human daily life actions in a university campus environment. JRDB-Act has been densely annotated with atomic actions, comprises over 2.8M action labels. JRDB-Act comes with social group identification annotations conducive to the task of grouping individuals based on their interactions in the scene.
  arXiv  Detail & Related papers  (2021-06-16T14:43:46Z)
• PHASE: PHysically-grounded Abstract Social Events for Machine Social Perception [50.551003004553806]
  We create a dataset of physically-grounded abstract social events, PHASE, that resemble a wide range of real-life social interactions. Phase is validated with human experiments demonstrating that humans perceive rich interactions in the social events. As a baseline model, we introduce a Bayesian inverse planning approach, SIMPLE, which outperforms state-of-the-art feed-forward neural networks.
  arXiv  Detail & Related papers  (2021-03-02T18:44:57Z)
• Discovering Underground Maps from Fashion [80.02941583103612]
  We propose a method to automatically create underground neighborhood maps of cities by analyzing how people dress. Using publicly available images from across a city, our method finds neighborhoods with a similar fashion sense and segments the map without supervision.
  arXiv  Detail & Related papers  (2020-12-04T23:40:59Z)
• Automatic Extraction of Urban Outdoor Perception from Geolocated Free-Texts [1.8419317899207144]
  We propose an automatic and generic approach to extract people's perceptions. We exemplify our approach in the context of urban outdoor areas in Chicago, New York City and London. We show that our approach can be helpful to better understand urban areas considering different perspectives.
  arXiv  Detail & Related papers  (2020-10-13T14:59:46Z)
• Joint Learning of Social Groups, Individuals Action and Sub-group Activities in Videos [23.15064911470468]
  We propose an end-to-end trainable framework for the social task. Our main contributions are: i) we propose an end-to-end trainable framework for the social task; ii) our proposed method sets the state-of-the-art results on two widely adopted benchmarks for the traditional group recognition activity task.
  arXiv  Detail & Related papers  (2020-07-06T10:42:11Z)
• Uncertainty-aware Score Distribution Learning for Action Quality Assessment [91.05846506274881]
  We propose an uncertainty-aware score distribution learning (USDL) approach for action quality assessment (AQA) Specifically, we regard an action as an instance associated with a score distribution, which describes the probability of different evaluated scores. Under the circumstance where fine-grained score labels are available, we devise a multi-path uncertainty-aware score distributions learning (MUSDL) method to explore the disentangled components of a score.
  arXiv  Detail & Related papers  (2020-06-13T15:41:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.