ZADU: A Python Library for Evaluating the Reliability of Dimensionality Reduction Embeddings

• URL: http://arxiv.org/abs/2308.00282v2
• Date: Fri, 11 Aug 2023 04:39:33 GMT
• Title: ZADU: A Python Library for Evaluating the Reliability of Dimensionality Reduction Embeddings
• Abstract summary: Dimensionality reduction (DR) techniques inherently distort the original structure of input high-dimensional data, producing imperfect low-dimensional embeddings. We present ZADU, a Python library that provides distortion measures and enables comprehensive evaluation of DR embeddings. As an application of ZADU, we present ZADUVis that allows users to easily create distortion visualizations that depict the extent to which each region of an embedding suffers from distortions.
• Score: 11.08175113417855
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Dimensionality reduction (DR) techniques inherently distort the original structure of input high-dimensional data, producing imperfect low-dimensional embeddings. Diverse distortion measures have thus been proposed to evaluate the reliability of DR embeddings. However, implementing and executing distortion measures in practice has so far been time-consuming and tedious. To address this issue, we present ZADU, a Python library that provides distortion measures. ZADU is not only easy to install and execute but also enables comprehensive evaluation of DR embeddings through three key features. First, the library covers a wide range of distortion measures. Second, it automatically optimizes the execution of distortion measures, substantially reducing the running time required to execute multiple measures. Last, the library informs how individual points contribute to the overall distortions, facilitating the detailed analysis of DR embeddings. By simulating a real-world scenario of optimizing DR embeddings, we verify that our optimization scheme substantially reduces the time required to execute distortion measures. Finally, as an application of ZADU, we present another library called ZADUVis that allows users to easily create distortion visualizations that depict the extent to which each region of an embedding suffers from distortions.

Related papers

• At FullTilt: Real-Time Open-Set 3D Macromolecule Detection Directly from Tilted 2D Projections [2.9465623430708905]
  VRAM constraints prohibit processing an entire 3D tomogram, forcing current methods to rely on slow sliding-window inference over extracted subvolumes.<n>We propose FullTilt, an end-to-end framework that redefines 3D detection by operating directly on aligned 2D tilt-series.<n>FullTilt achieves state-of-the-art zero-shot performance while drastically reducing VRAM requirements, paving the way for rapid, large-scale visual analysis.
  arXiv  Detail & Related papers  (2026-04-12T18:28:57Z)
• Tail-Aware Post-Training Quantization for 3D Geometry Models [58.79500829118265]
  Post-Training Quantization (PTQ) enables efficient inference without retraining.<n>PTQ fails to transfer effectively to 3D models due to intricate feature distributions and prohibitive calibration overhead.<n>We propose TAPTQ, a Tail-Aware Post-Training Quantization pipeline for 3D geometric learning.
  arXiv  Detail & Related papers  (2026-02-02T07:21:15Z)
• Cascaded Robust Rectification for Arbitrary Document Images [45.30113042855903]
  Document rectification in real-world scenarios poses significant challenges due to extreme variations in camera perspectives and physical distortions.<n>We introduce a novel multi-stage framework that progressively reverses distinct distortion types in a coarse-to-fine manner.<n>Our framework first performs a global affine transformation to correct perspective distortions arising from the camera's viewpoint, then rectifies geometric deformations resulting from physical paper curling and folding, and finally employs a content-aware iterative process to eliminate fine-grained content distortions.
  arXiv  Detail & Related papers  (2025-11-28T12:56:16Z)
• Domain Adaptation of Drag Reduction Policy to Partial Measurements [2.3020018305241337]
  We propose a method to adapt feedback control policies obtained from full-state measurements to setups with only partial measurements.<n>Our approach is demonstrated in a simulated environment by minimising the aerodynamic drag of a simplified road vehicle.
  arXiv  Detail & Related papers  (2025-07-06T09:30:43Z)
• DDL: A Large-Scale Datasets for Deepfake Detection and Localization in Diversified Real-World Scenarios [51.916287988122406]
  We present a novel large-scale deepfake detection and localization (textbfDDL) dataset containing over $textbf1.4M+$ forged samples.<n>Our DDL not only provides a more challenging benchmark for complex real-world forgeries but also offers crucial support for building next-generation deepfake detection, localization, and interpretability methods.
  arXiv  Detail & Related papers  (2025-06-29T15:29:03Z)
• TeZO: Empowering the Low-Rankness on the Temporal Dimension in the Zeroth-Order Optimization for Fine-tuning LLMs [58.19080159470868]
  We propose a novel low-rank ZO estimator, TeZO, which captures the low-rankness across both the model and temporal dimension. Specifically, we represent ZO perturbations along the temporal dimension as a 3D tensor and employ Canonical Polyadic Decomposition (CPD) to extract each low-rank 2D matrix.
  arXiv  Detail & Related papers  (2025-01-31T11:34:03Z)
• Geometry-Aware Instrumental Variable Regression [56.16884466478886]
  We propose a transport-based IV estimator that takes into account the geometry of the data manifold through data-derivative information. We provide a simple plug-and-play implementation of our method that performs on par with related estimators in standard settings.
  arXiv  Detail & Related papers  (2024-05-19T17:49:33Z)
• Evaluating Unsupervised Dimensionality Reduction Methods for Pretrained Sentence Embeddings [28.35953315232521]
  Sentence embeddings produced by Pretrained Language Models (PLMs) have received wide attention from the NLP community. High dimensionality of the sentence embeddings produced by PLMs is problematic when representing large numbers of sentences in memory- or compute-constrained devices. We evaluate unsupervised dimensionality reduction methods to reduce the dimensionality of sentence embeddings produced by PLMs.
  arXiv  Detail & Related papers  (2024-03-20T21:58:32Z)
• RobustCalib: Robust Lidar-Camera Extrinsic Calibration with Consistency Learning [42.90987864456673]
  Current methods for LiDAR-camera extrinsics estimation depend on offline targets and human efforts. We propose a novel approach to address the extrinsic calibration problem in a robust, automatic, and single-shot manner. We conduct comprehensive experiments on different datasets, and the results demonstrate that our method achieves accurate and robust performance.
  arXiv  Detail & Related papers  (2023-12-02T09:29:50Z)
• PREM: A Simple Yet Effective Approach for Node-Level Graph Anomaly Detection [65.24854366973794]
  Node-level graph anomaly detection (GAD) plays a critical role in identifying anomalous nodes from graph-structured data in domains such as medicine, social networks, and e-commerce. We introduce a simple method termed PREprocessing and Matching (PREM for short) to improve the efficiency of GAD. Our approach streamlines GAD, reducing time and memory consumption while maintaining powerful anomaly detection capabilities.
  arXiv  Detail & Related papers  (2023-10-18T02:59:57Z)
• Bias-Aware Minimisation: Understanding and Mitigating Estimator Bias in Private SGD [56.01810892677744]
  We show a connection between per-sample gradient norms and the estimation bias of the private gradient oracle used in DP-SGD. We propose Bias-Aware Minimisation (BAM) that allows for the provable reduction of private gradient estimator bias.
  arXiv  Detail & Related papers  (2023-08-23T09:20:41Z)
• Uncertainty-Aware Adaptation for Self-Supervised 3D Human Pose Estimation [70.32536356351706]
  We introduce MRP-Net that constitutes a common deep network backbone with two output heads subscribing to two diverse configurations. We derive suitable measures to quantify prediction uncertainty at both pose and joint level. We present a comprehensive evaluation of the proposed approach and demonstrate state-of-the-art performance on benchmark datasets.
  arXiv  Detail & Related papers  (2022-03-29T07:14:58Z)
• Camera Distortion-aware 3D Human Pose Estimation in Video with Optimization-based Meta-Learning [23.200130129530653]
  Existing 3D human pose estimation algorithms trained on distortion-free datasets suffer performance drop when applied to new scenarios with a specific camera distortion. We propose a simple yet effective model for 3D human pose estimation in video that can quickly adapt to any distortion environment.
  arXiv  Detail & Related papers  (2021-11-30T01:35:04Z)
• Efficient Real-Time Radial Distortion Correction for UAVs [1.7149364927872015]
  We present a novel algorithm for onboard radial distortion correction for unmanned aerial vehicles (UAVs) equipped with an inertial measurement unit (IMU) This approach makes calibration procedures redundant, thus allowing for exchange of optics extemporaneously. We propose a fast and robust minimal solver for simultaneously estimating the focal length, radial distortion profile and motion parameters from homographies.
  arXiv  Detail & Related papers  (2020-10-08T18:34:56Z)
• Evaluating representations by the complexity of learning low-loss predictors [55.94170724668857]
  We consider the problem of evaluating representations of data for use in solving a downstream task. We propose to measure the quality of a representation by the complexity of learning a predictor on top of the representation that achieves low loss on a task of interest.
  arXiv  Detail & Related papers  (2020-09-15T22:06:58Z)
• Monocular Real-Time Volumetric Performance Capture [28.481131687883256]
  We present the first approach to volumetric performance capture and novel-view rendering at real-time speed from monocular video. Our system reconstructs a fully textured 3D human from each frame by leveraging Pixel-Aligned Implicit Function (PIFu) We also introduce an Online Hard Example Mining (OHEM) technique that effectively suppresses failure modes due to the rare occurrence of challenging examples.
  arXiv  Detail & Related papers  (2020-07-28T04:45:13Z)
• Extrapolation for Large-batch Training in Deep Learning [72.61259487233214]
  We show that a host of variations can be covered in a unified framework that we propose. We prove the convergence of this novel scheme and rigorously evaluate its empirical performance on ResNet, LSTM, and Transformer.
  arXiv  Detail & Related papers  (2020-06-10T08:22:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.