See the Change, Keep the Flow: Unsupervised Action Segmentation via Spectral-Temporal Representation Learning

• URL: http://arxiv.org/abs/2608.29611v1
• Date: Sun, 30 Aug 2026 07:04:28 GMT
• Title: See the Change, Keep the Flow: Unsupervised Action Segmentation via Spectral-Temporal Representation Learning
• Abstract summary: SpecT-OT is a spectral-temporal representation learning framework built upon an unbalanced optimal transport pseudo-labeling concept.<n>We show that SpecT-OT achieves the best results on 13 of 15 metrics, including 4.1-point MoF and 7.4-point F1 gains over the baseline on Breakfast and Desktop Assembly.
• Score: 28.808122183182576
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Unsupervised action segmentation aims to discover latent action categories and their temporal organization without action annotations. Optimal transport-based methods provide structured frame-to-action assignments, however, their pseudo-label quality is fundamentally conditioned on the representation space used to construct the transport cost. We argue that reliable OT pseudo-labeling requires a representation geometry that is simultaneously sensitive to discriminative action changes and coherent along local temporal progressions. Based on this insight, we propose SpecT-OT, a spectral-temporal representation learning framework built upon an unbalanced optimal transport pseudo-labeling concept. SpecT-OT introduces a Spectral Reparameterization Projector (SRP), which parameterizes projector weights with fixed Fourier bases and learnable coefficients to improve the modeling of rapidly varying discriminative features, and Temporal Affinity Regularization (TAR), which imposes distance-aware, label-free constraints on pairwise frame affinities to stabilize local temporal structure. The two components jointly produce more discriminative and temporally stable transport costs, yielding more reliable pseudo-labels for iterative representation learning. Experiments on four benchmarks demonstrate strong performance compared with state-of-the-art methods. SpecT-OT achieves the best results on 13 of 15 metrics, including 4.1-point MoF and 7.4-point F1 gains over the baseline on Breakfast and Desktop Assembly, respectively.

Related papers

• FM-ChangeNet: Learning Change through Pathwise Feature Transport [0.28675177318965045]
  FM-ChangeNet is a pathwise-supervised framework for change detection.<n>We construct intermediate latent states and learn a time-conditioned velocity field.<n> Experiments on remote sensing benchmarks show that the proposed framework produces more structured and robust change representations.
  arXiv  Detail & Related papers  (2026-07-06T07:42:58Z)
• AnchorMoE: Interpretable Time Series Classification via Anchor-Routed MoE [51.994752158953084]
  AnchorMoE is an interpretable-by-construction classification framework.<n>It encodes multi-view representations of local patches and routes them to specialized experts.<n>Experiments on real-world and synthetic benchmarks demonstrate that AnchorMoE achieves highly competitive classification performance.
  arXiv  Detail & Related papers  (2026-06-02T13:30:54Z)
• Hierarchical Consistency Learning for Test-time Adaptation in Camouflage Perception [50.278200968044665]
  Camouflaged object detection (COD) aims to localize targets that exhibit minimal perceptual differences from backgrounds through physical attributes.<n>Existing methods, constrained by the static train-then-freeze paradigm, suffer from domain rigidity and dependency annotation.<n>We propose the hierarchical consistency learning framework, which integrates test-time adaptation for dynamic representation recalibration.
  arXiv  Detail & Related papers  (2026-05-25T09:57:46Z)
• Beyond Point-Wise Matching: Structural Representation Alignment for Accelerating Diffusion Transformers [93.3976834364707]
  We propose sREPA, a structural REPresentation Alignment framework to enforce consistency in the relational geometry of feature maps.<n>By encouraging the model to internalize holistic spatial layouts and structural correlations from pre-trained features, sREPA achieves faster and more stable convergence.
  arXiv  Detail & Related papers  (2026-05-16T12:01:04Z)
• Hybrid Congestion Classification Framework Using Flow-Guided Attention and Empirical Mode Decomposition [6.830580384536778]
  This study proposes FLO-EMD, a hybrid approach that couples motion-guided attention with empirical, data-driven temporal decomposition.<n>Experiments on 1,050 five-second clips from four surveillance networks show FLO-EMD 97.5% overall test accuracy.
  arXiv  Detail & Related papers  (2026-05-06T10:59:32Z)
• Cross-Domain Transfer with Self-Supervised Spectral-Spatial Modeling for Hyperspectral Image Classification [5.784164305429653]
  This paper proposes a self-supervised cross-domain transfer framework.<n>It learns transferable spectral-spatial joint representations without source labels.<n> Experimental results demonstrate stable classification performance and strong cross-domain adaptability.
  arXiv  Detail & Related papers  (2026-01-26T02:52:35Z)
• Bidirectional Feature-aligned Motion Transformation for Efficient Dynamic Point Cloud Compression [97.66080040613726]
  We propose a Bidirectional Feature-aligned Motion Transformation (Bi-FMT) framework that implicitly models motion in the feature space.<n>Bi-FMT aligns features across both past and future frames to produce temporally consistent latent representations.<n>We show Bi-FMT surpasses D-DPCC and AdaDPCC in both compression efficiency and runtime.
  arXiv  Detail & Related papers  (2025-09-18T03:51:06Z)
• ERIS: An Energy-Guided Feature Disentanglement Framework for Out-of-Distribution Time Series Classification [51.07970070817353]
  An ideal time series classification (TSC) should be able to capture invariant representations.<n>Current methods are largely unguided, lacking the semantic direction required to isolate truly universal features.<n>We propose an end-to-end Energy-Regularized Information for Shift-Robustness framework to enable guided and reliable feature disentanglement.
  arXiv  Detail & Related papers  (2025-08-19T12:13:41Z)
• Time-o1: Time-Series Forecasting Needs Transformed Label Alignment [50.54348432664401]
  Time-o1 is a transformation-augmented learning objective tailored for time-series forecasting.<n>The central idea is to transform the label sequence into decorrelated components with discriminated significance.<n>Time-o1 achieves state-of-the-art performance and is compatible with various forecast models.
  arXiv  Detail & Related papers  (2025-05-23T13:00:35Z)
• Multi-body SE(3) Equivariance for Unsupervised Rigid Segmentation and Motion Estimation [49.56131393810713]
  We present an SE(3) equivariant architecture and a training strategy to tackle this task in an unsupervised manner. Our method excels in both model performance and computational efficiency, with only 0.25M parameters and 0.92G FLOPs.
  arXiv  Detail & Related papers  (2023-06-08T22:55:32Z)
• Spatiotemporal Multi-scale Bilateral Motion Network for Gait Recognition [3.1240043488226967]
  In this paper, motivated by optical flow, the bilateral motion-oriented features are proposed. We develop a set of multi-scale temporal representations that force the motion context to be richly described at various levels of temporal resolution.
  arXiv  Detail & Related papers  (2022-09-26T01:36:22Z)
• Joint Spatial-Temporal and Appearance Modeling with Transformer for Multiple Object Tracking [59.79252390626194]
  We propose a novel solution named TransSTAM, which leverages Transformer to model both the appearance features of each object and the spatial-temporal relationships among objects. The proposed method is evaluated on multiple public benchmarks including MOT16, MOT17, and MOT20, and it achieves a clear performance improvement in both IDF1 and HOTA.
  arXiv  Detail & Related papers  (2022-05-31T01:19:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.