An Empirical Audit of Input Encoders for Multi-Channel Signal Transformers

• URL: http://arxiv.org/abs/2606.04752v2
• Date: Mon, 08 Jun 2026 08:15:04 GMT
• Title: An Empirical Audit of Input Encoders for Multi-Channel Signal Transformers
• Abstract summary: Transformers consuming multi-channel scalar signals must embed $C$ simultaneous values into one $d_textmodel$-dimensional vector per time step.<n>We audit eight input encoders -- a shared-scalar baseline, per-channel linear projections, an explicit regulariser, a nonlinearoretic, block-partitioned concatenation, channel-independent and channel-as-token.<n>We find that the standard per-channel linear projection matches every alternative up to small, statistically real but practically modest differences.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Transformers consuming multi-channel scalar signals must embed $C$ simultaneous values into one $d_{\text{model}}$-dimensional vector per time step. We audit eight input encoders -- a shared-scalar baseline, per-channel linear projections, an orthogonality regulariser, a nonlinear MLP, block-partitioned concatenation, channel-independent and channel-as-token architectures, and a projected positional encoding -- on a synthetic benchmark where channel identity is informative and on ETTh1, scored by next-step negative log-likelihood. The headline is practical near-equivalence within a wide "top tier": the standard per-channel linear projection matches every alternative up to small, statistically real but practically modest differences. A direct geometric probe attributes this to a spontaneous orthogonalisation of the per-channel projections: they end up near-orthogonal with no explicit regulariser, letting the standard linear recover channel identity from the summed embedding. Two encoders lose decisively: the shared-scalar baseline collapses for information-theoretic reasons we make explicit, and the channel-independent PatchTST-spirit baseline overfits universally on the synthetic benchmark and underperforms on both. Paired tests resolve two small gaps: projecting the sinusoidal positional encoding through a learned linear layer edges the rest at small $C$ by extending this orthogonality to the positional subspace; a nonlinear MLP stem edges them at the largest $C$, with the gap shrinking under more training data. The practical recommendation: use the standard per-channel linear projection by default; reach for something more elaborate only when the task calls for it.

Related papers

• DUET -- Dual User Embedding Transformers for Offsite Conversion Prediction [8.972672149050627]
  Offsite conversion rate (OCVR) prediction is an important ranking problem in computational recommendation systems.<n>This task presents a modeling challenge: click signals are abundant and exhibit short temporal horizons, whereas conversion signals are inherently sparse, long-delayed, and frequently unattributed.<n>We propose DUET, a framework that explicitly partitions user behavioral data into two domain-coherent streams -- clicks and conversions.
  arXiv  Detail & Related papers  (2026-06-08T23:13:58Z)
• Metric--Phase Fields: Decoupling Distance and Sign for Thin-Structure Reconstruction from Unoriented Point Clouds [67.30236447078187]
  We introduce Metric-Phase Fields (MPFs), a decoupled implicit representation that separates metric proximity from topological phase.<n>Experiments on both synthetic and scanned thin-shell and thin-plate shapes demonstrate that MPFs preserve thin and layered structures more faithfully than recent SDF-based methods.
  arXiv  Detail & Related papers  (2026-05-25T07:08:57Z)
• Learning to Recall with Transformers Beyond Orthogonal Embeddings [42.18876773867171]
  We analyze a transformer with random embeddings trained with gradient descent on a simple token-retrieval task.<n>Our analysis tracks the early phase'' of gradient descent and yields explicit formulas for the model's storage capacity.
  arXiv  Detail & Related papers  (2026-03-16T21:17:01Z)
• Gated Differential Linear Attention: A Linear-Time Decoder for High-Fidelity Medical Segmentation [15.30336007288786]
  PVT-GDLA is a decoder-centric Transformer that restores sharp, long-range dependencies at linear time.<n>It achieves state-of-the-art accuracy across CT, MRI, ultrasound, and dermoscopy benchmarks under equal training budgets.
  arXiv  Detail & Related papers  (2026-03-03T08:26:08Z)
• EqDeepRx: Learning a Scalable MIMO Receiver [6.732584013520367]
  This paper presents EqDeepRx, a practical deep-learning-aided multiple-input multiple-output (MIMO) receiver.<n>At the core of the receiver model is a shared-weight DetectorNN that operates independently on each spatial stream or layer.<n>5G/6G-compliant end-to-end simulations across multiple channel scenarios, pilot patterns, and inter-cell interference conditions show improved error rate and spectral efficiency.
  arXiv  Detail & Related papers  (2026-02-12T11:22:30Z)
• Gated Removal of Normalization in Transformers Enables Stable Training and Efficient Inference [2.1665689529884697]
  We introduce TaperNorm, a drop-in replacement for RMSNorm/LayerNorm that behaves exactly like the standard normalizer early in training.<n>A single global gate is held at $g=1$ during gate warmup, used to calibrate the scaling branch via EMAs, and then cosine-decayed to $g=0$.<n>On an efficiency microbenchmark, folding internal scalings yields up to $1.22times$ higher throughput in last-token logits mode.
  arXiv  Detail & Related papers  (2026-02-11T01:40:34Z)
• CP Loss: Channel-wise Perceptual Loss for Time Series Forecasting [67.3477355449697]
  We propose a Channel-wise Perceptual Loss (CP Loss) for time-series data.<n>We learn a unique perceptual space for each channel that is adapted to its characteristics.<n>Loss is calculated within these perception spaces to optimize the model.
  arXiv  Detail & Related papers  (2026-01-25T15:31:37Z)
• GSPN-2: Efficient Parallel Sequence Modeling [101.33780567131716]
  Generalized Spatial Propagation Network (GSPN) addresses this by replacing quadratic self-attention with a line-scan propagation scheme.<n>GSPN-2 establishes a new efficiency frontier for modeling global spatial context in vision applications.
  arXiv  Detail & Related papers  (2025-11-28T07:26:45Z)
• Unveiling Induction Heads: Provable Training Dynamics and Feature Learning in Transformers [54.20763128054692]
  We study how a two-attention-layer transformer is trained to perform ICL on $n$-gram Markov chain data. We prove that the gradient flow with respect to a cross-entropy ICL loss converges to a limiting model.
  arXiv  Detail & Related papers  (2024-09-09T18:10:26Z)
• Transformers as Support Vector Machines [54.642793677472724]
  We establish a formal equivalence between the optimization geometry of self-attention and a hard-margin SVM problem. We characterize the implicit bias of 1-layer transformers optimized with gradient descent. We believe these findings inspire the interpretation of transformers as a hierarchy of SVMs that separates and selects optimal tokens.
  arXiv  Detail & Related papers  (2023-08-31T17:57:50Z)
• Low PAPR MIMO-OFDM Design Based on Convolutional Autoencoder [20.544993155126967]
  A new framework for peak-to-average power ratio ($mathsfPAPR$) reduction and waveform design is presented. A convolutional-autoencoder ($mathsfCAE$) architecture is presented. We show that a single trained model covers the tasks of $mathsfPAPR$ reduction, spectrum design, and $mathsfMIMO$ detection together over a wide range of SNR levels.
  arXiv  Detail & Related papers  (2023-01-11T11:35:10Z)
• Efficient Nearest Neighbor Search for Cross-Encoder Models using Matrix Factorization [60.91600465922932]
  We present an approach that avoids the use of a dual-encoder for retrieval, relying solely on the cross-encoder. Our approach provides test-time recall-vs-computational cost trade-offs superior to the current widely-used methods.
  arXiv  Detail & Related papers  (2022-10-23T00:32:04Z)
• Error Correction Code Transformer [92.10654749898927]
  We propose to extend for the first time the Transformer architecture to the soft decoding of linear codes at arbitrary block lengths. We encode each channel's output dimension to high dimension for better representation of the bits information to be processed separately. The proposed approach demonstrates the extreme power and flexibility of Transformers and outperforms existing state-of-the-art neural decoders by large margins at a fraction of their time complexity.
  arXiv  Detail & Related papers  (2022-03-27T15:25:58Z)
• Just Least Squares: Binary Compressive Sampling with Low Generative Intrinsic Dimension [12.67951378791069]
  We consider recovering $n$ dimensional signals from $m$ binary measurements corrupted by noises and sign flips. Although the binary measurements model is highly nonlinear, we propose a least square decoder.
  arXiv  Detail & Related papers  (2021-11-29T12:06:47Z)
• Distributed Sparse Feature Selection in Communication-Restricted Networks [6.9257380648471765]
  We propose and theoretically analyze a new distributed scheme for sparse linear regression and feature selection. In order to infer the causal dimensions from the whole dataset, we propose a simple, yet effective method for information sharing in the network.
  arXiv  Detail & Related papers  (2021-11-02T05:02:24Z)
• Inductive Bias of Multi-Channel Linear Convolutional Networks with Bounded Weight Norm [15.08164172607321]
  We study the function space characterization of the inductive bias resulting from controlling the $ell$ norm of the weights in linear convolutional networks. For sufficiently large $C$, the induced regularizer for $K=1$ and $K=D$ are the nuclear norm and the $ell_2,1$ group-sparse norm, respectively.
  arXiv  Detail & Related papers  (2021-02-24T12:01:23Z)
• Analytic Signal Phase in $N-D$ by Linear Symmetry Tensor--fingerprint modeling [69.35569554213679]
  We show that the Analytic Signal phase, and its gradient have a hitherto unstudied discontinuity in $2-D $ and higher dimensions. This shortcoming can result in severe artifacts whereas the problem does not exist in $1-D $ signals. We suggest the use of Linear Symmetry phase, relying on more than one set of Gabor filters, but with a negligible computational add-on.
  arXiv  Detail & Related papers  (2020-05-16T21:17:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.