Prototype Training with Dual Pseudo-Inverse and Optimized Hidden Activations

• URL: http://arxiv.org/abs/2508.09787v1
• Date: Wed, 13 Aug 2025 13:13:32 GMT
• Title: Prototype Training with Dual Pseudo-Inverse and Optimized Hidden Activations
• Abstract summary: Proto-PINV+H is a fast training paradigm that combines closed-form weight computation with optimisation of a small set of synthetic inputs.<n>On MNIST (60k train, 10k test) and Fashion-MNIST (60k train, 10k test), our method reaches 97.8% and 89.3% test accuracy on the official 10k test sets, respectively.
• Score: 0.14504054468850663
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present Proto-PINV+H, a fast training paradigm that combines closed-form weight computation with gradient-based optimisation of a small set of synthetic inputs, soft labels, and-crucially-hidden activations. At each iteration we recompute all weight matrices in closed form via two (or more) ridge-regularised pseudo-inverse solves, while updating only the prototypes with Adam. The trainable degrees of freedom are thus shifted from weight space to data/activation space. On MNIST (60k train, 10k test) and Fashion-MNIST (60k train, 10k test), our method reaches 97.8% and 89.3% test accuracy on the official 10k test sets, respectively, in 3.9s--4.5s using approximately 130k trainable parameters and only 250 epochs on an RTX 5060 (16GB). We provide a multi-layer extension (optimised activations at each hidden stage), learnable ridge parameters, optional PCA/PLS projections, and theory linking the condition number of prototype matrices to generalisation. The approach yields favourable accuracy--speed--size trade-offs against ELM, random-feature ridge, and shallow MLPs trained by back-propagation.

Related papers

• Unextractable Protocol Models: Collaborative Training and Inference without Weight Materialization [58.14514930760722]
  We consider a decentralized setup in which the participants collaboratively train and serve a large neural network.<n>In this setup, we explore the possibility of unmaterializable weights, where a full weight set is never available to any one participant.<n>We introduce Unextractable Protocol Models (UPMs): a training and inference framework that leverages the sharded model setup.
  arXiv  Detail & Related papers  (2026-05-22T10:24:57Z)
• Prototype-Based Test-Time Adaptation of Vision-Language Models [58.28899599773308]
  Prototype-Based Test-Time Adaptation (PTA)<n>PTA uses a set of class-specific knowledge prototypes to accumulate knowledge from test samples.<n>PTA achieves extremely high efficiency while achieving state-of-the-art performance.
  arXiv  Detail & Related papers  (2026-04-23T07:20:56Z)
• Compressing Transformer Language Models via Matrix Product Operator Decomposition: A Case Study on PicoGPT [0.0]
  Transformer-based language models achieve strong performance across NLP tasks, but their quadratic parameter scaling with hidden dimension makes deployment on resource-constrained hardware expensive.<n>We study Matrix Product Operator decomposition as a principled compression method for transformers.<n>MPO factorises weight matrices into chains of low-rank cores, with approximation quality controlled by the bond dimension chi.
  arXiv  Detail & Related papers  (2026-03-30T14:57:47Z)
• OPTIMA: Optimal One-shot Pruning for LLMs via Quadratic Programming Reconstruction [12.653025902977001]
  Post-training model pruning is a promising solution, yet it faces a trade-off that zero weights are fast but degrade accuracy.<n>One-shot methods such as SparseGPT offer a practical trade-off in optimality by applying efficient, approximate weight updates.<n>We introduce OPTIMA, a practical one-shot post-training pruning method that balances accuracy and scalability.
  arXiv  Detail & Related papers  (2025-12-15T20:41:29Z)
• Fantastic Pretraining Optimizers and Where to Find Them [59.56075036649332]
  AdamW has long been the dominant gradients in language model pretraining.<n>Speedup of matrix-based matrices is inversely proportional to model scale.
  arXiv  Detail & Related papers  (2025-09-02T07:43:22Z)
• Gaussian Weight Sampling for Scalable, Efficient and Stable Pseudo-Quantization Training [5.908471365011943]
  Pseudo-quantization training (PQT) addresses the issues of fully quantized training (FQT)<n>We show that PQT with weight sampling is (3) stable while pre-training GPT2 and Llama2 models with up to 1B parameters and 300B tokens.
  arXiv  Detail & Related papers  (2025-05-16T12:14:12Z)
• Democratizing AI: Open-source Scalable LLM Training on GPU-based Supercomputers [65.35142508909892]
  We present a novel four-dimensional hybrid parallel algorithm implemented in a highly scalable, portable, open-source framework called AxoNN.<n>We demonstrate fine-tuning of a 405-billion parameter LLM using AxoNN on Frontier.
  arXiv  Detail & Related papers  (2025-02-12T06:05:52Z)
• HALO: Hadamard-Assisted Lower-Precision Optimization for LLMs [45.37278584462772]
  We present HALO, a novel quantization-aware training approach for Transformers.<n>Our approach ensures that all large matrix multiplications during the forward and backward passes are executed in lower precision.<n>Applying to LLAMA-family models, HALO achieves near-full-precision-equivalent results during fine-tuning on various tasks.
  arXiv  Detail & Related papers  (2025-01-05T18:41:54Z)
• Diffusion Sampling Correction via Approximately 10 Parameters [8.577537076809316]
  We propose PCA-based Adaptive Search (PAS) for Diffusion Probabilistic Models (DPMs)<n>We employ PCA to obtain a few basis vectors to span the high-dimensional sampling space, which enables us to learn just a set of coordinates to correct the sampling direction.<n>We then design an adaptive search strategy that further enhances the sampling efficiency and reduces the number of stored parameters to approximately 10.
  arXiv  Detail & Related papers  (2024-11-10T15:57:53Z)
• Video Prediction Transformers without Recurrence or Convolution [65.93130697098658]
  We propose PredFormer, a framework entirely based on Gated Transformers.<n>We provide a comprehensive analysis of 3D Attention in the context of video prediction.<n>The significant improvements in both accuracy and efficiency highlight the potential of PredFormer.
  arXiv  Detail & Related papers  (2024-10-07T03:52:06Z)
• Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMs [54.05511925104712]
  We propose a simple, effective, and data-efficient method called Step-DPO. Step-DPO treats individual reasoning steps as units for preference optimization rather than evaluating answers holistically. Our findings demonstrate that as few as 10K preference data pairs and fewer than 500 Step-DPO training steps can yield a nearly 3% gain in accuracy on MATH for models with over 70B parameters.
  arXiv  Detail & Related papers  (2024-06-26T17:43:06Z)
• SVFT: Parameter-Efficient Fine-Tuning with Singular Vectors [80.6043267994434]
  We propose SVFT, a simple approach that fundamentally differs from existing methods. SVFT updates (W) as a sparse combination of outer products of its singular vectors, training only the coefficients (scales) of these sparse combinations. Experiments on language and vision benchmarks show that SVFT recovers up to 96% of full fine-tuning performance while training only 0.006 to 0.25% of parameters.
  arXiv  Detail & Related papers  (2024-05-30T01:27:43Z)
• Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuning [126.84770886628833]
  Existing finetuning methods either tune all parameters of the pretrained model (full finetuning) or only tune the last linear layer (linear probing) We propose a new parameter-efficient finetuning method termed as SSF, representing that researchers only need to Scale and Shift the deep Features extracted by a pre-trained model to catch up with the performance full finetuning.
  arXiv  Detail & Related papers  (2022-10-17T08:14:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.