CurveFP: Co-Designing Numerical Representation and Product Arithmetic for Language Models

• URL: http://arxiv.org/abs/2608.10010v2
• Date: Wed, 12 Aug 2026 03:58:39 GMT
• Title: CurveFP: Co-Designing Numerical Representation and Product Arithmetic for Language Models
• Abstract summary: We introduce CurveFP, a block-scaled family that distributes magnitudes across interleaved logarithmic curves.<n>We instantiate the algebra as CurveFP8 E4C3/E5C2 for training and CurveFP7 E3C3 for compact inference.<n>On four 7B-9B models, CurveFP7 beats tensorwise FP8 perplexity with one fewer element bit and stays within 1.32% of native quality.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Low-precision formats usually optimize scalar fidelity while inheriting conventional product arithmetic. We introduce CurveFP, a block-scaled family that distributes magnitudes across interleaved logarithmic curves. Uniform curve indices make every nonzero product an exact sign and integer-index update, while a rational radix exposes the finite phase schedule required for accumulation. We instantiate the algebra as CurveFP8 E4C3/E5C2 for training and CurveFP7 E3C3 for compact inference. On four 7B-9B models, CurveFP7 beats tensorwise FP8 perplexity with one fewer element bit and stays within 1.32% of native quality. CurveFP8 lowers error in all 36 paired training-GEMM comparisons. Across three matched 3B-token pretraining triplets, it reaches mean BF16-inference perplexity 22.5366 versus 22.5407 for FP8 and has a lower format penalty in every seed. Downstream evaluation shows transfer parity and a consistent WikiText-103 gain. In a preliminary 4x4 Nangate45 spatial accelerator tile, CurveFP8 uses one fewer product register and 4.6% less area than timing-closing FP8 at 500 MHz. These results support CurveFP as a numerical and arithmetic co-design, while leaving system-level efficiency to future study.

Related papers

• CubicQuant: Parametric Non-Uniform Codebooks for High-Throughput LLM Inference with 1-8-Bit Weights [0.0]
  We introduce CubicQuant, a non-uniform, parametric non-uniform scalar format for large-language-model inference.<n>We derive population distortion under Uniform, Gaussian, and Laplace distributions, formulate continuous and Dynamic-A8-carrier-aware fitting objectives, and describe direct packed-weight GPU execution.<n>Preliminary H200 kernel measurements show a workload-dependent crossover: model-dtype execution is faster for narrow GEMV, while Dynamic A8 becomes favorable as row count grows.
  arXiv  Detail & Related papers  (2026-08-07T03:36:07Z)
• Characterizing the Impact of NVFP4 Quantization for Low-Power Edge AI Deployment [9.460818703756205]
  Energy-efficient neural-network inference at the edge requires reducing arithmetic cost, memory traffic, energy, and storage overhead while maintaining acceptable accuracy.<n>This paper presents an ablation-focused study of NVFP4 quantization for edge-efficient neural networks.
  arXiv  Detail & Related papers  (2026-06-03T03:09:59Z)
• The Illusion of Equivalence: Systematic FP16 Divergence in KV-Cached Autoregressive Inference [2.954681536906518]
  KV caching is a ubiquitous optimization in autoregressive transformer inference.<n>We show that FP16 KV cache inference is fundamentally non-equivalent to recomputation.
  arXiv  Detail & Related papers  (2026-04-16T15:59:40Z)
• Forge-UGC: FX optimization and register-graph engine for universal graph compiler [2.587194279527956]
  Existing frameworks such as OpenVINO and ONNX often use opaque compilation pipelines.<n> Forge-UGC addresses this with a hardware-agnostic design that separates graph capture, optimization, intermediate representation lowering, and backend scheduling.
  arXiv  Detail & Related papers  (2026-04-14T04:39:16Z)
• BPDQ: Bit-Plane Decomposition Quantization on a Variable Grid for Large Language Models [56.504879072674015]
  We propose Bit-Plane Decomposition Quantization (BPDQ), which constructs a variable quantization grid via bit-planes and scalar coefficients.<n>BPDQ enables serving Qwen2.5-72B on a single GTX 3090 with 83.85% GSM8K accuracy (vs. 90.83% at 16-bit)
  arXiv  Detail & Related papers  (2026-02-04T02:54:37Z)
• FP8-Flow-MoE: A Casting-Free FP8 Recipe without Double Quantization Error [3.281844093101284]
  Training large Mixture-of-Experts (MoE) models remains computationally prohibitive due to their extreme compute and memory demands.<n>We propose FP8-Flow-MoE, a training recipe featuring a quantization-consistent FP8-centric dataflow with a scaling-aware computation and fused FP8 operators.
  arXiv  Detail & Related papers  (2025-11-04T06:36:59Z)
• INT v.s. FP: A Comprehensive Study of Fine-Grained Low-bit Quantization Formats [51.72056104795248]
  Modern AI hardware, such as Nvidia's Blackwell architecture, is increasingly embracing low-precision floating-point (FP) formats.<n>This paper systematically investigates the trade-offs between FP and integer (INT) formats.<n>We reveal a critical performance crossover: while FP excels in coarse-grained quantization, the comparison at fine-grained (block-wise) levels is more nuanced.
  arXiv  Detail & Related papers  (2025-10-29T15:11:53Z)
• Speeding Up MACE: Low-Precision Tricks for Equivarient Force Fields [51.95157731126864]
  Machine-learning force fields can deliver accurate molecular dynamics (MD) at high computational cost.<n>This thesis aims to make MACE cheaper and faster by identifying computational bottlenecks and evaluating low-precision execution policies.
  arXiv  Detail & Related papers  (2025-10-23T14:02:34Z)
• Towards Fully FP8 GEMM LLM Training at Scale [77.97607456493257]
  Existing approaches often rely on suboptimal fine-grained FP8 kernels or fall back to higher-precision matrix multiplications.<n>We introduce a new class of LLM architectures that, for the first time, support FP8 computation for all GEMMs within transformer blocks during both forward and backward passes.<n>This enables unprecedented throughput gains, particularly at scale, while matching the downstream performance of standard BF16 training.
  arXiv  Detail & Related papers  (2025-05-26T21:04:14Z)
• "Give Me BF16 or Give Me Death"? Accuracy-Performance Trade-Offs in LLM Quantization [67.3213104337679]
  Quantization is a powerful tool for accelerating large language model (LLM) inference, but the accuracy-performance trade-offs across different formats remain unclear.<n>We conduct the most comprehensive empirical study to date, evaluating FP8, INT8, and INT4 quantization across academic benchmarks and real-world tasks.
  arXiv  Detail & Related papers  (2024-11-04T18:21:59Z)
• FPN-fusion: Enhanced Linear Complexity Time Series Forecasting Model [1.9780020273082008]
  This study presents a novel time series prediction model, FPN-fusion, designed with linear computational complexity. Our model introduces two key innovations: first, a Feature Pyramid Network (FPN) is employed to effectively capture time series data characteristics, bypassing the traditional decomposition into trend and seasonal components. Empirically, FPN-fusion outperforms DLiner in 31 out of 32 test cases on eight open-source datasets, with an average reduction of 16.8% in mean squared error (MSE) and 11.8% in mean absolute error (MAE)
  arXiv  Detail & Related papers  (2024-06-06T14:34:26Z)
• FP8-LM: Training FP8 Large Language Models [47.17804713425323]
  In this paper, we propose a new FP8 automatic mixed-precision framework for training large language models. Experiment results show that, during the training of GPT-175B model on H100 GPU platform, our FP8 mixed-precision training framework not only achieved a remarkable 39% reduction in real memory usage but also ran 75% faster than the widely adopted BF16 framework.
  arXiv  Detail & Related papers  (2023-10-27T17:59:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.