HiFloat4 Format for Language Model Inference

• URL: http://arxiv.org/abs/2602.11287v2
• Date: Fri, 13 Feb 2026 05:28:01 GMT
• Title: HiFloat4 Format for Language Model Inference
• Abstract summary: This paper introduces HiFloat4 (HiF4), a block floating-point data format tailored for deep learning.<n>Each HiF4 unit packs 64 4-bit elements with 32 bits of shared scaling metadata, averaging 4.5 bits per value.<n>Results show that HiF4 achieves higher average accuracy than the state-of-the-art NVFP4 format across multiple models and diverse downstream tasks.
• Score: 25.863121704892734
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper introduces HiFloat4 (HiF4), a block floating-point data format tailored for deep learning. Each HiF4 unit packs 64 4-bit elements with 32 bits of shared scaling metadata, averaging 4.5 bits per value. The metadata specifies a three-level scaling hierarchy, capturing inter- and intra-group dynamic range while improving the utilization of the representational space. In addition, the large 64-element group size enables matrix multiplications to be executed in a highly fixed-point manner, significantly reducing hardware area and power consumption. To evaluate the proposed format, we conducted inference experiments on several language models, including LLaMA, Qwen, Mistral, DeepSeek-V3.1 and LongCat. Results show that HiF4 achieves higher average accuracy than the state-of-the-art NVFP4 format across multiple models and diverse downstream tasks.

Related papers

• DepthART: Scaling Foundation Monocular Depth to Tiny Models [51.10174763031444]
  DepthART is a compact MDE model for on-device deployment across diverse scenes.<n>We show how DepthART consistently surpasses previous tiny baselines in both zero-shot generalization and metric accuracy.<n>We also provide a scalable model family, with DepthART-S reaching 347/245 FPS (strict FP32) on an GTX A6000 at $2242/4482$, 102 FPS (TF32) on a Orin NX 8GB, and over 15 FPS (FP32) on a Jetson Nano 4GB.
  arXiv  Detail & Related papers  (2026-07-19T06:54:52Z)
• HiFloat4 Format for Language Model Pre-training on Ascend NPUs [32.1837830814629]
  Recent work has demonstrated that 4-bit floating-point (FP4) formats can be successfully applied to linear GEMM operations in large language models (LLMs)<n>In this work, we investigate the recently proposed HiFloat4 FP4 format for Huawei Ascend NPUs and systematically compare it with MXFP4 in large-scale training settings.
  arXiv  Detail & Related papers  (2026-04-09T23:50:56Z)
• Adaptive Block-Scaled Data Types [7.634741713346681]
  NVFP4 suffers from its error distribution, resulting in large amounts of quantization error on near-maximal values.<n>We design new Adaptive Block-Scaled Data Types that can adapt to the distribution of their input values.<n> IF4 outperforms existing 4-bit block-scaled formats, achieving lower loss during quantized training and achieving higher accuracy on many tasks in post-training quantization.
  arXiv  Detail & Related papers  (2026-03-30T17:59:33Z)
• Unleashing Low-Bit Inference on Ascend NPUs: A Comprehensive Evaluation of HiFloat Formats [42.6259787270868]
  We evaluate HiFloat (HiF8 and HiF4), a family of formats tailored for Ascend NPUs.<n>HiFloat is fully compatible with state-of-the-art post-training quantization frameworks.
  arXiv  Detail & Related papers  (2026-02-13T05:41:31Z)
• Four Over Six: More Accurate NVFP4 Quantization with Adaptive Block Scaling [13.357423392911036]
  We introduce Four Over Six (4/6), a modification to the NVFP4 quantization algorithm that evaluates two potential scale factors for each block of values.<n>We find that for some blocks, scaling to smaller FP4 values makes the distribution of representable values more uniform.<n>We also find that 4/6 can be easily incorporated into many different post-training quantization methods and generally improves downstream accuracy.
  arXiv  Detail & Related papers  (2025-12-01T18:59:45Z)
• MiniCPM-V 4.5: Cooking Efficient MLLMs via Architecture, Data, and Training Recipe [68.04078852416248]
  MiniCPM-V 4.5 is an 8B parameter model designed for high efficiency and strong performance.<n>We introduce three core improvements in model architecture, data strategy and training method.<n>MiniCPM-V 4.5 achieves state-of-the-art performance among models under 30B size.
  arXiv  Detail & Related papers  (2025-09-16T19:41:48Z)
• Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs [195.24565517943802]
  We introduce Phi-4-Mini and Phi-4-Multimodal, compact yet highly capable language and multimodal models.<n>Phi-4-Mini is a 3.8-billion- parameter language model trained on high-quality web and synthetic data.<n>Phi-4-Multimodal is a multimodal model that integrates text, vision, and speech/audio input modalities into a single model.
  arXiv  Detail & Related papers  (2025-03-03T17:05:52Z)
• ShareGPT4V: Improving Large Multi-Modal Models with Better Captions [81.95879920888716]
  We introduce ShareGPT4V, a dataset featuring 1.2 million descriptive captions. This dataset surpasses existing datasets in diversity and information content, covering world knowledge, object properties, spatial relationships, and aesthetic evaluations. We further incorporate ShareGPT4V data into both the pre-training and SFT phases, obtaining ShareGPT4V-7B, a superior LMM.
  arXiv  Detail & Related papers  (2023-11-21T18:58:11Z)
• Efficient Post-training Quantization with FP8 Formats [14.543387418837154]
  We study the advantages of FP8 data formats for post-training quantization across 75 unique network architectures. E4M3 is better suited for NLP models, whereas E3M4 performs marginally better than E4M3 on computer vision tasks.
  arXiv  Detail & Related papers  (2023-09-26T00:58:36Z)
• A Multi-dimensional Deep Structured State Space Approach to Speech Enhancement Using Small-footprint Models [45.90759340302879]
  We explore several S4-based deep architectures in time (T) and time-frequency (TF) domains. The proposed TF-domain S4-based model is 78.6% smaller in size, yet it still achieves competitive results with a PESQ score of 3.15 with data augmentation.
  arXiv  Detail & Related papers  (2023-06-01T04:19:57Z)
• FP8 Formats for Deep Learning [49.54015320992368]
  We propose an 8-bit floating point (FP8) binary interchange format consisting of two encodings. E4M3's dynamic range is extended by not representing infinities and having only one mantissa bit-pattern for NaNs. We demonstrate the efficacy of the FP8 format on a variety of image and language tasks, effectively matching the result quality achieved by 16-bit training sessions.
  arXiv  Detail & Related papers  (2022-09-12T17:39:55Z)
• EFloat: Entropy-coded Floating Point Format for Deep Learning [2.3204178451683264]
  EFloat format encodes frequent exponent values with Huffman codes to minimize the average exponent field width. The proposed encoding concept may be beneficial to low-precision formats including 8-bit floats.
  arXiv  Detail & Related papers  (2021-02-04T15:58:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.