A Modern Multimodal Assistant on a 6 GB 2011 GPU: Stage-Validated, All-GPU CUDA Inference for Fermi

• URL: http://arxiv.org/abs/2607.14568v1
• Date: Thu, 16 Jul 2026 04:48:44 GMT
• Title: A Modern Multimodal Assistant on a 6 GB 2011 GPU: Stage-Validated, All-GPU CUDA Inference for Fermi
• Abstract summary: This report keeps the hardware and asks what a model that fits can do.<n>We deploy Mini-V-4.6, a modern multimodal assistant pairing a SigLIP2 encoder and window-attention merger.<n>The system answers an image question end-to-end in 1.7s.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: A companion study ran a 35B mixture-of-experts model on a 2011 NVIDIA Tesla C2075 (Fermi, sm_20, 6GB) as a GPU-prefill/CPU-decode hybrid, because the 4-bit model did not fit in device memory (arXiv:2606.24031). This report keeps the hardware and asks what a model that fits can do: we deploy MiniCPM-V-4.6, a modern multimodal assistant pairing a SigLIP2 vision encoder and window-attention merger (16x visual token compression) with a compact hybrid gated-delta-net backbone, entirely on the GPU. Three results. (i) An all-GPU engine built on measured foundations: projections that dequantize 8-bit weights once and call the vendor SGEMM still in the last Fermi toolchain (64% of FP32 peak; our best hand-written GEMM hit 37%, wrongly called the ceiling); a chunked delta-rule rewrite of the recurrent layers, 2.8x faster than the sequential scan once attribution exposed one bad kernel; and a measured negative: 4-bit weights make decode slower than 8-bit here, since Fermi issues nibble-unpacking shifts at half rate. (ii) The vision side is a port with a proof obligation: we translate tower, merger, and projector to sm_20 CUDA, validating every stage against a locally generated reference forward (full tower 1.4e-5). One failure, position-embedding bucketization differing on exact rational ties, generalizes to a rule: float tie-breaking in index arithmetic is implementation-defined; call the reference operator, do not reimplement it. (iii) Long context exposes an O(N^2) wall short benchmarks hide: prefill falls from 114 tok/s at 2k tokens to 21 at 10k in a naive attention kernel; per-head vendor-GEMM calls writing into the existing score buffer (zero extra memory) restore a flat profile (408 at 2k, 361 at 10k; 17x), verified by exact needle retrieval from 60% depth. The same rewrite cuts image encoding 6x, to 0.93s. The system answers an image question end-to-end in 1.7s.

Related papers

• Unfolding the Leech Lattice: Fused Multi-Shell Decoding and VRAM Layouts for 2-Bit LLM Weights [8.550323073861415]
  Leech-lattice vector quantization holds the strongest reported 2-bit quality under its own evaluation protocol.<n>This paper supplies one and measures its serving cost for decode-phase GEMV at batch 1.
  arXiv  Detail & Related papers  (2026-09-02T14:26:06Z)
• Daedalus-150M: A Convolution-Attention Hybrid Designed for CPU Inference [0.0]
  Small language models are usually built like large ones and then squeezed onto a CPU afterwards.<n>We did the opposite: we fixed the target first, one user, one token at a time, 4-bit weights, ordinary CPU, and chose the architecture to suit it.<n>The model scores 47.31 on a five-task benchmark against a bar of 42.20 that was fixed before training began.<n>It beats GPT-2 124M, Pythia-160M, OPT-125M and GPT-neo-125M, all trained on three to six times more data, and exceeds MobileLLM-125M'
  arXiv  Detail & Related papers  (2026-08-20T16:09:43Z)
• Jetson-ORB-SLAM3: Accuracy-Preserving GPU Implementation for Edge Computing Devices [0.09558392439655011]
  We present an accuracy-preserving GPU implementation of ORB-SLAM3 for the NVIDIA Jetson Orin Nano.<n>In monocular-inertial mode the system sustains 32FPS mean over the eleven EuRoC sequences.
  arXiv  Detail & Related papers  (2026-08-18T15:07:37Z)
• DepthART: Scaling Foundation Monocular Depth to Tiny Models [51.10174763031444]
  DepthART is a compact MDE model for on-device deployment across diverse scenes.<n>We show how DepthART consistently surpasses previous tiny baselines in both zero-shot generalization and metric accuracy.<n>We also provide a scalable model family, with DepthART-S reaching 347/245 FPS (strict FP32) on an GTX A6000 at $2242/4482$, 102 FPS (TF32) on a Orin NX 8GB, and over 15 FPS (FP32) on a Jetson Nano 4GB.
  arXiv  Detail & Related papers  (2026-07-19T06:54:52Z)
• TileMaxSim: IO-Aware GPU MaxSim Scoring with Dimension Tiling and Fused Product Quantization [3.3723515662362265]
  Multi-vector retrieval models such as ColBERT achieve state-of-the-art accuracy through fine-grained token-level MaxSim scoring.<n>Existing GPU implementations leave most hardware performance unused.<n>We present TileMaxSim, a family of IO-aware Triton kernels that close this gap.
  arXiv  Detail & Related papers  (2026-06-24T23:03:56Z)
• Scaling DoRA: High-Rank Adaptation via Factored Norms and Fused Kernels [83.99688944263843]
  Weight-De Low-Rank Adaptation (DoRA) extends LoRA by decoupling weight magnitude from direction.<n>At d_in = 8192 and rank r = 384, a single module's norm requires about 512 MB of transient working memory in bf16.<n>A factored norm decomposes the squared norm into base, cross, and Gram terms computable through O(d_out r + r2) intermediates, eliminating the dense product.
  arXiv  Detail & Related papers  (2026-03-23T17:57:24Z)
• Architecture-Aware LLM Inference Optimization on AMD Instinct GPUs: A Comprehensive Benchmark and Deployment Study [0.0]
  Cross-architecture evaluation of production LLM inference on AMD Instinct MI325X GPUs.<n> benchmarking four models spanning 235B to 1 trillion parameters across three architectural families.
  arXiv  Detail & Related papers  (2026-02-27T13:21:48Z)
• Chronicals: A High-Performance Framework for LLM Fine-Tuning with 3.51x Speedup over Unsloth [0.0]
  We present Chronicals, an open-source training framework achieving 3.5x speedup over Unsloth.<n>We provide complete mathematical foundations: online softmax correctness, FlashAttention IO complexity O(N2 d2 M-1), LoRA+ learning rate gradient approximations.
  arXiv  Detail & Related papers  (2026-01-06T00:00:55Z)
• GPU-Accelerated Interpretable Generalization for Rapid Cyberattack Detection and Forensics [0.0]
  IG mechanism recently published in IEEE Transactions on Information Forensics and Security delivers state-of-the-art, evidence-based intrusion detection.<n>We present IG-GPU, a PyTorch re-architecture that offloads all pairwise intersections and subset evaluations to commodity GPU.<n>In 15k-record NSL-KDD dataset, IG-GPU shows a 116-fold speed-up over the multi-core CPU implementation of IG.
  arXiv  Detail & Related papers  (2025-07-16T12:38:19Z)
• 70% Size, 100% Accuracy: Lossless LLM Compression for Efficient GPU Inference via Dynamic-Length Float [52.079202872069835]
  Large-scale AI models, such as Large Language Models (LLMs) and Diffusion Models (DMs) have grown rapidly in size.<n>We introduce Dynamic-Length Float (DFloat11), a compression framework that reduces LLM and DM size by 30% while preserving outputs that are bit-for-bit identical to the original model.
  arXiv  Detail & Related papers  (2025-04-15T22:38:38Z)
• Speedy MASt3R [68.47052557089631]
  MASt3R redefines image matching as a 3D task by leveraging DUSt3R and introducing a fast reciprocal matching scheme.<n>Fast MASt3R achieves a 54% reduction in inference time (198 ms to 91 ms per image pair) without sacrificing accuracy.<n>This advancement enables real-time 3D understanding, benefiting applications like mixed reality navigation and large-scale 3D scene reconstruction.
  arXiv  Detail & Related papers  (2025-03-13T03:56:22Z)
• HeadInfer: Memory-Efficient LLM Inference by Head-wise Offloading [79.38548165722229]
  HEADINFER offloads the KV cache to CPU RAM while avoiding the need to fully store the KV cache for any transformer layer on the GPU.<n>We demonstrate HEADINFER maintains computational efficiency while significantly reducing memory footprint.
  arXiv  Detail & Related papers  (2025-02-18T06:26:05Z)
• Fast Inference with Kronecker-Sparse Matrices [4.387337528923525]
  Existing GPU kernels for KS matrix multiplication suffer from high data movement costs.<n>We propose a fused, output-stationary GPU kernel that eliminates these overheads.<n>We demonstrate in FP32 end-to-end latency reductions of up to 22% in ViT-S/16 and 16% in GPT-2 medium.
  arXiv  Detail & Related papers  (2024-05-23T19:36:10Z)
• Efficient and Generic 1D Dilated Convolution Layer for Deep Learning [52.899995651639436]
  We introduce our efficient implementation of a generic 1D convolution layer covering a wide range of parameters. It is optimized for x86 CPU architectures, in particular, for architectures containing Intel AVX-512 and AVX-512 BFloat16 instructions. We demonstrate the performance of our optimized 1D convolution layer by utilizing it in the end-to-end neural network training with real genomics datasets.
  arXiv  Detail & Related papers  (2021-04-16T09:54:30Z)
• Hybrid Models for Learning to Branch [81.93868699246214]
  We propose a new hybrid architecture for efficient branching on CPU machines. The proposed architecture combines the expressive power of GNNs with computationally inexpensive multi-layer perceptrons (MLP) for branching.
  arXiv  Detail & Related papers  (2020-06-26T21:03:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.