CLiFT: Compressive Light-Field Tokens for Compute-Efficient and Adaptive Neural Rendering

• URL: http://arxiv.org/abs/2507.08776v2
• Date: Mon, 14 Jul 2025 01:25:49 GMT
• Title: CLiFT: Compressive Light-Field Tokens for Compute-Efficient and Adaptive Neural Rendering
• Abstract summary: This paper proposes a neural rendering approach that represents a scene as "compressed light-field tokens (CLiFTs)"<n>CLiFT enables compute-efficient rendering by compressed tokens, while being capable of changing the number of tokens to represent a scene or render a novel view with one trained network.
• Score: 20.916387646203273
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper proposes a neural rendering approach that represents a scene as "compressed light-field tokens (CLiFTs)", retaining rich appearance and geometric information of a scene. CLiFT enables compute-efficient rendering by compressed tokens, while being capable of changing the number of tokens to represent a scene or render a novel view with one trained network. Concretely, given a set of images, multi-view encoder tokenizes the images with the camera poses. Latent-space K-means selects a reduced set of rays as cluster centroids using the tokens. The multi-view ``condenser'' compresses the information of all the tokens into the centroid tokens to construct CLiFTs. At test time, given a target view and a compute budget (i.e., the number of CLiFTs), the system collects the specified number of nearby tokens and synthesizes a novel view using a compute-adaptive renderer. Extensive experiments on RealEstate10K and DL3DV datasets quantitatively and qualitatively validate our approach, achieving significant data reduction with comparable rendering quality and the highest overall rendering score, while providing trade-offs of data size, rendering quality, and rendering speed.

Related papers

• RADIO1D: Elastic Representations for Condensed Vision Modeling [69.84453279129937]
  This paper challenges the assumption that vision-language models (VLMs) require fixed patch-based 2D vision features.<n>We introduce RADIO1D, which compresses images into a compact, variable-length 1D token sequence using multi-teacher knowledge distillation and an autoencoder design.<n>The resulting representations exhibit strong hierarchical summarization, enabling accurate scene understanding - even with a single token - and support improved composition-aware image retrieval.
  arXiv  Detail & Related papers  (2026-07-03T22:58:54Z)
• Unified Driving Tokens: Representation- and Geometry-Guided Discrete Tokenizer for Driving World Models and Planning [13.385292682258147]
  We present a representation-guided and geometry-enhanced tokenizer that learns discrete tokens under joint supervision.<n> Experiments on NAVSIM show improved reconstruction fidelity and representation consistency, competitive planning performance under a fixed decoder, and better generative quality under matched settings.
  arXiv  Detail & Related papers  (2026-06-01T09:02:32Z)
• SceneTok: A Compressed, Diffusable Token Space for 3D Scenes [18.094424146923892]
  We present SceneTok, a novel tokenizer for encoding view sets of scenes into a compressed and diffusable set of unstructured tokens.<n>We show that the compression is 1-3 orders of magnitude stronger than for other representations while still reaching state-of-the-art reconstruction quality.
  arXiv  Detail & Related papers  (2026-02-21T16:08:17Z)
• ViCO: A Training Strategy towards Semantic Aware Dynamic High-Resolution [71.69364653858447]
  Existing Multimodal Large Language Models (MLLMs) suffer from increased inference costs due to the additional vision tokens introduced by image inputs.<n>We propose Visual Consistency Learning (ViCO), a novel training algorithm that enables the model to represent images of varying complexities using different numbers of vision tokens.<n> Experimental results demonstrate that our method can reduce the number of vision tokens by up to 50% while maintaining the model's perception, reasoning, and OCR capabilities.
  arXiv  Detail & Related papers  (2025-10-14T17:58:10Z)
• Neural Discrete Token Representation Learning for Extreme Token Reduction in Video Large Language Models [50.214593234229255]
  We introduce the novel task of Extreme Short Token Reduction, which aims to represent entire videos using a minimal set of discrete tokens.<n>On the Extreme Short Token Reduction task, our VQToken compresses sequences to just 0.07 percent of their original length while incurring only a 0.66 percent drop in accuracy on the NextQA-MC benchmark.
  arXiv  Detail & Related papers  (2025-03-21T09:46:31Z)
• LookupViT: Compressing visual information to a limited number of tokens [36.83826969693139]
  Vision Transformers (ViT) have emerged as the de-facto choice for numerous industry grade vision solutions. But their inference cost can be prohibitive for many settings, as they compute self-attention in each layer which suffers from complexity in the number of tokens. In this work, we introduce LookupViT, that exploits this information sparsity to reduce ViT inference cost.
  arXiv  Detail & Related papers  (2024-07-17T17:22:43Z)
• Understanding the Effect of using Semantically Meaningful Tokens for Visual Representation Learning [40.08368469646114]
  We provide semantically-meaningful visual tokens to transformer encoders within a vision-language pre-training framework.<n>We demonstrate notable improvements over ViTs in learned representation quality across text-to-image and image-to-text retrieval tasks.
  arXiv  Detail & Related papers  (2024-05-26T01:46:22Z)
• Leveraging Cross-Modal Neighbor Representation for Improved CLIP Classification [54.96876797812238]
  We present a novel CrOss-moDal nEighbor Representation(CODER) based on the distance structure between images and their neighbor texts. The key to construct a high-quality CODER lies in how to create a vast amount of high-quality and diverse texts to match with images. Experiment results across various datasets and models confirm CODER's effectiveness.
  arXiv  Detail & Related papers  (2024-04-27T02:04:36Z)
• Subobject-level Image Tokenization [60.80949852899857]
  Patch-based image tokenization ignores the morphology of the visual world.<n>Inspired by subword tokenization, we introduce subobject-level adaptive token segmentation.<n>We show that subobject tokenization enables faster convergence and better generalization while using fewer visual tokens.
  arXiv  Detail & Related papers  (2024-02-22T06:47:44Z)
• StrokeNUWA: Tokenizing Strokes for Vector Graphic Synthesis [112.25071764647683]
  StrokeNUWA is a pioneering work exploring a better visual representation ''stroke tokens'' on vector graphics. equipped with stroke tokens, StrokeNUWA can significantly surpass traditional LLM-based and optimization-based methods. StrokeNUWA achieves up to a 94x speedup in inference over the speed of prior methods with an exceptional SVG code compression ratio of 6.9%.
  arXiv  Detail & Related papers  (2024-01-30T15:20:26Z)
• SAMPLING: Scene-adaptive Hierarchical Multiplane Images Representation for Novel View Synthesis from a Single Image [60.52991173059486]
  We introduce SAMPLING, a Scene-adaptive Hierarchical Multiplane Images Representation for Novel View Synthesis from a Single Image. Our method demonstrates considerable performance gains in large-scale unbounded outdoor scenes using a single image on the KITTI dataset.
  arXiv  Detail & Related papers  (2023-09-12T15:33:09Z)
• Cascaded Cross-Attention Networks for Data-Efficient Whole-Slide Image Classification Using Transformers [0.11219061154635457]
  Whole-Slide Imaging allows for the capturing and digitization of high-resolution images of histological specimen. transformer architecture has been proposed as a possible candidate for effectively leveraging the high-resolution information. We propose a novel cascaded cross-attention network (CCAN) based on the cross-attention mechanism that scales linearly with the number of extracted patches.
  arXiv  Detail & Related papers  (2023-05-11T16:42:24Z)
• Modeling Image Composition for Complex Scene Generation [77.10533862854706]
  We present a method that achieves state-of-the-art results on layout-to-image generation tasks. After compressing RGB images into patch tokens, we propose the Transformer with Focal Attention (TwFA) for exploring dependencies of object-to-object, object-to-patch and patch-to-patch.
  arXiv  Detail & Related papers  (2022-06-02T08:34:25Z)
• Learning light field synthesis with Multi-Plane Images: scene encoding as a recurrent segmentation task [30.058283056074426]
  This paper addresses the problem of view synthesis from large baseline light fields by turning a sparse set of input views into a Multi-plane Image (MPI) Because available datasets are scarce, we propose a lightweight network that does not require extensive training. Our model does not learn to estimate RGB layers but only encodes the scene geometry within MPI alpha layers, which comes down to a segmentation task.
  arXiv  Detail & Related papers  (2020-02-12T14:35:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.