TESSERA v2: Scaling Pixel-wise Earth Foundation Models

• URL: http://arxiv.org/abs/2607.03949v1
• Date: Sat, 04 Jul 2026 16:52:34 GMT
• Title: TESSERA v2: Scaling Pixel-wise Earth Foundation Models
• Abstract summary: We present the largest controlled scaling study for Earth-observation pixel-wise foundation models to date.<n> 395 training runs on 1,024 GH200 superchips within a fixed pixel-wise Barlow Twins family.<n>We distill pixel-wise models into compact students for embeddings-as-data deployment.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Pixel-wise Earth-observation (EO) foundation models are now achieving state-of-the-art performance via generated spatial embeddings. However, how these models scale and how best to spend a pretraining budget remain poorly understood. We present the largest controlled scaling study for EO to date: 395 training runs on 1,024 GH200 superchips within a fixed pixel-wise Barlow Twins family, each evaluated on 15 downstream tasks. We find that pretraining loss barely predicts downstream performance (|Pearson r| < 0.2), so selecting models by loss wastes a large share of the compute. We also find that, as the training budget grows, the encoder and the data should grow together while the projector stays fixed, which gives a simple rule for allocating compute. Using this rule, we train a family of pixel-wise models (0.5B and 1B, with a 2B model in training) and distill them into compact students for embeddings-as-data deployment. The 21-million-parameter distilled TESSERA v2-1B-M in aggregate outperforms all open and proprietary models tested, some of which are orders of magnitude larger. These students produce Matryoshka representations that are inexpensive to serve: a 16-dimensional prefix keeps 92% of the full 128-dimensional performance at 1/8 of the storage. Upon completion of training we plan to release v2 global embeddings covering 2017-2025. Together, these results give a concrete, empirically grounded recipe for scaling pixel-wise EO foundation models: train large encoders, select by downstream performance, and distil into flexible student models. All code will be released at https://github.com/ucam-eo/tessera.

Related papers

• How Far Can 5,500 Hours of Driving Take You? A Scaling Law Analysis of Video Diffusion Models [57.34034448977433]
  Video generation for autonomous driving is expensive to collect, bound by privacy requirements, and cannot be scraped at will.<n>We present a systematic scaling-law study of video diffusion models trained from scratch on driving data.<n>We train the largest video diffusion model trained from scratch on driving data: it sets a new open-source state of the art for driving video generation.
  arXiv  Detail & Related papers  (2026-08-28T14:56:01Z)
• A Frozen Pixel-Space Diffusion Model Can Guide Itself with Its Own Samples [53.8898272536272]
  Pixel-space diffusion models aim to learn an end-to-end generator directly over raw pixels.<n>Recent work improves pixel diffusion through alternative prediction targets, training objectives, and architectures.<n>We show there is a cheaper, complementary strategy: textbfa frozen, pretrained pixel diffusion model can guide itself
  arXiv  Detail & Related papers  (2026-07-31T07:52:08Z)
• SeFi-Image: A Text-to-Image Foundation Model with Semantic-First Diffusion [0.0]
  SeFi-Image is a text-to-image foundation model built upon semantic-first diffusion.<n>We instantiate SeFi-Image at three model scales, 1B, 2B, and 5B parameters.<n>SeFi-Image achieves strong performance on a wide range of benchmarks, including GenEval, DPG, LongTextBench, OneIG, and DMDG-2K.
  arXiv  Detail & Related papers  (2026-06-21T16:10:22Z)
• Compute-Optimal Scaling for Value-Based Deep RL [99.680827753493]
  We investigate compute scaling for online, value-based deep RL.<n>Our analysis reveals a nuanced interplay between model size, batch size, and UTD.<n>We provide a mental model for understanding this phenomenon and build guidelines for choosing batch size and UTD.
  arXiv  Detail & Related papers  (2025-08-20T17:54:21Z)
• Training and Inference Efficiency of Encoder-Decoder Speech Models [25.031622057759492]
  We focus on the efficiency angle and ask the questions of whether we are training these speech models efficiently.<n>We show that negligence in mini-batch sampling leads to more than 50% being spent on padding.<n>We find that adjusting the model architecture to transfer model parameters from the decoder to the encoder results in a 3x inference speedup.
  arXiv  Detail & Related papers  (2025-03-07T20:57:43Z)
• David and Goliath: Small One-step Model Beats Large Diffusion with Score Post-training [8.352666876052616]
  We propose Diff-Instruct* (DI*), a data-efficient post-training approach for one-step text-to-image generative models.<n>Our method frames alignment as online reinforcement learning from human feedback.<n>Our 2.6B emphDI*-SDXL-1step model outperforms the 50-step 12B FLUX-dev model.
  arXiv  Detail & Related papers  (2024-10-28T10:26:19Z)
• OmniBal: Towards Fast Instruction-Tuning for Vision-Language Models via Omniverse Computation Balance [67.37017498784748]
  Large-scale 3D parallel training on vision-language instruction-tuning models leads to an imbalanced computation load across different devices.<n>We rebalance the computational load from data, model, and memory perspectives, achieving more balanced computation across devices.<n>Our method's efficacy and generalizability are further validated across various models and datasets.
  arXiv  Detail & Related papers  (2024-07-30T12:02:58Z)
• Language models scale reliably with over-training and on downstream tasks [121.69867718185125]
  Scaling laws are useful guides for derisking expensive training runs. However, there remain gaps between current studies and how language models are trained. In contrast, scaling laws mostly predict loss on inference, but models are usually compared on downstream task performance.
  arXiv  Detail & Related papers  (2024-03-13T13:54:00Z)
• DINOv2: Learning Robust Visual Features without Supervision [75.42921276202522]
  This work shows that existing pretraining methods, especially self-supervised methods, can produce such features if trained on enough curated data from diverse sources. Most of the technical contributions aim at accelerating and stabilizing the training at scale. In terms of data, we propose an automatic pipeline to build a dedicated, diverse, and curated image dataset instead of uncurated data, as typically done in the self-supervised literature.
  arXiv  Detail & Related papers  (2023-04-14T15:12:19Z)
• TinyMIM: An Empirical Study of Distilling MIM Pre-trained Models [31.16595289223858]
  Masked image modeling (MIM) performs strongly in pre-training large vision Transformers (ViTs) However, small models that are critical for real-world applications cannot or only marginally benefit from this pre-training approach. We explore distillation techniques to transfer the success of large MIM-based pre-trained models to smaller ones.
  arXiv  Detail & Related papers  (2023-01-03T18:59:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.