Daily Paper Briefing

Abstract Overview

MNIST-PRO reformulates MNIST digit recognition as a partially observable sequential search task in which an agent must move a glimpse window, accumulate evidence, and decide when to predict. The benchmark isolates perceptual-state construction from broader embodied-control complications by using a 2D POMDP setup with controllable lookback constraints. The study evaluates ten multimodal models across several memory representations, including raw visual history, free-form textual states, metric grid maps, and consolidated visual canvases. Across experiments, the paper examines accuracy alongside exploration behavior, stopping decisions, coverage, and revisit patterns to diagnose where agentic perception fails.

Novelty

The paper's main novelty is a controlled benchmark that turns a standard recognition dataset into a partially observable world to study agentic perception rather than passive classification. It systematically varies memory representations and visual lookback horizons to separate evidence acquisition, perceptual-state construction, and state interpretation.

Results

Models that achieve high recognition accuracy on fully visible MNIST suffer substantial performance drops under partial observability, particularly on multi-digit sequences. Memory representation significantly affects behavior: structured metric grid maps reduce redundant revisits, while programmatically consolidating visual canvases markedly improves accuracy. An autonomous agentic harness improved Gemini-3.7-Flash substantially, but persistent procedural memory and correctness feedback provided no additional accuracy gains.

Key Points

1. MNIST-PRO isolates agentic perception by converting MNIST into a glimpse-based POMDP with explicit sensing budgets, stepwise exploration, and controllable visual lookback.
2. A large gap emerges between full-observability recognition and partially observable task success, demonstrating that seeing visual fragments over time does not automatically translate into a usable perceptual state.
3. State representation strongly influences performance: metric grid maps reduce spatial tracking drift, while consolidated visual canvases improve downstream prediction accuracy by making accumulated evidence easier to interpret.

References

• arXiv: https://arxiv.org/abs/2608.31022v1
• Fugu-MT: https://fugumt.com/fugumt/paper_check/2608.31022v1
• Hugging Face Papers: https://huggingface.co/papers/2608.31022