Summary
This week's evidence emphasizes evaluating LLM agents beyond isolated answer accuracy, covering inference efficiency, robust tool use, and sustained multimodal reasoning. MOSAIC extends this focus to long-term conversational memory, separating failures in evidence storage, retrieval, delivery, and reading across text, images, video, and audio, while other new work diagnoses where agent failures originate.
Situation
The representative papers expose complementary gaps in evaluating capable, practical agents. M1 links reasoning performance to inference cost: it reports roughly threefold higher throughput for hybrid Mamba reasoning models than similarly sized transformers at large batch sizes, which can be converted into more test-time sampling. Thinking with Programming Vision shows that simple rotations and flips can sharply degrade state-of-the-art multimodal models, and motivates evaluation of code-mediated, multi-turn tool composition and error recovery. AMUSE targets another underexplored dimension, speaker identity, temporal consistency, and cross-turn reasoning, through zero-shot, guided, and agentic evaluation modes, finding that current multimodal models perform poorly in multi-speaker settings.
Current-week evidence from MOSAIC (2610.29249v1) extends these concerns to persistent conversational agents, with 923 questions whose evidence spans text, image, video, and audio across 11–24 sessions. Its matched diagnostic protocol separates write, retrieve, deliver, and read stages, distinguishing information lost during storage from retrieval misses, media-packing losses, and failures to integrate complete evidence. Together, these studies point toward evaluating not only whether an agent succeeds, but how its resource use, tool interactions, and handling of multimodal evidence account for success or failure.
Infographic (English)
Progress
Muse Glimmer 30B: An On-Device Agentic Model <See Details on Fugu-MT>
Muse Glimmer 30B is a distilled, open-weight model built for multi-step reasoning, tool use, and failure recovery on consumer hardware. It shifts agentic capability toward local deployment, using 4-bit quantization and speculative decoding to reach usable speeds on edge devices.
OpenVLHarness: Real World Visual Agentic Tool-Use with Multimodal Memory <See Details on Fugu-MT>
OpenVLHarness combines task-level tool interfaces and evidence rendering with persistent multimodal memory for visual agents. Across 22 benchmarks and eight backbones it improves average performance over base models by 9.7–12.6 points, suggesting evidence acquisition complements extra reasoning effort.
Steering Black-Box Large Language Model Agents with Pitfall Anticipation World Model <See Details on Fugu-MT>
PAWM extends agent reliability from recovering after errors to anticipating imminent pitfalls in black-box agents. Selective intervention improves Claude Sonnet 4.5 by 24.3% on ScienceWorld and 7.7% on WebArena-Lite.
MOSAIC: An Omni-Modal Benchmark for Long-Term Memory and Cross-Modal Reasoning in Conversational Agents <See Details on Fugu-MT>
MOSAIC evaluates long-term omni-modal memory with 923 questions over 11–24-session histories. Its stage-wise decomposition shows modality-dependent representation losses and substantial residual reading errors even under gold evidence.
A Mechanistic Study of Automated Failure Attribution in LLM-based Agentic Systems <See Details on Fugu-MT>
A mechanistic study finds that LLMs internally encode more failure-attribution information than they express when judging agent trajectories. Grafting, a simple manual intervention, routes this latent information into outputs and improves attribution over direct prompting.
Diagnostic Analysis for Large Language Model Reasoning <See Details on Fugu-MT>
A two-stage diagnostic evaluation tests whether reasoning structures remain reliable in open-ended multi-hop question answering. Generated-edge experiments reveal sensitivity to upstream graph errors that gold-graph settings hide.
Outlook
Outlook Summary
Multimodal agent evaluation is likely to become broader and more diagnostic, testing where evidence is lost or misunderstood rather than only whether answers are correct. CodeVision’s proposed expansion and OpenVLHarness findings support testing more varied tools and reusable memory, rewarding tool use when it helps. AMUSE’s reflective optimization could be tested against MOSAIC’s stage-specific failures to separate better retrieval from better evidence integration. Efficiency remains a parallel priority: M1 proposes improved kernels, attention integration, and faster long-sequence generation. On-device tests should measure reliability alongside memory use and throughput, while checking whether interventions repair the responsible failure step.
Infographic (English)
Three-Year Movement
Over three years, this path could turn diagnostic evaluation into bounded checks that determine whether updates to multimodal assistants are accepted. The first year would reproduce the weekly findings and test where evidence is lost or misused, rather than treating better retrieval as a complete solution. The mechanism comes from reliability engineering: introduce a controlled fault, replay the task, and check whether a targeted repair restores performance. Researchers would preserve evidence histories and compare targeted repairs with equally costly extra reasoning or changes that do not address the suspected fault.
Before practical use expands, repairs would need independently checked benefits across different tasks and models, including natural failures not used to develop them. These gains must fit stated memory and response-time limits, because faster processing of large batches does not establish faster responses for individual users. In the second year, successful repairs could become a reusable library, with each confirmed failure adding a regression test that later updates must pass. Pilot assistants could then request missing evidence or limited verification when measured benefits justify the cost, while uncertain cases go to human review. Teams would retain tested system snapshots so an unsuccessful update could be reversed without rebuilding the workflow.
By the third year, independent evaluators could challenge these checks with hidden natural failures and interactive attacks across different testing systems. A useful monitoring cue would be rejecting an update that raises average accuracy but repeatedly weakens evidence handling or speaker continuity. Rotating hidden tests and human audits would help reveal whether repair benefits survive beyond familiar examples after all processing overhead is counted. If natural and deliberately introduced failures require different repairs, the method should remain experimental rather than control broader update decisions. Even successful checks would support only specific workflows, since unpredictable generation can make replays diverge and cannot prove that an agent reasons correctly.
Over three years, this path could make selective diagnosis a practical alternative when testing every agent stage and configuration becomes too costly. Its starting point is measured expense, not an assumption that longer evaluations are always unaffordable or that quick checks are sufficient. During the first year, researchers would compare inexpensive warning tests with detailed stage-level diagnosis, measuring missed failures as carefully as false alarms. Flagged cases would receive deeper analysis, while audits of apparently successful cases would test whether quiet failures escape the initial screen.
The mechanism resembles health surveillance: cheap initial checks direct expensive confirmation, while shared, closely monitored testing systems track difficult problems in depth. Separate consumer-device tests would remain necessary because rapid large-batch processing does not show how quickly an assistant responds to one user. In the second year, expansion would depend on reducing total evaluation cost while preserving failure recall, meaning the share of actual failures detected. Confirmed failures would improve the screens, and any released testing capacity could support longer conversations and more complex visual tool use. Prototype teams could then run lightweight checks after changes and reserve deeper audits for releases, without treating a passing screen as proof of reliability.
By the third year, participating teams could routinely disclose sampling rates and detection limits, alongside schedules for independent audits of their evaluation process. These records could guide which workflows have enough evidence for supervised pilots, while sensitive settings retain full runs of designated adversarial tests. The central monitoring cue is lower total cost with stable detection of previously unseen failures, rather than simply a declining number of alerts. Repeatedly missed attacks or unfamiliar tool failures would require broader testing in the affected area, especially when one defect spreads across many tasks. If inexpensive screens fail to generalize, or comprehensive replay becomes cheap enough for routine use, this selective-testing pathway would lose its main justification.
Over three years, this path could grow from checking archive migrations into a bounded process for approving changes to internal assistants. The first year would reproduce stage-level evidence tests on permissioned archives, asking whether a replacement pipeline preserves usable information rather than merely retaining more data. Its mechanism resembles measuring successive sections of a water network: isolate a suspected loss, repair that section, and check whether useful service returns. Researchers would supply verified evidence at different processing stages, while human reviewers check whether the diagnosis identifies a fault whose repair improves answers.
A read-only testing component could compare old and replacement pipelines without interrupting the existing service, keeping early application separate from autonomous action. Targeted repairs would have to beat simpler upgrades under matched resource limits, with evaluation and correction together costing less than the existing acceptance process. Reusing the probes during a second migration would show whether the approach can become repeatable rather than remain a custom engineering project. In the second year, successful reuse could support recurring checks after pipeline changes, while researchers test whether diagnoses transfer across formats and models. Independent reviewers would challenge those checks with misleading evidence and prompt injections, and acceptance requirements could specify grounded answers within agreed resource limits.
By the third year, accumulated failure cases could support selective repairs, such as reprocessing a transcript segment, instead of rebuilding an entire archive. Tests in organizations not involved in development would check whether these benefits persist, while unresolved understanding failures or risky actions still require escalation. A useful monitoring cue would be reused tests across successive migrations, supported by retained source-to-answer traces and requests for independent acceptance reports. Each confirmed repair could add a regression case, making later checks cheaper and allowing archive owners to require evidence-path reports before approving changes. However, information is not a conserved fluid: complete evidence can still be misunderstood, and unstable diagnoses or cheaper successful alternatives would limit this approach.
1-Year / 3-Year Research-Application Infographic
References
- M1: Towards Scalable Test-Time Compute with Mamba Reasoning Models - Authors: Junxiong Wang, Wen-Ding Li, Daniele Paliotta, Daniel Ritter, Alexander M. Rush, Tri Dao / <See Details on Fugu-MT> / License: CC-BY-4.0
- Thinking with Programming Vision: Towards a Unified View for Thinking with Images - Authors: Zirun Guo, Minjie Hong, Feng Zhang, Kai Jia, Tao Jin / <See Details on Fugu-MT> / License: CC-BY-4.0
- AMUSE: Audio-Visual Benchmark and Alignment Framework for Agentic Multi-Speaker Understanding - Authors: Sanjoy Chowdhury, Karren D. Yang, Xudong Liu, Fartash Faghri, Pavan Kumar Anasosalu Vasu, Oncel Tuzel, Dinesh Manocha, Chun-Liang Li, Raviteja Vemulapalli / <See Details on Fugu-MT> / License: CC-BY-4.0
- Muse Glimmer 30B: An On-Device Agentic Model - Authors: Arash Ebrahim Abtahizadeh, Muna Aghamelu, Adil Ahmed, Juan C. Angeles Acuna, Alexis Audran-Reiss, Asaf Avrahamy, Yoram Bachrach, Can Balioglu, Amar Budhiraja, Onur Çelebi, Gaurav Chaurasia, Joy Chen, Lynn Cherif, Lucia Cipolina-Kun, Gal Cohen, Timothée Darcet, Beto de Paola, Digant Desai, Sam Devlin, Derek Dunfield, Ivan Evtimov, Maryam Fazel-Zarandi, Rob Fergus, Jean-Christophe Gagnon-Audet, Grant Gardner, Bhavul Gauri, Paris Giampouras, Sujan Kumar Gonugondla, Antoine Grosnit, Young Han, Michael Hassid, Monuril Islam, Masoud Jalili Sabet, Songhao Jia, Narine Kokhlikyan, Svetlana Kurtsman, Sandra Lefdal, Chee Hau Leow, Noam Levi, Mike Lewis, Sanae Lotfi, Xiyang Luo, Despoina Magka, Saeed Mahloujifar, Neal Mangaokar, Ishita Mediratta, Alexander H. Miller, Hela Momand, Saba Nazir, Kelvin Niu, Mattia Opper, Parth Pathak, Alberto Pepe, Danielle Pintz, Mikhail Plekhanov, Sidhartha Reddy Potu, Anton Protopopov, Shengyi Qian, Shashank Rajput, Joseph Rocca, Scott Roy, Luca Sbordone, Amin Shakeri, Tatiana Shavrina, Michael Shvartsman, Ruan Silva, Anya Sims, Eric Michael Smith, Hugo Touvron, Mabel Ubong, Isabel Urrego, Yanzheng Xiang, Zhiyuan Yao, Alex Yarmula, He Ye, Jade Yu, Luke Zettlemoyer, Arman Zharmagambetov, Perpetua Fathima Ahlijah, Tremo Ahmed, Enrique Alfonseca, Pedram Bashiri, Nikolay Bashlykov, Aaron Berman, Rohith Bhatia, Siddharth Bhattacharya, Prashant Bhende, Chloe Bi, Shuchao Bi, Clement Anthonioz Blanc, Allen Bolourchi, Andrea Caciolai, Erica Cai, Xiang Tony Cao, Yue Cao, Andrew Caples, Jinzhao Chang, Lele Chen, Moya Chen, Weijie Chen, Wenhu Chen, Xiaobai Chen, Yinyin Chen, Zhengxing Chen, Wei Cheng, Pavel Chikulaev, Edward Chou, Sy Choudhury, Brian Clerkin, Marta R. Costa-jussà, Curtis Davis, Myra Deng, Dujian Ding, Hantian Ding, Shuangrui Ding, Jesse Dodge, Shweta Doshi, Shirin Eghtesadi, Faith Eischen, Athyu Eleti, Alex Fabbri, Han Fang, Paula Farani, Aiman Farooq, Christoph Feichtenhofer, Darius Feher, Amanda Felix, Carl Chengyan Fu, Sean Gahagan, Yifan Gao, Itai Gat, Akashdeep Goel, Ahuva Goldstand, Bo He, David Heineman, Dylan Herman, Irene Hjelt, Joonas Hjelt, Alexander Holz, Rui Hou, Annie Hu, Yuan-Ting Hu, Bernie Huang, Jing Huang, Kun Huang, Pere-Lluís Huguet Cabot, Ashley Hunter, Srinivasan Iyer, Somya Jain, Michael Jiang, Song Jiang, Wenwen Jiang, Lin Jin, Jacob Kahn, Melanie Kambadur, Yingxin Kang, Justine Kao, Stephane Kasriel, Hossein Kavianihamedani, Ameya Khandekar, Yash Khare, Gefen Kohavi, Kavya Kopparapu, Praveen Krishnan, Venu Kuchibhotla, Ananya Kumar, Leona Lan, Jack Lanchantin, Hunter Lang, Nikolay Pavlovich Laptev, Yuanzhi Li, Yunlu Li, Asad Liaqat, Johnson Lin, Runwei Lin, Zhichao Lin, Shaun Lindsay, Yaron Lipman, Frederick Liu, Mengchen Liu, Qiang Liu, Yinghai Lu, Yujie Lu, Christopher Luc, Faye Ma, Pingchuan Ma, Tengyu Ma, Wufei Ma, Xiaodong Ma, Damir Maksyutov, Josh Malone, Udeep Manchanda, Thomas Mann, Kalyani Marathe, Roberto-Rafael Maura-Rivero, Ruslan Mavlyutov, Thomas Mesnard, Julian Michael, Sagar Miglani, Todor Mihaylov, Naureen Mitha, Archi Mitra, Leo Mok, Naila Murray, Muthukumaravel Muthuraman, Mergen Nachin, Nese Demir Nalci, Saaketh Narayan, Alyssa Newcomb, Peter Ney, Joycelyn Ng, Devin Norder, Maxwell Nye, Rajvardhan Oak, Cody Ohlsen, Artidoro Pagnoni, Yi Pan, Nilesh Pandey, Guan Pang, Richard Yuanzhe Pang, Mahesh Pasupuleti, Parth Patel, Yash Patel, Vladan Petrovic, Norman Ponte, Yundi Qian, Zengyi Qin, Jack Rae, Emmanuel Ramirez, Ramya Raghavendra, Nikhila Ravi, Hongyu Ren, Riandy Riandy, Harrison Rudolph, Kalyan Saladi, Eduardo Sánchez, Karthik Abinav Sankararaman, Aman Shankar, Paresh Shenoy, Alana Shine, Hamid Shojanazeri, Iva Simon-Bubalo, Mitesh Kumar Singh, Tiffany Siqi, Pawan Sisodiya, Ali Sol, Sten Sootla, Leslie St. Pierre, DiJia Su, Suraj Subramanian, Brian Sun, Tony Sun, Zhiqing Sun, Natacha Supper, Jason Talpalar, Ellen Xiaoqing Tan, Liang Tan, Jean-Christophe Testud, Parth Thakkar, Joseph Tighe, Ty Toledano, T Tse, Igor Tufanov, Alex Vaughan, Josh Walters, Alex Wang, Bethany Wang, Meng Wang, Rocky Wang, Weiyue Wang, Xiaolan Wang, Xin Wang, Zijian Wang, Bram Wasti, Quentin Wellens, Yeming Wen, Mary Williamson, Yuncheng Wu, Fei Xia, Tian Xie, David Xu, Jacob Xu, Puxin Xu, Zheng Xu, Shen Yan, Megan Yang, Yining Yang, Nick Yannopoulos, Kevin Yao, Michihiro Yasunaga, Haoxuan You, Michael You, Dingli Yu, Tianhe Yu, Summer Yue, Xiang Yue, Shuwei Zeng, Hanwen Zha, Chen Zhang, Hongming Zhang, Kaiyu Zhang, Meng Zhang, Mingyang Zhang, Yi Zhang, Shengjia Zhao, Chao Zhou, Claire Zhou, Shuyan Zhou, Yanjun Zhou, Yaxuan Zhou, Rosalie Zhu, Yun Zhu / <See Details on Fugu-MT> / License: CC BY 4.0
- OpenVLHarness: Real World Visual Agentic Tool-Use with Multimodal Memory - Authors: Yu Zhou, Cheng-Fu Yang, Rui Sun, Di Wu, Sibo Peng, Mingyang Zhang, Zhen Yang, Ruiyi Zhang, Nanyun Peng, Zhe Gan, Kai-Wei Chang / <See Details on Fugu-MT> / License: CC BY 4.0
- A Mechanistic Study of Automated Failure Attribution in LLM-based Agentic Systems - Authors: Wendy Zheng, Liang Wu, Yinhan He, Mayank Darbari, Jundong Li, Liangjie Hong / <See Details on Fugu-MT> / License: CC BY 4.0
- MOSAIC: An Omni-Modal Benchmark for Long-Term Memory and Cross-Modal Reasoning in Conversational Agents - Authors: Hong Ting Tsang, Boyi Zhang, Haoyu Huang, Jiaxin Bai, Yufei Li, Yisen Gao, Zhongwei Xie, Hongyu Luo, Jeff Z. Pan, Yangqiu Song / <See Details on Fugu-MT> / License: CC BY 4.0