Daily Paper Briefing

Abstract Overview

VoiceMem is a memory architecture for real-time spoken interaction that separates memory into a factual "left brain" and an affective/persona-oriented "right brain," connected through streaming memory I/O. The left brain uses a schema-entity index with emergent clustering to keep retrieval dense under a small top-k budget, while the right brain models both persistent persona traits and entity-linked emotional attitudes across short and long horizons. The paper also describes a broader infrastructure around this design, including memory-aware speech-language-model training, the ChatMem-400K data pipeline, ChatMem-Bench for long-horizon audio evaluation, and a decoupled deployment stack with interchangeable memory backends. Overall, the work targets the combination of accuracy, emotional personalization, and low-latency retrieval needed for voice assistants.

Novelty

The paper's main novelty is a unified streaming dual-brain memory design for voice interaction that explicitly splits factual retrieval from affective/persona modeling, rather than treating memory as a single text-centric store. It is also distinctive in combining this architecture with a four-stage low-latency retrieval pipeline, a decoupled upper-layer routing/lower-layer engine design, and a new long-horizon audio benchmarking and training pipeline.

Results

Across text benchmarks, VoiceMem reports an average factual-memory score of 76.39 and a persona-memory score of 74.16, outperforming its Mem0 backend on factual memory and surpassing the strongest persona baseline by 1.89 points. On ChatMem-Bench, it achieves a 68.73 average and leads 11 of 14 categories, with especially large gains on paralinguistic and environmental audio-memory tasks. The system also reports 91.2 on LoCoMo with 430 memory tokens and 134 ms retrieval latency, and the upper-layer index improves multiple backends by 15.76 to 29.52 points on LoCoMo.

Key Points

1. VoiceMem separates factual memory and emotional/persona memory into parallel "left brain" and "right brain" components, with joint retrieval during real-time dialogue.
2. Its retrieval design emphasizes dense top-5 memory access through schema-entity routing, emergent clustering, and a four-stage streaming process that fits within voice-assistant latency constraints.
3. The paper supports the method with both system infrastructure and evaluation assets, including memory-aware SLM training, ChatMem-400K/ChatMem-Bench, and experiments showing gains across text, persona, and long-horizon audio-memory benchmarks.

References

• arXiv: https://arxiv.org/abs/2608.26005v1
• Fugu-MT: https://fugumt.com/fugumt/paper_check/2608.26005v1
• Hugging Face Papers: https://huggingface.co/papers/2608.26005
• Project: https://xzf-thu.github.io/VoiceMem/