MobileMem is a comprehensive benchmarking framework for evaluating on-device memory systems in realistic mobile environments.
MobileMem is built from multiple heterogeneous sources to enable comprehensive on-device memory modeling.
- 2025-08-01 — We publicly release the MobileMem dataset.
- 2025-06-01 — We launch the MobileMem project.
MobileMem offers two benchmark tracks. Choose the path that fits your needs and navigate to the corresponding resources.
For an interactive visualization of the MobileMem data, visit the Dataset Explorer branch.
The textual benchmark for evaluating memory systems on long-term, knowledge-intensive mobile agent trajectories.
| Section | Description | Quick Link |
|---|---|---|
| 📥 Data Access | Download the synthesized KEME trajectories and QA pairs from HuggingFace. | Link |
| ⚙️ How to Evaluate | Detailed evaluation guide for reproducing leaderboard results is available in the MemBase repository. | Link |
| 🛠️ Data Construction | Reproduce the KEME synthesis pipeline from scratch. | Link |
The multimodal benchmark for evaluating on-device memory with realistic mobile images and dialogues.
| Section | Description | Quick Link |
|---|---|---|
| 📥 Data Access | Download the MobileMem-Omni dataset, including images and dialogues. | Link |
| ⚙️ How to Evaluate | Evaluate your models on various memory task types using our provided scripts. | Link |
| 🛠️ Data Construction | Rebuild the entire MobileMem-Omni dataset with the provided pipeline. | Link |
The repository is organized into two main tracks, each containing everything you need for data access, evaluation, and construction.
MobileMem/
├── text/ # 📖 Text Track
│ ├── README.md # Track-specific guide and dataset download
│ ├── keme/ # 🛠️ KEME synthesis pipeline code
│ └── eval/ # ⚙️ Evaluation scripts for text track
├── omni/ # 🖼️ Omni Track
│ ├── README.md # Track-specific guide and dataset download
│ ├── src/ # 🛠️ Data construction pipeline code
│ ├── eval/ # ⚙️ Evaluation scripts for omni track
│ └── asset/ # 🎨 Images used in this README
└── README.md # This fileWe recommend using MemTrace to perform an in-depth error analysis. MemTrace helps you visualize and diagnose where and why your memory system fails, making it easier to pinpoint areas for improvement. For an example of how to use MemTrace, please refer to the tutorial in MemBase.
If this work or datasets is helpful, please kindly cite as this: