An open memory module from Stanford University compresses entire collections into 16 tokens injected into a frozen LLM. Gains are seen on FactKG and other test sets, with unfavorable cases like PubMedQA. Code and checkpoints are published under Apache 2.0, but execution is aimed at high-end GPUs and numbers are self-reported.
Heterogeneous results and self-reported figures
Memorilla ranks behind on three benchmarks. On PubMedQA, the module reaches 59.00% when LLMLingua-2 displays 69.00%. This result is also one of the lowest reported, with 59.00% reported compared to 69.00% for LLMLingua-2. The team explains that PubMedQA summaries only total 324 tokens, leaving little room for compression. MemoRAG maintains the advantage over PersonalizationV4 (67.88% against 66.65%) and over NarrativeQA in ROUGE-L (20.51 against 19.31), however mobilizing approximately 690 times and 50 times more document tokens. All these scores and comparisons are provided by the research team, which specifies having evaluated the baselines on the same frozen Qwen3-8B.
Code released under Apache 2.0 and available resources
The project is accessible under the Apache 2.0 license, with code, training recipes and checkpoints posted online, including checkpoints of 194 million parameters hosted on Hugging Face. Installation is done via uv under Linux before loading with MemoryVLLM.from_pretrained, a first launch downloading approximately 25 GB. PersonalizationV4, a synthetic corpus of 149 users and 200 conversations each, is published under CC BY 4.0 license.
What changes latent memory limited to 16 tokens
Memorilla searches a complete document storage and delivers to the frozen LLM a compact summary of 16 tokens, a fixed size regardless of the collection. By transforming the collection into 16 vectors, the goal is to replace the inclusion of entire documents in context for a frozen Qwen3-8B. On FactKG, they report 84.33% compared to 80.43% for the full context, with 108 times fewer document tokens. On an H100, the system responds in 0.74 s with 512 documents (around 128K tokens), while the complete context stretches from 0.75 s to 19.9 s, or 27 times slower in this case; the memory module itself takes about 2ms.
The three-step pipeline and training
The treatment follows three phases. First, a frozen Qwen3-Embedding-4B encodes each document once, without re-encoding old ones when appending. Subsequently, sixteen learned slots connect to the question and memory via two layers of eight-headed cross-attention. In the next step, these slots are transformed into the input embedding space of Qwen3-8B and added as a prefix to the query. Only the memory module is trained, comprising 194,123,776 parameters linked to the 8B decoder. The training is organized in three phases: approximately 1.98 million examples of paraphrase reconstruction from Wikipedia, followed by approximately 1.34 million examples taken from ten sets of questions and answers and summaries, then refinement carried out on eight benchmarks. The whole recipe takes around 19 hours on four H100 GPUs.
Deciphered AI dev tools, agents and new features, every evening in 5 minutes. Free.
Included upon registration: our selection of the best AI guides & comparisons.
Choose your pace
Free · No spam · Unsubscribe in 1 click
Quantified results: averages, winners and losers depending on the case
Beyond FactKG, Memorilla achieves 44.60% on PersonaMem-v2 versus 30.90% for xRAG and, in another reported comparison, 44.60% versus 27.92% for the full context. On TriviaQA, the exact match score of 57.00 slightly exceeds MemoRAG at 56.60. On LaMP-4 and LaMP-7, 18.15 and 48.37 in RED-L are noted, the best scores. On average over eight benchmarks, Memorilla displays 49.68 compared to 41.92 for xRAG, 40.75 for LLMLingua-2 and 38.02 for Mem0. The comparative tables indicate a geometric average consumption per request of 16 document tokens for Memorilla, 78 for MemoRAG, 42 for Mem0 and 121 for xRAG. The accuracies on FactKG break down to 84.33, 80.38, 72.79 and 78.98 respectively; on PersonaMem-v2 at 44.60, 28.76, 27.48 and 30.90; on PubMedQA at 59.00, 67.00, 24.00 and 36.00; and on NarrativeQA in RED-L at 19.31, 20.51, 13.80 and 16.92. Compression peaks when 16 tokens replace 67,385 document tokens on NarrativeQA (reported reduction of 4,212x). In the agent context, eight memory tokens increase the average success of a Qwen3-4B TextWorld agent from 18.6% to 24.2%.
Runtime, hardware constraints and acceleration options
The system targets Linux, CUDA and Python 3.11 compatible GPUs. Quickstart requires an 80GB GPU, with recipes conducted on an 80GB H100, and no quantified variants are provided. In execution, a latency of 0.74 s is noted with 512 documents on an H100, when the complete context goes up to 19.9 s. An optional memory cache pre-computes attention keys and values: on a B200, a query takes 8.1 ms at 1M documents and 33 ms at 4M, and goes down to 5.0 ms at 1M with torch.compile; the cache requires 80 GiB to 4M documents. On the integration side, the installation is done via uv and the checkpoints are loaded with MemoryVLLM.from_pretrained; each checkpoint must be associated with the corresponding decoder, and the custom data comes in parquet files with collection_id, question, answer and documents.

