NeurIPS 2026 · Track on Evaluations and Datasets
EGOSTREAM
A Diagnostic Benchmark for Streaming Episodic Memory in Egocentric Vision
EgoStream is a streaming video benchmark for episodic memory: it measures what a multimodal LLM remembers from a first-person video stream, for how long, and when an answer stops being valid as the world changes. It ships with a unified framework to compare KV-cache memory-management policies under one protocol.
Egocentric video
Streaming only
Evidence grounded
7 memory dimensions
7 recall regimes
Benchmark
Streaming recall, measured carefully.
Each question is grounded in a specific evidence moment, assigned a memory category and asked again at several delays inside its Answer Validity Window. The result is a diagnostic profile of what a streaming model retains and when it forgets, not a single aggregate score.
Ego4D Episodic Memory VQA · 498
EgoLife · 433
EgoTempo · 425
Multi-Hop EgoQA · 578
HD-EPIC · 316
Benchmark streaming video LLMs
Online, query-agnostic ingestion with a bounded memory budget: the setting of wearable assistants and embodied agents, not offline video QA.
Compare KV-cache memory policies
Pruning, merging, adaptive head budgets, offloading and retrieval evaluated inside one cache implementation, same backbone, same prompts, same budget.
Measure episodic memory over time
Seven cognitive dimensions probed at seven delays, from instant recall to more than eight hours, with forgetting separated from scene change.
Build egocentric memory assistants
Where did I put the scissors, who was with me, did I close the door, what should I buy next: the questions a first-person assistant must answer.
Memory dimensions
What, where, when, how, who, why, and what next.
Questions are organised along seven dimensions of episodic memory, building on Tulving's account of episodic memory and on cognitive work on spatial, temporal, event, social, causal and prospective memory. Each question receives one primary label and optional secondary tags.
Detail
Spatial
Temporal
Event
Social
Causal
Prospective
| Category | Probe | Focus | Example query |
|---|---|---|---|
| Detail | What | object identity, attributes, counts | What did I put in the microwave? |
| Spatial | Where | location, placement, spatial relations | Where did I put the pair of scissors? |
| Temporal | When | order, sequence, time-indexed states | Where was the knife before I picked it up? |
| Event | How / Occurred | action gist, execution, state changes | Did I close the circuit breaker box? |
| Social | Who | people, interactions, co-presence | Who was with me when I ate? |
| Causal | Why | reasons, explanations for past events | Why didn't we buy the chestnut kernels? |
| Prospective | What next | future plans and intended actions | What should I buy to decorate my room? |
Answer Validity Window
Validity matters.
The AVW is the span, after the last evidence moment, during which the annotated answer stays factually correct. Query times are sampled only inside it, so a wrong answer means the model forgot, not that the world moved on. Every window was reviewed by humans; 1.96% needed adjustment.
Streaming protocol
Only what was seen so far.
Models process frames sequentially at 1 FPS and keep a compact internal memory. The question is revealed only at recall time, after the ingestion of the stream up to that timestamp, so memory management cannot peek at the query.
Recall regimes
A log-scaled view of memory, grounded in memory science.
Delays follow the classical paradigm of probing memory at several retention intervals, from Ebbinghaus' forgetting curve to consolidation timescales, so that the same question can test working memory and long-term retention.
Instant0s
Short0-5s
Short-mid5-30s
Mid30s-3m
Mid-long3m-30m
Long30m-8h
Ultra-long8h+
| Regime | Delay | Memory mechanism | Psychological basis |
|---|---|---|---|
| Instant | 0 s | Perceptual buffer | Encoding and immediate retrieval |
| Short-term | 0-5 s | Working memory | Active maintenance |
| Short-mid-term | 5-30 s | Working memory | Attention-dependent retention |
| Mid-term | 30 s-3 min | Transitional | Early synaptic consolidation |
| Mid-long-term | 3-30 min | Intermediate | Synaptic consolidation |
| Long-term | 30 min-8 h | Long-term | Late synaptic consolidation |
| Ultra-long-term | over 8 h | Long-term | Hippocampal-cortical transfer |
Results
Same accuracy, different memories.
Twenty-one memory-management policies were run on a frozen Qwen3-VL-8B under a 9,804-token active-cache budget (1 FPS, 896×512 frames, single A100), with the ranking confirmed on InternVL 3.5-8B and Molmo 2-8B.
Key findings
No in-cache policy exceeds about 45%.
A plain sliding window scores 29.53% semantic / 28.33% temporal accuracy; protecting the system prompt as sink tokens (StreamingVLM) lifts it to 44.74% / 39.05% at the same latency.
Adaptive head budgets (Ada-KV) give the best semantic average, 45.88%, with the best detail retention (57.04%); intra-frame cosine pruning reaches 45.63% with the best spatial retention (60.51%) at 2.28 s per frame.
Pruning beats merging on every backbone: discarding redundant tokens preserves fine-grained details and temporal structure, averaging blurs them (45.12% vs 41.87% with the pseudo-attention scorer).
Quantized offloading with retrieval (StreamingTOM) is the fastest policy, 1.43 s per frame, and rescues ultra-long-term recall (34.21%) at a small cost in average accuracy (42.86%).
Pre-prefill token compression (STC) reaches 47.18% semantic accuracy at 0.23 s per frame, above every in-cache policy; a text-only baseline scores 25.42%, chance level, so the questions cannot be solved from priors.
Budget matters with diminishing returns: 2.6K to 9.8K tokens raises semantic accuracy from 40.88% to 45.19%; 50.8K tokens peaks at 47.08% while nearly tripling processing time.
Positioning
How EgoStream compares with related benchmarks.
Existing egocentric and streaming VideoQA benchmarks cover parts of the problem. EgoStream is the first to combine an egocentric, online, evidence-grounded protocol with an explicit memory taxonomy, Answer Validity Windows and multiple recall regimes per question (paper Table 1).
| Benchmark | Egocentric | Online / streaming | Evidence grounding | Memory taxonomy | Answer Validity Window | Multiple recall regimes |
|---|---|---|---|---|---|---|
| MovieChat-1K | ✗ | ✗ | ✗ | △ | ✗ | ✗ |
| StreamingBench | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ |
| EgoSchema | ✓ | ✗ | ✓ | ✗ | ✗ | ✗ |
| MM-Ego | ✓ | ✗ | ✓ | △ | ✗ | ✗ |
| EgoTextVQA | ✓ | ✓ | ✓ | ✗ | ✗ | ✗ |
| EgoTempo | ✓ | △ | ✓ | ✗ | ✗ | ✗ |
| OVO-Bench | △ | ✓ | ✗ | △ | ✗ | ✗ |
| GroundVQA | ✓ | ✗ | ✓ | ✗ | ✗ | ✗ |
| Multi-Hop EgoQA | ✓ | ✗ | ✓ | ✗ | ✗ | ✗ |
| HD-EPIC | ✓ | ✗ | ✓ | ✗ | ✗ | ✗ |
| EgoLifeQA | ✓ | ✗ | ✓ | △ | ✗ | ✗ |
| EgoStream (ours) | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
Framework
One KV cache, every memory policy.
The released code isolates memory management as the only variable: the vision encoder, projector, decoder and prompts stay fixed while a plug-in memory manager operates on the mutable region of the cache. One JSON file describes each experiment of the paper.
Sliding window, sinks, pruning, merging
FIFO, attention sinks, protected recent frames, intra- and inter-frame pruning with cosine, pseudo-attention, value-norm and sink-guided scorers, token merging, composable pipelines.
Ada-KV, offloading, retrieval, STC
Adaptive per-head budgets, disk and int8 quantized offloading with ReKV-style retrieval at question time, StreamingTOM-like pipelines and STC pre-prefill compression.
Qwen3-VL, InternVL 3.5, Molmo 2
The backbone is chosen from the config; the same runner, protocol and budget apply to all three families.
Configs, evaluation, data scripts
One config per table row, an evaluation script that breaks accuracy down by category and regime, and scripts that rebuild the clips from the five source datasets.
git clone https://github.com/coai-iplab/EgoStream.git
cd EgoStream && pip install -r requirements.txt
python run_single_video.py --config configs/single_video/qwen3vl_8b_prune_intra_cs.jsonFAQ
Questions people ask about EgoStream.
What is EgoStream?
EgoStream is a diagnostic benchmark for streaming episodic memory in egocentric vision, published at NeurIPS 2026 (Track on Evaluations and Datasets). It contains 2,250 curated questions about first-person videos, organised in seven memory dimensions and expanded into 8,528 recall-conditioned evaluations, together with a unified framework that compares KV-cache memory-management policies in streaming multimodal LLMs.
Is EgoStream a streaming video benchmark?
Yes. Models must process the video sequentially and answer each question using only the frames observed up to the query time, while keeping a compact internal memory instead of an unbounded frame buffer. Ingestion is query-agnostic: the question is revealed only at recall time.
Which KV-cache memory-management methods does EgoStream evaluate?
Sliding windows (FIFO), attention sinks, protected recent-frame tokens, intra-frame and inter-frame token pruning (cosine redundancy, pseudo-attention, value norm, sink-guided attention), token merging, Ada-KV adaptive head budgets, disk offloading and int8 quantized offloading with retrieval, and pre-prefill token compression (STC). They reproduce the behaviour of StreamingVLM, StreamMem, InfiniPot-V, StreamingTOM, ReKV, CaM and Ada-KV inside one KV-cache implementation.
What is the Answer Validity Window (AVW)?
The AVW is the time span, after all the evidence for a question has been seen, during which the annotated answer remains factually valid as the scene keeps changing. Sampling query times inside the AVW lets EgoStream measure forgetting without confusing it with answers that became obsolete.
What are the recall regimes?
Seven log-spaced delays between the end of the evidence and the question: instant (0 s), short (0-5 s), short-mid (5-30 s), mid (30 s-3 min), mid-long (3-30 min), long (30 min-8 h) and ultra-long (over 8 h). They follow memory-consolidation timescales from cognitive science, from working memory to hippocampal-cortical transfer.
Which memory dimensions are tested?
Detail (what), spatial (where), temporal (when), event (how / did it occur), social (who), causal (why) and prospective (what next). The taxonomy follows the classic episodic-memory framework of Tulving and later work on spatial, temporal, event, social, causal and prospective memory.
Which source datasets are the questions built from?
Ego4D Episodic Memory VQA (498 questions), EgoLife (433), EgoTempo (425), Multi-Hop EgoQA (578) and HD-EPIC (316). Every question was validated by human annotators; videos stay with the original datasets and are never redistributed.
How does EgoStream differ from EgoSchema, StreamingBench or EgoLifeQA?
Those benchmarks cover at most some of the following: egocentric video, online streaming, evidence grounding and a memory taxonomy. EgoStream is the only one that combines all four with Answer Validity Windows and multiple recall regimes per question, so that it measures what is remembered and for how long rather than only aggregate accuracy.
What do current models score on EgoStream?
On Qwen3-VL-8B with a 9,804-token budget the best in-cache policies reach about 45% semantic accuracy (Ada-KV 45.88%, intra-frame cosine pruning 45.63%), against 29.53% for a plain sliding window and 25.42% for a text-only baseline; every in-cache policy runs slower than real time (more than 1 s per frame on an A100).
Is the data available?
The streaming framework, every run configuration of the paper and the data preparation scripts are released on GitHub. The question annotations will be published on this page; videos must be obtained from the five source datasets under their own licenses.
How do I cite EgoStream?
Cite the NeurIPS 2026 paper: Rosario Forte, Giuseppe Lando and Antonino Furnari, "EgoStream: A Diagnostic Benchmark for Streaming Episodic Memory in Egocentric Vision", Advances in Neural Information Processing Systems (NeurIPS), Track on Evaluations and Datasets, 2026. BibTeX is provided on this page.
Citation
Cite EgoStream.
Rosario Forte, Giuseppe Lando, Antonino Furnari. EgoStream: A Diagnostic Benchmark for Streaming Episodic Memory in Egocentric Vision. Advances in Neural Information Processing Systems (NeurIPS), Track on Evaluations and Datasets, 2026.
@inproceedings{forte2026egostream,
title = {{EgoStream}: A Diagnostic Benchmark for Streaming Episodic Memory in Egocentric Vision},
author = {Forte, Rosario and Lando, Giuseppe and Furnari, Antonino},
booktitle = {Advances in Neural Information Processing Systems (NeurIPS), Track on Evaluations and Datasets},
year = {2026},
url = {https://arxiv.org/abs/2605.31557}
}