Descriptions:
Feed it 67 videos from the 2022 World Cup and ask for the near misses, the shots that almost scored but did not, each with a reason, and it returns them. Ask it to track Messi across the entire corpus and describe the camera framing, and it finds the moment he slaloms past a sliding defender. James Le’s point is that this is unusual because video has no memory. Almost every video AI system answers each query from scratch, and a bigger context window does not fix it, because the real problem is that there is no durable representation to retrieve into.
His fix is to stop treating video as a bag of frames and start treating it as a spatial temporal volume, then build a memory layer over it: a context graph of time bounded moments, the entities and where they appear, the relationships between them, and corpus level themes. At TwelveLabs that is an embedding encoder, a context store, and a video language model exposed as an API. The design rules are blunt: ingest once and reason many times, store primitives not answers, ground every claim to a timestamp, and let intent decide what to remember, because brand safety and sports highlights need different things from the same footage.
Speaker info:
– https://x.com/le_james94
– https://www.linkedin.com/in/khanhnamle94/
– https://jameskle.com/
Timestamps:
0:00 – Video has no memory
0:50 – Video is a spatial temporal volume, not a bag of frames
2:06 – Three problems: wrong context, wrong memory, weak reasoning
3:36 – Five properties that make video memory hard
4:53 – The TwelveLabs stack: Marengo, the context store, and Pegasus
5:56 – Search versus memory
7:48 – The context graph
9:04 – Five design principles for a video memory layer
10:45 – From a static model to a video worker
12:51 – Demo: sports and tracking Messi across the World Cup
15:37 – Demos: traffic security and ad placement
