Embodied-RAG: General Non-parametric Embodied Memory for Retrieval and Generation

project page: https://quanting-xie.github.io/Embodied-RAG-web/

Embodied-RAG introduces a memory system for embodied agents in large environments. It builds a topological map with ego-centric images, linking each node to captions generated by a VLM (GPT4o). Using agglomerative clustering, it creates a “semantic forest” to capture different levels of semantic detail. For retrieval, it selects chains of nodes closest to the query, aiding navigation and explanations. This method tackles challenges like sparse memory, continuous space, and varying semantic granularity in indoor and outdoor settings. 

(they provide more demos on the project page)