Exquisitor at the Lifelog Search Challenge 2024 : Blending Conversational Search with User Relevance Feedback

Abstract

The past decade has seen a rapid expansion of personal and interpersonal multimedia collections. These collections offer a wealth of information about individuals, including their interests, health, and significant life events. While automated techniques can assist in structuring and organizing these collections, they often have limitations in helping users effectively navigate and find relevant items within such large datasets. The Lifelog Search Challenge (LSC) provides a valuable benchmark for evaluating interactive retrieval systems designed for personal multimedia collections. Exquisitor utilizes a large-scale user relevance feedback (URF) approach for searching through large collections. To address challenges in highly descriptive retrieval tasks where the relevance feedback model may fail to identify essential elements, we have enhanced Exquisitor with conversational search capabilities powered by a Vision Language Model (VLM) and refined the features underlying the URF model. Furthermore, Exquisitor has been updated with a streamlined user interface that enables seamless switching between conversational search and URF modes.