source: arxiv machine learning: multimodal colrag-tf: triple-filtered retrieval for complex pdfs
level: research
multimodal colrag-tf is a retrieval system designed for complex pdf documents that mix text, images, and tables. it uses four types of signals: dense text embeddings, bm25 keyword matching, knowledge graph triples, and image similarity. the system was tested on 43 japanese disaster lesson pdfs, creating an index of 2,403 content blocks. a hybrid ocr pipeline and llm-based caption generation help handle scanned or image-heavy pages.
the system extracts 11,414 openie triples from the documents and indexes them with faiss for fast lookup. this triple filtering helps with multi-hop reasoning by connecting related facts across different parts of the documents. a coarse-to-fine retrieval approach narrows the search from volume to chapter to block before final scoring. the fusion of all four signals aims to improve retrieval accuracy for questions that require combining information from multiple sources.
early experiments show that combining these different retrieval methods can outperform single-mode approaches. the knowledge graph component is especially useful for questions that need reasoning across document boundaries. the system is built to handle domain-specific terminology and multimodal content that often trips up standard rag pipelines. the work focuses on japanese disaster reports but the architecture could apply to other technical document collections.
why it matters: better retrieval over messy pdfs can improve ai assistants that need to answer questions from real-world document collections, such as technical manuals or research archives.
source: arxiv machine learning: multimodal colrag-tf: triple-filtered retrieval for complex pdfs