RAGCol: RAG-Based Automatic Video Colorization Through Text Caption Generation and Knowledge Enrichment

Automatic video colorization is a challenging task where multiple plausible colorizations can be deployed for any black-and-white film. For single photos, it is possible to have human knowledge guide the colorization process through text prompts, but for all of the frames and entities shown in a video, it becomes more difficult to achieve. With recent advances in automatic video colorization, natural language processing and knowledge enrichment, it is feasible to leverage external knowledge in automatic text-guided video colorization. To realize this possibility, we propose RAGCol, a knowledge-enriched video colorization system which adapts the retrieval augmented generation (RAG) framework to an automated colorization pipeline. We validated our RAGCol on the DAVIS [46] and Videvo [29] datasets. RAGCol demonstrated an average improvement of 9% over the previous state-of-the-art L-CAD [8] across the PSNR, SSIM, FID and FVD metrics. In a user study, we found that videos colorized by RAGCol were preferred by 74% on average over contemporary colorizers by human evaluators.