HybridContextQA: A Hybrid Approach for Complex Question Answering using Knowledge Graph Construction and Context Retrieval with LLMs

Augmenting domain-specific knowledge with Large Language Models (LLMs) to answer complex conditional questions is an important area of research. LLMs are good at answering general domain questions, however, their performance decreases when applied to a specific domain with complex conditional questions. We hypothesize that extracting context from relevant documents and Knowledge Graphs (KGs), and then feeding this combined knowledge to the LLM prompts, can provide better context to answer the complex conditional questions. To test our hypothesis, we propose a hybrid approach called Hybrid Context for Complex Question-Answering (HybridContextQA) that can extract relevant context from documents as well as from a KG. To implement this, we create a Retrieval-Augmented Generation (RAG)-based hybrid context retrieval pipeline. This pipeline creates a KG from the provided documents and stores it in a Neo4j graph store. An LLM is used to automatically create a KG from the provided documents. The pipeline also stores the context extracted from the documents in vector form in a vector database. This combined context from KG and vector store can then be used for answering the complex conditional questions of that domain using an LLM. We perform our experiments on a complex question-answering (QA) dataset called ConditionalQA. This dataset contains complex questions with conditional answers. We also compare the proposed approach with other approaches such as Code Prompt, Text Prompt, and Think-on-Graph. We find that the HybridContextQA approach performs better than the existing approaches for multiple LLMs, including Mistral and Mixtral. We also conduct comprehensive experiments to analyze the contribution of the context from KG and vector form. We release the code implementing the HybridContextQA approach and the end-to-end pipeline with LLM prompts.