Skip to content

Latest commit

 

History

History
175 lines (165 loc) · 23 KB

Knowledge Graph Question Answering (KGQA).md

File metadata and controls

175 lines (165 loc) · 23 KB

Knowledge Graph Question Answering (KGQA)

Note: This task is highly related with Entity Linking and Disambiguation as well as Relation Linking and Disambiguation.

📝 Survey and Summary

  1. Core techniques of question answering systems over knowledge bases: a survey (Knowledge and Information Systems 2017) [Paper]
  2. A Survey on Complex Question Answering over Knowledge Base: Recent Advances and Challenges (2020) [Paper]
  3. Question Answering Summary (not limited to KBQA) [GitHub]
  4. Introduction to Neural Network based Approaches for Question Answering over Knowledge Graphs (2019) [Paper]
  5. Awesome KGQA [GitHub]
  6. What is in the KGQA Benchmark Datasets? Survey on Challenges in Datasets for Question Answering on Knowledge Graphs (Journal on Data Semantics, 2021)[Paper]
  7. Complex Knowledge Base Question Answering: A Survey [Paper]
  8. Knowledge Graphs & LLMs: Multi-Hop Question Answering [Neo4j Developer Blog] [Another Similar Discussion]
  9. Multilingual Question Answering Systems for Knowledge Graphs—A Survey (Semantic Web 2023) [Paper]

📝 Papers

General KGQA

  1. Question Answering Over Knowledge Graphs: Question Understanding Via Template Decomposition (VLDB 2018) [Paper]🌟
  2. KBQA: Learning Question Answering over QA Corpora and Knowledge Bases
  3. SPARQA: Skeleton-based Semantic Parsing for Complex Questions over Knowledge Bases [Paper]
  4. Improving Multi-hop Question Answering over Knowledge Graphs using Knowledge Base Embeddings (ACL 2020) [Paper]

Efficiency issue.

  1. AskNow: A Framework for Natural Language Query Formalization in SPARQL (ESWC 2016)
  2. Answering Natural Language Questions by Subgraph Matching over Knowledge Graphs (TKDE 2018) [Paper]
  3. Natural language question answering over RDF: a graph data driven approach (SIGMOD 2014) 🌟
  4. Complex Factoid Question Answering with a Free-Text Knowledge Graph (WWW 2020)
  5. Automated template generation for question answering over knowledge graphs （WWW 2017） [Paper]
  6. Never-Ending Learning for Open-Domain Question Answering over Knowledge Bases (WWW 2018) [Paper]
  7. Learning to Retrieve Reasoning Paths over Wikipedia Graph for Question Answering (ICLR 2020)
  8. An Interpretable Reasoning Network for Multi-Relation Question Answering (COLING 2018)
  9. Pattern-revising Enhanced Simple Question Answering over Knowledge Bases (COLING 2018)
  10. TEQUILA: Temporal Question Answering over Knowledge Bases (CIKM 2018)
  11. FreebaseQA: A New Factoid QA Data Set Matching Trivia-Style Question-Answer Pairs with Freebase (NAACL 2019) [Paper]
  12. Knowledge Graph Embedding Based Question Answering (WSDM 2019) [Paper] [Code]
  13. Graph-Based Reasoning over Heterogeneous External Knowledge for Commonsense Question Answering (AAAI 2020) [Paper]
  14. Open Question Answering Over Curated and Extracted Knowledge Bases (KDD 2014) [Paper]
  15. RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering (2021) [Paper]
  16. Natural language question/answering let users talk with the knowledge graph (CIKM 2017)
  17. Asking Clarification Questions in Knowledge-Based Question Answering (EMNLP 2019)
  18. What's Missing: A Knowledge Gap Guided Approach for Multi-hop Question Answering (EMNLP 2019)
  19. KagNet: Learning to Answer Commonsense Questions with Knowledge-Aware Graph Networks (EMNLP 2019)
  20. Message Passing for Complex Question Answering over Knowledge Graphs (CIKM 2019) [Paper]
  21. Keyword Search on RDF Graphs — A Query Graph Assembly Approach (CIKM 2017) [Paper] 🌟
  22. Semantic Guided and Response Times Bounded Top-k Similarity Search over Knowledge Graphs (ICDE 2020) [Paper] 🌟
  23. Beyond I.I.D.: Three Levels of Generalization for Question Answering on Knowledge Bases (WWW 2021) [Paper]
  24. Query Graph Generation for Answering Multi-hop Complex Questions from Knowledge Bases (ACL 2020) [Paper] [Code]
  25. CBench: Towards Better Evaluation of Question Answering Over Knowledge Graphs [Paper] (VLDB 2021) 🌟
  26. SAKE: Spatial Question Answering over Knowledge Graph Based on Embedding Techniques (ICDE 2021) 🌟
  27. BeamQA: Multi-hop Knowledge Graph Question Answering with Sequence-to-Sequence Prediction and Beam Search (SIGIR 2023) [Paper]🌟
  28. Joint Knowledge Graph Completion and Question Answering (KDD 2022) [Paper]
  29. Would You Ask it that Way?: Measuring and Improving Question Naturalness for Knowledge Graph Question Answering (SIGIR 2022) [Paper]
  30. Sequence-to-Sequence Knowledge Graph Completion and Question Answering (ACL 2022) [Paper]
  31. Knowledge Graph Question Answering with Ambiguous Query (WWW 2023) [Paper]
  32. Efficient and Accurate Contextual Re-Ranking for Knowledge Graph Question Answering (LREC-COLING 2024) [Paper]
  33. STaR: Space and Time-aware Statistic Query Answering (CIKM 2024) [Paper]
  34. DiaKoP: Dialogue-based Knowledge-oriented Programming for Neural-symbolic Knowledge Base Question Answering (CIKM 2024) [Paper]
  35. FZI-WIM at AVeriTeC Shared Task: Real-World Fact-Checking with Question Answering (EMNLP 2024) [Paper] - Related to knowledge store instead of knowledge graph.
  36. Enhancing Complex Question Answering over Knowledge Graphs through Evidence Pattern Retrieval (SIGIR 2024) [Paper]
  37. NuTrea: neural tree search for context-guided multi-hop KGQA (NeurIPS 2023) [Paper]
  38. Uniqorn: Unified question answering over RDF knowledge graphs and natural language text (Journal of Web Semantics, 2024) [Paper]
  39. Reasoning with Trees: Faithful Question Answering over Knowledge Graph (COLING 2025)
  40. Aligning Complex Knowledge Graph Question Answering as Knowledge-Aware Constrained Code Generation (COLING 2025)

Multiple Hop QA

  1. Scalable Multi-Hop Relational Reasoning for Knowledge-Aware Question Answering (EMNLP 2020) [Video]
  2. QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answering (Jure's group, NAACL-HLT 2021) [Paper]
  3. Cognitive Graph for Multi-Hop Reading Comprehension at Scale (ACL 2019) [Paper] [Code]
  • Note: Cognitive Graph is not directly equal to Knowledge Graph. You can view CG as a (dynamic, partial, local) KG generated instantly from the query.
  1. Complex Question Answering on knowledge graphs using machine translation and multi-task learning (EACL 2021) [Paper]
  2. Improving Multi-hop Question Answering over Knowledge Graphs using Knowledge Base Embeddings (ACL 2020) [Paper] More than 300 stars in Github in Dec 2023!

Multiple-Options QA

  1. RiddleSense: Reasoning about Riddle Questions Featuring Linguistic Creativity and Commonsense Knowledge (ACL 2021) [Paper]
  2. QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answering (Jure's group, NAACL-HLT 2021) [Paper]

Multiligual QA

  1. Improving Zero-Shot Cross-lingual Transfer for Multilingual Question Answering over Knowledge Graph (ACL 2021) [Paper]
  2. A System for Answering Simple Questions in Multiple Languages (ACL 2023, demo) [Paper]

LLM for KGQA, or KG for QA based on LLM 🔥

  • It becomes a hot topic in both academics and industries, especially with the developement of KG for RAG.
  1. Leveraging LLMs in Scholarly Knowledge Graph Question Answering (Arxiv, Nov 2023) [Paper]
  2. Answering Questions Over Knowledge Graphs Using Logic Programming Along with Language Models (submitted to ICLR 2023 but not revised) [Paper]
  3. Retrieve-Rewrite-Answer: A KG-to-Text Enhanced LLMs Framework for Knowledge Graph Question Answering (ijckg, Sep 2023) [Paper]
  4. Language Models as Controlled Natural Language Semantic Parsers for Knowledge Graph Question Answering (from Amazon Science, 2023) [Paper]
  5. Is GPT fit for KGQA? – Preliminary Results (CEUR Workshop 2023) [Paper]
  • Only GPT3 and GPT3.5 are used, so the experiemnts prove that there is limitation.
  1. Bring Your Own KG: Self-Supervised Program Synthesis for Zero-shot KGQA (Amazon and UMass, Nov 2023) [Paper]
  • In BYOKG, exploration leverages an LLM-backed symbolic agent that generates a diverse set of queryprogram exemplars, which are then used to ground a retrieval-augmented reasoning procedure to predict programs for arbitrary questions.
  1. A Benchmark to Understand the Role of Knowledge Graphs on Large Language Model's Accuracy for Question Answering on Enterprise SQL Databases (Arxiv, Nov 2023) [Paper]
  2. Knowledge-augmented language model prompting for zero-shot knowledge graph question answering (ACL 2023, KAIST + MBZUAI + Amazon) [Paper]
  3. Empowering Language Models with Knowledge Graph Reasoning for Open-Domain Question Answering (EMNLP 2022) [Paper]
  4. KnowledgeNavigator: Leveraging Large Language Models for Enhanced Reasoning over Knowledge Graph (Arxiv, Dec 2023) [Paper]
  • The baselines also include StructGPT (A general framework for large language model to reason over structured data, Arxiv 2023) and TOG (Think-on-graph: Deep and responsible reasoning of large language model with knowledge graph, ICLR 2024 poster)
  1. Pretrained transformers for simple question answering over knowledge graphs (Web–ISWC 2019)
  • The first work to utilize LLMs as classifiers for relation prediction.
  1. An empirical study of pre-trained language models in simple knowledge graph question answering (Arxiv 2023)
  • It introduce two LLM-based KGQA frameworks that adopt LLMs to detect mentioned entities and relations. Then, they query the answer in KGs using the extracted entity-relation pairs.
  1. QA-GNN: QAGNN: Reasoning with language models and knowledge graphs for question answering (ACL 2021)
  • It uses LLMs to encode the question and candidate answer pairs, which are adopted to estimate the importance of relative KG entities. The entities are retrieved to form a subgraph, where an answer reasoning is conducted by a GNN.
  1. A bert-based approach with relationaware attention for knowledge base question answering (IJCNN 2021)
  • It use LLMs to calculate the similarities between relations and questions to retrieve related facts.
  1. Subgraph retrieval enhanced model for multi-hop knowledge base question answering (ACL 2022)
  • a LLM-based path retriever to retrieve question-related relations hop-byhop and construct several paths.
  1. Can ChatGPT Replace Traditional KBQA Models? An In-Depth Analysis of the Question Answering Performance of the GPT LLM Family (ISWC 2023) [Paper]
  2. ReasoningLM: Enabling Structural Subgraph Reasoning in Pre-trained Language Models for Question Answering over Knowledge Graph (ACL 2023) [Paper]
  3. Language Models as Controlled Natural Language Semantic Parsers for Knowledge Graph Question Answering (ECAI 2023) [Paper]
  4. LLM-Based SPARQL Generation with Selected Schema from Large Scale Knowledge Base (CCKS2023 CKBQA competition, F1 score is 75.63% on CKBQA dataset) [Paper]
  5. Improving Subgraph Extraction Algorithms for One-Shot SPARQL Query Generation with Large Language Models (ISWC 2023) [Paper]
  6. A Structure and Content Prompt-based Method for Knowledge Graph Question Answering over Scholarly Data (ISWC 2023) [Paper]
  7. Bring Your Own KG: Self-Supervised Program Synthesis for Zero-Shot KGQA (Arxiv 2023) [Paper]
  8. Check for updates LLM-Based SPARQL Generation with Selected Schema from Large Scale Knowledge Base (Springer Nature Singapore 2023)
  9. A Knowledge Graph Question Answering Approach to IoT Forensics (IoTDI 2023) [Paper]
  10. Don’t Generate, Discriminate: A Proposal for Grounding Language Models to Real-World Environments (ACL 2023) [Paper]
  • Pangu consists of a symbolic agent and a neural LM working in a concerted fashion: The agent explores the environment to incrementally construct valid plans, and the LM evaluates the plausibility of the candidate plans to guide the search process.
  • The evaluation is conducted on KBQA.
  1. FlexKBQA: A Flexible LLM-Powered Framework for Few-Shot Knowledge Base Question Answering (AAAI 2024) [Paper]
  2. Knowledge Graph Prompting for Multi-Document Question Answering (AAAI 2024) [Paper]
  3. CyberQ: Generating Questions and Answers for Cybersecurity Education Using Knowledge Graph-Augmented LLMs (AAAI 2024) [Paper]
  4. Grape: Knowledge Graph Enhanced Passage Reader for Open-domain Question Answering (EMNLP 2022) [Paper]
  5. Chatgpt versus traditional question answering for knowledge graphs: Current status and future directions towards knowledge graph chatbots (Arxiv 2023, with 77+ citations in Aug 2024) [Paper]
  6. Cross-Data Knowledge Graph Construction for LLM-enabled Educational Question-Answering System: A Case Study at HCMUT (AIQAM 2024) [Paper]
  7. Retrieval-augmented generation with knowledge graphs for customer service question answering (SIGIR 2024) [Paper]
  8. Knowledge Graph aided LLM based ESG Question-Answering from News ((KG4S2024, co-located with ESWC2024) [Paper]
  9. Improving LLM-based KGQA for multi-hop Question Answering with implicit reasoning in few-shot examples (KaLLM 2024) [Paper]
  10. Towards Faithful Knowledge Graph Explanation Through Deep Alignment in Commonsense Question Answering (EMNLP 2024) [Paper]
  11. Retrieval and Reasoning on KGs: Integrate Knowledge Graphs into Large Language Models for Complex Question Answering (EMNLP 2024) [Paper]
  12. Training-free Deep Concept Injection Enables Language Models for Video Question Answering (EMNLP 2024) [Paper]
  13. A GAIL Fine-Tuned LLM Enhanced Framework for Low-Resource Knowledge Graph Question Answering (CIKM 2024) [Paper]
  14. Generate-then-Ground in Retrieval-Augmented Generation for Multi-hop Question Answering (ACL 2024) [Paper]
  15. RGR-KBQA: Generating Logical Forms for Question Answering Using Knowledge-Graph-Enhanced Large Language Model (COLING 2025) [Paper]
  16. Awakening Augmented Generation: Learning to Awaken Internal Knowledge of Large Language Models for Question Answering (COLING 2025)

SPARQL

  1. Enhancing SPARQL Generation by Triplet-order-sensitive Pre-training (CIKM 2024) [Paper]

📊 Leaderboard and Benchmarks

  1. QALD-9： The 9th challenge on question answering over linked data (QALD-9) (invited paper) (CEUR Workshop 2018)
  2. Lc-quad 2.0: A large dataset for complex question answering over wikidata and dbpedia (ISWC 2019)
  3. Knowledge Graph Question Answering Leaderboard: A Community Resource to Prevent a Replication Crisis (LREC 2022) [Paper] [Link]
  4. MetaQA: Variational reasoning for question answering with knowledge graph (AAAI 2018)
  • A large scale multi-hop KGQA dataset with more than 400k questions in the movie domain. It has 1-hop, 2-hop, and 3-hop questions.
  • It also provides a KG with 135k triples, 43k entities, and nine relations.
  1. WebQuestionsSP: The value of semantic parse labeling for knowledge base question answering (ACL 2016)
  • a smaller QA dataset with 4,737 questions. The questions in this dataset are 1-hop and 2-hop questions and are answerable through Freebase KG.
  1. ComplexWebQuestions: The web as a knowledge-base for answering complex questions (ACL 2018)
  2. GraphQ: On generating characteristic-rich question sets for QA evaluation (EMNLP 2016)
  3. MKQA: a linguistically diverse benchmark for multilingual open domain question answering (ACL 2021)
  4. GrailQA: Beyond IID: three levels of generalization for question answering on knowledge bases (WWW 2021)
  5. KQA Pro: a dataset with explicit compositional programs for complex question answering over knowledge base (ACL 2022)
  6. Spider4SPARQL: A Complex Benchmark for Evaluating Knowledge Graph Question Answering Systems (Arxiv, Sep 2023) [Github]
  7. GQA: A new dataset for real-world visual reasoning and compositional question answering (CVPR 2019)
  8. A discussion of KGQA datasets: Knowledge Graph Question Answering Datasets and Their Generalizability: Are They Enough for Future Research? (SIGIR 2022) [Paper]
  9. Leaderboard: Knowledge Graph Question Answering Leaderboard: A Community Resource to Prevent a Replication Crisis (LREC 2022) [Paper]
  10. ToolQA: A Dataset for LLM Question Answering with External Tools (NeurIPS 2023, Datasets and Benchmarks Track) [Paper]
  11. QuALITY: Question Answering with Long Input Texts, Yes! (NAACL 2022) [Link]
  12. GeoQuestions1089: The question answering system GeoQA2 and a new benchmark for its evaluation (International Journal of Applied Earth Observation and Geoinformation 2024) [Paper]
  13. CommonsenseQA in-house (IH) data split: Commonsenseqa: A question answering challenge targeting commonsense knowledge (NAACL 2019)
  14. OpenBookQA: Can a suit of armor conduct electricity? a new dataset for open book question answering (EMNLP 2018)
  15. RAC: Retrieval-augmented Conversation Dataset for Open-domain Question Answering in Conversational Settings (EMNLP 2024, industry) [Paper]
  16. 2WikiMultiHop Dataset: Constructing a multihop QA dataset for comprehensive evaluation of reasoning steps (COLING 2020)
  17. HotpotQA (HQA): HotpotQA: A dataset for diverse, explainable multi-hop question answering (EMNLP 2018)
  18. MuSiQue (MQA): MuSiQue: Multihop questions via single-hop question composition (TACL 2022)
  19. StrategyQA (SQA): Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies (TACL 2021)
  20. CQW: The web as a knowledge-base for answering complex questions (NAACL 2018)
  21. WebQSP: The value of semantic parse labeling for knowledge base question answering (ACL 2016)

🧐 Others

Related Readings

  1. Unsupervised Question Decomposition for Question Answering (EMNLP 2020) [Paper] [Code]
  2. The Web as a Knowledge-base for Answering Complex Questions (NAACL-HLT 2018) [Paper]
  3. Scalable Join Processing on Very Large RDF Graphs (SIGMOD 2009) [Paper] 🌟

Some thoughts

  • Consider Combine embedding and subgraph matching for KGQA?