Fix issue with duplicate contents in one document list (#1256)

If 2 documents in document_list have ideantical contents they both get added to the database because thet are not present in DB yet when self.vector_db.doc_exists(document) filter is applied. Added additional filter to fix that Co-authored-by: Dirk Brand <[email protected]> Co-authored-by: Yash Pratap Solanky <[email protected]>
agno-agi · Jan 8, 2025 · 9ac6594 · 9ac6594
1 parent 7442a8e
commit 9ac6594
Showing 1 changed file with 1 addition and 0 deletions.
diff --git a/phi/knowledge/base.py b/phi/knowledge/base.py
@@ -84,6 +84,7 @@ def load(
             else:
                 # Filter out documents which already exist in the vector db
                 if skip_existing:
+                    document_list = {document.content: document for document in document_list}.values()
                     documents_to_load = [
                         document for document in document_list if not self.vector_db.doc_exists(document)
                     ]