From 5dc39edb061456eb71f5b27b8c33423e47eec895 Mon Sep 17 00:00:00 2001 From: Jian-Guo Zhang Date: Wed, 28 Jul 2021 15:39:29 -0500 Subject: [PATCH 1/2] Update README.md Add more instructions for processing small files --- parlai/agents/rag/README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/parlai/agents/rag/README.md b/parlai/agents/rag/README.md index 5fcfeb07551..9b35774f887 100644 --- a/parlai/agents/rag/README.md +++ b/parlai/agents/rag/README.md @@ -155,6 +155,8 @@ python index_dense_embeddings.py --retriever-embedding-size 768 \ --embeddings-dir /tmp/wiki_passage_embeddings/ --embeddings-name wiki_passages ``` +Note the default index factory setting is `IVF4096_HNSW128,PQ128`, if you are processing small files, you may encounter errors such as `Error: 'nx >= k' failed`, then you need to set `--compressed-indexer-factory` to other indexes in the [index factory in FAISS](https://github.com/facebookresearch/faiss/wiki/The-index-factory)) such as `HNSW32`. + ## Directory Structure / Custom Components I will outline here the structure of the RAG directory, and where you might want to add custom components if you so desire. From 9b6ad2610b092b06af567bba4fe63c815b8fdc1b Mon Sep 17 00:00:00 2001 From: Jian-Guo Zhang Date: Wed, 28 Jul 2021 15:43:10 -0500 Subject: [PATCH 2/2] Update README.md --- parlai/agents/rag/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/parlai/agents/rag/README.md b/parlai/agents/rag/README.md index 9b35774f887..50e0a523419 100644 --- a/parlai/agents/rag/README.md +++ b/parlai/agents/rag/README.md @@ -155,7 +155,7 @@ python index_dense_embeddings.py --retriever-embedding-size 768 \ --embeddings-dir /tmp/wiki_passage_embeddings/ --embeddings-name wiki_passages ``` -Note the default index factory setting is `IVF4096_HNSW128,PQ128`, if you are processing small files, you may encounter errors such as `Error: 'nx >= k' failed`, then you need to set `--compressed-indexer-factory` to other indexes in the [index factory in FAISS](https://github.com/facebookresearch/faiss/wiki/The-index-factory)) such as `HNSW32`. +Note the default index factory setting is `IVF4096_HNSW128,PQ128`, if you are processing small files, you may encounter errors such as `Error: 'nx >= k' failed`, then you need to set `--compressed-indexer-factory` to other indexes in the [index factory in FAISS](https://github.com/facebookresearch/faiss/wiki/The-index-factory) such as `HNSW32`. ## Directory Structure / Custom Components