[feat] deeplake retriever implementation

[AniLeo-01]

No description provided.

[okhat]

This looks good, but we usually like to have the import for external dependencies happen inside the class, so that it does not affect people who don’t need the library

[AniLeo-01]

@okhat could you please take a look 👀. I've made the changes that you have requested. Also, sorry for the inconvenience.

[okhat]

Thanks! I think that import code needs to be inside init for example

[AniLeo-01]

But putting the import inside init would lead to an error in the type hinting of the deeplake_client which would be VectorStore.

def __init__(
        self,
        deeplake_vectorstore_name: str,
        deeplake_client: VectorStore, #this will throw: "VectorStore" is not defined
        k: int = 3,
    ):
        try:
          from deeplake import VectorStore
        except ImportError:
          raise ImportError(
              "The 'deeplake' extra is required to use DeepLakeRM. Install it with `pip install dspy-ai[deeplake]`"
          )
        self._deeplake_vectorstore_name = deeplake_vectorstore_name
        self._deeplake_client = deeplake_client

        super().__init__(k=k)

[AniLeo-01]

@okhat Except for that import part, I have fixed the code and made it workable (it's working if the import check is made outside init). Should I create a notebook for the demonstration of the retriever?

[okhat]

This type won't work right? The import is happening inside the function

[okhat]

(also if possible please conduct a test with and without the dependencies installed to make sure it will work in both cases)

[AniLeo-01]

Using a generic Type type hint would resolve the issue. This is useful if we have a function that expects an argument of type VectorStore, but we do not want to specify the exact subclass of VectorStore that is required.

Suggested change

	deeplake_client: VectorStore,
	deeplake_client: Type['VectorStore'],

[AniLeo-01]

I think it's better to remove the type hint, as it's not enforcing any type either way.

[AniLeo-01]

I've tested the code with/without the dependencies, it's working now! Throwing the import error if the dependencies are not installed which is expected, and running fine if installation is done.

[insop]

Could you also update this document?

• docs/retrieval_models_client.md

[insop]

Would this be handled as this PR?

• Refactored ChromadbRM to use generic EmbeddingFunction #400

[AniLeo-01]

Unfortunately, deeplake doesn't provide any wrapper class for EmbeddingFunction, so we have to create it from scratch. I have implemented the openAI because it's the most popular and as a template as well.

[AniLeo-01]

Supporting my above comment with this excerpt from the official documentation:

> embedding_function (Optional[Any], optional) – Function or class that converts the embeddable data into embeddings. Input to embedding_function is a list of data and output is a list of embeddings. Defaults to None.

Suggest me what options I would keep for embedding function, and I would definitely update them in the code.

[insop]

Then, looks good to me, please add the entry to the document that I mentioned.
Thank you

[AniLeo-01]

Sure!