[Community] [Text Splitters]: Fix TEI support and html section splitter #22595
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
dosubot
bot
added
size:S
plageon
changed the title
baskaryan
reviewed
Jun 6, 2024
libs/community/langchain_community/embeddings/huggingface_hub.py
Outdated
Show resolved
Hide resolved
baskaryan
reviewed
Jun 6, 2024
dosubot
bot
added
size:XS
This was referenced Jun 7, 2024
efriis
pushed a commit
that referenced
this pull request
Jul 3, 2024
Update former pull request: #22595. Modified `libs/partners/huggingface/langchain_huggingface/embeddings/huggingface_endpoint.py`, where the API call function does not match current [Text Embeddings Inference API](https://huggingface.github.io/text-embeddings-inference/#/Text%20Embeddings%20Inference/embed). One example is: ```json { "inputs": "string", "normalize": true, "truncate": false } ``` Parameters in `_model_kwargs` are not passed properly in the latest version. By the way, the issue *[why cause 413? #50](huggingface/text-embeddings-inference#50 might be solved.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
[Community]: Fix HuggingFaceHubEmbeddings
Modified
langchain_community.embeddings.HuggingFaceHubEmbeddings, where the API call function does not match current Text Embeddings Inference API.One example is:
{ "inputs": "string", "normalize": true, "truncate": false }Parameters in
_model_kwargsare not passed properly in the latest version. By the way, the issue why cause 413? #50 might be solved.[Text Splitters]: Fix HTMLSectionSplitter
Modified
langchain_text_splitters.HTMLSectionSplitter, where in the latest versiondictdata structure is used to store sections from a html document, in functionsplit_html_by_headers. The header/section element names serve as dict keys. This can be a problem when duplicate header/section element names are present in a single html document. Latter ones can replace former ones with the same name. Therefore some contents can be miss after html text splitting is conducted.Using a list to store sections can hopefully solve the problem.