-
Notifications
You must be signed in to change notification settings - Fork 60
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix: stackoverflow caused by textspliter #165
Conversation
|
Sorry for the incorrect example. It's too causal. I just want to provide an example, actually users will initialize through As for how to fix this problem, I don't want to avoid this problem by simply checking. Instead, allow the user's input to determine how to split it, start on line 871 in the |
Thank you for your contribution! 🚀 |
Pull Request Template
Description
When separators do not include "", textspliter may result in infinite recursive calls, i.e., stack overflow.
Here's an example:
The input parameters are:
error like this:
Type of change
Add the newSeparators array to avoid entering an infinite loop call.
Inspired by https://github.com/langchain-ai/langchain/blob/00a09e1b7117f3bde14a44748510fcccc95f9de5/libs/langchain/langchain/text_splitter.py.