feat: implement text chunking processor with fixed token length and delimiter algorithm #1858
| Job | Run time |
|---|---|
| 8m 6s | |
| 1m 37s | |
| 7m 59s | |
| 1m 47s | |
| 8m 13s | |
| 1m 47s | |
| 5s | |
| 8m 48s | |
| 2m 43s | |
| 8m 54s | |
| 2m 37s | |
| 8m 23s | |
| 2m 42s | |
| 1h 3m 41s |
| Job | Run time |
|---|---|
| 8m 6s | |
| 1m 37s | |
| 7m 59s | |
| 1m 47s | |
| 8m 13s | |
| 1m 47s | |
| 5s | |
| 8m 48s | |
| 2m 43s | |
| 8m 54s | |
| 2m 37s | |
| 8m 23s | |
| 2m 42s | |
| 1h 3m 41s |