feat: implement text chunking processor with fixed token length and delimiter algorithm #556
| Job | Run time |
|---|---|
| 5m 50s | |
| 5m 50s | |
| 4m 15s | |
| 4m 15s | |
| 5m 57s | |
| 7m 14s | |
| 4m 13s | |
| 4m 13s | |
| 5m 46s | |
| 5m 46s | |
| 4m 59s | |
| 4m 59s | |
| 5m 53s | |
| 7m 30s | |
| 4m 54s | |
| 4m 54s | |
| 5m 53s | |
| 5m 59s | |
| 5m 6s | |
| 5m 6s | |
| 5m 58s | |
| 7m 5s | |
| 4m 48s | |
| 4m 48s | |
| 5m 2s | |
| 5m 2s | |
| 6m 9s | |
| 6m 9s | |
| 6m 12s | |
| 6m 12s | |
| 6m 22s | |
| 6m 22s | |
| 3m 41s | |
| 3m 41s | |
| 3m 51s | |
| 3m 51s | |
| 5m 0s | |
| 5m 0s | |
| 4m 37s | |
| 4m 37s | |
| 4m 55s | |
| 4m 55s | |
| 4m 35s | |
| 4m 35s | |
| 4m 52s | |
| 4m 52s | |
| 5m 46s | |
| 5m 46s | |
| 5m 58s | |
| 5m 58s | |
| 6m 3s | |
| 6m 3s | |
| 4m 0s | |
| 4m 0s | |
| 3m 47s | |
| 3m 47s | |
| 5m 4s | |
| 5m 4s | |
| 5m 1s | |
| 5m 1s | |
| 5m 19s | |
| 5m 19s | |
| 4m 48s | |
| 4m 48s | |
| 4m 32s | |
| 4m 32s | |
| 5m 52s | |
| 5m 52s | |
| 5m 50s | |
| 5m 50s | |
| 6m 15s | |
| 6m 15s | |
| 6h 18m 13s |