Token indices sequence length

[saboor2632]

I am facing this issue whenever i run my code for scraping bbc or any other site
Error:
Token indices sequence length is longer than the specified maximum sequence length for this model (5102 > 1024). Running this sequence through the model will result in indexing errors

It does not give me complete results

[Qunlexie]

I have this same issue as well especially when working with Ollama LLAMA models

[VinciGit00]

for big websites you should use openai

[PeriniM]

@Qunlexie @saboor2632 there is indeed an issue with the method to calculate chunks for ollama models, in tokenizer_ollama.py. We are using get_num_tokens from langchain lib but it is always limiting the model to 1024 tokens. Even if we change the num of tokens from the graph config using the model_tokens param we could only set it to 1024. It would be better to calculate the num of chunks ourself, using for example openai tokenizer from tiktoken and approximate the num of tokens per chunk. wdyt

[Qunlexie]

We are using get_num_tokens from langchain lib but it is always limiting the model to 1024 tokens. Even if we change the num of tokens from the graph config using the model_tokens param we could only set it to 1024.

Is this a bug that should be raised to langchain? I belive the limit to the tokens is the cause of the issue where only parts of the web page is being retrieved rather all of it.

It would be better to calculate the num of chunks ourself, using for example openai tokenizer from tiktoken and approximate the num of tokens per chunk. wdyt

How would this work? Do you have a practical example? for how this will work with ScrpapeGraph. I really believe that getting Ollama to work properly is key for open source.

Happy to get your thoughts

[Pal-dont-want-to-work]

I face the same problem, hoping author can fix it
Token indices sequence length is longer than the specified maximum sequence length for this model (1548 > 1024). Running this sequence through the model will result in indexing errors

[PeriniM]

Hey @saboor2632 @Qunlexie @Pal-dont-want-to-work the issue is now fixed in v1.36!
You can try with:

graph_config = {
    "llm": {
        "model": "ollama/llama3.2",
        "temperature": 0,
        "model_tokens": 4096
    },
    "verbose": True,
    "headless": False
}

[Qunlexie]

Thanks @PeriniM for the fix. The import error seems to have been resolved.

It does not give me complete results

However this part of the original question is still true. I believe this is because the parser runs into several issues when parsing ollama models into json (which is more likely at higher model_tokens).

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/langchain_core/output_parsers/json.py", line 83, in parse_result
    return parse_json_markdown(text)
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<string>", line 20, in <module>
  File "/usr/local/lib/python3.10/dist-packages/scrapegraphai/graphs/smart_scraper_graph.py", line 292, in run
    self.final_state, self.execution_info = self.graph.execute(inputs)
  .... 
  File "/usr/local/lib/python3.10/dist-packages/langchain_core/output_parsers/json.py", line 86, in parse_result
    raise OutputParserException(msg, llm_output=text) from e
langchain_core.exceptions.OutputParserException: Invalid json output: Here is the text reformatted to be more readable:

You can recreate this by:

graph_config = {
   'llm': {
      'model': 'ollama/llama3.2',
      'temperature': 0.0,
      'model_tokens': 4096,
      'base_url': 'http://localhost:11434'
   },
   'verbose': True,
   'headless': False
}

smart_scraper_graph = SmartScraperGraph(
   prompt='List me all the titles and link to the story',
   source='https://www.vanguardngr.com/2023/page/1',
   config=graph_config
)

result = smart_scraper_graph.run()

I wonder if the new structured output by ollama could solve this?
Wdyt?

[PeriniM]

Hey @Qunlexie in the new version is already present the new structured output for ollama models, I guess merging many chunks (big websiites) into a valid json is difficult for a 3b model, can you try with a bigger model and lmk? or increase the model_tokens in the graph config

[VinciGit00]

try with OpenAI, if it works with that is because of the model