Completion suggestor with synonyms analyzer - getting TokenStream expanded to 450 finite strings. Only <= 256 finite strings are supported

[seme1]

I’m using the completion suggester with a custom [synonymize] analyzer that expands the words in each indexed document with its synonyms.

When trying to index some documents with an input filed containing many synonyms, I get the following message:

java.lang.IllegalArgumentException: TokenStream expanded to 450 finite strings. Only <= 256 finite strings are supported at org.elasticsearch.search.suggest.completion.CompletionTokenStream.incrementToken(CompletionTokenStream.java:66) at org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:634) at org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:365) at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:321) at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:234) at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:450) at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1477) at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1256) at org.elasticsearch.index.engine.InternalEngine.innerIndex(InternalEngine.java:530) at org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:457) at org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:601) at org.elasticsearch.index.engine.Engine$Index.execute(Engine.java:836) at org.elasticsearch.action.index.TransportIndexAction.executeIndexRequestOnPrimary(TransportIndexAction.java:237) at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:326) at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:119) at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:68) at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryPhase.doRun(TransportReplicationAction.java:639) at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:279) at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:271) at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:75) at org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:376) at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

I tried setting the max_token_length to 900, but it does not have any effect and the above exception message continues to show up.

Here is my schema:

{ "mappings":{ "pname_ar":{ "properties":{ "name": {"type":"string"}, "suggest": { "type":"completion", "analyzer": "synonimize", "search_analyzer":"autocomplete", "preserve_position_increments":false, "payloads":true, "context": { "catalog": { "type": "category" } } } ,"max_token_length":900 } } }, "settings": { "analysis": { "filter": { "addy_synonym_filter": { "type": "synonym", "synonyms": [__my synonyms list is placed here__] } }, "analyzer": { "autocomplete": { "type": "custom", "tokenizer": "lowercase" ,"max_token_length":900 }, "synonimize": { "type":"custom", "tokenizer": "bigStandardTokenizer", "filter": [ "lowercase", "addy_synonym_filter" ] ,"max_token_length":900 } },"tokenizer": { "bigStandardTokenizer":{ "type":"standard", "max_token_length":900 } } } } }

As you can see, I tried adding “max_token_length” at different tokens and they don’t seem to have an effect on the above mentioned error message.

This seems to be a bug to me. ES version used is 2.3

[javanna]

@jimczi would you have time to have a look?

[jimczi]

Thanks for starting the discussion on the forum, I just answered https://discuss.elastic.co/t/completion-suggestor-with-synonyms-analyzer-getting-tokenstream-expanded-to-450-finite-strings-only-256-finite-strings-are-supported/98153.
@seme1 we're trying to answer the questions on the forum first before creating an issue so no need to duplicate ;).
I close this issue, we can continue the discussion on the forum