Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Search can fail when size is 0 and long sort optimization enabled. #56923

Closed
toby-jn opened this issue May 18, 2020 · 4 comments · Fixed by #57044
Closed

Search can fail when size is 0 and long sort optimization enabled. #56923

toby-jn opened this issue May 18, 2020 · 4 comments · Fixed by #57044
Labels
>bug :Search/Search Search-related issues that do not fall into other categories Team:Search Meta label for search team

Comments

@toby-jn
Copy link

toby-jn commented May 18, 2020

Elasticsearch version (bin/elasticsearch --version):

Version: 7.7.0, Build: oss/docker/81a1e9eda8e6183f5237786246f6dced26a10eaf/2020-05-12T02:01:37.602180Z, JVM: 14

Plugins installed: []

JVM version (java -version):

openjdk version "14" 2020-03-17
OpenJDK Runtime Environment AdoptOpenJDK (build 14+36)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 14+36, mixed mode, sharing)

OS version (uname -a if on a Unix-like system):

Linux elasticsearch-0 4.19.102+ #1 SMP Tue Feb 11 05:33:03 PST 2020 x86_64 x86_64 x86_64 GNU/Linux

Description of the problem including expected versus actual behavior:

After upgrading to 7.7.0 from 7.5.1, when doing a search with track_total_hits=true, size=0 and sort I get an error:

curl -X POST '127.0.0.1:9200/v33.tcpevent-000174/_search?pretty' -H content-type:application/json --data '{"size":0,"sort":[{"@timestamp":{"order":"desc"}},{"conn_uuid":{"order":"desc"}}],"track_total_hits":true}'
{
  "error" : {
    "root_cause" : [
      {
        "type" : "illegal_argument_exception",
        "reason" : "numHits must be > 0; please use TotalHitCountCollector if you just need the total hit count"
      }
    ],
    "type" : "search_phase_execution_exception",
    "reason" : "all shards failed",
    "phase" : "query",
    "grouped" : true,
    "failed_shards" : [
      {
        "shard" : 1,
        "index" : "v33.tcpevent-000174",
        "node" : "j07kSSNmSFueB_E9i911MQ",
        "reason" : {
          "type" : "illegal_argument_exception",
          "reason" : "numHits must be > 0; please use TotalHitCountCollector if you just need the total hit count"
        }
      }
    ],
    "caused_by" : {
      "type" : "illegal_argument_exception",
      "reason" : "numHits must be > 0; please use TotalHitCountCollector if you just need the total hit count",
      "caused_by" : {
        "type" : "illegal_argument_exception",
        "reason" : "numHits must be > 0; please use TotalHitCountCollector if you just need the total hit count"
      }
    }
  },
  "status" : 400
}

The index settings have:

        "sort" : {
          "field" : "@timestamp",
          "order" : "desc"
        }

If I set size to 1 or change the sort parameter to match the index sort then the query works. Both these queries work:

{"size":0,"sort":[{"@timestamp":{"order":"desc"}}],"track_total_hits":true}
{"size":1,"sort":[{"@timestamp":{"order":"desc"}},{"conn_uuid":{"order":"desc"}}],"track_total_hits":true}

I've tried to create a minimal reproducible setup for this error, but using the same mappings and query I've not been able to trigger the error.

Logs:
Query run with error trace:

curl -s -X POST '127.0.0.1:9200/v33.tcpevent-000174/_search?pretty&error_trace=true' -H content-type:application/json --data '{"size":0,"sort":[{"@timestamp":{"order":"desc"}},{"conn_uuid":{"order":"desc"}}],"track_total_hits":1000}'
{
  "error" : {
    "root_cause" : [
      {
        "type" : "illegal_argument_exception",
        "reason" : "numHits must be > 0; please use TotalHitCountCollector if you just need the total hit count",
        "stack_trace" : "[numHits must be > 0; please use TotalHitCountCollector if you just need the total hit count]; nested: IllegalArgumentException[numHits must be > 0; please use TotalHitCountCollector if you just need the total hit count];\n\tat org.elasticsearch.ElasticsearchException.guessRootCauses(ElasticsearchException.java:644)\n\tat org.elasticsearch.action.search.SearchPhaseExecutionException.guessRootCauses(SearchPhaseExecutionException.java:167)\n\tat org.elasticsearch.ElasticsearchException.guessRootCauses(ElasticsearchException.java:626)\n\tat org.elasticsearch.ElasticsearchException.generateFailureXContent(ElasticsearchException.java:572)\n\tat org.elasticsearch.rest.BytesRestResponse.build(BytesRestResponse.java:138)\n\tat org.elasticsearch.rest.BytesRestResponse.<init>(BytesRestResponse.java:96)\n\tat org.elasticsearch.rest.BytesRestResponse.<init>(BytesRestResponse.java:91)\n\tat org.elasticsearch.rest.action.RestActionListener.onFailure(RestActionListener.java:58)\n\tat org.elasticsearch.rest.action.RestCancellableNodeClient$1.onFailure(RestCancellableNodeClient.java:106)\n\tat org.elasticsearch.action.support.TransportAction$1.onFailure(TransportAction.java:79)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.raisePhaseFailure(AbstractSearchAsyncAction.java:571)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:551)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:309)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:580)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:393)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.access$100(AbstractSearchAsyncAction.java:68)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:245)\n\tat org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73)\n\tat org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59)\n\tat org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:402)\n\tat org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1139)\n\tat org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:244)\n\tat org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225)\n\tat org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:242)\n\tat org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:234)\n\tat org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:137)\n\tat org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:103)\n\tat org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:676)\n\tat org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:62)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)\n\tat io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)\n\tat io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)\n\tat io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:227)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)\n\tat io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)\n\tat io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)\n\tat io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578)\n\tat io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)\n\tat io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)\n\tat io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat java.base/java.lang.Thread.run(Thread.java:832)\nCaused by: java.lang.IllegalArgumentException: numHits must be > 0; please use TotalHitCountCollector if you just need the total hit count\n\tat org.apache.lucene.search.TopFieldCollector.create(TopFieldCollector.java:464)\n\tat org.apache.lucene.search.TopFieldCollector$1.newCollector(TopFieldCollector.java:502)\n\tat org.apache.lucene.search.TopFieldCollector$1.newCollector(TopFieldCollector.java:495)\n\tat org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:166)\n\tat org.elasticsearch.search.query.QueryPhase.searchWithCollectorManager(QueryPhase.java:402)\n\tat org.elasticsearch.search.query.QueryPhase.executeInternal(QueryPhase.java:297)\n\tat org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:151)\n\tat org.elasticsearch.indices.IndicesService.lambda$loadIntoContext$21(IndicesService.java:1344)\n\tat org.elasticsearch.indices.IndicesService.lambda$cacheShardLevelResult$22(IndicesService.java:1396)\n\tat org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:176)\n\tat org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:159)\n\tat org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:433)\n\tat org.elasticsearch.indices.IndicesRequestCache.getOrCompute(IndicesRequestCache.java:125)\n\tat org.elasticsearch.indices.IndicesService.cacheShardLevelResult(IndicesService.java:1402)\n\tat org.elasticsearch.indices.IndicesService.loadIntoContext(IndicesService.java:1341)\n\tat org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:359)\n\tat org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:434)\n\tat org.elasticsearch.search.SearchService.access$200(SearchService.java:135)\n\tat org.elasticsearch.search.SearchService$2.lambda$onResponse$0(SearchService.java:395)\n\tat org.elasticsearch.search.SearchService.lambda$runAsync$0(SearchService.java:411)\n\tat org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)\n\tat java.lang.Thread.run(Thread.java:832)\n"
      }
    ],
    "type" : "search_phase_execution_exception",
    "reason" : "all shards failed",
    "phase" : "query",
    "grouped" : true,
    "failed_shards" : [
      {
        "shard" : 2,
        "index" : "v33.tcpevent-000174",
        "node" : "yQLAAU2QTN6ORtWRIhqsFQ",
        "reason" : {
          "type" : "illegal_argument_exception",
          "reason" : "numHits must be > 0; please use TotalHitCountCollector if you just need the total hit count",
          "stack_trace" : "java.lang.IllegalArgumentException: numHits must be > 0; please use TotalHitCountCollector if you just need the total hit count\n\tat org.apache.lucene.search.TopFieldCollector.create(TopFieldCollector.java:464)\n\tat org.apache.lucene.search.TopFieldCollector$1.newCollector(TopFieldCollector.java:502)\n\tat org.apache.lucene.search.TopFieldCollector$1.newCollector(TopFieldCollector.java:495)\n\tat org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:166)\n\tat org.elasticsearch.search.query.QueryPhase.searchWithCollectorManager(QueryPhase.java:402)\n\tat org.elasticsearch.search.query.QueryPhase.executeInternal(QueryPhase.java:297)\n\tat org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:151)\n\tat org.elasticsearch.indices.IndicesService.lambda$loadIntoContext$21(IndicesService.java:1344)\n\tat org.elasticsearch.indices.IndicesService.lambda$cacheShardLevelResult$22(IndicesService.java:1396)\n\tat org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:176)\n\tat org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:159)\n\tat org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:433)\n\tat org.elasticsearch.indices.IndicesRequestCache.getOrCompute(IndicesRequestCache.java:125)\n\tat org.elasticsearch.indices.IndicesService.cacheShardLevelResult(IndicesService.java:1402)\n\tat org.elasticsearch.indices.IndicesService.loadIntoContext(IndicesService.java:1341)\n\tat org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:359)\n\tat org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:434)\n\tat org.elasticsearch.search.SearchService.access$200(SearchService.java:135)\n\tat org.elasticsearch.search.SearchService$2.lambda$onResponse$0(SearchService.java:395)\n\tat org.elasticsearch.search.SearchService.lambda$runAsync$0(SearchService.java:411)\n\tat org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)\n\tat java.lang.Thread.run(Thread.java:832)\n"
        }
      }
    ],
    "caused_by" : {
      "type" : "illegal_argument_exception",
      "reason" : "numHits must be > 0; please use TotalHitCountCollector if you just need the total hit count",
      "caused_by" : {
        "type" : "illegal_argument_exception",
        "reason" : "numHits must be > 0; please use TotalHitCountCollector if you just need the total hit count",
        "stack_trace" : "java.lang.IllegalArgumentException: numHits must be > 0; please use TotalHitCountCollector if you just need the total hit count\n\tat org.apache.lucene.search.TopFieldCollector.create(TopFieldCollector.java:464)\n\tat org.apache.lucene.search.TopFieldCollector$1.newCollector(TopFieldCollector.java:502)\n\tat org.apache.lucene.search.TopFieldCollector$1.newCollector(TopFieldCollector.java:495)\n\tat org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:166)\n\tat org.elasticsearch.search.query.QueryPhase.searchWithCollectorManager(QueryPhase.java:402)\n\tat org.elasticsearch.search.query.QueryPhase.executeInternal(QueryPhase.java:297)\n\tat org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:151)\n\tat org.elasticsearch.indices.IndicesService.lambda$loadIntoContext$21(IndicesService.java:1344)\n\tat org.elasticsearch.indices.IndicesService.lambda$cacheShardLevelResult$22(IndicesService.java:1396)\n\tat org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:176)\n\tat org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:159)\n\tat org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:433)\n\tat org.elasticsearch.indices.IndicesRequestCache.getOrCompute(IndicesRequestCache.java:125)\n\tat org.elasticsearch.indices.IndicesService.cacheShardLevelResult(IndicesService.java:1402)\n\tat org.elasticsearch.indices.IndicesService.loadIntoContext(IndicesService.java:1341)\n\tat org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:359)\n\tat org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:434)\n\tat org.elasticsearch.search.SearchService.access$200(SearchService.java:135)\n\tat org.elasticsearch.search.SearchService$2.lambda$onResponse$0(SearchService.java:395)\n\tat org.elasticsearch.search.SearchService.lambda$runAsync$0(SearchService.java:411)\n\tat org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)\n\tat java.lang.Thread.run(Thread.java:832)\n"
      },
      "stack_trace" : "[numHits must be > 0; please use TotalHitCountCollector if you just need the total hit count]; nested: IllegalArgumentException[numHits must be > 0; please use TotalHitCountCollector if you just need the total hit count];\n\tat org.elasticsearch.ElasticsearchException.guessRootCauses(ElasticsearchException.java:644)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:307)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:580)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:393)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.access$100(AbstractSearchAsyncAction.java:68)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:245)\n\tat org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73)\n\tat org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59)\n\tat org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:402)\n\tat org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1139)\n\tat org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:244)\n\tat org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225)\n\tat org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:242)\n\tat org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:234)\n\tat org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:137)\n\tat org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:103)\n\tat org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:676)\n\tat org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:62)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)\n\tat io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)\n\tat io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)\n\tat io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:227)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)\n\tat io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)\n\tat io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)\n\tat io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578)\n\tat io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)\n\tat io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)\n\tat io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat java.base/java.lang.Thread.run(Thread.java:832)\nCaused by: java.lang.IllegalArgumentException: numHits must be > 0; please use TotalHitCountCollector if you just need the total hit count\n\tat org.apache.lucene.search.TopFieldCollector.create(TopFieldCollector.java:464)\n\tat org.apache.lucene.search.TopFieldCollector$1.newCollector(TopFieldCollector.java:502)\n\tat org.apache.lucene.search.TopFieldCollector$1.newCollector(TopFieldCollector.java:495)\n\tat org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:166)\n\tat org.elasticsearch.search.query.QueryPhase.searchWithCollectorManager(QueryPhase.java:402)\n\tat org.elasticsearch.search.query.QueryPhase.executeInternal(QueryPhase.java:297)\n\tat org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:151)\n\tat org.elasticsearch.indices.IndicesService.lambda$loadIntoContext$21(IndicesService.java:1344)\n\tat org.elasticsearch.indices.IndicesService.lambda$cacheShardLevelResult$22(IndicesService.java:1396)\n\tat org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:176)\n\tat org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:159)\n\tat org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:433)\n\tat org.elasticsearch.indices.IndicesRequestCache.getOrCompute(IndicesRequestCache.java:125)\n\tat org.elasticsearch.indices.IndicesService.cacheShardLevelResult(IndicesService.java:1402)\n\tat org.elasticsearch.indices.IndicesService.loadIntoContext(IndicesService.java:1341)\n\tat org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:359)\n\tat org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:434)\n\tat org.elasticsearch.search.SearchService.access$200(SearchService.java:135)\n\tat org.elasticsearch.search.SearchService$2.lambda$onResponse$0(SearchService.java:395)\n\tat org.elasticsearch.search.SearchService.lambda$runAsync$0(SearchService.java:411)\n\tat org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)\n\tat java.lang.Thread.run(Thread.java:832)\n"
    },
    "stack_trace" : "Failed to execute phase [query], all shards failed; shardFailures {[yQLAAU2QTN6ORtWRIhqsFQ][v33.tcpevent-000174][2]: RemoteTransportException[[elasticsearch-2][10.32.130.157:9300][indices:data/read/search[phase/query]]]; nested: QueryPhaseExecutionException[Query Failed [Failed to execute main query]]; nested: IllegalArgumentException[numHits must be > 0; please use TotalHitCountCollector if you just need the total hit count]; }{[6RDmCBlcRM-dVtnwzcXTog][v33.tcpevent-000174][0]: RemoteTransportException[[elasticsearch-0][10.32.136.152:9300][indices:data/read/search[phase/query]]]; nested: QueryPhaseExecutionException[Query Failed [Failed to execute main query]]; nested: IllegalArgumentException[numHits must be > 0; please use TotalHitCountCollector if you just need the total hit count]; }{[VX6qSJggQAmNL_HPnz6hhw][v33.tcpevent-000174][1]: RemoteTransportException[[elasticsearch-3][10.32.134.121:9300][indices:data/read/search[phase/query]]]; nested: QueryPhaseExecutionException[Query Failed [Failed to execute main query]]; nested: IllegalArgumentException[numHits must be > 0; please use TotalHitCountCollector if you just need the total hit count]; }\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:551)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:309)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:580)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:393)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.access$100(AbstractSearchAsyncAction.java:68)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:245)\n\tat org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73)\n\tat org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59)\n\tat org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:402)\n\tat org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1139)\n\tat org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:244)\n\tat org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225)\n\tat org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:242)\n\tat org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:234)\n\tat org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:137)\n\tat org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:103)\n\tat org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:676)\n\tat org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:62)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)\n\tat io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)\n\tat io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)\n\tat io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:227)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)\n\tat io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)\n\tat io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)\n\tat io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578)\n\tat io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)\n\tat io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)\n\tat io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat java.base/java.lang.Thread.run(Thread.java:832)\nCaused by: [numHits must be > 0; please use TotalHitCountCollector if you just need the total hit count]; nested: IllegalArgumentException[numHits must be > 0; please use TotalHitCountCollector if you just need the total hit count];\n\tat org.elasticsearch.ElasticsearchException.guessRootCauses(ElasticsearchException.java:644)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:307)\n\t... 40 more\nCaused by: java.lang.IllegalArgumentException: numHits must be > 0; please use TotalHitCountCollector if you just need the total hit count\n\tat org.apache.lucene.search.TopFieldCollector.create(TopFieldCollector.java:464)\n\tat org.apache.lucene.search.TopFieldCollector$1.newCollector(TopFieldCollector.java:502)\n\tat org.apache.lucene.search.TopFieldCollector$1.newCollector(TopFieldCollector.java:495)\n\tat org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:166)\n\tat org.elasticsearch.search.query.QueryPhase.searchWithCollectorManager(QueryPhase.java:402)\n\tat org.elasticsearch.search.query.QueryPhase.executeInternal(QueryPhase.java:297)\n\tat org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:151)\n\tat org.elasticsearch.indices.IndicesService.lambda$loadIntoContext$21(IndicesService.java:1344)\n\tat org.elasticsearch.indices.IndicesService.lambda$cacheShardLevelResult$22(IndicesService.java:1396)\n\tat org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:176)\n\tat org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:159)\n\tat org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:433)\n\tat org.elasticsearch.indices.IndicesRequestCache.getOrCompute(IndicesRequestCache.java:125)\n\tat org.elasticsearch.indices.IndicesService.cacheShardLevelResult(IndicesService.java:1402)\n\tat org.elasticsearch.indices.IndicesService.loadIntoContext(IndicesService.java:1341)\n\tat org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:359)\n\tat org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:434)\n\tat org.elasticsearch.search.SearchService.access$200(SearchService.java:135)\n\tat org.elasticsearch.search.SearchService$2.lambda$onResponse$0(SearchService.java:395)\n\tat org.elasticsearch.search.SearchService.lambda$runAsync$0(SearchService.java:411)\n\tat org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)\n\tat java.lang.Thread.run(Thread.java:832)\n"
  },
  "status" : 400
}
@toby-jn toby-jn added >bug needs:triage Requires assignment of a team area label labels May 18, 2020
@jtibshirani jtibshirani added :Search/Search Search-related issues that do not fall into other categories and removed needs:triage Requires assignment of a team area label labels May 18, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-search (:Search/Search)

@elasticmachine elasticmachine added the Team:Search Meta label for search team label May 18, 2020
@toby-jn
Copy link
Author

toby-jn commented May 18, 2020

It looks to me like the error is coming from: https://github.com/elastic/elasticsearch/blob/v7.7.0/server/src/main/java/org/elasticsearch/search/query/QueryPhase.java#L376
Though I'm not sure why it's not easier to reproduce this error.

@jtibshirani
Copy link
Contributor

@toby-jn thanks for raising this bug. From digging into the stack trace, it looks like this happens when (1) the search size is 0, and (2) the 'long sort' optimization is enabled.

We added the optimization in 7.6 to speed up searches that sort on a long field (#48804). The optimization only kicks in if there are > 512 documents, and the sort doesn't match the index sort exactly (otherwise we could use the normal index sort optimization). This could help explain why you aren't able to put together a simple reproduction using just a few documents.

We will look into a fix. In the meantime, a workaround would be to remove the sort section for these searches that just count total hits, since sorting won't have an affect on the result.

@toby-jn
Copy link
Author

toby-jn commented May 18, 2020

Just managed to reproduce it, but looks like you beat me to why it wasn't working before, needed to add more documents for testing. Now able to reliably reproduce this with the following script on a fresh 7.7.0 install:

curl -X PUT 127.0.0.1:9200/twitter -H 'Content-Type: application/json' -d'
{
    "settings" : {
        "index" : {
            "sort.field" : "date",
            "sort.order": "desc"
        }
    },
    "mappings": {
        "properties": {
            "date": {
                "type": "date"
            },
            "title": {
                "type": "keyword"
            }
        }
    }
}
'

# index 20k documents
yes '{"index": {}}
{"date": "2020-05-18T19:21:36Z","title": "title1"}' | head -n 40000 | curl -X POST 127.0.0.1:9200/twitter/_bulk?refresh -H 'Content-Type: application/json' --data-binary @- > /dev/null

# works
curl -X POST '127.0.0.1:9200/twitter/_search?error_trace=true&pretty' -H content-type:application/json --data '{"size":0,"sort":[{"date":{"order":"desc"}}],"track_total_hits":true}'

# error
curl -X POST '127.0.0.1:9200/twitter/_search?error_trace=true&pretty' -H content-type:application/json --data '{"size":0,"sort":[{"date":{"order":"desc"}},{"title":{"order":"asc"}}],"track_total_hits":true}'

@jtibshirani jtibshirani changed the title "numHits must be > 0; please use TotalHitCountCollector if you just need the total hit count" error in 7.7 Search can fail when size is 0 and long sort optimization enabled. May 20, 2020
mayya-sharipova added a commit to mayya-sharipova/elasticsearch that referenced this issue May 21, 2020
@mayya-sharipova
Don't run sort optimization on size=0
23b152d 
Sort optimization creates TopFieldCollector that errors
when size=0. This ensures that sort optimization is not
run when size=0.

Closes elastic#56923
@mayya-sharipova mayya-sharipova mentioned this issue May 21, 2020
Merged
mayya-sharipova added a commit that referenced this issue May 21, 2020
@mayya-sharipova
Don't run sort optimization on size=0 (#57044)
bdfb137 
Sort optimization creates TopFieldCollector that errors
when size=0. This ensures that sort optimization is not
run when size=0.

Closes #56923
mayya-sharipova added a commit that referenced this issue May 21, 2020
@mayya-sharipova
Don't run sort optimization on size=0 (#57044)
4cf49bc 
Sort optimization creates TopFieldCollector that errors
when size=0. This ensures that sort optimization is not
run when size=0.

Closes #56923
mayya-sharipova added a commit that referenced this issue May 21, 2020
@mayya-sharipova
Don't run sort optimization on size=0 (#57044)
904fe1d 
Sort optimization creates TopFieldCollector that errors
when size=0. This ensures that sort optimization is not
run when size=0.

Closes #56923
mayya-sharipova added a commit that referenced this issue May 21, 2020
@mayya-sharipova
Don't run sort optimization on size=0 (#57044)
676c8ae 
Sort optimization creates TopFieldCollector that errors
when size=0. This ensures that sort optimization is not
run when size=0.

Closes #56923
@g-w g-w mentioned this issue Jun 4, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
>bug :Search/Search Search-related issues that do not fall into other categories Team:Search Meta label for search team
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Don't run sort optimization on size=0 mayya-sharipova/elasticsearch
3 participants
@jtibshirani @elasticmachine @toby-jn