Implement hidden indices

docs/reference/index-modules.asciidoc

@@ -79,6 +79,13 @@ indices.
     Indicates whether <<query-filter-context, cached filters>> are pre-loaded for
     nested queries. Possible values are `true` (default) and `false`.
 
+`index.hidden`::
+
+    Indicates whether the index should be hidden by default. Hidden indices are not
+    returned by default when using a wildcard expression. This behavior is controlled
+    per request through the use of the `expand_wildcards` parameter. Possible values are
+    `true` and `false` (default).
+
 [float]
 [[dynamic-index-settings]]
 === Dynamic index settings

docs/reference/rest-api/common-parms.asciidoc

@@ -53,7 +53,7 @@ Specifies what to do when the request:
 --
 * Contains wildcard expressions and there are no {transforms} that match.
 * Contains the `_all` string or no identifiers and there are no matches.
-* Contains wildcard expressions and there are only partial matches. 
+* Contains wildcard expressions and there are only partial matches.
 
 The default value is `true`, which returns an empty `transforms` array when
 there are no matches and the subset of results when there are partial matches.
@@ -69,7 +69,7 @@ Specifies what to do when the request:
 --
 * Contains wildcard expressions and there are no {transforms} that match.
 * Contains the `_all` string or no identifiers and there are no matches.
-* Contains wildcard expressions and there are only partial matches. 
+* Contains wildcard expressions and there are only partial matches.
 
 The default value is `true`, which returns a successful acknowledgement message
 when there are no matches. When there are only partial matches, the API stops
@@ -90,7 +90,7 @@ end::analyzer[]
 
 tag::analyze_wildcard[]
 `analyze_wildcard`::
-(Optional, boolean) If `true`, wildcard and prefix queries are 
+(Optional, boolean) If `true`, wildcard and prefix queries are
 analyzed. Defaults to `false`.
 end::analyze_wildcard[]
 
@@ -118,7 +118,7 @@ end::completion-fields[]
 
 tag::default_operator[]
 `default_operator`::
-(Optional, string) The default operator for query string query: AND or OR. 
+(Optional, string) The default operator for query string query: AND or OR.
 Defaults to `OR`.
 end::default_operator[]
 
@@ -144,7 +144,7 @@ end::detailed[]
 
 tag::df[]
 `df`::
-(Optional, string) Field to use as default where no field prefix is 
+(Optional, string) Field to use as default where no field prefix is
 given in the query string.
 end::df[]
 
@@ -181,14 +181,17 @@ Expand only to open indices.
 `closed`::
 Expand only to closed indices.
 
+`hidden`::
+Expansion of wildcards will include hidden indices.
+
 `none`::
 Wildcard expressions are not accepted.
 --
 end::expand-wildcards[]
 
 tag::field_statistics[]
 `field_statistics`::
-(Optional, boolean) If `true`, the response includes the document count, sum of document frequencies, 
+(Optional, boolean) If `true`, the response includes the document count, sum of document frequencies,
 and sum of total term frequencies.
 Defaults to `true`.
 end::field_statistics[]
@@ -297,9 +300,9 @@ end::help[]
 
 tag::bulk-id[]
 `_id`::
-(Optional, string) 
+(Optional, string)
 The document ID.
-If no ID is specified, a document ID is automatically generated. 
+If no ID is specified, a document ID is automatically generated.
 end::bulk-id[]
 
 tag::if_primary_term[]
@@ -316,7 +319,7 @@ end::if_seq_no[]
 
 tag::ignore_throttled[]
 `ignore_throttled`::
-(Optional, boolean) If `true`, concrete, expanded or aliased indices are 
+(Optional, boolean) If `true`, concrete, expanded or aliased indices are
 ignored when throttled.
 end::ignore_throttled[]
 
@@ -362,7 +365,7 @@ end::index[]
 tag::bulk-index[]
 `_index`::
 (Optional, string)
-The name of the target index. 
+The name of the target index.
 Required if not specified as a path parameter.
 end::bulk-index[]
 
@@ -447,7 +450,7 @@ end::index-template[]
 
 tag::lenient[]
 `lenient`::
-(Optional, boolean) If `true`, format-based query failures (such as 
+(Optional, boolean) If `true`, format-based query failures (such as
 providing text to a numeric field) will be ignored. Defaults to `false`.
 end::lenient[]
 
@@ -618,13 +621,13 @@ end::search-q[]
 
 tag::query[]
 `query`::
-(Optional, <<query-dsl,query object>>) Defines the search definition using the 
+(Optional, <<query-dsl,query object>>) Defines the search definition using the
 <<query-dsl,Query DSL>>.
 end::query[]
 
 tag::realtime[]
 `realtime`::
-(Optional, boolean) If `true`, the request is real-time as opposed to near-real-time. 
+(Optional, boolean) If `true`, the request is real-time as opposed to near-real-time.
 Defaults to `true`. See <<realtime>>.
 end::realtime[]
 
@@ -638,7 +641,7 @@ end::refresh[]
 
 tag::request_cache[]
 `request_cache`::
-(Optional, boolean) If `true`, the request cache is used for this request. 
+(Optional, boolean) If `true`, the request cache is used for this request.
 Defaults to the index-level setting.
 end::request_cache[]
 
@@ -668,14 +671,14 @@ end::cat-s[]
 
 tag::scroll[]
 `scroll`::
-(Optional, <<time-units, time units>>) Specifies how long a consistent view of 
+(Optional, <<time-units, time units>>) Specifies how long a consistent view of
 the index should be maintained for scrolled search.
 end::scroll[]
 
 tag::scroll_size[]
 `scroll_size`::
-(Optional, integer) Size of the scroll request that powers the operation. 
-Defaults to 100. 
+(Optional, integer) Size of the scroll request that powers the operation.
+Defaults to 100.
 end::scroll_size[]
 
 tag::search_timeout[]
@@ -731,7 +734,7 @@ end::size-transforms[]
 
 tag::slices[]
 `slices`::
-(Optional, integer) The number of slices this task should be divided into. 
+(Optional, integer) The number of slices this task should be divided into.
 Defaults to 1 meaning the task isn't sliced into subtasks.
 end::slices[]
 
@@ -742,24 +745,24 @@ end::sort[]
 
 tag::source[]
 `_source`::
-(Optional, string) True or false to return the `_source` field or not, or a 
+(Optional, string) True or false to return the `_source` field or not, or a
 list of fields to return.
 end::source[]
 
 tag::source_excludes[]
 `_source_excludes`::
-(Optional, string) A list of fields to exclude from the returned `_source` 
+(Optional, string) A list of fields to exclude from the returned `_source`
 field.
 end::source_excludes[]
 
 tag::source_includes[]
 `_source_includes`::
-(Optional, string) A list of fields to extract and return from the `_source` 
+(Optional, string) A list of fields to extract and return from the `_source`
 field.
 end::source_includes[]
 
 tag::source-transforms[]
-The source of the data for the {transform}. 
+The source of the data for the {transform}.
 end::source-transforms[]
 
 tag::source-index-transforms[]
@@ -823,13 +826,13 @@ end::task-id[]
 
 tag::term_statistics[]
 `term_statistics`::
-(Optional, boolean) If `true`, the response includes term frequency and document frequency. 
+(Optional, boolean) If `true`, the response includes term frequency and document frequency.
 Defaults to `false`.
 end::term_statistics[]
 
 tag::terminate_after[]
 `terminate_after`::
-(Optional, integer) The maximum number of documents to collect for each shard, 
+(Optional, integer) The maximum number of documents to collect for each shard,
 upon reaching which the query execution will terminate early.
 end::terminate_after[]
 
@@ -871,7 +874,7 @@ end::transform-id-wildcard[]
 
 tag::cat-v[]
 `v`::
-(Optional, boolean) If `true`, the response includes column headings. 
+(Optional, boolean) If `true`, the response includes column headings.
 Defaults to `false`.
 end::cat-v[]
 
@@ -912,6 +915,6 @@ end::wait_for_active_shards[]
 
 tag::wait_for_completion[]
 `wait_for_completion`::
-(Optional, boolean) If `true`, the request blocks until the operation is complete. 
+(Optional, boolean) If `true`, the request blocks until the operation is complete.
 Defaults to `true`.
 end::wait_for_completion[]

server/src/main/java/org/elasticsearch/action/admin/cluster/health/ClusterHealthRequest.java

@@ -39,7 +39,7 @@
 public class ClusterHealthRequest extends MasterNodeReadRequest<ClusterHealthRequest> implements IndicesRequest.Replaceable {
 
     private String[] indices;
-    private IndicesOptions indicesOptions = IndicesOptions.lenientExpand();
+    private IndicesOptions indicesOptions = IndicesOptions.lenientExpandHidden();
     private TimeValue timeout = new TimeValue(30, TimeUnit.SECONDS);
     private ClusterHealthStatus waitForStatus;
     private boolean waitForNoRelocatingShards = false;

server/src/main/java/org/elasticsearch/action/admin/indices/rollover/TransportRolloverAction.java

@@ -43,7 +43,6 @@
 import org.elasticsearch.cluster.metadata.MetaData;
 import org.elasticsearch.cluster.metadata.MetaDataCreateIndexService;
 import org.elasticsearch.cluster.metadata.MetaDataIndexAliasesService;
-import org.elasticsearch.cluster.metadata.MetaDataIndexTemplateService;
 import org.elasticsearch.cluster.service.ClusterService;
 import org.elasticsearch.common.Nullable;
 import org.elasticsearch.common.inject.Inject;
@@ -63,6 +62,8 @@
 import java.util.regex.Pattern;
 import java.util.stream.Collectors;
 
+import static org.elasticsearch.cluster.metadata.MetaDataIndexTemplateService.findTemplates;
+
 /**
  * Main class to swap the index pointed to by an alias, given some conditions
  */
@@ -122,7 +123,9 @@ protected void masterOperation(Task task, final RolloverRequest rolloverRequest,
             : generateRolloverIndexName(sourceProvidedName, indexNameExpressionResolver);
         final String rolloverIndexName = indexNameExpressionResolver.resolveDateMathExpression(unresolvedName);
         MetaDataCreateIndexService.validateIndexName(rolloverIndexName, state); // will fail if the index already exists
-        checkNoDuplicatedAliasInIndexTemplate(metaData, rolloverIndexName, rolloverRequest.getAlias());
+        final Boolean isHidden = IndexMetaData.INDEX_HIDDEN_SETTING.exists(rolloverRequest.getCreateIndexRequest().settings()) ?
+            IndexMetaData.INDEX_HIDDEN_SETTING.get(rolloverRequest.getCreateIndexRequest().settings()) : null;
+        checkNoDuplicatedAliasInIndexTemplate(metaData, rolloverIndexName, rolloverRequest.getAlias(), isHidden);
         IndicesStatsRequest statsRequest = new IndicesStatsRequest().indices(rolloverRequest.getAlias())
             .clear()
             .indicesOptions(IndicesOptions.fromOptions(true, false, true, true))
@@ -291,8 +294,9 @@ static CreateIndexClusterStateUpdateRequest prepareCreateIndexRequest(final Stri
      * the rollover alias will point to multiple indices. This causes indexing requests to be rejected.
      * To avoid this, we make sure that there is no duplicated alias in index templates before creating a new index.
      */
-    static void checkNoDuplicatedAliasInIndexTemplate(MetaData metaData, String rolloverIndexName, String rolloverRequestAlias) {
-        final List<IndexTemplateMetaData> matchedTemplates = MetaDataIndexTemplateService.findTemplates(metaData, rolloverIndexName);
+    static void checkNoDuplicatedAliasInIndexTemplate(MetaData metaData, String rolloverIndexName, String rolloverRequestAlias,
+                                                      @Nullable Boolean isHidden) {
+        final List<IndexTemplateMetaData> matchedTemplates = findTemplates(metaData, rolloverIndexName, isHidden);
         for (IndexTemplateMetaData template : matchedTemplates) {
             if (template.aliases().containsKey(rolloverRequestAlias)) {
                 throw new IllegalArgumentException(String.format(Locale.ROOT,

server/src/main/java/org/elasticsearch/action/bulk/TransportBulkAction.java

@@ -307,7 +307,7 @@ static boolean resolvePipelines(final DocWriteRequest<?> originalRequest, final
                 }
             } else if (indexRequest.index() != null) {
                 // the index does not exist yet (and this is a valid request), so match index templates to look for pipelines
-                List<IndexTemplateMetaData> templates = MetaDataIndexTemplateService.findTemplates(metaData, indexRequest.index());
+                List<IndexTemplateMetaData> templates = MetaDataIndexTemplateService.findTemplates(metaData, indexRequest.index(), null);
                 assert (templates != null);
                 // order of templates are highest order first
                 for (final IndexTemplateMetaData template : templates) {

server/src/main/java/org/elasticsearch/action/search/MultiSearchRequest.java

@@ -23,6 +23,7 @@
 import org.elasticsearch.action.ActionRequestValidationException;
 import org.elasticsearch.action.CompositeIndicesRequest;
 import org.elasticsearch.action.support.IndicesOptions;
+import org.elasticsearch.action.support.IndicesOptions.WildcardStates;
 import org.elasticsearch.common.CheckedBiConsumer;
 import org.elasticsearch.common.bytes.BytesReference;
 import org.elasticsearch.common.io.stream.StreamInput;
@@ -304,15 +305,7 @@ public static void writeSearchRequestParams(SearchRequest request, XContentBuild
             xContentBuilder.field("index", request.indices());
         }
         if (request.indicesOptions() != null && request.indicesOptions() != SearchRequest.DEFAULT_INDICES_OPTIONS) {
-            if (request.indicesOptions().expandWildcardsOpen() && request.indicesOptions().expandWildcardsClosed()) {
-                xContentBuilder.field("expand_wildcards", "all");
-            } else if (request.indicesOptions().expandWildcardsOpen()) {
-                xContentBuilder.field("expand_wildcards", "open");
-            } else if (request.indicesOptions().expandWildcardsClosed()) {
-                xContentBuilder.field("expand_wildcards", "closed");
-            } else {
-                xContentBuilder.field("expand_wildcards", "none");
-            }
+            WildcardStates.toXContent(request.indicesOptions().getExpandWildcards(), xContentBuilder);
             xContentBuilder.field("ignore_unavailable", request.indicesOptions().ignoreUnavailable());
             xContentBuilder.field("allow_no_indices", request.indicesOptions().allowNoIndices());
         }