done plugins k-NN libraries not found in LD_LIBRARY_PATH. Updating path to: :/tmp/tmpmpoc3k76/1/local-test-cluster/opensearch-2.12.0/plugins/opensearch-knn/lib. Starting OpenSearch [2024-02-06T19:39:51,997][INFO ][o.o.n.Node ] [node_name_9200] version[2.12.0], pid[19875], build[tar/f67ed1796749376401c5cc617eff2b7903325a9d/2024-02-06T15:43:50.690828893Z], OS[Linux/5.10.205-172.807.amzn2int.x86_64/amd64], JVM[Amazon.com Inc./OpenJDK 64-Bit Server VM/21.0.2/21.0.2+13-LTS] [2024-02-06T19:39:51,999][INFO ][o.o.n.Node ] [node_name_9200] JVM home [/local/home/gaievski/.sdkman/candidates/java/21.0.2-amzn], using bundled JDK/JRE [false] [2024-02-06T19:39:52,000][INFO ][o.o.n.Node ] [node_name_9200] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.security.manager=allow, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-13612305738166679138, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.security.manager=allow, --add-modules=jdk.incubator.vector, -Djava.util.concurrent.ForkJoinPool.common.threadFactory=org.opensearch.secure_sm.SecuredForkJoinWorkerThreadFactory, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=/tmp/tmpmpoc3k76/1/local-test-cluster/opensearch-2.12.0/config/opensearch-performance-analyzer/opensearch_security.policy, --add-opens=jdk.attach/sun.tools.attach=ALL-UNNAMED, -XX:MaxDirectMemorySize=536870912, -Dopensearch.path.home=/tmp/tmpmpoc3k76/1/local-test-cluster/opensearch-2.12.0, -Dopensearch.path.conf=/tmp/tmpmpoc3k76/1/local-test-cluster/opensearch-2.12.0/config, -Dopensearch.distribution.type=tar, -Dopensearch.bundled_jdk=true] [2024-02-06T19:39:52,775][INFO ][o.o.i.r.ReindexPlugin ] [node_name_9200] ReindexPlugin reloadSPI called [2024-02-06T19:39:52,777][INFO ][o.o.i.r.ReindexPlugin ] [node_name_9200] Unable to find any implementation for RemoteReindexExtension [2024-02-06T19:39:52,799][INFO ][o.o.p.PluginsService ] [node_name_9200] loaded module [aggs-matrix-stats] [2024-02-06T19:39:52,799][INFO ][o.o.p.PluginsService ] [node_name_9200] loaded module [analysis-common] [2024-02-06T19:39:52,800][INFO ][o.o.p.PluginsService ] [node_name_9200] loaded module [geo] [2024-02-06T19:39:52,800][INFO ][o.o.p.PluginsService ] [node_name_9200] loaded module [ingest-common] [2024-02-06T19:39:52,800][INFO ][o.o.p.PluginsService ] [node_name_9200] loaded module [ingest-geoip] [2024-02-06T19:39:52,800][INFO ][o.o.p.PluginsService ] [node_name_9200] loaded module [ingest-user-agent] [2024-02-06T19:39:52,801][INFO ][o.o.p.PluginsService ] [node_name_9200] loaded module [lang-expression] [2024-02-06T19:39:52,801][INFO ][o.o.p.PluginsService ] [node_name_9200] loaded module [lang-mustache] [2024-02-06T19:39:52,801][INFO ][o.o.p.PluginsService ] [node_name_9200] loaded module [lang-painless] [2024-02-06T19:39:52,801][INFO ][o.o.p.PluginsService ] [node_name_9200] loaded module [mapper-extras] [2024-02-06T19:39:52,801][INFO ][o.o.p.PluginsService ] [node_name_9200] loaded module [opensearch-dashboards] [2024-02-06T19:39:52,802][INFO ][o.o.p.PluginsService ] [node_name_9200] loaded module [parent-join] [2024-02-06T19:39:52,802][INFO ][o.o.p.PluginsService ] [node_name_9200] loaded module [percolator] [2024-02-06T19:39:52,802][INFO ][o.o.p.PluginsService ] [node_name_9200] loaded module [rank-eval] [2024-02-06T19:39:52,802][INFO ][o.o.p.PluginsService ] [node_name_9200] loaded module [reindex] [2024-02-06T19:39:52,803][INFO ][o.o.p.PluginsService ] [node_name_9200] loaded module [repository-url] [2024-02-06T19:39:52,803][INFO ][o.o.p.PluginsService ] [node_name_9200] loaded module [search-pipeline-common] [2024-02-06T19:39:52,803][INFO ][o.o.p.PluginsService ] [node_name_9200] loaded module [systemd] [2024-02-06T19:39:52,803][INFO ][o.o.p.PluginsService ] [node_name_9200] loaded module [transport-netty4] [2024-02-06T19:39:52,804][INFO ][o.o.p.PluginsService ] [node_name_9200] loaded plugin [opensearch-job-scheduler] [2024-02-06T19:39:52,804][INFO ][o.o.p.PluginsService ] [node_name_9200] loaded plugin [opensearch-knn] [2024-02-06T19:39:52,804][INFO ][o.o.p.PluginsService ] [node_name_9200] loaded plugin [opensearch-ml] [2024-02-06T19:39:52,804][INFO ][o.o.p.PluginsService ] [node_name_9200] loaded plugin [opensearch-neural-search] [2024-02-06T19:39:52,851][INFO ][o.o.e.ExtensionsManager ] [node_name_9200] ExtensionsManager initialized [2024-02-06T19:39:52,860][INFO ][o.a.l.s.MemorySegmentIndexInputProvider] [node_name_9200] Using MemorySegmentIndexInput with Java 21; to disable start with -Dorg.apache.lucene.store.MMapDirectory.enableMemorySegments=false [2024-02-06T19:39:52,866][INFO ][o.o.e.NodeEnvironment ] [node_name_9200] using [1] data paths, mounts [[/ (/dev/nvme0n1p1)]], net usable_space [278.1gb], net total_space [492gb], types [ext4] [2024-02-06T19:39:52,867][INFO ][o.o.e.NodeEnvironment ] [node_name_9200] heap size [1gb], compressed ordinary object pointers [true] [2024-02-06T19:39:52,891][INFO ][o.o.n.Node ] [node_name_9200] node name [node_name_9200], node ID [OAYF21-xRbu9dc9StXHecg], cluster name [opensearchcluster1], roles [ingest, remote_cluster_client, data, cluster_manager] [2024-02-06T19:39:54,748][INFO ][o.o.n.p.NeuralSearch ] [node_name_9200] Registering hybrid query phase searcher with feature flag [plugins.neural_search.hybrid_search_disabled] [2024-02-06T19:39:55,203][INFO ][o.o.m.b.MLCircuitBreakerService] [node_name_9200] Registered ML memory breaker. [2024-02-06T19:39:55,203][INFO ][o.o.m.b.MLCircuitBreakerService] [node_name_9200] Registered ML disk breaker. [2024-02-06T19:39:55,204][INFO ][o.o.m.b.MLCircuitBreakerService] [node_name_9200] Registered ML native memory breaker. [2024-02-06T19:39:55,290][INFO ][o.r.Reflections ] [node_name_9200] Reflections took 42 ms to scan 1 urls, producing 21 keys and 58 values [2024-02-06T19:39:55,511][INFO ][o.o.t.NettyAllocator ] [node_name_9200] creating NettyAllocator with the following configs: [name=unpooled, suggested_max_allocation_size=256kb, factors={opensearch.unsafe.use_unpooled_allocator=null, g1gc_enabled=true, g1gc_region_size=1mb, heap_size=1gb}] [2024-02-06T19:39:55,593][INFO ][o.o.d.DiscoveryModule ] [node_name_9200] using discovery type [zen] and seed hosts providers [settings] [2024-02-06T19:39:55,886][WARN ][o.o.g.DanglingIndicesState] [node_name_9200] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually [2024-02-06T19:39:56,192][INFO ][o.o.n.Node ] [node_name_9200] initialized [2024-02-06T19:39:56,192][INFO ][o.o.n.Node ] [node_name_9200] starting ... [2024-02-06T19:39:56,398][INFO ][o.o.t.TransportService ] [node_name_9200] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300} [2024-02-06T19:39:56,400][INFO ][o.o.t.TransportService ] [node_name_9200] Remote clusters initialized successfully. [2024-02-06T19:39:56,532][WARN ][o.o.b.BootstrapChecks ] [node_name_9200] max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144] [2024-02-06T19:39:56,533][WARN ][o.o.b.BootstrapChecks ] [node_name_9200] the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_cluster_manager_nodes / cluster.initial_master_nodes] must be configured [2024-02-06T19:39:56,539][INFO ][o.o.c.c.ClusterBootstrapService] [node_name_9200] no discovery configuration found, will perform best-effort cluster bootstrapping after [3s] unless existing cluster-manager is discovered [2024-02-06T19:39:59,542][INFO ][o.o.c.c.Coordinator ] [node_name_9200] setting initial configuration to VotingConfiguration{OAYF21-xRbu9dc9StXHecg} [2024-02-06T19:39:59,654][INFO ][o.o.c.s.MasterService ] [node_name_9200] elected-as-cluster-manager ([1] nodes joined)[{node_name_9200}{OAYF21-xRbu9dc9StXHecg}{oDgJYYhSQuOXUd4-9IIZRA}{127.0.0.1}{127.0.0.1:9300}{dimr}{shard_indexing_pressure_enabled=true} elect leader, _BECOME_CLUSTER_MANAGER_TASK_, _FINISH_ELECTION_], term: 1, version: 1, delta: cluster-manager node changed {previous [], current [{node_name_9200}{OAYF21-xRbu9dc9StXHecg}{oDgJYYhSQuOXUd4-9IIZRA}{127.0.0.1}{127.0.0.1:9300}{dimr}{shard_indexing_pressure_enabled=true}]} [2024-02-06T19:39:59,685][INFO ][o.o.c.c.CoordinationState] [node_name_9200] cluster UUID set to [3QSda_kXRyKiN6dL5zsbRA] [2024-02-06T19:39:59,704][INFO ][o.o.c.s.ClusterApplierService] [node_name_9200] cluster-manager node changed {previous [], current [{node_name_9200}{OAYF21-xRbu9dc9StXHecg}{oDgJYYhSQuOXUd4-9IIZRA}{127.0.0.1}{127.0.0.1:9300}{dimr}{shard_indexing_pressure_enabled=true}]}, term: 1, version: 1, reason: Publication{term=1, version=1} [2024-02-06T19:39:59,715][INFO ][o.o.d.PeerFinder ] [node_name_9200] setting findPeersInterval to [1s] as node commission status = [true] for local node [{node_name_9200}{OAYF21-xRbu9dc9StXHecg}{oDgJYYhSQuOXUd4-9IIZRA}{127.0.0.1}{127.0.0.1:9300}{dimr}{shard_indexing_pressure_enabled=true}] [2024-02-06T19:39:59,719][INFO ][o.o.h.AbstractHttpServerTransport] [node_name_9200] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200} [2024-02-06T19:39:59,719][INFO ][o.o.n.Node ] [node_name_9200] started [2024-02-06T19:39:59,728][INFO ][o.o.g.GatewayService ] [node_name_9200] recovered [0] indices into cluster_state [2024-02-06T19:40:09,724][INFO ][o.o.m.a.MLModelAutoReDeployer] [node_name_9200] Index not found, not performing auto reloading! [2024-02-06T19:40:09,726][INFO ][o.o.m.c.MLCommonsClusterManagerEventListener] [node_name_9200] Starting ML sync up job... [2024-02-06T19:40:19,750][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[.plugins-ml-config/10LKspi_RPavNCn18x3RkA] [2024-02-06T19:40:19,846][INFO ][o.o.c.m.MetadataCreateIndexService] [node_name_9200] [.plugins-ml-config] creating index, cause [api], templates [], shards [1]/[1] [2024-02-06T19:40:19,852][INFO ][o.o.c.r.a.AllocationService] [node_name_9200] updating number_of_replicas to [0] for indices [.plugins-ml-config] [2024-02-06T19:40:20,035][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[.plugins-ml-config/10LKspi_RPavNCn18x3RkA] [2024-02-06T19:40:20,382][INFO ][o.o.c.r.a.AllocationService] [node_name_9200] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.plugins-ml-config][0]]]). [2024-02-06T19:40:20,406][INFO ][o.o.m.e.i.MLIndicesHandler] [node_name_9200] create index:.plugins-ml-config [2024-02-06T19:40:20,490][INFO ][o.o.m.c.MLSyncUpCron ] [node_name_9200] ML configuration initialized successfully [2024-02-06T19:40:37,737][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[neuralsearchknnindexforvalidation/ZMa-6H0xRXa1RANaHTybiQ] [2024-02-06T19:40:37,752][INFO ][o.o.c.m.MetadataCreateIndexService] [node_name_9200] [neuralsearchknnindexforvalidation] creating index, cause [api], templates [], shards [1]/[1] [2024-02-06T19:40:37,775][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[neuralsearchknnindexforvalidation/ZMa-6H0xRXa1RANaHTybiQ] [2024-02-06T19:40:37,877][INFO ][o.a.l.i.v.PanamaVectorizationProvider] [node_name_9200] Java vector incubator API enabled; uses preferredBitSize=512; FMA enabled [2024-02-06T19:40:37,915][INFO ][o.o.c.m.MetadataDeleteIndexService] [node_name_9200] [neuralsearchknnindexforvalidation/ZMa-6H0xRXa1RANaHTybiQ] deleting index [2024-02-06T19:40:38,534][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [true] to [false] [2024-02-06T19:40:38,534][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [true] to [false] [2024-02-06T19:40:38,585][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.allow_registering_model_via_url] from [false] to [true] [2024-02-06T19:40:38,622][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[my-nlp-index/4U_0jMyZQyOwob6Pf4kf_Q] [2024-02-06T19:40:38,625][INFO ][o.o.c.m.MetadataCreateIndexService] [node_name_9200] [my-nlp-index] creating index, cause [api], templates [], shards [3]/[1] [2024-02-06T19:40:38,644][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[my-nlp-index/4U_0jMyZQyOwob6Pf4kf_Q] [2024-02-06T19:40:38,824][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[.plugins-ml-model-group/I4dVheIbQtCK5GqformioQ] [2024-02-06T19:40:38,839][INFO ][o.o.c.m.MetadataCreateIndexService] [node_name_9200] [.plugins-ml-model-group] creating index, cause [api], templates [], shards [1]/[1] [2024-02-06T19:40:38,839][INFO ][o.o.c.r.a.AllocationService] [node_name_9200] updating number_of_replicas to [0] for indices [.plugins-ml-model-group] [2024-02-06T19:40:38,858][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[.plugins-ml-model-group/I4dVheIbQtCK5GqformioQ] [2024-02-06T19:40:38,918][INFO ][o.o.m.e.i.MLIndicesHandler] [node_name_9200] create index:.plugins-ml-model-group [2024-02-06T19:40:38,953][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[.plugins-ml-task/eSJNhk7IRDiwJKYLGhIEXw] [2024-02-06T19:40:38,959][INFO ][o.o.c.m.MetadataCreateIndexService] [node_name_9200] [.plugins-ml-task] creating index, cause [api], templates [], shards [1]/[1] [2024-02-06T19:40:38,960][INFO ][o.o.c.r.a.AllocationService] [node_name_9200] updating number_of_replicas to [0] for indices [.plugins-ml-task] [2024-02-06T19:40:38,979][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[.plugins-ml-task/eSJNhk7IRDiwJKYLGhIEXw] [2024-02-06T19:40:39,037][INFO ][o.o.m.e.i.MLIndicesHandler] [node_name_9200] create index:.plugins-ml-task [2024-02-06T19:40:39,099][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[.plugins-ml-model/NeKKIo51S7SmdhXE1dM0qA] [2024-02-06T19:40:39,107][INFO ][o.o.c.m.MetadataCreateIndexService] [node_name_9200] [.plugins-ml-model] creating index, cause [api], templates [], shards [1]/[1] [2024-02-06T19:40:39,108][INFO ][o.o.c.r.a.AllocationService] [node_name_9200] updating number_of_replicas to [0] for indices [.plugins-ml-model] [2024-02-06T19:40:39,126][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[.plugins-ml-model/NeKKIo51S7SmdhXE1dM0qA] [2024-02-06T19:40:39,186][INFO ][o.o.m.e.i.MLIndicesHandler] [node_name_9200] create index:.plugins-ml-model [2024-02-06T19:40:39,204][INFO ][o.o.m.m.MLModelManager ] [node_name_9200] create new model meta doc wpDvf40Bamv3vXxyz9iU for register model task wZDvf40Bamv3vXxyztj- [2024-02-06T19:40:40,007][INFO ][stdout ] [node_name_9200] Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 1% |█ | traced_small_model.zip Downloading: 2% |█ | traced_small_model.zip Downloading: 3% |██ | traced_small_model.zip Downloading: 4% |██ | traced_small_model.zip Downloading: 5% |███ | traced_small_model.zip Downloading: 6% |███ | traced_small_model.zip Downloading: 7% |███ | traced_small_model.zip Downloading: 8% |████ | traced_small_model.zip Downloading: 9% |████ | traced_small_model.zip Downloading: 10% |█████ | traced_small_model.zip Downloading: 11% |█████ | traced_small_model.zip Downloading: 12% |█████ | traced_small_model.zip Downloading: 13% |██████ | traced_small_model.zip Downloading: 14% |██████ | traced_small_model.zip Downloading: 15% |███████ | traced_small_model.zip Downloading: 16% |███████ | traced_small_model.zip Downloading: 17% |███████ | traced_small_model.zip Downloading: 18% |████████ | traced_small_model.zip Downloading: 19% |████████ | traced_small_model.zip Downloading: 20% |█████████ | traced_small_model.zip Downloading: 21% |█████████ | traced_small_model.zip Downloading: 22% |█████████ | traced_small_model.zip Downloading: 23% |██████████ | traced_small_model.zip Downloading: 24% |██████████ | traced_small_model.zip Downloading: 25% |███████████ | traced_small_model.zip Downloading: 26% |███████████ | traced_small_model.zip Downloading: 27% |███████████ | traced_small_model.zip Downloading: 28% |████████████ | traced_small_model.zip Downloading: 29% |████████████ | traced_small_model.zip Downloading: 30% |█████████████ | traced_small_model.zip Downloading: 31% |█████████████ | traced_small_model.zip Downloading: 32% |█████████████ | traced_small_model.zip Downloading: 33% |██████████████ | traced_small_model.zip Downloading: 34% |██████████████ | traced_small_model.zip Downloading: 35% |███████████████ | traced_small_model.zip Downloading: 36% |███████████████ | traced_small_model.zip Downloading: 37% |███████████████ | traced_small_model.zip Downloading: 38% |████████████████ | traced_small_model.zip Downloading: 39% |████████████████ | traced_small_model.zip Downloading: 40% |█████████████████ | traced_small_model.zip Downloading: 41% |█████████████████ | traced_small_model.zip Downloading: 42% |█████████████████ | traced_small_model.zip Downloading: 43% |██████████████████ | traced_small_model.zip Downloading: 44% |██████████████████ | traced_small_model.zip Downloading: 45% |███████████████████ | traced_small_model.zip Downloading: 46% |███████████████████ | traced_small_model.zip Downloading: 47% |███████████████████ | traced_small_model.zip Downloading: 48% |████████████████████ | traced_small_model.zip Downloading: 49% |████████████████████ | traced_small_model.zip Downloading: 50% |█████████████████████ | traced_small_model.zip Downloading: 51% |█████████████████████ | traced_small_model.zip Downloading: 52% |█████████████████████ | traced_small_model.zip Downloading: 53% |██████████████████████ | traced_small_model.zip Downloading: 54% |██████████████████████ | traced_small_model.zip Downloading: 55% |███████████████████████ | traced_small_model.zip Downloading: 56% |███████████████████████ | traced_small_model.zip Downloading: 57% |███████████████████████ | traced_small_model.zip Downloading: 58% |████████████████████████ | traced_small_model.zip Downloading: 59% |████████████████████████ | traced_small_model.zip Downloading: 60% |█████████████████████████ | traced_small_model.zip Downloading: 61% |█████████████████████████ | traced_small_model.zip Downloading: 62% |█████████████████████████ | traced_small_model.zip Downloading: 63% |██████████████████████████ | traced_small_model.zip Downloading: 64% |██████████████████████████ | traced_small_model.zip Downloading: 65% |███████████████████████████ | traced_small_model.zip Downloading: 66% |███████████████████████████ | traced_small_model.zip Downloading: 67% |███████████████████████████ | traced_small_model.zip Downloading: 68% |████████████████████████████ | traced_small_model.zip Downloading: 69% |████████████████████████████ | traced_small_model.zip Downloading: 70% |█████████████████████████████ | traced_small_model.zip Downloading: 71% |█████████████████████████████ | traced_small_model.zip Downloading: 72% |█████████████████████████████ | traced_small_model.zip Downloading: 73% |██████████████████████████████ | traced_small_model.zip Downloading: 74% |██████████████████████████████ | traced_small_model.zip Downloading: 75% |███████████████████████████████ | traced_small_model.zip Downloading: 76% |███████████████████████████████ | traced_small_model.zip Downloading: 77% |███████████████████████████████ | traced_small_model.zip Downloading: 78% |████████████████████████████████ | traced_small_model.zip Downloading: 79% |████████████████████████████████ | traced_small_model.zip Downloading: 80% |█████████████████████████████████ | traced_small_model.zip Downloading: 81% |█████████████████████████████████ | traced_small_model.zip Downloading: 82% |█████████████████████████████████ | traced_small_model.zip Downloading: 83% |██████████████████████████████████ | traced_small_model.zip Downloading: 84% |██████████████████████████████████ | traced_small_model.zip Downloading: 85% |███████████████████████████████████ | traced_small_model.zip Downloading: 86% |███████████████████████████████████ | traced_small_model.zip Downloading: 87% |███████████████████████████████████ | traced_small_model.zip Downloading: 88% |████████████████████████████████████ | traced_small_model.zip Downloading: 89% |████████████████████████████████████ | traced_small_model.zip Downloading: 90% |█████████████████████████████████████ | traced_small_model.zip Downloading: 91% |█████████████████████████████████████ | traced_small_model.zip Downloading: 92% |█████████████████████████████████████ | traced_small_model.zip Downloading: 93% |██████████████████████████████████████ | traced_small_model.zip Downloading: 94% |██████████████████████████████████████ | traced_small_model.zip Downloading: 95% |███████████████████████████████████████ | traced_small_model.zip Downloading: 96% |███████████████████████████████████████ | traced_small_model.zip Downloading: 97% |███████████████████████████████████████ | traced_small_model.zip Downloading: 98% |████████████████████████████████████████| traced_small_model.zip Downloading: 99% |████████████████████████████████████████| traced_small_model.zip Downloading: 100% |████████████████████████████████████████| traced_small_model.zip [2024-02-06T19:40:40,285][INFO ][o.o.m.m.MLModelManager ] [node_name_9200] Model registered successfully, model id: wpDvf40Bamv3vXxyz9iU, task id: wZDvf40Bamv3vXxyztj- [2024-02-06T19:40:42,080][INFO ][o.o.m.a.d.TransportDeployModelAction] [node_name_9200] Will deploy model on these nodes: OAYF21-xRbu9dc9StXHecg [2024-02-06T19:40:42,138][ERROR][o.o.m.m.MLModelManager ] [node_name_9200] No controller is deployed because the model wpDvf40Bamv3vXxyz9iU is expected not having an enabled model controller. Please use the create model controller api to create one if this is unexpected. [2024-02-06T19:40:42,419][INFO ][a.d.u.Ec2Utils ] [node_name_9200] DJL will collect telemetry to help us better understand our users’ needs, diagnose issues, and deliver additional features. If you would like to learn more or opt-out please go to: https://docs.djl.ai/docs/telemetry.html for more information. [2024-02-06T19:40:42,426][WARN ][a.d.u.c.CudaUtils ] [node_name_9200] Access denied during loading cudart library. [2024-02-06T19:40:42,662][INFO ][a.d.p.j.LibUtils ] [node_name_9200] Downloading https://publish.djl.ai/pytorch/1.13.1/cpu-precxx11/linux-x86_64/native/lib/libgomp-a34b3233.so.1.gz ... [2024-02-06T19:40:42,680][INFO ][a.d.p.j.LibUtils ] [node_name_9200] Downloading https://publish.djl.ai/pytorch/1.13.1/cpu-precxx11/linux-x86_64/native/lib/libc10.so.gz ... [2024-02-06T19:40:42,709][INFO ][a.d.p.j.LibUtils ] [node_name_9200] Downloading https://publish.djl.ai/pytorch/1.13.1/cpu-precxx11/linux-x86_64/native/lib/libtorch_cpu.so.gz ... [2024-02-06T19:40:46,437][INFO ][a.d.p.j.LibUtils ] [node_name_9200] Downloading https://publish.djl.ai/pytorch/1.13.1/cpu-precxx11/linux-x86_64/native/lib/libtorch.so.gz ... [2024-02-06T19:40:46,445][INFO ][a.d.p.j.LibUtils ] [node_name_9200] Downloading https://publish.djl.ai/pytorch/1.13.1/cpu-precxx11/linux-x86_64/native/lib/libstdc%2B%2B.so.6.gz ... [2024-02-06T19:40:46,760][INFO ][a.d.p.j.LibUtils ] [node_name_9200] Downloading jni https://publish.djl.ai/pytorch/1.13.1/jnilib/0.21.0/linux-x86_64/cpu-precxx11/libdjl_torch.so to cache ... [2024-02-06T19:40:46,833][INFO ][a.d.p.e.PtEngine ] [node_name_9200] Number of inter-op threads is 1 [2024-02-06T19:40:46,834][INFO ][a.d.p.e.PtEngine ] [node_name_9200] Number of intra-op threads is 1 [2024-02-06T19:40:47,094][INFO ][a.d.h.t.j.LibUtils ] [node_name_9200] Extracting native/lib/linux-x86_64/libtokenizers.so to cache ... [2024-02-06T19:40:47,285][INFO ][o.o.m.e.a.DLModel ] [node_name_9200] Model wpDvf40Bamv3vXxyz9iU is successfully deployed on 1 devices [2024-02-06T19:40:47,288][INFO ][o.o.m.a.MLModelAutoReDeployer] [node_name_9200] No models needs to be auto redeployed! [2024-02-06T19:40:47,290][INFO ][o.o.m.a.f.TransportForwardAction] [node_name_9200] deploy model done with state: DEPLOYED, model id: wpDvf40Bamv3vXxyz9iU [2024-02-06T19:40:47,291][INFO ][o.o.m.a.d.TransportDeployModelOnNodeAction] [node_name_9200] deploy model task done w5Dvf40Bamv3vXxy2tji [2024-02-06T19:40:49,204][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[my-nlp-index/4U_0jMyZQyOwob6Pf4kf_Q] [2024-02-06T19:40:49,207][INFO ][o.o.c.s.IndexScopedSettings] [node_name_9200] [my-nlp-index] updating [index.search.default_pipeline] from [_none] to [search-pipeline] [2024-02-06T19:40:49,224][INFO ][o.o.c.s.IndexScopedSettings] [node_name_9200] [my-nlp-index] updating [index.search.default_pipeline] from [_none] to [search-pipeline] [2024-02-06T19:40:49,270][INFO ][o.r.Reflections ] [node_name_9200] Reflections took 7 ms to scan 1 urls, producing 6 keys and 34 values [2024-02-06T19:40:49,280][INFO ][o.r.Reflections ] [node_name_9200] Reflections took 6 ms to scan 1 urls, producing 8 keys and 38 values [2024-02-06T19:40:49,289][INFO ][o.r.Reflections ] [node_name_9200] Reflections took 6 ms to scan 1 urls, producing 5 keys and 17 values [2024-02-06T19:40:49,298][INFO ][o.r.Reflections ] [node_name_9200] Reflections took 6 ms to scan 1 urls, producing 5 keys and 11 values [2024-02-06T19:40:49,305][INFO ][o.r.Reflections ] [node_name_9200] Reflections took 5 ms to scan 1 urls, producing 5 keys and 15 values [2024-02-06T19:40:49,315][INFO ][o.r.Reflections ] [node_name_9200] Reflections took 7 ms to scan 1 urls, producing 11 keys and 54 values [2024-02-06T19:40:49,322][INFO ][o.r.Reflections ] [node_name_9200] Reflections took 4 ms to scan 1 urls, producing 11 keys and 24 values [2024-02-06T19:40:49,746][INFO ][o.o.m.c.MLSyncUpCron ] [node_name_9200] Refresh model state: {wpDvf40Bamv3vXxyz9iU=DEPLOY_FAILED} [2024-02-06T19:40:52,617][INFO ][o.o.m.a.m.DeleteModelTransportAction] [node_name_9200] Model controller not deleted due to no model controller found for model: wpDvf40Bamv3vXxyz9iU [2024-02-06T19:40:52,645][INFO ][o.o.c.m.MetadataDeleteIndexService] [node_name_9200] [my-nlp-index/4U_0jMyZQyOwob6Pf4kf_Q] deleting index [2024-02-06T19:40:52,718][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [false] to [true] [2024-02-06T19:40:52,719][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [false] to [true] [2024-02-06T19:40:52,721][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.allow_registering_model_via_url] from [true] to [false] [2024-02-06T19:40:52,761][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [true] to [false] [2024-02-06T19:40:52,762][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [true] to [false] [2024-02-06T19:40:52,807][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.allow_registering_model_via_url] from [false] to [true] [2024-02-06T19:40:52,825][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[my-nlp-index/5okSKsEITMmz79o9DXUcpA] [2024-02-06T19:40:52,828][INFO ][o.o.c.m.MetadataCreateIndexService] [node_name_9200] [my-nlp-index] creating index, cause [api], templates [], shards [3]/[1] [2024-02-06T19:40:52,848][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[my-nlp-index/5okSKsEITMmz79o9DXUcpA] [2024-02-06T19:40:53,039][INFO ][o.o.m.m.MLModelManager ] [node_name_9200] create new model meta doc xpDwf40Bamv3vXxyBdiY for register model task xZDwf40Bamv3vXxyBdh6 [2024-02-06T19:40:53,736][INFO ][stdout ] [node_name_9200] Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 1% |█ | traced_small_model.zip Downloading: 2% |█ | traced_small_model.zip Downloading: 3% |██ | traced_small_model.zip Downloading: 4% |██ | traced_small_model.zip Downloading: 5% |███ | traced_small_model.zip Downloading: 6% |███ | traced_small_model.zip Downloading: 7% |███ | traced_small_model.zip Downloading: 8% |████ | traced_small_model.zip Downloading: 9% |████ | traced_small_model.zip Downloading: 10% |█████ | traced_small_model.zip Downloading: 11% |█████ | traced_small_model.zip Downloading: 12% |█████ | traced_small_model.zip Downloading: 13% |██████ | traced_small_model.zip Downloading: 14% |██████ | traced_small_model.zip Downloading: 15% |███████ | traced_small_model.zip Downloading: 16% |███████ | traced_small_model.zip Downloading: 17% |███████ | traced_small_model.zip Downloading: 18% |████████ | traced_small_model.zip Downloading: 19% |████████ | traced_small_model.zip Downloading: 20% |█████████ | traced_small_model.zip Downloading: 21% |█████████ | traced_small_model.zip Downloading: 22% |█████████ | traced_small_model.zip Downloading: 23% |██████████ | traced_small_model.zip Downloading: 24% |██████████ | traced_small_model.zip Downloading: 25% |███████████ | traced_small_model.zip Downloading: 26% |███████████ | traced_small_model.zip Downloading: 27% |███████████ | traced_small_model.zip Downloading: 28% |████████████ | traced_small_model.zip Downloading: 29% |████████████ | traced_small_model.zip Downloading: 30% |█████████████ | traced_small_model.zip Downloading: 31% |█████████████ | traced_small_model.zip Downloading: 32% |█████████████ | traced_small_model.zip Downloading: 33% |██████████████ | traced_small_model.zip Downloading: 34% |██████████████ | traced_small_model.zip Downloading: 35% |███████████████ | traced_small_model.zip Downloading: 36% |███████████████ | traced_small_model.zip Downloading: 37% |███████████████ | traced_small_model.zip Downloading: 38% |████████████████ | traced_small_model.zip Downloading: 39% |████████████████ | traced_small_model.zip Downloading: 40% |█████████████████ | traced_small_model.zip Downloading: 41% |█████████████████ | traced_small_model.zip Downloading: 42% |█████████████████ | traced_small_model.zip Downloading: 43% |██████████████████ | traced_small_model.zip Downloading: 44% |██████████████████ | traced_small_model.zip Downloading: 45% |███████████████████ | traced_small_model.zip Downloading: 46% |███████████████████ | traced_small_model.zip Downloading: 47% |███████████████████ | traced_small_model.zip Downloading: 48% |████████████████████ | traced_small_model.zip Downloading: 49% |████████████████████ | traced_small_model.zip Downloading: 50% |█████████████████████ | traced_small_model.zip Downloading: 51% |█████████████████████ | traced_small_model.zip Downloading: 52% |█████████████████████ | traced_small_model.zip Downloading: 53% |██████████████████████ | traced_small_model.zip Downloading: 54% |██████████████████████ | traced_small_model.zip Downloading: 55% |███████████████████████ | traced_small_model.zip Downloading: 56% |███████████████████████ | traced_small_model.zip Downloading: 57% |███████████████████████ | traced_small_model.zip Downloading: 58% |████████████████████████ | traced_small_model.zip Downloading: 59% |████████████████████████ | traced_small_model.zip Downloading: 60% |█████████████████████████ | traced_small_model.zip Downloading: 61% |█████████████████████████ | traced_small_model.zip Downloading: 62% |█████████████████████████ | traced_small_model.zip Downloading: 63% |██████████████████████████ | traced_small_model.zip Downloading: 64% |██████████████████████████ | traced_small_model.zip Downloading: 65% |███████████████████████████ | traced_small_model.zip Downloading: 66% |███████████████████████████ | traced_small_model.zip Downloading: 67% |███████████████████████████ | traced_small_model.zip Downloading: 68% |████████████████████████████ | traced_small_model.zip Downloading: 69% |████████████████████████████ | traced_small_model.zip Downloading: 70% |█████████████████████████████ | traced_small_model.zip Downloading: 71% |█████████████████████████████ | traced_small_model.zip Downloading: 72% |█████████████████████████████ | traced_small_model.zip Downloading: 73% |██████████████████████████████ | traced_small_model.zip Downloading: 74% |██████████████████████████████ | traced_small_model.zip Downloading: 75% |███████████████████████████████ | traced_small_model.zip Downloading: 76% |███████████████████████████████ | traced_small_model.zip Downloading: 77% |███████████████████████████████ | traced_small_model.zip Downloading: 78% |████████████████████████████████ | traced_small_model.zip Downloading: 79% |████████████████████████████████ | traced_small_model.zip Downloading: 80% |█████████████████████████████████ | traced_small_model.zip Downloading: 81% |█████████████████████████████████ | traced_small_model.zip Downloading: 82% |█████████████████████████████████ | traced_small_model.zip Downloading: 83% |██████████████████████████████████ | traced_small_model.zip Downloading: 84% |██████████████████████████████████ | traced_small_model.zip Downloading: 85% |███████████████████████████████████ | traced_small_model.zip Downloading: 86% |███████████████████████████████████ | traced_small_model.zip Downloading: 87% |███████████████████████████████████ | traced_small_model.zip Downloading: 88% |████████████████████████████████████ | traced_small_model.zip Downloading: 89% |████████████████████████████████████ | traced_small_model.zip Downloading: 90% |█████████████████████████████████████ | traced_small_model.zip Downloading: 91% |█████████████████████████████████████ | traced_small_model.zip Downloading: 92% |█████████████████████████████████████ | traced_small_model.zip Downloading: 93% |██████████████████████████████████████ | traced_small_model.zip Downloading: 94% |██████████████████████████████████████ | traced_small_model.zip Downloading: 95% |███████████████████████████████████████ | traced_small_model.zip Downloading: 96% |███████████████████████████████████████ | traced_small_model.zip Downloading: 97% |███████████████████████████████████████ | traced_small_model.zip Downloading: 98% |████████████████████████████████████████| traced_small_model.zip Downloading: 99% |████████████████████████████████████████| traced_small_model.zip Downloading: 100% |████████████████████████████████████████| traced_small_model.zip [2024-02-06T19:40:53,895][INFO ][o.o.m.m.MLModelManager ] [node_name_9200] Model registered successfully, model id: xpDwf40Bamv3vXxyBdiY, task id: xZDwf40Bamv3vXxyBdh6 [2024-02-06T19:40:55,019][INFO ][o.o.m.a.d.TransportDeployModelAction] [node_name_9200] Will deploy model on these nodes: OAYF21-xRbu9dc9StXHecg [2024-02-06T19:40:55,065][ERROR][o.o.m.m.MLModelManager ] [node_name_9200] No controller is deployed because the model xpDwf40Bamv3vXxyBdiY is expected not having an enabled model controller. Please use the create model controller api to create one if this is unexpected. [2024-02-06T19:40:55,222][INFO ][o.o.m.e.a.DLModel ] [node_name_9200] Model xpDwf40Bamv3vXxyBdiY is successfully deployed on 1 devices [2024-02-06T19:40:55,224][INFO ][o.o.m.a.MLModelAutoReDeployer] [node_name_9200] No models needs to be auto redeployed! [2024-02-06T19:40:55,224][INFO ][o.o.m.a.f.TransportForwardAction] [node_name_9200] deploy model done with state: DEPLOYED, model id: xpDwf40Bamv3vXxyBdiY [2024-02-06T19:40:55,225][INFO ][o.o.m.a.d.TransportDeployModelOnNodeAction] [node_name_9200] deploy model task done x5Dwf40Bamv3vXxyDdhs [2024-02-06T19:40:57,093][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[my-nlp-index/5okSKsEITMmz79o9DXUcpA] [2024-02-06T19:40:57,096][INFO ][o.o.c.s.IndexScopedSettings] [node_name_9200] [my-nlp-index] updating [index.search.default_pipeline] from [_none] to [search-pipeline] [2024-02-06T19:40:57,111][INFO ][o.o.c.s.IndexScopedSettings] [node_name_9200] [my-nlp-index] updating [index.search.default_pipeline] from [_none] to [search-pipeline] [2024-02-06T19:41:00,194][INFO ][o.o.m.a.m.DeleteModelTransportAction] [node_name_9200] Model controller not deleted due to no model controller found for model: xpDwf40Bamv3vXxyBdiY [2024-02-06T19:41:00,204][INFO ][o.o.c.m.MetadataDeleteIndexService] [node_name_9200] [my-nlp-index/5okSKsEITMmz79o9DXUcpA] deleting index [2024-02-06T19:41:00,271][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [false] to [true] [2024-02-06T19:41:00,271][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [false] to [true] [2024-02-06T19:41:00,272][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.allow_registering_model_via_url] from [true] to [false] [2024-02-06T19:41:00,348][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [true] to [false] [2024-02-06T19:41:00,349][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [true] to [false] [2024-02-06T19:41:00,389][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.allow_registering_model_via_url] from [false] to [true] [2024-02-06T19:41:00,403][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-neural-multi-doc-one-shard-index/pMDKuBbATqeNsRWBk6OXHw] [2024-02-06T19:41:00,405][INFO ][o.o.c.m.MetadataCreateIndexService] [node_name_9200] [test-neural-multi-doc-one-shard-index] creating index, cause [api], templates [], shards [1]/[1] [2024-02-06T19:41:00,423][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-neural-multi-doc-one-shard-index/pMDKuBbATqeNsRWBk6OXHw] [2024-02-06T19:41:00,491][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-neural-multi-doc-one-shard-index/pMDKuBbATqeNsRWBk6OXHw] [2024-02-06T19:41:00,495][INFO ][o.o.c.m.MetadataMappingService] [node_name_9200] [test-neural-multi-doc-one-shard-index/pMDKuBbATqeNsRWBk6OXHw] update_mapping [_doc] [2024-02-06T19:41:00,664][INFO ][o.o.m.m.MLModelManager ] [node_name_9200] create new model meta doc ypDwf40Bamv3vXxyI9hr for register model task yZDwf40Bamv3vXxyI9hB [2024-02-06T19:41:00,835][INFO ][stdout ] [node_name_9200] Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 1% |█ | traced_small_model.zip Downloading: 2% |█ | traced_small_model.zip Downloading: 3% |██ | traced_small_model.zip Downloading: 4% |██ | traced_small_model.zip Downloading: 5% |███ | traced_small_model.zip Downloading: 6% |███ | traced_small_model.zip Downloading: 7% |███ | traced_small_model.zip Downloading: 8% |████ | traced_small_model.zip Downloading: 9% |████ | traced_small_model.zip Downloading: 10% |█████ | traced_small_model.zip Downloading: 11% |█████ | traced_small_model.zip Downloading: 12% |█████ | traced_small_model.zip Downloading: 13% |██████ | traced_small_model.zip Downloading: 14% |██████ | traced_small_model.zip Downloading: 15% |███████ | traced_small_model.zip Downloading: 16% |███████ | traced_small_model.zip Downloading: 17% |███████ | traced_small_model.zip Downloading: 18% |████████ | traced_small_model.zip Downloading: 19% |████████ | traced_small_model.zip Downloading: 20% |█████████ | traced_small_model.zip Downloading: 21% |█████████ | traced_small_model.zip Downloading: 22% |█████████ | traced_small_model.zip Downloading: 23% |██████████ | traced_small_model.zip Downloading: 24% |██████████ | traced_small_model.zip Downloading: 25% |███████████ | traced_small_model.zip Downloading: 26% |███████████ | traced_small_model.zip Downloading: 27% |███████████ | traced_small_model.zip Downloading: 28% |████████████ | traced_small_model.zip Downloading: 29% |████████████ | traced_small_model.zip Downloading: 30% |█████████████ | traced_small_model.zip Downloading: 31% |█████████████ | traced_small_model.zip Downloading: 32% |█████████████ | traced_small_model.zip Downloading: 33% |██████████████ | traced_small_model.zip Downloading: 34% |██████████████ | traced_small_model.zip Downloading: 35% |███████████████ | traced_small_model.zip Downloading: 36% |███████████████ | traced_small_model.zip Downloading: 37% |███████████████ | traced_small_model.zip Downloading: 38% |████████████████ | traced_small_model.zip Downloading: 39% |████████████████ | traced_small_model.zip Downloading: 40% |█████████████████ | traced_small_model.zip Downloading: 41% |█████████████████ | traced_small_model.zip Downloading: 42% |█████████████████ | traced_small_model.zip Downloading: 43% |██████████████████ | traced_small_model.zip Downloading: 44% |██████████████████ | traced_small_model.zip Downloading: 45% |███████████████████ | traced_small_model.zip Downloading: 46% |███████████████████ | traced_small_model.zip Downloading: 47% |███████████████████ | traced_small_model.zip Downloading: 48% |████████████████████ | traced_small_model.zip Downloading: 49% |████████████████████ | traced_small_model.zip Downloading: 50% |█████████████████████ | traced_small_model.zip Downloading: 51% |█████████████████████ | traced_small_model.zip Downloading: 52% |█████████████████████ | traced_small_model.zip Downloading: 53% |██████████████████████ | traced_small_model.zip Downloading: 54% |██████████████████████ | traced_small_model.zip Downloading: 55% |███████████████████████ | traced_small_model.zip Downloading: 56% |███████████████████████ | traced_small_model.zip Downloading: 57% |███████████████████████ | traced_small_model.zip Downloading: 58% |████████████████████████ | traced_small_model.zip Downloading: 59% |████████████████████████ | traced_small_model.zip Downloading: 60% |█████████████████████████ | traced_small_model.zip Downloading: 61% |█████████████████████████ | traced_small_model.zip Downloading: 62% |█████████████████████████ | traced_small_model.zip Downloading: 63% |██████████████████████████ | traced_small_model.zip Downloading: 64% |██████████████████████████ | traced_small_model.zip Downloading: 65% |███████████████████████████ | traced_small_model.zip Downloading: 66% |███████████████████████████ | traced_small_model.zip Downloading: 67% |███████████████████████████ | traced_small_model.zip Downloading: 68% |████████████████████████████ | traced_small_model.zip Downloading: 69% |████████████████████████████ | traced_small_model.zip Downloading: 70% |█████████████████████████████ | traced_small_model.zip Downloading: 71% |█████████████████████████████ | traced_small_model.zip Downloading: 72% |█████████████████████████████ | traced_small_model.zip Downloading: 73% |██████████████████████████████ | traced_small_model.zip Downloading: 74% |██████████████████████████████ | traced_small_model.zip Downloading: 75% |███████████████████████████████ | traced_small_model.zip Downloading: 76% |███████████████████████████████ | traced_small_model.zip Downloading: 77% |███████████████████████████████ | traced_small_model.zip Downloading: 78% |████████████████████████████████ | traced_small_model.zip Downloading: 79% |████████████████████████████████ | traced_small_model.zip Downloading: 80% |█████████████████████████████████ | traced_small_model.zip Downloading: 81% |█████████████████████████████████ | traced_small_model.zip Downloading: 82% |█████████████████████████████████ | traced_small_model.zip Downloading: 83% |██████████████████████████████████ | traced_small_model.zip Downloading: 84% |██████████████████████████████████ | traced_small_model.zip Downloading: 85% |███████████████████████████████████ | traced_small_model.zip Downloading: 86% |███████████████████████████████████ | traced_small_model.zip Downloading: 87% |███████████████████████████████████ | traced_small_model.zip Downloading: 88% |████████████████████████████████████ | traced_small_model.zip Downloading: 89% |████████████████████████████████████ | traced_small_model.zip Downloading: 90% |█████████████████████████████████████ | traced_small_model.zip Downloading: 91% |█████████████████████████████████████ | traced_small_model.zip Downloading: 92% |█████████████████████████████████████ | traced_small_model.zip Downloading: 93% |██████████████████████████████████████ | traced_small_model.zip Downloading: 94% |██████████████████████████████████████ | traced_small_model.zip Downloading: 95% |███████████████████████████████████████ | traced_small_model.zip Downloading: 96% |███████████████████████████████████████ | traced_small_model.zip Downloading: 97% |███████████████████████████████████████ | traced_small_model.zip Downloading: 98% |████████████████████████████████████████| traced_small_model.zip Downloading: 99% |████████████████████████████████████████| traced_small_model.zip Downloading: 100% |████████████████████████████████████████| traced_small_model.zip [2024-02-06T19:41:01,056][INFO ][o.o.m.m.MLModelManager ] [node_name_9200] Model registered successfully, model id: ypDwf40Bamv3vXxyI9hr, task id: yZDwf40Bamv3vXxyI9hB [2024-02-06T19:41:02,649][INFO ][o.o.m.a.d.TransportDeployModelAction] [node_name_9200] Will deploy model on these nodes: OAYF21-xRbu9dc9StXHecg [2024-02-06T19:41:02,680][ERROR][o.o.m.m.MLModelManager ] [node_name_9200] No controller is deployed because the model ypDwf40Bamv3vXxyI9hr is expected not having an enabled model controller. Please use the create model controller api to create one if this is unexpected. [2024-02-06T19:41:02,830][INFO ][o.o.m.e.a.DLModel ] [node_name_9200] Model ypDwf40Bamv3vXxyI9hr is successfully deployed on 1 devices [2024-02-06T19:41:02,832][INFO ][o.o.m.a.MLModelAutoReDeployer] [node_name_9200] No models needs to be auto redeployed! [2024-02-06T19:41:02,832][INFO ][o.o.m.a.f.TransportForwardAction] [node_name_9200] deploy model done with state: DEPLOYED, model id: ypDwf40Bamv3vXxyI9hr [2024-02-06T19:41:02,833][INFO ][o.o.m.a.d.TransportDeployModelOnNodeAction] [node_name_9200] deploy model task done y5Dwf40Bamv3vXxyK9g5 [2024-02-06T19:41:04,675][INFO ][o.o.n.p.f.NormalizationProcessorFactory] [node_name_9200] Creating search phase results processor of type [normalization-processor] with normalization [MinMaxScoreNormalizationTechnique(TECHNIQUE_NAME=min_max)] and combination [ArithmeticMeanScoreCombinationTechnique(TECHNIQUE_NAME=arithmetic_mean)] [2024-02-06T19:41:04,695][INFO ][o.o.n.p.f.NormalizationProcessorFactory] [node_name_9200] Creating search phase results processor of type [normalization-processor] with normalization [MinMaxScoreNormalizationTechnique(TECHNIQUE_NAME=min_max)] and combination [ArithmeticMeanScoreCombinationTechnique(TECHNIQUE_NAME=arithmetic_mean)] [2024-02-06T19:41:07,773][INFO ][o.o.m.a.m.DeleteModelTransportAction] [node_name_9200] Model controller not deleted due to no model controller found for model: ypDwf40Bamv3vXxyI9hr [2024-02-06T19:41:07,782][INFO ][o.o.c.m.MetadataDeleteIndexService] [node_name_9200] [test-neural-multi-doc-one-shard-index/pMDKuBbATqeNsRWBk6OXHw] deleting index [2024-02-06T19:41:07,845][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [false] to [true] [2024-02-06T19:41:07,845][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [false] to [true] [2024-02-06T19:41:07,846][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.allow_registering_model_via_url] from [true] to [false] [2024-02-06T19:41:07,880][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [true] to [false] [2024-02-06T19:41:07,880][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [true] to [false] [2024-02-06T19:41:07,917][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.allow_registering_model_via_url] from [false] to [true] [2024-02-06T19:41:07,930][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-neural-multi-doc-three-shards-index/-N7FHHDlRWmRYxgsfLsbXg] [2024-02-06T19:41:07,933][INFO ][o.o.c.m.MetadataCreateIndexService] [node_name_9200] [test-neural-multi-doc-three-shards-index] creating index, cause [api], templates [], shards [3]/[1] [2024-02-06T19:41:07,949][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-neural-multi-doc-three-shards-index/-N7FHHDlRWmRYxgsfLsbXg] [2024-02-06T19:41:08,047][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-neural-multi-doc-three-shards-index/-N7FHHDlRWmRYxgsfLsbXg] [2024-02-06T19:41:08,051][INFO ][o.o.c.m.MetadataMappingService] [node_name_9200] [test-neural-multi-doc-three-shards-index/-N7FHHDlRWmRYxgsfLsbXg] update_mapping [_doc] [2024-02-06T19:41:08,225][INFO ][o.o.m.m.MLModelManager ] [node_name_9200] create new model meta doc zpDwf40Bamv3vXxyQNj0 for register model task zZDwf40Bamv3vXxyQNjb [2024-02-06T19:41:08,407][INFO ][stdout ] [node_name_9200] Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 1% |█ | traced_small_model.zip Downloading: 2% |█ | traced_small_model.zip Downloading: 3% |██ | traced_small_model.zip Downloading: 4% |██ | traced_small_model.zip Downloading: 5% |███ | traced_small_model.zip Downloading: 6% |███ | traced_small_model.zip Downloading: 7% |███ | traced_small_model.zip Downloading: 8% |████ | traced_small_model.zip Downloading: 9% |████ | traced_small_model.zip Downloading: 10% |█████ | traced_small_model.zip Downloading: 11% |█████ | traced_small_model.zip Downloading: 12% |█████ | traced_small_model.zip Downloading: 13% |██████ | traced_small_model.zip Downloading: 14% |██████ | traced_small_model.zip Downloading: 15% |███████ | traced_small_model.zip Downloading: 16% |███████ | traced_small_model.zip Downloading: 17% |███████ | traced_small_model.zip Downloading: 18% |████████ | traced_small_model.zip Downloading: 19% |████████ | traced_small_model.zip Downloading: 20% |█████████ | traced_small_model.zip Downloading: 21% |█████████ | traced_small_model.zip Downloading: 22% |█████████ | traced_small_model.zip Downloading: 23% |██████████ | traced_small_model.zip Downloading: 24% |██████████ | traced_small_model.zip Downloading: 25% |███████████ | traced_small_model.zip Downloading: 26% |███████████ | traced_small_model.zip Downloading: 27% |███████████ | traced_small_model.zip Downloading: 28% |████████████ | traced_small_model.zip Downloading: 29% |████████████ | traced_small_model.zip Downloading: 30% |█████████████ | traced_small_model.zip Downloading: 31% |█████████████ | traced_small_model.zip Downloading: 32% |█████████████ | traced_small_model.zip Downloading: 33% |██████████████ | traced_small_model.zip Downloading: 34% |██████████████ | traced_small_model.zip Downloading: 35% |███████████████ | traced_small_model.zip Downloading: 36% |███████████████ | traced_small_model.zip Downloading: 37% |███████████████ | traced_small_model.zip Downloading: 38% |████████████████ | traced_small_model.zip Downloading: 39% |████████████████ | traced_small_model.zip Downloading: 40% |█████████████████ | traced_small_model.zip Downloading: 41% |█████████████████ | traced_small_model.zip Downloading: 42% |█████████████████ | traced_small_model.zip Downloading: 43% |██████████████████ | traced_small_model.zip Downloading: 44% |██████████████████ | traced_small_model.zip Downloading: 45% |███████████████████ | traced_small_model.zip Downloading: 46% |███████████████████ | traced_small_model.zip Downloading: 47% |███████████████████ | traced_small_model.zip Downloading: 48% |████████████████████ | traced_small_model.zip Downloading: 49% |████████████████████ | traced_small_model.zip Downloading: 50% |█████████████████████ | traced_small_model.zip Downloading: 51% |█████████████████████ | traced_small_model.zip Downloading: 52% |█████████████████████ | traced_small_model.zip Downloading: 53% |██████████████████████ | traced_small_model.zip Downloading: 54% |██████████████████████ | traced_small_model.zip Downloading: 55% |███████████████████████ | traced_small_model.zip Downloading: 56% |███████████████████████ | traced_small_model.zip Downloading: 57% |███████████████████████ | traced_small_model.zip Downloading: 58% |████████████████████████ | traced_small_model.zip Downloading: 59% |████████████████████████ | traced_small_model.zip Downloading: 60% |█████████████████████████ | traced_small_model.zip Downloading: 61% |█████████████████████████ | traced_small_model.zip Downloading: 62% |█████████████████████████ | traced_small_model.zip Downloading: 63% |██████████████████████████ | traced_small_model.zip Downloading: 64% |██████████████████████████ | traced_small_model.zip Downloading: 65% |███████████████████████████ | traced_small_model.zip Downloading: 66% |███████████████████████████ | traced_small_model.zip Downloading: 67% |███████████████████████████ | traced_small_model.zip Downloading: 68% |████████████████████████████ | traced_small_model.zip Downloading: 69% |████████████████████████████ | traced_small_model.zip Downloading: 70% |█████████████████████████████ | traced_small_model.zip Downloading: 71% |█████████████████████████████ | traced_small_model.zip Downloading: 72% |█████████████████████████████ | traced_small_model.zip Downloading: 73% |██████████████████████████████ | traced_small_model.zip Downloading: 74% |██████████████████████████████ | traced_small_model.zip Downloading: 75% |███████████████████████████████ | traced_small_model.zip Downloading: 76% |███████████████████████████████ | traced_small_model.zip Downloading: 77% |███████████████████████████████ | traced_small_model.zip Downloading: 78% |████████████████████████████████ | traced_small_model.zip Downloading: 79% |████████████████████████████████ | traced_small_model.zip Downloading: 80% |█████████████████████████████████ | traced_small_model.zip Downloading: 81% |█████████████████████████████████ | traced_small_model.zip Downloading: 82% |█████████████████████████████████ | traced_small_model.zip Downloading: 83% |██████████████████████████████████ | traced_small_model.zip Downloading: 84% |██████████████████████████████████ | traced_small_model.zip Downloading: 85% |███████████████████████████████████ | traced_small_model.zip Downloading: 86% |███████████████████████████████████ | traced_small_model.zip Downloading: 87% |███████████████████████████████████ | traced_small_model.zip Downloading: 88% |████████████████████████████████████ | traced_small_model.zip Downloading: 89% |████████████████████████████████████ | traced_small_model.zip Downloading: 90% |█████████████████████████████████████ | traced_small_model.zip Downloading: 91% |█████████████████████████████████████ | traced_small_model.zip Downloading: 92% |█████████████████████████████████████ | traced_small_model.zip Downloading: 93% |██████████████████████████████████████ | traced_small_model.zip Downloading: 94% |██████████████████████████████████████ | traced_small_model.zip Downloading: 95% |███████████████████████████████████████ | traced_small_model.zip Downloading: 96% |███████████████████████████████████████ | traced_small_model.zip Downloading: 97% |███████████████████████████████████████ | traced_small_model.zip Downloading: 98% |████████████████████████████████████████| traced_small_model.zip Downloading: 99% |████████████████████████████████████████| traced_small_model.zip Downloading: 100% |████████████████████████████████████████| traced_small_model.zip [2024-02-06T19:41:08,561][INFO ][o.o.m.m.MLModelManager ] [node_name_9200] Model registered successfully, model id: zpDwf40Bamv3vXxyQNj0, task id: zZDwf40Bamv3vXxyQNjb [2024-02-06T19:41:10,208][INFO ][o.o.m.a.d.TransportDeployModelAction] [node_name_9200] Will deploy model on these nodes: OAYF21-xRbu9dc9StXHecg [2024-02-06T19:41:10,240][ERROR][o.o.m.m.MLModelManager ] [node_name_9200] No controller is deployed because the model zpDwf40Bamv3vXxyQNj0 is expected not having an enabled model controller. Please use the create model controller api to create one if this is unexpected. [2024-02-06T19:41:10,393][INFO ][o.o.m.e.a.DLModel ] [node_name_9200] Model zpDwf40Bamv3vXxyQNj0 is successfully deployed on 1 devices [2024-02-06T19:41:10,394][INFO ][o.o.m.a.MLModelAutoReDeployer] [node_name_9200] No models needs to be auto redeployed! [2024-02-06T19:41:10,395][INFO ][o.o.m.a.f.TransportForwardAction] [node_name_9200] deploy model done with state: DEPLOYED, model id: zpDwf40Bamv3vXxyQNj0 [2024-02-06T19:41:10,395][INFO ][o.o.m.a.d.TransportDeployModelOnNodeAction] [node_name_9200] deploy model task done z5Dwf40Bamv3vXxySNjB [2024-02-06T19:41:12,236][INFO ][o.o.n.p.f.NormalizationProcessorFactory] [node_name_9200] Creating search phase results processor of type [normalization-processor] with normalization [MinMaxScoreNormalizationTechnique(TECHNIQUE_NAME=min_max)] and combination [ArithmeticMeanScoreCombinationTechnique(TECHNIQUE_NAME=arithmetic_mean)] [2024-02-06T19:41:12,253][INFO ][o.o.n.p.f.NormalizationProcessorFactory] [node_name_9200] Creating search phase results processor of type [normalization-processor] with normalization [MinMaxScoreNormalizationTechnique(TECHNIQUE_NAME=min_max)] and combination [ArithmeticMeanScoreCombinationTechnique(TECHNIQUE_NAME=arithmetic_mean)] [2024-02-06T19:41:15,295][INFO ][o.o.m.a.m.DeleteModelTransportAction] [node_name_9200] Model controller not deleted due to no model controller found for model: zpDwf40Bamv3vXxyQNj0 [2024-02-06T19:41:15,303][INFO ][o.o.c.m.MetadataDeleteIndexService] [node_name_9200] [test-neural-multi-doc-three-shards-index/-N7FHHDlRWmRYxgsfLsbXg] deleting index [2024-02-06T19:41:15,366][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [false] to [true] [2024-02-06T19:41:15,366][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [false] to [true] [2024-02-06T19:41:15,367][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.allow_registering_model_via_url] from [true] to [false] [2024-02-06T19:41:15,400][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [true] to [false] [2024-02-06T19:41:15,401][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [true] to [false] [2024-02-06T19:41:15,439][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.allow_registering_model_via_url] from [false] to [true] [2024-02-06T19:41:15,452][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-neural-multi-doc-three-shards-index/n8gLo_UmRBCeIwcjq59trA] [2024-02-06T19:41:15,454][INFO ][o.o.c.m.MetadataCreateIndexService] [node_name_9200] [test-neural-multi-doc-three-shards-index] creating index, cause [api], templates [], shards [3]/[1] [2024-02-06T19:41:15,470][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-neural-multi-doc-three-shards-index/n8gLo_UmRBCeIwcjq59trA] [2024-02-06T19:41:15,560][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-neural-multi-doc-three-shards-index/n8gLo_UmRBCeIwcjq59trA] [2024-02-06T19:41:15,564][INFO ][o.o.c.m.MetadataMappingService] [node_name_9200] [test-neural-multi-doc-three-shards-index/n8gLo_UmRBCeIwcjq59trA] update_mapping [_doc] [2024-02-06T19:41:15,708][INFO ][o.o.m.m.MLModelManager ] [node_name_9200] create new model meta doc 0pDwf40Bamv3vXxyXtgx for register model task 0ZDwf40Bamv3vXxyXtgb [2024-02-06T19:41:15,905][INFO ][stdout ] [node_name_9200] Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 1% |█ | traced_small_model.zip Downloading: 2% |█ | traced_small_model.zip Downloading: 3% |██ | traced_small_model.zip Downloading: 4% |██ | traced_small_model.zip Downloading: 5% |███ | traced_small_model.zip Downloading: 6% |███ | traced_small_model.zip Downloading: 7% |███ | traced_small_model.zip Downloading: 8% |████ | traced_small_model.zip Downloading: 9% |████ | traced_small_model.zip Downloading: 10% |█████ | traced_small_model.zip Downloading: 11% |█████ | traced_small_model.zip Downloading: 12% |█████ | traced_small_model.zip Downloading: 13% |██████ | traced_small_model.zip Downloading: 14% |██████ | traced_small_model.zip Downloading: 15% |███████ | traced_small_model.zip Downloading: 16% |███████ | traced_small_model.zip Downloading: 17% |███████ | traced_small_model.zip Downloading: 18% |████████ | traced_small_model.zip Downloading: 19% |████████ | traced_small_model.zip Downloading: 20% |█████████ | traced_small_model.zip Downloading: 21% |█████████ | traced_small_model.zip Downloading: 22% |█████████ | traced_small_model.zip Downloading: 23% |██████████ | traced_small_model.zip Downloading: 24% |██████████ | traced_small_model.zip Downloading: 25% |███████████ | traced_small_model.zip Downloading: 26% |███████████ | traced_small_model.zip Downloading: 27% |███████████ | traced_small_model.zip Downloading: 28% |████████████ | traced_small_model.zip Downloading: 29% |████████████ | traced_small_model.zip Downloading: 30% |█████████████ | traced_small_model.zip Downloading: 31% |█████████████ | traced_small_model.zip Downloading: 32% |█████████████ | traced_small_model.zip Downloading: 33% |██████████████ | traced_small_model.zip Downloading: 34% |██████████████ | traced_small_model.zip Downloading: 35% |███████████████ | traced_small_model.zip Downloading: 36% |███████████████ | traced_small_model.zip Downloading: 37% |███████████████ | traced_small_model.zip Downloading: 38% |████████████████ | traced_small_model.zip Downloading: 39% |████████████████ | traced_small_model.zip Downloading: 40% |█████████████████ | traced_small_model.zip Downloading: 41% |█████████████████ | traced_small_model.zip Downloading: 42% |█████████████████ | traced_small_model.zip Downloading: 43% |██████████████████ | traced_small_model.zip Downloading: 44% |██████████████████ | traced_small_model.zip Downloading: 45% |███████████████████ | traced_small_model.zip Downloading: 46% |███████████████████ | traced_small_model.zip Downloading: 47% |███████████████████ | traced_small_model.zip Downloading: 48% |████████████████████ | traced_small_model.zip Downloading: 49% |████████████████████ | traced_small_model.zip Downloading: 50% |█████████████████████ | traced_small_model.zip Downloading: 51% |█████████████████████ | traced_small_model.zip Downloading: 52% |█████████████████████ | traced_small_model.zip Downloading: 53% |██████████████████████ | traced_small_model.zip Downloading: 54% |██████████████████████ | traced_small_model.zip Downloading: 55% |███████████████████████ | traced_small_model.zip Downloading: 56% |███████████████████████ | traced_small_model.zip Downloading: 57% |███████████████████████ | traced_small_model.zip Downloading: 58% |████████████████████████ | traced_small_model.zip Downloading: 59% |████████████████████████ | traced_small_model.zip Downloading: 60% |█████████████████████████ | traced_small_model.zip Downloading: 61% |█████████████████████████ | traced_small_model.zip Downloading: 62% |█████████████████████████ | traced_small_model.zip Downloading: 63% |██████████████████████████ | traced_small_model.zip Downloading: 64% |██████████████████████████ | traced_small_model.zip Downloading: 65% |███████████████████████████ | traced_small_model.zip Downloading: 66% |███████████████████████████ | traced_small_model.zip Downloading: 67% |███████████████████████████ | traced_small_model.zip Downloading: 68% |████████████████████████████ | traced_small_model.zip Downloading: 69% |████████████████████████████ | traced_small_model.zip Downloading: 70% |█████████████████████████████ | traced_small_model.zip Downloading: 71% |█████████████████████████████ | traced_small_model.zip Downloading: 72% |█████████████████████████████ | traced_small_model.zip Downloading: 73% |██████████████████████████████ | traced_small_model.zip Downloading: 74% |██████████████████████████████ | traced_small_model.zip Downloading: 75% |███████████████████████████████ | traced_small_model.zip Downloading: 76% |███████████████████████████████ | traced_small_model.zip Downloading: 77% |███████████████████████████████ | traced_small_model.zip Downloading: 78% |████████████████████████████████ | traced_small_model.zip Downloading: 79% |████████████████████████████████ | traced_small_model.zip Downloading: 80% |█████████████████████████████████ | traced_small_model.zip Downloading: 81% |█████████████████████████████████ | traced_small_model.zip Downloading: 82% |█████████████████████████████████ | traced_small_model.zip Downloading: 83% |██████████████████████████████████ | traced_small_model.zip Downloading: 84% |██████████████████████████████████ | traced_small_model.zip Downloading: 85% |███████████████████████████████████ | traced_small_model.zip Downloading: 86% |███████████████████████████████████ | traced_small_model.zip Downloading: 87% |███████████████████████████████████ | traced_small_model.zip Downloading: 88% |████████████████████████████████████ | traced_small_model.zip Downloading: 89% |████████████████████████████████████ | traced_small_model.zip Downloading: 90% |█████████████████████████████████████ | traced_small_model.zip Downloading: 91% |█████████████████████████████████████ | traced_small_model.zip Downloading: 92% |█████████████████████████████████████ | traced_small_model.zip Downloading: 93% |██████████████████████████████████████ | traced_small_model.zip Downloading: 94% |██████████████████████████████████████ | traced_small_model.zip Downloading: 95% |███████████████████████████████████████ | traced_small_model.zip Downloading: 96% |███████████████████████████████████████ | traced_small_model.zip Downloading: 97% |███████████████████████████████████████ | traced_small_model.zip Downloading: 98% |████████████████████████████████████████| traced_small_model.zip Downloading: 99% |████████████████████████████████████████| traced_small_model.zip Downloading: 100% |████████████████████████████████████████| traced_small_model.zip [2024-02-06T19:41:16,061][INFO ][o.o.m.m.MLModelManager ] [node_name_9200] Model registered successfully, model id: 0pDwf40Bamv3vXxyXtgx, task id: 0ZDwf40Bamv3vXxyXtgb [2024-02-06T19:41:17,697][INFO ][o.o.m.a.d.TransportDeployModelAction] [node_name_9200] Will deploy model on these nodes: OAYF21-xRbu9dc9StXHecg [2024-02-06T19:41:17,788][ERROR][o.o.m.m.MLModelManager ] [node_name_9200] No controller is deployed because the model 0pDwf40Bamv3vXxyXtgx is expected not having an enabled model controller. Please use the create model controller api to create one if this is unexpected. [2024-02-06T19:41:17,939][INFO ][o.o.m.e.a.DLModel ] [node_name_9200] Model 0pDwf40Bamv3vXxyXtgx is successfully deployed on 1 devices [2024-02-06T19:41:17,940][INFO ][o.o.m.a.MLModelAutoReDeployer] [node_name_9200] No models needs to be auto redeployed! [2024-02-06T19:41:17,940][INFO ][o.o.m.a.f.TransportForwardAction] [node_name_9200] deploy model done with state: DEPLOYED, model id: 0pDwf40Bamv3vXxyXtgx [2024-02-06T19:41:17,941][INFO ][o.o.m.a.d.TransportDeployModelOnNodeAction] [node_name_9200] deploy model task done 05Dwf40Bamv3vXxyZtgC [2024-02-06T19:41:19,748][INFO ][o.o.n.p.f.NormalizationProcessorFactory] [node_name_9200] Creating search phase results processor of type [normalization-processor] with normalization [MinMaxScoreNormalizationTechnique(TECHNIQUE_NAME=min_max)] and combination [ArithmeticMeanScoreCombinationTechnique(TECHNIQUE_NAME=arithmetic_mean)] [2024-02-06T19:41:19,766][INFO ][o.o.n.p.f.NormalizationProcessorFactory] [node_name_9200] Creating search phase results processor of type [normalization-processor] with normalization [MinMaxScoreNormalizationTechnique(TECHNIQUE_NAME=min_max)] and combination [ArithmeticMeanScoreCombinationTechnique(TECHNIQUE_NAME=arithmetic_mean)] [2024-02-06T19:41:22,820][INFO ][o.o.m.a.m.DeleteModelTransportAction] [node_name_9200] Model controller not deleted due to no model controller found for model: 0pDwf40Bamv3vXxyXtgx [2024-02-06T19:41:22,830][INFO ][o.o.c.m.MetadataDeleteIndexService] [node_name_9200] [test-neural-multi-doc-three-shards-index/n8gLo_UmRBCeIwcjq59trA] deleting index [2024-02-06T19:41:22,897][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [false] to [true] [2024-02-06T19:41:22,897][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [false] to [true] [2024-02-06T19:41:22,897][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.allow_registering_model_via_url] from [true] to [false] [2024-02-06T19:41:22,928][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [true] to [false] [2024-02-06T19:41:22,928][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [true] to [false] [2024-02-06T19:41:22,962][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.allow_registering_model_via_url] from [false] to [true] [2024-02-06T19:41:22,973][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-neural-multi-doc-one-shard-index/nldWab3HRTCK7t5BD_JXDw] [2024-02-06T19:41:22,976][INFO ][o.o.c.m.MetadataCreateIndexService] [node_name_9200] [test-neural-multi-doc-one-shard-index] creating index, cause [api], templates [], shards [1]/[1] [2024-02-06T19:41:22,994][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-neural-multi-doc-one-shard-index/nldWab3HRTCK7t5BD_JXDw] [2024-02-06T19:41:23,054][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-neural-multi-doc-one-shard-index/nldWab3HRTCK7t5BD_JXDw] [2024-02-06T19:41:23,057][INFO ][o.o.c.m.MetadataMappingService] [node_name_9200] [test-neural-multi-doc-one-shard-index/nldWab3HRTCK7t5BD_JXDw] update_mapping [_doc] [2024-02-06T19:41:23,204][INFO ][o.o.m.m.MLModelManager ] [node_name_9200] create new model meta doc 1pDwf40Bamv3vXxye9h2 for register model task 1ZDwf40Bamv3vXxye9hf [2024-02-06T19:41:23,392][INFO ][stdout ] [node_name_9200] Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 1% |█ | traced_small_model.zip Downloading: 2% |█ | traced_small_model.zip Downloading: 3% |██ | traced_small_model.zip Downloading: 4% |██ | traced_small_model.zip Downloading: 5% |███ | traced_small_model.zip Downloading: 6% |███ | traced_small_model.zip Downloading: 7% |███ | traced_small_model.zip Downloading: 8% |████ | traced_small_model.zip Downloading: 9% |████ | traced_small_model.zip Downloading: 10% |█████ | traced_small_model.zip Downloading: 11% |█████ | traced_small_model.zip Downloading: 12% |█████ | traced_small_model.zip Downloading: 13% |██████ | traced_small_model.zip Downloading: 14% |██████ | traced_small_model.zip Downloading: 15% |███████ | traced_small_model.zip Downloading: 16% |███████ | traced_small_model.zip Downloading: 17% |███████ | traced_small_model.zip Downloading: 18% |████████ | traced_small_model.zip Downloading: 19% |████████ | traced_small_model.zip Downloading: 20% |█████████ | traced_small_model.zip Downloading: 21% |█████████ | traced_small_model.zip Downloading: 22% |█████████ | traced_small_model.zip Downloading: 23% |██████████ | traced_small_model.zip Downloading: 24% |██████████ | traced_small_model.zip Downloading: 25% |███████████ | traced_small_model.zip Downloading: 26% |███████████ | traced_small_model.zip Downloading: 27% |███████████ | traced_small_model.zip Downloading: 28% |████████████ | traced_small_model.zip Downloading: 29% |████████████ | traced_small_model.zip Downloading: 30% |█████████████ | traced_small_model.zip Downloading: 31% |█████████████ | traced_small_model.zip Downloading: 32% |█████████████ | traced_small_model.zip Downloading: 33% |██████████████ | traced_small_model.zip Downloading: 34% |██████████████ | traced_small_model.zip Downloading: 35% |███████████████ | traced_small_model.zip Downloading: 36% |███████████████ | traced_small_model.zip Downloading: 37% |███████████████ | traced_small_model.zip Downloading: 38% |████████████████ | traced_small_model.zip Downloading: 39% |████████████████ | traced_small_model.zip Downloading: 40% |█████████████████ | traced_small_model.zip Downloading: 41% |█████████████████ | traced_small_model.zip Downloading: 42% |█████████████████ | traced_small_model.zip Downloading: 43% |██████████████████ | traced_small_model.zip Downloading: 44% |██████████████████ | traced_small_model.zip Downloading: 45% |███████████████████ | traced_small_model.zip Downloading: 46% |███████████████████ | traced_small_model.zip Downloading: 47% |███████████████████ | traced_small_model.zip Downloading: 48% |████████████████████ | traced_small_model.zip Downloading: 49% |████████████████████ | traced_small_model.zip Downloading: 50% |█████████████████████ | traced_small_model.zip Downloading: 51% |█████████████████████ | traced_small_model.zip Downloading: 52% |█████████████████████ | traced_small_model.zip Downloading: 53% |██████████████████████ | traced_small_model.zip Downloading: 54% |██████████████████████ | traced_small_model.zip Downloading: 55% |███████████████████████ | traced_small_model.zip Downloading: 56% |███████████████████████ | traced_small_model.zip Downloading: 57% |███████████████████████ | traced_small_model.zip Downloading: 58% |████████████████████████ | traced_small_model.zip Downloading: 59% |████████████████████████ | traced_small_model.zip Downloading: 60% |█████████████████████████ | traced_small_model.zip Downloading: 61% |█████████████████████████ | traced_small_model.zip Downloading: 62% |█████████████████████████ | traced_small_model.zip Downloading: 63% |██████████████████████████ | traced_small_model.zip Downloading: 64% |██████████████████████████ | traced_small_model.zip Downloading: 65% |███████████████████████████ | traced_small_model.zip Downloading: 66% |███████████████████████████ | traced_small_model.zip Downloading: 67% |███████████████████████████ | traced_small_model.zip Downloading: 68% |████████████████████████████ | traced_small_model.zip Downloading: 69% |████████████████████████████ | traced_small_model.zip Downloading: 70% |█████████████████████████████ | traced_small_model.zip Downloading: 71% |█████████████████████████████ | traced_small_model.zip Downloading: 72% |█████████████████████████████ | traced_small_model.zip Downloading: 73% |██████████████████████████████ | traced_small_model.zip Downloading: 74% |██████████████████████████████ | traced_small_model.zip Downloading: 75% |███████████████████████████████ | traced_small_model.zip Downloading: 76% |███████████████████████████████ | traced_small_model.zip Downloading: 77% |███████████████████████████████ | traced_small_model.zip Downloading: 78% |████████████████████████████████ | traced_small_model.zip Downloading: 79% |████████████████████████████████ | traced_small_model.zip Downloading: 80% |█████████████████████████████████ | traced_small_model.zip Downloading: 81% |█████████████████████████████████ | traced_small_model.zip Downloading: 82% |█████████████████████████████████ | traced_small_model.zip Downloading: 83% |██████████████████████████████████ | traced_small_model.zip Downloading: 84% |██████████████████████████████████ | traced_small_model.zip Downloading: 85% |███████████████████████████████████ | traced_small_model.zip Downloading: 86% |███████████████████████████████████ | traced_small_model.zip Downloading: 87% |███████████████████████████████████ | traced_small_model.zip Downloading: 88% |████████████████████████████████████ | traced_small_model.zip Downloading: 89% |████████████████████████████████████ | traced_small_model.zip Downloading: 90% |█████████████████████████████████████ | traced_small_model.zip Downloading: 91% |█████████████████████████████████████ | traced_small_model.zip Downloading: 92% |█████████████████████████████████████ | traced_small_model.zip Downloading: 93% |██████████████████████████████████████ | traced_small_model.zip Downloading: 94% |██████████████████████████████████████ | traced_small_model.zip Downloading: 95% |███████████████████████████████████████ | traced_small_model.zip Downloading: 96% |███████████████████████████████████████ | traced_small_model.zip Downloading: 97% |███████████████████████████████████████ | traced_small_model.zip Downloading: 98% |████████████████████████████████████████| traced_small_model.zip Downloading: 99% |████████████████████████████████████████| traced_small_model.zip Downloading: 100% |████████████████████████████████████████| traced_small_model.zip [2024-02-06T19:41:23,546][INFO ][o.o.m.m.MLModelManager ] [node_name_9200] Model registered successfully, model id: 1pDwf40Bamv3vXxye9h2, task id: 1ZDwf40Bamv3vXxye9hf [2024-02-06T19:41:25,188][INFO ][o.o.m.a.d.TransportDeployModelAction] [node_name_9200] Will deploy model on these nodes: OAYF21-xRbu9dc9StXHecg [2024-02-06T19:41:25,218][ERROR][o.o.m.m.MLModelManager ] [node_name_9200] No controller is deployed because the model 1pDwf40Bamv3vXxye9h2 is expected not having an enabled model controller. Please use the create model controller api to create one if this is unexpected. [2024-02-06T19:41:25,372][INFO ][o.o.m.e.a.DLModel ] [node_name_9200] Model 1pDwf40Bamv3vXxye9h2 is successfully deployed on 1 devices [2024-02-06T19:41:25,373][INFO ][o.o.m.a.MLModelAutoReDeployer] [node_name_9200] No models needs to be auto redeployed! [2024-02-06T19:41:25,373][INFO ][o.o.m.a.f.TransportForwardAction] [node_name_9200] deploy model done with state: DEPLOYED, model id: 1pDwf40Bamv3vXxye9h2 [2024-02-06T19:41:25,374][INFO ][o.o.m.a.d.TransportDeployModelOnNodeAction] [node_name_9200] deploy model task done 15Dwf40Bamv3vXxyg9hE [2024-02-06T19:41:27,213][INFO ][o.o.n.p.f.NormalizationProcessorFactory] [node_name_9200] Creating search phase results processor of type [normalization-processor] with normalization [MinMaxScoreNormalizationTechnique(TECHNIQUE_NAME=min_max)] and combination [ArithmeticMeanScoreCombinationTechnique(TECHNIQUE_NAME=arithmetic_mean)] [2024-02-06T19:41:27,233][INFO ][o.o.n.p.f.NormalizationProcessorFactory] [node_name_9200] Creating search phase results processor of type [normalization-processor] with normalization [MinMaxScoreNormalizationTechnique(TECHNIQUE_NAME=min_max)] and combination [ArithmeticMeanScoreCombinationTechnique(TECHNIQUE_NAME=arithmetic_mean)] [2024-02-06T19:41:30,281][INFO ][o.o.m.a.m.DeleteModelTransportAction] [node_name_9200] Model controller not deleted due to no model controller found for model: 1pDwf40Bamv3vXxye9h2 [2024-02-06T19:41:30,289][INFO ][o.o.c.m.MetadataDeleteIndexService] [node_name_9200] [test-neural-multi-doc-one-shard-index/nldWab3HRTCK7t5BD_JXDw] deleting index [2024-02-06T19:41:30,343][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [false] to [true] [2024-02-06T19:41:30,344][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [false] to [true] [2024-02-06T19:41:30,344][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.allow_registering_model_via_url] from [true] to [false] [2024-02-06T19:41:30,373][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [true] to [false] [2024-02-06T19:41:30,373][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [true] to [false] [2024-02-06T19:41:30,409][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.allow_registering_model_via_url] from [false] to [true] [2024-02-06T19:41:30,419][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-neural-multi-doc-three-shards-index/TShSDVrbRXaqWaxpL6vIJg] [2024-02-06T19:41:30,422][INFO ][o.o.c.m.MetadataCreateIndexService] [node_name_9200] [test-neural-multi-doc-three-shards-index] creating index, cause [api], templates [], shards [3]/[1] [2024-02-06T19:41:30,438][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-neural-multi-doc-three-shards-index/TShSDVrbRXaqWaxpL6vIJg] [2024-02-06T19:41:30,528][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-neural-multi-doc-three-shards-index/TShSDVrbRXaqWaxpL6vIJg] [2024-02-06T19:41:30,532][INFO ][o.o.c.m.MetadataMappingService] [node_name_9200] [test-neural-multi-doc-three-shards-index/TShSDVrbRXaqWaxpL6vIJg] update_mapping [_doc] [2024-02-06T19:41:30,680][INFO ][o.o.m.m.MLModelManager ] [node_name_9200] create new model meta doc 2pDwf40Bamv3vXxymNit for register model task 2ZDwf40Bamv3vXxymNiX [2024-02-06T19:41:30,892][INFO ][stdout ] [node_name_9200] Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 1% |█ | traced_small_model.zip Downloading: 2% |█ | traced_small_model.zip Downloading: 3% |██ | traced_small_model.zip Downloading: 4% |██ | traced_small_model.zip Downloading: 5% |███ | traced_small_model.zip Downloading: 6% |███ | traced_small_model.zip Downloading: 7% |███ | traced_small_model.zip Downloading: 8% |████ | traced_small_model.zip Downloading: 9% |████ | traced_small_model.zip Downloading: 10% |█████ | traced_small_model.zip Downloading: 11% |█████ | traced_small_model.zip Downloading: 12% |█████ | traced_small_model.zip Downloading: 13% |██████ | traced_small_model.zip Downloading: 14% |██████ | traced_small_model.zip Downloading: 15% |███████ | traced_small_model.zip Downloading: 16% |███████ | traced_small_model.zip Downloading: 17% |███████ | traced_small_model.zip Downloading: 18% |████████ | traced_small_model.zip Downloading: 19% |████████ | traced_small_model.zip Downloading: 20% |█████████ | traced_small_model.zip Downloading: 21% |█████████ | traced_small_model.zip Downloading: 22% |█████████ | traced_small_model.zip Downloading: 23% |██████████ | traced_small_model.zip Downloading: 24% |██████████ | traced_small_model.zip Downloading: 25% |███████████ | traced_small_model.zip Downloading: 26% |███████████ | traced_small_model.zip Downloading: 27% |███████████ | traced_small_model.zip Downloading: 28% |████████████ | traced_small_model.zip Downloading: 29% |████████████ | traced_small_model.zip Downloading: 30% |█████████████ | traced_small_model.zip Downloading: 31% |█████████████ | traced_small_model.zip Downloading: 32% |█████████████ | traced_small_model.zip Downloading: 33% |██████████████ | traced_small_model.zip Downloading: 34% |██████████████ | traced_small_model.zip Downloading: 35% |███████████████ | traced_small_model.zip Downloading: 36% |███████████████ | traced_small_model.zip Downloading: 37% |███████████████ | traced_small_model.zip Downloading: 38% |████████████████ | traced_small_model.zip Downloading: 39% |████████████████ | traced_small_model.zip Downloading: 40% |█████████████████ | traced_small_model.zip Downloading: 41% |█████████████████ | traced_small_model.zip Downloading: 42% |█████████████████ | traced_small_model.zip Downloading: 43% |██████████████████ | traced_small_model.zip Downloading: 44% |██████████████████ | traced_small_model.zip Downloading: 45% |███████████████████ | traced_small_model.zip Downloading: 46% |███████████████████ | traced_small_model.zip Downloading: 47% |███████████████████ | traced_small_model.zip Downloading: 48% |████████████████████ | traced_small_model.zip Downloading: 49% |████████████████████ | traced_small_model.zip Downloading: 50% |█████████████████████ | traced_small_model.zip Downloading: 51% |█████████████████████ | traced_small_model.zip Downloading: 52% |█████████████████████ | traced_small_model.zip Downloading: 53% |██████████████████████ | traced_small_model.zip Downloading: 54% |██████████████████████ | traced_small_model.zip Downloading: 55% |███████████████████████ | traced_small_model.zip Downloading: 56% |███████████████████████ | traced_small_model.zip Downloading: 57% |███████████████████████ | traced_small_model.zip Downloading: 58% |████████████████████████ | traced_small_model.zip Downloading: 59% |████████████████████████ | traced_small_model.zip Downloading: 60% |█████████████████████████ | traced_small_model.zip Downloading: 61% |█████████████████████████ | traced_small_model.zip Downloading: 62% |█████████████████████████ | traced_small_model.zip Downloading: 63% |██████████████████████████ | traced_small_model.zip Downloading: 64% |██████████████████████████ | traced_small_model.zip Downloading: 65% |███████████████████████████ | traced_small_model.zip Downloading: 66% |███████████████████████████ | traced_small_model.zip Downloading: 67% |███████████████████████████ | traced_small_model.zip Downloading: 68% |████████████████████████████ | traced_small_model.zip Downloading: 69% |████████████████████████████ | traced_small_model.zip Downloading: 70% |█████████████████████████████ | traced_small_model.zip Downloading: 71% |█████████████████████████████ | traced_small_model.zip Downloading: 72% |█████████████████████████████ | traced_small_model.zip Downloading: 73% |██████████████████████████████ | traced_small_model.zip Downloading: 74% |██████████████████████████████ | traced_small_model.zip Downloading: 75% |███████████████████████████████ | traced_small_model.zip Downloading: 76% |███████████████████████████████ | traced_small_model.zip Downloading: 77% |███████████████████████████████ | traced_small_model.zip Downloading: 78% |████████████████████████████████ | traced_small_model.zip Downloading: 79% |████████████████████████████████ | traced_small_model.zip Downloading: 80% |█████████████████████████████████ | traced_small_model.zip Downloading: 81% |█████████████████████████████████ | traced_small_model.zip Downloading: 82% |█████████████████████████████████ | traced_small_model.zip Downloading: 83% |██████████████████████████████████ | traced_small_model.zip Downloading: 84% |██████████████████████████████████ | traced_small_model.zip Downloading: 85% |███████████████████████████████████ | traced_small_model.zip Downloading: 86% |███████████████████████████████████ | traced_small_model.zip Downloading: 87% |███████████████████████████████████ | traced_small_model.zip Downloading: 88% |████████████████████████████████████ | traced_small_model.zip Downloading: 89% |████████████████████████████████████ | traced_small_model.zip Downloading: 90% |█████████████████████████████████████ | traced_small_model.zip Downloading: 91% |█████████████████████████████████████ | traced_small_model.zip Downloading: 92% |█████████████████████████████████████ | traced_small_model.zip Downloading: 93% |██████████████████████████████████████ | traced_small_model.zip Downloading: 94% |██████████████████████████████████████ | traced_small_model.zip Downloading: 95% |███████████████████████████████████████ | traced_small_model.zip Downloading: 96% |███████████████████████████████████████ | traced_small_model.zip Downloading: 97% |███████████████████████████████████████ | traced_small_model.zip Downloading: 98% |████████████████████████████████████████| traced_small_model.zip Downloading: 99% |████████████████████████████████████████| traced_small_model.zip Downloading: 100% |████████████████████████████████████████| traced_small_model.zip [2024-02-06T19:41:31,048][INFO ][o.o.m.m.MLModelManager ] [node_name_9200] Model registered successfully, model id: 2pDwf40Bamv3vXxymNit, task id: 2ZDwf40Bamv3vXxymNiX [2024-02-06T19:41:32,666][INFO ][o.o.m.a.d.TransportDeployModelAction] [node_name_9200] Will deploy model on these nodes: OAYF21-xRbu9dc9StXHecg [2024-02-06T19:41:32,694][ERROR][o.o.m.m.MLModelManager ] [node_name_9200] No controller is deployed because the model 2pDwf40Bamv3vXxymNit is expected not having an enabled model controller. Please use the create model controller api to create one if this is unexpected. [2024-02-06T19:41:32,842][INFO ][o.o.m.e.a.DLModel ] [node_name_9200] Model 2pDwf40Bamv3vXxymNit is successfully deployed on 1 devices [2024-02-06T19:41:32,844][INFO ][o.o.m.a.MLModelAutoReDeployer] [node_name_9200] No models needs to be auto redeployed! [2024-02-06T19:41:32,844][INFO ][o.o.m.a.f.TransportForwardAction] [node_name_9200] deploy model done with state: DEPLOYED, model id: 2pDwf40Bamv3vXxymNit [2024-02-06T19:41:32,844][INFO ][o.o.m.a.d.TransportDeployModelOnNodeAction] [node_name_9200] deploy model task done 25Dwf40Bamv3vXxyoNh7 [2024-02-06T19:41:34,689][INFO ][o.o.n.p.f.NormalizationProcessorFactory] [node_name_9200] Creating search phase results processor of type [normalization-processor] with normalization [MinMaxScoreNormalizationTechnique(TECHNIQUE_NAME=min_max)] and combination [ArithmeticMeanScoreCombinationTechnique(TECHNIQUE_NAME=arithmetic_mean)] [2024-02-06T19:41:34,706][INFO ][o.o.n.p.f.NormalizationProcessorFactory] [node_name_9200] Creating search phase results processor of type [normalization-processor] with normalization [MinMaxScoreNormalizationTechnique(TECHNIQUE_NAME=min_max)] and combination [ArithmeticMeanScoreCombinationTechnique(TECHNIQUE_NAME=arithmetic_mean)] [2024-02-06T19:41:37,751][INFO ][o.o.m.a.m.DeleteModelTransportAction] [node_name_9200] Model controller not deleted due to no model controller found for model: 2pDwf40Bamv3vXxymNit [2024-02-06T19:41:37,759][INFO ][o.o.c.m.MetadataDeleteIndexService] [node_name_9200] [test-neural-multi-doc-three-shards-index/TShSDVrbRXaqWaxpL6vIJg] deleting index [2024-02-06T19:41:37,820][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [false] to [true] [2024-02-06T19:41:37,821][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [false] to [true] [2024-02-06T19:41:37,821][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.allow_registering_model_via_url] from [true] to [false] [2024-02-06T19:41:37,889][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [true] to [false] [2024-02-06T19:41:37,889][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [true] to [false] [2024-02-06T19:41:37,925][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.allow_registering_model_via_url] from [false] to [true] [2024-02-06T19:41:37,937][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-score-combination-neural-multi-doc-one-shard-index/XJahbZnlS0uCia-p9vn64A] [2024-02-06T19:41:37,939][INFO ][o.o.c.m.MetadataCreateIndexService] [node_name_9200] [test-score-combination-neural-multi-doc-one-shard-index] creating index, cause [api], templates [], shards [1]/[1] [2024-02-06T19:41:37,955][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-score-combination-neural-multi-doc-one-shard-index/XJahbZnlS0uCia-p9vn64A] [2024-02-06T19:41:38,028][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-score-combination-neural-multi-doc-one-shard-index/XJahbZnlS0uCia-p9vn64A] [2024-02-06T19:41:38,031][INFO ][o.o.c.m.MetadataMappingService] [node_name_9200] [test-score-combination-neural-multi-doc-one-shard-index/XJahbZnlS0uCia-p9vn64A] update_mapping [_doc] [2024-02-06T19:41:38,162][INFO ][o.o.m.m.MLModelManager ] [node_name_9200] create new model meta doc 3pDwf40Bamv3vXxytdjl for register model task 3ZDwf40Bamv3vXxytdjP [2024-02-06T19:41:38,361][INFO ][stdout ] [node_name_9200] Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 1% |█ | traced_small_model.zip Downloading: 2% |█ | traced_small_model.zip Downloading: 3% |██ | traced_small_model.zip Downloading: 4% |██ | traced_small_model.zip Downloading: 5% |███ | traced_small_model.zip Downloading: 6% |███ | traced_small_model.zip Downloading: 7% |███ | traced_small_model.zip Downloading: 8% |████ | traced_small_model.zip Downloading: 9% |████ | traced_small_model.zip Downloading: 10% |█████ | traced_small_model.zip Downloading: 11% |█████ | traced_small_model.zip Downloading: 12% |█████ | traced_small_model.zip Downloading: 13% |██████ | traced_small_model.zip Downloading: 14% |██████ | traced_small_model.zip Downloading: 15% |███████ | traced_small_model.zip Downloading: 16% |███████ | traced_small_model.zip Downloading: 17% |███████ | traced_small_model.zip Downloading: 18% |████████ | traced_small_model.zip Downloading: 19% |████████ | traced_small_model.zip Downloading: 20% |█████████ | traced_small_model.zip Downloading: 21% |█████████ | traced_small_model.zip Downloading: 22% |█████████ | traced_small_model.zip Downloading: 23% |██████████ | traced_small_model.zip Downloading: 24% |██████████ | traced_small_model.zip Downloading: 25% |███████████ | traced_small_model.zip Downloading: 26% |███████████ | traced_small_model.zip Downloading: 27% |███████████ | traced_small_model.zip Downloading: 28% |████████████ | traced_small_model.zip Downloading: 29% |████████████ | traced_small_model.zip Downloading: 30% |█████████████ | traced_small_model.zip Downloading: 31% |█████████████ | traced_small_model.zip Downloading: 32% |█████████████ | traced_small_model.zip Downloading: 33% |██████████████ | traced_small_model.zip Downloading: 34% |██████████████ | traced_small_model.zip Downloading: 35% |███████████████ | traced_small_model.zip Downloading: 36% |███████████████ | traced_small_model.zip Downloading: 37% |███████████████ | traced_small_model.zip Downloading: 38% |████████████████ | traced_small_model.zip Downloading: 39% |████████████████ | traced_small_model.zip Downloading: 40% |█████████████████ | traced_small_model.zip Downloading: 41% |█████████████████ | traced_small_model.zip Downloading: 42% |█████████████████ | traced_small_model.zip Downloading: 43% |██████████████████ | traced_small_model.zip Downloading: 44% |██████████████████ | traced_small_model.zip Downloading: 45% |███████████████████ | traced_small_model.zip Downloading: 46% |███████████████████ | traced_small_model.zip Downloading: 47% |███████████████████ | traced_small_model.zip Downloading: 48% |████████████████████ | traced_small_model.zip Downloading: 49% |████████████████████ | traced_small_model.zip Downloading: 50% |█████████████████████ | traced_small_model.zip Downloading: 51% |█████████████████████ | traced_small_model.zip Downloading: 52% |█████████████████████ | traced_small_model.zip Downloading: 53% |██████████████████████ | traced_small_model.zip Downloading: 54% |██████████████████████ | traced_small_model.zip Downloading: 55% |███████████████████████ | traced_small_model.zip Downloading: 56% |███████████████████████ | traced_small_model.zip Downloading: 57% |███████████████████████ | traced_small_model.zip Downloading: 58% |████████████████████████ | traced_small_model.zip Downloading: 59% |████████████████████████ | traced_small_model.zip Downloading: 60% |█████████████████████████ | traced_small_model.zip Downloading: 61% |█████████████████████████ | traced_small_model.zip Downloading: 62% |█████████████████████████ | traced_small_model.zip Downloading: 63% |██████████████████████████ | traced_small_model.zip Downloading: 64% |██████████████████████████ | traced_small_model.zip Downloading: 65% |███████████████████████████ | traced_small_model.zip Downloading: 66% |███████████████████████████ | traced_small_model.zip Downloading: 67% |███████████████████████████ | traced_small_model.zip Downloading: 68% |████████████████████████████ | traced_small_model.zip Downloading: 69% |████████████████████████████ | traced_small_model.zip Downloading: 70% |█████████████████████████████ | traced_small_model.zip Downloading: 71% |█████████████████████████████ | traced_small_model.zip Downloading: 72% |█████████████████████████████ | traced_small_model.zip Downloading: 73% |██████████████████████████████ | traced_small_model.zip Downloading: 74% |██████████████████████████████ | traced_small_model.zip Downloading: 75% |███████████████████████████████ | traced_small_model.zip Downloading: 76% |███████████████████████████████ | traced_small_model.zip Downloading: 77% |███████████████████████████████ | traced_small_model.zip Downloading: 78% |████████████████████████████████ | traced_small_model.zip Downloading: 79% |████████████████████████████████ | traced_small_model.zip Downloading: 80% |█████████████████████████████████ | traced_small_model.zip Downloading: 81% |█████████████████████████████████ | traced_small_model.zip Downloading: 82% |█████████████████████████████████ | traced_small_model.zip Downloading: 83% |██████████████████████████████████ | traced_small_model.zip Downloading: 84% |██████████████████████████████████ | traced_small_model.zip Downloading: 85% |███████████████████████████████████ | traced_small_model.zip Downloading: 86% |███████████████████████████████████ | traced_small_model.zip Downloading: 87% |███████████████████████████████████ | traced_small_model.zip Downloading: 88% |████████████████████████████████████ | traced_small_model.zip Downloading: 89% |████████████████████████████████████ | traced_small_model.zip Downloading: 90% |█████████████████████████████████████ | traced_small_model.zip Downloading: 91% |█████████████████████████████████████ | traced_small_model.zip Downloading: 92% |█████████████████████████████████████ | traced_small_model.zip Downloading: 93% |██████████████████████████████████████ | traced_small_model.zip Downloading: 94% |██████████████████████████████████████ | traced_small_model.zip Downloading: 95% |███████████████████████████████████████ | traced_small_model.zip Downloading: 96% |███████████████████████████████████████ | traced_small_model.zip Downloading: 97% |███████████████████████████████████████ | traced_small_model.zip Downloading: 98% |████████████████████████████████████████| traced_small_model.zip Downloading: 99% |████████████████████████████████████████| traced_small_model.zip Downloading: 100% |████████████████████████████████████████| traced_small_model.zip [2024-02-06T19:41:38,518][INFO ][o.o.m.m.MLModelManager ] [node_name_9200] Model registered successfully, model id: 3pDwf40Bamv3vXxytdjl, task id: 3ZDwf40Bamv3vXxytdjP [2024-02-06T19:41:40,146][INFO ][o.o.m.a.d.TransportDeployModelAction] [node_name_9200] Will deploy model on these nodes: OAYF21-xRbu9dc9StXHecg [2024-02-06T19:41:40,173][INFO ][o.o.i.b.HierarchyCircuitBreakerService] [node_name_9200] attempting to trigger G1GC due to high heap usage [1021651312] [2024-02-06T19:41:40,194][INFO ][o.o.i.b.HierarchyCircuitBreakerService] [node_name_9200] GC did bring memory usage down, before [1021651312], after [321551904], allocations [47], duration [21] [2024-02-06T19:41:40,206][ERROR][o.o.m.m.MLModelManager ] [node_name_9200] No controller is deployed because the model 3pDwf40Bamv3vXxytdjl is expected not having an enabled model controller. Please use the create model controller api to create one if this is unexpected. [2024-02-06T19:41:40,352][INFO ][o.o.m.e.a.DLModel ] [node_name_9200] Model 3pDwf40Bamv3vXxytdjl is successfully deployed on 1 devices [2024-02-06T19:41:40,353][INFO ][o.o.m.a.MLModelAutoReDeployer] [node_name_9200] No models needs to be auto redeployed! [2024-02-06T19:41:40,353][INFO ][o.o.m.a.f.TransportForwardAction] [node_name_9200] deploy model done with state: DEPLOYED, model id: 3pDwf40Bamv3vXxytdjl [2024-02-06T19:41:40,354][INFO ][o.o.m.a.d.TransportDeployModelOnNodeAction] [node_name_9200] deploy model task done 35Dwf40Bamv3vXxyvdiy [2024-02-06T19:41:42,210][INFO ][o.o.n.p.f.NormalizationProcessorFactory] [node_name_9200] Creating search phase results processor of type [normalization-processor] with normalization [MinMaxScoreNormalizationTechnique(TECHNIQUE_NAME=min_max)] and combination [HarmonicMeanScoreCombinationTechnique(TECHNIQUE_NAME=harmonic_mean)] [2024-02-06T19:41:42,226][INFO ][o.o.n.p.f.NormalizationProcessorFactory] [node_name_9200] Creating search phase results processor of type [normalization-processor] with normalization [MinMaxScoreNormalizationTechnique(TECHNIQUE_NAME=min_max)] and combination [HarmonicMeanScoreCombinationTechnique(TECHNIQUE_NAME=harmonic_mean)] [2024-02-06T19:41:42,258][INFO ][o.o.n.p.f.NormalizationProcessorFactory] [node_name_9200] Creating search phase results processor of type [normalization-processor] with normalization [L2ScoreNormalizationTechnique(TECHNIQUE_NAME=l2)] and combination [HarmonicMeanScoreCombinationTechnique(TECHNIQUE_NAME=harmonic_mean)] [2024-02-06T19:41:42,273][INFO ][o.o.n.p.f.NormalizationProcessorFactory] [node_name_9200] Creating search phase results processor of type [normalization-processor] with normalization [L2ScoreNormalizationTechnique(TECHNIQUE_NAME=l2)] and combination [HarmonicMeanScoreCombinationTechnique(TECHNIQUE_NAME=harmonic_mean)] [2024-02-06T19:41:45,316][INFO ][o.o.m.a.m.DeleteModelTransportAction] [node_name_9200] Model controller not deleted due to no model controller found for model: 3pDwf40Bamv3vXxytdjl [2024-02-06T19:41:45,323][INFO ][o.o.c.m.MetadataDeleteIndexService] [node_name_9200] [test-score-combination-neural-multi-doc-one-shard-index/XJahbZnlS0uCia-p9vn64A] deleting index [2024-02-06T19:41:45,382][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [false] to [true] [2024-02-06T19:41:45,382][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [false] to [true] [2024-02-06T19:41:45,383][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.allow_registering_model_via_url] from [true] to [false] [2024-02-06T19:41:45,417][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [true] to [false] [2024-02-06T19:41:45,418][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [true] to [false] [2024-02-06T19:41:45,455][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.allow_registering_model_via_url] from [false] to [true] [2024-02-06T19:41:45,465][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-score-combination-neural-multi-doc-one-shard-index/kIxB37jVTq2J8vuahvfPjA] [2024-02-06T19:41:45,467][INFO ][o.o.c.m.MetadataCreateIndexService] [node_name_9200] [test-score-combination-neural-multi-doc-one-shard-index] creating index, cause [api], templates [], shards [1]/[1] [2024-02-06T19:41:45,485][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-score-combination-neural-multi-doc-one-shard-index/kIxB37jVTq2J8vuahvfPjA] [2024-02-06T19:41:45,545][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-score-combination-neural-multi-doc-one-shard-index/kIxB37jVTq2J8vuahvfPjA] [2024-02-06T19:41:45,548][INFO ][o.o.c.m.MetadataMappingService] [node_name_9200] [test-score-combination-neural-multi-doc-one-shard-index/kIxB37jVTq2J8vuahvfPjA] update_mapping [_doc] [2024-02-06T19:41:45,681][INFO ][o.o.m.m.MLModelManager ] [node_name_9200] create new model meta doc 4pDwf40Bamv3vXxy09hG for register model task 4ZDwf40Bamv3vXxy09gv [2024-02-06T19:41:45,877][INFO ][stdout ] [node_name_9200] Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 1% |█ | traced_small_model.zip Downloading: 2% |█ | traced_small_model.zip Downloading: 3% |██ | traced_small_model.zip Downloading: 4% |██ | traced_small_model.zip Downloading: 5% |███ | traced_small_model.zip Downloading: 6% |███ | traced_small_model.zip Downloading: 7% |███ | traced_small_model.zip Downloading: 8% |████ | traced_small_model.zip Downloading: 9% |████ | traced_small_model.zip Downloading: 10% |█████ | traced_small_model.zip Downloading: 11% |█████ | traced_small_model.zip Downloading: 12% |█████ | traced_small_model.zip Downloading: 13% |██████ | traced_small_model.zip Downloading: 14% |██████ | traced_small_model.zip Downloading: 15% |███████ | traced_small_model.zip Downloading: 16% |███████ | traced_small_model.zip Downloading: 17% |███████ | traced_small_model.zip Downloading: 18% |████████ | traced_small_model.zip Downloading: 19% |████████ | traced_small_model.zip Downloading: 20% |█████████ | traced_small_model.zip Downloading: 21% |█████████ | traced_small_model.zip Downloading: 22% |█████████ | traced_small_model.zip Downloading: 23% |██████████ | traced_small_model.zip Downloading: 24% |██████████ | traced_small_model.zip Downloading: 25% |███████████ | traced_small_model.zip Downloading: 26% |███████████ | traced_small_model.zip Downloading: 27% |███████████ | traced_small_model.zip Downloading: 28% |████████████ | traced_small_model.zip Downloading: 29% |████████████ | traced_small_model.zip Downloading: 30% |█████████████ | traced_small_model.zip Downloading: 31% |█████████████ | traced_small_model.zip Downloading: 32% |█████████████ | traced_small_model.zip Downloading: 33% |██████████████ | traced_small_model.zip Downloading: 34% |██████████████ | traced_small_model.zip Downloading: 35% |███████████████ | traced_small_model.zip Downloading: 36% |███████████████ | traced_small_model.zip Downloading: 37% |███████████████ | traced_small_model.zip Downloading: 38% |████████████████ | traced_small_model.zip Downloading: 39% |████████████████ | traced_small_model.zip Downloading: 40% |█████████████████ | traced_small_model.zip Downloading: 41% |█████████████████ | traced_small_model.zip Downloading: 42% |█████████████████ | traced_small_model.zip Downloading: 43% |██████████████████ | traced_small_model.zip Downloading: 44% |██████████████████ | traced_small_model.zip Downloading: 45% |███████████████████ | traced_small_model.zip Downloading: 46% |███████████████████ | traced_small_model.zip Downloading: 47% |███████████████████ | traced_small_model.zip Downloading: 48% |████████████████████ | traced_small_model.zip Downloading: 49% |████████████████████ | traced_small_model.zip Downloading: 50% |█████████████████████ | traced_small_model.zip Downloading: 51% |█████████████████████ | traced_small_model.zip Downloading: 52% |█████████████████████ | traced_small_model.zip Downloading: 53% |██████████████████████ | traced_small_model.zip Downloading: 54% |██████████████████████ | traced_small_model.zip Downloading: 55% |███████████████████████ | traced_small_model.zip Downloading: 56% |███████████████████████ | traced_small_model.zip Downloading: 57% |███████████████████████ | traced_small_model.zip Downloading: 58% |████████████████████████ | traced_small_model.zip Downloading: 59% |████████████████████████ | traced_small_model.zip Downloading: 60% |█████████████████████████ | traced_small_model.zip Downloading: 61% |█████████████████████████ | traced_small_model.zip Downloading: 62% |█████████████████████████ | traced_small_model.zip Downloading: 63% |██████████████████████████ | traced_small_model.zip Downloading: 64% |██████████████████████████ | traced_small_model.zip Downloading: 65% |███████████████████████████ | traced_small_model.zip Downloading: 66% |███████████████████████████ | traced_small_model.zip Downloading: 67% |███████████████████████████ | traced_small_model.zip Downloading: 68% |████████████████████████████ | traced_small_model.zip Downloading: 69% |████████████████████████████ | traced_small_model.zip Downloading: 70% |█████████████████████████████ | traced_small_model.zip Downloading: 71% |█████████████████████████████ | traced_small_model.zip Downloading: 72% |█████████████████████████████ | traced_small_model.zip Downloading: 73% |██████████████████████████████ | traced_small_model.zip Downloading: 74% |██████████████████████████████ | traced_small_model.zip Downloading: 75% |███████████████████████████████ | traced_small_model.zip Downloading: 76% |███████████████████████████████ | traced_small_model.zip Downloading: 77% |███████████████████████████████ | traced_small_model.zip Downloading: 78% |████████████████████████████████ | traced_small_model.zip Downloading: 79% |████████████████████████████████ | traced_small_model.zip Downloading: 80% |█████████████████████████████████ | traced_small_model.zip Downloading: 81% |█████████████████████████████████ | traced_small_model.zip Downloading: 82% |█████████████████████████████████ | traced_small_model.zip Downloading: 83% |██████████████████████████████████ | traced_small_model.zip Downloading: 84% |██████████████████████████████████ | traced_small_model.zip Downloading: 85% |███████████████████████████████████ | traced_small_model.zip Downloading: 86% |███████████████████████████████████ | traced_small_model.zip Downloading: 87% |███████████████████████████████████ | traced_small_model.zip Downloading: 88% |████████████████████████████████████ | traced_small_model.zip Downloading: 89% |████████████████████████████████████ | traced_small_model.zip Downloading: 90% |█████████████████████████████████████ | traced_small_model.zip Downloading: 91% |█████████████████████████████████████ | traced_small_model.zip Downloading: 92% |█████████████████████████████████████ | traced_small_model.zip Downloading: 93% |██████████████████████████████████████ | traced_small_model.zip Downloading: 94% |██████████████████████████████████████ | traced_small_model.zip Downloading: 95% |███████████████████████████████████████ | traced_small_model.zip Downloading: 96% |███████████████████████████████████████ | traced_small_model.zip Downloading: 97% |███████████████████████████████████████ | traced_small_model.zip Downloading: 98% |████████████████████████████████████████| traced_small_model.zip Downloading: 99% |████████████████████████████████████████| traced_small_model.zip Downloading: 100% |████████████████████████████████████████| traced_small_model.zip [2024-02-06T19:41:46,032][INFO ][o.o.m.m.MLModelManager ] [node_name_9200] Model registered successfully, model id: 4pDwf40Bamv3vXxy09hG, task id: 4ZDwf40Bamv3vXxy09gv [2024-02-06T19:41:47,665][INFO ][o.o.m.a.d.TransportDeployModelAction] [node_name_9200] Will deploy model on these nodes: OAYF21-xRbu9dc9StXHecg [2024-02-06T19:41:47,692][ERROR][o.o.m.m.MLModelManager ] [node_name_9200] No controller is deployed because the model 4pDwf40Bamv3vXxy09hG is expected not having an enabled model controller. Please use the create model controller api to create one if this is unexpected. [2024-02-06T19:41:47,840][INFO ][o.o.m.e.a.DLModel ] [node_name_9200] Model 4pDwf40Bamv3vXxy09hG is successfully deployed on 1 devices [2024-02-06T19:41:47,841][INFO ][o.o.m.a.MLModelAutoReDeployer] [node_name_9200] No models needs to be auto redeployed! [2024-02-06T19:41:47,842][INFO ][o.o.m.a.f.TransportForwardAction] [node_name_9200] deploy model done with state: DEPLOYED, model id: 4pDwf40Bamv3vXxy09hG [2024-02-06T19:41:47,842][INFO ][o.o.m.a.d.TransportDeployModelOnNodeAction] [node_name_9200] deploy model task done 45Dwf40Bamv3vXxy29gS [2024-02-06T19:41:49,685][INFO ][o.o.n.p.f.NormalizationProcessorFactory] [node_name_9200] Creating search phase results processor of type [normalization-processor] with normalization [MinMaxScoreNormalizationTechnique(TECHNIQUE_NAME=min_max)] and combination [GeometricMeanScoreCombinationTechnique(TECHNIQUE_NAME=geometric_mean)] [2024-02-06T19:41:49,702][INFO ][o.o.n.p.f.NormalizationProcessorFactory] [node_name_9200] Creating search phase results processor of type [normalization-processor] with normalization [MinMaxScoreNormalizationTechnique(TECHNIQUE_NAME=min_max)] and combination [GeometricMeanScoreCombinationTechnique(TECHNIQUE_NAME=geometric_mean)] [2024-02-06T19:41:49,732][INFO ][o.o.n.p.f.NormalizationProcessorFactory] [node_name_9200] Creating search phase results processor of type [normalization-processor] with normalization [L2ScoreNormalizationTechnique(TECHNIQUE_NAME=l2)] and combination [GeometricMeanScoreCombinationTechnique(TECHNIQUE_NAME=geometric_mean)] [2024-02-06T19:41:49,748][INFO ][o.o.n.p.f.NormalizationProcessorFactory] [node_name_9200] Creating search phase results processor of type [normalization-processor] with normalization [L2ScoreNormalizationTechnique(TECHNIQUE_NAME=l2)] and combination [GeometricMeanScoreCombinationTechnique(TECHNIQUE_NAME=geometric_mean)] [2024-02-06T19:41:52,790][INFO ][o.o.m.a.m.DeleteModelTransportAction] [node_name_9200] Model controller not deleted due to no model controller found for model: 4pDwf40Bamv3vXxy09hG [2024-02-06T19:41:52,799][INFO ][o.o.c.m.MetadataDeleteIndexService] [node_name_9200] [test-score-combination-neural-multi-doc-one-shard-index/kIxB37jVTq2J8vuahvfPjA] deleting index [2024-02-06T19:41:52,861][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [false] to [true] [2024-02-06T19:41:52,862][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [false] to [true] [2024-02-06T19:41:52,862][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.allow_registering_model_via_url] from [true] to [false] [2024-02-06T19:41:52,891][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [true] to [false] [2024-02-06T19:41:52,891][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [true] to [false] [2024-02-06T19:41:52,925][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.allow_registering_model_via_url] from [false] to [true] [2024-02-06T19:41:52,935][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-score-combination-neural-multi-doc-three-shards-index/T5tOGYa0TOCy0cfMqAggfA] [2024-02-06T19:41:52,938][INFO ][o.o.c.m.MetadataCreateIndexService] [node_name_9200] [test-score-combination-neural-multi-doc-three-shards-index] creating index, cause [api], templates [], shards [3]/[1] [2024-02-06T19:41:52,955][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-score-combination-neural-multi-doc-three-shards-index/T5tOGYa0TOCy0cfMqAggfA] [2024-02-06T19:41:53,040][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-score-combination-neural-multi-doc-three-shards-index/T5tOGYa0TOCy0cfMqAggfA] [2024-02-06T19:41:53,043][INFO ][o.o.c.m.MetadataMappingService] [node_name_9200] [test-score-combination-neural-multi-doc-three-shards-index/T5tOGYa0TOCy0cfMqAggfA] update_mapping [_doc] [2024-02-06T19:41:53,196][INFO ][o.o.m.m.MLModelManager ] [node_name_9200] create new model meta doc 5pDwf40Bamv3vXxy8Nif for register model task 5ZDwf40Bamv3vXxy8Nh- [2024-02-06T19:41:53,374][INFO ][stdout ] [node_name_9200] Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 1% |█ | traced_small_model.zip Downloading: 2% |█ | traced_small_model.zip Downloading: 3% |██ | traced_small_model.zip Downloading: 4% |██ | traced_small_model.zip Downloading: 5% |███ | traced_small_model.zip Downloading: 6% |███ | traced_small_model.zip Downloading: 7% |███ | traced_small_model.zip Downloading: 8% |████ | traced_small_model.zip Downloading: 9% |████ | traced_small_model.zip Downloading: 10% |█████ | traced_small_model.zip Downloading: 11% |█████ | traced_small_model.zip Downloading: 12% |█████ | traced_small_model.zip Downloading: 13% |██████ | traced_small_model.zip Downloading: 14% |██████ | traced_small_model.zip Downloading: 15% |███████ | traced_small_model.zip Downloading: 16% |███████ | traced_small_model.zip Downloading: 17% |███████ | traced_small_model.zip Downloading: 18% |████████ | traced_small_model.zip Downloading: 19% |████████ | traced_small_model.zip Downloading: 20% |█████████ | traced_small_model.zip Downloading: 21% |█████████ | traced_small_model.zip Downloading: 22% |█████████ | traced_small_model.zip Downloading: 23% |██████████ | traced_small_model.zip Downloading: 24% |██████████ | traced_small_model.zip Downloading: 25% |███████████ | traced_small_model.zip Downloading: 26% |███████████ | traced_small_model.zip Downloading: 27% |███████████ | traced_small_model.zip Downloading: 28% |████████████ | traced_small_model.zip Downloading: 29% |████████████ | traced_small_model.zip Downloading: 30% |█████████████ | traced_small_model.zip Downloading: 31% |█████████████ | traced_small_model.zip Downloading: 32% |█████████████ | traced_small_model.zip Downloading: 33% |██████████████ | traced_small_model.zip Downloading: 34% |██████████████ | traced_small_model.zip Downloading: 35% |███████████████ | traced_small_model.zip Downloading: 36% |███████████████ | traced_small_model.zip Downloading: 37% |███████████████ | traced_small_model.zip Downloading: 38% |████████████████ | traced_small_model.zip Downloading: 39% |████████████████ | traced_small_model.zip Downloading: 40% |█████████████████ | traced_small_model.zip Downloading: 41% |█████████████████ | traced_small_model.zip Downloading: 42% |█████████████████ | traced_small_model.zip Downloading: 43% |██████████████████ | traced_small_model.zip Downloading: 44% |██████████████████ | traced_small_model.zip Downloading: 45% |███████████████████ | traced_small_model.zip Downloading: 46% |███████████████████ | traced_small_model.zip Downloading: 47% |███████████████████ | traced_small_model.zip Downloading: 48% |████████████████████ | traced_small_model.zip Downloading: 49% |████████████████████ | traced_small_model.zip Downloading: 50% |█████████████████████ | traced_small_model.zip Downloading: 51% |█████████████████████ | traced_small_model.zip Downloading: 52% |█████████████████████ | traced_small_model.zip Downloading: 53% |██████████████████████ | traced_small_model.zip Downloading: 54% |██████████████████████ | traced_small_model.zip Downloading: 55% |███████████████████████ | traced_small_model.zip Downloading: 56% |███████████████████████ | traced_small_model.zip Downloading: 57% |███████████████████████ | traced_small_model.zip Downloading: 58% |████████████████████████ | traced_small_model.zip Downloading: 59% |████████████████████████ | traced_small_model.zip Downloading: 60% |█████████████████████████ | traced_small_model.zip Downloading: 61% |█████████████████████████ | traced_small_model.zip Downloading: 62% |█████████████████████████ | traced_small_model.zip Downloading: 63% |██████████████████████████ | traced_small_model.zip Downloading: 64% |██████████████████████████ | traced_small_model.zip Downloading: 65% |███████████████████████████ | traced_small_model.zip Downloading: 66% |███████████████████████████ | traced_small_model.zip Downloading: 67% |███████████████████████████ | traced_small_model.zip Downloading: 68% |████████████████████████████ | traced_small_model.zip Downloading: 69% |████████████████████████████ | traced_small_model.zip Downloading: 70% |█████████████████████████████ | traced_small_model.zip Downloading: 71% |█████████████████████████████ | traced_small_model.zip Downloading: 72% |█████████████████████████████ | traced_small_model.zip Downloading: 73% |██████████████████████████████ | traced_small_model.zip Downloading: 74% |██████████████████████████████ | traced_small_model.zip Downloading: 75% |███████████████████████████████ | traced_small_model.zip Downloading: 76% |███████████████████████████████ | traced_small_model.zip Downloading: 77% |███████████████████████████████ | traced_small_model.zip Downloading: 78% |████████████████████████████████ | traced_small_model.zip Downloading: 79% |████████████████████████████████ | traced_small_model.zip Downloading: 80% |█████████████████████████████████ | traced_small_model.zip Downloading: 81% |█████████████████████████████████ | traced_small_model.zip Downloading: 82% |█████████████████████████████████ | traced_small_model.zip Downloading: 83% |██████████████████████████████████ | traced_small_model.zip Downloading: 84% |██████████████████████████████████ | traced_small_model.zip Downloading: 85% |███████████████████████████████████ | traced_small_model.zip Downloading: 86% |███████████████████████████████████ | traced_small_model.zip Downloading: 87% |███████████████████████████████████ | traced_small_model.zip Downloading: 88% |████████████████████████████████████ | traced_small_model.zip Downloading: 89% |████████████████████████████████████ | traced_small_model.zip Downloading: 90% |█████████████████████████████████████ | traced_small_model.zip Downloading: 91% |█████████████████████████████████████ | traced_small_model.zip Downloading: 92% |█████████████████████████████████████ | traced_small_model.zip Downloading: 93% |██████████████████████████████████████ | traced_small_model.zip Downloading: 94% |██████████████████████████████████████ | traced_small_model.zip Downloading: 95% |███████████████████████████████████████ | traced_small_model.zip Downloading: 96% |███████████████████████████████████████ | traced_small_model.zip Downloading: 97% |███████████████████████████████████████ | traced_small_model.zip Downloading: 98% |████████████████████████████████████████| traced_small_model.zip Downloading: 99% |████████████████████████████████████████| traced_small_model.zip Downloading: 100% |████████████████████████████████████████| traced_small_model.zip [2024-02-06T19:41:53,522][INFO ][o.o.m.m.MLModelManager ] [node_name_9200] Model registered successfully, model id: 5pDwf40Bamv3vXxy8Nif, task id: 5ZDwf40Bamv3vXxy8Nh- [2024-02-06T19:41:55,168][INFO ][o.o.m.a.d.TransportDeployModelAction] [node_name_9200] Will deploy model on these nodes: OAYF21-xRbu9dc9StXHecg [2024-02-06T19:41:55,198][ERROR][o.o.m.a.f.TransportForwardAction] [node_name_9200] deploy model failed on all nodes, model id: 5pDwf40Bamv3vXxy8Nif [2024-02-06T19:41:55,198][INFO ][o.o.m.a.f.TransportForwardAction] [node_name_9200] deploy model done with state: DEPLOY_FAILED, model id: 5pDwf40Bamv3vXxy8Nif [2024-02-06T19:41:55,199][INFO ][o.o.m.a.d.TransportDeployModelOnNodeAction] [node_name_9200] deploy model task done 55Dwf40Bamv3vXxy-Nhg [2024-02-06T19:41:57,190][INFO ][o.o.n.p.f.NormalizationProcessorFactory] [node_name_9200] Creating search phase results processor of type [normalization-processor] with normalization [MinMaxScoreNormalizationTechnique(TECHNIQUE_NAME=min_max)] and combination [ArithmeticMeanScoreCombinationTechnique(TECHNIQUE_NAME=arithmetic_mean)] [2024-02-06T19:41:57,206][INFO ][o.o.n.p.f.NormalizationProcessorFactory] [node_name_9200] Creating search phase results processor of type [normalization-processor] with normalization [MinMaxScoreNormalizationTechnique(TECHNIQUE_NAME=min_max)] and combination [ArithmeticMeanScoreCombinationTechnique(TECHNIQUE_NAME=arithmetic_mean)] [2024-02-06T19:41:57,230][INFO ][o.o.n.p.f.NormalizationProcessorFactory] [node_name_9200] Creating search phase results processor of type [normalization-processor] with normalization [MinMaxScoreNormalizationTechnique(TECHNIQUE_NAME=min_max)] and combination [ArithmeticMeanScoreCombinationTechnique(TECHNIQUE_NAME=arithmetic_mean)] [2024-02-06T19:41:57,244][INFO ][o.o.n.p.f.NormalizationProcessorFactory] [node_name_9200] Creating search phase results processor of type [normalization-processor] with normalization [MinMaxScoreNormalizationTechnique(TECHNIQUE_NAME=min_max)] and combination [ArithmeticMeanScoreCombinationTechnique(TECHNIQUE_NAME=arithmetic_mean)] [2024-02-06T19:41:57,266][INFO ][o.o.n.p.f.NormalizationProcessorFactory] [node_name_9200] Creating search phase results processor of type [normalization-processor] with normalization [MinMaxScoreNormalizationTechnique(TECHNIQUE_NAME=min_max)] and combination [ArithmeticMeanScoreCombinationTechnique(TECHNIQUE_NAME=arithmetic_mean)] [2024-02-06T19:41:57,281][INFO ][o.o.n.p.f.NormalizationProcessorFactory] [node_name_9200] Creating search phase results processor of type [normalization-processor] with normalization [MinMaxScoreNormalizationTechnique(TECHNIQUE_NAME=min_max)] and combination [ArithmeticMeanScoreCombinationTechnique(TECHNIQUE_NAME=arithmetic_mean)] [2024-02-06T19:41:57,304][INFO ][o.o.n.p.f.NormalizationProcessorFactory] [node_name_9200] Creating search phase results processor of type [normalization-processor] with normalization [MinMaxScoreNormalizationTechnique(TECHNIQUE_NAME=min_max)] and combination [ArithmeticMeanScoreCombinationTechnique(TECHNIQUE_NAME=arithmetic_mean)] [2024-02-06T19:41:57,318][INFO ][o.o.n.p.f.NormalizationProcessorFactory] [node_name_9200] Creating search phase results processor of type [normalization-processor] with normalization [MinMaxScoreNormalizationTechnique(TECHNIQUE_NAME=min_max)] and combination [ArithmeticMeanScoreCombinationTechnique(TECHNIQUE_NAME=arithmetic_mean)] [2024-02-06T19:42:00,350][INFO ][o.o.m.a.m.DeleteModelTransportAction] [node_name_9200] Model controller not deleted due to no model controller found for model: 5pDwf40Bamv3vXxy8Nif [2024-02-06T19:42:00,357][INFO ][o.o.c.m.MetadataDeleteIndexService] [node_name_9200] [test-score-combination-neural-multi-doc-three-shards-index/T5tOGYa0TOCy0cfMqAggfA] deleting index [2024-02-06T19:42:00,414][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [false] to [true] [2024-02-06T19:42:00,414][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [false] to [true] [2024-02-06T19:42:00,415][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.allow_registering_model_via_url] from [true] to [false] [2024-02-06T19:42:00,416][INFO ][o.o.i.b.HierarchyCircuitBreakerService] [node_name_9200] attempting to trigger G1GC due to high heap usage [1020952096] [2024-02-06T19:42:00,426][INFO ][o.o.i.b.HierarchyCircuitBreakerService] [node_name_9200] GC did bring memory usage down, before [1020952096], after [507777512], allocations [49], duration [10] [2024-02-06T19:42:00,490][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [true] to [false] [2024-02-06T19:42:00,490][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [true] to [false] [2024-02-06T19:42:00,528][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.allow_registering_model_via_url] from [false] to [true] [2024-02-06T19:42:00,533][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-score-normalization-neural-multi-doc-one-shard-index/kRDXeaXcTz-kASYLM9T9Rg] [2024-02-06T19:42:00,535][INFO ][o.o.c.m.MetadataCreateIndexService] [node_name_9200] [test-score-normalization-neural-multi-doc-one-shard-index] creating index, cause [api], templates [], shards [1]/[1] [2024-02-06T19:42:00,549][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-score-normalization-neural-multi-doc-one-shard-index/kRDXeaXcTz-kASYLM9T9Rg] [2024-02-06T19:42:00,604][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-score-normalization-neural-multi-doc-one-shard-index/kRDXeaXcTz-kASYLM9T9Rg] [2024-02-06T19:42:00,606][INFO ][o.o.c.m.MetadataMappingService] [node_name_9200] [test-score-normalization-neural-multi-doc-one-shard-index/kRDXeaXcTz-kASYLM9T9Rg] update_mapping [_doc] [2024-02-06T19:42:00,733][INFO ][o.o.m.m.MLModelManager ] [node_name_9200] create new model meta doc 6pDxf40Bamv3vXxyDtgS for register model task 6ZDxf40Bamv3vXxyDdj6 [2024-02-06T19:42:00,921][INFO ][stdout ] [node_name_9200] Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 1% |█ | traced_small_model.zip Downloading: 2% |█ | traced_small_model.zip Downloading: 3% |██ | traced_small_model.zip Downloading: 4% |██ | traced_small_model.zip Downloading: 5% |███ | traced_small_model.zip Downloading: 6% |███ | traced_small_model.zip Downloading: 7% |███ | traced_small_model.zip Downloading: 8% |████ | traced_small_model.zip Downloading: 9% |████ | traced_small_model.zip Downloading: 10% |█████ | traced_small_model.zip Downloading: 11% |█████ | traced_small_model.zip Downloading: 12% |█████ | traced_small_model.zip Downloading: 13% |██████ | traced_small_model.zip Downloading: 14% |██████ | traced_small_model.zip Downloading: 15% |███████ | traced_small_model.zip Downloading: 16% |███████ | traced_small_model.zip Downloading: 17% |███████ | traced_small_model.zip Downloading: 18% |████████ | traced_small_model.zip Downloading: 19% |████████ | traced_small_model.zip Downloading: 20% |█████████ | traced_small_model.zip Downloading: 21% |█████████ | traced_small_model.zip Downloading: 22% |█████████ | traced_small_model.zip Downloading: 23% |██████████ | traced_small_model.zip Downloading: 24% |██████████ | traced_small_model.zip Downloading: 25% |███████████ | traced_small_model.zip Downloading: 26% |███████████ | traced_small_model.zip Downloading: 27% |███████████ | traced_small_model.zip Downloading: 28% |████████████ | traced_small_model.zip Downloading: 29% |████████████ | traced_small_model.zip Downloading: 30% |█████████████ | traced_small_model.zip Downloading: 31% |█████████████ | traced_small_model.zip Downloading: 32% |█████████████ | traced_small_model.zip Downloading: 33% |██████████████ | traced_small_model.zip Downloading: 34% |██████████████ | traced_small_model.zip Downloading: 35% |███████████████ | traced_small_model.zip Downloading: 36% |███████████████ | traced_small_model.zip Downloading: 37% |███████████████ | traced_small_model.zip Downloading: 38% |████████████████ | traced_small_model.zip Downloading: 39% |████████████████ | traced_small_model.zip Downloading: 40% |█████████████████ | traced_small_model.zip Downloading: 41% |█████████████████ | traced_small_model.zip Downloading: 42% |█████████████████ | traced_small_model.zip Downloading: 43% |██████████████████ | traced_small_model.zip Downloading: 44% |██████████████████ | traced_small_model.zip Downloading: 45% |███████████████████ | traced_small_model.zip Downloading: 46% |███████████████████ | traced_small_model.zip Downloading: 47% |███████████████████ | traced_small_model.zip Downloading: 48% |████████████████████ | traced_small_model.zip Downloading: 49% |████████████████████ | traced_small_model.zip Downloading: 50% |█████████████████████ | traced_small_model.zip Downloading: 51% |█████████████████████ | traced_small_model.zip Downloading: 52% |█████████████████████ | traced_small_model.zip Downloading: 53% |██████████████████████ | traced_small_model.zip Downloading: 54% |██████████████████████ | traced_small_model.zip Downloading: 55% |███████████████████████ | traced_small_model.zip Downloading: 56% |███████████████████████ | traced_small_model.zip Downloading: 57% |███████████████████████ | traced_small_model.zip Downloading: 58% |████████████████████████ | traced_small_model.zip Downloading: 59% |████████████████████████ | traced_small_model.zip Downloading: 60% |█████████████████████████ | traced_small_model.zip Downloading: 61% |█████████████████████████ | traced_small_model.zip Downloading: 62% |█████████████████████████ | traced_small_model.zip Downloading: 63% |██████████████████████████ | traced_small_model.zip Downloading: 64% |██████████████████████████ | traced_small_model.zip Downloading: 65% |███████████████████████████ | traced_small_model.zip Downloading: 66% |███████████████████████████ | traced_small_model.zip Downloading: 67% |███████████████████████████ | traced_small_model.zip Downloading: 68% |████████████████████████████ | traced_small_model.zip Downloading: 69% |████████████████████████████ | traced_small_model.zip Downloading: 70% |█████████████████████████████ | traced_small_model.zip Downloading: 71% |█████████████████████████████ | traced_small_model.zip Downloading: 72% |█████████████████████████████ | traced_small_model.zip Downloading: 73% |██████████████████████████████ | traced_small_model.zip Downloading: 74% |██████████████████████████████ | traced_small_model.zip Downloading: 75% |███████████████████████████████ | traced_small_model.zip Downloading: 76% |███████████████████████████████ | traced_small_model.zip Downloading: 77% |███████████████████████████████ | traced_small_model.zip Downloading: 78% |████████████████████████████████ | traced_small_model.zip Downloading: 79% |████████████████████████████████ | traced_small_model.zip Downloading: 80% |█████████████████████████████████ | traced_small_model.zip Downloading: 81% |█████████████████████████████████ | traced_small_model.zip Downloading: 82% |█████████████████████████████████ | traced_small_model.zip Downloading: 83% |██████████████████████████████████ | traced_small_model.zip Downloading: 84% |██████████████████████████████████ | traced_small_model.zip Downloading: 85% |███████████████████████████████████ | traced_small_model.zip Downloading: 86% |███████████████████████████████████ | traced_small_model.zip Downloading: 87% |███████████████████████████████████ | traced_small_model.zip Downloading: 88% |████████████████████████████████████ | traced_small_model.zip Downloading: 89% |████████████████████████████████████ | traced_small_model.zip Downloading: 90% |█████████████████████████████████████ | traced_small_model.zip Downloading: 91% |█████████████████████████████████████ | traced_small_model.zip Downloading: 92% |█████████████████████████████████████ | traced_small_model.zip Downloading: 93% |██████████████████████████████████████ | traced_small_model.zip Downloading: 94% |██████████████████████████████████████ | traced_small_model.zip Downloading: 95% |███████████████████████████████████████ | traced_small_model.zip Downloading: 96% |███████████████████████████████████████ | traced_small_model.zip Downloading: 97% |███████████████████████████████████████ | traced_small_model.zip Downloading: 98% |████████████████████████████████████████| traced_small_model.zip Downloading: 99% |████████████████████████████████████████| traced_small_model.zip Downloading: 100% |████████████████████████████████████████| traced_small_model.zip [2024-02-06T19:42:01,102][INFO ][o.o.m.m.MLModelManager ] [node_name_9200] Model registered successfully, model id: 6pDxf40Bamv3vXxyDtgS, task id: 6ZDxf40Bamv3vXxyDdj6 [2024-02-06T19:42:02,717][INFO ][o.o.m.a.d.TransportDeployModelAction] [node_name_9200] Will deploy model on these nodes: OAYF21-xRbu9dc9StXHecg [2024-02-06T19:42:02,742][ERROR][o.o.m.m.MLModelManager ] [node_name_9200] No controller is deployed because the model 6pDxf40Bamv3vXxyDtgS is expected not having an enabled model controller. Please use the create model controller api to create one if this is unexpected. [2024-02-06T19:42:02,896][INFO ][o.o.m.e.a.DLModel ] [node_name_9200] Model 6pDxf40Bamv3vXxyDtgS is successfully deployed on 1 devices [2024-02-06T19:42:02,897][INFO ][o.o.m.a.MLModelAutoReDeployer] [node_name_9200] No models needs to be auto redeployed! [2024-02-06T19:42:02,898][INFO ][o.o.m.a.f.TransportForwardAction] [node_name_9200] deploy model done with state: DEPLOYED, model id: 6pDxf40Bamv3vXxyDtgS [2024-02-06T19:42:02,898][INFO ][o.o.m.a.d.TransportDeployModelOnNodeAction] [node_name_9200] deploy model task done 65Dxf40Bamv3vXxyFdjd [2024-02-06T19:42:04,738][INFO ][o.o.n.p.f.NormalizationProcessorFactory] [node_name_9200] Creating search phase results processor of type [normalization-processor] with normalization [MinMaxScoreNormalizationTechnique(TECHNIQUE_NAME=min_max)] and combination [ArithmeticMeanScoreCombinationTechnique(TECHNIQUE_NAME=arithmetic_mean)] [2024-02-06T19:42:04,756][INFO ][o.o.n.p.f.NormalizationProcessorFactory] [node_name_9200] Creating search phase results processor of type [normalization-processor] with normalization [MinMaxScoreNormalizationTechnique(TECHNIQUE_NAME=min_max)] and combination [ArithmeticMeanScoreCombinationTechnique(TECHNIQUE_NAME=arithmetic_mean)] [2024-02-06T19:42:04,798][INFO ][o.o.n.p.f.NormalizationProcessorFactory] [node_name_9200] Creating search phase results processor of type [normalization-processor] with normalization [MinMaxScoreNormalizationTechnique(TECHNIQUE_NAME=min_max)] and combination [HarmonicMeanScoreCombinationTechnique(TECHNIQUE_NAME=harmonic_mean)] [2024-02-06T19:42:04,813][INFO ][o.o.n.p.f.NormalizationProcessorFactory] [node_name_9200] Creating search phase results processor of type [normalization-processor] with normalization [MinMaxScoreNormalizationTechnique(TECHNIQUE_NAME=min_max)] and combination [HarmonicMeanScoreCombinationTechnique(TECHNIQUE_NAME=harmonic_mean)] [2024-02-06T19:42:04,843][INFO ][o.o.n.p.f.NormalizationProcessorFactory] [node_name_9200] Creating search phase results processor of type [normalization-processor] with normalization [MinMaxScoreNormalizationTechnique(TECHNIQUE_NAME=min_max)] and combination [GeometricMeanScoreCombinationTechnique(TECHNIQUE_NAME=geometric_mean)] [2024-02-06T19:42:04,857][INFO ][o.o.n.p.f.NormalizationProcessorFactory] [node_name_9200] Creating search phase results processor of type [normalization-processor] with normalization [MinMaxScoreNormalizationTechnique(TECHNIQUE_NAME=min_max)] and combination [GeometricMeanScoreCombinationTechnique(TECHNIQUE_NAME=geometric_mean)] [2024-02-06T19:42:07,899][INFO ][o.o.m.a.m.DeleteModelTransportAction] [node_name_9200] Model controller not deleted due to no model controller found for model: 6pDxf40Bamv3vXxyDtgS [2024-02-06T19:42:07,907][INFO ][o.o.c.m.MetadataDeleteIndexService] [node_name_9200] [test-score-normalization-neural-multi-doc-one-shard-index/kRDXeaXcTz-kASYLM9T9Rg] deleting index [2024-02-06T19:42:07,968][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [false] to [true] [2024-02-06T19:42:07,968][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [false] to [true] [2024-02-06T19:42:07,969][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.allow_registering_model_via_url] from [true] to [false] [2024-02-06T19:42:08,000][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [true] to [false] [2024-02-06T19:42:08,001][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [true] to [false] [2024-02-06T19:42:08,033][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.allow_registering_model_via_url] from [false] to [true] [2024-02-06T19:42:08,038][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-score-normalization-neural-multi-doc-one-shard-index/1Fahum3vSMW1ykHcMtQ2Pg] [2024-02-06T19:42:08,041][INFO ][o.o.c.m.MetadataCreateIndexService] [node_name_9200] [test-score-normalization-neural-multi-doc-one-shard-index] creating index, cause [api], templates [], shards [1]/[1] [2024-02-06T19:42:08,063][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-score-normalization-neural-multi-doc-one-shard-index/1Fahum3vSMW1ykHcMtQ2Pg] [2024-02-06T19:42:08,119][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-score-normalization-neural-multi-doc-one-shard-index/1Fahum3vSMW1ykHcMtQ2Pg] [2024-02-06T19:42:08,122][INFO ][o.o.c.m.MetadataMappingService] [node_name_9200] [test-score-normalization-neural-multi-doc-one-shard-index/1Fahum3vSMW1ykHcMtQ2Pg] update_mapping [_doc] [2024-02-06T19:42:08,249][INFO ][o.o.m.m.MLModelManager ] [node_name_9200] create new model meta doc 7pDxf40Bamv3vXxyK9ht for register model task 7ZDxf40Bamv3vXxyK9hZ [2024-02-06T19:42:08,456][INFO ][stdout ] [node_name_9200] Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 1% |█ | traced_small_model.zip Downloading: 2% |█ | traced_small_model.zip Downloading: 3% |██ | traced_small_model.zip Downloading: 4% |██ | traced_small_model.zip Downloading: 5% |███ | traced_small_model.zip Downloading: 6% |███ | traced_small_model.zip Downloading: 7% |███ | traced_small_model.zip Downloading: 8% |████ | traced_small_model.zip Downloading: 9% |████ | traced_small_model.zip Downloading: 10% |█████ | traced_small_model.zip Downloading: 11% |█████ | traced_small_model.zip Downloading: 12% |█████ | traced_small_model.zip Downloading: 13% |██████ | traced_small_model.zip Downloading: 14% |██████ | traced_small_model.zip Downloading: 15% |███████ | traced_small_model.zip Downloading: 16% |███████ | traced_small_model.zip Downloading: 17% |███████ | traced_small_model.zip Downloading: 18% |████████ | traced_small_model.zip Downloading: 19% |████████ | traced_small_model.zip Downloading: 20% |█████████ | traced_small_model.zip Downloading: 21% |█████████ | traced_small_model.zip Downloading: 22% |█████████ | traced_small_model.zip Downloading: 23% |██████████ | traced_small_model.zip Downloading: 24% |██████████ | traced_small_model.zip Downloading: 25% |███████████ | traced_small_model.zip Downloading: 26% |███████████ | traced_small_model.zip Downloading: 27% |███████████ | traced_small_model.zip Downloading: 28% |████████████ | traced_small_model.zip Downloading: 29% |████████████ | traced_small_model.zip Downloading: 30% |█████████████ | traced_small_model.zip Downloading: 31% |█████████████ | traced_small_model.zip Downloading: 32% |█████████████ | traced_small_model.zip Downloading: 33% |██████████████ | traced_small_model.zip Downloading: 34% |██████████████ | traced_small_model.zip Downloading: 35% |███████████████ | traced_small_model.zip Downloading: 36% |███████████████ | traced_small_model.zip Downloading: 37% |███████████████ | traced_small_model.zip Downloading: 38% |████████████████ | traced_small_model.zip Downloading: 39% |████████████████ | traced_small_model.zip Downloading: 40% |█████████████████ | traced_small_model.zip Downloading: 41% |█████████████████ | traced_small_model.zip Downloading: 42% |█████████████████ | traced_small_model.zip Downloading: 43% |██████████████████ | traced_small_model.zip Downloading: 44% |██████████████████ | traced_small_model.zip Downloading: 45% |███████████████████ | traced_small_model.zip Downloading: 46% |███████████████████ | traced_small_model.zip Downloading: 47% |███████████████████ | traced_small_model.zip Downloading: 48% |████████████████████ | traced_small_model.zip Downloading: 49% |████████████████████ | traced_small_model.zip Downloading: 50% |█████████████████████ | traced_small_model.zip Downloading: 51% |█████████████████████ | traced_small_model.zip Downloading: 52% |█████████████████████ | traced_small_model.zip Downloading: 53% |██████████████████████ | traced_small_model.zip Downloading: 54% |██████████████████████ | traced_small_model.zip Downloading: 55% |███████████████████████ | traced_small_model.zip Downloading: 56% |███████████████████████ | traced_small_model.zip Downloading: 57% |███████████████████████ | traced_small_model.zip Downloading: 58% |████████████████████████ | traced_small_model.zip Downloading: 59% |████████████████████████ | traced_small_model.zip Downloading: 60% |█████████████████████████ | traced_small_model.zip Downloading: 61% |█████████████████████████ | traced_small_model.zip Downloading: 62% |█████████████████████████ | traced_small_model.zip Downloading: 63% |██████████████████████████ | traced_small_model.zip Downloading: 64% |██████████████████████████ | traced_small_model.zip Downloading: 65% |███████████████████████████ | traced_small_model.zip Downloading: 66% |███████████████████████████ | traced_small_model.zip Downloading: 67% |███████████████████████████ | traced_small_model.zip Downloading: 68% |████████████████████████████ | traced_small_model.zip Downloading: 69% |████████████████████████████ | traced_small_model.zip Downloading: 70% |█████████████████████████████ | traced_small_model.zip Downloading: 71% |█████████████████████████████ | traced_small_model.zip Downloading: 72% |█████████████████████████████ | traced_small_model.zip Downloading: 73% |██████████████████████████████ | traced_small_model.zip Downloading: 74% |██████████████████████████████ | traced_small_model.zip Downloading: 75% |███████████████████████████████ | traced_small_model.zip Downloading: 76% |███████████████████████████████ | traced_small_model.zip Downloading: 77% |███████████████████████████████ | traced_small_model.zip Downloading: 78% |████████████████████████████████ | traced_small_model.zip Downloading: 79% |████████████████████████████████ | traced_small_model.zip Downloading: 80% |█████████████████████████████████ | traced_small_model.zip Downloading: 81% |█████████████████████████████████ | traced_small_model.zip Downloading: 82% |█████████████████████████████████ | traced_small_model.zip Downloading: 83% |██████████████████████████████████ | traced_small_model.zip Downloading: 84% |██████████████████████████████████ | traced_small_model.zip Downloading: 85% |███████████████████████████████████ | traced_small_model.zip Downloading: 86% |███████████████████████████████████ | traced_small_model.zip Downloading: 87% |███████████████████████████████████ | traced_small_model.zip Downloading: 88% |████████████████████████████████████ | traced_small_model.zip Downloading: 89% |████████████████████████████████████ | traced_small_model.zip Downloading: 90% |█████████████████████████████████████ | traced_small_model.zip Downloading: 91% |█████████████████████████████████████ | traced_small_model.zip Downloading: 92% |█████████████████████████████████████ | traced_small_model.zip Downloading: 93% |██████████████████████████████████████ | traced_small_model.zip Downloading: 94% |██████████████████████████████████████ | traced_small_model.zip Downloading: 95% |███████████████████████████████████████ | traced_small_model.zip Downloading: 96% |███████████████████████████████████████ | traced_small_model.zip Downloading: 97% |███████████████████████████████████████ | traced_small_model.zip Downloading: 98% |████████████████████████████████████████| traced_small_model.zip Downloading: 99% |████████████████████████████████████████| traced_small_model.zip Downloading: 100% |████████████████████████████████████████| traced_small_model.zip [2024-02-06T19:42:08,611][INFO ][o.o.m.m.MLModelManager ] [node_name_9200] Model registered successfully, model id: 7pDxf40Bamv3vXxyK9ht, task id: 7ZDxf40Bamv3vXxyK9hZ [2024-02-06T19:42:10,233][INFO ][o.o.m.a.d.TransportDeployModelAction] [node_name_9200] Will deploy model on these nodes: OAYF21-xRbu9dc9StXHecg [2024-02-06T19:42:10,263][ERROR][o.o.m.m.MLModelManager ] [node_name_9200] No controller is deployed because the model 7pDxf40Bamv3vXxyK9ht is expected not having an enabled model controller. Please use the create model controller api to create one if this is unexpected. [2024-02-06T19:42:10,416][INFO ][o.o.m.e.a.DLModel ] [node_name_9200] Model 7pDxf40Bamv3vXxyK9ht is successfully deployed on 1 devices [2024-02-06T19:42:10,418][INFO ][o.o.m.a.MLModelAutoReDeployer] [node_name_9200] No models needs to be auto redeployed! [2024-02-06T19:42:10,418][INFO ][o.o.m.a.f.TransportForwardAction] [node_name_9200] deploy model done with state: DEPLOYED, model id: 7pDxf40Bamv3vXxyK9ht [2024-02-06T19:42:10,418][INFO ][o.o.m.a.d.TransportDeployModelOnNodeAction] [node_name_9200] deploy model task done 75Dxf40Bamv3vXxyM9g5 [2024-02-06T19:42:12,253][INFO ][o.o.n.p.f.NormalizationProcessorFactory] [node_name_9200] Creating search phase results processor of type [normalization-processor] with normalization [L2ScoreNormalizationTechnique(TECHNIQUE_NAME=l2)] and combination [ArithmeticMeanScoreCombinationTechnique(TECHNIQUE_NAME=arithmetic_mean)] [2024-02-06T19:42:12,268][INFO ][o.o.n.p.f.NormalizationProcessorFactory] [node_name_9200] Creating search phase results processor of type [normalization-processor] with normalization [L2ScoreNormalizationTechnique(TECHNIQUE_NAME=l2)] and combination [ArithmeticMeanScoreCombinationTechnique(TECHNIQUE_NAME=arithmetic_mean)] [2024-02-06T19:42:12,302][INFO ][o.o.n.p.f.NormalizationProcessorFactory] [node_name_9200] Creating search phase results processor of type [normalization-processor] with normalization [L2ScoreNormalizationTechnique(TECHNIQUE_NAME=l2)] and combination [HarmonicMeanScoreCombinationTechnique(TECHNIQUE_NAME=harmonic_mean)] [2024-02-06T19:42:12,316][INFO ][o.o.n.p.f.NormalizationProcessorFactory] [node_name_9200] Creating search phase results processor of type [normalization-processor] with normalization [L2ScoreNormalizationTechnique(TECHNIQUE_NAME=l2)] and combination [HarmonicMeanScoreCombinationTechnique(TECHNIQUE_NAME=harmonic_mean)] [2024-02-06T19:42:12,341][INFO ][o.o.n.p.f.NormalizationProcessorFactory] [node_name_9200] Creating search phase results processor of type [normalization-processor] with normalization [L2ScoreNormalizationTechnique(TECHNIQUE_NAME=l2)] and combination [GeometricMeanScoreCombinationTechnique(TECHNIQUE_NAME=geometric_mean)] [2024-02-06T19:42:12,354][INFO ][o.o.n.p.f.NormalizationProcessorFactory] [node_name_9200] Creating search phase results processor of type [normalization-processor] with normalization [L2ScoreNormalizationTechnique(TECHNIQUE_NAME=l2)] and combination [GeometricMeanScoreCombinationTechnique(TECHNIQUE_NAME=geometric_mean)] [2024-02-06T19:42:15,393][INFO ][o.o.m.a.m.DeleteModelTransportAction] [node_name_9200] Model controller not deleted due to no model controller found for model: 7pDxf40Bamv3vXxyK9ht [2024-02-06T19:42:15,401][INFO ][o.o.c.m.MetadataDeleteIndexService] [node_name_9200] [test-score-normalization-neural-multi-doc-one-shard-index/1Fahum3vSMW1ykHcMtQ2Pg] deleting index [2024-02-06T19:42:15,455][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [false] to [true] [2024-02-06T19:42:15,456][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [false] to [true] [2024-02-06T19:42:15,456][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.allow_registering_model_via_url] from [true] to [false] [2024-02-06T19:42:15,525][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [true] to [false] [2024-02-06T19:42:15,525][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [true] to [false] [2024-02-06T19:42:15,558][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.allow_registering_model_via_url] from [false] to [true] [2024-02-06T19:42:15,612][INFO ][o.o.m.m.MLModelManager ] [node_name_9200] create new model meta doc 8pDxf40Bamv3vXxySNgv for register model task 8ZDxf40Bamv3vXxySNga [2024-02-06T19:42:15,838][INFO ][stdout ] [node_name_9200] Downloading: 8% |████ | tokenize-idf-0915.zip Downloading: 16% |███████ | tokenize-idf-0915.zip Downloading: 25% |███████████ | tokenize-idf-0915.zip Downloading: 33% |██████████████ | tokenize-idf-0915.zip Downloading: 41% |█████████████████ | tokenize-idf-0915.zip Downloading: 50% |█████████████████████ | tokenize-idf-0915.zip Downloading: 58% |████████████████████████ | tokenize-idf-0915.zip Downloading: 67% |███████████████████████████ | tokenize-idf-0915.zip Downloading: 75% |███████████████████████████████ | tokenize-idf-0915.zip Downloading: 83% |██████████████████████████████████ | tokenize-idf-0915.zip Downloading: 92% |█████████████████████████████████████ | tokenize-idf-0915.zip Downloading: 100% |████████████████████████████████████████| tokenize-idf-0915.zip [2024-02-06T19:42:15,860][INFO ][o.o.m.m.MLModelManager ] [node_name_9200] Model registered successfully, model id: 8pDxf40Bamv3vXxySNgv, task id: 8ZDxf40Bamv3vXxySNga [2024-02-06T19:42:17,594][INFO ][o.o.m.a.d.TransportDeployModelAction] [node_name_9200] Will deploy model on these nodes: OAYF21-xRbu9dc9StXHecg [2024-02-06T19:42:17,622][ERROR][o.o.m.m.MLModelManager ] [node_name_9200] No controller is deployed because the model 8pDxf40Bamv3vXxySNgv is expected not having an enabled model controller. Please use the create model controller api to create one if this is unexpected. [2024-02-06T19:42:17,655][INFO ][o.o.m.e.a.t.SparseTokenizerModel] [node_name_9200] sparse tokenize Model 8pDxf40Bamv3vXxySNgv is successfully deployed [2024-02-06T19:42:17,656][INFO ][o.o.m.a.MLModelAutoReDeployer] [node_name_9200] No models needs to be auto redeployed! [2024-02-06T19:42:17,656][INFO ][o.o.m.a.f.TransportForwardAction] [node_name_9200] deploy model done with state: DEPLOYED, model id: 8pDxf40Bamv3vXxySNgv [2024-02-06T19:42:17,656][INFO ][o.o.m.a.d.TransportDeployModelOnNodeAction] [node_name_9200] deploy model task done 85Dxf40Bamv3vXxyT9j7 [2024-02-06T19:42:19,634][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[sparse_encoding_index/3Z8V1Ps6Sf-0vl2kgK1N4w] [2024-02-06T19:42:19,638][INFO ][o.o.c.m.MetadataCreateIndexService] [node_name_9200] [sparse_encoding_index] creating index, cause [api], templates [], shards [1]/[1] [2024-02-06T19:42:19,654][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[sparse_encoding_index/3Z8V1Ps6Sf-0vl2kgK1N4w] [2024-02-06T19:42:19,774][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[sparse_encoding_index/3Z8V1Ps6Sf-0vl2kgK1N4w] [2024-02-06T19:42:19,777][INFO ][o.o.c.m.MetadataMappingService] [node_name_9200] [sparse_encoding_index/3Z8V1Ps6Sf-0vl2kgK1N4w] update_mapping [_doc] [2024-02-06T19:42:22,845][INFO ][o.o.m.a.m.DeleteModelTransportAction] [node_name_9200] Model controller not deleted due to no model controller found for model: 8pDxf40Bamv3vXxySNgv [2024-02-06T19:42:22,853][INFO ][o.o.c.m.MetadataDeleteIndexService] [node_name_9200] [sparse_encoding_index/3Z8V1Ps6Sf-0vl2kgK1N4w] deleting index [2024-02-06T19:42:22,907][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [false] to [true] [2024-02-06T19:42:22,908][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [false] to [true] [2024-02-06T19:42:22,908][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.allow_registering_model_via_url] from [true] to [false] [2024-02-06T19:42:22,972][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [true] to [false] [2024-02-06T19:42:22,973][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [true] to [false] [2024-02-06T19:42:23,005][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.allow_registering_model_via_url] from [false] to [true] [2024-02-06T19:42:23,050][WARN ][o.o.m.a.f.TransportForwardAction] [node_name_9200] Memory Circuit Breaker is open, please check your resources! [2024-02-06T19:42:23,050][WARN ][o.o.m.a.r.TransportRegisterModelAction] [node_name_9200] Memory Circuit Breaker is open, please check your resources! [2024-02-06T19:42:24,102][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [false] to [true] [2024-02-06T19:42:24,102][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [false] to [true] [2024-02-06T19:42:24,103][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.allow_registering_model_via_url] from [true] to [false] [2024-02-06T19:42:24,189][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [true] to [false] [2024-02-06T19:42:24,189][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [true] to [false] [2024-02-06T19:42:24,221][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.allow_registering_model_via_url] from [false] to [true] [2024-02-06T19:42:24,289][INFO ][o.o.m.m.MLModelManager ] [node_name_9200] create new model meta doc -ZDxf40Bamv3vXxyatgV for register model task -JDxf40Bamv3vXxyadjz [2024-02-06T19:42:24,480][INFO ][stdout ] [node_name_9200] Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 1% |█ | traced_small_model.zip Downloading: 2% |█ | traced_small_model.zip Downloading: 3% |██ | traced_small_model.zip Downloading: 4% |██ | traced_small_model.zip Downloading: 5% |███ | traced_small_model.zip Downloading: 6% |███ | traced_small_model.zip Downloading: 7% |███ | traced_small_model.zip Downloading: 8% |████ | traced_small_model.zip Downloading: 9% |████ | traced_small_model.zip Downloading: 10% |█████ | traced_small_model.zip Downloading: 11% |█████ | traced_small_model.zip Downloading: 12% |█████ | traced_small_model.zip Downloading: 13% |██████ | traced_small_model.zip Downloading: 14% |██████ | traced_small_model.zip Downloading: 15% |███████ | traced_small_model.zip Downloading: 16% |███████ | traced_small_model.zip Downloading: 17% |███████ | traced_small_model.zip Downloading: 18% |████████ | traced_small_model.zip Downloading: 19% |████████ | traced_small_model.zip Downloading: 20% |█████████ | traced_small_model.zip Downloading: 21% |█████████ | traced_small_model.zip Downloading: 22% |█████████ | traced_small_model.zip Downloading: 23% |██████████ | traced_small_model.zip Downloading: 24% |██████████ | traced_small_model.zip Downloading: 25% |███████████ | traced_small_model.zip Downloading: 26% |███████████ | traced_small_model.zip Downloading: 27% |███████████ | traced_small_model.zip Downloading: 28% |████████████ | traced_small_model.zip Downloading: 29% |████████████ | traced_small_model.zip Downloading: 30% |█████████████ | traced_small_model.zip Downloading: 31% |█████████████ | traced_small_model.zip Downloading: 32% |█████████████ | traced_small_model.zip Downloading: 33% |██████████████ | traced_small_model.zip Downloading: 34% |██████████████ | traced_small_model.zip Downloading: 35% |███████████████ | traced_small_model.zip Downloading: 36% |███████████████ | traced_small_model.zip Downloading: 37% |███████████████ | traced_small_model.zip Downloading: 38% |████████████████ | traced_small_model.zip Downloading: 39% |████████████████ | traced_small_model.zip Downloading: 40% |█████████████████ | traced_small_model.zip Downloading: 41% |█████████████████ | traced_small_model.zip Downloading: 42% |█████████████████ | traced_small_model.zip Downloading: 43% |██████████████████ | traced_small_model.zip Downloading: 44% |██████████████████ | traced_small_model.zip Downloading: 45% |███████████████████ | traced_small_model.zip Downloading: 46% |███████████████████ | traced_small_model.zip Downloading: 47% |███████████████████ | traced_small_model.zip Downloading: 48% |████████████████████ | traced_small_model.zip Downloading: 49% |████████████████████ | traced_small_model.zip Downloading: 50% |█████████████████████ | traced_small_model.zip Downloading: 51% |█████████████████████ | traced_small_model.zip Downloading: 52% |█████████████████████ | traced_small_model.zip Downloading: 53% |██████████████████████ | traced_small_model.zip Downloading: 54% |██████████████████████ | traced_small_model.zip Downloading: 55% |███████████████████████ | traced_small_model.zip Downloading: 56% |███████████████████████ | traced_small_model.zip Downloading: 57% |███████████████████████ | traced_small_model.zip Downloading: 58% |████████████████████████ | traced_small_model.zip Downloading: 59% |████████████████████████ | traced_small_model.zip Downloading: 60% |█████████████████████████ | traced_small_model.zip Downloading: 61% |█████████████████████████ | traced_small_model.zip Downloading: 62% |█████████████████████████ | traced_small_model.zip Downloading: 63% |██████████████████████████ | traced_small_model.zip Downloading: 64% |██████████████████████████ | traced_small_model.zip Downloading: 65% |███████████████████████████ | traced_small_model.zip Downloading: 66% |███████████████████████████ | traced_small_model.zip Downloading: 67% |███████████████████████████ | traced_small_model.zip Downloading: 68% |████████████████████████████ | traced_small_model.zip Downloading: 69% |████████████████████████████ | traced_small_model.zip Downloading: 70% |█████████████████████████████ | traced_small_model.zip Downloading: 71% |█████████████████████████████ | traced_small_model.zip Downloading: 72% |█████████████████████████████ | traced_small_model.zip Downloading: 73% |██████████████████████████████ | traced_small_model.zip Downloading: 74% |██████████████████████████████ | traced_small_model.zip Downloading: 75% |███████████████████████████████ | traced_small_model.zip Downloading: 76% |███████████████████████████████ | traced_small_model.zip Downloading: 77% |███████████████████████████████ | traced_small_model.zip Downloading: 78% |████████████████████████████████ | traced_small_model.zip Downloading: 79% |████████████████████████████████ | traced_small_model.zip Downloading: 80% |█████████████████████████████████ | traced_small_model.zip Downloading: 81% |█████████████████████████████████ | traced_small_model.zip Downloading: 82% |█████████████████████████████████ | traced_small_model.zip Downloading: 83% |██████████████████████████████████ | traced_small_model.zip Downloading: 84% |██████████████████████████████████ | traced_small_model.zip Downloading: 85% |███████████████████████████████████ | traced_small_model.zip Downloading: 86% |███████████████████████████████████ | traced_small_model.zip Downloading: 87% |███████████████████████████████████ | traced_small_model.zip Downloading: 88% |████████████████████████████████████ | traced_small_model.zip Downloading: 89% |████████████████████████████████████ | traced_small_model.zip Downloading: 90% |█████████████████████████████████████ | traced_small_model.zip Downloading: 91% |█████████████████████████████████████ | traced_small_model.zip Downloading: 92% |█████████████████████████████████████ | traced_small_model.zip Downloading: 93% |██████████████████████████████████████ | traced_small_model.zip Downloading: 94% |██████████████████████████████████████ | traced_small_model.zip Downloading: 95% |███████████████████████████████████████ | traced_small_model.zip Downloading: 96% |███████████████████████████████████████ | traced_small_model.zip Downloading: 97% |███████████████████████████████████████ | traced_small_model.zip Downloading: 98% |████████████████████████████████████████| traced_small_model.zip Downloading: 99% |████████████████████████████████████████| traced_small_model.zip Downloading: 100% |████████████████████████████████████████| traced_small_model.zip [2024-02-06T19:42:24,632][INFO ][o.o.m.m.MLModelManager ] [node_name_9200] Model registered successfully, model id: -ZDxf40Bamv3vXxyatgV, task id: -JDxf40Bamv3vXxyadjz [2024-02-06T19:42:26,258][INFO ][o.o.m.a.d.TransportDeployModelAction] [node_name_9200] Will deploy model on these nodes: OAYF21-xRbu9dc9StXHecg [2024-02-06T19:42:26,285][ERROR][o.o.m.m.MLModelManager ] [node_name_9200] No controller is deployed because the model -ZDxf40Bamv3vXxyatgV is expected not having an enabled model controller. Please use the create model controller api to create one if this is unexpected. [2024-02-06T19:42:26,435][INFO ][o.o.m.e.a.DLModel ] [node_name_9200] Model -ZDxf40Bamv3vXxyatgV is successfully deployed on 1 devices [2024-02-06T19:42:26,437][INFO ][o.o.m.a.MLModelAutoReDeployer] [node_name_9200] No models needs to be auto redeployed! [2024-02-06T19:42:26,437][INFO ][o.o.m.a.f.TransportForwardAction] [node_name_9200] deploy model done with state: DEPLOYED, model id: -ZDxf40Bamv3vXxyatgV [2024-02-06T19:42:26,437][INFO ][o.o.m.a.d.TransportDeployModelOnNodeAction] [node_name_9200] deploy model task done -pDxf40Bamv3vXxycdjT [2024-02-06T19:42:28,297][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[text_image_embedding_index/SgcQtk-YRU6AKu_cHMQbRw] [2024-02-06T19:42:28,299][INFO ][o.o.c.m.MetadataCreateIndexService] [node_name_9200] [text_image_embedding_index] creating index, cause [api], templates [], shards [1]/[0] [2024-02-06T19:42:28,314][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[text_image_embedding_index/SgcQtk-YRU6AKu_cHMQbRw] [2024-02-06T19:42:28,352][INFO ][o.o.c.r.a.AllocationService] [node_name_9200] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[text_image_embedding_index][0]]]). [2024-02-06T19:42:28,369][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[text_image_embedding_index/SgcQtk-YRU6AKu_cHMQbRw] [2024-02-06T19:42:28,373][INFO ][o.o.c.m.MetadataMappingService] [node_name_9200] [text_image_embedding_index/SgcQtk-YRU6AKu_cHMQbRw] update_mapping [_doc] [2024-02-06T19:42:31,428][INFO ][o.o.m.a.m.DeleteModelTransportAction] [node_name_9200] Model controller not deleted due to no model controller found for model: -ZDxf40Bamv3vXxyatgV [2024-02-06T19:42:31,435][INFO ][o.o.c.m.MetadataDeleteIndexService] [node_name_9200] [text_image_embedding_index/SgcQtk-YRU6AKu_cHMQbRw] deleting index [2024-02-06T19:42:31,482][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [false] to [true] [2024-02-06T19:42:31,482][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [false] to [true] [2024-02-06T19:42:31,483][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.allow_registering_model_via_url] from [true] to [false] [2024-02-06T19:42:31,508][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [true] to [false] [2024-02-06T19:42:31,509][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [true] to [false] [2024-02-06T19:42:31,542][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.allow_registering_model_via_url] from [false] to [true] [2024-02-06T19:42:31,596][INFO ][o.o.m.m.MLModelManager ] [node_name_9200] create new model meta doc _pDxf40Bamv3vXxyhtie for register model task _ZDxf40Bamv3vXxyhtiK [2024-02-06T19:42:31,855][INFO ][stdout ] [node_name_9200] Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 1% |█ | traced_small_model.zip Downloading: 2% |█ | traced_small_model.zip Downloading: 3% |██ | traced_small_model.zip Downloading: 4% |██ | traced_small_model.zip Downloading: 5% |███ | traced_small_model.zip Downloading: 6% |███ | traced_small_model.zip Downloading: 7% |███ | traced_small_model.zip Downloading: 8% |████ | traced_small_model.zip Downloading: 9% |████ | traced_small_model.zip Downloading: 10% |█████ | traced_small_model.zip Downloading: 11% |█████ | traced_small_model.zip Downloading: 12% |█████ | traced_small_model.zip Downloading: 13% |██████ | traced_small_model.zip Downloading: 14% |██████ | traced_small_model.zip Downloading: 15% |███████ | traced_small_model.zip Downloading: 16% |███████ | traced_small_model.zip Downloading: 17% |███████ | traced_small_model.zip Downloading: 18% |████████ | traced_small_model.zip Downloading: 19% |████████ | traced_small_model.zip Downloading: 20% |█████████ | traced_small_model.zip Downloading: 21% |█████████ | traced_small_model.zip Downloading: 22% |█████████ | traced_small_model.zip Downloading: 23% |██████████ | traced_small_model.zip Downloading: 24% |██████████ | traced_small_model.zip Downloading: 25% |███████████ | traced_small_model.zip Downloading: 26% |███████████ | traced_small_model.zip Downloading: 27% |███████████ | traced_small_model.zip Downloading: 28% |████████████ | traced_small_model.zip Downloading: 29% |████████████ | traced_small_model.zip Downloading: 30% |█████████████ | traced_small_model.zip Downloading: 31% |█████████████ | traced_small_model.zip Downloading: 32% |█████████████ | traced_small_model.zip Downloading: 33% |██████████████ | traced_small_model.zip Downloading: 34% |██████████████ | traced_small_model.zip Downloading: 35% |███████████████ | traced_small_model.zip Downloading: 36% |███████████████ | traced_small_model.zip Downloading: 37% |███████████████ | traced_small_model.zip Downloading: 38% |████████████████ | traced_small_model.zip Downloading: 39% |████████████████ | traced_small_model.zip Downloading: 40% |█████████████████ | traced_small_model.zip Downloading: 41% |█████████████████ | traced_small_model.zip Downloading: 42% |█████████████████ | traced_small_model.zip Downloading: 43% |██████████████████ | traced_small_model.zip Downloading: 44% |██████████████████ | traced_small_model.zip Downloading: 45% |███████████████████ | traced_small_model.zip Downloading: 46% |███████████████████ | traced_small_model.zip Downloading: 47% |███████████████████ | traced_small_model.zip Downloading: 48% |████████████████████ | traced_small_model.zip Downloading: 49% |████████████████████ | traced_small_model.zip Downloading: 50% |█████████████████████ | traced_small_model.zip Downloading: 51% |█████████████████████ | traced_small_model.zip Downloading: 52% |█████████████████████ | traced_small_model.zip Downloading: 53% |██████████████████████ | traced_small_model.zip Downloading: 54% |██████████████████████ | traced_small_model.zip Downloading: 55% |███████████████████████ | traced_small_model.zip Downloading: 56% |███████████████████████ | traced_small_model.zip Downloading: 57% |███████████████████████ | traced_small_model.zip Downloading: 58% |████████████████████████ | traced_small_model.zip Downloading: 59% |████████████████████████ | traced_small_model.zip Downloading: 60% |█████████████████████████ | traced_small_model.zip Downloading: 61% |█████████████████████████ | traced_small_model.zip Downloading: 62% |█████████████████████████ | traced_small_model.zip Downloading: 63% |██████████████████████████ | traced_small_model.zip Downloading: 64% |██████████████████████████ | traced_small_model.zip Downloading: 65% |███████████████████████████ | traced_small_model.zip Downloading: 66% |███████████████████████████ | traced_small_model.zip Downloading: 67% |███████████████████████████ | traced_small_model.zip Downloading: 68% |████████████████████████████ | traced_small_model.zip Downloading: 69% |████████████████████████████ | traced_small_model.zip Downloading: 70% |█████████████████████████████ | traced_small_model.zip Downloading: 71% |█████████████████████████████ | traced_small_model.zip Downloading: 72% |█████████████████████████████ | traced_small_model.zip Downloading: 73% |██████████████████████████████ | traced_small_model.zip Downloading: 74% |██████████████████████████████ | traced_small_model.zip Downloading: 75% |███████████████████████████████ | traced_small_model.zip Downloading: 76% |███████████████████████████████ | traced_small_model.zip Downloading: 77% |███████████████████████████████ | traced_small_model.zip Downloading: 78% |████████████████████████████████ | traced_small_model.zip Downloading: 79% |████████████████████████████████ | traced_small_model.zip Downloading: 80% |█████████████████████████████████ | traced_small_model.zip Downloading: 81% |█████████████████████████████████ | traced_small_model.zip Downloading: 82% |█████████████████████████████████ | traced_small_model.zip Downloading: 83% |██████████████████████████████████ | traced_small_model.zip Downloading: 84% |██████████████████████████████████ | traced_small_model.zip Downloading: 85% |███████████████████████████████████ | traced_small_model.zip Downloading: 86% |███████████████████████████████████ | traced_small_model.zip Downloading: 87% |███████████████████████████████████ | traced_small_model.zip Downloading: 88% |████████████████████████████████████ | traced_small_model.zip Downloading: 89% |████████████████████████████████████ | traced_small_model.zip Downloading: 90% |█████████████████████████████████████ | traced_small_model.zip Downloading: 91% |█████████████████████████████████████ | traced_small_model.zip Downloading: 92% |█████████████████████████████████████ | traced_small_model.zip Downloading: 93% |██████████████████████████████████████ | traced_small_model.zip Downloading: 94% |██████████████████████████████████████ | traced_small_model.zip Downloading: 95% |███████████████████████████████████████ | traced_small_model.zip Downloading: 96% |███████████████████████████████████████ | traced_small_model.zip Downloading: 97% |███████████████████████████████████████ | traced_small_model.zip Downloading: 98% |████████████████████████████████████████| traced_small_model.zip Downloading: 99% |████████████████████████████████████████| traced_small_model.zip Downloading: 100% |████████████████████████████████████████| traced_small_model.zip [2024-02-06T19:42:32,007][INFO ][o.o.m.m.MLModelManager ] [node_name_9200] Model registered successfully, model id: _pDxf40Bamv3vXxyhtie, task id: _ZDxf40Bamv3vXxyhtiK [2024-02-06T19:42:33,578][INFO ][o.o.m.a.d.TransportDeployModelAction] [node_name_9200] Will deploy model on these nodes: OAYF21-xRbu9dc9StXHecg [2024-02-06T19:42:33,621][ERROR][o.o.m.a.f.TransportForwardAction] [node_name_9200] deploy model failed on all nodes, model id: _pDxf40Bamv3vXxyhtie [2024-02-06T19:42:33,621][INFO ][o.o.m.a.f.TransportForwardAction] [node_name_9200] deploy model done with state: DEPLOY_FAILED, model id: _pDxf40Bamv3vXxyhtie [2024-02-06T19:42:33,622][INFO ][o.o.m.a.d.TransportDeployModelOnNodeAction] [node_name_9200] deploy model task done _5Dxf40Bamv3vXxyjthr [2024-02-06T19:42:35,631][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[text_image_embedding_index/3NPL1aZDRVSqm467KPZMkg] [2024-02-06T19:42:35,633][INFO ][o.o.c.m.MetadataCreateIndexService] [node_name_9200] [text_image_embedding_index] creating index, cause [api], templates [], shards [1]/[0] [2024-02-06T19:42:35,648][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[text_image_embedding_index/3NPL1aZDRVSqm467KPZMkg] [2024-02-06T19:42:35,687][INFO ][o.o.c.r.a.AllocationService] [node_name_9200] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[text_image_embedding_index][0]]]). [2024-02-06T19:42:38,733][INFO ][o.o.m.a.m.DeleteModelTransportAction] [node_name_9200] Model controller not deleted due to no model controller found for model: _pDxf40Bamv3vXxyhtie [2024-02-06T19:42:38,741][INFO ][o.o.c.m.MetadataDeleteIndexService] [node_name_9200] [text_image_embedding_index/3NPL1aZDRVSqm467KPZMkg] deleting index [2024-02-06T19:42:38,792][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [false] to [true] [2024-02-06T19:42:38,792][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [false] to [true] [2024-02-06T19:42:38,793][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.allow_registering_model_via_url] from [true] to [false] [2024-02-06T19:42:38,854][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [true] to [false] [2024-02-06T19:42:38,854][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [true] to [false] [2024-02-06T19:42:38,887][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.allow_registering_model_via_url] from [false] to [true] [2024-02-06T19:42:38,891][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-hybrid-multi-doc-nested-type-single-shard-index/AxgP2PysR6mkUZB2KPJt3A] [2024-02-06T19:42:38,893][INFO ][o.o.c.m.MetadataCreateIndexService] [node_name_9200] [test-hybrid-multi-doc-nested-type-single-shard-index] creating index, cause [api], templates [], shards [1]/[1] [2024-02-06T19:42:38,908][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-hybrid-multi-doc-nested-type-single-shard-index/AxgP2PysR6mkUZB2KPJt3A] [2024-02-06T19:42:38,967][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-hybrid-multi-doc-nested-type-single-shard-index/AxgP2PysR6mkUZB2KPJt3A] [2024-02-06T19:42:38,970][INFO ][o.o.c.m.MetadataMappingService] [node_name_9200] [test-hybrid-multi-doc-nested-type-single-shard-index/AxgP2PysR6mkUZB2KPJt3A] update_mapping [_doc] [2024-02-06T19:42:39,027][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-hybrid-multi-doc-nested-type-single-shard-index/AxgP2PysR6mkUZB2KPJt3A] [2024-02-06T19:42:39,030][INFO ][o.o.c.m.MetadataMappingService] [node_name_9200] [test-hybrid-multi-doc-nested-type-single-shard-index/AxgP2PysR6mkUZB2KPJt3A] update_mapping [_doc] [2024-02-06T19:42:39,090][WARN ][o.o.m.a.f.TransportForwardAction] [node_name_9200] Memory Circuit Breaker is open, please check your resources! [2024-02-06T19:42:39,091][WARN ][o.o.m.a.r.TransportRegisterModelAction] [node_name_9200] Memory Circuit Breaker is open, please check your resources! [2024-02-06T19:42:40,102][INFO ][o.o.c.m.MetadataDeleteIndexService] [node_name_9200] [test-hybrid-multi-doc-nested-type-single-shard-index/AxgP2PysR6mkUZB2KPJt3A] deleting index [2024-02-06T19:42:40,139][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-hybrid-vector-doc-field-index/V0nVHdKfR-OBMtepLY5H-Q] [2024-02-06T19:42:40,141][INFO ][o.o.c.m.MetadataCreateIndexService] [node_name_9200] [test-hybrid-vector-doc-field-index] creating index, cause [api], templates [], shards [3]/[1] [2024-02-06T19:42:40,156][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-hybrid-vector-doc-field-index/V0nVHdKfR-OBMtepLY5H-Q] [2024-02-06T19:42:40,238][INFO ][o.o.i.b.HierarchyCircuitBreakerService] [node_name_9200] attempting to trigger G1GC due to high heap usage [1022357632] [2024-02-06T19:42:40,248][INFO ][o.o.i.b.HierarchyCircuitBreakerService] [node_name_9200] GC did bring memory usage down, before [1022357632], after [561633848], allocations [47], duration [10] [2024-02-06T19:42:40,251][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-hybrid-vector-doc-field-index/V0nVHdKfR-OBMtepLY5H-Q] [2024-02-06T19:42:40,254][INFO ][o.o.c.m.MetadataMappingService] [node_name_9200] [test-hybrid-vector-doc-field-index/V0nVHdKfR-OBMtepLY5H-Q] update_mapping [_doc] [2024-02-06T19:42:40,356][INFO ][o.o.m.m.MLModelManager ] [node_name_9200] create new model meta doc BJDxf40Bamv3vXxyqNna for register model task A5Dxf40Bamv3vXxyqNnF [2024-02-06T19:42:40,545][INFO ][stdout ] [node_name_9200] Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 1% |█ | traced_small_model.zip Downloading: 2% |█ | traced_small_model.zip Downloading: 3% |██ | traced_small_model.zip Downloading: 4% |██ | traced_small_model.zip Downloading: 5% |███ | traced_small_model.zip Downloading: 6% |███ | traced_small_model.zip Downloading: 7% |███ | traced_small_model.zip Downloading: 8% |████ | traced_small_model.zip Downloading: 9% |████ | traced_small_model.zip Downloading: 10% |█████ | traced_small_model.zip Downloading: 11% |█████ | traced_small_model.zip Downloading: 12% |█████ | traced_small_model.zip Downloading: 13% |██████ | traced_small_model.zip Downloading: 14% |██████ | traced_small_model.zip Downloading: 15% |███████ | traced_small_model.zip Downloading: 16% |███████ | traced_small_model.zip Downloading: 17% |███████ | traced_small_model.zip Downloading: 18% |████████ | traced_small_model.zip Downloading: 19% |████████ | traced_small_model.zip Downloading: 20% |█████████ | traced_small_model.zip Downloading: 21% |█████████ | traced_small_model.zip Downloading: 22% |█████████ | traced_small_model.zip Downloading: 23% |██████████ | traced_small_model.zip Downloading: 24% |██████████ | traced_small_model.zip Downloading: 25% |███████████ | traced_small_model.zip Downloading: 26% |███████████ | traced_small_model.zip Downloading: 27% |███████████ | traced_small_model.zip Downloading: 28% |████████████ | traced_small_model.zip Downloading: 29% |████████████ | traced_small_model.zip Downloading: 30% |█████████████ | traced_small_model.zip Downloading: 31% |█████████████ | traced_small_model.zip Downloading: 32% |█████████████ | traced_small_model.zip Downloading: 33% |██████████████ | traced_small_model.zip Downloading: 34% |██████████████ | traced_small_model.zip Downloading: 35% |███████████████ | traced_small_model.zip Downloading: 36% |███████████████ | traced_small_model.zip Downloading: 37% |███████████████ | traced_small_model.zip Downloading: 38% |████████████████ | traced_small_model.zip Downloading: 39% |████████████████ | traced_small_model.zip Downloading: 40% |█████████████████ | traced_small_model.zip Downloading: 41% |█████████████████ | traced_small_model.zip Downloading: 42% |█████████████████ | traced_small_model.zip Downloading: 43% |██████████████████ | traced_small_model.zip Downloading: 44% |██████████████████ | traced_small_model.zip Downloading: 45% |███████████████████ | traced_small_model.zip Downloading: 46% |███████████████████ | traced_small_model.zip Downloading: 47% |███████████████████ | traced_small_model.zip Downloading: 48% |████████████████████ | traced_small_model.zip Downloading: 49% |████████████████████ | traced_small_model.zip Downloading: 50% |█████████████████████ | traced_small_model.zip Downloading: 51% |█████████████████████ | traced_small_model.zip Downloading: 52% |█████████████████████ | traced_small_model.zip Downloading: 53% |██████████████████████ | traced_small_model.zip Downloading: 54% |██████████████████████ | traced_small_model.zip Downloading: 55% |███████████████████████ | traced_small_model.zip Downloading: 56% |███████████████████████ | traced_small_model.zip Downloading: 57% |███████████████████████ | traced_small_model.zip Downloading: 58% |████████████████████████ | traced_small_model.zip Downloading: 59% |████████████████████████ | traced_small_model.zip Downloading: 60% |█████████████████████████ | traced_small_model.zip Downloading: 61% |█████████████████████████ | traced_small_model.zip Downloading: 62% |█████████████████████████ | traced_small_model.zip Downloading: 63% |██████████████████████████ | traced_small_model.zip Downloading: 64% |██████████████████████████ | traced_small_model.zip Downloading: 65% |███████████████████████████ | traced_small_model.zip Downloading: 66% |███████████████████████████ | traced_small_model.zip Downloading: 67% |███████████████████████████ | traced_small_model.zip Downloading: 68% |████████████████████████████ | traced_small_model.zip Downloading: 69% |████████████████████████████ | traced_small_model.zip Downloading: 70% |█████████████████████████████ | traced_small_model.zip Downloading: 71% |█████████████████████████████ | traced_small_model.zip Downloading: 72% |█████████████████████████████ | traced_small_model.zip Downloading: 73% |██████████████████████████████ | traced_small_model.zip Downloading: 74% |██████████████████████████████ | traced_small_model.zip Downloading: 75% |███████████████████████████████ | traced_small_model.zip Downloading: 76% |███████████████████████████████ | traced_small_model.zip Downloading: 77% |███████████████████████████████ | traced_small_model.zip Downloading: 78% |████████████████████████████████ | traced_small_model.zip Downloading: 79% |████████████████████████████████ | traced_small_model.zip Downloading: 80% |█████████████████████████████████ | traced_small_model.zip Downloading: 81% |█████████████████████████████████ | traced_small_model.zip Downloading: 82% |█████████████████████████████████ | traced_small_model.zip Downloading: 83% |██████████████████████████████████ | traced_small_model.zip Downloading: 84% |██████████████████████████████████ | traced_small_model.zip Downloading: 85% |███████████████████████████████████ | traced_small_model.zip Downloading: 86% |███████████████████████████████████ | traced_small_model.zip Downloading: 87% |███████████████████████████████████ | traced_small_model.zip Downloading: 88% |████████████████████████████████████ | traced_small_model.zip Downloading: 89% |████████████████████████████████████ | traced_small_model.zip Downloading: 90% |█████████████████████████████████████ | traced_small_model.zip Downloading: 91% |█████████████████████████████████████ | traced_small_model.zip Downloading: 92% |█████████████████████████████████████ | traced_small_model.zip Downloading: 93% |██████████████████████████████████████ | traced_small_model.zip Downloading: 94% |██████████████████████████████████████ | traced_small_model.zip Downloading: 95% |███████████████████████████████████████ | traced_small_model.zip Downloading: 96% |███████████████████████████████████████ | traced_small_model.zip Downloading: 97% |███████████████████████████████████████ | traced_small_model.zip Downloading: 98% |████████████████████████████████████████| traced_small_model.zip Downloading: 99% |████████████████████████████████████████| traced_small_model.zip Downloading: 100% |████████████████████████████████████████| traced_small_model.zip [2024-02-06T19:42:40,695][INFO ][o.o.m.m.MLModelManager ] [node_name_9200] Model registered successfully, model id: BJDxf40Bamv3vXxyqNna, task id: A5Dxf40Bamv3vXxyqNnF [2024-02-06T19:42:42,341][INFO ][o.o.m.a.d.TransportDeployModelAction] [node_name_9200] Will deploy model on these nodes: OAYF21-xRbu9dc9StXHecg [2024-02-06T19:42:42,369][ERROR][o.o.m.m.MLModelManager ] [node_name_9200] No controller is deployed because the model BJDxf40Bamv3vXxyqNna is expected not having an enabled model controller. Please use the create model controller api to create one if this is unexpected. [2024-02-06T19:42:42,520][INFO ][o.o.m.e.a.DLModel ] [node_name_9200] Model BJDxf40Bamv3vXxyqNna is successfully deployed on 1 devices [2024-02-06T19:42:42,521][INFO ][o.o.m.a.MLModelAutoReDeployer] [node_name_9200] No models needs to be auto redeployed! [2024-02-06T19:42:42,522][INFO ][o.o.m.a.f.TransportForwardAction] [node_name_9200] deploy model done with state: DEPLOYED, model id: BJDxf40Bamv3vXxyqNna [2024-02-06T19:42:42,522][INFO ][o.o.m.a.d.TransportDeployModelOnNodeAction] [node_name_9200] deploy model task done BZDxf40Bamv3vXxysNmm [2024-02-06T19:42:44,361][INFO ][o.o.n.p.f.NormalizationProcessorFactory] [node_name_9200] Creating search phase results processor of type [normalization-processor] with normalization [MinMaxScoreNormalizationTechnique(TECHNIQUE_NAME=min_max)] and combination [ArithmeticMeanScoreCombinationTechnique(TECHNIQUE_NAME=arithmetic_mean)] [2024-02-06T19:42:44,375][INFO ][o.o.n.p.f.NormalizationProcessorFactory] [node_name_9200] Creating search phase results processor of type [normalization-processor] with normalization [MinMaxScoreNormalizationTechnique(TECHNIQUE_NAME=min_max)] and combination [ArithmeticMeanScoreCombinationTechnique(TECHNIQUE_NAME=arithmetic_mean)] [2024-02-06T19:42:47,417][INFO ][o.o.m.a.m.DeleteModelTransportAction] [node_name_9200] Model controller not deleted due to no model controller found for model: BJDxf40Bamv3vXxyqNna [2024-02-06T19:42:47,425][INFO ][o.o.c.m.MetadataDeleteIndexService] [node_name_9200] [test-hybrid-vector-doc-field-index/V0nVHdKfR-OBMtepLY5H-Q] deleting index [2024-02-06T19:42:47,468][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-hybrid-vector-doc-field-index/otBjJ7fCR2qucsT0pfwZzQ] [2024-02-06T19:42:47,470][INFO ][o.o.c.m.MetadataCreateIndexService] [node_name_9200] [test-hybrid-vector-doc-field-index] creating index, cause [api], templates [], shards [3]/[1] [2024-02-06T19:42:47,487][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-hybrid-vector-doc-field-index/otBjJ7fCR2qucsT0pfwZzQ] [2024-02-06T19:42:47,573][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-hybrid-vector-doc-field-index/otBjJ7fCR2qucsT0pfwZzQ] [2024-02-06T19:42:47,576][INFO ][o.o.c.m.MetadataMappingService] [node_name_9200] [test-hybrid-vector-doc-field-index/otBjJ7fCR2qucsT0pfwZzQ] update_mapping [_doc] [2024-02-06T19:42:47,696][INFO ][o.o.m.m.MLModelManager ] [node_name_9200] create new model meta doc CJDxf40Bamv3vXxyxdmD for register model task B5Dxf40Bamv3vXxyxdlc [2024-02-06T19:42:47,883][INFO ][stdout ] [node_name_9200] Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 1% |█ | traced_small_model.zip Downloading: 2% |█ | traced_small_model.zip Downloading: 3% |██ | traced_small_model.zip Downloading: 4% |██ | traced_small_model.zip Downloading: 5% |███ | traced_small_model.zip Downloading: 6% |███ | traced_small_model.zip Downloading: 7% |███ | traced_small_model.zip Downloading: 8% |████ | traced_small_model.zip Downloading: 9% |████ | traced_small_model.zip Downloading: 10% |█████ | traced_small_model.zip Downloading: 11% |█████ | traced_small_model.zip Downloading: 12% |█████ | traced_small_model.zip Downloading: 13% |██████ | traced_small_model.zip Downloading: 14% |██████ | traced_small_model.zip Downloading: 15% |███████ | traced_small_model.zip Downloading: 16% |███████ | traced_small_model.zip Downloading: 17% |███████ | traced_small_model.zip Downloading: 18% |████████ | traced_small_model.zip Downloading: 19% |████████ | traced_small_model.zip Downloading: 20% |█████████ | traced_small_model.zip Downloading: 21% |█████████ | traced_small_model.zip Downloading: 22% |█████████ | traced_small_model.zip Downloading: 23% |██████████ | traced_small_model.zip Downloading: 24% |██████████ | traced_small_model.zip Downloading: 25% |███████████ | traced_small_model.zip Downloading: 26% |███████████ | traced_small_model.zip Downloading: 27% |███████████ | traced_small_model.zip Downloading: 28% |████████████ | traced_small_model.zip Downloading: 29% |████████████ | traced_small_model.zip Downloading: 30% |█████████████ | traced_small_model.zip Downloading: 31% |█████████████ | traced_small_model.zip Downloading: 32% |█████████████ | traced_small_model.zip Downloading: 33% |██████████████ | traced_small_model.zip Downloading: 34% |██████████████ | traced_small_model.zip Downloading: 35% |███████████████ | traced_small_model.zip Downloading: 36% |███████████████ | traced_small_model.zip Downloading: 37% |███████████████ | traced_small_model.zip Downloading: 38% |████████████████ | traced_small_model.zip Downloading: 39% |████████████████ | traced_small_model.zip Downloading: 40% |█████████████████ | traced_small_model.zip Downloading: 41% |█████████████████ | traced_small_model.zip Downloading: 42% |█████████████████ | traced_small_model.zip Downloading: 43% |██████████████████ | traced_small_model.zip Downloading: 44% |██████████████████ | traced_small_model.zip Downloading: 45% |███████████████████ | traced_small_model.zip Downloading: 46% |███████████████████ | traced_small_model.zip Downloading: 47% |███████████████████ | traced_small_model.zip Downloading: 48% |████████████████████ | traced_small_model.zip Downloading: 49% |████████████████████ | traced_small_model.zip Downloading: 50% |█████████████████████ | traced_small_model.zip Downloading: 51% |█████████████████████ | traced_small_model.zip Downloading: 52% |█████████████████████ | traced_small_model.zip Downloading: 53% |██████████████████████ | traced_small_model.zip Downloading: 54% |██████████████████████ | traced_small_model.zip Downloading: 55% |███████████████████████ | traced_small_model.zip Downloading: 56% |███████████████████████ | traced_small_model.zip Downloading: 57% |███████████████████████ | traced_small_model.zip Downloading: 58% |████████████████████████ | traced_small_model.zip Downloading: 59% |████████████████████████ | traced_small_model.zip Downloading: 60% |█████████████████████████ | traced_small_model.zip Downloading: 61% |█████████████████████████ | traced_small_model.zip Downloading: 62% |█████████████████████████ | traced_small_model.zip Downloading: 63% |██████████████████████████ | traced_small_model.zip Downloading: 64% |██████████████████████████ | traced_small_model.zip Downloading: 65% |███████████████████████████ | traced_small_model.zip Downloading: 66% |███████████████████████████ | traced_small_model.zip Downloading: 67% |███████████████████████████ | traced_small_model.zip Downloading: 68% |████████████████████████████ | traced_small_model.zip Downloading: 69% |████████████████████████████ | traced_small_model.zip Downloading: 70% |█████████████████████████████ | traced_small_model.zip Downloading: 71% |█████████████████████████████ | traced_small_model.zip Downloading: 72% |█████████████████████████████ | traced_small_model.zip Downloading: 73% |██████████████████████████████ | traced_small_model.zip Downloading: 74% |██████████████████████████████ | traced_small_model.zip Downloading: 75% |███████████████████████████████ | traced_small_model.zip Downloading: 76% |███████████████████████████████ | traced_small_model.zip Downloading: 77% |███████████████████████████████ | traced_small_model.zip Downloading: 78% |████████████████████████████████ | traced_small_model.zip Downloading: 79% |████████████████████████████████ | traced_small_model.zip Downloading: 80% |█████████████████████████████████ | traced_small_model.zip Downloading: 81% |█████████████████████████████████ | traced_small_model.zip Downloading: 82% |█████████████████████████████████ | traced_small_model.zip Downloading: 83% |██████████████████████████████████ | traced_small_model.zip Downloading: 84% |██████████████████████████████████ | traced_small_model.zip Downloading: 85% |███████████████████████████████████ | traced_small_model.zip Downloading: 86% |███████████████████████████████████ | traced_small_model.zip Downloading: 87% |███████████████████████████████████ | traced_small_model.zip Downloading: 88% |████████████████████████████████████ | traced_small_model.zip Downloading: 89% |████████████████████████████████████ | traced_small_model.zip Downloading: 90% |█████████████████████████████████████ | traced_small_model.zip Downloading: 91% |█████████████████████████████████████ | traced_small_model.zip Downloading: 92% |█████████████████████████████████████ | traced_small_model.zip Downloading: 93% |██████████████████████████████████████ | traced_small_model.zip Downloading: 94% |██████████████████████████████████████ | traced_small_model.zip Downloading: 95% |███████████████████████████████████████ | traced_small_model.zip Downloading: 96% |███████████████████████████████████████ | traced_small_model.zip Downloading: 97% |███████████████████████████████████████ | traced_small_model.zip Downloading: 98% |████████████████████████████████████████| traced_small_model.zip Downloading: 99% |████████████████████████████████████████| traced_small_model.zip Downloading: 100% |████████████████████████████████████████| traced_small_model.zip [2024-02-06T19:42:48,036][INFO ][o.o.m.m.MLModelManager ] [node_name_9200] Model registered successfully, model id: CJDxf40Bamv3vXxyxdmD, task id: B5Dxf40Bamv3vXxyxdlc [2024-02-06T19:42:49,675][INFO ][o.o.m.a.d.TransportDeployModelAction] [node_name_9200] Will deploy model on these nodes: OAYF21-xRbu9dc9StXHecg [2024-02-06T19:42:49,704][ERROR][o.o.m.m.MLModelManager ] [node_name_9200] No controller is deployed because the model CJDxf40Bamv3vXxyxdmD is expected not having an enabled model controller. Please use the create model controller api to create one if this is unexpected. [2024-02-06T19:42:49,857][INFO ][o.o.m.e.a.DLModel ] [node_name_9200] Model CJDxf40Bamv3vXxyxdmD is successfully deployed on 1 devices [2024-02-06T19:42:49,858][INFO ][o.o.m.a.MLModelAutoReDeployer] [node_name_9200] No models needs to be auto redeployed! [2024-02-06T19:42:49,858][INFO ][o.o.m.a.f.TransportForwardAction] [node_name_9200] deploy model done with state: DEPLOYED, model id: CJDxf40Bamv3vXxyxdmD [2024-02-06T19:42:49,859][INFO ][o.o.m.a.d.TransportDeployModelOnNodeAction] [node_name_9200] deploy model task done CZDxf40Bamv3vXxyzdlM [2024-02-06T19:42:51,697][INFO ][o.o.n.p.f.NormalizationProcessorFactory] [node_name_9200] Creating search phase results processor of type [normalization-processor] with normalization [MinMaxScoreNormalizationTechnique(TECHNIQUE_NAME=min_max)] and combination [ArithmeticMeanScoreCombinationTechnique(TECHNIQUE_NAME=arithmetic_mean)] [2024-02-06T19:42:51,713][INFO ][o.o.n.p.f.NormalizationProcessorFactory] [node_name_9200] Creating search phase results processor of type [normalization-processor] with normalization [MinMaxScoreNormalizationTechnique(TECHNIQUE_NAME=min_max)] and combination [ArithmeticMeanScoreCombinationTechnique(TECHNIQUE_NAME=arithmetic_mean)] [2024-02-06T19:42:54,749][INFO ][o.o.m.a.m.DeleteModelTransportAction] [node_name_9200] Model controller not deleted due to no model controller found for model: CJDxf40Bamv3vXxyxdmD [2024-02-06T19:42:54,756][INFO ][o.o.c.m.MetadataDeleteIndexService] [node_name_9200] [test-hybrid-vector-doc-field-index/otBjJ7fCR2qucsT0pfwZzQ] deleting index [2024-02-06T19:42:54,802][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-hybrid-multi-doc-index/UgabihgfTi-oo3zy0VBVTQ] [2024-02-06T19:42:54,805][INFO ][o.o.c.m.MetadataCreateIndexService] [node_name_9200] [test-hybrid-multi-doc-index] creating index, cause [api], templates [], shards [3]/[1] [2024-02-06T19:42:54,820][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-hybrid-multi-doc-index/UgabihgfTi-oo3zy0VBVTQ] [2024-02-06T19:42:54,906][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-hybrid-multi-doc-index/UgabihgfTi-oo3zy0VBVTQ] [2024-02-06T19:42:54,908][INFO ][o.o.c.m.MetadataMappingService] [node_name_9200] [test-hybrid-multi-doc-index/UgabihgfTi-oo3zy0VBVTQ] update_mapping [_doc] [2024-02-06T19:42:54,984][WARN ][o.o.m.a.f.TransportForwardAction] [node_name_9200] Memory Circuit Breaker is open, please check your resources! [2024-02-06T19:42:54,984][WARN ][o.o.m.a.r.TransportRegisterModelAction] [node_name_9200] Memory Circuit Breaker is open, please check your resources! [2024-02-06T19:42:55,988][INFO ][o.o.i.b.HierarchyCircuitBreakerService] [node_name_9200] attempting to trigger G1GC due to high heap usage [1021958712] [2024-02-06T19:42:55,999][INFO ][o.o.i.b.HierarchyCircuitBreakerService] [node_name_9200] GC did bring memory usage down, before [1021958712], after [719521400], allocations [47], duration [11] [2024-02-06T19:42:56,007][INFO ][o.o.c.m.MetadataDeleteIndexService] [node_name_9200] [test-hybrid-multi-doc-index/UgabihgfTi-oo3zy0VBVTQ] deleting index [2024-02-06T19:42:56,046][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-hybrid-multi-doc-single-shard-index/nEoywVyiQpCiZIFwmerI8Q] [2024-02-06T19:42:56,048][INFO ][o.o.c.m.MetadataCreateIndexService] [node_name_9200] [test-hybrid-multi-doc-single-shard-index] creating index, cause [api], templates [], shards [1]/[1] [2024-02-06T19:42:56,064][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-hybrid-multi-doc-single-shard-index/nEoywVyiQpCiZIFwmerI8Q] [2024-02-06T19:42:56,120][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-hybrid-multi-doc-single-shard-index/nEoywVyiQpCiZIFwmerI8Q] [2024-02-06T19:42:56,123][INFO ][o.o.c.m.MetadataMappingService] [node_name_9200] [test-hybrid-multi-doc-single-shard-index/nEoywVyiQpCiZIFwmerI8Q] update_mapping [_doc] [2024-02-06T19:42:56,235][INFO ][o.o.m.m.MLModelManager ] [node_name_9200] create new model meta doc DpDxf40Bamv3vXxy5tnh for register model task DZDxf40Bamv3vXxy5tnN [2024-02-06T19:42:56,406][INFO ][stdout ] [node_name_9200] Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 1% |█ | traced_small_model.zip Downloading: 2% |█ | traced_small_model.zip Downloading: 3% |██ | traced_small_model.zip Downloading: 4% |██ | traced_small_model.zip Downloading: 5% |███ | traced_small_model.zip Downloading: 6% |███ | traced_small_model.zip Downloading: 7% |███ | traced_small_model.zip Downloading: 8% |████ | traced_small_model.zip Downloading: 9% |████ | traced_small_model.zip Downloading: 10% |█████ | traced_small_model.zip Downloading: 11% |█████ | traced_small_model.zip Downloading: 12% |█████ | traced_small_model.zip Downloading: 13% |██████ | traced_small_model.zip Downloading: 14% |██████ | traced_small_model.zip Downloading: 15% |███████ | traced_small_model.zip Downloading: 16% |███████ | traced_small_model.zip Downloading: 17% |███████ | traced_small_model.zip Downloading: 18% |████████ | traced_small_model.zip Downloading: 19% |████████ | traced_small_model.zip Downloading: 20% |█████████ | traced_small_model.zip Downloading: 21% |█████████ | traced_small_model.zip Downloading: 22% |█████████ | traced_small_model.zip Downloading: 23% |██████████ | traced_small_model.zip Downloading: 24% |██████████ | traced_small_model.zip Downloading: 25% |███████████ | traced_small_model.zip Downloading: 26% |███████████ | traced_small_model.zip Downloading: 27% |███████████ | traced_small_model.zip Downloading: 28% |████████████ | traced_small_model.zip Downloading: 29% |████████████ | traced_small_model.zip Downloading: 30% |█████████████ | traced_small_model.zip Downloading: 31% |█████████████ | traced_small_model.zip Downloading: 32% |█████████████ | traced_small_model.zip Downloading: 33% |██████████████ | traced_small_model.zip Downloading: 34% |██████████████ | traced_small_model.zip Downloading: 35% |███████████████ | traced_small_model.zip Downloading: 36% |███████████████ | traced_small_model.zip Downloading: 37% |███████████████ | traced_small_model.zip Downloading: 38% |████████████████ | traced_small_model.zip Downloading: 39% |████████████████ | traced_small_model.zip Downloading: 40% |█████████████████ | traced_small_model.zip Downloading: 41% |█████████████████ | traced_small_model.zip Downloading: 42% |█████████████████ | traced_small_model.zip Downloading: 43% |██████████████████ | traced_small_model.zip Downloading: 44% |██████████████████ | traced_small_model.zip Downloading: 45% |███████████████████ | traced_small_model.zip Downloading: 46% |███████████████████ | traced_small_model.zip Downloading: 47% |███████████████████ | traced_small_model.zip Downloading: 48% |████████████████████ | traced_small_model.zip Downloading: 49% |████████████████████ | traced_small_model.zip Downloading: 50% |█████████████████████ | traced_small_model.zip Downloading: 51% |█████████████████████ | traced_small_model.zip Downloading: 52% |█████████████████████ | traced_small_model.zip Downloading: 53% |██████████████████████ | traced_small_model.zip Downloading: 54% |██████████████████████ | traced_small_model.zip Downloading: 55% |███████████████████████ | traced_small_model.zip Downloading: 56% |███████████████████████ | traced_small_model.zip Downloading: 57% |███████████████████████ | traced_small_model.zip Downloading: 58% |████████████████████████ | traced_small_model.zip Downloading: 59% |████████████████████████ | traced_small_model.zip Downloading: 60% |█████████████████████████ | traced_small_model.zip Downloading: 61% |█████████████████████████ | traced_small_model.zip Downloading: 62% |█████████████████████████ | traced_small_model.zip Downloading: 63% |██████████████████████████ | traced_small_model.zip Downloading: 64% |██████████████████████████ | traced_small_model.zip Downloading: 65% |███████████████████████████ | traced_small_model.zip Downloading: 66% |███████████████████████████ | traced_small_model.zip Downloading: 67% |███████████████████████████ | traced_small_model.zip Downloading: 68% |████████████████████████████ | traced_small_model.zip Downloading: 69% |████████████████████████████ | traced_small_model.zip Downloading: 70% |█████████████████████████████ | traced_small_model.zip Downloading: 71% |█████████████████████████████ | traced_small_model.zip Downloading: 72% |█████████████████████████████ | traced_small_model.zip Downloading: 73% |██████████████████████████████ | traced_small_model.zip Downloading: 74% |██████████████████████████████ | traced_small_model.zip Downloading: 75% |███████████████████████████████ | traced_small_model.zip Downloading: 76% |███████████████████████████████ | traced_small_model.zip Downloading: 77% |███████████████████████████████ | traced_small_model.zip Downloading: 78% |████████████████████████████████ | traced_small_model.zip Downloading: 79% |████████████████████████████████ | traced_small_model.zip Downloading: 80% |█████████████████████████████████ | traced_small_model.zip Downloading: 81% |█████████████████████████████████ | traced_small_model.zip Downloading: 82% |█████████████████████████████████ | traced_small_model.zip Downloading: 83% |██████████████████████████████████ | traced_small_model.zip Downloading: 84% |██████████████████████████████████ | traced_small_model.zip Downloading: 85% |███████████████████████████████████ | traced_small_model.zip Downloading: 86% |███████████████████████████████████ | traced_small_model.zip Downloading: 87% |███████████████████████████████████ | traced_small_model.zip Downloading: 88% |████████████████████████████████████ | traced_small_model.zip Downloading: 89% |████████████████████████████████████ | traced_small_model.zip Downloading: 90% |█████████████████████████████████████ | traced_small_model.zip Downloading: 91% |█████████████████████████████████████ | traced_small_model.zip Downloading: 92% |█████████████████████████████████████ | traced_small_model.zip Downloading: 93% |██████████████████████████████████████ | traced_small_model.zip Downloading: 94% |██████████████████████████████████████ | traced_small_model.zip Downloading: 95% |███████████████████████████████████████ | traced_small_model.zip Downloading: 96% |███████████████████████████████████████ | traced_small_model.zip Downloading: 97% |███████████████████████████████████████ | traced_small_model.zip Downloading: 98% |████████████████████████████████████████| traced_small_model.zip Downloading: 99% |████████████████████████████████████████| traced_small_model.zip Downloading: 100% |████████████████████████████████████████| traced_small_model.zip [2024-02-06T19:42:56,556][INFO ][o.o.m.m.MLModelManager ] [node_name_9200] Model registered successfully, model id: DpDxf40Bamv3vXxy5tnh, task id: DZDxf40Bamv3vXxy5tnN [2024-02-06T19:42:58,220][INFO ][o.o.m.a.d.TransportDeployModelAction] [node_name_9200] Will deploy model on these nodes: OAYF21-xRbu9dc9StXHecg [2024-02-06T19:42:58,264][ERROR][o.o.m.m.MLModelManager ] [node_name_9200] No controller is deployed because the model DpDxf40Bamv3vXxy5tnh is expected not having an enabled model controller. Please use the create model controller api to create one if this is unexpected. [2024-02-06T19:42:58,416][INFO ][o.o.m.e.a.DLModel ] [node_name_9200] Model DpDxf40Bamv3vXxy5tnh is successfully deployed on 1 devices [2024-02-06T19:42:58,417][INFO ][o.o.m.a.MLModelAutoReDeployer] [node_name_9200] No models needs to be auto redeployed! [2024-02-06T19:42:58,417][INFO ][o.o.m.a.f.TransportForwardAction] [node_name_9200] deploy model done with state: DEPLOYED, model id: DpDxf40Bamv3vXxy5tnh [2024-02-06T19:42:58,417][INFO ][o.o.m.a.d.TransportDeployModelOnNodeAction] [node_name_9200] deploy model task done D5Dxf40Bamv3vXxy7tmt [2024-02-06T19:43:00,258][INFO ][o.o.n.p.f.NormalizationProcessorFactory] [node_name_9200] Creating search phase results processor of type [normalization-processor] with normalization [MinMaxScoreNormalizationTechnique(TECHNIQUE_NAME=min_max)] and combination [ArithmeticMeanScoreCombinationTechnique(TECHNIQUE_NAME=arithmetic_mean)] [2024-02-06T19:43:00,278][INFO ][o.o.n.p.f.NormalizationProcessorFactory] [node_name_9200] Creating search phase results processor of type [normalization-processor] with normalization [MinMaxScoreNormalizationTechnique(TECHNIQUE_NAME=min_max)] and combination [ArithmeticMeanScoreCombinationTechnique(TECHNIQUE_NAME=arithmetic_mean)] [2024-02-06T19:43:00,305][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-hybrid-multi-doc-nested-type-single-shard-index/hCwmtaPaRtWkrt8dYbKkgw] [2024-02-06T19:43:00,308][INFO ][o.o.c.m.MetadataCreateIndexService] [node_name_9200] [test-hybrid-multi-doc-nested-type-single-shard-index] creating index, cause [api], templates [], shards [1]/[1] [2024-02-06T19:43:00,325][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-hybrid-multi-doc-nested-type-single-shard-index/hCwmtaPaRtWkrt8dYbKkgw] [2024-02-06T19:43:00,384][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-hybrid-multi-doc-nested-type-single-shard-index/hCwmtaPaRtWkrt8dYbKkgw] [2024-02-06T19:43:00,386][INFO ][o.o.c.m.MetadataMappingService] [node_name_9200] [test-hybrid-multi-doc-nested-type-single-shard-index/hCwmtaPaRtWkrt8dYbKkgw] update_mapping [_doc] [2024-02-06T19:43:00,443][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-hybrid-multi-doc-nested-type-single-shard-index/hCwmtaPaRtWkrt8dYbKkgw] [2024-02-06T19:43:00,446][INFO ][o.o.c.m.MetadataMappingService] [node_name_9200] [test-hybrid-multi-doc-nested-type-single-shard-index/hCwmtaPaRtWkrt8dYbKkgw] update_mapping [_doc] [2024-02-06T19:43:03,507][INFO ][o.o.m.a.m.DeleteModelTransportAction] [node_name_9200] Model controller not deleted due to no model controller found for model: DpDxf40Bamv3vXxy5tnh [2024-02-06T19:43:03,514][INFO ][o.o.c.m.MetadataDeleteIndexService] [node_name_9200] [test-hybrid-multi-doc-single-shard-index/nEoywVyiQpCiZIFwmerI8Q] deleting index [2024-02-06T19:43:03,540][INFO ][o.o.c.m.MetadataDeleteIndexService] [node_name_9200] [test-hybrid-multi-doc-nested-type-single-shard-index/hCwmtaPaRtWkrt8dYbKkgw] deleting index [2024-02-06T19:43:03,575][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-hybrid-multi-doc-nested-type-single-shard-index/R8CaqRuaTVyOqAHWNjSSjA] [2024-02-06T19:43:03,577][INFO ][o.o.c.m.MetadataCreateIndexService] [node_name_9200] [test-hybrid-multi-doc-nested-type-single-shard-index] creating index, cause [api], templates [], shards [1]/[1] [2024-02-06T19:43:03,594][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-hybrid-multi-doc-nested-type-single-shard-index/R8CaqRuaTVyOqAHWNjSSjA] [2024-02-06T19:43:03,652][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-hybrid-multi-doc-nested-type-single-shard-index/R8CaqRuaTVyOqAHWNjSSjA] [2024-02-06T19:43:03,655][INFO ][o.o.c.m.MetadataMappingService] [node_name_9200] [test-hybrid-multi-doc-nested-type-single-shard-index/R8CaqRuaTVyOqAHWNjSSjA] update_mapping [_doc] [2024-02-06T19:43:03,715][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-hybrid-multi-doc-nested-type-single-shard-index/R8CaqRuaTVyOqAHWNjSSjA] [2024-02-06T19:43:03,718][INFO ][o.o.c.m.MetadataMappingService] [node_name_9200] [test-hybrid-multi-doc-nested-type-single-shard-index/R8CaqRuaTVyOqAHWNjSSjA] update_mapping [_doc] [2024-02-06T19:43:03,793][INFO ][o.o.m.m.MLModelManager ] [node_name_9200] create new model meta doc EpDyf40Bamv3vXxyBNln for register model task EZDyf40Bamv3vXxyBNlS [2024-02-06T19:43:03,986][INFO ][stdout ] [node_name_9200] Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 1% |█ | traced_small_model.zip Downloading: 2% |█ | traced_small_model.zip Downloading: 3% |██ | traced_small_model.zip Downloading: 4% |██ | traced_small_model.zip Downloading: 5% |███ | traced_small_model.zip Downloading: 6% |███ | traced_small_model.zip Downloading: 7% |███ | traced_small_model.zip Downloading: 8% |████ | traced_small_model.zip Downloading: 9% |████ | traced_small_model.zip Downloading: 10% |█████ | traced_small_model.zip Downloading: 11% |█████ | traced_small_model.zip Downloading: 12% |█████ | traced_small_model.zip Downloading: 13% |██████ | traced_small_model.zip Downloading: 14% |██████ | traced_small_model.zip Downloading: 15% |███████ | traced_small_model.zip Downloading: 16% |███████ | traced_small_model.zip Downloading: 17% |███████ | traced_small_model.zip Downloading: 18% |████████ | traced_small_model.zip Downloading: 19% |████████ | traced_small_model.zip Downloading: 20% |█████████ | traced_small_model.zip Downloading: 21% |█████████ | traced_small_model.zip Downloading: 22% |█████████ | traced_small_model.zip Downloading: 23% |██████████ | traced_small_model.zip Downloading: 24% |██████████ | traced_small_model.zip Downloading: 25% |███████████ | traced_small_model.zip Downloading: 26% |███████████ | traced_small_model.zip Downloading: 27% |███████████ | traced_small_model.zip Downloading: 28% |████████████ | traced_small_model.zip Downloading: 29% |████████████ | traced_small_model.zip Downloading: 30% |█████████████ | traced_small_model.zip Downloading: 31% |█████████████ | traced_small_model.zip Downloading: 32% |█████████████ | traced_small_model.zip Downloading: 33% |██████████████ | traced_small_model.zip Downloading: 34% |██████████████ | traced_small_model.zip Downloading: 35% |███████████████ | traced_small_model.zip Downloading: 36% |███████████████ | traced_small_model.zip Downloading: 37% |███████████████ | traced_small_model.zip Downloading: 38% |████████████████ | traced_small_model.zip Downloading: 39% |████████████████ | traced_small_model.zip Downloading: 40% |█████████████████ | traced_small_model.zip Downloading: 41% |█████████████████ | traced_small_model.zip Downloading: 42% |█████████████████ | traced_small_model.zip Downloading: 43% |██████████████████ | traced_small_model.zip Downloading: 44% |██████████████████ | traced_small_model.zip Downloading: 45% |███████████████████ | traced_small_model.zip Downloading: 46% |███████████████████ | traced_small_model.zip Downloading: 47% |███████████████████ | traced_small_model.zip Downloading: 48% |████████████████████ | traced_small_model.zip Downloading: 49% |████████████████████ | traced_small_model.zip Downloading: 50% |█████████████████████ | traced_small_model.zip Downloading: 51% |█████████████████████ | traced_small_model.zip Downloading: 52% |█████████████████████ | traced_small_model.zip Downloading: 53% |██████████████████████ | traced_small_model.zip Downloading: 54% |██████████████████████ | traced_small_model.zip Downloading: 55% |███████████████████████ | traced_small_model.zip Downloading: 56% |███████████████████████ | traced_small_model.zip Downloading: 57% |███████████████████████ | traced_small_model.zip Downloading: 58% |████████████████████████ | traced_small_model.zip Downloading: 59% |████████████████████████ | traced_small_model.zip Downloading: 60% |█████████████████████████ | traced_small_model.zip Downloading: 61% |█████████████████████████ | traced_small_model.zip Downloading: 62% |█████████████████████████ | traced_small_model.zip Downloading: 63% |██████████████████████████ | traced_small_model.zip Downloading: 64% |██████████████████████████ | traced_small_model.zip Downloading: 65% |███████████████████████████ | traced_small_model.zip Downloading: 66% |███████████████████████████ | traced_small_model.zip Downloading: 67% |███████████████████████████ | traced_small_model.zip Downloading: 68% |████████████████████████████ | traced_small_model.zip Downloading: 69% |████████████████████████████ | traced_small_model.zip Downloading: 70% |█████████████████████████████ | traced_small_model.zip Downloading: 71% |█████████████████████████████ | traced_small_model.zip Downloading: 72% |█████████████████████████████ | traced_small_model.zip Downloading: 73% |██████████████████████████████ | traced_small_model.zip Downloading: 74% |██████████████████████████████ | traced_small_model.zip Downloading: 75% |███████████████████████████████ | traced_small_model.zip Downloading: 76% |███████████████████████████████ | traced_small_model.zip Downloading: 77% |███████████████████████████████ | traced_small_model.zip Downloading: 78% |████████████████████████████████ | traced_small_model.zip Downloading: 79% |████████████████████████████████ | traced_small_model.zip Downloading: 80% |█████████████████████████████████ | traced_small_model.zip Downloading: 81% |█████████████████████████████████ | traced_small_model.zip Downloading: 82% |█████████████████████████████████ | traced_small_model.zip Downloading: 83% |██████████████████████████████████ | traced_small_model.zip Downloading: 84% |██████████████████████████████████ | traced_small_model.zip Downloading: 85% |███████████████████████████████████ | traced_small_model.zip Downloading: 86% |███████████████████████████████████ | traced_small_model.zip Downloading: 87% |███████████████████████████████████ | traced_small_model.zip Downloading: 88% |████████████████████████████████████ | traced_small_model.zip Downloading: 89% |████████████████████████████████████ | traced_small_model.zip Downloading: 90% |█████████████████████████████████████ | traced_small_model.zip Downloading: 91% |█████████████████████████████████████ | traced_small_model.zip Downloading: 92% |█████████████████████████████████████ | traced_small_model.zip Downloading: 93% |██████████████████████████████████████ | traced_small_model.zip Downloading: 94% |██████████████████████████████████████ | traced_small_model.zip Downloading: 95% |███████████████████████████████████████ | traced_small_model.zip Downloading: 96% |███████████████████████████████████████ | traced_small_model.zip Downloading: 97% |███████████████████████████████████████ | traced_small_model.zip Downloading: 98% |████████████████████████████████████████| traced_small_model.zip Downloading: 99% |████████████████████████████████████████| traced_small_model.zip Downloading: 100% |████████████████████████████████████████| traced_small_model.zip [2024-02-06T19:43:04,138][INFO ][o.o.m.m.MLModelManager ] [node_name_9200] Model registered successfully, model id: EpDyf40Bamv3vXxyBNln, task id: EZDyf40Bamv3vXxyBNlS [2024-02-06T19:43:05,778][INFO ][o.o.m.a.d.TransportDeployModelAction] [node_name_9200] Will deploy model on these nodes: OAYF21-xRbu9dc9StXHecg [2024-02-06T19:43:05,805][ERROR][o.o.m.m.MLModelManager ] [node_name_9200] No controller is deployed because the model EpDyf40Bamv3vXxyBNln is expected not having an enabled model controller. Please use the create model controller api to create one if this is unexpected. [2024-02-06T19:43:05,977][INFO ][o.o.m.e.a.DLModel ] [node_name_9200] Model EpDyf40Bamv3vXxyBNln is successfully deployed on 1 devices [2024-02-06T19:43:05,978][INFO ][o.o.m.a.MLModelAutoReDeployer] [node_name_9200] No models needs to be auto redeployed! [2024-02-06T19:43:05,978][INFO ][o.o.m.a.f.TransportForwardAction] [node_name_9200] deploy model done with state: DEPLOYED, model id: EpDyf40Bamv3vXxyBNln [2024-02-06T19:43:05,978][INFO ][o.o.m.a.d.TransportDeployModelOnNodeAction] [node_name_9200] deploy model task done E5Dyf40Bamv3vXxyDNky [2024-02-06T19:43:07,796][INFO ][o.o.n.p.f.NormalizationProcessorFactory] [node_name_9200] Creating search phase results processor of type [normalization-processor] with normalization [MinMaxScoreNormalizationTechnique(TECHNIQUE_NAME=min_max)] and combination [ArithmeticMeanScoreCombinationTechnique(TECHNIQUE_NAME=arithmetic_mean)] [2024-02-06T19:43:07,811][INFO ][o.o.n.p.f.NormalizationProcessorFactory] [node_name_9200] Creating search phase results processor of type [normalization-processor] with normalization [MinMaxScoreNormalizationTechnique(TECHNIQUE_NAME=min_max)] and combination [ArithmeticMeanScoreCombinationTechnique(TECHNIQUE_NAME=arithmetic_mean)] [2024-02-06T19:43:10,847][INFO ][o.o.m.a.m.DeleteModelTransportAction] [node_name_9200] Model controller not deleted due to no model controller found for model: EpDyf40Bamv3vXxyBNln [2024-02-06T19:43:10,854][INFO ][o.o.c.m.MetadataDeleteIndexService] [node_name_9200] [test-hybrid-multi-doc-nested-type-single-shard-index/R8CaqRuaTVyOqAHWNjSSjA] deleting index [2024-02-06T19:43:10,934][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-neural-multi-vector-field-index/39W_thTgR2y4B_7H-bniEA] [2024-02-06T19:43:10,936][INFO ][o.o.c.m.MetadataCreateIndexService] [node_name_9200] [test-neural-multi-vector-field-index] creating index, cause [api], templates [], shards [3]/[1] [2024-02-06T19:43:10,954][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-neural-multi-vector-field-index/39W_thTgR2y4B_7H-bniEA] [2024-02-06T19:43:11,094][INFO ][o.o.m.m.MLModelManager ] [node_name_9200] create new model meta doc FpDyf40Bamv3vXxyINnp for register model task FZDyf40Bamv3vXxyINnW [2024-02-06T19:43:11,275][INFO ][stdout ] [node_name_9200] Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 1% |█ | traced_small_model.zip Downloading: 2% |█ | traced_small_model.zip Downloading: 3% |██ | traced_small_model.zip Downloading: 4% |██ | traced_small_model.zip Downloading: 5% |███ | traced_small_model.zip Downloading: 6% |███ | traced_small_model.zip Downloading: 7% |███ | traced_small_model.zip Downloading: 8% |████ | traced_small_model.zip Downloading: 9% |████ | traced_small_model.zip Downloading: 10% |█████ | traced_small_model.zip Downloading: 11% |█████ | traced_small_model.zip Downloading: 12% |█████ | traced_small_model.zip Downloading: 13% |██████ | traced_small_model.zip Downloading: 14% |██████ | traced_small_model.zip Downloading: 15% |███████ | traced_small_model.zip Downloading: 16% |███████ | traced_small_model.zip Downloading: 17% |███████ | traced_small_model.zip Downloading: 18% |████████ | traced_small_model.zip Downloading: 19% |████████ | traced_small_model.zip Downloading: 20% |█████████ | traced_small_model.zip Downloading: 21% |█████████ | traced_small_model.zip Downloading: 22% |█████████ | traced_small_model.zip Downloading: 23% |██████████ | traced_small_model.zip Downloading: 24% |██████████ | traced_small_model.zip Downloading: 25% |███████████ | traced_small_model.zip Downloading: 26% |███████████ | traced_small_model.zip Downloading: 27% |███████████ | traced_small_model.zip Downloading: 28% |████████████ | traced_small_model.zip Downloading: 29% |████████████ | traced_small_model.zip Downloading: 30% |█████████████ | traced_small_model.zip Downloading: 31% |█████████████ | traced_small_model.zip Downloading: 32% |█████████████ | traced_small_model.zip Downloading: 33% |██████████████ | traced_small_model.zip Downloading: 34% |██████████████ | traced_small_model.zip Downloading: 35% |███████████████ | traced_small_model.zip Downloading: 36% |███████████████ | traced_small_model.zip Downloading: 37% |███████████████ | traced_small_model.zip Downloading: 38% |████████████████ | traced_small_model.zip Downloading: 39% |████████████████ | traced_small_model.zip Downloading: 40% |█████████████████ | traced_small_model.zip Downloading: 41% |█████████████████ | traced_small_model.zip Downloading: 42% |█████████████████ | traced_small_model.zip Downloading: 43% |██████████████████ | traced_small_model.zip Downloading: 44% |██████████████████ | traced_small_model.zip Downloading: 45% |███████████████████ | traced_small_model.zip Downloading: 46% |███████████████████ | traced_small_model.zip Downloading: 47% |███████████████████ | traced_small_model.zip Downloading: 48% |████████████████████ | traced_small_model.zip Downloading: 49% |████████████████████ | traced_small_model.zip Downloading: 50% |█████████████████████ | traced_small_model.zip Downloading: 51% |█████████████████████ | traced_small_model.zip Downloading: 52% |█████████████████████ | traced_small_model.zip Downloading: 53% |██████████████████████ | traced_small_model.zip Downloading: 54% |██████████████████████ | traced_small_model.zip Downloading: 55% |███████████████████████ | traced_small_model.zip Downloading: 56% |███████████████████████ | traced_small_model.zip Downloading: 57% |███████████████████████ | traced_small_model.zip Downloading: 58% |████████████████████████ | traced_small_model.zip Downloading: 59% |████████████████████████ | traced_small_model.zip Downloading: 60% |█████████████████████████ | traced_small_model.zip Downloading: 61% |█████████████████████████ | traced_small_model.zip Downloading: 62% |█████████████████████████ | traced_small_model.zip Downloading: 63% |██████████████████████████ | traced_small_model.zip Downloading: 64% |██████████████████████████ | traced_small_model.zip Downloading: 65% |███████████████████████████ | traced_small_model.zip Downloading: 66% |███████████████████████████ | traced_small_model.zip Downloading: 67% |███████████████████████████ | traced_small_model.zip Downloading: 68% |████████████████████████████ | traced_small_model.zip Downloading: 69% |████████████████████████████ | traced_small_model.zip Downloading: 70% |█████████████████████████████ | traced_small_model.zip Downloading: 71% |█████████████████████████████ | traced_small_model.zip Downloading: 72% |█████████████████████████████ | traced_small_model.zip Downloading: 73% |██████████████████████████████ | traced_small_model.zip Downloading: 74% |██████████████████████████████ | traced_small_model.zip Downloading: 75% |███████████████████████████████ | traced_small_model.zip Downloading: 76% |███████████████████████████████ | traced_small_model.zip Downloading: 77% |███████████████████████████████ | traced_small_model.zip Downloading: 78% |████████████████████████████████ | traced_small_model.zip Downloading: 79% |████████████████████████████████ | traced_small_model.zip Downloading: 80% |█████████████████████████████████ | traced_small_model.zip Downloading: 81% |█████████████████████████████████ | traced_small_model.zip Downloading: 82% |█████████████████████████████████ | traced_small_model.zip Downloading: 83% |██████████████████████████████████ | traced_small_model.zip Downloading: 84% |██████████████████████████████████ | traced_small_model.zip Downloading: 85% |███████████████████████████████████ | traced_small_model.zip Downloading: 86% |███████████████████████████████████ | traced_small_model.zip Downloading: 87% |███████████████████████████████████ | traced_small_model.zip Downloading: 88% |████████████████████████████████████ | traced_small_model.zip Downloading: 89% |████████████████████████████████████ | traced_small_model.zip Downloading: 90% |█████████████████████████████████████ | traced_small_model.zip Downloading: 91% |█████████████████████████████████████ | traced_small_model.zip Downloading: 92% |█████████████████████████████████████ | traced_small_model.zip Downloading: 93% |██████████████████████████████████████ | traced_small_model.zip Downloading: 94% |██████████████████████████████████████ | traced_small_model.zip Downloading: 95% |███████████████████████████████████████ | traced_small_model.zip Downloading: 96% |███████████████████████████████████████ | traced_small_model.zip Downloading: 97% |███████████████████████████████████████ | traced_small_model.zip Downloading: 98% |████████████████████████████████████████| traced_small_model.zip Downloading: 99% |████████████████████████████████████████| traced_small_model.zip Downloading: 100% |████████████████████████████████████████| traced_small_model.zip [2024-02-06T19:43:11,442][INFO ][o.o.m.m.MLModelManager ] [node_name_9200] Model registered successfully, model id: FpDyf40Bamv3vXxyINnp, task id: FZDyf40Bamv3vXxyINnW [2024-02-06T19:43:13,076][INFO ][o.o.m.a.d.TransportDeployModelAction] [node_name_9200] Will deploy model on these nodes: OAYF21-xRbu9dc9StXHecg [2024-02-06T19:43:13,101][ERROR][o.o.m.a.f.TransportForwardAction] [node_name_9200] deploy model failed on all nodes, model id: FpDyf40Bamv3vXxyINnp [2024-02-06T19:43:13,102][INFO ][o.o.m.a.f.TransportForwardAction] [node_name_9200] deploy model done with state: DEPLOY_FAILED, model id: FpDyf40Bamv3vXxyINnp [2024-02-06T19:43:13,102][INFO ][o.o.m.a.d.TransportDeployModelOnNodeAction] [node_name_9200] deploy model task done F5Dyf40Bamv3vXxyKNm1 [2024-02-06T19:43:18,105][INFO ][o.o.m.a.m.DeleteModelTransportAction] [node_name_9200] Model controller not deleted due to no model controller found for model: FpDyf40Bamv3vXxyINnp [2024-02-06T19:43:18,112][INFO ][o.o.c.m.MetadataDeleteIndexService] [node_name_9200] [test-neural-multi-vector-field-index/39W_thTgR2y4B_7H-bniEA] deleting index [2024-02-06T19:43:18,165][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [false] to [true] [2024-02-06T19:43:18,165][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [false] to [true] [2024-02-06T19:43:18,166][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.allow_registering_model_via_url] from [true] to [false] [2024-02-06T19:43:18,197][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [true] to [false] [2024-02-06T19:43:18,197][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [true] to [false] [2024-02-06T19:43:18,229][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.allow_registering_model_via_url] from [false] to [true] [2024-02-06T19:43:18,238][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-neural-basic-index/cWbwnmnxRMmekIlibiAyug] [2024-02-06T19:43:18,240][INFO ][o.o.c.m.MetadataCreateIndexService] [node_name_9200] [test-neural-basic-index] creating index, cause [api], templates [], shards [3]/[1] [2024-02-06T19:43:18,256][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-neural-basic-index/cWbwnmnxRMmekIlibiAyug] [2024-02-06T19:43:18,396][INFO ][o.o.m.m.MLModelManager ] [node_name_9200] create new model meta doc GpDyf40Bamv3vXxyPdlv for register model task GZDyf40Bamv3vXxyPdlZ [2024-02-06T19:43:18,601][INFO ][stdout ] [node_name_9200] Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 1% |█ | traced_small_model.zip Downloading: 2% |█ | traced_small_model.zip Downloading: 3% |██ | traced_small_model.zip Downloading: 4% |██ | traced_small_model.zip Downloading: 5% |███ | traced_small_model.zip Downloading: 6% |███ | traced_small_model.zip Downloading: 7% |███ | traced_small_model.zip Downloading: 8% |████ | traced_small_model.zip Downloading: 9% |████ | traced_small_model.zip Downloading: 10% |█████ | traced_small_model.zip Downloading: 11% |█████ | traced_small_model.zip Downloading: 12% |█████ | traced_small_model.zip Downloading: 13% |██████ | traced_small_model.zip Downloading: 14% |██████ | traced_small_model.zip Downloading: 15% |███████ | traced_small_model.zip Downloading: 16% |███████ | traced_small_model.zip Downloading: 17% |███████ | traced_small_model.zip Downloading: 18% |████████ | traced_small_model.zip Downloading: 19% |████████ | traced_small_model.zip Downloading: 20% |█████████ | traced_small_model.zip Downloading: 21% |█████████ | traced_small_model.zip Downloading: 22% |█████████ | traced_small_model.zip Downloading: 23% |██████████ | traced_small_model.zip Downloading: 24% |██████████ | traced_small_model.zip Downloading: 25% |███████████ | traced_small_model.zip Downloading: 26% |███████████ | traced_small_model.zip Downloading: 27% |███████████ | traced_small_model.zip Downloading: 28% |████████████ | traced_small_model.zip Downloading: 29% |████████████ | traced_small_model.zip Downloading: 30% |█████████████ | traced_small_model.zip Downloading: 31% |█████████████ | traced_small_model.zip Downloading: 32% |█████████████ | traced_small_model.zip Downloading: 33% |██████████████ | traced_small_model.zip Downloading: 34% |██████████████ | traced_small_model.zip Downloading: 35% |███████████████ | traced_small_model.zip Downloading: 36% |███████████████ | traced_small_model.zip Downloading: 37% |███████████████ | traced_small_model.zip Downloading: 38% |████████████████ | traced_small_model.zip Downloading: 39% |████████████████ | traced_small_model.zip Downloading: 40% |█████████████████ | traced_small_model.zip Downloading: 41% |█████████████████ | traced_small_model.zip Downloading: 42% |█████████████████ | traced_small_model.zip Downloading: 43% |██████████████████ | traced_small_model.zip Downloading: 44% |██████████████████ | traced_small_model.zip Downloading: 45% |███████████████████ | traced_small_model.zip Downloading: 46% |███████████████████ | traced_small_model.zip Downloading: 47% |███████████████████ | traced_small_model.zip Downloading: 48% |████████████████████ | traced_small_model.zip Downloading: 49% |████████████████████ | traced_small_model.zip Downloading: 50% |█████████████████████ | traced_small_model.zip Downloading: 51% |█████████████████████ | traced_small_model.zip Downloading: 52% |█████████████████████ | traced_small_model.zip Downloading: 53% |██████████████████████ | traced_small_model.zip Downloading: 54% |██████████████████████ | traced_small_model.zip Downloading: 55% |███████████████████████ | traced_small_model.zip Downloading: 56% |███████████████████████ | traced_small_model.zip Downloading: 57% |███████████████████████ | traced_small_model.zip Downloading: 58% |████████████████████████ | traced_small_model.zip Downloading: 59% |████████████████████████ | traced_small_model.zip Downloading: 60% |█████████████████████████ | traced_small_model.zip Downloading: 61% |█████████████████████████ | traced_small_model.zip Downloading: 62% |█████████████████████████ | traced_small_model.zip Downloading: 63% |██████████████████████████ | traced_small_model.zip Downloading: 64% |██████████████████████████ | traced_small_model.zip Downloading: 65% |███████████████████████████ | traced_small_model.zip Downloading: 66% |███████████████████████████ | traced_small_model.zip Downloading: 67% |███████████████████████████ | traced_small_model.zip Downloading: 68% |████████████████████████████ | traced_small_model.zip Downloading: 69% |████████████████████████████ | traced_small_model.zip Downloading: 70% |█████████████████████████████ | traced_small_model.zip Downloading: 71% |█████████████████████████████ | traced_small_model.zip Downloading: 72% |█████████████████████████████ | traced_small_model.zip Downloading: 73% |██████████████████████████████ | traced_small_model.zip Downloading: 74% |██████████████████████████████ | traced_small_model.zip Downloading: 75% |███████████████████████████████ | traced_small_model.zip Downloading: 76% |███████████████████████████████ | traced_small_model.zip Downloading: 77% |███████████████████████████████ | traced_small_model.zip Downloading: 78% |████████████████████████████████ | traced_small_model.zip Downloading: 79% |████████████████████████████████ | traced_small_model.zip Downloading: 80% |█████████████████████████████████ | traced_small_model.zip Downloading: 81% |█████████████████████████████████ | traced_small_model.zip Downloading: 82% |█████████████████████████████████ | traced_small_model.zip Downloading: 83% |██████████████████████████████████ | traced_small_model.zip Downloading: 84% |██████████████████████████████████ | traced_small_model.zip Downloading: 85% |███████████████████████████████████ | traced_small_model.zip Downloading: 86% |███████████████████████████████████ | traced_small_model.zip Downloading: 87% |███████████████████████████████████ | traced_small_model.zip Downloading: 88% |████████████████████████████████████ | traced_small_model.zip Downloading: 89% |████████████████████████████████████ | traced_small_model.zip Downloading: 90% |█████████████████████████████████████ | traced_small_model.zip Downloading: 91% |█████████████████████████████████████ | traced_small_model.zip Downloading: 92% |█████████████████████████████████████ | traced_small_model.zip Downloading: 93% |██████████████████████████████████████ | traced_small_model.zip Downloading: 94% |██████████████████████████████████████ | traced_small_model.zip Downloading: 95% |███████████████████████████████████████ | traced_small_model.zip Downloading: 96% |███████████████████████████████████████ | traced_small_model.zip Downloading: 97% |███████████████████████████████████████ | traced_small_model.zip Downloading: 98% |████████████████████████████████████████| traced_small_model.zip Downloading: 99% |████████████████████████████████████████| traced_small_model.zip Downloading: 100% |████████████████████████████████████████| traced_small_model.zip [2024-02-06T19:43:18,752][INFO ][o.o.m.m.MLModelManager ] [node_name_9200] Model registered successfully, model id: GpDyf40Bamv3vXxyPdlv, task id: GZDyf40Bamv3vXxyPdlZ [2024-02-06T19:43:20,376][INFO ][o.o.m.a.d.TransportDeployModelAction] [node_name_9200] Will deploy model on these nodes: OAYF21-xRbu9dc9StXHecg [2024-02-06T19:43:20,403][ERROR][o.o.m.m.MLModelManager ] [node_name_9200] No controller is deployed because the model GpDyf40Bamv3vXxyPdlv is expected not having an enabled model controller. Please use the create model controller api to create one if this is unexpected. [2024-02-06T19:43:20,552][INFO ][o.o.m.e.a.DLModel ] [node_name_9200] Model GpDyf40Bamv3vXxyPdlv is successfully deployed on 1 devices [2024-02-06T19:43:20,553][INFO ][o.o.m.a.MLModelAutoReDeployer] [node_name_9200] No models needs to be auto redeployed! [2024-02-06T19:43:20,554][INFO ][o.o.m.a.f.TransportForwardAction] [node_name_9200] deploy model done with state: DEPLOYED, model id: GpDyf40Bamv3vXxyPdlv [2024-02-06T19:43:20,554][INFO ][o.o.m.a.d.TransportDeployModelOnNodeAction] [node_name_9200] deploy model task done G5Dyf40Bamv3vXxyRdk5 [2024-02-06T19:43:25,443][INFO ][o.o.m.a.m.DeleteModelTransportAction] [node_name_9200] Model controller not deleted due to no model controller found for model: GpDyf40Bamv3vXxyPdlv [2024-02-06T19:43:25,450][INFO ][o.o.c.m.MetadataDeleteIndexService] [node_name_9200] [test-neural-basic-index/cWbwnmnxRMmekIlibiAyug] deleting index [2024-02-06T19:43:25,501][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [false] to [true] [2024-02-06T19:43:25,501][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [false] to [true] [2024-02-06T19:43:25,502][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.allow_registering_model_via_url] from [true] to [false] [2024-02-06T19:43:25,526][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [true] to [false] [2024-02-06T19:43:25,526][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [true] to [false] [2024-02-06T19:43:25,555][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.allow_registering_model_via_url] from [false] to [true] [2024-02-06T19:43:25,563][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-neural-multi-doc-index/Oxp9aWzRT8q4FPX1RJrWfQ] [2024-02-06T19:43:25,564][INFO ][o.o.c.m.MetadataCreateIndexService] [node_name_9200] [test-neural-multi-doc-index] creating index, cause [api], templates [], shards [3]/[1] [2024-02-06T19:43:25,578][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-neural-multi-doc-index/Oxp9aWzRT8q4FPX1RJrWfQ] [2024-02-06T19:43:25,746][INFO ][o.o.m.m.MLModelManager ] [node_name_9200] create new model meta doc HpDyf40Bamv3vXxyWtkn for register model task HZDyf40Bamv3vXxyWtkI [2024-02-06T19:43:25,926][INFO ][stdout ] [node_name_9200] Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 1% |█ | traced_small_model.zip Downloading: 2% |█ | traced_small_model.zip Downloading: 3% |██ | traced_small_model.zip Downloading: 4% |██ | traced_small_model.zip Downloading: 5% |███ | traced_small_model.zip Downloading: 6% |███ | traced_small_model.zip Downloading: 7% |███ | traced_small_model.zip Downloading: 8% |████ | traced_small_model.zip Downloading: 9% |████ | traced_small_model.zip Downloading: 10% |█████ | traced_small_model.zip Downloading: 11% |█████ | traced_small_model.zip Downloading: 12% |█████ | traced_small_model.zip Downloading: 13% |██████ | traced_small_model.zip Downloading: 14% |██████ | traced_small_model.zip Downloading: 15% |███████ | traced_small_model.zip Downloading: 16% |███████ | traced_small_model.zip Downloading: 17% |███████ | traced_small_model.zip Downloading: 18% |████████ | traced_small_model.zip Downloading: 19% |████████ | traced_small_model.zip Downloading: 20% |█████████ | traced_small_model.zip Downloading: 21% |█████████ | traced_small_model.zip Downloading: 22% |█████████ | traced_small_model.zip Downloading: 23% |██████████ | traced_small_model.zip Downloading: 24% |██████████ | traced_small_model.zip Downloading: 25% |███████████ | traced_small_model.zip Downloading: 26% |███████████ | traced_small_model.zip Downloading: 27% |███████████ | traced_small_model.zip Downloading: 28% |████████████ | traced_small_model.zip Downloading: 29% |████████████ | traced_small_model.zip Downloading: 30% |█████████████ | traced_small_model.zip Downloading: 31% |█████████████ | traced_small_model.zip Downloading: 32% |█████████████ | traced_small_model.zip Downloading: 33% |██████████████ | traced_small_model.zip Downloading: 34% |██████████████ | traced_small_model.zip Downloading: 35% |███████████████ | traced_small_model.zip Downloading: 36% |███████████████ | traced_small_model.zip Downloading: 37% |███████████████ | traced_small_model.zip Downloading: 38% |████████████████ | traced_small_model.zip Downloading: 39% |████████████████ | traced_small_model.zip Downloading: 40% |█████████████████ | traced_small_model.zip Downloading: 41% |█████████████████ | traced_small_model.zip Downloading: 42% |█████████████████ | traced_small_model.zip Downloading: 43% |██████████████████ | traced_small_model.zip Downloading: 44% |██████████████████ | traced_small_model.zip Downloading: 45% |███████████████████ | traced_small_model.zip Downloading: 46% |███████████████████ | traced_small_model.zip Downloading: 47% |███████████████████ | traced_small_model.zip Downloading: 48% |████████████████████ | traced_small_model.zip Downloading: 49% |████████████████████ | traced_small_model.zip Downloading: 50% |█████████████████████ | traced_small_model.zip Downloading: 51% |█████████████████████ | traced_small_model.zip Downloading: 52% |█████████████████████ | traced_small_model.zip Downloading: 53% |██████████████████████ | traced_small_model.zip Downloading: 54% |██████████████████████ | traced_small_model.zip Downloading: 55% |███████████████████████ | traced_small_model.zip Downloading: 56% |███████████████████████ | traced_small_model.zip Downloading: 57% |███████████████████████ | traced_small_model.zip Downloading: 58% |████████████████████████ | traced_small_model.zip Downloading: 59% |████████████████████████ | traced_small_model.zip Downloading: 60% |█████████████████████████ | traced_small_model.zip Downloading: 61% |█████████████████████████ | traced_small_model.zip Downloading: 62% |█████████████████████████ | traced_small_model.zip Downloading: 63% |██████████████████████████ | traced_small_model.zip Downloading: 64% |██████████████████████████ | traced_small_model.zip Downloading: 65% |███████████████████████████ | traced_small_model.zip Downloading: 66% |███████████████████████████ | traced_small_model.zip Downloading: 67% |███████████████████████████ | traced_small_model.zip Downloading: 68% |████████████████████████████ | traced_small_model.zip Downloading: 69% |████████████████████████████ | traced_small_model.zip Downloading: 70% |█████████████████████████████ | traced_small_model.zip Downloading: 71% |█████████████████████████████ | traced_small_model.zip Downloading: 72% |█████████████████████████████ | traced_small_model.zip Downloading: 73% |██████████████████████████████ | traced_small_model.zip Downloading: 74% |██████████████████████████████ | traced_small_model.zip Downloading: 75% |███████████████████████████████ | traced_small_model.zip Downloading: 76% |███████████████████████████████ | traced_small_model.zip Downloading: 77% |███████████████████████████████ | traced_small_model.zip Downloading: 78% |████████████████████████████████ | traced_small_model.zip Downloading: 79% |████████████████████████████████ | traced_small_model.zip Downloading: 80% |█████████████████████████████████ | traced_small_model.zip Downloading: 81% |█████████████████████████████████ | traced_small_model.zip Downloading: 82% |█████████████████████████████████ | traced_small_model.zip Downloading: 83% |██████████████████████████████████ | traced_small_model.zip Downloading: 84% |██████████████████████████████████ | traced_small_model.zip Downloading: 85% |███████████████████████████████████ | traced_small_model.zip Downloading: 86% |███████████████████████████████████ | traced_small_model.zip Downloading: 87% |███████████████████████████████████ | traced_small_model.zip Downloading: 88% |████████████████████████████████████ | traced_small_model.zip Downloading: 89% |████████████████████████████████████ | traced_small_model.zip Downloading: 90% |█████████████████████████████████████ | traced_small_model.zip Downloading: 91% |█████████████████████████████████████ | traced_small_model.zip Downloading: 92% |█████████████████████████████████████ | traced_small_model.zip Downloading: 93% |██████████████████████████████████████ | traced_small_model.zip Downloading: 94% |██████████████████████████████████████ | traced_small_model.zip Downloading: 95% |███████████████████████████████████████ | traced_small_model.zip Downloading: 96% |███████████████████████████████████████ | traced_small_model.zip Downloading: 97% |███████████████████████████████████████ | traced_small_model.zip Downloading: 98% |████████████████████████████████████████| traced_small_model.zip Downloading: 99% |████████████████████████████████████████| traced_small_model.zip Downloading: 100% |████████████████████████████████████████| traced_small_model.zip [2024-02-06T19:43:26,076][INFO ][o.o.m.m.MLModelManager ] [node_name_9200] Model registered successfully, model id: HpDyf40Bamv3vXxyWtkn, task id: HZDyf40Bamv3vXxyWtkI [2024-02-06T19:43:27,719][INFO ][o.o.m.a.d.TransportDeployModelAction] [node_name_9200] Will deploy model on these nodes: OAYF21-xRbu9dc9StXHecg [2024-02-06T19:43:27,745][ERROR][o.o.m.m.MLModelManager ] [node_name_9200] No controller is deployed because the model HpDyf40Bamv3vXxyWtkn is expected not having an enabled model controller. Please use the create model controller api to create one if this is unexpected. [2024-02-06T19:43:27,895][INFO ][o.o.m.e.a.DLModel ] [node_name_9200] Model HpDyf40Bamv3vXxyWtkn is successfully deployed on 1 devices [2024-02-06T19:43:27,897][INFO ][o.o.m.a.MLModelAutoReDeployer] [node_name_9200] No models needs to be auto redeployed! [2024-02-06T19:43:27,897][INFO ][o.o.m.a.f.TransportForwardAction] [node_name_9200] deploy model done with state: DEPLOYED, model id: HpDyf40Bamv3vXxyWtkn [2024-02-06T19:43:27,897][INFO ][o.o.m.a.d.TransportDeployModelOnNodeAction] [node_name_9200] deploy model task done H5Dyf40Bamv3vXxyYdno [2024-02-06T19:43:32,769][INFO ][o.o.m.a.m.DeleteModelTransportAction] [node_name_9200] Model controller not deleted due to no model controller found for model: HpDyf40Bamv3vXxyWtkn [2024-02-06T19:43:32,776][INFO ][o.o.c.m.MetadataDeleteIndexService] [node_name_9200] [test-neural-multi-doc-index/Oxp9aWzRT8q4FPX1RJrWfQ] deleting index [2024-02-06T19:43:32,833][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [false] to [true] [2024-02-06T19:43:32,833][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [false] to [true] [2024-02-06T19:43:32,833][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.allow_registering_model_via_url] from [true] to [false] [2024-02-06T19:43:32,859][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [true] to [false] [2024-02-06T19:43:32,859][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [true] to [false] [2024-02-06T19:43:32,890][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.allow_registering_model_via_url] from [false] to [true] [2024-02-06T19:43:32,898][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-neural-nested-index/72HX5EJSSn6ExIfyU3rFmw] [2024-02-06T19:43:32,899][INFO ][o.o.c.m.MetadataCreateIndexService] [node_name_9200] [test-neural-nested-index] creating index, cause [api], templates [], shards [3]/[1] [2024-02-06T19:43:32,914][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-neural-nested-index/72HX5EJSSn6ExIfyU3rFmw] [2024-02-06T19:43:33,034][WARN ][o.o.m.a.f.TransportForwardAction] [node_name_9200] Memory Circuit Breaker is open, please check your resources! [2024-02-06T19:43:33,034][WARN ][o.o.m.a.r.TransportRegisterModelAction] [node_name_9200] Memory Circuit Breaker is open, please check your resources! [2024-02-06T19:43:34,037][INFO ][o.o.i.b.HierarchyCircuitBreakerService] [node_name_9200] attempting to trigger G1GC due to high heap usage [1020494240] [2024-02-06T19:43:34,050][INFO ][o.o.i.b.HierarchyCircuitBreakerService] [node_name_9200] GC did bring memory usage down, before [1020494240], after [692269312], allocations [47], duration [13] [2024-02-06T19:43:34,050][INFO ][o.o.c.m.MetadataDeleteIndexService] [node_name_9200] [test-neural-nested-index/72HX5EJSSn6ExIfyU3rFmw] deleting index [2024-02-06T19:43:34,102][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [false] to [true] [2024-02-06T19:43:34,102][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [false] to [true] [2024-02-06T19:43:34,103][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.allow_registering_model_via_url] from [true] to [false] [2024-02-06T19:43:34,128][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [true] to [false] [2024-02-06T19:43:34,128][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [true] to [false] [2024-02-06T19:43:34,158][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.allow_registering_model_via_url] from [false] to [true] [2024-02-06T19:43:34,166][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-neural-basic-index/bg8B2KS9TXCEiNfCZ5RSZA] [2024-02-06T19:43:34,168][INFO ][o.o.c.m.MetadataCreateIndexService] [node_name_9200] [test-neural-basic-index] creating index, cause [api], templates [], shards [3]/[1] [2024-02-06T19:43:34,182][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-neural-basic-index/bg8B2KS9TXCEiNfCZ5RSZA] [2024-02-06T19:43:34,320][INFO ][o.o.m.m.MLModelManager ] [node_name_9200] create new model meta doc JJDyf40Bamv3vXxye9mm for register model task I5Dyf40Bamv3vXxye9mR [2024-02-06T19:43:34,512][INFO ][stdout ] [node_name_9200] Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 1% |█ | traced_small_model.zip Downloading: 2% |█ | traced_small_model.zip Downloading: 3% |██ | traced_small_model.zip Downloading: 4% |██ | traced_small_model.zip Downloading: 5% |███ | traced_small_model.zip Downloading: 6% |███ | traced_small_model.zip Downloading: 7% |███ | traced_small_model.zip Downloading: 8% |████ | traced_small_model.zip Downloading: 9% |████ | traced_small_model.zip Downloading: 10% |█████ | traced_small_model.zip Downloading: 11% |█████ | traced_small_model.zip Downloading: 12% |█████ | traced_small_model.zip Downloading: 13% |██████ | traced_small_model.zip Downloading: 14% |██████ | traced_small_model.zip Downloading: 15% |███████ | traced_small_model.zip Downloading: 16% |███████ | traced_small_model.zip Downloading: 17% |███████ | traced_small_model.zip Downloading: 18% |████████ | traced_small_model.zip Downloading: 19% |████████ | traced_small_model.zip Downloading: 20% |█████████ | traced_small_model.zip Downloading: 21% |█████████ | traced_small_model.zip Downloading: 22% |█████████ | traced_small_model.zip Downloading: 23% |██████████ | traced_small_model.zip Downloading: 24% |██████████ | traced_small_model.zip Downloading: 25% |███████████ | traced_small_model.zip Downloading: 26% |███████████ | traced_small_model.zip Downloading: 27% |███████████ | traced_small_model.zip Downloading: 28% |████████████ | traced_small_model.zip Downloading: 29% |████████████ | traced_small_model.zip Downloading: 30% |█████████████ | traced_small_model.zip Downloading: 31% |█████████████ | traced_small_model.zip Downloading: 32% |█████████████ | traced_small_model.zip Downloading: 33% |██████████████ | traced_small_model.zip Downloading: 34% |██████████████ | traced_small_model.zip Downloading: 35% |███████████████ | traced_small_model.zip Downloading: 36% |███████████████ | traced_small_model.zip Downloading: 37% |███████████████ | traced_small_model.zip Downloading: 38% |████████████████ | traced_small_model.zip Downloading: 39% |████████████████ | traced_small_model.zip Downloading: 40% |█████████████████ | traced_small_model.zip Downloading: 41% |█████████████████ | traced_small_model.zip Downloading: 42% |█████████████████ | traced_small_model.zip Downloading: 43% |██████████████████ | traced_small_model.zip Downloading: 44% |██████████████████ | traced_small_model.zip Downloading: 45% |███████████████████ | traced_small_model.zip Downloading: 46% |███████████████████ | traced_small_model.zip Downloading: 47% |███████████████████ | traced_small_model.zip Downloading: 48% |████████████████████ | traced_small_model.zip Downloading: 49% |████████████████████ | traced_small_model.zip Downloading: 50% |█████████████████████ | traced_small_model.zip Downloading: 51% |█████████████████████ | traced_small_model.zip Downloading: 52% |█████████████████████ | traced_small_model.zip Downloading: 53% |██████████████████████ | traced_small_model.zip Downloading: 54% |██████████████████████ | traced_small_model.zip Downloading: 55% |███████████████████████ | traced_small_model.zip Downloading: 56% |███████████████████████ | traced_small_model.zip Downloading: 57% |███████████████████████ | traced_small_model.zip Downloading: 58% |████████████████████████ | traced_small_model.zip Downloading: 59% |████████████████████████ | traced_small_model.zip Downloading: 60% |█████████████████████████ | traced_small_model.zip Downloading: 61% |█████████████████████████ | traced_small_model.zip Downloading: 62% |█████████████████████████ | traced_small_model.zip Downloading: 63% |██████████████████████████ | traced_small_model.zip Downloading: 64% |██████████████████████████ | traced_small_model.zip Downloading: 65% |███████████████████████████ | traced_small_model.zip Downloading: 66% |███████████████████████████ | traced_small_model.zip Downloading: 67% |███████████████████████████ | traced_small_model.zip Downloading: 68% |████████████████████████████ | traced_small_model.zip Downloading: 69% |████████████████████████████ | traced_small_model.zip Downloading: 70% |█████████████████████████████ | traced_small_model.zip Downloading: 71% |█████████████████████████████ | traced_small_model.zip Downloading: 72% |█████████████████████████████ | traced_small_model.zip Downloading: 73% |██████████████████████████████ | traced_small_model.zip Downloading: 74% |██████████████████████████████ | traced_small_model.zip Downloading: 75% |███████████████████████████████ | traced_small_model.zip Downloading: 76% |███████████████████████████████ | traced_small_model.zip Downloading: 77% |███████████████████████████████ | traced_small_model.zip Downloading: 78% |████████████████████████████████ | traced_small_model.zip Downloading: 79% |████████████████████████████████ | traced_small_model.zip Downloading: 80% |█████████████████████████████████ | traced_small_model.zip Downloading: 81% |█████████████████████████████████ | traced_small_model.zip Downloading: 82% |█████████████████████████████████ | traced_small_model.zip Downloading: 83% |██████████████████████████████████ | traced_small_model.zip Downloading: 84% |██████████████████████████████████ | traced_small_model.zip Downloading: 85% |███████████████████████████████████ | traced_small_model.zip Downloading: 86% |███████████████████████████████████ | traced_small_model.zip Downloading: 87% |███████████████████████████████████ | traced_small_model.zip Downloading: 88% |████████████████████████████████████ | traced_small_model.zip Downloading: 89% |████████████████████████████████████ | traced_small_model.zip Downloading: 90% |█████████████████████████████████████ | traced_small_model.zip Downloading: 91% |█████████████████████████████████████ | traced_small_model.zip Downloading: 92% |█████████████████████████████████████ | traced_small_model.zip Downloading: 93% |██████████████████████████████████████ | traced_small_model.zip Downloading: 94% |██████████████████████████████████████ | traced_small_model.zip Downloading: 95% |███████████████████████████████████████ | traced_small_model.zip Downloading: 96% |███████████████████████████████████████ | traced_small_model.zip Downloading: 97% |███████████████████████████████████████ | traced_small_model.zip Downloading: 98% |████████████████████████████████████████| traced_small_model.zip Downloading: 99% |████████████████████████████████████████| traced_small_model.zip Downloading: 100% |████████████████████████████████████████| traced_small_model.zip [2024-02-06T19:43:34,664][INFO ][o.o.m.m.MLModelManager ] [node_name_9200] Model registered successfully, model id: JJDyf40Bamv3vXxye9mm, task id: I5Dyf40Bamv3vXxye9mR [2024-02-06T19:43:36,303][INFO ][o.o.m.a.d.TransportDeployModelAction] [node_name_9200] Will deploy model on these nodes: OAYF21-xRbu9dc9StXHecg [2024-02-06T19:43:36,327][ERROR][o.o.m.m.MLModelManager ] [node_name_9200] No controller is deployed because the model JJDyf40Bamv3vXxye9mm is expected not having an enabled model controller. Please use the create model controller api to create one if this is unexpected. [2024-02-06T19:43:36,479][INFO ][o.o.m.e.a.DLModel ] [node_name_9200] Model JJDyf40Bamv3vXxye9mm is successfully deployed on 1 devices [2024-02-06T19:43:36,480][INFO ][o.o.m.a.MLModelAutoReDeployer] [node_name_9200] No models needs to be auto redeployed! [2024-02-06T19:43:36,480][INFO ][o.o.m.a.f.TransportForwardAction] [node_name_9200] deploy model done with state: DEPLOYED, model id: JJDyf40Bamv3vXxye9mm [2024-02-06T19:43:36,480][INFO ][o.o.m.a.d.TransportDeployModelOnNodeAction] [node_name_9200] deploy model task done JZDyf40Bamv3vXxyg9lv [2024-02-06T19:43:41,345][INFO ][o.o.m.a.m.DeleteModelTransportAction] [node_name_9200] Model controller not deleted due to no model controller found for model: JJDyf40Bamv3vXxye9mm [2024-02-06T19:43:41,352][INFO ][o.o.c.m.MetadataDeleteIndexService] [node_name_9200] [test-neural-basic-index/bg8B2KS9TXCEiNfCZ5RSZA] deleting index [2024-02-06T19:43:41,410][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [false] to [true] [2024-02-06T19:43:41,410][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [false] to [true] [2024-02-06T19:43:41,410][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.allow_registering_model_via_url] from [true] to [false] [2024-02-06T19:43:41,443][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [true] to [false] [2024-02-06T19:43:41,443][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [true] to [false] [2024-02-06T19:43:41,485][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.allow_registering_model_via_url] from [false] to [true] [2024-02-06T19:43:41,492][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-neural-text-and-vector-field-index/scKSweAMQJmMCnUQG284ng] [2024-02-06T19:43:41,494][INFO ][o.o.c.m.MetadataCreateIndexService] [node_name_9200] [test-neural-text-and-vector-field-index] creating index, cause [api], templates [], shards [3]/[1] [2024-02-06T19:43:41,509][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-neural-text-and-vector-field-index/scKSweAMQJmMCnUQG284ng] [2024-02-06T19:43:41,596][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-neural-text-and-vector-field-index/scKSweAMQJmMCnUQG284ng] [2024-02-06T19:43:41,598][INFO ][o.o.c.m.MetadataMappingService] [node_name_9200] [test-neural-text-and-vector-field-index/scKSweAMQJmMCnUQG284ng] update_mapping [_doc] [2024-02-06T19:43:41,673][INFO ][o.o.m.m.MLModelManager ] [node_name_9200] create new model meta doc KJDyf40Bamv3vXxymNlb for register model task J5Dyf40Bamv3vXxymNlH [2024-02-06T19:43:41,870][INFO ][stdout ] [node_name_9200] Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 1% |█ | traced_small_model.zip Downloading: 2% |█ | traced_small_model.zip Downloading: 3% |██ | traced_small_model.zip Downloading: 4% |██ | traced_small_model.zip Downloading: 5% |███ | traced_small_model.zip Downloading: 6% |███ | traced_small_model.zip Downloading: 7% |███ | traced_small_model.zip Downloading: 8% |████ | traced_small_model.zip Downloading: 9% |████ | traced_small_model.zip Downloading: 10% |█████ | traced_small_model.zip Downloading: 11% |█████ | traced_small_model.zip Downloading: 12% |█████ | traced_small_model.zip Downloading: 13% |██████ | traced_small_model.zip Downloading: 14% |██████ | traced_small_model.zip Downloading: 15% |███████ | traced_small_model.zip Downloading: 16% |███████ | traced_small_model.zip Downloading: 17% |███████ | traced_small_model.zip Downloading: 18% |████████ | traced_small_model.zip Downloading: 19% |████████ | traced_small_model.zip Downloading: 20% |█████████ | traced_small_model.zip Downloading: 21% |█████████ | traced_small_model.zip Downloading: 22% |█████████ | traced_small_model.zip Downloading: 23% |██████████ | traced_small_model.zip Downloading: 24% |██████████ | traced_small_model.zip Downloading: 25% |███████████ | traced_small_model.zip Downloading: 26% |███████████ | traced_small_model.zip Downloading: 27% |███████████ | traced_small_model.zip Downloading: 28% |████████████ | traced_small_model.zip Downloading: 29% |████████████ | traced_small_model.zip Downloading: 30% |█████████████ | traced_small_model.zip Downloading: 31% |█████████████ | traced_small_model.zip Downloading: 32% |█████████████ | traced_small_model.zip Downloading: 33% |██████████████ | traced_small_model.zip Downloading: 34% |██████████████ | traced_small_model.zip Downloading: 35% |███████████████ | traced_small_model.zip Downloading: 36% |███████████████ | traced_small_model.zip Downloading: 37% |███████████████ | traced_small_model.zip Downloading: 38% |████████████████ | traced_small_model.zip Downloading: 39% |████████████████ | traced_small_model.zip Downloading: 40% |█████████████████ | traced_small_model.zip Downloading: 41% |█████████████████ | traced_small_model.zip Downloading: 42% |█████████████████ | traced_small_model.zip Downloading: 43% |██████████████████ | traced_small_model.zip Downloading: 44% |██████████████████ | traced_small_model.zip Downloading: 45% |███████████████████ | traced_small_model.zip Downloading: 46% |███████████████████ | traced_small_model.zip Downloading: 47% |███████████████████ | traced_small_model.zip Downloading: 48% |████████████████████ | traced_small_model.zip Downloading: 49% |████████████████████ | traced_small_model.zip Downloading: 50% |█████████████████████ | traced_small_model.zip Downloading: 51% |█████████████████████ | traced_small_model.zip Downloading: 52% |█████████████████████ | traced_small_model.zip Downloading: 53% |██████████████████████ | traced_small_model.zip Downloading: 54% |██████████████████████ | traced_small_model.zip Downloading: 55% |███████████████████████ | traced_small_model.zip Downloading: 56% |███████████████████████ | traced_small_model.zip Downloading: 57% |███████████████████████ | traced_small_model.zip Downloading: 58% |████████████████████████ | traced_small_model.zip Downloading: 59% |████████████████████████ | traced_small_model.zip Downloading: 60% |█████████████████████████ | traced_small_model.zip Downloading: 61% |█████████████████████████ | traced_small_model.zip Downloading: 62% |█████████████████████████ | traced_small_model.zip Downloading: 63% |██████████████████████████ | traced_small_model.zip Downloading: 64% |██████████████████████████ | traced_small_model.zip Downloading: 65% |███████████████████████████ | traced_small_model.zip Downloading: 66% |███████████████████████████ | traced_small_model.zip Downloading: 67% |███████████████████████████ | traced_small_model.zip Downloading: 68% |████████████████████████████ | traced_small_model.zip Downloading: 69% |████████████████████████████ | traced_small_model.zip Downloading: 70% |█████████████████████████████ | traced_small_model.zip Downloading: 71% |█████████████████████████████ | traced_small_model.zip Downloading: 72% |█████████████████████████████ | traced_small_model.zip Downloading: 73% |██████████████████████████████ | traced_small_model.zip Downloading: 74% |██████████████████████████████ | traced_small_model.zip Downloading: 75% |███████████████████████████████ | traced_small_model.zip Downloading: 76% |███████████████████████████████ | traced_small_model.zip Downloading: 77% |███████████████████████████████ | traced_small_model.zip Downloading: 78% |████████████████████████████████ | traced_small_model.zip Downloading: 79% |████████████████████████████████ | traced_small_model.zip Downloading: 80% |█████████████████████████████████ | traced_small_model.zip Downloading: 81% |█████████████████████████████████ | traced_small_model.zip Downloading: 82% |█████████████████████████████████ | traced_small_model.zip Downloading: 83% |██████████████████████████████████ | traced_small_model.zip Downloading: 84% |██████████████████████████████████ | traced_small_model.zip Downloading: 85% |███████████████████████████████████ | traced_small_model.zip Downloading: 86% |███████████████████████████████████ | traced_small_model.zip Downloading: 87% |███████████████████████████████████ | traced_small_model.zip Downloading: 88% |████████████████████████████████████ | traced_small_model.zip Downloading: 89% |████████████████████████████████████ | traced_small_model.zip Downloading: 90% |█████████████████████████████████████ | traced_small_model.zip Downloading: 91% |█████████████████████████████████████ | traced_small_model.zip Downloading: 92% |█████████████████████████████████████ | traced_small_model.zip Downloading: 93% |██████████████████████████████████████ | traced_small_model.zip Downloading: 94% |██████████████████████████████████████ | traced_small_model.zip Downloading: 95% |███████████████████████████████████████ | traced_small_model.zip Downloading: 96% |███████████████████████████████████████ | traced_small_model.zip Downloading: 97% |███████████████████████████████████████ | traced_small_model.zip Downloading: 98% |████████████████████████████████████████| traced_small_model.zip Downloading: 99% |████████████████████████████████████████| traced_small_model.zip Downloading: 100% |████████████████████████████████████████| traced_small_model.zip [2024-02-06T19:43:42,024][INFO ][o.o.m.m.MLModelManager ] [node_name_9200] Model registered successfully, model id: KJDyf40Bamv3vXxymNlb, task id: J5Dyf40Bamv3vXxymNlH [2024-02-06T19:43:43,654][INFO ][o.o.m.a.d.TransportDeployModelAction] [node_name_9200] Will deploy model on these nodes: OAYF21-xRbu9dc9StXHecg [2024-02-06T19:43:43,681][ERROR][o.o.m.m.MLModelManager ] [node_name_9200] No controller is deployed because the model KJDyf40Bamv3vXxymNlb is expected not having an enabled model controller. Please use the create model controller api to create one if this is unexpected. [2024-02-06T19:43:43,837][INFO ][o.o.m.e.a.DLModel ] [node_name_9200] Model KJDyf40Bamv3vXxymNlb is successfully deployed on 1 devices [2024-02-06T19:43:43,838][INFO ][o.o.m.a.MLModelAutoReDeployer] [node_name_9200] No models needs to be auto redeployed! [2024-02-06T19:43:43,838][INFO ][o.o.m.a.f.TransportForwardAction] [node_name_9200] deploy model done with state: DEPLOYED, model id: KJDyf40Bamv3vXxymNlb [2024-02-06T19:43:43,839][INFO ][o.o.m.a.d.TransportDeployModelOnNodeAction] [node_name_9200] deploy model task done KZDyf40Bamv3vXxyoNkn [2024-02-06T19:43:48,704][INFO ][o.o.m.a.m.DeleteModelTransportAction] [node_name_9200] Model controller not deleted due to no model controller found for model: KJDyf40Bamv3vXxymNlb [2024-02-06T19:43:48,710][INFO ][o.o.c.m.MetadataDeleteIndexService] [node_name_9200] [test-neural-text-and-vector-field-index/scKSweAMQJmMCnUQG284ng] deleting index [2024-02-06T19:43:48,760][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [false] to [true] [2024-02-06T19:43:48,760][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [false] to [true] [2024-02-06T19:43:48,761][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.allow_registering_model_via_url] from [true] to [false] [2024-02-06T19:43:48,788][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [true] to [false] [2024-02-06T19:43:48,789][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [true] to [false] [2024-02-06T19:43:48,817][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.allow_registering_model_via_url] from [false] to [true] [2024-02-06T19:43:48,824][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-neural-basic-index/kd-ffrh8TSSk1mJqfRoQ5Q] [2024-02-06T19:43:48,826][INFO ][o.o.c.m.MetadataCreateIndexService] [node_name_9200] [test-neural-basic-index] creating index, cause [api], templates [], shards [3]/[1] [2024-02-06T19:43:48,839][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-neural-basic-index/kd-ffrh8TSSk1mJqfRoQ5Q] [2024-02-06T19:43:48,972][INFO ][o.o.m.m.MLModelManager ] [node_name_9200] create new model meta doc LJDyf40Bamv3vXxytNni for register model task K5Dyf40Bamv3vXxytNnO [2024-02-06T19:43:49,140][INFO ][stdout ] [node_name_9200] Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 1% |█ | traced_small_model.zip Downloading: 2% |█ | traced_small_model.zip Downloading: 3% |██ | traced_small_model.zip Downloading: 4% |██ | traced_small_model.zip Downloading: 5% |███ | traced_small_model.zip Downloading: 6% |███ | traced_small_model.zip Downloading: 7% |███ | traced_small_model.zip Downloading: 8% |████ | traced_small_model.zip Downloading: 9% |████ | traced_small_model.zip Downloading: 10% |█████ | traced_small_model.zip Downloading: 11% |█████ | traced_small_model.zip Downloading: 12% |█████ | traced_small_model.zip Downloading: 13% |██████ | traced_small_model.zip Downloading: 14% |██████ | traced_small_model.zip Downloading: 15% |███████ | traced_small_model.zip Downloading: 16% |███████ | traced_small_model.zip Downloading: 17% |███████ | traced_small_model.zip Downloading: 18% |████████ | traced_small_model.zip Downloading: 19% |████████ | traced_small_model.zip Downloading: 20% |█████████ | traced_small_model.zip Downloading: 21% |█████████ | traced_small_model.zip Downloading: 22% |█████████ | traced_small_model.zip Downloading: 23% |██████████ | traced_small_model.zip Downloading: 24% |██████████ | traced_small_model.zip Downloading: 25% |███████████ | traced_small_model.zip Downloading: 26% |███████████ | traced_small_model.zip Downloading: 27% |███████████ | traced_small_model.zip Downloading: 28% |████████████ | traced_small_model.zip Downloading: 29% |████████████ | traced_small_model.zip Downloading: 30% |█████████████ | traced_small_model.zip Downloading: 31% |█████████████ | traced_small_model.zip Downloading: 32% |█████████████ | traced_small_model.zip Downloading: 33% |██████████████ | traced_small_model.zip Downloading: 34% |██████████████ | traced_small_model.zip Downloading: 35% |███████████████ | traced_small_model.zip Downloading: 36% |███████████████ | traced_small_model.zip Downloading: 37% |███████████████ | traced_small_model.zip Downloading: 38% |████████████████ | traced_small_model.zip Downloading: 39% |████████████████ | traced_small_model.zip Downloading: 40% |█████████████████ | traced_small_model.zip Downloading: 41% |█████████████████ | traced_small_model.zip Downloading: 42% |█████████████████ | traced_small_model.zip Downloading: 43% |██████████████████ | traced_small_model.zip Downloading: 44% |██████████████████ | traced_small_model.zip Downloading: 45% |███████████████████ | traced_small_model.zip Downloading: 46% |███████████████████ | traced_small_model.zip Downloading: 47% |███████████████████ | traced_small_model.zip Downloading: 48% |████████████████████ | traced_small_model.zip Downloading: 49% |████████████████████ | traced_small_model.zip Downloading: 50% |█████████████████████ | traced_small_model.zip Downloading: 51% |█████████████████████ | traced_small_model.zip Downloading: 52% |█████████████████████ | traced_small_model.zip Downloading: 53% |██████████████████████ | traced_small_model.zip Downloading: 54% |██████████████████████ | traced_small_model.zip Downloading: 55% |███████████████████████ | traced_small_model.zip Downloading: 56% |███████████████████████ | traced_small_model.zip Downloading: 57% |███████████████████████ | traced_small_model.zip Downloading: 58% |████████████████████████ | traced_small_model.zip Downloading: 59% |████████████████████████ | traced_small_model.zip Downloading: 60% |█████████████████████████ | traced_small_model.zip Downloading: 61% |█████████████████████████ | traced_small_model.zip Downloading: 62% |█████████████████████████ | traced_small_model.zip Downloading: 63% |██████████████████████████ | traced_small_model.zip Downloading: 64% |██████████████████████████ | traced_small_model.zip Downloading: 65% |███████████████████████████ | traced_small_model.zip Downloading: 66% |███████████████████████████ | traced_small_model.zip Downloading: 67% |███████████████████████████ | traced_small_model.zip Downloading: 68% |████████████████████████████ | traced_small_model.zip Downloading: 69% |████████████████████████████ | traced_small_model.zip Downloading: 70% |█████████████████████████████ | traced_small_model.zip Downloading: 71% |█████████████████████████████ | traced_small_model.zip Downloading: 72% |█████████████████████████████ | traced_small_model.zip Downloading: 73% |██████████████████████████████ | traced_small_model.zip Downloading: 74% |██████████████████████████████ | traced_small_model.zip Downloading: 75% |███████████████████████████████ | traced_small_model.zip Downloading: 76% |███████████████████████████████ | traced_small_model.zip Downloading: 77% |███████████████████████████████ | traced_small_model.zip Downloading: 78% |████████████████████████████████ | traced_small_model.zip Downloading: 79% |████████████████████████████████ | traced_small_model.zip Downloading: 80% |█████████████████████████████████ | traced_small_model.zip Downloading: 81% |█████████████████████████████████ | traced_small_model.zip Downloading: 82% |█████████████████████████████████ | traced_small_model.zip Downloading: 83% |██████████████████████████████████ | traced_small_model.zip Downloading: 84% |██████████████████████████████████ | traced_small_model.zip Downloading: 85% |███████████████████████████████████ | traced_small_model.zip Downloading: 86% |███████████████████████████████████ | traced_small_model.zip Downloading: 87% |███████████████████████████████████ | traced_small_model.zip Downloading: 88% |████████████████████████████████████ | traced_small_model.zip Downloading: 89% |████████████████████████████████████ | traced_small_model.zip Downloading: 90% |█████████████████████████████████████ | traced_small_model.zip Downloading: 91% |█████████████████████████████████████ | traced_small_model.zip Downloading: 92% |█████████████████████████████████████ | traced_small_model.zip Downloading: 93% |██████████████████████████████████████ | traced_small_model.zip Downloading: 94% |██████████████████████████████████████ | traced_small_model.zip Downloading: 95% |███████████████████████████████████████ | traced_small_model.zip Downloading: 96% |███████████████████████████████████████ | traced_small_model.zip Downloading: 97% |███████████████████████████████████████ | traced_small_model.zip Downloading: 98% |████████████████████████████████████████| traced_small_model.zip Downloading: 99% |████████████████████████████████████████| traced_small_model.zip Downloading: 100% |████████████████████████████████████████| traced_small_model.zip [2024-02-06T19:43:49,289][INFO ][o.o.m.m.MLModelManager ] [node_name_9200] Model registered successfully, model id: LJDyf40Bamv3vXxytNni, task id: K5Dyf40Bamv3vXxytNnO [2024-02-06T19:43:50,958][INFO ][o.o.m.a.d.TransportDeployModelAction] [node_name_9200] Will deploy model on these nodes: OAYF21-xRbu9dc9StXHecg [2024-02-06T19:43:50,995][ERROR][o.o.m.m.MLModelManager ] [node_name_9200] No controller is deployed because the model LJDyf40Bamv3vXxytNni is expected not having an enabled model controller. Please use the create model controller api to create one if this is unexpected. [2024-02-06T19:43:51,140][INFO ][o.o.m.e.a.DLModel ] [node_name_9200] Model LJDyf40Bamv3vXxytNni is successfully deployed on 1 devices [2024-02-06T19:43:51,141][INFO ][o.o.m.a.MLModelAutoReDeployer] [node_name_9200] No models needs to be auto redeployed! [2024-02-06T19:43:51,142][INFO ][o.o.m.a.f.TransportForwardAction] [node_name_9200] deploy model done with state: DEPLOYED, model id: LJDyf40Bamv3vXxytNni [2024-02-06T19:43:51,142][INFO ][o.o.m.a.d.TransportDeployModelOnNodeAction] [node_name_9200] deploy model task done LZDyf40Bamv3vXxyvNmu [2024-02-06T19:43:56,015][INFO ][o.o.m.a.m.DeleteModelTransportAction] [node_name_9200] Model controller not deleted due to no model controller found for model: LJDyf40Bamv3vXxytNni [2024-02-06T19:43:56,021][INFO ][o.o.c.m.MetadataDeleteIndexService] [node_name_9200] [test-neural-basic-index/kd-ffrh8TSSk1mJqfRoQ5Q] deleting index [2024-02-06T19:43:56,081][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [false] to [true] [2024-02-06T19:43:56,081][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [false] to [true] [2024-02-06T19:43:56,081][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.allow_registering_model_via_url] from [true] to [false] [2024-02-06T19:43:56,106][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [true] to [false] [2024-02-06T19:43:56,106][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [true] to [false] [2024-02-06T19:43:56,135][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.allow_registering_model_via_url] from [false] to [true] [2024-02-06T19:43:56,143][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-neural-basic-index/-s-uu4QBSZCHi5usrxZFBg] [2024-02-06T19:43:56,144][INFO ][o.o.c.m.MetadataCreateIndexService] [node_name_9200] [test-neural-basic-index] creating index, cause [api], templates [], shards [3]/[1] [2024-02-06T19:43:56,160][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-neural-basic-index/-s-uu4QBSZCHi5usrxZFBg] [2024-02-06T19:43:56,312][INFO ][o.o.m.m.MLModelManager ] [node_name_9200] create new model meta doc MJDyf40Bamv3vXxy0dmM for register model task L5Dyf40Bamv3vXxy0dls [2024-02-06T19:43:56,523][INFO ][stdout ] [node_name_9200] Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 0% |█ | traced_small_model.zip Downloading: 1% |█ | traced_small_model.zip Downloading: 2% |█ | traced_small_model.zip Downloading: 3% |██ | traced_small_model.zip Downloading: 4% |██ | traced_small_model.zip Downloading: 5% |███ | traced_small_model.zip Downloading: 6% |███ | traced_small_model.zip Downloading: 7% |███ | traced_small_model.zip Downloading: 8% |████ | traced_small_model.zip Downloading: 9% |████ | traced_small_model.zip Downloading: 10% |█████ | traced_small_model.zip Downloading: 11% |█████ | traced_small_model.zip Downloading: 12% |█████ | traced_small_model.zip Downloading: 13% |██████ | traced_small_model.zip Downloading: 14% |██████ | traced_small_model.zip Downloading: 15% |███████ | traced_small_model.zip Downloading: 16% |███████ | traced_small_model.zip Downloading: 17% |███████ | traced_small_model.zip Downloading: 18% |████████ | traced_small_model.zip Downloading: 19% |████████ | traced_small_model.zip Downloading: 20% |█████████ | traced_small_model.zip Downloading: 21% |█████████ | traced_small_model.zip Downloading: 22% |█████████ | traced_small_model.zip Downloading: 23% |██████████ | traced_small_model.zip Downloading: 24% |██████████ | traced_small_model.zip Downloading: 25% |███████████ | traced_small_model.zip Downloading: 26% |███████████ | traced_small_model.zip Downloading: 27% |███████████ | traced_small_model.zip Downloading: 28% |████████████ | traced_small_model.zip Downloading: 29% |████████████ | traced_small_model.zip Downloading: 30% |█████████████ | traced_small_model.zip Downloading: 31% |█████████████ | traced_small_model.zip Downloading: 32% |█████████████ | traced_small_model.zip Downloading: 33% |██████████████ | traced_small_model.zip Downloading: 34% |██████████████ | traced_small_model.zip Downloading: 35% |███████████████ | traced_small_model.zip Downloading: 36% |███████████████ | traced_small_model.zip Downloading: 37% |███████████████ | traced_small_model.zip Downloading: 38% |████████████████ | traced_small_model.zip Downloading: 39% |████████████████ | traced_small_model.zip Downloading: 40% |█████████████████ | traced_small_model.zip Downloading: 41% |█████████████████ | traced_small_model.zip Downloading: 42% |█████████████████ | traced_small_model.zip Downloading: 43% |██████████████████ | traced_small_model.zip Downloading: 44% |██████████████████ | traced_small_model.zip Downloading: 45% |███████████████████ | traced_small_model.zip Downloading: 46% |███████████████████ | traced_small_model.zip Downloading: 47% |███████████████████ | traced_small_model.zip Downloading: 48% |████████████████████ | traced_small_model.zip Downloading: 49% |████████████████████ | traced_small_model.zip Downloading: 50% |█████████████████████ | traced_small_model.zip Downloading: 51% |█████████████████████ | traced_small_model.zip Downloading: 52% |█████████████████████ | traced_small_model.zip Downloading: 53% |██████████████████████ | traced_small_model.zip Downloading: 54% |██████████████████████ | traced_small_model.zip Downloading: 55% |███████████████████████ | traced_small_model.zip Downloading: 56% |███████████████████████ | traced_small_model.zip Downloading: 57% |███████████████████████ | traced_small_model.zip Downloading: 58% |████████████████████████ | traced_small_model.zip Downloading: 59% |████████████████████████ | traced_small_model.zip Downloading: 60% |█████████████████████████ | traced_small_model.zip Downloading: 61% |█████████████████████████ | traced_small_model.zip Downloading: 62% |█████████████████████████ | traced_small_model.zip Downloading: 63% |██████████████████████████ | traced_small_model.zip Downloading: 64% |██████████████████████████ | traced_small_model.zip Downloading: 65% |███████████████████████████ | traced_small_model.zip Downloading: 66% |███████████████████████████ | traced_small_model.zip Downloading: 67% |███████████████████████████ | traced_small_model.zip Downloading: 68% |████████████████████████████ | traced_small_model.zip Downloading: 69% |████████████████████████████ | traced_small_model.zip Downloading: 70% |█████████████████████████████ | traced_small_model.zip Downloading: 71% |█████████████████████████████ | traced_small_model.zip Downloading: 72% |█████████████████████████████ | traced_small_model.zip Downloading: 73% |██████████████████████████████ | traced_small_model.zip Downloading: 74% |██████████████████████████████ | traced_small_model.zip Downloading: 75% |███████████████████████████████ | traced_small_model.zip Downloading: 76% |███████████████████████████████ | traced_small_model.zip Downloading: 77% |███████████████████████████████ | traced_small_model.zip Downloading: 78% |████████████████████████████████ | traced_small_model.zip Downloading: 79% |████████████████████████████████ | traced_small_model.zip Downloading: 80% |█████████████████████████████████ | traced_small_model.zip Downloading: 81% |█████████████████████████████████ | traced_small_model.zip Downloading: 82% |█████████████████████████████████ | traced_small_model.zip Downloading: 83% |██████████████████████████████████ | traced_small_model.zip Downloading: 84% |██████████████████████████████████ | traced_small_model.zip Downloading: 85% |███████████████████████████████████ | traced_small_model.zip Downloading: 86% |███████████████████████████████████ | traced_small_model.zip Downloading: 87% |███████████████████████████████████ | traced_small_model.zip Downloading: 88% |████████████████████████████████████ | traced_small_model.zip Downloading: 89% |████████████████████████████████████ | traced_small_model.zip Downloading: 90% |█████████████████████████████████████ | traced_small_model.zip Downloading: 91% |█████████████████████████████████████ | traced_small_model.zip Downloading: 92% |█████████████████████████████████████ | traced_small_model.zip Downloading: 93% |██████████████████████████████████████ | traced_small_model.zip Downloading: 94% |██████████████████████████████████████ | traced_small_model.zip Downloading: 95% |███████████████████████████████████████ | traced_small_model.zip Downloading: 96% |███████████████████████████████████████ | traced_small_model.zip Downloading: 97% |███████████████████████████████████████ | traced_small_model.zip Downloading: 98% |████████████████████████████████████████| traced_small_model.zip Downloading: 99% |████████████████████████████████████████| traced_small_model.zip Downloading: 100% |████████████████████████████████████████| traced_small_model.zip [2024-02-06T19:43:56,672][INFO ][o.o.m.m.MLModelManager ] [node_name_9200] Model registered successfully, model id: MJDyf40Bamv3vXxy0dmM, task id: L5Dyf40Bamv3vXxy0dls [2024-02-06T19:43:58,283][INFO ][o.o.m.a.d.TransportDeployModelAction] [node_name_9200] Will deploy model on these nodes: OAYF21-xRbu9dc9StXHecg [2024-02-06T19:43:58,309][ERROR][o.o.m.m.MLModelManager ] [node_name_9200] No controller is deployed because the model MJDyf40Bamv3vXxy0dmM is expected not having an enabled model controller. Please use the create model controller api to create one if this is unexpected. [2024-02-06T19:43:58,456][INFO ][o.o.m.e.a.DLModel ] [node_name_9200] Model MJDyf40Bamv3vXxy0dmM is successfully deployed on 1 devices [2024-02-06T19:43:58,457][INFO ][o.o.m.a.MLModelAutoReDeployer] [node_name_9200] No models needs to be auto redeployed! [2024-02-06T19:43:58,457][INFO ][o.o.m.a.f.TransportForwardAction] [node_name_9200] deploy model done with state: DEPLOYED, model id: MJDyf40Bamv3vXxy0dmM [2024-02-06T19:43:58,458][INFO ][o.o.m.a.d.TransportDeployModelOnNodeAction] [node_name_9200] deploy model task done MZDyf40Bamv3vXxy2dlL [2024-02-06T19:44:03,324][INFO ][o.o.m.a.m.DeleteModelTransportAction] [node_name_9200] Model controller not deleted due to no model controller found for model: MJDyf40Bamv3vXxy0dmM [2024-02-06T19:44:03,330][INFO ][o.o.c.m.MetadataDeleteIndexService] [node_name_9200] [test-neural-basic-index/-s-uu4QBSZCHi5usrxZFBg] deleting index [2024-02-06T19:44:03,380][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [false] to [true] [2024-02-06T19:44:03,380][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [false] to [true] [2024-02-06T19:44:03,380][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.allow_registering_model_via_url] from [true] to [false] [2024-02-06T19:44:03,446][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [true] to [false] [2024-02-06T19:44:03,446][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [true] to [false] [2024-02-06T19:44:03,478][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.allow_registering_model_via_url] from [false] to [true] [2024-02-06T19:44:03,483][INFO ][o.o.i.b.HierarchyCircuitBreakerService] [node_name_9200] attempting to trigger G1GC due to high heap usage [1024635720] [2024-02-06T19:44:03,492][INFO ][o.o.i.b.HierarchyCircuitBreakerService] [node_name_9200] GC did bring memory usage down, before [1024635720], after [648996304], allocations [43], duration [9] [2024-02-06T19:44:03,494][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-sparse-text-and-field-index/PEKgkJ71Q0ykckkizlW09A] [2024-02-06T19:44:03,496][INFO ][o.o.c.m.MetadataCreateIndexService] [node_name_9200] [test-sparse-text-and-field-index] creating index, cause [api], templates [], shards [1]/[1] [2024-02-06T19:44:03,511][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-sparse-text-and-field-index/PEKgkJ71Q0ykckkizlW09A] [2024-02-06T19:44:03,569][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-sparse-text-and-field-index/PEKgkJ71Q0ykckkizlW09A] [2024-02-06T19:44:03,571][INFO ][o.o.c.m.MetadataMappingService] [node_name_9200] [test-sparse-text-and-field-index/PEKgkJ71Q0ykckkizlW09A] update_mapping [_doc] [2024-02-06T19:44:03,647][INFO ][o.o.m.m.MLModelManager ] [node_name_9200] create new model meta doc NJDyf40Bamv3vXxy7tkz for register model task M5Dyf40Bamv3vXxy7tke [2024-02-06T19:44:03,861][INFO ][stdout ] [node_name_9200] Downloading: 8% |████ | tokenize-idf-0915.zip Downloading: 16% |███████ | tokenize-idf-0915.zip Downloading: 25% |███████████ | tokenize-idf-0915.zip Downloading: 33% |██████████████ | tokenize-idf-0915.zip Downloading: 41% |█████████████████ | tokenize-idf-0915.zip Downloading: 50% |█████████████████████ | tokenize-idf-0915.zip Downloading: 58% |████████████████████████ | tokenize-idf-0915.zip Downloading: 67% |███████████████████████████ | tokenize-idf-0915.zip Downloading: 75% |███████████████████████████████ | tokenize-idf-0915.zip Downloading: 83% |██████████████████████████████████ | tokenize-idf-0915.zip Downloading: 92% |█████████████████████████████████████ | tokenize-idf-0915.zip Downloading: 100% |████████████████████████████████████████| tokenize-idf-0915.zip [2024-02-06T19:44:03,882][INFO ][o.o.m.m.MLModelManager ] [node_name_9200] Model registered successfully, model id: NJDyf40Bamv3vXxy7tkz, task id: M5Dyf40Bamv3vXxy7tke [2024-02-06T19:44:05,630][INFO ][o.o.m.a.d.TransportDeployModelAction] [node_name_9200] Will deploy model on these nodes: OAYF21-xRbu9dc9StXHecg [2024-02-06T19:44:05,654][ERROR][o.o.m.m.MLModelManager ] [node_name_9200] No controller is deployed because the model NJDyf40Bamv3vXxy7tkz is expected not having an enabled model controller. Please use the create model controller api to create one if this is unexpected. [2024-02-06T19:44:05,688][INFO ][o.o.m.e.a.t.SparseTokenizerModel] [node_name_9200] sparse tokenize Model NJDyf40Bamv3vXxy7tkz is successfully deployed [2024-02-06T19:44:05,689][INFO ][o.o.m.a.MLModelAutoReDeployer] [node_name_9200] No models needs to be auto redeployed! [2024-02-06T19:44:05,689][INFO ][o.o.m.a.f.TransportForwardAction] [node_name_9200] deploy model done with state: DEPLOYED, model id: NJDyf40Bamv3vXxy7tkz [2024-02-06T19:44:05,690][INFO ][o.o.m.a.d.TransportDeployModelOnNodeAction] [node_name_9200] deploy model task done NZDyf40Bamv3vXxy9dn- [2024-02-06T19:44:10,671][INFO ][o.o.m.a.m.DeleteModelTransportAction] [node_name_9200] Model controller not deleted due to no model controller found for model: NJDyf40Bamv3vXxy7tkz [2024-02-06T19:44:10,678][INFO ][o.o.c.m.MetadataDeleteIndexService] [node_name_9200] [test-sparse-text-and-field-index/PEKgkJ71Q0ykckkizlW09A] deleting index [2024-02-06T19:44:10,729][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [false] to [true] [2024-02-06T19:44:10,730][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [false] to [true] [2024-02-06T19:44:10,730][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.allow_registering_model_via_url] from [true] to [false] [2024-02-06T19:44:10,755][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [true] to [false] [2024-02-06T19:44:10,755][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [true] to [false] [2024-02-06T19:44:10,786][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.allow_registering_model_via_url] from [false] to [true] [2024-02-06T19:44:10,794][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-sparse-text-and-field-index/1U-W0IOHQay1KuREsBLG6A] [2024-02-06T19:44:10,797][INFO ][o.o.c.m.MetadataCreateIndexService] [node_name_9200] [test-sparse-text-and-field-index] creating index, cause [api], templates [], shards [1]/[1] [2024-02-06T19:44:10,810][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-sparse-text-and-field-index/1U-W0IOHQay1KuREsBLG6A] [2024-02-06T19:44:10,863][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-sparse-text-and-field-index/1U-W0IOHQay1KuREsBLG6A] [2024-02-06T19:44:10,865][INFO ][o.o.c.m.MetadataMappingService] [node_name_9200] [test-sparse-text-and-field-index/1U-W0IOHQay1KuREsBLG6A] update_mapping [_doc] [2024-02-06T19:44:10,939][INFO ][o.o.m.m.MLModelManager ] [node_name_9200] create new model meta doc OJDzf40Bamv3vXxyCtmt for register model task N5Dzf40Bamv3vXxyCtmY [2024-02-06T19:44:11,030][INFO ][stdout ] [node_name_9200] Downloading: 8% |████ | tokenize-idf-0915.zip Downloading: 16% |███████ | tokenize-idf-0915.zip Downloading: 25% |███████████ | tokenize-idf-0915.zip Downloading: 33% |██████████████ | tokenize-idf-0915.zip Downloading: 41% |█████████████████ | tokenize-idf-0915.zip Downloading: 50% |█████████████████████ | tokenize-idf-0915.zip Downloading: 58% |████████████████████████ | tokenize-idf-0915.zip Downloading: 67% |███████████████████████████ | tokenize-idf-0915.zip Downloading: 75% |███████████████████████████████ | tokenize-idf-0915.zip Downloading: 83% |██████████████████████████████████ | tokenize-idf-0915.zip Downloading: 92% |█████████████████████████████████████ | tokenize-idf-0915.zip Downloading: 100% |████████████████████████████████████████| tokenize-idf-0915.zip [2024-02-06T19:44:11,049][INFO ][o.o.m.m.MLModelManager ] [node_name_9200] Model registered successfully, model id: OJDzf40Bamv3vXxyCtmt, task id: N5Dzf40Bamv3vXxyCtmY [2024-02-06T19:44:12,919][INFO ][o.o.m.a.d.TransportDeployModelAction] [node_name_9200] Will deploy model on these nodes: OAYF21-xRbu9dc9StXHecg [2024-02-06T19:44:12,945][ERROR][o.o.m.m.MLModelManager ] [node_name_9200] No controller is deployed because the model OJDzf40Bamv3vXxyCtmt is expected not having an enabled model controller. Please use the create model controller api to create one if this is unexpected. [2024-02-06T19:44:12,980][INFO ][o.o.m.e.a.t.SparseTokenizerModel] [node_name_9200] sparse tokenize Model OJDzf40Bamv3vXxyCtmt is successfully deployed [2024-02-06T19:44:12,980][INFO ][o.o.m.a.MLModelAutoReDeployer] [node_name_9200] No models needs to be auto redeployed! [2024-02-06T19:44:12,981][INFO ][o.o.m.a.f.TransportForwardAction] [node_name_9200] deploy model done with state: DEPLOYED, model id: OJDzf40Bamv3vXxyCtmt [2024-02-06T19:44:12,981][INFO ][o.o.m.a.d.TransportDeployModelOnNodeAction] [node_name_9200] deploy model task done OZDzf40Bamv3vXxyEtl3 [2024-02-06T19:44:17,961][INFO ][o.o.m.a.m.DeleteModelTransportAction] [node_name_9200] Model controller not deleted due to no model controller found for model: OJDzf40Bamv3vXxyCtmt [2024-02-06T19:44:17,967][INFO ][o.o.c.m.MetadataDeleteIndexService] [node_name_9200] [test-sparse-text-and-field-index/1U-W0IOHQay1KuREsBLG6A] deleting index [2024-02-06T19:44:18,014][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [false] to [true] [2024-02-06T19:44:18,014][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [false] to [true] [2024-02-06T19:44:18,014][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.allow_registering_model_via_url] from [true] to [false] [2024-02-06T19:44:18,038][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [true] to [false] [2024-02-06T19:44:18,039][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [true] to [false] [2024-02-06T19:44:18,069][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.allow_registering_model_via_url] from [false] to [true] [2024-02-06T19:44:18,075][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-sparse-basic-index/IyYa1_yjR56Et4gm1rZsBw] [2024-02-06T19:44:18,077][INFO ][o.o.c.m.MetadataCreateIndexService] [node_name_9200] [test-sparse-basic-index] creating index, cause [api], templates [], shards [1]/[1] [2024-02-06T19:44:18,092][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-sparse-basic-index/IyYa1_yjR56Et4gm1rZsBw] [2024-02-06T19:44:18,212][INFO ][o.o.m.m.MLModelManager ] [node_name_9200] create new model meta doc PJDzf40Bamv3vXxyJ9kY for register model task O5Dzf40Bamv3vXxyJtn2 [2024-02-06T19:44:18,315][INFO ][stdout ] [node_name_9200] Downloading: 8% |████ | tokenize-idf-0915.zip Downloading: 16% |███████ | tokenize-idf-0915.zip Downloading: 25% |███████████ | tokenize-idf-0915.zip Downloading: 33% |██████████████ | tokenize-idf-0915.zip Downloading: 41% |█████████████████ | tokenize-idf-0915.zip Downloading: 50% |█████████████████████ | tokenize-idf-0915.zip Downloading: 58% |████████████████████████ | tokenize-idf-0915.zip Downloading: 67% |███████████████████████████ | tokenize-idf-0915.zip Downloading: 75% |███████████████████████████████ | tokenize-idf-0915.zip Downloading: 83% |██████████████████████████████████ | tokenize-idf-0915.zip Downloading: 92% |█████████████████████████████████████ | tokenize-idf-0915.zip Downloading: 100% |████████████████████████████████████████| tokenize-idf-0915.zip [2024-02-06T19:44:18,354][INFO ][o.o.m.m.MLModelManager ] [node_name_9200] Model registered successfully, model id: PJDzf40Bamv3vXxyJ9kY, task id: O5Dzf40Bamv3vXxyJtn2 [2024-02-06T19:44:20,195][INFO ][o.o.m.a.d.TransportDeployModelAction] [node_name_9200] Will deploy model on these nodes: OAYF21-xRbu9dc9StXHecg [2024-02-06T19:44:20,221][ERROR][o.o.m.m.MLModelManager ] [node_name_9200] No controller is deployed because the model PJDzf40Bamv3vXxyJ9kY is expected not having an enabled model controller. Please use the create model controller api to create one if this is unexpected. [2024-02-06T19:44:20,255][INFO ][o.o.m.e.a.t.SparseTokenizerModel] [node_name_9200] sparse tokenize Model PJDzf40Bamv3vXxyJ9kY is successfully deployed [2024-02-06T19:44:20,256][INFO ][o.o.m.a.MLModelAutoReDeployer] [node_name_9200] No models needs to be auto redeployed! [2024-02-06T19:44:20,256][INFO ][o.o.m.a.f.TransportForwardAction] [node_name_9200] deploy model done with state: DEPLOYED, model id: PJDzf40Bamv3vXxyJ9kY [2024-02-06T19:44:20,256][INFO ][o.o.m.a.d.TransportDeployModelOnNodeAction] [node_name_9200] deploy model task done PZDzf40Bamv3vXxyLtnj [2024-02-06T19:44:25,230][INFO ][o.o.m.a.m.DeleteModelTransportAction] [node_name_9200] Model controller not deleted due to no model controller found for model: PJDzf40Bamv3vXxyJ9kY [2024-02-06T19:44:25,237][INFO ][o.o.c.m.MetadataDeleteIndexService] [node_name_9200] [test-sparse-basic-index/IyYa1_yjR56Et4gm1rZsBw] deleting index [2024-02-06T19:44:25,289][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [false] to [true] [2024-02-06T19:44:25,289][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [false] to [true] [2024-02-06T19:44:25,289][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.allow_registering_model_via_url] from [true] to [false] [2024-02-06T19:44:25,314][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [true] to [false] [2024-02-06T19:44:25,314][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [true] to [false] [2024-02-06T19:44:25,343][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.allow_registering_model_via_url] from [false] to [true] [2024-02-06T19:44:25,350][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-sparse-basic-index/5wtzyBKgQDWDbRludhfpBg] [2024-02-06T19:44:25,352][INFO ][o.o.c.m.MetadataCreateIndexService] [node_name_9200] [test-sparse-basic-index] creating index, cause [api], templates [], shards [1]/[1] [2024-02-06T19:44:25,365][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-sparse-basic-index/5wtzyBKgQDWDbRludhfpBg] [2024-02-06T19:44:25,472][INFO ][o.o.m.m.MLModelManager ] [node_name_9200] create new model meta doc QJDzf40Bamv3vXxyQ9l0 for register model task P5Dzf40Bamv3vXxyQ9lg [2024-02-06T19:44:25,565][INFO ][stdout ] [node_name_9200] Downloading: 8% |████ | tokenize-idf-0915.zip Downloading: 16% |███████ | tokenize-idf-0915.zip Downloading: 25% |███████████ | tokenize-idf-0915.zip Downloading: 33% |██████████████ | tokenize-idf-0915.zip Downloading: 41% |█████████████████ | tokenize-idf-0915.zip Downloading: 50% |█████████████████████ | tokenize-idf-0915.zip Downloading: 58% |████████████████████████ | tokenize-idf-0915.zip Downloading: 67% |███████████████████████████ | tokenize-idf-0915.zip Downloading: 75% |███████████████████████████████ | tokenize-idf-0915.zip Downloading: 83% |██████████████████████████████████ | tokenize-idf-0915.zip Downloading: 92% |█████████████████████████████████████ | tokenize-idf-0915.zip Downloading: 100% |████████████████████████████████████████| tokenize-idf-0915.zip [2024-02-06T19:44:25,589][INFO ][o.o.m.m.MLModelManager ] [node_name_9200] Model registered successfully, model id: QJDzf40Bamv3vXxyQ9l0, task id: P5Dzf40Bamv3vXxyQ9lg [2024-02-06T19:44:27,455][INFO ][o.o.m.a.d.TransportDeployModelAction] [node_name_9200] Will deploy model on these nodes: OAYF21-xRbu9dc9StXHecg [2024-02-06T19:44:27,481][ERROR][o.o.m.m.MLModelManager ] [node_name_9200] No controller is deployed because the model QJDzf40Bamv3vXxyQ9l0 is expected not having an enabled model controller. Please use the create model controller api to create one if this is unexpected. [2024-02-06T19:44:27,529][INFO ][o.o.m.e.a.t.SparseTokenizerModel] [node_name_9200] sparse tokenize Model QJDzf40Bamv3vXxyQ9l0 is successfully deployed [2024-02-06T19:44:27,529][INFO ][o.o.m.a.MLModelAutoReDeployer] [node_name_9200] No models needs to be auto redeployed! [2024-02-06T19:44:27,529][INFO ][o.o.m.a.f.TransportForwardAction] [node_name_9200] deploy model done with state: DEPLOYED, model id: QJDzf40Bamv3vXxyQ9l0 [2024-02-06T19:44:27,530][INFO ][o.o.m.a.d.TransportDeployModelOnNodeAction] [node_name_9200] deploy model task done QZDzf40Bamv3vXxyS9k_ [2024-02-06T19:44:29,744][INFO ][o.o.m.c.MLSyncUpCron ] [node_name_9200] Refresh model state: {QJDzf40Bamv3vXxyQ9l0=DEPLOY_FAILED} [2024-02-06T19:44:32,491][INFO ][o.o.m.a.m.DeleteModelTransportAction] [node_name_9200] Model controller not deleted due to no model controller found for model: QJDzf40Bamv3vXxyQ9l0 [2024-02-06T19:44:32,498][INFO ][o.o.c.m.MetadataDeleteIndexService] [node_name_9200] [test-sparse-basic-index/5wtzyBKgQDWDbRludhfpBg] deleting index [2024-02-06T19:44:32,546][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [false] to [true] [2024-02-06T19:44:32,547][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [false] to [true] [2024-02-06T19:44:32,547][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.allow_registering_model_via_url] from [true] to [false] [2024-02-06T19:44:32,572][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [true] to [false] [2024-02-06T19:44:32,572][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [true] to [false] [2024-02-06T19:44:32,602][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.allow_registering_model_via_url] from [false] to [true] [2024-02-06T19:44:32,609][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-sparse-basic-index/DFumF3GhQkWm9Bg6xUSVuQ] [2024-02-06T19:44:32,611][INFO ][o.o.c.m.MetadataCreateIndexService] [node_name_9200] [test-sparse-basic-index] creating index, cause [api], templates [], shards [1]/[1] [2024-02-06T19:44:32,624][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-sparse-basic-index/DFumF3GhQkWm9Bg6xUSVuQ] [2024-02-06T19:44:32,739][INFO ][o.o.m.m.MLModelManager ] [node_name_9200] create new model meta doc RJDzf40Bamv3vXxyX9nX for register model task Q5Dzf40Bamv3vXxyX9nF [2024-02-06T19:44:32,831][INFO ][stdout ] [node_name_9200] Downloading: 0% |█ | tokenize-idf-0915.zip Downloading: 1% |█ | tokenize-idf-0915.zip Downloading: 2% |█ | tokenize-idf-0915.zip Downloading: 3% |██ | tokenize-idf-0915.zip Downloading: 4% |██ | tokenize-idf-0915.zip Downloading: 5% |███ | tokenize-idf-0915.zip Downloading: 6% |███ | tokenize-idf-0915.zip Downloading: 7% |███ | tokenize-idf-0915.zip Downloading: 16% |███████ | tokenize-idf-0915.zip Downloading: 24% |██████████ | tokenize-idf-0915.zip Downloading: 33% |██████████████ | tokenize-idf-0915.zip Downloading: 41% |█████████████████ | tokenize-idf-0915.zip Downloading: 49% |████████████████████ | tokenize-idf-0915.zip Downloading: 58% |████████████████████████ | tokenize-idf-0915.zip Downloading: 66% |███████████████████████████ | tokenize-idf-0915.zip Downloading: 75% |███████████████████████████████ | tokenize-idf-0915.zip Downloading: 83% |██████████████████████████████████ | tokenize-idf-0915.zip Downloading: 91% |█████████████████████████████████████ | tokenize-idf-0915.zip Downloading: 100% |████████████████████████████████████████| tokenize-idf-0915.zip [2024-02-06T19:44:32,851][INFO ][o.o.m.m.MLModelManager ] [node_name_9200] Model registered successfully, model id: RJDzf40Bamv3vXxyX9nX, task id: Q5Dzf40Bamv3vXxyX9nF [2024-02-06T19:44:34,722][INFO ][o.o.m.a.d.TransportDeployModelAction] [node_name_9200] Will deploy model on these nodes: OAYF21-xRbu9dc9StXHecg [2024-02-06T19:44:34,747][ERROR][o.o.m.m.MLModelManager ] [node_name_9200] No controller is deployed because the model RJDzf40Bamv3vXxyX9nX is expected not having an enabled model controller. Please use the create model controller api to create one if this is unexpected. [2024-02-06T19:44:34,780][INFO ][o.o.m.e.a.t.SparseTokenizerModel] [node_name_9200] sparse tokenize Model RJDzf40Bamv3vXxyX9nX is successfully deployed [2024-02-06T19:44:34,781][INFO ][o.o.m.a.MLModelAutoReDeployer] [node_name_9200] No models needs to be auto redeployed! [2024-02-06T19:44:34,781][INFO ][o.o.m.a.f.TransportForwardAction] [node_name_9200] deploy model done with state: DEPLOYED, model id: RJDzf40Bamv3vXxyX9nX [2024-02-06T19:44:34,781][INFO ][o.o.m.a.d.TransportDeployModelOnNodeAction] [node_name_9200] deploy model task done RZDzf40Bamv3vXxyZ9mj [2024-02-06T19:44:39,759][INFO ][o.o.m.a.m.DeleteModelTransportAction] [node_name_9200] Model controller not deleted due to no model controller found for model: RJDzf40Bamv3vXxyX9nX [2024-02-06T19:44:39,766][INFO ][o.o.c.m.MetadataDeleteIndexService] [node_name_9200] [test-sparse-basic-index/DFumF3GhQkWm9Bg6xUSVuQ] deleting index [2024-02-06T19:44:39,818][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [false] to [true] [2024-02-06T19:44:39,818][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [false] to [true] [2024-02-06T19:44:39,819][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.allow_registering_model_via_url] from [true] to [false] [2024-02-06T19:44:39,844][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [true] to [false] [2024-02-06T19:44:39,844][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [true] to [false] [2024-02-06T19:44:39,874][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.allow_registering_model_via_url] from [false] to [true] [2024-02-06T19:44:39,882][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-sparse-multi-field-index/WWVw8MJ7SWqQm5GNdkjnGQ] [2024-02-06T19:44:39,884][INFO ][o.o.c.m.MetadataCreateIndexService] [node_name_9200] [test-sparse-multi-field-index] creating index, cause [api], templates [], shards [1]/[1] [2024-02-06T19:44:39,898][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-sparse-multi-field-index/WWVw8MJ7SWqQm5GNdkjnGQ] [2024-02-06T19:44:40,044][INFO ][o.o.m.m.MLModelManager ] [node_name_9200] create new model meta doc SJDzf40Bamv3vXxyfNlG for register model task R5Dzf40Bamv3vXxyfNkw [2024-02-06T19:44:40,128][INFO ][stdout ] [node_name_9200] Downloading: 8% |████ | tokenize-idf-0915.zip Downloading: 16% |███████ | tokenize-idf-0915.zip Downloading: 25% |███████████ | tokenize-idf-0915.zip Downloading: 33% |██████████████ | tokenize-idf-0915.zip Downloading: 41% |█████████████████ | tokenize-idf-0915.zip Downloading: 50% |█████████████████████ | tokenize-idf-0915.zip Downloading: 58% |████████████████████████ | tokenize-idf-0915.zip Downloading: 67% |███████████████████████████ | tokenize-idf-0915.zip Downloading: 75% |███████████████████████████████ | tokenize-idf-0915.zip Downloading: 83% |██████████████████████████████████ | tokenize-idf-0915.zip Downloading: 92% |█████████████████████████████████████ | tokenize-idf-0915.zip Downloading: 100% |████████████████████████████████████████| tokenize-idf-0915.zip [2024-02-06T19:44:40,147][INFO ][o.o.m.m.MLModelManager ] [node_name_9200] Model registered successfully, model id: SJDzf40Bamv3vXxyfNlG, task id: R5Dzf40Bamv3vXxyfNkw [2024-02-06T19:44:41,999][INFO ][o.o.m.a.d.TransportDeployModelAction] [node_name_9200] Will deploy model on these nodes: OAYF21-xRbu9dc9StXHecg [2024-02-06T19:44:42,023][ERROR][o.o.m.m.MLModelManager ] [node_name_9200] No controller is deployed because the model SJDzf40Bamv3vXxyfNlG is expected not having an enabled model controller. Please use the create model controller api to create one if this is unexpected. [2024-02-06T19:44:42,055][INFO ][o.o.m.e.a.t.SparseTokenizerModel] [node_name_9200] sparse tokenize Model SJDzf40Bamv3vXxyfNlG is successfully deployed [2024-02-06T19:44:42,056][INFO ][o.o.m.a.MLModelAutoReDeployer] [node_name_9200] No models needs to be auto redeployed! [2024-02-06T19:44:42,056][INFO ][o.o.m.a.f.TransportForwardAction] [node_name_9200] deploy model done with state: DEPLOYED, model id: SJDzf40Bamv3vXxyfNlG [2024-02-06T19:44:42,056][INFO ][o.o.m.a.d.TransportDeployModelOnNodeAction] [node_name_9200] deploy model task done SZDzf40Bamv3vXxyhNkQ [2024-02-06T19:44:47,033][INFO ][o.o.m.a.m.DeleteModelTransportAction] [node_name_9200] Model controller not deleted due to no model controller found for model: SJDzf40Bamv3vXxyfNlG [2024-02-06T19:44:47,040][INFO ][o.o.c.m.MetadataDeleteIndexService] [node_name_9200] [test-sparse-multi-field-index/WWVw8MJ7SWqQm5GNdkjnGQ] deleting index [2024-02-06T19:44:47,085][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [false] to [true] [2024-02-06T19:44:47,085][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [false] to [true] [2024-02-06T19:44:47,086][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.allow_registering_model_via_url] from [true] to [false] [2024-02-06T19:44:47,108][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [true] to [false] [2024-02-06T19:44:47,108][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [true] to [false] [2024-02-06T19:44:47,136][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.allow_registering_model_via_url] from [false] to [true] [2024-02-06T19:44:47,143][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-sparse-basic-index/RTxiBEcJRYGo8GghYOofWA] [2024-02-06T19:44:47,145][INFO ][o.o.c.m.MetadataCreateIndexService] [node_name_9200] [test-sparse-basic-index] creating index, cause [api], templates [], shards [1]/[1] [2024-02-06T19:44:47,159][INFO ][o.o.p.PluginsService ] [node_name_9200] PluginService:onIndexModule index:[test-sparse-basic-index/RTxiBEcJRYGo8GghYOofWA] [2024-02-06T19:44:47,266][INFO ][o.o.m.m.MLModelManager ] [node_name_9200] create new model meta doc TJDzf40Bamv3vXxymNmW for register model task S5Dzf40Bamv3vXxymNmC [2024-02-06T19:44:47,372][INFO ][stdout ] [node_name_9200] Downloading: 0% |█ | tokenize-idf-0915.zip Downloading: 1% |█ | tokenize-idf-0915.zip Downloading: 2% |█ | tokenize-idf-0915.zip Downloading: 3% |██ | tokenize-idf-0915.zip Downloading: 4% |██ | tokenize-idf-0915.zip Downloading: 5% |███ | tokenize-idf-0915.zip Downloading: 6% |███ | tokenize-idf-0915.zip Downloading: 7% |███ | tokenize-idf-0915.zip Downloading: 16% |███████ | tokenize-idf-0915.zip Downloading: 24% |██████████ | tokenize-idf-0915.zip Downloading: 33% |██████████████ | tokenize-idf-0915.zip Downloading: 41% |█████████████████ | tokenize-idf-0915.zip Downloading: 49% |████████████████████ | tokenize-idf-0915.zip Downloading: 58% |████████████████████████ | tokenize-idf-0915.zip Downloading: 66% |███████████████████████████ | tokenize-idf-0915.zip Downloading: 75% |███████████████████████████████ | tokenize-idf-0915.zip Downloading: 83% |██████████████████████████████████ | tokenize-idf-0915.zip Downloading: 91% |█████████████████████████████████████ | tokenize-idf-0915.zip Downloading: 100% |████████████████████████████████████████| tokenize-idf-0915.zip [2024-02-06T19:44:47,391][INFO ][o.o.m.m.MLModelManager ] [node_name_9200] Model registered successfully, model id: TJDzf40Bamv3vXxymNmW, task id: S5Dzf40Bamv3vXxymNmC [2024-02-06T19:44:49,248][INFO ][o.o.m.a.d.TransportDeployModelAction] [node_name_9200] Will deploy model on these nodes: OAYF21-xRbu9dc9StXHecg [2024-02-06T19:44:49,301][ERROR][o.o.m.m.MLModelManager ] [node_name_9200] No controller is deployed because the model TJDzf40Bamv3vXxymNmW is expected not having an enabled model controller. Please use the create model controller api to create one if this is unexpected. [2024-02-06T19:44:49,334][INFO ][o.o.m.e.a.t.SparseTokenizerModel] [node_name_9200] sparse tokenize Model TJDzf40Bamv3vXxymNmW is successfully deployed [2024-02-06T19:44:49,334][INFO ][o.o.m.a.MLModelAutoReDeployer] [node_name_9200] No models needs to be auto redeployed! [2024-02-06T19:44:49,335][INFO ][o.o.m.a.f.TransportForwardAction] [node_name_9200] deploy model done with state: DEPLOYED, model id: TJDzf40Bamv3vXxymNmW [2024-02-06T19:44:49,335][INFO ][o.o.m.a.d.TransportDeployModelOnNodeAction] [node_name_9200] deploy model task done TZDzf40Bamv3vXxyoNlg [2024-02-06T19:44:54,281][INFO ][o.o.m.a.m.DeleteModelTransportAction] [node_name_9200] Model controller not deleted due to no model controller found for model: TJDzf40Bamv3vXxymNmW [2024-02-06T19:44:54,287][INFO ][o.o.c.m.MetadataDeleteIndexService] [node_name_9200] [test-sparse-basic-index/RTxiBEcJRYGo8GghYOofWA] deleting index [2024-02-06T19:44:54,340][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [false] to [true] [2024-02-06T19:44:54,340][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.only_run_on_ml_node] from [false] to [true] [2024-02-06T19:44:54,341][INFO ][o.o.c.s.ClusterSettings ] [node_name_9200] updating [plugins.ml_commons.allow_registering_model_via_url] from [true] to [false]