[Bug] [Elasticsearch source connector] Read Es 7.x ERROR

[981433814]

Search before asking

• I had searched in the issues and found no similar issues.

What happened

The current version of es is 7.9.3. The following error will be displayed when using Elasticsearch source connector. Is Elasticsearch 7.x supported now

SeaTunnel Version

2.1.0

SeaTunnel Config

source {
elasticsearch {
    hosts = ["xxxxx:9200"]
    index = "xxxxxx"
    es.net.http.header.Authorization = "xxxxxxxxx"
    es.read.field.include = "xxxx, xxxx"
    result_table_name = "xxxxx"
}
}

Running Command

$SEATUNNEL_HOME/bin/start-seatunnel-spark.sh --master local[4] --deploy-mode client --config $SEATUNNEL_HOME/config/$CONF_NAME.conf

Error Exception

Exception in thread "main" org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Cannot detect ES version - typically this happens if the network/Elasticsearch cluster is not accessible or when targeting a WAN/Cloud instance without the proper setting 'es.nodes.wan.only'
        at org.elasticsearch.hadoop.rest.InitializationUtils.discoverClusterInfo(InitializationUtils.java:344)
        at org.elasticsearch.spark.sql.ElasticsearchRelation.cfg$lzycompute(DefaultSource.scala:225)
        at org.elasticsearch.spark.sql.ElasticsearchRelation.cfg(DefaultSource.scala:223)
        at org.elasticsearch.spark.sql.ElasticsearchRelation.lazySchema$lzycompute(DefaultSource.scala:229)
        at org.elasticsearch.spark.sql.ElasticsearchRelation.lazySchema(DefaultSource.scala:229)
        at org.elasticsearch.spark.sql.ElasticsearchRelation$$anonfun$schema$1.apply(DefaultSource.scala:233)
        at org.elasticsearch.spark.sql.ElasticsearchRelation$$anonfun$schema$1.apply(DefaultSource.scala:233)
        at scala.Option.getOrElse(Option.scala:121)
        at org.elasticsearch.spark.sql.ElasticsearchRelation.schema(DefaultSource.scala:233)
        at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:417)
        at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:242)
        at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:230)
        at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:197)
        at org.apache.seatunnel.spark.source.Elasticsearch.getData(Elasticsearch.scala:62)
        at org.apache.seatunnel.spark.source.Elasticsearch.getData(Elasticsearch.scala:28)
        at org.apache.seatunnel.spark.batch.SparkBatchExecution.registerInputTempView(SparkBatchExecution.java:54)
        at org.apache.seatunnel.spark.batch.SparkBatchExecution.lambda$start$0(SparkBatchExecution.java:95)
        at java.util.ArrayList.forEach(ArrayList.java:1257)
        at org.apache.seatunnel.spark.batch.SparkBatchExecution.start(SparkBatchExecution.java:95)
        at org.apache.seatunnel.Seatunnel.entryPoint(Seatunnel.java:107)
        at org.apache.seatunnel.Seatunnel.run(Seatunnel.java:65)
        at org.apache.seatunnel.SeatunnelSpark.main(SeatunnelSpark.java:29)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
        at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:845)
        at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
        at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
        at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
        at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Unsupported/Unknown Elasticsearch version 7.9.3
        at org.elasticsearch.hadoop.util.EsMajorVersion.parse(EsMajorVersion.java:87)
        at org.elasticsearch.hadoop.rest.RestClient.mainInfo(RestClient.java:703)
        at org.elasticsearch.hadoop.rest.InitializationUtils.discoverClusterInfo(InitializationUtils.java:334)
        ... 33 more

Flink or Spark Version

No response

Java or Scala Version

No response

Screenshots

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[laglangyue]

I think you use the Cnnector-v1,which is based on ES-hadoop, you can add spark conf and try again.
es.nodes.wan.only = true
https://www.elastic.co/guide/en/elasticsearch/hadoop/master/configuration.html
If not work，you should use Connector-v2.

[981433814]

<elasticsearch-spark.version>6.8.3</elasticsearch-spark.version>
This version of the package can only recognize version 6.x of es

我想你使用的是基于ES-hadoop的Cnnector-v1，你可以添加spark conf再试一次。 es.nodes.wan.only = true https://www.elastic.co/guide/en/elasticsearch/hadoop/master/configuration.html 如果不行，你应该使用 Connector-v2。

[laglangyue]

<elasticsearch-spark.version>6.8.3</elasticsearch-spark.version> This version of the package can only recognize version 6.x of es

我想你使用的是基于ES-hadoop的Cnnector-v1，你可以添加spark conf再试一次。 es.nodes.wan.only = true https://www.elastic.co/guide/en/elasticsearch/hadoop/master/configuration.html 如果不行，你应该使用 Connector-v2。

so you need to use Connector-v2,please upgrade your version.
if you want to enhance Connector-v1 for es7, may not be accepted by member

[laglangyue]

Hello, I think the issue can be closed, please close the issue in time.
@981433814
Look forward to your next isuse, PR will be better!

[981433814]

I mentioned pr, could you please help me review it #3716