Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug] [Connector-hive-v2] can't get table from hive3.x by the seatunnel 2.3.1-release #4451

Closed
3 tasks done
zengxiaosong opened this issue Mar 30, 2023 · 15 comments · Fixed by #5261
Closed
3 tasks done

[Bug] [Connector-hive-v2] can't get table from hive3.x by the seatunnel 2.3.1-release #4451

zengxiaosong opened this issue Mar 30, 2023 · 15 comments · Fixed by #5261
Labels
bug stale

Comments

@zengxiaosong
Copy link

Search before asking

  • I had searched in the issues and found no similar issues.

What happened

there are a problem when i start with hive 3.x . i can't get the table from hive although Kerberos authentication successful and connection is opened.

SeaTunnel Version

seatunnel2.3.1 zeta

SeaTunnel Config

standalone 

env {
  execution.parallelism = 1
  job.mode = "BATCH"
  job.name = "mysql_hive_test"
}

source {

  Hive {
    table_name = "seatunnel_test.test_person_yxf_part2"
    metastore_uri = "thrift://ambari-31.snowleopard.cn:9083,thrift://ambari-32.snowleopard.cn:9083"
    kerberos_principal = "hive/[email protected]"
    kerberos_keytab_path = "/etc/security/keytabs/hive.service.keytab"
    hdfs_site_path = "/etc/hadoop/conf/hdfs-site.xml"
  }
}

sink {
    Console {}
}

Running Command

./bin/seatunnel.sh --config ./config/hive-mysql-tbds.conf -e local

Error Exception

(1):  
 WARN  hive.metastore - set_ugi() not successful, Likely cause: new client talking to old server. Continuing without it.
org.apache.thrift.transport.TTransportException: null
        at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) ~[hive-exec-2.3.9.jar:2.3.9]
        at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) ~[hive-exec-2.3.9.jar:2.3.9]
        at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:380) ~[hive-exec-2.3.9.jar:2.3.9]
        at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:230) ~[hive-exec-2.3.9.jar:2.3.9]
        at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77) ~[hive-exec-2.3.9.jar:2.3.9]
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_set_ugi(ThriftHiveMetastore.java:4247) ~[hive-exec-2.3.9.jar:2.3.9]
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.set_ugi(ThriftHiveMetastore.java:4233) ~[hive-exec-2.3.9.jar:2.3.9]
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:496) [hive-exec-2.3.9.jar:2.3.9]
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:245) [hive-exec-2.3.9.jar:2.3.9]
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:128) [hive-exec-2.3.9.jar:2.3.9]
        at org.apache.seatunnel.connectors.seatunnel.hive.utils.HiveMetaStoreProxy.<init>(HiveMetaStoreProxy.java:58) [connector-hive-2.3.1.jar:2.3.1]
        at org.apache.seatunnel.connectors.seatunnel.hive.utils.HiveMetaStoreProxy.getInstance(HiveMetaStoreProxy.java:74) [connector-hive-2.3.1.jar:2.3.1]
        at org.apache.seatunnel.connectors.seatunnel.hive.config.HiveConfig.getTableInfo(HiveConfig.java:59) [connector-hive-2.3.1.jar:2.3.1]
        at org.apache.seatunnel.connectors.seatunnel.hive.source.HiveSource.prepare(HiveSource.java:123) [connector-hive-2.3.1.jar:2.3.1]
        at org.apache.seatunnel.engine.core.parse.ConnectorInstanceLoader.loadSourceInstance(ConnectorInstanceLoader.java:64) [seatunnel-starter.jar:2.3.1]
        at org.apache.seatunnel.engine.core.parse.JobConfigParser.sampleAnalyze(JobConfigParser.java:371) [seatunnel-starter.jar:2.3.1]
        at org.apache.seatunnel.engine.core.parse.JobConfigParser.parse(JobConfigParser.java:132) [seatunnel-starter.jar:2.3.1]
        at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:112) [seatunnel-starter.jar:2.3.1]
        at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.getLogicalDag(JobExecutionEnvironment.java:155) [seatunnel-starter.jar:2.3.1]
        at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.execute(JobExecutionEnvironment.java:147) [seatunnel-starter.jar:2.3.1]
        at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:140) [seatunnel-starter.jar:2.3.1]
        at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40) [seatunnel-starter.jar:2.3.1]
        at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34) [seatunnel-starter.jar:2.3.1]
 ERROR org.apache.seatunnel.core.starter.SeaTunnel - Exception StackTrace:org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed
        at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:181)
        at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
        at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)


(2)：
Caused by: org.apache.seatunnel.connectors.seatunnel.hive.exception.HiveConnectorException: ErrorCode:[HIVE-03], ErrorDescription:[Get hive table information from hive metastore service failed] - Get table [seatunnel_test.test_person_yxf_part2] information failed
        at org.apache.seatunnel.connectors.seatunnel.hive.utils.HiveMetaStoreProxy.getTable(HiveMetaStoreProxy.java:87)
        at org.apache.seatunnel.connectors.seatunnel.hive.config.HiveConfig.getTableInfo(HiveConfig.java:60)
        at org.apache.seatunnel.connectors.seatunnel.hive.source.HiveSource.prepare(HiveSource.java:123)
        at org.apache.seatunnel.engine.core.parse.ConnectorInstanceLoader.loadSourceInstance(ConnectorInstanceLoader.java:64)
        at org.apache.seatunnel.engine.core.parse.JobConfigParser.sampleAnalyze(JobConfigParser.java:371)
        at org.apache.seatunnel.engine.core.parse.JobConfigParser.parse(JobConfigParser.java:132)
        at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:112)
        at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.getLogicalDag(JobExecutionEnvironment.java:155)
        at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.execute(JobExecutionEnvironment.java:147)
        at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:140)
        ... 2 more

Flink or Spark Version

No response

Java or Scala Version

java8

Screenshots

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct
@zengxiaosong
Copy link
Author

the hive version is 3.1.0

@zengxiaosong
Copy link
Author

all logs:

JAVA_OPTS: -Dhazelcast.client.config=/opt/apache-seatunnel-incubating-2.3.1/config/hazelcast-client.yaml -Dseatunnel.config=/opt/apache-seatunnel-incubating-2.3.1/config/seatunnel.yaml -Dhazelcast.config=/opt/apache-seatunnel-incubating-2.3.1/config/hazelcast.yaml -Dlog4j2.configurationFile=/opt/apache-seatunnel-incubating-2.3.1/config/log4j2_client.properties -Dseatunnel.logs.path=/opt/apache-seatunnel-incubating-2.3.1/logs -Dseatunnel.logs.file_name=seatunnel-starter-client-1680141163115 -Xms128m -Xmx512m -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:/tmp/seatunnel-4431960592068079776 -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=32 -XX:GCLogFileSize=64m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/data/seatunnel/
2023-03-30 09:52:43,922 WARN org.apache.seatunnel.core.starter.seatunnel.args.ClientCommandArgs$MasterTypeValidator -

-e and --deploy-mode will be deprecated in 2.3.1, please use -m and --master instead of it

三月 30, 2023 9:52:43 上午 com.hazelcast.internal.config.AbstractConfigLocator
信息: Loading configuration '/opt/apache-seatunnel-incubating-2.3.1/config/seatunnel.yaml' from System property 'seatunnel.config'
三月 30, 2023 9:52:43 上午 com.hazelcast.internal.config.AbstractConfigLocator
信息: Using configuration file at /opt/apache-seatunnel-incubating-2.3.1/config/seatunnel.yaml
三月 30, 2023 9:52:43 上午 org.apache.seatunnel.engine.common.config.SeaTunnelConfig
信息: seatunnel.home is /opt/apache-seatunnel-incubating-2.3.1
三月 30, 2023 9:52:44 上午 com.hazelcast.internal.config.AbstractConfigLocator
信息: Loading configuration '/opt/apache-seatunnel-incubating-2.3.1/config/hazelcast.yaml' from System property 'hazelcast.config'
三月 30, 2023 9:52:44 上午 com.hazelcast.internal.config.AbstractConfigLocator
信息: Using configuration file at /opt/apache-seatunnel-incubating-2.3.1/config/hazelcast.yaml
三月 30, 2023 9:52:44 上午 com.hazelcast.internal.config.AbstractConfigLocator
信息: Loading configuration '/opt/apache-seatunnel-incubating-2.3.1/config/seatunnel.yaml' from System property 'seatunnel.config'
三月 30, 2023 9:52:44 上午 com.hazelcast.internal.config.AbstractConfigLocator
信息: Using configuration file at /opt/apache-seatunnel-incubating-2.3.1/config/seatunnel.yaml
三月 30, 2023 9:52:44 上午 com.hazelcast.internal.config.AbstractConfigLocator
信息: Loading configuration '/opt/apache-seatunnel-incubating-2.3.1/config/hazelcast.yaml' from System property 'hazelcast.config'
三月 30, 2023 9:52:44 上午 com.hazelcast.internal.config.AbstractConfigLocator
信息: Using configuration file at /opt/apache-seatunnel-incubating-2.3.1/config/hazelcast.yaml
2023-03-30 09:52:44,417 WARN com.hazelcast.instance.AddressPicker - [LOCAL] [seatunnel-823219] [5.1] You configured your member address as host name. Please be aware of that your dns can be spoofed. Make sure that your dns configurations are correct.
2023-03-30 09:52:44,417 INFO com.hazelcast.instance.AddressPicker - [LOCAL] [seatunnel-823219] [5.1] Resolving domain name 'localhost' to address(es): [127.0.0.1, 0:0:0:0:0:0:0:1]
2023-03-30 09:52:44,418 INFO com.hazelcast.instance.AddressPicker - [LOCAL] [seatunnel-823219] [5.1] Interfaces is disabled, trying to pick one address from TCP-IP config addresses: [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1]
2023-03-30 09:52:44,441 INFO org.apache.seatunnel.engine.server.SeaTunnelServer - SeaTunnel server start...
2023-03-30 09:52:44,443 INFO com.hazelcast.system - [localhost]:5801 [seatunnel-823219] [5.1] Based on Hazelcast IMDG version: 5.1.0 (20220228 - 21f20e7)
2023-03-30 09:52:44,443 INFO com.hazelcast.system - [localhost]:5801 [seatunnel-823219] [5.1] Cluster name: seatunnel-823219
2023-03-30 09:52:44,443 INFO com.hazelcast.system - [localhost]:5801 [seatunnel-823219] [5.1]

/ | | | | |
\ --. ___ __ _ | | _ _ _ __ _ __ ___ | | --. \ / _ \ / ` | | | | | | || ' \ | ' \ / _ | |
/_/ /| __/| (| | | | | || || | | || | | || /| |
_/ _| _,| _/ _,||| |||| || _||_|

2023-03-30 09:52:44,443 INFO com.hazelcast.system - [localhost]:5801 [seatunnel-823219] [5.1] Copyright © 2021-2022 The Apache Software Foundation. Apache SeaTunnel, SeaTunnel, and its feather logo are trademarks of The Apache Software Foundation.
2023-03-30 09:52:44,443 INFO com.hazelcast.system - [localhost]:5801 [seatunnel-823219] [5.1] Integrity Checker is disabled. Fail-fast on corrupted executables will not be performed.
To enable integrity checker do one of the following:

  • Change member config using Java API: config.setIntegrityCheckerEnabled(true);
  • Change XML/YAML configuration property: Set hazelcast.integrity-checker.enabled to true
  • Add system property: -Dhz.integritychecker.enabled=true (for Hazelcast embedded, works only when loading config via Config.load)
  • Add environment variable: HZ_INTEGRITYCHECKER_ENABLED=true (recommended when running container image. For Hazelcast embedded, works only when loading config via Config.load)
    2023-03-30 09:52:44,446 INFO com.hazelcast.system - [localhost]:5801 [seatunnel-823219] [5.1] The Jet engine is disabled.
    To enable the Jet engine on the members, do one of the following:
  • Change member config using Java API: config.getJetConfig().setEnabled(true)
  • Change XML/YAML configuration property: Set hazelcast.jet.enabled to true
  • Add system property: -Dhz.jet.enabled=true (for Hazelcast embedded, works only when loading config via Config.load)
  • Add environment variable: HZ_JET_ENABLED=true (recommended when running container image. For Hazelcast embedded, works only when loading config via Config.load)
    2023-03-30 09:52:44,877 INFO com.hazelcast.system.security - [localhost]:5801 [seatunnel-823219] [5.1] Enable DEBUG/FINE log level for log category com.hazelcast.system.security or use -Dhazelcast.security.recommendations system property to see 🔒 security recommendations and the status of current config.
    2023-03-30 09:52:44,941 INFO com.hazelcast.instance.impl.Node - [localhost]:5801 [seatunnel-823219] [5.1] Using TCP/IP discovery
    2023-03-30 09:52:44,942 WARN com.hazelcast.cp.CPSubsystem - [localhost]:5801 [seatunnel-823219] [5.1] CP Subsystem is not enabled. CP data structures will operate in UNSAFE mode! Please note that UNSAFE mode will not provide strong consistency guarantees.
    三月 30, 2023 9:52:45 上午 com.hazelcast.internal.config.AbstractConfigLocator
    信息: Loading configuration '/opt/apache-seatunnel-incubating-2.3.1/config/seatunnel.yaml' from System property 'seatunnel.config'
    三月 30, 2023 9:52:45 上午 com.hazelcast.internal.config.AbstractConfigLocator
    信息: Using configuration file at /opt/apache-seatunnel-incubating-2.3.1/config/seatunnel.yaml
    三月 30, 2023 9:52:45 上午 com.hazelcast.internal.config.AbstractConfigLocator
    信息: Loading configuration '/opt/apache-seatunnel-incubating-2.3.1/config/hazelcast.yaml' from System property 'hazelcast.config'
    三月 30, 2023 9:52:45 上午 com.hazelcast.internal.config.AbstractConfigLocator
    信息: Using configuration file at /opt/apache-seatunnel-incubating-2.3.1/config/hazelcast.yaml
    2023-03-30 09:52:45,128 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [localhost]:5801 [seatunnel-823219] [5.1] Created new BusWork : 1730129134
    2023-03-30 09:52:45,139 WARN org.apache.seatunnel.engine.server.service.slot.DefaultSlotService - failed send heartbeat to resource manager, will retry later. this address: [localhost]:5801
    2023-03-30 09:52:45,142 INFO org.apache.seatunnel.engine.server.CoordinatorService - [localhost]:5801 [seatunnel-823219] [5.1]
 CoordinatorService Thread Pool Status

activeCount : 0
corePoolSize : 0
maximumPoolSize : 2147483647
poolSize : 0
completedTaskCount : 0
taskCount : 0

2023-03-30 09:52:45,203 INFO com.hazelcast.internal.diagnostics.Diagnostics - [localhost]:5801 [seatunnel-823219] [5.1] Diagnostics disabled. To enable add -Dhazelcast.diagnostics.enabled=true to the JVM arguments.
2023-03-30 09:52:45,210 INFO com.hazelcast.core.LifecycleService - [localhost]:5801 [seatunnel-823219] [5.1] [localhost]:5801 is STARTING
2023-03-30 09:52:45,232 INFO com.hazelcast.internal.cluster.ClusterService - [localhost]:5801 [seatunnel-823219] [5.1]

Members {size:1, ver:1} [
Member [localhost]:5801 - a5d08ac1-a938-4c8a-be7f-4fa206977ab8 this
]

2023-03-30 09:52:45,241 INFO com.hazelcast.core.LifecycleService - [localhost]:5801 [seatunnel-823219] [5.1] [localhost]:5801 is STARTED
三月 30, 2023 9:52:45 上午 com.hazelcast.internal.config.AbstractConfigLocator
信息: Loading configuration '/opt/apache-seatunnel-incubating-2.3.1/config/hazelcast-client.yaml' from System property 'hazelcast.client.config'
三月 30, 2023 9:52:45 上午 com.hazelcast.internal.config.AbstractConfigLocator
信息: Using configuration file at /opt/apache-seatunnel-incubating-2.3.1/config/hazelcast-client.yaml
2023-03-30 09:52:45,270 INFO com.hazelcast.internal.partition.impl.PartitionStateManager - [localhost]:5801 [seatunnel-823219] [5.1] Initializing cluster partition table arrangement...
2023-03-30 09:52:45,474 INFO com.hazelcast.client.impl.spi.ClientInvocationService - hz.client_1 [seatunnel-823219] [5.1] Running with 2 response threads, dynamic=true
2023-03-30 09:52:45,485 INFO com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel-823219] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is STARTING
2023-03-30 09:52:45,486 INFO com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel-823219] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is STARTED
2023-03-30 09:52:45,492 INFO com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel-823219] [5.1] Trying to connect to cluster: seatunnel-823219
2023-03-30 09:52:45,493 INFO com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel-823219] [5.1] Trying to connect to [localhost]:5801
2023-03-30 09:52:45,512 INFO com.hazelcast.client.impl.protocol.task.AuthenticationMessageTask - [localhost]:5801 [seatunnel-823219] [5.1] Received auth from Connection[id=1, /127.0.0.1:5801->/127.0.0.1:47926, qualifier=null, endpoint=[127.0.0.1]:47926, remoteUuid=bea0232d-1875-4bd5-8055-cc54d4d61773, alive=true, connectionType=JVM, planeIndex=-1], successfully authenticated, clientUuid: bea0232d-1875-4bd5-8055-cc54d4d61773, client name: hz.client_1, client version: 5.1
2023-03-30 09:52:45,514 INFO com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel-823219] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is CLIENT_CONNECTED
2023-03-30 09:52:45,514 INFO com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel-823219] [5.1] Authenticated with server [localhost]:5801:a5d08ac1-a938-4c8a-be7f-4fa206977ab8, server version: 5.1, local address: /127.0.0.1:47926
2023-03-30 09:52:45,515 INFO com.hazelcast.internal.diagnostics.Diagnostics - hz.client_1 [seatunnel-823219] [5.1] Diagnostics disabled. To enable add -Dhazelcast.diagnostics.enabled=true to the JVM arguments.
2023-03-30 09:52:45,521 INFO com.hazelcast.client.impl.spi.ClientClusterService - hz.client_1 [seatunnel-823219] [5.1]

Members [1] {
Member [localhost]:5801 - a5d08ac1-a938-4c8a-be7f-4fa206977ab8
}

2023-03-30 09:52:45,540 INFO com.hazelcast.client.impl.statistics.ClientStatisticsService - Client statistics is enabled with period 5 seconds.
2023-03-30 09:52:45,576 INFO org.apache.seatunnel.engine.client.job.JobExecutionEnvironment - add common jar in plugins :[file:/opt/apache-seatunnel-incubating-2.3.1/plugins/jdbc/lib/mysql-connector-j-8.0.31.jar]
2023-03-30 09:52:45,581 INFO org.apache.seatunnel.core.starter.utils.ConfigBuilder - Loading config file from path: ./config/hive-mysql-tbds.conf
2023-03-30 09:52:45,627 INFO org.apache.seatunnel.core.starter.utils.ConfigShadeUtils - Load config shade spi: [base64]
2023-03-30 09:52:45,669 INFO org.apache.seatunnel.core.starter.utils.ConfigBuilder - Parsed config file: {
"env" : {
"execution.parallelism" : 1,
"job.mode" : "BATCH",
"job.name" : "mysql_hive_test"
},
"source" : [
{
"hdfs_site_path" : "/etc/hadoop/conf/hdfs-site.xml",
"kerberos_principal" : "hive/[email protected]",
"metastore_uri" : "thrift://ambari-31.snowleopard.cn:9083,thrift://ambari-32.snowleopard.cn:9083",
"plugin_name" : "Hive",
"table_name" : "seatunnel_test.test_person_yxf_part2",
"kerberos_keytab_path" : "/etc/security/keytabs/hive.service.keytab"
}
],
"sink" : [
{
"plugin_name" : "Console"
}
]
}

2023-03-30 09:52:45,686 INFO org.apache.seatunnel.core.starter.utils.ConfigBuilder - Loading config file from path: ./config/hive-mysql-tbds.conf
2023-03-30 09:52:45,689 INFO org.apache.seatunnel.core.starter.utils.ConfigBuilder - Parsed config file: {
"env" : {
"execution.parallelism" : 1,
"job.mode" : "BATCH",
"job.name" : "mysql_hive_test"
},
"source" : [
{
"hdfs_site_path" : "/etc/hadoop/conf/hdfs-site.xml",
"kerberos_principal" : "hive/[email protected]",
"metastore_uri" : "thrift://ambari-31.snowleopard.cn:9083,thrift://ambari-32.snowleopard.cn:9083",
"plugin_name" : "Hive",
"table_name" : "seatunnel_test.test_person_yxf_part2",
"kerberos_keytab_path" : "/etc/security/keytabs/hive.service.keytab"
}
],
"sink" : [
{
"plugin_name" : "Console"
}
]
}

2023-03-30 09:52:45,697 INFO org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Load SeaTunnelSource Plugin from /opt/apache-seatunnel-incubating-2.3.1/connectors/seatunnel
2023-03-30 09:52:45,707 INFO org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Discovery plugin jar: Hive at: file:/opt/apache-seatunnel-incubating-2.3.1/connectors/seatunnel/connector-hive-2.3.1.jar
2023-03-30 09:52:45,715 INFO org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Load plugin: PluginIdentifier{engineType='seatunnel', pluginType='source', pluginName='Hive'} from path: file:/opt/apache-seatunnel-incubating-2.3.1/connectors/seatunnel/connector-hive-2.3.1.jar use classloader: org.apache.seatunnel.engine.common.loader.SeaTunnelChildFirstClassLoader
2023-03-30 09:52:45,745 INFO org.apache.hadoop.hive.conf.HiveConf - Found configuration file null
2023-03-30 09:52:46,100 WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2023-03-30 09:52:46,116 INFO org.apache.seatunnel.connectors.seatunnel.file.sink.util.FileSystemUtils - Start Kerberos authentication using principal hive/[email protected] and keytab /etc/security/keytabs/hive.service.keytab
2023-03-30 09:52:46,319 INFO org.apache.hadoop.security.UserGroupInformation - Login successful for user hive/[email protected] using keytab file /etc/security/keytabs/hive.service.keytab
2023-03-30 09:52:46,319 INFO org.apache.seatunnel.connectors.seatunnel.file.sink.util.FileSystemUtils - Kerberos authentication successful
2023-03-30 09:52:46,335 INFO hive.metastore - Trying to connect to metastore with URI thrift://ambari-31.snowleopard.cn:9083
2023-03-30 09:52:46,353 INFO hive.metastore - Opened a connection to metastore, current connections: 1
2023-03-30 09:52:46,431 WARN hive.metastore - set_ugi() not successful, Likely cause: new client talking to old server. Continuing without it.
org.apache.thrift.transport.TTransportException: null
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) ~[hive-exec-2.3.9.jar:2.3.9]
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) ~[hive-exec-2.3.9.jar:2.3.9]
at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:380) ~[hive-exec-2.3.9.jar:2.3.9]
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:230) ~[hive-exec-2.3.9.jar:2.3.9]
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77) ~[hive-exec-2.3.9.jar:2.3.9]
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_set_ugi(ThriftHiveMetastore.java:4247) ~[hive-exec-2.3.9.jar:2.3.9]
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.set_ugi(ThriftHiveMetastore.java:4233) ~[hive-exec-2.3.9.jar:2.3.9]
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:496) [hive-exec-2.3.9.jar:2.3.9]
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:245) [hive-exec-2.3.9.jar:2.3.9]
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:128) [hive-exec-2.3.9.jar:2.3.9]
at org.apache.seatunnel.connectors.seatunnel.hive.utils.HiveMetaStoreProxy.(HiveMetaStoreProxy.java:58) [connector-hive-2.3.1.jar:2.3.1]
at org.apache.seatunnel.connectors.seatunnel.hive.utils.HiveMetaStoreProxy.getInstance(HiveMetaStoreProxy.java:74) [connector-hive-2.3.1.jar:2.3.1]
at org.apache.seatunnel.connectors.seatunnel.hive.config.HiveConfig.getTableInfo(HiveConfig.java:59) [connector-hive-2.3.1.jar:2.3.1]
at org.apache.seatunnel.connectors.seatunnel.hive.source.HiveSource.prepare(HiveSource.java:123) [connector-hive-2.3.1.jar:2.3.1]
at org.apache.seatunnel.engine.core.parse.ConnectorInstanceLoader.loadSourceInstance(ConnectorInstanceLoader.java:64) [seatunnel-starter.jar:2.3.1]
at org.apache.seatunnel.engine.core.parse.JobConfigParser.sampleAnalyze(JobConfigParser.java:371) [seatunnel-starter.jar:2.3.1]
at org.apache.seatunnel.engine.core.parse.JobConfigParser.parse(JobConfigParser.java:132) [seatunnel-starter.jar:2.3.1]
at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:112) [seatunnel-starter.jar:2.3.1]
at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.getLogicalDag(JobExecutionEnvironment.java:155) [seatunnel-starter.jar:2.3.1]
at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.execute(JobExecutionEnvironment.java:147) [seatunnel-starter.jar:2.3.1]
at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:140) [seatunnel-starter.jar:2.3.1]
at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40) [seatunnel-starter.jar:2.3.1]
at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34) [seatunnel-starter.jar:2.3.1]
2023-03-30 09:52:46,435 INFO hive.metastore - Connected to metastore.
2023-03-30 09:52:46,441 INFO com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel-823219] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is SHUTTING_DOWN
2023-03-30 09:52:46,447 INFO com.hazelcast.internal.server.tcp.TcpServerConnection - [localhost]:5801 [seatunnel-823219] [5.1] Connection[id=1, /127.0.0.1:5801->/127.0.0.1:47926, qualifier=null, endpoint=[127.0.0.1]:47926, remoteUuid=bea0232d-1875-4bd5-8055-cc54d4d61773, alive=false, connectionType=JVM, planeIndex=-1] closed. Reason: Connection closed by the other side
2023-03-30 09:52:46,448 INFO com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel-823219] [5.1] Removed connection to endpoint: [localhost]:5801:a5d08ac1-a938-4c8a-be7f-4fa206977ab8, connection: ClientConnection{alive=false, connectionId=1, channel=NioChannel{/127.0.0.1:47926->localhost/127.0.0.1:5801}, remoteAddress=[localhost]:5801, lastReadTime=2023-03-30 09:52:45.570, lastWriteTime=2023-03-30 09:52:45.568, closedTime=2023-03-30 09:52:46.446, connected server version=5.1}
2023-03-30 09:52:46,448 INFO com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel-823219] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is CLIENT_DISCONNECTED
2023-03-30 09:52:46,450 INFO com.hazelcast.client.impl.ClientEndpointManager - [localhost]:5801 [seatunnel-823219] [5.1] Destroying ClientEndpoint{connection=Connection[id=1, /127.0.0.1:5801->/127.0.0.1:47926, qualifier=null, endpoint=[127.0.0.1]:47926, remoteUuid=bea0232d-1875-4bd5-8055-cc54d4d61773, alive=false, connectionType=JVM, planeIndex=-1], clientUuid=bea0232d-1875-4bd5-8055-cc54d4d61773, clientName=hz.client_1, authenticated=true, clientVersion=5.1, creationTime=1680141165508, latest clientAttributes=lastStatisticsCollectionTime=1680141165540,enterprise=false,clientType=JVM,clientVersion=5.1,clusterConnectionTimestamp=1680141165500,clientAddress=127.0.0.1,clientName=hz.client_1,credentials.principal=null,os.committedVirtualMemorySize=17780465664,os.freePhysicalMemorySize=43084734464,os.freeSwapSpaceSize=0,os.maxFileDescriptorCount=65536,os.openFileDescriptorCount=60,os.processCpuTime=7910000000,os.systemLoadAverage=0.64,os.totalPhysicalMemorySize=67383083008,os.totalSwapSpaceSize=0,runtime.availableProcessors=32,runtime.freeMemory=173637280,runtime.maxMemory=477626368,runtime.totalMemory=218103808,runtime.uptime=2173,runtime.usedMemory=44466528, labels=[]}
2023-03-30 09:52:46,451 INFO com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel-823219] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is SHUTDOWN
2023-03-30 09:52:46,451 INFO org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand - Closed SeaTunnel client......
2023-03-30 09:52:46,452 INFO com.hazelcast.core.LifecycleService - [localhost]:5801 [seatunnel-823219] [5.1] [localhost]:5801 is SHUTTING_DOWN
2023-03-30 09:52:46,455 INFO com.hazelcast.internal.partition.impl.MigrationManager - [localhost]:5801 [seatunnel-823219] [5.1] Shutdown request of Member [localhost]:5801 - a5d08ac1-a938-4c8a-be7f-4fa206977ab8 this is handled
2023-03-30 09:52:46,462 INFO com.hazelcast.instance.impl.Node - [localhost]:5801 [seatunnel-823219] [5.1] Shutting down connection manager...
2023-03-30 09:52:46,463 INFO com.hazelcast.instance.impl.Node - [localhost]:5801 [seatunnel-823219] [5.1] Shutting down node engine...
2023-03-30 09:52:46,472 INFO org.apache.seatunnel.engine.server.SeaTunnelServer - master node check interrupted
2023-03-30 09:52:49,599 INFO com.hazelcast.instance.impl.NodeExtension - [localhost]:5801 [seatunnel-823219] [5.1] Destroying node NodeExtension.
2023-03-30 09:52:49,600 INFO com.hazelcast.instance.impl.Node - [localhost]:5801 [seatunnel-823219] [5.1] Hazelcast Shutdown is completed in 3146 ms.
2023-03-30 09:52:49,600 INFO com.hazelcast.core.LifecycleService - [localhost]:5801 [seatunnel-823219] [5.1] [localhost]:5801 is SHUTDOWN
2023-03-30 09:52:49,600 INFO org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand - Closed HazelcastInstance ......
2023-03-30 09:52:49,600 ERROR org.apache.seatunnel.core.starter.SeaTunnel -

===============================================================================

2023-03-30 09:52:49,600 ERROR org.apache.seatunnel.core.starter.SeaTunnel - Fatal Error,

2023-03-30 09:52:49,600 ERROR org.apache.seatunnel.core.starter.SeaTunnel - Please submit bug report in https://github.com/apache/incubator-seatunnel/issues

2023-03-30 09:52:49,601 ERROR org.apache.seatunnel.core.starter.SeaTunnel - Reason:SeaTunnel job executed failed

2023-03-30 09:52:49,602 ERROR org.apache.seatunnel.core.starter.SeaTunnel - Exception StackTrace:org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed
at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:181)
at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by: org.apache.seatunnel.connectors.seatunnel.hive.exception.HiveConnectorException: ErrorCode:[HIVE-03], ErrorDescription:[Get hive table information from hive metastore service failed] - Get table [seatunnel_test.test_person_yxf_part2] information failed
at org.apache.seatunnel.connectors.seatunnel.hive.utils.HiveMetaStoreProxy.getTable(HiveMetaStoreProxy.java:87)
at org.apache.seatunnel.connectors.seatunnel.hive.config.HiveConfig.getTableInfo(HiveConfig.java:60)
at org.apache.seatunnel.connectors.seatunnel.hive.source.HiveSource.prepare(HiveSource.java:123)
at org.apache.seatunnel.engine.core.parse.ConnectorInstanceLoader.loadSourceInstance(ConnectorInstanceLoader.java:64)
at org.apache.seatunnel.engine.core.parse.JobConfigParser.sampleAnalyze(JobConfigParser.java:371)
at org.apache.seatunnel.engine.core.parse.JobConfigParser.parse(JobConfigParser.java:132)
at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:112)
at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.getLogicalDag(JobExecutionEnvironment.java:155)
at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.execute(JobExecutionEnvironment.java:147)
at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:140)
... 2 more
Caused by: org.apache.thrift.transport.TTransportException
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table(ThriftHiveMetastore.java:1514)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table(ThriftHiveMetastore.java:1500)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:1346)
at org.apache.seatunnel.connectors.seatunnel.hive.utils.HiveMetaStoreProxy.getTable(HiveMetaStoreProxy.java:83)
... 11 more

2023-03-30 09:52:49,602 ERROR org.apache.seatunnel.core.starter.SeaTunnel -

Exception in thread "main" org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed
at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:181)
at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by: org.apache.seatunnel.connectors.seatunnel.hive.exception.HiveConnectorException: ErrorCode:[HIVE-03], ErrorDescription:[Get hive table information from hive metastore service failed] - Get table [seatunnel_test.test_person_yxf_part2] information failed
at org.apache.seatunnel.connectors.seatunnel.hive.utils.HiveMetaStoreProxy.getTable(HiveMetaStoreProxy.java:87)
at org.apache.seatunnel.connectors.seatunnel.hive.config.HiveConfig.getTableInfo(HiveConfig.java:60)
at org.apache.seatunnel.connectors.seatunnel.hive.source.HiveSource.prepare(HiveSource.java:123)
at org.apache.seatunnel.engine.core.parse.ConnectorInstanceLoader.loadSourceInstance(ConnectorInstanceLoader.java:64)
at org.apache.seatunnel.engine.core.parse.JobConfigParser.sampleAnalyze(JobConfigParser.java:371)
at org.apache.seatunnel.engine.core.parse.JobConfigParser.parse(JobConfigParser.java:132)
at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:112)
at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.getLogicalDag(JobExecutionEnvironment.java:155)
at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.execute(JobExecutionEnvironment.java:147)
at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:140)
... 2 more
Caused by: org.apache.thrift.transport.TTransportException
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table(ThriftHiveMetastore.java:1514)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table(ThriftHiveMetastore.java:1500)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:1346)
at org.apache.seatunnel.connectors.seatunnel.hive.utils.HiveMetaStoreProxy.getTable(HiveMetaStoreProxy.java:83)
... 11 more

@chenchurong
Copy link

check u hadoop-client & hive-client version is same with u server version

@github-actions
Copy link

This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.

@github-actions github-actions bot added the stale label Apr 30, 2023
@SinyoWong
Copy link

SinyoWong commented May 10, 2023 •
edited
Loading

same problem;
my versions:
hive version is 3.1.0;
hadoop version is 3.1.1;

@Bingz2
Copy link
Contributor

Bingz2 commented May 16, 2023

same problem;
my versions:
hive version is 1.1.0;
hadoop version is 2.6.0;
seatunnel used hive-exec-2.3.9.jar

@zhengqiangtan
Copy link

same problem;
my versions:
Hadoop 3.1.2
hive-3.1.3-hadoop3.1-1.0.0
spark-2.4.8-hadoop3.2-1.0.5

seatunnel use hive-exec-3.1.3.jar

@SinyoWong
Copy link

Hi, big bro, we are searching for help!
@TyrantLucifer

@itbtt
Copy link

itbtt commented Jun 8, 2023

1

@williamxulab
Copy link

same problem,
hive version:3.1.3
hadoop version:3.3.4

@TyrantLucifer
Copy link
Member

TyrantLucifer commented Jun 15, 2023 •
edited
Loading

Could you please offer more information of your hive metastore server authentication?

@TyrantLucifer
Copy link
Member

TyrantLucifer commented Jun 15, 2023 •
edited
Loading

I will aim to this problem submit a patch, hey guys please wait my news

@shuwei8626
Copy link

same problem;
my versions:
hive version is 2.1.1;
hadoop version is 3.0.0;

seatunnel used hive-exec-2.3.9.jar

@EricJoy2048
Copy link
Member

@XiaoJiang521 PTAL

@Hisoka-X Hisoka-X linked a pull request Aug 24, 2023 that will close this issue
[Hotfix][Connector-V2][Hive] fix the bug that hive-site.xml can not be injected #5261
Merged
5 tasks
@SinyoWong SinyoWong mentioned this issue Aug 30, 2023
Merged
5 tasks
@Light-Towers
Copy link

请问这个问题是被解决了吗？ 可以支持 Hive3.x 了吗？

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug stale
Projects
None yet
Milestone
No milestone