Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] [Maxcompute] Failed to parse some maxcompute type #3893

Closed
3 tasks done
stdnt-xiao opened this issue Jan 7, 2023 · 2 comments · Fixed by #3894
Closed
3 tasks done

[Bug] [Maxcompute] Failed to parse some maxcompute type #3893

stdnt-xiao opened this issue Jan 7, 2023 · 2 comments · Fixed by #3894

Comments

@stdnt-xiao
Copy link
Contributor

Search before asking

  • I had searched in the issues and found no similar issues.

What happened

Use maxcompute connector,I got prints like [B@79338251... when parse data of String type.
as follows:
WechatIMG355

SeaTunnel Version

dev & 2.3.1

SeaTunnel Config

#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#    http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
######
###### This config file is a demonstration of streaming processing in seatunnel config
######

######
###### Sample of maxcompute data type
######
#  DROP TABLE IF EXISTS fake_source;
#
#  CREATE TABLE IF NOT EXISTS fake_source(c1 TINYINT,c2 SMALLINT,c3 INT,c4 BIGINT,c5 FLOAT ,c6 DOUBLE
#  ,c7 VARCHAR(10),c8 CHAR(10),c9 STRING,c10 DATE,c11 DATETIME ,c12 TIMESTAMP ,c13 BOOLEAN,c14 BINARY
#  ,c15 MAP<STRING,STRING>,c16 ARRAY<INT>,c17 STRUCT<s1:STRING,s2:INT,s3:ARRAY<FLOAT>>);
#
#  INSERT INTO fake_source(c1, c2, c3, c4, c5, c6, c7,c8,c9,c10,c11,c12,c13,c14,c15,c16,c17) VALUES (
#  CAST(-128 AS  TINYINT ),CAST(-32768 AS SMALLINT ) ,0,10000000000000,0.01,0.0000000000000001
#  ,CAST("varchar" as VARCHAR(10)),CAST("char" as CHAR(10)),"hello0",CAST("2022-12-31" as DATE )
#  ,CAST("2022-12-31 23:59:59" as DATETIME  ),CAST("2022-12-31 23:59:59.999" as TIMESTAMP ),FALSE,CAST("bytes" AS BINARY )
#  ,MAP("int",1,"str","hello"),ARRAY("11","22"),named_struct("s1","s1","s2",100,"s3",array(1.1, 2.2)));
#
#  SELECT * FROM fake_source;
#
#  DROP TABLE IF EXISTS fake_sink;
#
#  CREATE TABLE IF NOT EXISTS fake_sink LIKE fake_source;
#
#  SELECT * FROM fake_sink;
#

env {
  # You can set spark configuration here
  # see available properties defined by spark: https://spark.apache.org/docs/latest/configuration.html#available-properties
  #job.mode = BATCH
  job.name = "SeaTunnel"
  spark.executor.instances = 1
  spark.executor.cores = 1
  spark.executor.memory = "1g"
  spark.master = local
}

source {
  # This is a example source plugin **only for test and demonstrate the feature source plugin**
  Maxcompute {
    accessId="<your access id>"
    accesskey="<your access Key>"
    endpoint="<http://service.odps.aliyun.com/api>"
    project="<your project>"
    table_name="<your table name>"
    #partition_spec="<your partition spec>"
    #split_row = 10000
  }

  # If you would like to get more information about how to configure seatunnel and see full list of source plugins,
  # please go to https://seatunnel.apache.org/docs/category/source-v2
}

transform {
  sql {
    source_table_name = "fake"
    sql = "select * from fake"
  }

  # If you would like to get more information about how to configure seatunnel and see full list of transform plugins,
  # please go to https://seatunnel.apache.org/docs/category/transform
}

sink {
  Maxcompute {
    accessId="<your access id>"
    accesskey="<your access Key>"
    endpoint="<http://service.odps.aliyun.com/api>"
    project="<your project>"
    table_name="<your table name>"
    #partition_spec="<your partition spec>"
    #overwrite = false
  }

  # If you would like to get more information about how to configure seatunnel and see full list of sink plugins,
  # please go to https://seatunnel.apache.org/docs/category/sink-v2
}

Running Command

bin/start-seatunnel-spark-connector-v2.sh -e client -m local[1] --config ./config/v2.batch.config.template

Error Exception

Use maxcompute source,I got print like [B@79338251... when parse data of String type.

Flink or Spark Version

spark default

Java or Scala Version

java 8

Screenshots

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

@github-actions
Copy link

github-actions bot commented Feb 7, 2023

This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.

@github-actions github-actions bot added the stale label Feb 7, 2023
@github-actions
Copy link

This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.

TyrantLucifer pushed a commit to stdnt-xiao/incubator-seatunnel that referenced this issue Feb 15, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant