Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature][PostgreSQL-jdbc] Supports GEOMETRY data type for PostgreSQL… #4673

Merged
merged 8 commits into from
May 15, 2023

Conversation

zhilinli123
Copy link
Contributor

@zhilinli123 zhilinli123 commented Apr 25, 2023

Purpose of this pull request

issues:#4652

Check list

@EricJoy2048
Copy link
Member

Please reference https://github.com/apache/incubator-seatunnel/pull/4590/files and finished the document update.

@zhilinli123
Copy link
Contributor Author

zhilinli123 commented Apr 26, 2023

Please reference https://github.com/apache/incubator-seatunnel/pull/4590/files and finished the document update.
After the current merger, I will complete the pg connector documentation @EricJoy2048
pg connector documentation has been applied for

driver = "org.postgresql.Driver"
user = "test"
password = "test"
query = "select * from spatial_data"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you test all supported field types?
A rigorous test can refer to the following information:

  1. The input and output of the test should include all supported data types.
  2. The input data should be deterministic and the values of each row and column can be verified in the output.
  3. In the output table, the values of each column in each row of data should be verified to ensure that there are no type conversion errors (precision loss, etc.) or missing values (a non null value in the input becomes null in the output) during the synchronization process.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think so, because currently pr supports geometric types, and if I wanted to test all types, I think I could open one and create a separate pr header such as Pg e2e to test it
@EricJoy2048

Copy link
Contributor

@ic4y ic4y left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. When obtaining the GEOMETRY type directly through (AbstractJdbcRowConverter.toInternal), you can only get a string of numbers that are not understandable.

https://github.com/apache/incubator-seatunnel/blob/5430ca9621ffdd7f7ec183235eed29764ecd6205/seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/internal/converter/AbstractJdbcRowConverter.java#L53-L55

Therefore, it is necessary to introduce the PostGIS dependency, and then obtain the PGgeometry instance through the rs.getObject method and then call toString to get an understandable string.

 <dependency>
            <groupId>net.postgis</groupId>
            <artifactId>postgis-jdbc</artifactId>
            <version>2.5.0</version>
  </dependency>
  1. Spatial data types not only include GEOMETRY but also GEOGRAPHY, which also needs to be supported.

@zhilinli123
Copy link
Contributor Author

  1. When obtaining the GEOMETRY type directly through (AbstractJdbcRowConverter.toInternal), you can only get a string of numbers that are not understandable.

https://github.com/apache/incubator-seatunnel/blob/5430ca9621ffdd7f7ec183235eed29764ecd6205/seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/internal/converter/AbstractJdbcRowConverter.java#L53-L55

Therefore, it is necessary to introduce the PostGIS dependency, and then obtain the PGgeometry instance through the rs.getObject method and then call toString to get an understandable string.

 <dependency>
            <groupId>net.postgis</groupId>
            <artifactId>postgis-jdbc</artifactId>
            <version>2.5.0</version>
  </dependency>
  1. Spatial data types not only include GEOMETRY but also GEOGRAPHY, which also needs to be supported.

Done

@@ -0,0 +1,45 @@
#
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@zhilinli123
Copy link
Contributor Author

RUN CI thanks

ic4y
ic4y previously approved these changes May 4, 2023
Copy link
Contributor

@ic4y ic4y left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And the postgis dependency needs to be added to the seatunnel-dist pom.xml

switch (seaTunnelDataType.getSqlType()) {
case STRING:
if (metaDataColumnType.equals(PG_GEOMETRY)) {
fields[fieldIndex] = ((PGgeometry) columnObj).getValue();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggest using the method rs.getObject(resultSetIndex).toString(), because this way it won't throw an error when the postgis package is not imported (you will get a string of numbers), and you can also get the desired data when the postgis package is imported.
like

                case STRING:
                    String columnTypeName = metaData.getColumnTypeName(resultSetIndex).toUpperCase(Locale.ROOT);
                    if (columnTypeName.equals("GEOGRAPHY") || columnTypeName.equals("GEOMETRY")) {
                        fields[fieldIndex] = rs.getObject(resultSetIndex) == null ? null : rs.getObject(resultSetIndex).toString();
                        break;
                    }
                    fields[fieldIndex] = rs.getString(resultSetIndex);
                    break;

@zhilinli123 zhilinli123 closed this May 4, 2023
@zhilinli123 zhilinli123 reopened this May 4, 2023
Copy link
Contributor

@ic4y ic4y left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rename JdbcPostgisIT.java to JdbcPostgresIT.java

@zhilinli123
Copy link
Contributor Author

zhilinli123 commented May 9, 2023

@EricJoy2048 @ic4y help review thanks

@zhilinli123
Copy link
Contributor Author

@EricJoy2048 @hailin0 Please check it out. Thank you

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants