Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature][Jdbc] Support read multiple tables #5581

Merged
merged 42 commits into from
Oct 25, 2023

Conversation

hailin0
Copy link
Member

@hailin0 hailin0 commented Sep 28, 2023

Purpose of this pull request

close #5482

Support read table list for jdbc connector

Does this PR introduce any user-facing change?

Compatible with older versions config:

source {
    Jdbc {
        url = "jdbc:mysql://127.0.0.1:3306/test"
        driver = "com.mysql.cj.jdbc.Driver"
        connection_check_timeout_sec = 100
        user = "root"
        password = "root@123"

        query = "select * from test_1"
        partition_column = "id"
        partition_num = 100
    }
}

Support new config of table path:

source {
    Jdbc {
        url = "jdbc:mysql://127.0.0.1:3306/test"
        driver = "com.mysql.cj.jdbc.Driver"
        connection_check_timeout_sec = 100
        user = "root"
        password = "root@123"

        table_path = "test.test_1"

        #split.size = 8096
        #split.even-distribution.factor.upper-bound = 100
        #split.even-distribution.factor.lower-bound = 0.05
        #split.sample-sharding.threshold = 1000
        #split.inverse-sampling.rate = 1000
    }
}

Support read table list config:

source {
    Jdbc {
        url = "jdbc:mysql://127.0.0.1:3306/test"
        driver = "com.mysql.cj.jdbc.Driver"
        connection_check_timeout_sec = 100
        user = "root"
        password = "root@123"

        table_list = [
            {
                table_path = "test.test_1"
            },
            {
                table_path = "test.test_2"
            },
            {
                query = "select id, name from test.test_3 where id > 1000"
                table_path = "test.test_3"
            }
        ]
        #split.size = 8096
        #split.even-distribution.factor.upper-bound = 100
        #split.even-distribution.factor.lower-bound = 0.05
        #split.sample-sharding.threshold = 1000
        #split.inverse-sampling.rate = 1000
    }
}

How was this patch tested?

Check list

@hailin0 hailin0 changed the title [Feature][Jdbc] Support read table list through table-path/query-sql [Feature][Jdbc] Support read multiple tables Oct 10, 2023
@hailin0
Copy link
Member Author

hailin0 commented Oct 11, 2023

Waiting for #5595#5604#5620

@hailin0 hailin0 marked this pull request as ready for review October 20, 2023 07:04
@hailin0 hailin0 linked an issue Oct 20, 2023 that may be closed by this pull request
3 tasks
@hailin0 hailin0 closed this Oct 24, 2023
@hailin0 hailin0 reopened this Oct 24, 2023
ruanwenjun
ruanwenjun previously approved these changes Oct 24, 2023
Copy link
Member

@ruanwenjun ruanwenjun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@@ -47,7 +50,7 @@ public class MySqlCatalog extends AbstractJdbcCatalog {
private static final MysqlDataTypeConvertor DATA_TYPE_CONVERTOR = new MysqlDataTypeConvertor();

private static final String SELECT_COLUMNS_SQL_TEMPLATE =
"SELECT * FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_SCHEMA = '%s' AND TABLE_NAME ='%s'";
"SELECT * FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_SCHEMA = '%s' AND TABLE_NAME ='%s' ORDER BY ORDINAL_POSITION ASC";
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

orderby

if (rawValue instanceof List) {
return ((List<?>) rawValue)
.stream()
.map(value -> convertValue(convertToJsonString(value), clazz))
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fix bug

table_list = [{xxx=xxx},{xxx=xxx}]

Copy link
Member

@ruanwenjun ruanwenjun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@hailin0 hailin0 merged commit 33fa8ff into apache:dev Oct 25, 2023
@hailin0 hailin0 mentioned this pull request Oct 30, 2023
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants