Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

mysql bigint field inserto to es , find error data #964

Open
LockGit opened this issue Aug 11, 2017 · 3 comments
Open

mysql bigint field inserto to es , find error data #964

LockGit opened this issue Aug 11, 2017 · 3 comments

Comments

@LockGit
Copy link

LockGit commented Aug 11, 2017

example:
CREATE TABLE product_info (
item_id bigint(20) NOT NULL DEFAULT '0' ,
);
whene item_id too long , use jdbc insert es from mysql will error;

also, we can test on sense
`
PUT lock/test/5
{
"name":"del_5",
"age":20,
"num":"-8629845578222121431",
"bignum":-8629845578222121431
}


GET lock/test/_search
{
"query": {
"term": {
"name": {
"value": "del_5"
}
}
}
}

result:
{
"took": 1,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"failed": 0
},
"hits": {
"total": 1,
"max_score": 0.15342641,
"hits": [
{
"_index": "lock",
"_type": "test",
"_id": "5",
"_score": 0.15342641,
"_source": {
"name": "del_5",
"age": 20,
"num": "-8629845578222121431",
"bignum": -8629845578222121000
}
}
]
}
}
`
we can see bignum change to 8629845578222121000 , rather than 8629845578222121431

it's precision question .

how too fix this question when use elasticsearch-jdbc tool index data to elasticsearch from mysql.
don't change mysql table structure !

@LockGit
Copy link
Author

LockGit commented Aug 11, 2017

find a way ,
can change SQL to below when use elasticsearch-jdbc tool index data to elasticsearch from mysql , don't change mysql table structure !

elasticsearch-jdbc-2.3.3.1/bin/mysql-xxx.sh:

"statement": "select convert(bignum,CHAR) as bignum from xxx where xxxxx";

change to string type index to es

@jprante
Copy link
Owner

jprante commented Aug 11, 2017

The fact is that database and JDBC know about big integer, Java knows big integer, but ES does not (and Javascript/JSON is silent about this).
JDBC importer tries to convert to JSON numbers by using Java, and ES tries again to index a numeric value from the JSON number.
JDBC importer tries by default to downgrade to numeric values to retain numbers. JDBC importer uses scale and precision defaults, they can be changed, but result must fit into Java double.
If you index strings, all numeric properties are lost.

@LockGit
Copy link
Author

LockGit commented Aug 11, 2017

@jprante thx, if i index long type, -8629845578222121431 will change to -8629845578222121000 , but correct data is -8629845578222121431,the -8629845578222121000 is an error data. so too long type number can only change index Strings.
you have good idea ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants