[Feature] Thrift support for realtime and batch ingestion #3418

du00cs · 2016-09-01T10:17:55Z

In our company thrift is used heavily, thrift serialized objects are stored to HDFS as SequenceFile and bytes to Kafka. I try to make it easier for thrift data to be ingested into druid without ETL in simple case.
Thanks to the flexible JSONSpec, the Implementation is rather simple, deserialize thrift bytes to thrift object and then serialize it to json. And then parse it with JSONSpec.

Is anybody using thrift too? Hope this useful for somebody like me.

fjy · 2016-09-13T20:19:56Z

...contrib/thrift-extensions/src/main/java/io/druid/data/input/thrift/ThriftInputRowParser.java

+  {
+    this.protocol = protocol;
+
+    if (protocol == null || protocol.equals("compact")) {


can we use Jackson to deserialize the protocol into a TCompactProtocol or TBinaryProtocol? this iwll avoid the if/else check

Any example? I am not familiar with Jackson and don't know how to make it after some search.

@fjy Sorry, but how to do that? Any tips?

fjy · 2016-11-08T22:54:42Z

@du00cs there's some overlap between this and https://github.com/druid-io/druid/pull/3633/files

Can we come to an agreement around which version to use?

du00cs · 2016-11-09T03:32:05Z

@fjy Protocol aware is awesome and I missed thread safe problem of deserializer. Something should be checked more carefully like how and when a thrift class is added to classpath. Implement a TBase to Map is too complicated and error prone. I should argue that using JSONPath will make it much simpler and more expressive and be consistent with what JsonSpec can do.

justsmoke · 2016-11-09T08:37:04Z

@du00cs I just had a look at your code, specify thrift jar in runtime is really good, and serialize to simple json can solve the problem of nested field, so I think your solution is better. Also, I noticed that your batch ingestion is based on SequenceFileInputFormat, can you help add support for LzoThriftBlockInputFormat (this is implemented in project elephant-bird which is also quite popular)

du00cs · 2016-11-10T01:43:37Z

@justsmoke I must admit that your protocol aware and taking thread safe into account is much better. We can do it together. Take multiple format into account should be supported.

du00cs · 2016-11-15T10:41:11Z

@justsmoke I have merged your work. Can you help me to test if LzoBlockThriftInputFormat is supported? I tried to set up the environment but failed.

fjy · 2016-11-16T00:14:45Z

@justsmoke @du00cs thanks, we can focus review on this PR

du00cs · 2016-11-16T02:12:03Z

@justsmoke update with a change to avoid code style check failure. Thrift generator is platform dependent, I use scrooge-generator as a substitution to generate thrift files. You may need to change the thrift and scrooge version to match your case.

justsmoke · 2016-11-16T06:45:38Z

@du00cs ok, I will test the LzoBlockThriftInputFormat and give you feedback in two days

justsmoke · 2016-11-17T07:14:55Z

@du00cs When using LzoBlockThriftInputFormat, the thrift jar had to be included in classpath. This is different from other cases, and you may need to add some description in documentation. After the thrift jar is added to classpath, the thrift object is successfully parsed.

du00cs · 2016-11-17T07:18:14Z

@justsmoke if the thrift jar is specified by tmpjars like

{
"jobProperties" : {
        "tmpjars": "/user/xxx/druid/test/book.jar"
      }
}

the job will succeed?

justsmoke · 2016-11-17T09:36:17Z

@du00cs No, using config: "elephantbird.class.for.MultiInputFormat" : "${YOUR_THRIFT_CLASS_NAME}" and include jar in classpath is the only way to get it work. (because thrift deserialize is done within the elephant-bird lib)

du00cs · 2016-11-17T10:30:49Z

@justsmoke A little confusing

{
"jobProperties" : {
        "tmpjars": "/user/xxx/druid/test/book.jar",
        "elephantbird.class.for.MultiInputFormat" : "your.class.name"
      }
}

fails? I can not understand why. What's more, the path of tmpjars should be a HDFS path. Can you try it once more?

gzm55 · 2016-11-17T15:17:38Z

...ontrib/thrift-extensions/src/main/java/io/druid/data/input/thrift/ThriftDeserialization.java

@@ -0,0 +1,107 @@
+/**
+ * Copyright (c) 2016 XiaoMi Inc. All Rights Reserved.


this 'Copyright' is not appropriate here.

Sorry, it's just a mistake.

justsmoke · 2016-11-18T01:37:30Z

@du00cs Because deserialization is done by elephant-bird (after calling get on thriftWritable object, we get a TBase directly), so the parameter tmpjars is actually not used.

1. thrift binary is platform dependent, use scrooge to generate java files to avoid style check failure 2. stream and hadoop ingesion are both supported, input format can be sequence file and lzo thrift block file. 3. base64 and protocol aware change header

fjy · 2016-12-06T18:54:34Z

@du00cs @justsmoke there's a few merge conflicts. Do you guys feel this is ready?

du00cs · 2016-12-07T01:26:56Z

@justsmoke Can you send me a sample file of LzoBlockThriftInputFormat (and also the thrfit file) ? I want to look more into it.

nishantmonu51

LGTM, seems good to go after conflicts are resolved.

nishantmonu51 · 2016-12-09T19:55:28Z

docs/content/development/extensions-contrib/thrift.md

+| ----------- | ----------- | ---------------------------------------- | -------- |
+| type        | String      | This should say `thrift`                 | yes      |
+| parseSpec   | JSON Object | Specifies the timestamp and dimensions of the data. Should be a Json parseSpec. | yes      |
+| thriftJar   | String      | path of thrift jar, if not provided, it will try to find the thrift class in classpath. Thrift jar in batch ingestion should be uploaded to HDFS first and configure `jobProperties` with `"tmpjars":"/path/to/your/thrift.jar"` | no       |


druid uploads the jars added to classpath of middlemanager to HDFS. can you test by adding the thrift jar to middlemanager classpath work for batch ingestion ?
In that case use may not need to add explicitly upload it to HDFS.

Adding thrift jar to middle manager's class path will indeed work for batch ingestion, but an HDFS or some path is much more flexible. You don't have to put a jar into each nodes. Maybe pull from somewhere by http is the best. In this way, batch and realtime ways are unified. I should try to make it.

fjy · 2016-12-09T23:11:50Z

@du00cs there's a few conflicts

merge & fix pom conflicts

fjy · 2016-12-13T18:05:12Z

👍

justsmoke · 2016-12-16T03:06:04Z

@du00cs I am using this extension with tranquility, but get the following exception: java.lang.IllegalArgumentException: requirement failed: Expected ByteBufferInputRowParser, got[io.druid.data.input.thrift.ThriftInputRowParser]. It seems that tranquility only supports ByteBufferInputRowParser ?

du00cs · 2016-12-16T05:37:59Z

@justsmoke You are right. I submit a pull request at Sep. [Fix] Remove data type and InputRowParser type bindings , may be some help.

* Thrift ingestion plugin 1. thrift binary is platform dependent, use scrooge to generate java files to avoid style check failure 2. stream and hadoop ingesion are both supported, input format can be sequence file and lzo thrift block file. 3. base64 and protocol aware change header * fix conlicts in pom

du00cs force-pushed the feature-thrift branch 3 times, most recently from 1c216f8 to 205becc Compare September 6, 2016 09:18

fjy added this to the 0.9.3 milestone Sep 13, 2016

fjy added the Feature label Sep 13, 2016

fjy reviewed Sep 13, 2016
View reviewed changes

knoguchi mentioned this pull request Sep 25, 2016

flattenSpec: unified way to access elements of complex data (JSON, Avro, ProtoBuf, Thrift, ...) #3505

Closed

justsmoke mentioned this pull request Nov 1, 2016

druid thrift extensions #3633

Closed

du00cs force-pushed the feature-thrift branch 3 times, most recently from bff869d to d2f43d8 Compare November 15, 2016 10:38

du00cs force-pushed the feature-thrift branch from d2f43d8 to 5e3c4a7 Compare November 16, 2016 02:02

du00cs force-pushed the feature-thrift branch from 5e3c4a7 to ace0abe Compare November 16, 2016 02:18

gzm55 suggested changes Nov 17, 2016

View reviewed changes

Thrift ingestion plugin

ae3e799

1. thrift binary is platform dependent, use scrooge to generate java files to avoid style check failure 2. stream and hadoop ingesion are both supported, input format can be sequence file and lzo thrift block file. 3. base64 and protocol aware change header

du00cs force-pushed the feature-thrift branch from 8b1b2bb to ae3e799 Compare November 18, 2016 11:18

gianm assigned fjy and nishantmonu51 Nov 29, 2016

nishantmonu51 reviewed Dec 9, 2016

View reviewed changes

杜宁林 added 2 commits December 12, 2016 00:57

fix conlicts in pom

522e95a

Merge remote-tracking branch 'druid-io/master' into feature-thrift

94f08c1

merge & fix pom conflicts

fjy merged commit 469ab21 into apache:master Dec 13, 2016

gianm added the Release Notes label Feb 23, 2017

gianm mentioned this pull request Feb 28, 2017

Druid 0.10.0 release notes #3944

Closed

jihoonson mentioned this pull request Oct 7, 2017

Lazy initialization for JavaScript functions #4871

Merged

clambertus unassigned fjy and nishantmonu51 Jul 6, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature] Thrift support for realtime and batch ingestion #3418

[Feature] Thrift support for realtime and batch ingestion #3418

du00cs commented Sep 1, 2016

fjy Sep 13, 2016

du00cs Sep 14, 2016

du00cs Oct 11, 2016

fjy commented Nov 8, 2016

du00cs commented Nov 9, 2016

justsmoke commented Nov 9, 2016

du00cs commented Nov 10, 2016

du00cs commented Nov 15, 2016

fjy commented Nov 16, 2016

du00cs commented Nov 16, 2016

justsmoke commented Nov 16, 2016

justsmoke commented Nov 17, 2016

du00cs commented Nov 17, 2016

justsmoke commented Nov 17, 2016

du00cs commented Nov 17, 2016 •

edited

Loading

gzm55 Nov 17, 2016

du00cs Nov 18, 2016

justsmoke commented Nov 18, 2016

fjy commented Dec 6, 2016

du00cs commented Dec 7, 2016

nishantmonu51 left a comment

nishantmonu51 Dec 9, 2016

du00cs Dec 11, 2016

fjy commented Dec 9, 2016

fjy commented Dec 13, 2016

justsmoke commented Dec 16, 2016

du00cs commented Dec 16, 2016

		@@ -0,0 +1,107 @@
		/**
		* Copyright (c) 2016 XiaoMi Inc. All Rights Reserved.

[Feature] Thrift support for realtime and batch ingestion #3418

[Feature] Thrift support for realtime and batch ingestion #3418

Conversation

du00cs commented Sep 1, 2016

fjy Sep 13, 2016

Choose a reason for hiding this comment

du00cs Sep 14, 2016

Choose a reason for hiding this comment

du00cs Oct 11, 2016

Choose a reason for hiding this comment

fjy commented Nov 8, 2016

du00cs commented Nov 9, 2016

justsmoke commented Nov 9, 2016

du00cs commented Nov 10, 2016

du00cs commented Nov 15, 2016

fjy commented Nov 16, 2016

du00cs commented Nov 16, 2016

justsmoke commented Nov 16, 2016

justsmoke commented Nov 17, 2016

du00cs commented Nov 17, 2016

justsmoke commented Nov 17, 2016

du00cs commented Nov 17, 2016 • edited Loading

gzm55 Nov 17, 2016

Choose a reason for hiding this comment

du00cs Nov 18, 2016

Choose a reason for hiding this comment

justsmoke commented Nov 18, 2016

fjy commented Dec 6, 2016

du00cs commented Dec 7, 2016

nishantmonu51 left a comment

Choose a reason for hiding this comment

nishantmonu51 Dec 9, 2016

Choose a reason for hiding this comment

du00cs Dec 11, 2016

Choose a reason for hiding this comment

fjy commented Dec 9, 2016

fjy commented Dec 13, 2016

justsmoke commented Dec 16, 2016

du00cs commented Dec 16, 2016

du00cs commented Nov 17, 2016 •

edited

Loading