Interface changes revision-2 #58
Conversation
common.proto
Outdated
| @@ -1,8 +1,8 @@ | |||
| syntax = "proto3"; | |||
| option optimize_for = SPEED; | |||
| option java_multiple_files = true; | |||
| option go_package = "fivetran.com/fivetran_sdk"; | |||
| package fivetran_sdk; | |||
| option go_package = "fivetran.com/fivetran_sdk_r2"; | |||
There was a problem hiding this comment.
Why are we changing the package name?
There was a problem hiding this comment.
We are introducing breaking changes and will use the identifier revision (r2) to denote the version for these updates.
There was a problem hiding this comment.
@georgewfraser as we discussed during the previous meeting, we need to be able to run one version in production while we need to be able to build and release different interface version in the testers to give to the partners. Since the testers are in the engineering repo too, there is no way to have two versions of grpc interface in the same package. So we introduced the "_r2" for "revision 2". (We chose to replace "version" with "revision" to indicate we are not going to use versions)
destination_sdk.proto
Outdated
| @@ -55,7 +55,26 @@ message CreateTableResponse { | |||
| message AlterTableRequest { | |||
| map<string, string> configuration = 1; | |||
| string schema_name = 2; | |||
| Table table = 3; | |||
| string table_name = 3; | |||
| repeated SchemaDiff changes = 4; | |||
There was a problem hiding this comment.
Do our own DataWriter implementations get something like this? How is it generated?
There was a problem hiding this comment.
No, our SDKDataWriter generates the schema difference. The same approach is followed across all the DataWriters.
For eg: In SQLLikeDataWriter, we compute the difference in primary key, update in column type, added new columns etc - SQLLikeDataWriter#updateSchema, SQLLikeDataWriter#updateTable
There was a problem hiding this comment.
@georgewfraser no they don't "get" it from the core, they calculate it themselves. This is something new we introduced so the partner code doesn't have to calculate it. SdkDataWriter generates the diffs to pass to the partner code so they don't have to do it themselves.
It would be great if we can move this to the core so the other data writers didn't have to perform the diff either. (In fact, the core doesn't even do a simple compare between existing and new schema, calling DataWriter#updateSchema whether the two are different or not, expecting every data writer to figure it out)
|
One more q, how are we doing logging now? |
@georgewfraser Logging is handled the same way it is handled for donkey processes, that is, the partner code prints to STDOUT in a specific JSON format. FluentBit collects these, processes them and sends them to New Relic. |
Currently, the logs are sent as a separate LogEntry operation. Going forward Partner's can use STDOUT with specified JSON format. Note that, as of now the connections logs are not exposed to the Partners. |
manjutapali
force-pushed
the
pt-sdk-r2-intf-changes
branch
from
0a9978e to
5aeb6d3
Compare
Description
Updated the proto interfaces with
default value,place holderfor FormField.NAIVE_TIMEOpTypetoRecordTypeLogEntry, modified theUpdateResponsemethod.