Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mkhokh/8858 cursor fields for incremental update #14356

Merged

Conversation

mkhokh-33
Copy link
Contributor

@mkhokh-33 mkhokh-33 commented Jul 1, 2022

What

DB Sources: only show a table can sync incrementally if at least one column can be used as a cursor field
If table has no field types that could be used as cursor then there should be no incremental sync option for this table.
See an example screenshot bellow
non_incremental

How

We have a list of predefined fields that could be used as cursors, before assign sync modes to stream we will check that table has at least one field that could be used as cursor

Recommended reading order

  1. x.java

🚨 User Impact 🚨

Are there any breaking changes? What is the end result perceived by the user? If yes, please merge this PR with the 🚨🚨 emoji so changelog authors can further highlight this if needed.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

  • Community member? Grant edit access to maintainers (instructions)
  • Secrets in the connector's spec are annotated with airbyte_secret
  • Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
  • Code reviews completed
  • Documentation updated
    • Connector's README.md
    • Connector's bootstrap.md. See description and examples
    • docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
    • docs/integrations/README.md
    • airbyte-integrations/builds.md
  • PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

  • Create a non-forked branch based on this PR and test the below items on it
  • Build is successful
  • If new credentials are required for use in CI, add them to GSM. Instructions.
  • /test connector=connectors/<name> command is passing
  • New Connector version released on Dockerhub by running the /publish command described here
  • After the connector is published, connector added to connector index as described here
  • Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here
Updating a connector

Community member or Airbyter

  • Grant edit access to maintainers (instructions)
  • Secrets in the connector's spec are annotated with airbyte_secret
  • Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
  • Code reviews completed
  • Documentation updated
    • Connector's README.md
    • Connector's bootstrap.md. See description and examples
    • Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
  • PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

  • Create a non-forked branch based on this PR and test the below items on it
  • Build is successful
  • If new credentials are required for use in CI, add them to GSM. Instructions.
  • /test connector=connectors/<name> command is passing
  • New Connector version released on Dockerhub and connector version bumped by running the /publish command described here
Connector Generator
  • Issue acceptance criteria met
  • PR name follows PR naming conventions
  • If adding a new generator, add it to the list of scaffold modules being tested
  • The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
  • Documentation which references the generator is updated as needed

Tests

Unit

Put your unit tests output here.

Integration

Put your integration tests output here.

Acceptance

Put your acceptance tests output here.

mkhokh-33 added 4 commits July 1, 2022 19:59
…update

# Conflicts:
#	airbyte-integrations/connectors/source-jdbc/src/test/java/io/airbyte/integrations/source/jdbc/AbstractJdbcSourceAcceptanceTest.java
#	airbyte-integrations/connectors/source-mysql/src/main/java/io/airbyte/integrations/source/mysql/MySqlSource.java
#	airbyte-integrations/connectors/source-postgres/src/main/java/io/airbyte/integrations/source/postgres/PostgresSource.java
#	airbyte-integrations/connectors/source-postgres/src/test/java/io/airbyte/integrations/source/postgres/PostgresJdbcSourceAcceptanceTest.java
@mkhokh-33
Copy link
Contributor Author

mkhokh-33 commented Jul 4, 2022

/test connector=connectors/source-mysql

🕑 connectors/source-mysql https://github.com/airbytehq/airbyte/actions/runs/2611538527
✅ connectors/source-mysql https://github.com/airbytehq/airbyte/actions/runs/2611538527
No Python unittests run

Build Passed

Test summary info:

All Passed

@mkhokh-33
Copy link
Contributor Author

mkhokh-33 commented Jul 4, 2022

/test connector=connectors/source-mssql

🕑 connectors/source-mssql https://github.com/airbytehq/airbyte/actions/runs/2611539182
✅ connectors/source-mssql https://github.com/airbytehq/airbyte/actions/runs/2611539182
No Python unittests run

Build Passed

Test summary info:

All Passed

@mkhokh-33
Copy link
Contributor Author

mkhokh-33 commented Jul 4, 2022

/test connector=connectors/source-db2

🕑 connectors/source-db2 https://github.com/airbytehq/airbyte/actions/runs/2611539750
✅ connectors/source-db2 https://github.com/airbytehq/airbyte/actions/runs/2611539750
No Python unittests run

Build Passed

Test summary info:

All Passed

@mkhokh-33
Copy link
Contributor Author

mkhokh-33 commented Jul 4, 2022

/test connector=connectors/source-clickhouse

🕑 connectors/source-clickhouse https://github.com/airbytehq/airbyte/actions/runs/2611540613
❌ connectors/source-clickhouse https://github.com/airbytehq/airbyte/actions/runs/2611540613
🐛 https://gradle.com/s/7pm4t2i7ujrzq

Build Failed

Test summary info:

Could not find result summary

@mkhokh-33
Copy link
Contributor Author

mkhokh-33 commented Jul 4, 2022

/test connector=connectors/source-oracle

🕑 connectors/source-oracle https://github.com/airbytehq/airbyte/actions/runs/2611540961
✅ connectors/source-oracle https://github.com/airbytehq/airbyte/actions/runs/2611540961
No Python unittests run

Build Passed

Test summary info:

All Passed

@mkhokh-33
Copy link
Contributor Author

mkhokh-33 commented Jul 4, 2022

/test connector=connectors/source-postgres

🕑 connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/2611541606
✅ connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/2611541606
No Python unittests run

Build Passed

Test summary info:

All Passed

@mkhokh-33
Copy link
Contributor Author

mkhokh-33 commented Jul 4, 2022

/test connector=connectors/source-redshift

🕑 connectors/source-redshift https://github.com/airbytehq/airbyte/actions/runs/2611542149
❌ connectors/source-redshift https://github.com/airbytehq/airbyte/actions/runs/2611542149
🐛 https://gradle.com/s/x3mogd5ji3jws

Build Failed

Test summary info:

Could not find result summary

@mkhokh-33
Copy link
Contributor Author

mkhokh-33 commented Jul 4, 2022

/test connector=connectors/source-snowflake

🕑 connectors/source-snowflake https://github.com/airbytehq/airbyte/actions/runs/2611543344
❌ connectors/source-snowflake https://github.com/airbytehq/airbyte/actions/runs/2611543344
🐛 https://gradle.com/s/kwv36isakyqsg

Build Failed

Test summary info:

Could not find result summary

@octavia-squidington-iii octavia-squidington-iii temporarily deployed to more-secrets August 18, 2022 13:05 Inactive
@octavia-squidington-iii octavia-squidington-iii temporarily deployed to more-secrets August 18, 2022 13:33 Inactive
@octavia-squidington-iii octavia-squidington-iii temporarily deployed to more-secrets August 18, 2022 13:39 Inactive
@octavia-squidington-iii octavia-squidington-iii temporarily deployed to more-secrets August 18, 2022 13:50 Inactive
@danieldiamond
Copy link
Contributor

danieldiamond commented Aug 21, 2022

A couple of people running into issues now
java.lang.RuntimeException: java.lang.RuntimeException: java.time.format.DateTimeParseException: Text '0' could not be parsed at index 0

e.g.

@mkhokh-33
Copy link
Contributor Author

mkhokh-33 commented Aug 22, 2022

A couple of people running into issues now java.lang.RuntimeException: java.lang.RuntimeException: java.time.format.DateTimeParseException: Text '0' could not be parsed at index 0

e.g.

created issue to investigate - #15840

@subodh1810
Copy link
Contributor

@danieldiamond can you share the logs of the failed sync please.

rodireich pushed a commit that referenced this pull request Aug 25, 2022
* add logic to skip incremental mode for tables which do not have a field types that could not be used as cursor

* added tests for cursor fields incremental updates

* remove not used methods

* fix tests

* fix formatting

* revert RedshiftInsertDestinationAcceptanceTest

* fix ssl clickhouse test

* fix codereview comments

* fix clickhouse strict encrypt test

* fix postgres source after merge from master

* optimize ssl clickhouse test logic

* fix db2 test

* added check for cursor field should be not nullable, fix formatting

* fixed tests, formatting, issue after merge from master

* fix merge issues

* fix test for source-jdbc and remove some code used for local testing

* added description for cockroach db tests

* test changes

* refactoring for cursor field tests after code review comments

* refactoring for cursor field tests after code review comments

* fix code review comments

* remove unused imports

* fix formatting

* fix compilation error

* fix postgres and postgress ssl test

* Fixed bucket naming for S3

* remove CHAR, NCHAR from possible cursor types

* remove SYMBOL from possible cursor types

* removed redundant configs

* fixed mysql-strict-encrypt tests

* fixed mongodb-v2 tests

* minor format changes

* bump version

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

* auto-bump connector version [ci skip]

Co-authored-by: vmaltsev <[email protected]>
Co-authored-by: subodh <[email protected]>
Co-authored-by: Octavia Squidington III <[email protected]>
Copy link

@matimirandach matimirandach left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since this version the source connector is failing when it's traying to get the schema.

m (plural-airbyte-3), a secondary, and following a leader (plural-airbyte-0)
2022-10-14 07:09:24,230 INFO: no action. I am (plural-airbyte-2), a secondary, and following a leader (plural-airbyte-0)
2022-10-14 07:09:24,238 INFO: no action. I am (plural-airbyte-4), a secondary, and following a leader (plural-airbyte-0)
2022-10-14 07:09:24,246 INFO: no action. I am (plural-airbyte-5), a secondary, and following a leader (plural-airbyte-0)
2022-10-14 07:09:24,260 INFO: no action. I am (plural-airbyte-7), a secondary, and following a leader (plural-airbyte-0)
2022-10-14 07:09:24,267 INFO: no action. I am (plural-airbyte-1), a secondary, and following a leader (plural-airbyte-0)
2022-10-14 07:09:24,280 INFO: no action. I am (plural-airbyte-0), the leader with the lock
2022-10-14 07:09:24,274 INFO: no action. I am (plural-airbyte-6), a secondary, and following a leader (plural-airbyte-0)
2022-10-14 07:09:27.964 38 LOG {ticks: 0, maint: 0, retry: 0}
2022-10-14 07:07:07 �[32mINFO�[m i.a.w.p.KubePodProcess(close):714 - (pod: airbyte / redshift-discover-ecffb8a4-66ef-47a2-9562-f3d8cd7f40fa-0-swrgr) - Closed all resources for pod
Log4j2Appender says: (pod: airbyte / redshift-discover-ecffb8a4-66ef-47a2-9562-f3d8cd7f40fa-0-swrgr) - Closed all resources for pod
2022-10-14 07:07:07 �[32mINFO�[m i.a.w.t.TemporalAttemptExecution(lambda$getWorkerThread$2):161 - Completing future exceptionally...
io.airbyte.workers.exception.WorkerException: Discover job subprocess finished with exit code 1
at io.airbyte.workers.general.DefaultDiscoverCatalogWorker.run(DefaultDiscoverCatalogWorker.java:76) ~[io.airbyte-airbyte-workers-0.39.14-alpha.jar:?]
at io.airbyte.workers.general.DefaultDiscoverCatalogWorker.run(DefaultDiscoverCatalogWorker.java:26) ~[io.airbyte-airbyte-workers-0.39.14-alpha.jar:?]
at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:158) ~[io.airbyte-airbyte-workers-0.39.14-alpha.jar:?]
at java.lang.Thread.run(Thread.java:833) [?:?]
Log4j2Appender says: Completing future exceptionally...
2022-10-14 07:07:07 �[32mINFO�[m i.a.w.t.TemporalAttemptExecution(get):134 - Stopping cancellation check scheduling...
Log4j2Appender says: Stopping cancellation check scheduling...
2022-10-14 07:07:07 �[33mWARN�[m i.t.i.a.POJOActivityTaskHandler(activityFailureToResult):307 - Activity failure. ActivityId=81f39ae7-6663-39d3-a3bc-e85f5f61d972, activityType=Run, attempt=1
java.util.concurrent.ExecutionException: io.airbyte.workers.exception.WorkerException: Discover job subprocess finished with exit code 1
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) ~[?:?]
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) ~[?:?]
at io.airbyte.workers.temporal.TemporalAttemptExecution.get(TemporalAttemptExecution.java:132) ~[io.airbyte-airbyte-workers-0.39.14-alpha.jar:?]
at io.airbyte.workers.temporal.discover.catalog.DiscoverCatalogActivityImpl.run(DiscoverCatalogActivityImpl.java:84) ~[io.airbyte-airbyte-workers-0.39.14-alpha.jar:?]
at jdk.internal.reflect.GeneratedMethodAccessor831.invoke(Unknown Source) ~[?:?]
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
at java.lang.reflect.Method.invoke(Method.java:568) ~[?:?]
at io.temporal.internal.activity.POJOActivityTaskHandler$POJOActivityInboundCallsInterceptor.execute(POJOActivityTaskHandler.java:214) ~[temporal-sdk-1.8.1.jar:?]
at io.temporal.internal.activity.POJOActivityTaskHandler$POJOActivityImplementation.execute(POJOActivityTaskHandler.java:180) ~[temporal-sdk-1.8.1.jar:?]
at io.temporal.internal.activity.POJOActivityTaskHandler.handle(POJOActivityTaskHandler.java:120) ~[temporal-sdk-1.8.1.jar:?]
at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:204) ~[temporal-sdk-1.8.1.jar:?]
at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:164) ~[temporal-sdk-1.8.1.jar:?]
at io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:93) ~[temporal-sdk-1.8.1.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: io.airbyte.workers.exception.WorkerException: Discover job subprocess finished with exit code 1
at io.airbyte.workers.general.DefaultDiscoverCatalogWorker.run(DefaultDiscoverCatalogWorker.java:76) ~[io.airbyte-airbyte-workers-0.39.14-alpha.jar:?]
at io.airbyte.workers.general.DefaultDiscoverCatalogWorker.run(DefaultDiscoverCatalogWorker.java:26) ~[io.airbyte-airbyte-workers-0.39.14-alpha.jar:?]
at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:158) ~[io.airbyte-airbyte-workers-0.39.14-alpha.jar:?]
... 1 more
Log4j2Appender says: Activity failure. ActivityId=81f39ae7-6663-39d3-a3bc-e85f5f61d972, activityType=Run, attempt=1
2022-10-14 07:07:07 �[1;31mERROR�[m i.t.i.s.WorkflowExecuteRunnable(logWorkflowExecutionException):125 - Workflow execution failure WorkflowId=9f629641-fc9d-435f-9232-0f5ee43aefc7, RunId=93ac5c88-deee-4d36-a159-ad942a5a653b, WorkflowType=DiscoverCatalogWorkflow
io.temporal.failure.ActivityFailure: scheduledEventId=5, startedEventId=6, activityType='Run', activityId='81f39ae7-6663-39d3-a3bc-e85f5f61d972', identity='1@airbyte-worker-65ffd94695-mm6gz', retryState=RETRY_STATE_MAXIMUM_ATTEMPTS_REACHED
at java.lang.Thread.getStackTrace(Thread.java:1610) ~[?:?]
at io.temporal.internal.sync.ActivityStubBase.execute(ActivityStubBase.java:48) ~[temporal-sdk-1.8.1.jar:?]
at io.temporal.internal.sync.ActivityInvocationHandler.lambda$getActivityFunc$0(ActivityInvocationHandler.java:77) ~[temporal-sdk-1.8.1.jar:?]
at io.temporal.internal.sync.ActivityInvocationHandlerBase.invoke(ActivityInvocationHandlerBase.java:70) ~[temporal-sdk-1.8.1.jar:?]
at jdk.proxy2.$Proxy79.run(Unknown Source) ~[?:?]
at io.airbyte.workers.temporal.discover.catalog.DiscoverCatalogWorkflowImpl.run(DiscoverCatalogWorkflowImpl.java:28) ~[io.airbyte-airbyte-workers-0.39.14-alpha.jar:?]
at jdk.internal.reflect.GeneratedMethodAccessor829.invoke(Unknown Source) ~[?:?]
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
at java.lang.reflect.Method.invoke(Method.java:568) ~[?:?]
at io.temporal.internal.sync.POJOWorkflowImplementationFactory$POJOWorkflowImplementation$RootWorkflowInboundCallsInterceptor.execute(POJOWorkflowImplementationFactory.java:317) ~[temporal-sdk-1.8.1.jar:?]
at io.temporal.internal.sync.POJOWorkflowImplementationFactory$POJOWorkflowImplementation.execute(POJOWorkflowImplementationFactory.java:292) ~[temporal-sdk-1.8.1.jar:?]
at io.temporal.internal.sync.WorkflowExecuteRunnable.run(WorkflowExecuteRunnable.java:72) ~[temporal-sdk-1.8.1.jar:?]
at io.temporal.internal.sync.SyncWorkflow.lambda$start$0(SyncWorkflow.java:137) ~[temporal-sdk-1.8.1.jar:?]
at io.temporal.internal.sync.CancellationScopeImpl.run(CancellationScopeImpl.java:101) [temporal-sdk-1.8.1.jar:?]
at io.temporal.internal.sync.WorkflowThreadImpl$RunnableWrapper.run(WorkflowThreadImpl.java:111) [temporal-sdk-1.8.1.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) [?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: io.temporal.failure.ApplicationFailure: message='io.airbyte.workers.exception.WorkerException: Discover job subprocess finished with exit code 1', type='java.util.concurrent.ExecutionException', nonRetryable=false
at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) ~[?:?]
at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) ~[?:?]
at io.airbyte.workers.temporal.TemporalAttemptExecution.get(TemporalAttemptExecution.java:132) ~[io.airbyte-airbyte-workers-0.39.14-alpha.jar:?]
at io.airbyte.workers.temporal.discover.catalog.DiscoverCatalogActivityImpl.run(DiscoverCatalogActivityImpl.java:84) ~[io.airbyte-airbyte-workers-0.39.14-alpha.jar:?]
at jdk.internal.reflect.GeneratedMethodAccessor831.invoke(Unknown Source:0) ~[?:?]
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
at java.base/java.lang.reflect.Method.invoke(Method.java:568) ~[?:?]
at io.temporal.internal.activity.POJOActivityTaskHandler$POJOActivityInboundCallsInterceptor.execute(POJOActivityTaskHandler.java:214) ~[temporal-sdk-1.8.1.jar:?]
at io.temporal.internal.activity.POJOActivityTaskHandler$POJOActivityImplementation.execute(POJOActivityTaskHandler.java:180) ~[temporal-sdk-1.8.1.jar:?]
at io.temporal.internal.activity.POJOActivityTaskHandler.handle(POJOActivityTaskHandler.java:120) ~[temporal-sdk-1.8.1.jar:?]
at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:204) ~[temporal-sdk-1.8.1.jar:?]
at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:164) ~[temporal-sdk-1.8.1.jar:?]
at io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:93) ~[temporal-sdk-1.8.1.jar:?]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
at java.base/java.lang.Thread.run(Thread.java:833) ~[?:?]
Caused by: io.temporal.failure.ApplicationFailure: message='Discover job subprocess finished with exit code 1', type='io.airbyte.workers.exception.WorkerException', nonRetryable=false
at io.airbyte.workers.general.DefaultDiscoverCatalogWorker.run(DefaultDiscoverCatalogWorker.java:76) ~[io.airbyte-airbyte-workers-0.39.14-alpha.jar:?]
at io.airbyte.workers.general.DefaultDiscoverCatalogWorker.run(DefaultDiscoverCatalogWorker.java:26) ~[io.airbyte-airbyte-workers-0.39.14-alpha.jar:?]
at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:158) ~[io.airbyte-airbyte-workers-0.39.14-alpha.jar:?]
at java.base/java.lang.Thread.run(Thread.java:833) ~[?:?]
Terminated

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment