2024-05-16 16:15:01 platform > Docker volume job log path: /tmp/workspace/10969/0/logs.log
2024-05-16 16:15:01 platform > Executing worker wrapper. Airbyte version: 0.56.0
2024-05-16 16:15:01 platform > Attempt 0 to save workflow id for cancellation
2024-05-16 16:15:01 platform > start sync worker. job id: 10969 attempt id: 0
2024-05-16 16:15:01 platform > 
2024-05-16 16:15:01 platform > ----- START REPLICATION -----
2024-05-16 16:15:01 platform > 
2024-05-16 16:15:01 platform > Running destination...
2024-05-16 16:15:01 platform > Using default value for environment variable SIDECAR_KUBE_CPU_LIMIT: '2.0'
2024-05-16 16:15:01 platform > Using default value for environment variable SOCAT_KUBE_CPU_LIMIT: '2.0'
2024-05-16 16:15:01 platform > Using default value for environment variable SIDECAR_KUBE_CPU_REQUEST: '0.1'
2024-05-16 16:15:01 platform > Using default value for environment variable SOCAT_KUBE_CPU_REQUEST: '0.1'
2024-05-16 16:15:01 platform > Using default value for environment variable SIDECAR_KUBE_CPU_LIMIT: '2.0'
2024-05-16 16:15:01 platform > Using default value for environment variable SOCAT_KUBE_CPU_LIMIT: '2.0'
2024-05-16 16:15:01 platform > Using default value for environment variable SIDECAR_KUBE_CPU_REQUEST: '0.1'
2024-05-16 16:15:01 platform > Using default value for environment variable SOCAT_KUBE_CPU_REQUEST: '0.1'
2024-05-16 16:15:01 platform > Checking if airbyte/destination-bigquery:2.4.19 exists...
2024-05-16 16:15:01 platform > Checking if planetscale/airbyte-source:latest exists...
2024-05-16 16:15:01 platform > Creating docker container = airbyte-source-read-10969-0-fwgqn with resources io.airbyte.config.ResourceRequirements@3797b964[cpuRequest=0.5,cpuLimit=1,memoryRequest=1Gi,memoryLimit=2Gi,additionalProperties={}] and allowedHosts null
2024-05-16 16:15:01 platform > Preparing command: docker run --rm --init -i -w /data/10969/0 --log-driver none --name airbyte-source-read-10969-0-fwgqn -e CONCURRENT_SOURCE_STREAM_READ=false --network host -v airbyte_workspace:/data -v oss_local_root:/local -e DEPLOYMENT_MODE=OSS -e WORKER_CONNECTOR_IMAGE=planetscale/airbyte-source:latest -e AUTO_DETECT_SCHEMA=true -e LAUNCHDARKLY_KEY= -e SOCAT_KUBE_CPU_REQUEST=0.1 -e SOCAT_KUBE_CPU_LIMIT=2.0 -e FIELD_SELECTION_WORKSPACES= -e USE_STREAM_CAPABLE_STATE=true -e AIRBYTE_ROLE=dev -e WORKER_ENVIRONMENT=DOCKER -e APPLY_FIELD_SELECTION=false -e WORKER_JOB_ATTEMPT=0 -e OTEL_COLLECTOR_ENDPOINT=http://host.docker.internal:4317 -e FEATURE_FLAG_CLIENT=config -e AIRBYTE_VERSION=0.56.0 -e WORKER_JOB_ID=10969 --cpus=1 --memory-reservation=1Gi --memory=2Gi planetscale/airbyte-source:latest read --config source_config.json --catalog source_catalog.json --state input_state.json
2024-05-16 16:15:01 platform > planetscale/airbyte-source:latest was found locally.
2024-05-16 16:15:01 platform > Reading messages from protocol version 0.2.0
2024-05-16 16:15:01 platform > airbyte/destination-bigquery:2.4.19 was found locally.
2024-05-16 16:15:01 platform > Creating docker container = destination-bigquery-write-10969-0-kkqpr with resources io.airbyte.config.ResourceRequirements@59d691e[cpuRequest=0.5,cpuLimit=1,memoryRequest=1Gi,memoryLimit=2Gi,additionalProperties={}] and allowedHosts null
2024-05-16 16:15:01 platform > Preparing command: docker run --rm --init -i -w /data/10969/0 --log-driver none --name destination-bigquery-write-10969-0-kkqpr --network host -v airbyte_workspace:/data -v oss_local_root:/local -e DEPLOYMENT_MODE=OSS -e WORKER_CONNECTOR_IMAGE=airbyte/destination-bigquery:2.4.19 -e AUTO_DETECT_SCHEMA=true -e LAUNCHDARKLY_KEY= -e SOCAT_KUBE_CPU_REQUEST=0.1 -e SOCAT_KUBE_CPU_LIMIT=2.0 -e FIELD_SELECTION_WORKSPACES= -e USE_STREAM_CAPABLE_STATE=true -e AIRBYTE_ROLE=dev -e WORKER_ENVIRONMENT=DOCKER -e APPLY_FIELD_SELECTION=false -e WORKER_JOB_ATTEMPT=0 -e OTEL_COLLECTOR_ENDPOINT=http://host.docker.internal:4317 -e FEATURE_FLAG_CLIENT=config -e AIRBYTE_VERSION=0.56.0 -e WORKER_JOB_ID=10969 --cpus=1 --memory-reservation=1Gi --memory=2Gi airbyte/destination-bigquery:2.4.19 write --config destination_config.json --catalog destination_catalog.json
2024-05-16 16:15:01 platform > Writing messages to protocol version 0.2.0
2024-05-16 16:15:01 platform > Reading messages from protocol version 0.2.0
2024-05-16 16:15:01 platform > readFromSource: start
2024-05-16 16:15:01 platform > processMessage: start
2024-05-16 16:15:01 platform > Starting source heartbeat check. Will check threshold of 10800 seconds, every 1 minutes.
2024-05-16 16:15:01 platform > writeToDestination: start
2024-05-16 16:15:01 platform > readFromDestination: start
2024-05-16 16:15:02 source > PlanetScale Source :: Checking connection
2024-05-16 16:15:02 source > PlanetScale Source :: Syncing from tabletType "primary"
2024-05-16 16:15:02 source > PlanetScale Source :: [roofworx:primary:Task shard : -] peeking to see if there's any new rows
2024-05-16 16:15:02 source > PlanetScale Source :: new rows found, syncing rows for 1m0s
2024-05-16 16:15:02 source > PlanetScale Source :: [roofworx:primary:Task shard : -] syncing rows with cursor [shard:"-" keyspace:"roofworx" position:"MySQL56/190233c4-7cbc-11ee-9dcd-62270c1e7ad8:1-19510016,690d9190-cc6c-11ee-8a70-3e04e2c56d42:1-291827,895f5296-cce3-11ed-ba5c-767dd62768f0:1-216647,8c453a45-cce3-11ed-86c1-dac8568b2b4a:1-11747020,bc1c20d4-cce4-11ed-bf98-12b46bec4980:1-36534"]
2024-05-16 16:15:02 source > PlanetScale Source :: Syncing with cursor position : [MySQL56/190233c4-7cbc-11ee-9dcd-62270c1e7ad8:1-19510016,690d9190-cc6c-11ee-8a70-3e04e2c56d42:1-291827,895f5296-cce3-11ed-ba5c-767dd62768f0:1-216647,8c453a45-cce3-11ed-86c1-dac8568b2b4a:1-11747020,bc1c20d4-cce4-11ed-bf98-12b46bec4980:1-36534], using last known PK : false, stop cursor is : [MySQL56/190233c4-7cbc-11ee-9dcd-62270c1e7ad8:1-19510319,690d9190-cc6c-11ee-8a70-3e04e2c56d42:1-291827,895f5296-cce3-11ed-ba5c-767dd62768f0:1-216647,8c453a45-cce3-11ed-86c1-dac8568b2b4a:1-11747020,bc1c20d4-cce4-11ed-bf98-12b46bec4980:1-36534]
2024-05-16 16:15:02 source > PlanetScale Source :: DEBUG: SyncRequest.Cells = [planetscale_operator_default]
2024-05-16 16:15:03 source > PlanetScale Source :: [roofworx:primary:Task shard : -] Finished reading all rows for table [Task]
2024-05-16 16:15:03 platform > Total records read: 36 (12 KB)
2024-05-16 16:15:03 platform > Schema validation was performed to a max of 10 records with errors per stream.
2024-05-16 16:15:03 platform > Schema validation errors found for stream roofworx_Task. Error messages: [$.updatedAt: 2024-05-16 16:09:54.712 is an invalid date-time, $.dueDate: 2024-05-20 06:00:00.000 is an invalid date-time, $.updatedAt: 2024-05-16 16:07:26.853 is an invalid date-time, $.createdAt: 2024-05-16 16:10:03.806 is an invalid date-time, $.createdAt: 2024-04-22 14:52:11.402 is an invalid date-time, $.dueDate: 2024-05-16 06:00:00.000 is an invalid date-time, $.updatedAt: 2024-05-16 16:11:00.224 is an invalid date-time, $.createdAt: 2024-05-16 16:08:26.353 is an invalid date-time, $.updatedAt: 2024-05-16 16:07:17.755 is an invalid date-time, $.createdAt: 2024-05-16 16:08:26.367 is an invalid date-time, $.createdAt: 2024-05-16 16:11:00.366 is an invalid date-time, $.updatedAt: 2024-05-16 16:08:26.358 is an invalid date-time, $.updatedAt: 2024-05-16 16:09:39.673 is an invalid date-time, $.createdAt: 2024-05-16 15:12:06.904 is an invalid date-time, $.assigneeId: null found, string expected, $.onHoldScheduledEventId: null found, string expected, $.createdAt: 2024-05-16 15:57:06.502 is an invalid date-time, $.completionStatusReason: null found, string expected, $.createdAt: 2024-05-16 12:57:06.769 is an invalid date-time, $.dueDate: 2024-05-17 06:00:00.000 is an invalid date-time, $.completedAt: 2024-05-16 16:11:00.223 is an invalid date-time, $.updatedAt: 2024-05-16 16:09:57.971 is an invalid date-time, $.createdAt: 2024-05-10 16:14:06.034 is an invalid date-time, $.dueDate: 2024-05-24 06:00:00.000 is an invalid date-time, $.createdAt: 2024-04-24 15:38:45.448 is an invalid date-time, $.updatedAt: 2024-05-16 16:10:03.700 is an invalid date-time, $.updatedAt: 2024-05-16 16:12:20.126 is an invalid date-time, $.updatedAt: 2024-05-16 16:09:19.553 is an invalid date-time, $.createdAt: 2024-05-16 14:54:22.527 is an invalid date-time, $.updatedAt: 2024-05-16 16:10:03.807 is an invalid date-time, $.dueDate: null found, string expected, $.completedAt: 2024-05-16 16:10:03.699 is an invalid date-time, $.updatedAt: 2024-05-16 16:11:49.576 is an invalid date-time, $.updatedAt: 2024-05-16 16:08:53.957 is an invalid date-time, $.updatedAt: 2024-05-16 16:08:57.408 is an invalid date-time, $.createdAt: 2024-05-16 16:08:26.250 is an invalid date-time, $.createdAt: 2024-05-16 15:07:39.280 is an invalid date-time, $.completedAt: null found, string expected, $.updatedAt: 2024-05-16 16:11:00.367 is an invalid date-time, $.updatedAt: 2024-05-16 16:08:26.370 is an invalid date-time, $.updatedAt: 2024-05-16 16:09:22.409 is an invalid date-time, $.createdAt: 2024-04-24 15:51:18.080 is an invalid date-time, $.updatedAt: 2024-05-16 16:09:01.003 is an invalid date-time, $.updatedAt: 2024-05-16 16:12:57.259 is an invalid date-time, $.updatedAt: 2024-05-16 16:10:58.033 is an invalid date-time, $.createdAt: 2024-05-16 15:42:06.624 is an invalid date-time, $.updatedAt: 2024-05-16 16:08:26.270 is an invalid date-time, $.updatedAt: 2024-05-16 16:09:18.061 is an invalid date-time, $.createdAt: 2024-05-16 15:07:33.349 is an invalid date-time]
2024-05-16 16:15:03 platform > readFromSource: done. (source.isFinished:true, fromSource.isClosed:false)
2024-05-16 16:15:03 platform > thread status... heartbeat thread: false , replication thread: true
2024-05-16 16:15:04 platform > processMessage: done. (fromSource.isDone:true, forDest.isClosed:false)
2024-05-16 16:15:04 platform > writeToDestination: done. (forDest.isDone:true, isDestRunning:true)
2024-05-16 16:15:04 platform > thread status... timeout thread: false , replication thread: true
2024-05-16 16:15:06 destination > INFO main i.a.c.i.b.IntegrationCliParser$Companion(parseOptions):145 integration args: {catalog=destination_catalog.json, write=null, config=destination_config.json}
2024-05-16 16:15:06 destination > INFO main i.a.c.i.b.IntegrationRunner(runInternal):124 Running integration: io.airbyte.integrations.destination.bigquery.BigQueryDestination
2024-05-16 16:15:06 destination > INFO main i.a.c.i.b.IntegrationRunner(runInternal):125 Command: WRITE
2024-05-16 16:15:06 destination > INFO main i.a.c.i.b.IntegrationRunner(runInternal):126 Integration config: IntegrationConfig{command=WRITE, configPath='destination_config.json', catalogPath='destination_catalog.json', statePath='null'}
2024-05-16 16:15:06 destination > WARN main c.n.s.JsonMetaSchema(newValidator):278 Unknown keyword groups - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2024-05-16 16:15:06 destination > WARN main c.n.s.JsonMetaSchema(newValidator):278 Unknown keyword group - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2024-05-16 16:15:06 destination > WARN main c.n.s.JsonMetaSchema(newValidator):278 Unknown keyword order - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2024-05-16 16:15:06 destination > WARN main c.n.s.JsonMetaSchema(newValidator):278 Unknown keyword display_type - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2024-05-16 16:15:06 destination > WARN main c.n.s.JsonMetaSchema(newValidator):278 Unknown keyword airbyte_secret - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2024-05-16 16:15:06 destination > WARN main c.n.s.JsonMetaSchema(newValidator):278 Unknown keyword always_show - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2024-05-16 16:15:07 destination > INFO main i.a.i.d.b.BigQueryUtils(getLoadingMethod):323 Selected loading method is set to: GCS
2024-05-16 16:15:07 destination > INFO main i.a.i.b.d.t.CatalogParser(parseCatalog):109 Running sync with stream configs: [StreamConfig(id=StreamId(finalNamespace=planetscale, finalName=test_Task, rawNamespace=airbyte_internal, rawName=planetscale_raw__stream_test_Task, originalNamespace=planetscale, originalName=test_Task), syncMode=incremental, destinationSyncMode=append_dedup, primaryKey=[ColumnId(name=id, originalName=id, canonicalName=id)], cursor=Optional[ColumnId(name=id, originalName=id, canonicalName=id)], columns={ColumnId(name=id, originalName=id, canonicalName=id)=STRING, ColumnId(name=dueDate, originalName=dueDate, canonicalName=duedate)=TIMESTAMP_WITHOUT_TIMEZONE, ColumnId(name=taskType, originalName=taskType, canonicalName=tasktype)=STRING, ColumnId(name=createdAt, originalName=createdAt, canonicalName=createdat)=TIMESTAMP_WITHOUT_TIMEZONE, ColumnId(name=isDeleted, originalName=isDeleted, canonicalName=isdeleted)=INTEGER, ColumnId(name=projectId, originalName=projectId, canonicalName=projectid)=INTEGER, ColumnId(name=updatedAt, originalName=updatedAt, canonicalName=updatedat)=TIMESTAMP_WITHOUT_TIMEZONE, ColumnId(name=assigneeId, originalName=assigneeId, canonicalName=assigneeid)=STRING, ColumnId(name=completedAt, originalName=completedAt, canonicalName=completedat)=TIMESTAMP_WITHOUT_TIMEZONE, ColumnId(name=completionStatus, originalName=completionStatus, canonicalName=completionstatus)=STRING, ColumnId(name=completionStatusReason, originalName=completionStatusReason, canonicalName=completionstatusreason)=STRING, ColumnId(name=onHoldScheduledEventId, originalName=onHoldScheduledEventId, canonicalName=onholdscheduledeventid)=STRING})]
2024-05-16 16:15:07 destination > INFO main i.a.c.i.d.s.UploadFormatConfigFactory(getUploadFormatConfig):20 File upload format config: {"format_type":"CSV","flattening":"No flattening"}
2024-05-16 16:15:07 destination > INFO main i.a.i.d.b.BigQueryUtils(isKeepFilesInGcs):339 All tmp files will be removed from GCS when replication is finished
2024-05-16 16:15:09 destination > INFO main i.a.i.d.b.BigQueryStagingConsumerFactory(lambda$createWriteConfigs$0):109 BigQuery write config: BigQueryWriteConfig[streamName=test_Task, namespace=planetscale, datasetId=airbyte_internal, datasetLocation=US, tmpTableId=GenericData{classInfo=[datasetId, projectId, tableId], {datasetId=airbyte_internal, tableId=_airbyte_tmp_qkx_test_Task}}, targetTableId=GenericData{classInfo=[datasetId, projectId, tableId], {datasetId=airbyte_internal, tableId=planetscale_raw__stream_test_Task}}, tableSchema=Schema{fields=[Field{name=_airbyte_raw_id, type=STRING, mode=null, description=null, policyTags=null, maxLength=null, scale=null, precision=null, defaultValueExpression=null, collation=null, rangeElementType=null}, Field{name=_airbyte_extracted_at, type=TIMESTAMP, mode=null, description=null, policyTags=null, maxLength=null, scale=null, precision=null, defaultValueExpression=null, collation=null, rangeElementType=null}, Field{name=_airbyte_loaded_at, type=TIMESTAMP, mode=null, description=null, policyTags=null, maxLength=null, scale=null, precision=null, defaultValueExpression=null, collation=null, rangeElementType=null}, Field{name=_airbyte_data, type=STRING, mode=null, description=null, policyTags=null, maxLength=null, scale=null, precision=null, defaultValueExpression=null, collation=null, rangeElementType=null}]}, syncMode=append_dedup]
2024-05-16 16:15:09 destination > INFO main i.a.c.i.d.a.b.BufferManager(<init>):43 Max 'memory' available for buffer allocation 593 MB
2024-05-16 16:15:09 destination > INFO main i.a.c.i.b.IntegrationRunner$Companion(consumeWriteStream$io_airbyte_airbyte_cdk_java_airbyte_cdk_airbyte_cdk_core):423 Starting buffered read of input stream
2024-05-16 16:15:09 destination > INFO main i.a.c.i.d.a.FlushWorkers(start):74 Start async buffer supervisor
2024-05-16 16:15:09 destination > INFO main i.a.c.i.d.a.AsyncStreamConsumer(start):112 class io.airbyte.cdk.integrations.destination.async.AsyncStreamConsumer started.
2024-05-16 16:15:09 destination > INFO main i.a.i.d.b.BigQueryStagingConsumerFactory(lambda$onStartFunction$2):126 Preparing airbyte_raw tables in destination started for 1 streams
2024-05-16 16:15:09 destination > INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):89 [ASYNC QUEUE INFO] Global: max: 593.92 MB, allocated: 10 MB (10.0 MB), %% used: 0.016837142758506073 | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0
2024-05-16 16:15:09 destination > INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):128 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2024-05-16 16:15:14 destination > INFO type-and-dedupe i.a.i.b.d.t.BaseDestinationV1V2Migrator(migrateIfNecessary):19 Assessing whether migration is necessary for stream test_Task
2024-05-16 16:15:14 destination > INFO type-and-dedupe i.a.i.b.d.t.BaseDestinationV1V2Migrator(shouldMigrate):44 Checking whether v1 raw table _airbyte_raw_test_Task in dataset planetscale exists
2024-05-16 16:15:16 destination > INFO type-and-dedupe i.a.i.b.d.t.BaseDestinationV1V2Migrator(shouldMigrate):54 Migration Info: Required for Sync mode: true, No existing v2 raw tables: false, A v1 raw table exists: false
2024-05-16 16:15:16 destination > INFO type-and-dedupe i.a.i.b.d.t.BaseDestinationV1V2Migrator(migrateIfNecessary):31 No Migration Required for stream: test_Task
2024-05-16 16:15:16 destination > INFO type-and-dedupe i.a.i.d.b.t.BigQueryV2TableMigrator(migrateIfNecessary):76 No Data column Migration Required for stream planetscale_raw__stream_test_Task
2024-05-16 16:15:17 destination > INFO main i.a.i.d.b.t.BigQueryDestinationHandler(existingSchemaMatchesStreamConfig):228 Alter Table Report [] [] []; Clustering true; Partitioning true
2024-05-16 16:15:18 destination > INFO main i.a.i.d.b.BigQueryStagingConsumerFactory(lambda$onStartFunction$2):130 Preparing staging are in destination for schema: Schema{fields=[Field{name=_airbyte_raw_id, type=STRING, mode=null, description=null, policyTags=null, maxLength=null, scale=null, precision=null, defaultValueExpression=null, collation=null, rangeElementType=null}, Field{name=_airbyte_extracted_at, type=TIMESTAMP, mode=null, description=null, policyTags=null, maxLength=null, scale=null, precision=null, defaultValueExpression=null, collation=null, rangeElementType=null}, Field{name=_airbyte_loaded_at, type=TIMESTAMP, mode=null, description=null, policyTags=null, maxLength=null, scale=null, precision=null, defaultValueExpression=null, collation=null, rangeElementType=null}, Field{name=_airbyte_data, type=STRING, mode=null, description=null, policyTags=null, maxLength=null, scale=null, precision=null, defaultValueExpression=null, collation=null, rangeElementType=null}]}, stream: test_Task, target table: GenericData{classInfo=[datasetId, projectId, tableId], {datasetId=airbyte_internal, tableId=planetscale_raw__stream_test_Task}}, stage: test_Task
2024-05-16 16:15:18 destination > INFO main i.a.i.d.b.BigQueryGcsOperations(createSchemaIfNotExists):86 Creating dataset airbyte_internal
2024-05-16 16:15:19 destination > INFO main i.a.i.d.b.BigQueryGcsOperations(createTableIfNotExists):102 Creating target table GenericData{classInfo=[datasetId, projectId, tableId], {datasetId=airbyte_internal, tableId=planetscale_raw__stream_test_Task}}
2024-05-16 16:15:19 destination > INFO main i.a.i.d.b.BigQueryUtils(createPartitionedTableIfNotExists):218 Partitioned table ALREADY EXISTS: GenericData{classInfo=[datasetId, projectId, tableId], {datasetId=airbyte_internal, tableId=planetscale_raw__stream_test_Task}}
2024-05-16 16:15:19 destination > INFO main i.a.i.d.b.BigQueryGcsOperations(createStageIfNotExists):109 Creating staging path for stream test_Task (dataset airbyte_internal): datasync/airbyte_internal_test_Task/2024/05/16/16/58cc7ba5-a36d-4e6a-bc11-65682c869cb9/
2024-05-16 16:15:19 destination > INFO main i.a.i.b.d.t.DefaultTyperDeduper(prepareFinalTables):121 Preparing tables
2024-05-16 16:15:19 destination > INFO type-and-dedupe i.a.i.b.d.t.DefaultTyperDeduper(prepareTablesFuture$lambda$4):154 Final Table exists for stream test_Task
2024-05-16 16:15:19 destination > INFO main i.a.i.d.b.BigQueryStagingConsumerFactory(lambda$onStartFunction$2):151 Preparing tables in destination completed.
2024-05-16 16:15:20 destination > INFO main i.a.c.i.b.IntegrationRunner$Companion(consumeWriteStream$io_airbyte_airbyte_cdk_java_airbyte_cdk_airbyte_cdk_core):445 Finished buffered read of input stream
2024-05-16 16:15:20 destination > INFO main i.a.c.i.d.a.FlushWorkers(close):205 Closing flush workers -- waiting for all buffers to flush
2024-05-16 16:15:20 destination > INFO main i.a.c.i.d.a.FlushWorkers(close):249 REMAINING_BUFFERS_INFO
  Namespace: planetscale Stream: test_Task -- remaining records: 35
2024-05-16 16:15:20 destination > INFO main i.a.c.i.d.a.FlushWorkers(close):250 Waiting for all streams to flush.
2024-05-16 16:15:20 destination > INFO pool-5-thread-1 i.a.c.i.d.a.DetectStreamToFlush(getNextStreamToFlush):110 flushing: trigger info: planetscale - test_Task, time trigger: false , size trigger: true current threshold b: 0 bytes, queue size b: 17.83 KB, penalty b: 0 bytes, after penalty b: 17.83 KB
2024-05-16 16:15:20 destination > INFO pool-4-thread-1 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):136 Flush Worker (002ed) -- Worker picked up work.
2024-05-16 16:15:20 destination > INFO pool-4-thread-1 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):142 Flush Worker (002ed) -- Attempting to read from queue namespace: planetscale, stream: test_Task.
2024-05-16 16:15:20 destination > INFO pool-4-thread-1 i.a.c.i.d.a.GlobalMemoryManager(free):78 Freeing 10467498 bytes..
2024-05-16 16:15:20 destination > INFO pool-4-thread-1 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):164 Flush Worker (002ed) -- Batch contains: 35 records, 17.83 KB bytes.
2024-05-16 16:15:20 destination > INFO pool-4-thread-1 i.a.c.i.d.r.BaseSerializedBuffer(flush):156 Finished writing data to 88de77a4-8cd4-4201-9de6-14989b7283723775778884548178928.csv.gz (2 KB)
2024-05-16 16:15:20 destination > INFO pool-4-thread-1 i.a.i.d.b.BigQueryAsyncFlush(flush):62 Flushing CSV buffer for stream test_Task (2 KB) to staging
2024-05-16 16:15:20 destination > INFO pool-4-thread-1 i.a.i.d.b.BigQueryGcsOperations(uploadRecordsToStage):116 Uploading records to staging for stream test_Task (dataset airbyte_internal): datasync/airbyte_internal_test_Task/2024/05/16/16/58cc7ba5-a36d-4e6a-bc11-65682c869cb9/
2024-05-16 16:15:20 destination > INFO pool-4-thread-1 a.m.s.StreamTransferManager(getMultiPartOutputStreams):329 Initiated multipart upload to remi-monorepo-sync/datasync/airbyte_internal_test_Task/2024/05/16/16/58cc7ba5-a36d-4e6a-bc11-65682c869cb9/0.csv.gz with full ID ABPnzm7lY8ykOizyJCR1qqFZGHaGzd-v26Ebo8Vx-NucsZRxTtQJt1psCpPoL5s5cjQYBXQ
2024-05-16 16:15:20 destination > INFO pool-4-thread-1 a.m.s.MultiPartOutputStream(close):158 Called close() on [MultipartOutputStream for parts 1 - 10000]
2024-05-16 16:15:20 destination > INFO pool-4-thread-1 a.m.s.StreamTransferManager(complete):367 [Manager uploading to remi-monorepo-sync/datasync/airbyte_internal_test_Task/2024/05/16/16/58cc7ba5-a36d-4e6a-bc11-65682c869cb9/0.csv.gz with id ABPnzm7lY...s5cjQYBXQ]: Uploading leftover stream [Part number 1 containing 0.00 MB]
2024-05-16 16:15:21 destination > INFO main i.a.c.i.d.a.FlushWorkers(close):253 Closing flush workers -- all buffers flushed
2024-05-16 16:15:21 destination > INFO main i.a.c.i.d.a.s.GlobalAsyncStateManager(flushStates):155 Flushing states
2024-05-16 16:15:21 destination > INFO main i.a.c.i.d.a.s.GlobalAsyncStateManager(flushStates):209 Flushing states complete
2024-05-16 16:15:21 destination > INFO main i.a.c.i.d.a.GlobalMemoryManager(free):78 Freeing 0 bytes..
2024-05-16 16:15:21 destination > INFO main i.a.c.i.d.a.FlushWorkers(close):261 Closing flush workers -- supervisor shut down
2024-05-16 16:15:21 destination > INFO main i.a.c.i.d.a.FlushWorkers(close):263 Closing flush workers -- Starting worker pool shutdown..
2024-05-16 16:15:21 destination > INFO pool-4-thread-1 a.m.s.StreamTransferManager(uploadStreamPart):560 [Manager uploading to remi-monorepo-sync/datasync/airbyte_internal_test_Task/2024/05/16/16/58cc7ba5-a36d-4e6a-bc11-65682c869cb9/0.csv.gz with id ABPnzm7lY...s5cjQYBXQ]: Finished uploading [Part number 1 containing 0.00 MB]
2024-05-16 16:15:21 destination > INFO pool-4-thread-1 a.m.s.StreamTransferManager(complete):397 [Manager uploading to remi-monorepo-sync/datasync/airbyte_internal_test_Task/2024/05/16/16/58cc7ba5-a36d-4e6a-bc11-65682c869cb9/0.csv.gz with id ABPnzm7lY...s5cjQYBXQ]: Completed
2024-05-16 16:15:21 destination > INFO pool-4-thread-1 i.a.c.i.d.s.S3StorageOperations(loadDataIntoBucket):234 Uploaded buffer file to storage: 88de77a4-8cd4-4201-9de6-14989b7283723775778884548178928.csv.gz -> datasync/airbyte_internal_test_Task/2024/05/16/16/58cc7ba5-a36d-4e6a-bc11-65682c869cb9/0.csv.gz (filename: 0.csv.gz)
2024-05-16 16:15:21 destination > INFO pool-4-thread-1 i.a.c.i.d.s.S3StorageOperations(uploadRecordsToBucket):130 Successfully loaded records to stage datasync/airbyte_internal_test_Task/2024/05/16/16/58cc7ba5-a36d-4e6a-bc11-65682c869cb9/ with 0 re-attempt(s)
2024-05-16 16:15:21 destination > INFO pool-4-thread-1 i.a.i.d.b.BigQueryGcsOperations(copyIntoTableFromStage):134 Uploading records from staging files to target table GenericData{classInfo=[datasetId, projectId, tableId], {datasetId=airbyte_internal, tableId=planetscale_raw__stream_test_Task}} (dataset airbyte_internal): 0.csv.gz
2024-05-16 16:15:21 destination > INFO pool-4-thread-1 i.a.i.d.b.BigQueryGcsOperations(copyIntoTableFromStage):138 Uploading staged file: gs://remi-monorepo-sync/datasync/airbyte_internal_test_Task/2024/05/16/16/58cc7ba5-a36d-4e6a-bc11-65682c869cb9/0.csv.gz
2024-05-16 16:15:22 destination > INFO pool-4-thread-1 i.a.i.d.b.BigQueryGcsOperations(copyIntoTableFromStage):147 [JobId{project=remi-data-warehouse, job=b3fb3a34-4ba6-4e07-ba3c-82d8228ca9a9, location=US}] Created a new job to upload record(s) to target table GenericData{classInfo=[datasetId, projectId, tableId], {datasetId=airbyte_internal, tableId=planetscale_raw__stream_test_Task}} (dataset airbyte_internal): Job{job=JobId{project=remi-data-warehouse, job=b3fb3a34-4ba6-4e07-ba3c-82d8228ca9a9, location=US}, status=JobStatus{state=PENDING, error=null, executionErrors=null}, statistics=LoadStatistics{creationTime=1715876121877, endTime=null, startTime=null, numChildJobs=null, parentJobId=null, scriptStatistics=null, reservationUsage=null, transactionInfo=null, sessionInfo=null, inputBytes=null, inputFiles=null, outputBytes=null, outputRows=null, badRecords=null}, userEmail=remi-warehouse-service@remi-data-warehouse.iam.gserviceaccount.com, etag=I5ADrgSvwMojReBYjc1MYA==, generatedId=remi-data-warehouse:US.b3fb3a34-4ba6-4e07-ba3c-82d8228ca9a9, selfLink=https://bigquery.googleapis.com/bigquery/v2/projects/remi-data-warehouse/jobs/b3fb3a34-4ba6-4e07-ba3c-82d8228ca9a9?location=US, configuration=LoadJobConfiguration{type=LOAD, destinationTable=GenericData{classInfo=[datasetId, projectId, tableId], {datasetId=airbyte_internal, projectId=remi-data-warehouse, tableId=planetscale_raw__stream_test_Task}}, decimalTargetTypes=null, destinationEncryptionConfiguration=null, createDisposition=null, writeDisposition=WRITE_APPEND, formatOptions=CsvOptions{type=CSV, allowJaggedRows=null, allowQuotedNewLines=null, encoding=null, fieldDelimiter=null, quote=null, skipLeadingRows=null, preserveAsciiControlCharacters=null}, nullMarker=null, maxBadRecords=null, schema=Schema{fields=[Field{name=_airbyte_raw_id, type=STRING, mode=null, description=null, policyTags=null, maxLength=null, scale=null, precision=null, defaultValueExpression=null, collation=null, rangeElementType=null}, Field{name=_airbyte_extracted_at, type=TIMESTAMP, mode=null, description=null, policyTags=null, maxLength=null, scale=null, precision=null, defaultValueExpression=null, collation=null, rangeElementType=null}, Field{name=_airbyte_loaded_at, type=TIMESTAMP, mode=null, description=null, policyTags=null, maxLength=null, scale=null, precision=null, defaultValueExpression=null, collation=null, rangeElementType=null}, Field{name=_airbyte_data, type=STRING, mode=null, description=null, policyTags=null, maxLength=null, scale=null, precision=null, defaultValueExpression=null, collation=null, rangeElementType=null}]}, ignoreUnknownValue=null, sourceUris=[gs://remi-monorepo-sync/datasync/airbyte_internal_test_Task/2024/05/16/16/58cc7ba5-a36d-4e6a-bc11-65682c869cb9/0.csv.gz], fileSetSpecType=null, schemaUpdateOptions=null, autodetect=null, timePartitioning=null, clustering=null, useAvroLogicalTypes=null, labels=null, jobTimeoutMs=600000, rangePartitioning=null, hivePartitioningOptions=null, referenceFileSchemaUri=null, connectionProperties=null, createSession=null}}
2024-05-16 16:15:22 destination > INFO pool-4-thread-1 i.a.i.d.b.BigQueryUtils(waitForJobFinish):348 Waiting for job finish Job{job=JobId{project=remi-data-warehouse, job=b3fb3a34-4ba6-4e07-ba3c-82d8228ca9a9, location=US}, status=JobStatus{state=PENDING, error=null, executionErrors=null}, statistics=LoadStatistics{creationTime=1715876121877, endTime=null, startTime=null, numChildJobs=null, parentJobId=null, scriptStatistics=null, reservationUsage=null, transactionInfo=null, sessionInfo=null, inputBytes=null, inputFiles=null, outputBytes=null, outputRows=null, badRecords=null}, userEmail=remi-warehouse-service@remi-data-warehouse.iam.gserviceaccount.com, etag=I5ADrgSvwMojReBYjc1MYA==, generatedId=remi-data-warehouse:US.b3fb3a34-4ba6-4e07-ba3c-82d8228ca9a9, selfLink=https://bigquery.googleapis.com/bigquery/v2/projects/remi-data-warehouse/jobs/b3fb3a34-4ba6-4e07-ba3c-82d8228ca9a9?location=US, configuration=LoadJobConfiguration{type=LOAD, destinationTable=GenericData{classInfo=[datasetId, projectId, tableId], {datasetId=airbyte_internal, projectId=remi-data-warehouse, tableId=planetscale_raw__stream_test_Task}}, decimalTargetTypes=null, destinationEncryptionConfiguration=null, createDisposition=null, writeDisposition=WRITE_APPEND, formatOptions=CsvOptions{type=CSV, allowJaggedRows=null, allowQuotedNewLines=null, encoding=null, fieldDelimiter=null, quote=null, skipLeadingRows=null, preserveAsciiControlCharacters=null}, nullMarker=null, maxBadRecords=null, schema=Schema{fields=[Field{name=_airbyte_raw_id, type=STRING, mode=null, description=null, policyTags=null, maxLength=null, scale=null, precision=null, defaultValueExpression=null, collation=null, rangeElementType=null}, Field{name=_airbyte_extracted_at, type=TIMESTAMP, mode=null, description=null, policyTags=null, maxLength=null, scale=null, precision=null, defaultValueExpression=null, collation=null, rangeElementType=null}, Field{name=_airbyte_loaded_at, type=TIMESTAMP, mode=null, description=null, policyTags=null, maxLength=null, scale=null, precision=null, defaultValueExpression=null, collation=null, rangeElementType=null}, Field{name=_airbyte_data, type=STRING, mode=null, description=null, policyTags=null, maxLength=null, scale=null, precision=null, defaultValueExpression=null, collation=null, rangeElementType=null}]}, ignoreUnknownValue=null, sourceUris=[gs://remi-monorepo-sync/datasync/airbyte_internal_test_Task/2024/05/16/16/58cc7ba5-a36d-4e6a-bc11-65682c869cb9/0.csv.gz], fileSetSpecType=null, schemaUpdateOptions=null, autodetect=null, timePartitioning=null, clustering=null, useAvroLogicalTypes=null, labels=null, jobTimeoutMs=600000, rangePartitioning=null, hivePartitioningOptions=null, referenceFileSchemaUri=null, connectionProperties=null, createSession=null}}. Status: JobStatus{state=PENDING, error=null, executionErrors=null}
2024-05-16 16:15:26 destination > INFO pool-4-thread-1 i.a.i.d.b.BigQueryUtils(waitForJobFinish):350 Job finish Job{job=JobId{project=remi-data-warehouse, job=b3fb3a34-4ba6-4e07-ba3c-82d8228ca9a9, location=US}, status=JobStatus{state=PENDING, error=null, executionErrors=null}, statistics=LoadStatistics{creationTime=1715876121877, endTime=null, startTime=null, numChildJobs=null, parentJobId=null, scriptStatistics=null, reservationUsage=null, transactionInfo=null, sessionInfo=null, inputBytes=null, inputFiles=null, outputBytes=null, outputRows=null, badRecords=null}, userEmail=remi-warehouse-service@remi-data-warehouse.iam.gserviceaccount.com, etag=I5ADrgSvwMojReBYjc1MYA==, generatedId=remi-data-warehouse:US.b3fb3a34-4ba6-4e07-ba3c-82d8228ca9a9, selfLink=https://bigquery.googleapis.com/bigquery/v2/projects/remi-data-warehouse/jobs/b3fb3a34-4ba6-4e07-ba3c-82d8228ca9a9?location=US, configuration=LoadJobConfiguration{type=LOAD, destinationTable=GenericData{classInfo=[datasetId, projectId, tableId], {datasetId=airbyte_internal, projectId=remi-data-warehouse, tableId=planetscale_raw__stream_test_Task}}, decimalTargetTypes=null, destinationEncryptionConfiguration=null, createDisposition=null, writeDisposition=WRITE_APPEND, formatOptions=CsvOptions{type=CSV, allowJaggedRows=null, allowQuotedNewLines=null, encoding=null, fieldDelimiter=null, quote=null, skipLeadingRows=null, preserveAsciiControlCharacters=null}, nullMarker=null, maxBadRecords=null, schema=Schema{fields=[Field{name=_airbyte_raw_id, type=STRING, mode=null, description=null, policyTags=null, maxLength=null, scale=null, precision=null, defaultValueExpression=null, collation=null, rangeElementType=null}, Field{name=_airbyte_extracted_at, type=TIMESTAMP, mode=null, description=null, policyTags=null, maxLength=null, scale=null, precision=null, defaultValueExpression=null, collation=null, rangeElementType=null}, Field{name=_airbyte_loaded_at, type=TIMESTAMP, mode=null, description=null, policyTags=null, maxLength=null, scale=null, precision=null, defaultValueExpression=null, collation=null, rangeElementType=null}, Field{name=_airbyte_data, type=STRING, mode=null, description=null, policyTags=null, maxLength=null, scale=null, precision=null, defaultValueExpression=null, collation=null, rangeElementType=null}]}, ignoreUnknownValue=null, sourceUris=[gs://remi-monorepo-sync/datasync/airbyte_internal_test_Task/2024/05/16/16/58cc7ba5-a36d-4e6a-bc11-65682c869cb9/0.csv.gz], fileSetSpecType=null, schemaUpdateOptions=null, autodetect=null, timePartitioning=null, clustering=null, useAvroLogicalTypes=null, labels=null, jobTimeoutMs=600000, rangePartitioning=null, hivePartitioningOptions=null, referenceFileSchemaUri=null, connectionProperties=null, createSession=null}} with status JobStatus{state=PENDING, error=null, executionErrors=null}
2024-05-16 16:15:26 destination > INFO pool-4-thread-1 i.a.i.d.b.BigQueryGcsOperations(copyIntoTableFromStage):152 [JobId{project=remi-data-warehouse, job=b3fb3a34-4ba6-4e07-ba3c-82d8228ca9a9, location=US}] Target table GenericData{classInfo=[datasetId, projectId, tableId], {datasetId=airbyte_internal, tableId=planetscale_raw__stream_test_Task}} (dataset airbyte_internal) is successfully appended with staging files
2024-05-16 16:15:26 destination > INFO pool-4-thread-1 i.a.c.i.d.r.FileBuffer(deleteFile):74 Deleting tempFile data 88de77a4-8cd4-4201-9de6-14989b7283723775778884548178928.csv.gz
2024-05-16 16:15:26 destination > INFO pool-4-thread-1 i.a.c.i.d.a.s.GlobalAsyncStateManager(flushStates):155 Flushing states
2024-05-16 16:15:26 platform > Destination state message checksum is invalid: state record count 35.0 does not equal tracked record count 0.0 for stream _null.
2024-05-16 16:15:26 platform > starting state flush thread for connectionId f87d795b-e574-4f97-b0a9-a6e9c1e27b73
2024-05-16 16:15:26 destination > INFO pool-4-thread-1 i.a.c.i.d.a.s.GlobalAsyncStateManager(flushStates):209 Flushing states complete
2024-05-16 16:15:26 destination > INFO pool-4-thread-1 i.a.c.i.d.a.GlobalMemoryManager(free):78 Freeing 529 bytes..
2024-05-16 16:15:26 destination > INFO pool-4-thread-1 i.a.c.i.d.a.GlobalMemoryManager(free):78 Freeing 18262 bytes..
2024-05-16 16:15:26 destination > INFO pool-4-thread-1 i.a.c.i.d.a.FlushWorkers(flush$lambda$6):182 Flush Worker (002ed) -- Worker finished flushing. Current queue size: 0
2024-05-16 16:15:26 destination > INFO main i.a.c.i.d.a.FlushWorkers(close):268 Closing flush workers  -- workers shut down
2024-05-16 16:15:26 destination > INFO main i.a.c.i.d.a.b.BufferManager(close):68 Buffers cleared..
2024-05-16 16:15:26 destination > INFO main i.a.i.b.d.t.DefaultTyperDeduper(typeAndDedupe):292 Typing and deduping all tables
2024-05-16 16:15:26 destination > INFO type-and-dedupe i.a.i.b.d.t.TypeAndDedupeTransaction(executeTypeAndDedupe):37 Attempting typing and deduping for planetscale.test_Task with suffix 
2024-05-16 16:15:36 destination > INFO main i.a.i.d.b.BigQueryStagingConsumerFactory(lambda$onCloseFunction$3):171 Cleaning up destination started for 1 streams
2024-05-16 16:15:36 destination > INFO main i.a.i.d.b.BigQueryGcsOperations(dropStageIfExists):186 Cleaning up staging path for stream test_Task (dataset airbyte_internal): datasync/airbyte_internal_test_Task
2024-05-16 16:15:36 destination > INFO main i.a.c.i.d.g.GcsStorageOperations(cleanUpObjects):34 Deleting object datasync/airbyte_internal_test_Task/2024/05/16/16/58cc7ba5-a36d-4e6a-bc11-65682c869cb9/0.csv.gz
2024-05-16 16:15:36 destination > INFO main i.a.c.i.d.s.S3StorageOperations(cleanUpBucketObject):380 Storage bucket datasync/airbyte_internal_test_Task has been cleaned-up (1 objects were deleted)...
2024-05-16 16:15:36 destination > INFO main i.a.i.b.d.t.DefaultTyperDeduper(commitFinalTables):348 Committing final tables
2024-05-16 16:15:36 destination > INFO main i.a.i.b.d.t.DefaultTyperDeduper(cleanup):412 Cleaning Up type-and-dedupe thread pool
2024-05-16 16:15:36 destination > INFO main i.a.i.d.b.BigQueryStagingConsumerFactory(lambda$onCloseFunction$3):177 Cleaning up destination completed.
2024-05-16 16:15:36 destination > INFO main i.a.c.i.d.a.AsyncStreamConsumer(close):179 class io.airbyte.cdk.integrations.destination.async.AsyncStreamConsumer closed
2024-05-16 16:15:36 destination > INFO main i.a.c.i.b.IntegrationRunner(runInternal):267 Completed integration: io.airbyte.integrations.destination.bigquery.BigQueryDestination
2024-05-16 16:15:36 platform > readFromDestination: done. (writeToDestFailed:false, dest.isFinished:true)
2024-05-16 16:15:36 platform > thread status... timeout thread: false , replication thread: true
2024-05-16 16:15:36 platform > sync summary: {
  "status" : "completed",
  "recordsSynced" : 0,
  "bytesSynced" : 0,
  "startTime" : 1715876101248,
  "endTime" : 1715876136476,
  "totalStats" : {
    "bytesCommitted" : 12522,
    "bytesEmitted" : 12522,
    "destinationStateMessagesEmitted" : 1,
    "destinationWriteEndTime" : 1715876136474,
    "destinationWriteStartTime" : 1715876101249,
    "meanSecondsBeforeSourceStateMessageEmitted" : 0,
    "maxSecondsBeforeSourceStateMessageEmitted" : 1,
    "maxSecondsBetweenStateMessageEmittedandCommitted" : 22,
    "meanSecondsBetweenStateMessageEmittedandCommitted" : 22,
    "recordsEmitted" : 35,
    "recordsCommitted" : 35,
    "replicationEndTime" : 1715876136476,
    "replicationStartTime" : 1715876101248,
    "sourceReadEndTime" : 1715876103993,
    "sourceReadStartTime" : 1715876101250,
    "sourceStateMessagesEmitted" : 1
  },
  "streamStats" : [ {
    "streamName" : "Task",
    "streamNamespace" : "roofworx",
    "stats" : {
      "bytesCommitted" : 12522,
      "bytesEmitted" : 12522,
      "recordsEmitted" : 35,
      "recordsCommitted" : 35
    }
  } ],
  "performanceMetrics" : {
    "processFromSource" : {
      "elapsedTimeInNanos" : 16187471,
      "executionCount" : 36,
      "avgExecTimeInNanos" : 449651.97222222225
    },
    "readFromSource" : {
      "elapsedTimeInNanos" : 2539369592,
      "executionCount" : 56,
      "avgExecTimeInNanos" : 4.5345885571428575E7
    },
    "processFromDest" : {
      "elapsedTimeInNanos" : 4997534,
      "executionCount" : 1,
      "avgExecTimeInNanos" : 4997534.0
    },
    "writeToDest" : {
      "elapsedTimeInNanos" : 1328854,
      "executionCount" : 36,
      "avgExecTimeInNanos" : 36912.61111111111
    },
    "readFromDest" : {
      "elapsedTimeInNanos" : 35038152703,
      "executionCount" : 219,
      "avgExecTimeInNanos" : 1.5999156485388127E8
    }
  }
}
2024-05-16 16:15:36 platform > failures: [ ]
2024-05-16 16:15:36 platform > 
2024-05-16 16:15:36 platform > ----- END REPLICATION -----
2024-05-16 16:15:36 platform >