Airbyte Incremental | Append + Deduped Not Updating Records

[VolubleRain1]

I built a connector from PlanetScale to BigQuery using Airbyte and the official PlanetScale connector. The majority of tables seem to be incrementing and updating properly. However, one table isn't. Since I can't manually set the cursor like I can with a MySQL connector, what do I need to do for the connector to recognize the cursor and update the BigQuery table appropriately?

[notfelineit]

Hi @VolubleRain1 ,

Could you help me answer your question better? Could you provide logs on the one table that isn't incrementing and updating properly?

We do have a way to set the cursor to a starting GTID now in Airbyte: https://github.com/planetscale/airbyte-source?tab=readme-ov-file#3a-starting-from-a-specific-gtid

However, I'd like to get more information about that failure before being prescriptive. Thanks!

[VolubleRain1]

@notfelineit remihq is the org and roofworx is the db.

I did a reset on a batch of our streams today, and it looks like a similar error message is thrown on several other tables as well. . . I would expect after a data reset, we wouldn't have that error.

Thank you for your assistance in looking into this!!

[notfelineit]

Ok, thank you @VolubleRain1 ! I'll look at the logs for this specific database and see what's happening. Could you share your Airbyte logs again for the latest sync?

[VolubleRain1]

@notfelineit here they are!
default_workspace_job_10969_attempt_1_txt.txt

[VolubleRain1]

@notfelineit have you been able to look into this?

[notfelineit]

@VolubleRain1 I noticed this error:

2024-05-16 16:15:03 platform > Schema validation errors found for stream roofworx_Task. Error messages: [$.updatedAt: 2024-05-16 16:09:54.712 is an invalid date-time, $.dueDate: 2024-05-20 06:00:00.000 is an invalid date-time, $.updatedAt: 2024-05-16 16:07:26.853 is an invalid date-time, $.createdAt: 2024-05-16 16:10:03.806 is an invalid date-time, $.createdAt: 2024-04-22 14:52:11.402 is an invalid date-time, $.dueDate: 2024-05-16 06:00:00.000 is an invalid date-time, $.updatedAt: 2024-05-16 16:11:00.224 is an invalid date-time, $.createdAt: 2024-05-16 16:08:26.353 is an invalid date-time, $.updatedAt: 2024-05-16 16:07:17.755 is an invalid date-time, $.createdAt: 2024-05-16 16:08:26.367 is an invalid date-time, $.createdAt: 2024-05-16 16:11:00.366 is an invalid date-time, $.updatedAt: 2024-05-16 16:08:26.358 is an invalid date-time, $.updatedAt: 2024-05-16 16:09:39.673 is an invalid date-time, $.createdAt: 2024-05-16 15:12:06.904 is an invalid date-time, $.assigneeId: null found, string expected, $.onHoldScheduledEventId: null

it looks like the wrong data type is either being set, or sent. This error is happening at the Airbyte level. I'll check your database schema to see which type of error is happening.

[sneakersgames]

Is there a fix for this? I have the same issue on v0.50.35

2024-08-23 16:32:40 �[46mplatform�[0m > Schema validation errors found for stream sporza_teams. Error messages: [$.hth_points: null found, integer expected, $.league_week: null found, integer expected, $.transfers: null found, integer expected, $.league_id: null found, integer expected, $.freehit_budget: null found, number expected, $.bank: null found, integer expected, $.sponsor_id: null found, integer expected, $.triple_captain: null found, integer expected, $.points: number found, string expected, $.lat_powerup: null found, string expected, $.draftnr: null found, integer expected, $.team_stats: null found, string expected, $.prev_league: null found, integer expected, $.wildcard: null found, integer expected, $.waiver: null found, integer expected, $.prizes: null found, integer expected, $.visible: null found, integer expected, $.est_powerup: null found, string expected, $.freehit: null found, integer expected, $.payment_date: null found, string expected, $.payment_id: null found, integer expected, $.rank: null found, integer expected, $.badge: null found, string expected]