feat(source-errors): Percolates and fixes source errors #28356
Conversation
phixMe
changed the title
|
Size Change: 0 B Total Size: 1.18 MB ℹ️ View Unchanged
|
| .filter(should_sync=True) | ||
| .filter( | ||
| Q(should_sync=True) | Q(latest_error__isnull=False) | ||
| ) # OR to include schemas with errors or marked for sync |
There was a problem hiding this comment.
@Gilbert09 This does change what is returned in a test since it includes records with non-null errors. I think this is what we want since the latest_error will get nulled out when the connection is rectified.
There was a problem hiding this comment.
I guess my question here is whether we want the source to say Error when the errored schema is now disabled?
There was a problem hiding this comment.
Maybe we add a new visual state, like SUSPENDED or something like that?
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
There was a problem hiding this comment.
PR Summary
This PR enhances error reporting and visibility for external data sources in PostHog's data warehouse integration.
- Added
latest_errorfield propagation from jobs through schemas to sources in/posthog/warehouse/api/external_data_source.py - Implemented error tooltips in status columns for both source and schema tables in
/frontend/src/scenes/data-warehouse/settings/DataWarehouseManagedSourcesTable.tsx - Fixed status fallback behavior to prevent incorrect "Running" state when errors occur
- Modified schema prefetch logic to include errored schemas even when sync is disabled
- Added proper error state synchronization between jobs, schemas and sources in
/posthog/warehouse/external_data_source/jobs.py
6 file(s) reviewed, 3 comment(s)
Edit PR Review Bot Settings | Greptile
frontend/src/scenes/data-warehouse/settings/DataWarehouseManagedSourcesTable.tsx
Outdated
Show resolved
Hide resolved
frontend/src/scenes/data-warehouse/settings/DataWarehouseManagedSourcesTable.tsx
Outdated
Show resolved
Hide resolved
# Conflicts: # frontend/src/scenes/data-warehouse/settings/DataWarehouseManagedSourcesTable.tsx
📸 UI snapshots have been updated1 snapshot changes in total. 0 added, 1 modified, 0 deleted:
Triggered by this commit. |
📸 UI snapshots have been updated1 snapshot changes in total. 0 added, 1 modified, 0 deleted:
Triggered by this commit. |
Problem
Users currently have no idea why a source is not working since we don't really tell them anything regarding a failure. I also noticed some issues about percolating data between the source, schema, and job db tables and synced up the schema and job.
I also noticed that we fallback to an erroneous
Runningstate whenshould_syncgets disabled by an error that cannot be retried which creates confusing visual experience.Schema Page (w/ suspended state)
Changes
Does this work well for both Cloud and self-hosted?
Yes
How did you test this code?
Tried with failing connection a couple different ways