Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(projects): Treat fetch failures as pending #4140

Merged
merged 3 commits into from
Dec 3, 2024

Conversation

jjbayer
Copy link
Member

@jjbayer jjbayer commented Oct 15, 2024

When we fail to fetch a project state for any reason (max retries exceeded, redis error), we should not mark the project as invalid.

Currently we treat the project as if the DSN was invalid, and respond to clients with a 403 response. This might have made sense as a defensive measure in a pre-spooling world where envelopes could not be buffered for a longer period of time, but with spooling and time-based envelope eviction, we can now safely buffer them.

ref: INC-888

blocked by: https://github.com/getsentry/team-ingest/issues/345

@jjbayer jjbayer self-assigned this Oct 15, 2024
time.sleep(1) # Wait for project to be cached

# Relay still accepts events for this project
next_response = relay.send_event(42)
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With the old version, this call raises a 403 error.

@jjbayer jjbayer marked this pull request as ready for review October 15, 2024 08:39
@jjbayer jjbayer requested a review from a team as a code owner October 15, 2024 08:39
Copy link
Member

@Dav1dde Dav1dde left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This does have the downside of potentially exhausting buffers if a project consistently cannot be computed. Do we need more alerts for this case?

relay-server/src/services/projects/cache.rs Outdated Show resolved Hide resolved
@jjbayer
Copy link
Member Author

jjbayer commented Oct 15, 2024

This does have the downside of potentially exhausting buffers if a project consistently cannot be computed. Do we need more alerts for this case?

Good point, we should alert when the buffer gets close to its total capacity, or implement @iambriccardo 's automatic eviction policy. I added a task to the epic.

relay-server/src/services/projects/cache.rs Outdated Show resolved Hide resolved
@jjbayer
Copy link
Member Author

jjbayer commented Oct 15, 2024

Note: not merging this until https://github.com/getsentry/team-ingest/issues/345 is mature.

jjbayer added a commit that referenced this pull request Oct 15, 2024
Measure impact before we merge
#4140.
@jjbayer jjbayer force-pushed the fix/project-failure-is-pending branch from 688d23e to 2970ed0 Compare December 3, 2024 08:38
@jjbayer jjbayer merged commit 8667f9d into master Dec 3, 2024
23 checks passed
@jjbayer jjbayer deleted the fix/project-failure-is-pending branch December 3, 2024 09:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants