fix: Improve caching mechanism for URL indexing status #75
+47
−9
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Modify shouldRecheck function to always process non-indexed URLs - Ensure "Submitted and Indexed" URLs are only rechecked after cache timeout - Optimize batch processing to avoid unnecessary API calls
goenning
reviewed
Sep 27, 2024
src/index.ts
Outdated
| @@ -120,11 +120,20 @@ export const index = async (input: string = process.argv[2], options: IndexOptio | |||
| ]; | |||
|
|
|||
| const shouldRecheck = (status: Status, lastCheckedAt: string) => { | |||
| const shouldIndexIt = indexableStatuses.includes(status); | |||
| if (status !== Status.SubmittedAndIndexed) { | |||
There was a problem hiding this comment.
this seems wrong, there are some statuses that should not force a recheck
or am I missing something?
There was a problem hiding this comment.
Ah you're right, I overfit to my own personal issue.
My cache was saving "status": "Not found (404)", which is not currently part of the Status enum, so those were being skipped.
I should rather just add a new Status.NotFound to indexableStatus.
However, such things can be fixed quite quickly but because of the long cacheTimeout they'd be ignored for 2 weeks. Perhaps we can have a shorter timeout for NotFound or PageWithRedirect.
What do you think?
…ound) - Maintain efficient long-term caching for stable, indexed URLs - add NotFound to Status enum
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What did I change?
Why did I change it?
Every run on the script was trying the process the same urls and then timing out.
Related: #51 #59