Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update migrate_to_v1_vectorstore.py #1198

Merged
merged 1 commit into from
Oct 4, 2024
Merged

Conversation

Yash-2707
Copy link
Contributor

@Yash-2707 Yash-2707 commented Oct 2, 2024

Enhancement made in the code by error handling and logging batch processing and new functionalities like backup , progress tracking
Title: Enhance Vector Migration with Error Handling, Logging, Backup, and Progress Tracking

Description:

This PR introduces several enhancements to the vector migration scripts, focusing on robustness, maintainability, and user experience.

What kind of change does this PR introduce?

Feature: Adds new functionalities like backup, progress tracking, and improved error handling.
Enhancement: Improves existing code with better logging and batch processing.
Why was this change needed?

The original migration scripts lacked error handling and logging, making it difficult to diagnose issues during migration.
There was no backup mechanism, which posed a risk of data loss during migrations.
Lack of progress tracking made it difficult to estimate the migration progress.
The code could benefit from batch processing optimizations for MongoDB operations.
Other information:

This PR addresses these issues by:
Implementing detailed logging using Python's logging module.
Adding a backup mechanism to create copies of collections before migration.
Introducing progress bars using the tqdm library to provide visual feedback during migration.
Enhancing error handling to capture and log exceptions.
Optimizing MongoDB operations for batch processing.
These changes make the migration scripts more reliable, easier to maintain, and provide a better user experience.

  • What kind of change does this PR introduce? (Bug fix, feature, docs update, ...)

  • Why was this change needed? (You can also link to an open issue here)

  • Other information:

Enhancement made in the code by  error handling and logging batch processing and new functionalities like backup , progress tracking
Copy link

vercel bot commented Oct 2, 2024

@Yash-2707 is attempting to deploy a commit to the Arc53 Team on Vercel.

A member of the Team first needs to authorize it.

Copy link

vercel bot commented Oct 4, 2024

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
docs-gpt ✅ Ready (Inspect) Visit Preview 💬 Add feedback Oct 4, 2024 5:17pm

@dartpain dartpain merged commit 7859a0d into arc53:main Oct 4, 2024
3 of 4 checks passed
@dartpain
Copy link
Contributor

@holopin-bot @Yash-2707 Thank you!

Copy link

holopin-bot bot commented Oct 21, 2024

Congratulations @Yash-2707, the maintainer of this repository has issued you a badge! Here it is: https://holopin.io/claim/cm2jeiyn304240cjv9z5soef3

This badge can only be claimed by you, so make sure that your GitHub account is linked to your Holopin account. You can manage those preferences here: https://holopin.io/account.
Or if you're new to Holopin, you can simply sign up with GitHub, which will do the trick!

@dartpain
Copy link
Contributor

Hey @Yash-2707 please fill in this form if you are interested in a t-shirt
https://airtable.com/appikMaJwdHhC1SDP/pagoblCJ9W29wf6Hf/form

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
scripts Scripts
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants