Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Sql registry optimization #133

Closed
wants to merge 1 commit into from
Closed

Conversation

EXPEbdodla
Copy link
Collaborator

@EXPEbdodla EXPEbdodla commented Aug 19, 2024

What this PR does / why we need it:

Problem Summary: We are using SQL Registry with 150+ projects and ~600 feature views at the moment. They are going to grow in future.
1 . Starting up a Remote registry service is taking longer than expected due to proto() call during Registry initialization. This has to refresh all 150 projects metadata on start up.
2 . Refreshing the cache on a regular interval with cache_mode = thread taking longer even though most of the projects object (like entities, FVs, ODFVs) doesn't change so often.

Fixes:

  1. Running the proto() method using ThreadPoolExecutor to run in parallel with 5 default workers. This doesn't fully solve the problem. We may need to look into the lazy caching options which reduces the start up times and load only the projects for which it's serving the calls rather than loading all projects into the cache. Along with that we may need to control the size of cache_registry_proto object with some eviction policy for the cache.
  2. Leveraging the previously cached RegistryProto if the last_updated_timestamp of the project hasn't changed in the feast_metadata table. Reduces the number of database calls we make.
  3. Adding last_updated_timestamp column to ProjectMetadata table
  4. Added Indexes to the tables
  5. _maybe_init_project_metadata is called multiple times (Ex: In get, list and apply). Proto() method depends on list calls (~10 calls per project. With 150 projects, it just makes the function call 1500 times which is not needed)
  6. _get_all_projects() method is using individual tables to get the projects list instead of using feast_metadata table
  7. Added get_all_projects(), get_project_metadata() and delete_project() methods to SQL Registry.
  8. Updated test case for ODFV backward compatibility issue
  9. Added tests cases to ProjectMetadata object
  10. Added integration tests for the changes
  11. Switched sqlite fixture scope to "function". sqlite is not thread safe so added thread_pool_executor_worker_count configuration. If its 0, doesn't use ThreadPoolExecutor.

Questions:

  1. It make sense to refresh the proto() with all projects for remote registry service if SQL registry is the proxy. If users pass the project_name as part of Repo Config, proto() can only refresh the project passed. I'm not sure how to make the distinction when registry is a Proxy registry for Registry Service vs Regular Registry. Appreciate any thoughts on this.

Which issue(s) this PR fixes:

Misc

@EXPEbdodla EXPEbdodla force-pushed the sql_registry_optimization branch 2 times, most recently from 314e794 to cc6571b Compare August 20, 2024 20:14
@EXPEbdodla EXPEbdodla force-pushed the sql_registry_optimization branch 3 times, most recently from 1183040 to 9369d3f Compare August 20, 2024 23:07
@EXPEbdodla EXPEbdodla force-pushed the sql_registry_optimization branch from f6aefa8 to 3a68274 Compare August 20, 2024 23:39
@EXPEbdodla EXPEbdodla closed this Oct 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant