New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Speed up lookup #1

Open

alexfikl opened this issue Nov 10, 2023 · 0 comments

Owner

alexfikl commented Nov 10, 2023

Currently we're storing the UEFISCDI databases as list[{"name": ...,}, ...] in JSON files.

When matching a document, we would like to

Match the ISSN / eISSN exactly if available
Fuzzy match the journal name (generally the UEFISCDI names seem to be ASCII and missing apostrophes and stuff).

Requirements:

Fast load the file from disk.
Fast access and fuzzy search

Ideas:

Currently just using difflib.SequenceMatcher for the fuzzy matching. A bit slow, but not horrible.
Just pickle the database? Not sure if much faster..
Can we use SQLite? Can we convince FTS5 to be fuzzy?
What other storage formats would work for this? Ideally without a big dependency.

The text was updated successfully, but these errors were encountered:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment