fix: adjust storage options for MapeoManager and MapeoProject #235
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
achou11
force-pushed
the
226/storage-opts-update
branch
from
42758ad to
68adf44
Compare
achou11
commented
Aug 29, 2023
src/mapeo-project.js
Outdated
Comment on lines
65
to
82
| const coreManagerStorage = | ||
| storagePath !== undefined | ||
| typeof storage === 'string' | ||
| ? (name) => | ||
| new RandomAccessFile( | ||
| path.join(storagePath, CORE_STORAGE_FOLDER_NAME, name), | ||
| path.join(dbPath, storage, CORE_STORAGE_FOLDER_NAME, name), | ||
| { pool: filePool } | ||
| ) | ||
| : () => new RAM() | ||
| : (name) => storage(path.join(CORE_STORAGE_FOLDER_NAME, name)) | ||
|
|
||
| /** @type {ConstructorParameters<typeof DataStore>[0]['storage']} */ | ||
| const indexerStorage = | ||
| storagePath !== undefined | ||
| typeof storage === 'string' | ||
| ? (name) => | ||
| new RandomAccessFile( | ||
| path.join(storagePath, INDEXER_STORAGE_FOLDER_NAME, name), | ||
| path.join(dbPath, storage, INDEXER_STORAGE_FOLDER_NAME, name), | ||
| { pool: filePool } | ||
| ) | ||
| : () => new RAM() | ||
| : (name) => storage(path.join(INDEXER_STORAGE_FOLDER_NAME, name)) |
There was a problem hiding this comment.
If a path is passed, use a default random-access-file storage function
Inside MapeoProject, the storage function will still need to be wrapped to separate index storage from core storage.
Made the guess that these points applied here, but could be mistaken
src/mapeo-project.js
Outdated
| * @param {import('@mapeo/crypto').KeyManager} opts.keyManager mapeo/crypto KeyManager instance | ||
| * @param {Buffer} opts.projectKey 32-byte public key of the project creator core | ||
| * @param {Buffer} [opts.projectSecretKey] 32-byte secret key of the project creator core | ||
| * @param {Partial<Record<import('./core-manager/index.js').Namespace, Buffer>>} [opts.encryptionKeys] Encryption keys for each namespace | ||
| * @param {import('drizzle-orm/better-sqlite3').BetterSQLite3Database} opts.sharedDb | ||
| * @param {IndexWriter} opts.sharedIndexWriter | ||
| * @param {import('./types.js').StorageParam} opts.storage Folder to store all hypercore data | ||
| * @param {import('./core-manager/random-access-file-pool.js').RandomAccessFilePool} [opts.filePool] FilePool instance to use if opts.storage is a path |
There was a problem hiding this comment.
Create a single instance of RandomAccessFilePool in MapeoManager and use that for all storages
is this aligned with what you described here? wasn't explicitly noted to create an opt for MapeoProject but seemed to make sense given the changes required by other points
achou11
commented
Aug 29, 2023
src/mapeo-manager.js
Outdated
Comment on lines
65
to
68
| ? () => ({ | ||
| dbPath: ':memory:', | ||
| storage: () => new RAM(), | ||
| }) |
There was a problem hiding this comment.
have a feeling that this shouldn't be a function that creates a new RAM instance each time a project is created. is the idea to share the ram instance across all projects too?
|
Not quite what I meant, sorry the issue wasn't clear. I pushed 28d4f89 to show what I meant. I think this is done now, can you review? |
Corestore already creates a `core` folder, which resulted in weird paths `cores/cores/...`, so renamed the folder to be `corestore/cores`
There are a few things that Hypercore does in its default storage for managing lock files and sparse files https://github.com/holepunchto/hypercore/blob/7c9f8b07b33e60753470728b4a9a11bcb549e158/index.js#L177
|
Should merge and release digidem/multi-core-indexer#17 before merging, so that we can update the snapshot and fix the storage locations |
gmaclennan
requested changes
Aug 30, 2023
gmaclennan
approved these changes
Aug 30, 2023
|
your changes lgtm! thanks for making the fixes - was hoping to do this to take something off your plate but ended up with you doing most of the work anyways 😄 |
achou11
changed the title
gmaclennan
added a commit
that referenced
this pull request
Sep 6, 2023
* main: (25 commits) add initial implementation of MemberApi (#232) feat: $blobs.getUrl and $blobs.create methods (#184) chore: update manager e2e tests (#237) feat: add capabilities (#231) feat: coreOwnership integration [3/3] (#230) feat: CoreOwnership class w getOwner & getCoreKey [2/3] (#229) feat: handle `coreOwnership` records in `IndexWriter` [1/3] (#214) fix: adjust storage options for MapeoManager and MapeoProject (#235) implement addProject method for MapeoManager class (#215) implement listProjects method for MapeoManager class (#208) feat: expose blobStore.writerDriveId (#219) implement wrapper client containing createProject and getProject methods (#199) add project settings functionality to MapeoProject (#187) feat: Add encode/decode for project keys [3/3] (#203) feat: update protobuf for RPC [2/3] (#202) chore: move protobuf messages into src/generated [1/3] (#201) feat: Add internal `dataType.createWithDocId()` (#192) chore: explicitly set "mode" opt for encryptionKeys column creation (#186) chore: fix linting and type checking (#183) chore: consolidate encryption key columns in projectKeys table (#181) ...
This was referenced Nov 16, 2023
This was referenced Apr 11, 2024
This was referenced Jun 24, 2024
This was referenced Jul 25, 2024
This was referenced Aug 21, 2024
This was referenced Aug 29, 2024
This was referenced Sep 11, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #226
Tried my best to follow the checklist but there were some uncertainties, mostly around handling the
storageopt inMapeoProject