feat: complex docs (properties that are arrays or objects) #14
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This should match the transform done in mapeo core when reading from the db, so that the read documents have the same shape/type as the docs going in
gmaclennan
added a commit
to digidem/comapeo-core
that referenced
this pull request
Aug 5, 2023
Depends on: - [ ] digidem/mapeo-sqlite-indexer#11 - [ ] digidem/mapeo-sqlite-indexer#14 Makes more sense to e2e test rather than unit test, which depends on the new DataType class, which is next up.
This was referenced Aug 5, 2023
gmaclennan
added a commit
to digidem/comapeo-core
that referenced
this pull request
Aug 7, 2023
Depends on: - [ ] digidem/mapeo-sqlite-indexer#11 - [ ] digidem/mapeo-sqlite-indexer#14 Makes more sense to e2e test rather than unit test, which depends on the new DataType class, which is next up.
gmaclennan
changed the title
achou11
approved these changes
Aug 7, 2023
gmaclennan
added a commit
to digidem/comapeo-core
that referenced
this pull request
Aug 7, 2023
Depends on: - [ ] digidem/mapeo-sqlite-indexer#11 - [ ] digidem/mapeo-sqlite-indexer#14 Makes more sense to e2e test rather than unit test, which depends on the new DataType class, which is next up.
3 tasks
gmaclennan
added a commit
to digidem/comapeo-core
that referenced
this pull request
Aug 9, 2023
…er (#149) * feat: Generate SQL table schema from JSON Schemas Squashed commits: [a8267eb] Switch moduleResolution so @digidem/types work [1205a2b] Fixes and add forks [981070a] Add tests and fix accordingly [2dac678] Add tests and fixes [01ac065] add types for brittle [537ff7a] Add project, preset, field schemas [ad186ae] cleanup code and add comments [c23cf09] fix: extract correct schema w/out schemaName param [d8786c8] fix: docId as primary key [36eda92] fix enum type [f0c7eed] Remove db artifact [89468e2] WIP: Generate SQL table schema from JSON Schemas * skip test that is only for types TODO: Figure out best way to test types * new DataStore * ts-ignore to ts-expect-error Co-authored-by: Andrew Chou <[email protected]> * use versionId utils from @mapeo/schema * remove .solo * feat: Add IndexWriter for all things indexy Depends on: - [ ] digidem/mapeo-sqlite-indexer#11 - [ ] digidem/mapeo-sqlite-indexer#14 Makes more sense to e2e test rather than unit test, which depends on the new DataType class, which is next up. * Update to published @mapeo/[email protected] * new DataType * stricter DataStore types Ensures you can't create a dataType in the wrong DataStore * pass IndexWriter to DataStore, not indexEntries fn - More explicit relationship between indexWriter and Datastore - Allows some type-checking to ensure IndexWriter can process entries * Pass storage param to data store * chore: consistent naming - better-sqlite3 instances = sqlite - drizzle table definitions = table * ensure core.key is always set * Add drizzle migration files * Add MapeoProject class and update e2e tests * tests for updatedAt and createdAt * Explicit max file descriptors * rename test-e2e/project-crud.js --------- Co-authored-by: Andrew Chou <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
SQLite can only store numbers (INTEGER or REAL) or strings (TEXT): https://www.sqlite.org/datatype3.html
Nullable fields are set to NULL if missing.
In Mapeo Core we use Drizzle and custom fields to map JSONSchema types to SQLite types and back. This module needs to do the same.
I initially attempted to do this by integrating Drizzle into here, but that led to a 15x slowdown and issues with Drizzle. This PR instead does the conversion based on incoming data type and the column type read from the DB schema.
Because we use prepared statements, we need to pass a value for every property (even optional props), therefore the transform does:
nullif no default set)1(true) or0(false) in an integer collumnThe indexer does not do reads, so it's only writes that matter.