[Merged by Bors] - Add rowid to tables with inefficient clustered indices #5394
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## develop #5394 +/- ##
=========================================
+ Coverage 77.4% 77.6% +0.1%
=========================================
Files 265 265
Lines 30894 30928 +34
=========================================
+ Hits 23942 24012 +70
+ Misses 5431 5393 -38
- Partials 1521 1523 +2 ☔ View full report in Codecov by Sentry. |
|
what do you think about removing rowid from activesets table? |
|
@dshulyak it can be done but unless activesets are pruned, that will take outrageously long to time to complete. Also, forgot to mention that, but actually w/o updating activesets, the database size increase (before vacuum) is not that big currently. So we can either do the full update after the pruning is done and in this case I can add activesets table update to this PR, or we can update activesets in a separate PR that we can apply after pruning. |
|
we can safely drop all activesets before epoch 12 (current epoch) |
|
That will tremendously help with cache warmup step. |
dshulyak
approved these changes
Jan 15, 2024
dshulyak
reviewed
Jan 15, 2024
sql/migrations/state/0009_rowid.sql
Outdated
| @@ -0,0 +1,105 @@ | |||
| DROP INDEX atxs_by_pubkey_by_epoch_desc; | |||
There was a problem hiding this comment.
i think 0009 is taken now
ivan4th
force-pushed
the
fix/rowids
branch
2 times, most recently
from
11e890d to
adb2962
Compare
Stop using WITHOUT ROWID for tables with large row size
|
bors merge |
spacemesh-bors bot
pushed a commit
that referenced
this pull request
Jan 15, 2024
## Motivation `WITHOUT ROWID` doesn't work well for table rows over about 1/20th the size of a database page (50 bytes for 1KiB pages). This causes slower database access and increased disk usage. ## Changes Remove WITHOUT ROWID from problematic tables, add integer `n` column (also serving as rowid) for forthcoming syncv2 changes ## Test Plan **TODO**: verify that mainnet nodes function correctly after migration ## TODO - [x] Explain motivation or link existing issue(s) - [x] Test changes and document test plan - [x] Update [changelog](../CHANGELOG.md) as needed
|
Build failed: |
|
bors merge |
spacemesh-bors bot
pushed a commit
that referenced
this pull request
Jan 16, 2024
## Motivation `WITHOUT ROWID` doesn't work well for table rows over about 1/20th the size of a database page (50 bytes for 1KiB pages). This causes slower database access and increased disk usage. ## Changes Remove WITHOUT ROWID from problematic tables, add integer `n` column (also serving as rowid) for forthcoming syncv2 changes ## Test Plan **TODO**: verify that mainnet nodes function correctly after migration ## TODO - [x] Explain motivation or link existing issue(s) - [x] Test changes and document test plan - [x] Update [changelog](../CHANGELOG.md) as needed
|
Pull request successfully merged into develop. Build succeeded: |
spacemesh-bors
bot
changed the title
ivan4th
added a commit
that referenced
this pull request
Jan 16, 2024
`WITHOUT ROWID` doesn't work well for table rows over about 1/20th the size of a database page (50 bytes for 1KiB pages). This causes slower database access and increased disk usage. Remove WITHOUT ROWID from problematic tables, add integer `n` column (also serving as rowid) for forthcoming syncv2 changes **TODO**: verify that mainnet nodes function correctly after migration - [x] Explain motivation or link existing issue(s) - [x] Test changes and document test plan - [x] Update [changelog](../CHANGELOG.md) as needed
ivan4th
added a commit
that referenced
this pull request
Jan 16, 2024
`WITHOUT ROWID` doesn't work well for table rows over about 1/20th the size of a database page (50 bytes for 1KiB pages). This causes slower database access and increased disk usage. Remove WITHOUT ROWID from problematic tables, add integer `n` column (also serving as rowid) for forthcoming syncv2 changes **TODO**: verify that mainnet nodes function correctly after migration - [x] Explain motivation or link existing issue(s) - [x] Test changes and document test plan - [x] Update [changelog](../CHANGELOG.md) as needed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
WITHOUT ROWIDdoesn't work well for table rows over about 1/20th the size of a database page (50 bytes for 1KiB pages). This causes slower database access and increased disk usage.Changes
Remove WITHOUT ROWID from problematic tables, add integer
ncolumn (also serving as rowid) for forthcoming syncv2 changesTest Plan
TODO: verify that mainnet nodes function correctly after migration
TODO