Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Using NULL for column values show up as tombstones in Cassandra #240

Closed
mrwulf opened this issue Oct 18, 2017 · 3 comments
Closed

Using NULL for column values show up as tombstones in Cassandra #240

mrwulf opened this issue Oct 18, 2017 · 3 comments
Assignees
@michaelsembwever
Milestone
1.0.0

Comments

@mrwulf
Copy link

mrwulf commented Oct 18, 2017

I'm using the cassandra storage engine with Cassandra 3.11.0. In my Cassandra logs there are lots of messages like:

WARN  [ReadStage-5] 2017-10-18 11:04:30,610 ReadCommand.java:522 - Read 1547 live rows and 3311 tombstone cells for query SELECT * FROM reaper_db.repair_run WHERE id = 71d1c7e0-b426-11e7-8477-b7c57b58f240 LIMIT 5000 (see tombstone_warn_threshold)
WARN  [ReadStage-8] 2017-10-18 11:04:30,639 ReadCommand.java:522 - Read 2306 live rows and 2838 tombstone cells for query SELECT * FROM reaper_db.repair_run WHERE id = 6c33a9c0-aae4-11e7-bc2b-155801446996 LIMIT 5000 (see tombstone_warn_threshold)
WARN  [ReadStage-2] 2017-10-18 11:04:30,784 ReadCommand.java:522 - Read 2266 live rows and 6155 tombstone cells for query SELECT * FROM reaper_db.repair_run WHERE id = 6ae688f0-b23a-11e7-9d50-79c9c6ee5ac0 LIMIT 5000 (see tombstone_warn_threshold)
WARN  [ReadStage-3] 2017-10-18 11:04:30,853 ReadCommand.java:522 - Read 2266 live rows and 6049 tombstone cells for query SELECT * FROM reaper_db.repair_run WHERE id = 99202530-b2cd-11e7-a3fa-6904da3676a4 LIMIT 5000 (see tombstone_warn_threshold)

I believe this is being caused by writing null values instead of leaving columns unset (here is a blog referencing why not to use nulls). I could just change the warn threshold, but I would prefer to store only the limited, necessary data.
@mrwulf
Copy link
Author

mrwulf commented Oct 18, 2017

Actually, TLP did an article about it as well! http://thelastpickle.com/blog/2016/09/15/Null-bindings-on-prepared-statements-and-undesired-tombstone-creation.html

@mrwulf mrwulf mentioned this issue Oct 18, 2017
Closed
@mrwulf
Copy link
Author

mrwulf commented Oct 18, 2017

Can anyone speak to why the repair_run table is used for both repairs and individual segments being repaired?

@michaelsembwever michaelsembwever self-assigned this Oct 19, 2017
@michaelsembwever michaelsembwever added this to the 0.8.0 milestone Oct 19, 2017
michaelsembwever added a commit that referenced this issue Oct 19, 2017
@michaelsembwever
For Cassandra storage minimise the number of tombstones (NULL entries…
0518ceb 
…) in the repair_run table.

Remove RepairSegment.repairCommandId as no one was using/storing it.

Make explicit which fields in RepairSegment can be null and be nulled.
Hide the RepairSegment.Builder constructor.

Add Preconditions and asserts in ensuring lifecycle state is correct and null fields only exist as/when appropriate.
Preconditions are used when fast and providing fail-fast design. Asserts are used where expensive and/or are secondary checks.

ref: #240
michaelsembwever added a commit that referenced this issue Oct 19, 2017
@michaelsembwever
For Cassandra storage minimise the number of tombstones (NULL entries…
b793304 
…) in the repair_run table.

Remove RepairSegment.repairCommandId as no one was using/storing it.

Make explicit which fields in RepairSegment can be null and be nulled.
Hide the RepairSegment.Builder constructor.

Add Preconditions and asserts in ensuring lifecycle state is correct and null fields only exist as/when appropriate.
Preconditions are used when fast and providing fail-fast design. Asserts are used where expensive and/or are secondary checks.

ref: #240
@michaelsembwever michaelsembwever mentioned this issue Oct 19, 2017
Merged
michaelsembwever added a commit that referenced this issue Oct 19, 2017
@michaelsembwever
For Cassandra storage minimise the number of tombstones (NULL entries…
f58665a 
…) in the repair_run table.

Remove RepairSegment.repairCommandId as no one was using/storing it.

Make explicit which fields in RepairSegment can be null and be nulled.
Hide the RepairSegment.Builder constructor.

Add Preconditions and asserts in ensuring lifecycle state is correct and null fields only exist as/when appropriate.
Preconditions are used when fast and providing fail-fast design. Asserts are used where expensive and/or are secondary checks.

ref:
 - #240
 - #244
michaelsembwever added a commit that referenced this issue Oct 19, 2017
@michaelsembwever
For Cassandra storage minimise the number of tombstones (NULL entries…
6322293 
…) in the repair_run table.

Remove RepairSegment.repairCommandId as no one was using/storing it.

Make explicit which fields in RepairSegment can be null and be nulled.
Hide the RepairSegment.Builder constructor.

Add Preconditions and asserts in ensuring lifecycle state is correct and null fields only exist as/when appropriate.
Preconditions are used when fast and providing fail-fast design. Asserts are used where expensive and/or are secondary checks.

ref:
 - #240
 - #244
michaelsembwever added a commit that referenced this issue Oct 19, 2017
@michaelsembwever
For Cassandra storage minimise the number of tombstones (NULL entries…
091403b 
…) in the repair_run table.

Remove RepairSegment.repairCommandId as no one was using/storing it.

Make explicit which fields in RepairSegment can be null and be nulled.
Hide the RepairSegment.Builder constructor.

Add Preconditions and asserts in ensuring lifecycle state is correct and null fields only exist as/when appropriate.
Preconditions are used when fast and providing fail-fast design. Asserts are used where expensive and/or are secondary checks.

ref:
 - #240
 - #244
michaelsembwever added a commit that referenced this issue Oct 19, 2017
@michaelsembwever
For Cassandra storage minimise the number of tombstones (NULL entries…
baf08fe 
…) in the repair_run table.

Remove RepairSegment.repairCommandId as no one was using/storing it.

Make explicit which fields in RepairSegment can be null and be nulled.
Hide the RepairSegment.Builder constructor.

Add Preconditions and asserts in ensuring lifecycle state is correct and null fields only exist as/when appropriate.
Preconditions are used when fast and providing fail-fast design. Asserts are used where expensive and/or are secondary checks.

ref:
 - #240
 - #244
michaelsembwever added a commit that referenced this issue Oct 19, 2017
@michaelsembwever
For Cassandra storage minimise the number of tombstones (NULL entries…
03b6663 
…) in the repair_run table.

Remove RepairSegment.repairCommandId as no one was using/storing it.

Make explicit which fields in RepairSegment can be null and be nulled.
Hide the RepairSegment.Builder constructor.

Re-read segments once leader-election is done, ensuring we're not writing back stale data

Add Preconditions and asserts in ensuring lifecycle state is correct and null fields only exist as/when appropriate.
Preconditions are used when fast and providing fail-fast design. Asserts are used where expensive and/or are secondary checks.

ref:
 - #240
 - #244
michaelsembwever added a commit that referenced this issue Oct 19, 2017
@michaelsembwever
For Cassandra storage minimise the number of tombstones (NULL entries…
e717ad8 
…) in the repair_run table.

Remove RepairSegment.repairCommandId as no one was using/storing it.

Make explicit which fields in RepairSegment can be null and be nulled.
Hide the RepairSegment.Builder constructor.

Re-read segments once leader-election is done, ensuring we're not writing back stale data

Add Preconditions and asserts in ensuring lifecycle state is correct and null fields only exist as/when appropriate.
Preconditions are used when fast and providing fail-fast design. Asserts are used where expensive and/or are secondary checks.

ref:
 - #240
 - #244
@michaelsembwever
Copy link
Member

fixed in #244 which is now in master.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@michaelsembwever michaelsembwever

Labels
None yet
Projects
None yet
Milestone
1.0.0
Development

No branches or pull requests
2 participants
@michaelsembwever @mrwulf