Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Localized state management, and improved partition #1158

Merged
merged 23 commits into from
Apr 1, 2019
Merged

Localized state management, and improved partition #1158

merged 23 commits into from
Apr 1, 2019

Conversation

suhothayan
Copy link
Contributor

@suhothayan suhothayan commented Mar 27, 2019 •
edited
Loading

Purpose

Improve memory management of Siddhi with minimal impact on its performance. Improve data persistence. Fix behavior of "output first every " to emit the events as soon as they arrive.

Goals

Currently partition is not scaling as needed and it's causing OOM issues, and this fixes this concern.

Approach

Remove process logic cloning and move the partition state management to individual executions elements of Siddhi.
Improve bulk data cleaning for group-by states.
Remove unnecessary synchronizations.
Improve data persistence format to first have partition ID to help cleaning and migrating data partitions.
Remove unused Stream and State event pools
Fix behavior of "output first every " to emit the events as soon as they arrive.

Release note

Rearchitected partition and state processing for better memory management.
This has backward incompatible changes to extension APIs.

Documentation

Siddhi extensions are affected due to backward incompatible changes.
Partition cleaning using @purge(enable='true', interval='1 sec', idle.period='5 sec')
Fix behavior of "output first every " to emit the events as soon as they arrive.

@suhothayan
Add state classes
29d4b1e
@suhothayan
Merge branch 'query-context' into state-partition
e9a6664
@suhothayan
Add query state management, remove cloning, improve data storage format
fc771aa 
Removing cloning of query and other Siddhi operations for partitioning
 and allow each Siddhi element to handle a collection of state based on
 partition and group id. This also allow each Siddhi element to clean
 it's state appropriately.

Making Siddhi to snapshot the data having partition and group ids as
the root elements, allowing easy data migration in the future.

Node: Though this implementation improve memory management it can
introduce performance degradation due to multiple hash map lookups.
@suhothayan
Remove redundant state management and synchronizations
12638b5 
Remove redundant state management in SingleStateHolder
and remove redundant synchronizations.
Improve length window and filter query performance to Siddhi 4.x level
@suhothayan
Fix incremental data storage, improve StateHandler to handle group by
d744e74
@suhothayan
Fix concurrency and build issues.
a92c499
@suhothayan
Fix build issues
a7fa11a
@suhothayan
Add groupBy cleaning function, improve group by perf better than 4.x.
d1bb4a6 
Introduce synchronised and unsafe state holders.
@suhothayan
Add groupBy cleaning function, improve group by perf better than 4.x.
b4e1d5f 
Introduce synchronised and unsafe state holders.
@suhothayan
Add perf tests
7f9d964
@suhothayan
Remove unnecessary synchronizations
fce0c1f
@suhothayan
Make snapshot and restore synchronized.
f498eee
@suhothayan
Optimize group by data cleaning
809c5d6
@suhothayan
Remove event pools
693948e 
Remove Stream and State Event Pools

They are not used anymore
@suhothayan
Merge remote-tracking branch 'siddhi-io/master' into state-partition
78fa7a5 
# Conflicts:
#	modules/siddhi-core/src/main/java/io/siddhi/core/util/collection/operator/IncrementalAggregateCompileCondition.java
@codecov-io
Copy link

codecov-io commented Mar 27, 2019 •
edited
Loading

Codecov Report

Merging #1158 into master will increase coverage by 1.2%.
The diff coverage is 76.18%.

Impacted file tree graph

@@             Coverage Diff             @@
##             master    #1158     +/-   ##
===========================================
+ Coverage     60.13%   61.33%   +1.2%     
  Complexity       35       35             
===========================================
  Files           612      617      +5     
  Lines         24937    24711    -226     
  Branches       3754     3841     +87     
===========================================
+ Hits          14996    15157    +161     
+ Misses         8709     8268    -441     
- Partials       1232     1286     +54
Impacted Files Coverage Δ Complexity Δ
...alCompareConditionExpressionExecutorIntDouble.java 100% <ø> (ø) 0 <0> (ø) ⬇️
...e/event/stream/converter/StreamEventConverter.java 100% <ø> (ø) 0 <0> (ø) ⬇️
...cutor/math/divide/DivideExpressionExecutorInt.java 69.23% <ø> (+8.11%) 0 <0> (ø) ⬇️
...ompareConditionExpressionExecutorDoubleDouble.java 100% <ø> (ø) 0 <0> (ø) ⬇️
...executor/math/add/AddExpressionExecutorDouble.java 90% <ø> (+30%) 0 <0> (ø) ⬇️
...uery/output/callback/InsertIntoStreamCallback.java 70.58% <ø> (ø) 0 <0> (ø) ⬇️
...ore/query/input/stream/join/JoinStreamRuntime.java 100% <ø> (ø) 0 <0> (ø) ⬇️
...re/executor/math/mod/ModExpressionExecutorInt.java 69.23% <ø> (+8.11%) 0 <0> (ø) ⬇️
...anCompareConditionExpressionExecutorLongFloat.java 100% <ø> (ø) 0 <0> (ø) ⬇️
...utor/condition/AndConditionExpressionExecutor.java 68% <ø> (+4.66%) 0 <0> (ø) ⬇️
... and 374 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1cadfab...0f5ee5f. Read the comment docs.

@mohanvive mohanvive changed the title Localized state management, and improved parti Localized state management, and improved partition Mar 28, 2019
@suhothayan
Add partition purging
f86f8ff 
with following syntax

@purge(enable='true', interval='1 sec', retention.period='1 sec')
partition with (symbol of streamA)
begin
   @info(name = 'query1')
   from streamA#window.length(3)
   select symbol, avg(price) as total
   insert into StockQuote;
end
@suhothayan
Add Session window test to the build
7e7acf4
@suhothayan
Add Delay window test to the build
39d6677
@suhothayan
Improve partition creation notification
930588b
@suhothayan
Rename partition retention.period to idle.period
fa6860d 
Also improve readability of SnapshotService
@suhothayan
Fix behaviour of "output first every <time>"
7065699 
Make it to output the first event as soon as it arrives.
@suhothayan
Fix behaviour of "output first every"
61ab808 
Make it to output the first event as soon as it arrives.
@suhothayan suhothayan mentioned this pull request Mar 31, 2019
Closed
This was referenced Apr 1, 2019
Closed
Closed
@mohanvive mohanvive merged commit c57264b into siddhi-io:master Apr 1, 2019
@niveathika niveathika mentioned this pull request Apr 17, 2019
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mohanvive mohanvive mohanvive approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@suhothayan @codecov-io @mohanvive