Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Temperature-aware data partitioning #348

Merged
merged 178 commits into from
Oct 18, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
178 commits
Select commit Hold shift + click to select a range
b94f97a
added neew module monitoring
hennlo Mar 5, 2021
8d46a61
added inital depenedencies
hennlo Mar 5, 2021
cc34b2c
- Added InfluxDB dependency to gradle
Mar 5, 2021
bad8f95
- Support Read for InfluxDB
Mar 12, 2021
f42c2df
Merge branch 'master' into workload_monitoring
hennlo Mar 19, 2021
1b4da59
added statistics backend abstraction layer
hennlo Mar 20, 2021
9a8fb7b
added MonitoringEvent to track events
hennlo Mar 20, 2021
75bbff3
added backgroundjobs for monitoring
hennlo Mar 20, 2021
cba2d29
added mapDB for persisting the monitoring queue
hennlo Mar 21, 2021
0b2bf76
Added conenctrion to simple monitoring backend
hennlo Apr 1, 2021
3c1f079
added refreshable information page
hennlo Apr 2, 2021
e0c8f68
added neew module monitoring
hennlo Mar 5, 2021
df9fd40
added inital depenedencies
hennlo Mar 5, 2021
b774bc6
- Added InfluxDB dependency to gradle
Mar 5, 2021
2cb4bf1
- Support Read for InfluxDB
Mar 12, 2021
a0e8db6
added statistics backend abstraction layer
hennlo Mar 20, 2021
001387f
added MonitoringEvent to track events
hennlo Mar 20, 2021
2095806
added backgroundjobs for monitoring
hennlo Mar 20, 2021
767a46a
added mapDB for persisting the monitoring queue
hennlo Mar 21, 2021
b0048de
Added conenctrion to simple monitoring backend
hennlo Apr 1, 2021
402d84f
added refreshable information page
hennlo Apr 2, 2021
607d633
Merge remote-tracking branch 'origin/workload_monitoring' into worklo…
Apr 5, 2021
d1ee5f6
moved LogicalTable/Schema from dbms to core
hennlo Apr 5, 2021
d52a2ff
remove persistence from eventQueue
hennlo Apr 10, 2021
f923a1b
Merge branch 'master' into workload_monitoring
hennlo Apr 10, 2021
f1194e5
added table identification to monitor event
hennlo Apr 11, 2021
6de6569
chaged query information retireval to logicaltable
hennlo Apr 15, 2021
8d4134a
Added Event Subscription structure
hennlo Apr 16, 2021
e6facb1
added event brokerage
hennlo Apr 17, 2021
47478b4
extended Broker registritation
hennlo Apr 18, 2021
6ac17de
- Refactor MonitoringService.
Apr 22, 2021
3f985c0
- some refactoring and cleanup
Apr 23, 2021
8bff44f
- some bugfixed
Apr 23, 2021
168f80a
- refactor Transaction to use only MonitoringData and not whole Monit…
Apr 23, 2021
ee33001
- some classes moved to other packages
Apr 23, 2021
7d2ff42
Start Monitoring Service instantly
hennlo Apr 25, 2021
5bd8451
added comemnts and information
hennlo Apr 25, 2021
1755c9e
merge master
hennlo Apr 27, 2021
f7fa191
deactivate metric gathering
hennlo Apr 28, 2021
fc45776
- Change of concept to use metrics
Apr 28, 2021
fa0f4c3
initial structure of temp-awareness
hennlo Apr 29, 2021
5189b91
class cleanup
hennlo Apr 29, 2021
9dec4d7
fixed Event inheritance
hennlo Apr 30, 2021
2cb2464
added queue handling
hennlo May 1, 2021
a34e7bc
added DML event
hennlo May 1, 2021
4b2f9cf
some Feedback, discussion
May 3, 2021
7937eb4
- rename MonitoringMetric to MonitoringDataPoint
May 6, 2021
6db9080
improved event to transaction handling
hennlo May 8, 2021
929803b
added execution time to dml
hennlo May 8, 2021
a96c7e7
Merge branch 'workload_monitoring' into temp-aware_partitioning
hennlo May 8, 2021
ca45281
added initial modal for Temp-aprt
hennlo May 9, 2021
fe93550
added partition selection to monitor event
hennlo May 10, 2021
96355a4
extension to UI partition-creation for number of partitions
hennlo May 17, 2021
ea8b2df
refactorded monitoring module
hennlo May 17, 2021
37e5208
refactored partioning form core to dbms
hennlo May 17, 2021
83e3411
refactored partition to partitionGroup
hennlo May 17, 2021
ba01a25
fixed a major bug when partioned tables are reset when renamed
hennlo May 23, 2021
dfa4f9d
fixed bug when partitoned table ismerged, can't be dropped afterwards
hennlo May 23, 2021
0a3400c
fixed a bug when Unbound partGroup wouldn't create internal
hennlo May 24, 2021
61dcf99
modified retrieval of targetPartitionID to internal partition handling
hennlo May 24, 2021
dc8021d
added PartitionProperty for more sophisticated and variable partition…
hennlo May 24, 2021
e0df093
refactored schema mapping and physical table creation
hennlo Jun 30, 2021
d55d951
Merge branch 'master' into workload_monitoring
Jul 2, 2021
ea206c9
adapted dataMigrator to physicaltables
hennlo Jul 5, 2021
1ca8547
corrected physical execution with CCPs
hennlo Jul 5, 2021
3441b27
fixed bug with ambigious physical table create
hennlo Jul 6, 2021
f326761
switched to internal partition routing instead of partitionGroup
hennlo Jul 7, 2021
39e8e3e
querying on different physical partitions
hennlo Jul 8, 2021
e9799e2
removed most od worst case routing
hennlo Jul 9, 2021
d9f5e35
added frequency map for temp
hennlo Jul 11, 2021
25c4cfe
fixed a monitroing bug
hennlo Jul 11, 2021
e949566
added TEMPERATURE parsing & property generation
hennlo Jul 11, 2021
2bca412
optimized freqeuncy check
hennlo Jul 13, 2021
94cbe96
implemented partitioning via webui
hennlo Jul 14, 2021
783d6d4
extended ferqeuncyMap
hennlo Jul 14, 2021
c637b3e
implemented automatic physical table create/drop with copyData when p…
hennlo Jul 15, 2021
9025c7a
fixed bug in physical table adjustment
hennlo Jul 15, 2021
78707a5
fixed smaller changes
hennlo Jul 16, 2021
130d3b3
fixed a bug with partition distribution constraints
hennlo Jul 16, 2021
8611619
added temperature data copy
hennlo Jul 16, 2021
43242ec
optimized update handling
hennlo Jul 17, 2021
afdda3b
optimized temperature frequency ALL
hennlo Jul 17, 2021
c2a06e3
Merge branch 'master' into workload_monitoring
hennlo Jul 18, 2021
7d079a8
fixed bug in REST monitor
hennlo Jul 18, 2021
579c5aa
removed unused test
hennlo Jul 18, 2021
1da18fe
added configuration
hennlo Jul 18, 2021
ffbcce0
merged workload monitoring
hennlo Jul 19, 2021
36a8e2a
fixed merge bugs
hennlo Jul 19, 2021
e28efe2
added batch insrt for partitioning
hennlo Jul 20, 2021
8094968
Fix order of parameters in REST request parser
vogti Jul 21, 2021
8edd3ca
merged master into feature
hennlo Jul 21, 2021
079015b
Merge branch 'temp-aware_partitioning' of https://github.com/polyphen…
hennlo Jul 21, 2021
2d88214
extended taskscheduling
hennlo Jul 21, 2021
47a80c9
fixed errors in partition merge
hennlo Jul 21, 2021
1870a2a
added addiitonal partitioning tests
hennlo Jul 22, 2021
982592b
fixed test bugs
hennlo Jul 22, 2021
4c1d43f
remove physicalTableName for CCP
hennlo Jul 22, 2021
2f6a0b2
fixed some failing tests
hennlo Jul 22, 2021
c8733b1
fixed a bug with prepared statements
hennlo Jul 25, 2021
cc60a9a
added constraints on tests
hennlo Jul 26, 2021
524ce42
merged changes from monitoring
hennlo Jul 27, 2021
c09ca97
fixed bug in table creation of mongo and cottontail
hennlo Aug 3, 2021
72c3936
fixed minor ddl adjustment
hennlo Aug 3, 2021
c3a8b73
fixed issue with pushdown of joins
hennlo Aug 14, 2021
bee5662
fixed a bug with insert lookup error
hennlo Aug 16, 2021
5050f2c
fixed insert bug
hennlo Aug 18, 2021
e3a73d2
workaround for partitioned select
hennlo Aug 24, 2021
db956a0
fixed cottontail schema error
hennlo Aug 24, 2021
7ffb3b9
remove workaround
hennlo Aug 26, 2021
b34b1fa
fixed alias bug
hennlo Aug 26, 2021
f691ad7
Adjust formatting and year in copyright
vogti Sep 1, 2021
47abcaf
Import some changes from master
vogti Sep 1, 2021
89ce72c
Fix issue with queries on partitioned tables
vogti Sep 30, 2021
4baebfc
Merge branch 'master' into temp-aware_partitioning
hennlo Oct 1, 2021
83e8463
merged master
hennlo Oct 2, 2021
a65b237
Minor changes and fixed typos
vogti Oct 2, 2021
5413f69
Minor improvements
vogti Oct 2, 2021
b1b584c
fixed bug with partition merge
hennlo Oct 2, 2021
983b400
Merge remote-tracking branch 'origin/temp-aware_partitioning' into te…
hennlo Oct 2, 2021
bfdc34b
fixed minor bugs
hennlo Oct 3, 2021
358f3b1
Enable ComplexViewTest
vogti Oct 4, 2021
46c36c4
Remove debugging output, enable tests and minor changes to formatting
vogti Oct 4, 2021
f28733d
Fix support for partitioning on file store
vogti Oct 4, 2021
bf62922
Fix data sources
vogti Oct 4, 2021
e2f9f41
Fix names in ethereum data source
vogti Oct 4, 2021
85a7673
Undo changes to REST auth
vogti Oct 5, 2021
9bb775b
fixed bugs with partition merge
hennlo Oct 6, 2021
173ffc9
corrected testcase
hennlo Oct 7, 2021
a5c17e5
excluded cassandra partition tests
hennlo Oct 7, 2021
6c91b6d
Verify connection when deploying a Postgres or MonetDB store
vogti Oct 8, 2021
5b651c7
implemented datamigration for partitioned tables
hennlo Oct 9, 2021
765f738
improve partition handling
hennlo Oct 10, 2021
84de464
Catch batch update and delete statements on file store
vogti Oct 10, 2021
fa0bf5d
fixed a bug when partition table would remove rows
hennlo Oct 10, 2021
2e772d7
Merge remote-tracking branch 'origin/temp-aware_partitioning' into te…
hennlo Oct 10, 2021
0de3aa9
addeddatamigration tests for partitioning
hennlo Oct 10, 2021
b0bb38c
Fix support for batched inserts on horizontally partitioned tables
vogti Oct 11, 2021
f6fc604
Fix issues with data migration
vogti Oct 12, 2021
a063c12
Minor improvements
vogti Oct 12, 2021
b67fa8c
Improve DML performance
vogti Oct 12, 2021
70732ad
Minor changes to code formatting
vogti Oct 12, 2021
2988fd6
Minor code improvements
vogti Oct 12, 2021
992cf45
Improve code formatting
vogti Oct 12, 2021
1d0839f
Clean-up partition managers
vogti Oct 12, 2021
2756469
Restore order of methods in AbstractQueryProcessor
vogti Oct 12, 2021
12e1567
Batch UPDATE and DELETE
vogti Oct 13, 2021
22acf07
Test for multi value insert
vogti Oct 13, 2021
676e991
fixed bug in datamigrator when merging tables
hennlo Oct 15, 2021
b842b12
added multiInsert handling
hennlo Oct 15, 2021
8ea80a8
Added another test for multi value insert
hennlo Oct 15, 2021
d6a53bc
Added more tests for range partitioning
hennlo Oct 15, 2021
136285b
todo cleanup
hennlo Oct 15, 2021
b5af610
fixed psql index drop
hennlo Oct 15, 2021
c185405
added docs
hennlo Oct 16, 2021
db276ad
code cleanup
hennlo Oct 16, 2021
8435252
added LogicalFilter parsing
hennlo Oct 16, 2021
6fd5360
cleanup comments
hennlo Oct 16, 2021
9a8eb25
Some minor clean-up
vogti Oct 17, 2021
ace7c7d
Further improve DDL performance
vogti Oct 17, 2021
e9406f8
Fix issues with multi inserts on partitioned tables
vogti Oct 17, 2021
a565517
Fix bug in HorizontalPartitioningTest
vogti Oct 17, 2021
3cd9509
Fix issue with file store
vogti Oct 17, 2021
5370ae7
Trigger adapter schema creation in ddl manager
vogti Oct 17, 2021
73d873f
minor cleanups
hennlo Oct 17, 2021
047e7cf
Minor clean-up
vogti Oct 17, 2021
ed93087
Minor improvements to code formatting
vogti Oct 17, 2021
b9c9383
Fix formatting
vogti Oct 17, 2021
9f47b19
added JavaDoc for catalog methods
hennlo Oct 17, 2021
ca70bab
Merge remote-tracking branch 'origin/temp-aware_partitioning' into te…
hennlo Oct 17, 2021
e94f644
added missing JavaDoc for catalog methods
hennlo Oct 17, 2021
b4889fd
Remove empty line
vogti Oct 17, 2021
6573f70
extended index support for partitioned tables
hennlo Oct 17, 2021
9ae28ad
Fix issue with filters pushed down below union operations on partitio…
vogti Oct 17, 2021
9e33821
added test for partition Filter
hennlo Oct 17, 2021
bb297db
Minor changes to code style
vogti Oct 17, 2021
2935c4b
added test for partition Filter
hennlo Oct 17, 2021
c7d42b9
Fix issue with column name expansion
vogti Oct 18, 2021
3a43fb5
Minor fixes to code formatting
vogti Oct 18, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@
import org.polypheny.db.catalog.Catalog;
import org.polypheny.db.catalog.entity.CatalogColumn;
import org.polypheny.db.catalog.entity.CatalogColumnPlacement;
import org.polypheny.db.catalog.entity.CatalogPartitionPlacement;
import org.polypheny.db.catalog.entity.CatalogTable;
import org.polypheny.db.catalog.exceptions.UnknownColumnException;
import org.polypheny.db.catalog.exceptions.UnknownDatabaseException;
Expand Down Expand Up @@ -146,7 +147,8 @@ public String getPhysicalColumnName( String tableName, String logicalColumnName

public void updatePhysicalColumnName( long columnId, String updatedName, boolean updatePosition ) {
CatalogColumnPlacement placement = this.catalog.getColumnPlacement( this.storeId, columnId );
this.catalog.updateColumnPlacementPhysicalNames( this.storeId, columnId, placement.physicalTableName, placement.physicalTableName, updatedName, updatePosition );
CatalogPartitionPlacement partitionPlacement = catalog.getPartitionPlacement( this.storeId, catalog.getTable( placement.tableId ).partitionProperty.partitionIds.get( 0 ) );
this.catalog.updateColumnPlacementPhysicalNames( this.storeId, columnId, partitionPlacement.physicalTableName, updatedName, updatePosition );
}


Expand Down Expand Up @@ -193,4 +195,5 @@ public static String incrementNameRevision( String name ) {

return type + id + "r" + rev;
}

}
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,7 @@
import org.polypheny.db.catalog.entity.CatalogColumnPlacement;
import org.polypheny.db.catalog.entity.CatalogIndex;
import org.polypheny.db.catalog.entity.CatalogKey;
import org.polypheny.db.catalog.entity.CatalogPartitionPlacement;
import org.polypheny.db.catalog.entity.CatalogTable;
import org.polypheny.db.docker.DockerInstance;
import org.polypheny.db.docker.DockerManager;
Expand Down Expand Up @@ -208,9 +209,9 @@ public void createNewSchema( SchemaPlus rootSchema, String name ) {


@Override
public Table createTableSchema( CatalogTable catalogTable, List<CatalogColumnPlacement> columnPlacementsOnStore ) {
String physicalTableName = currentSchema.getConvention().physicalNameProvider.getPhysicalTableName( catalogTable.id );
return new CassandraTable( this.currentSchema, catalogTable.name, physicalTableName, false );
public Table createTableSchema( CatalogTable catalogTable, List<CatalogColumnPlacement> columnPlacementsOnStore, CatalogPartitionPlacement partitionPlacement ) {
String cassandraphysicalTableName = currentSchema.getConvention().physicalNameProvider.getPhysicalTableName( catalogTable.id );
return new CassandraTable( this.currentSchema, catalogTable.name, cassandraphysicalTableName, false );
}


Expand All @@ -221,7 +222,7 @@ public Schema getCurrentSchema() {


@Override
public void createTable( Context context, CatalogTable catalogTable ) {
public void createTable( Context context, CatalogTable catalogTable, List<Long> partitionIds ) {
// This check is probably not required due to the check below it.
if ( catalogTable.primaryKey == null ) {
throw new UnsupportedOperationException( "Cannot create Cassandra Table without a primary key!" );
Expand All @@ -247,7 +248,7 @@ public void createTable( Context context, CatalogTable catalogTable ) {
CassandraPhysicalNameProvider physicalNameProvider = new CassandraPhysicalNameProvider( this.getAdapterId() );
String physicalTableName = physicalNameProvider.getPhysicalTableName( catalogTable.id );
// List<CatalogColumn> columns = combinedTable.getColumns();
List<CatalogColumnPlacement> columns = catalog.getColumnPlacementsOnAdapter( getAdapterId(), catalogTable.id );
List<CatalogColumnPlacement> columns = catalog.getColumnPlacementsOnAdapterPerTable( getAdapterId(), catalogTable.id );
CatalogColumnPlacement primaryColumnPlacement = columns.stream().filter( c -> c.columnId == primaryKeyColumnLambda ).findFirst().get();
CatalogColumn catalogColumn = catalog.getColumn( primaryColumnPlacement.columnId );

Expand Down Expand Up @@ -275,20 +276,19 @@ public void createTable( Context context, CatalogTable catalogTable ) {
context.getStatement().getTransaction().registerInvolvedAdapter( this );
this.session.execute( createTable.build() );

for ( CatalogColumnPlacement placement : catalog.getColumnPlacementsOnAdapter( getAdapterId(), catalogTable.id ) ) {
for ( CatalogColumnPlacement placement : catalog.getColumnPlacementsOnAdapterPerTable( getAdapterId(), catalogTable.id ) ) {
catalog.updateColumnPlacementPhysicalNames(
getAdapterId(),
placement.columnId,
this.dbKeyspace, // TODO MV: physical schema name
physicalTableName,
physicalNameProvider.generatePhysicalColumnName( placement.columnId ),
true );
}
}


@Override
public void dropTable( Context context, CatalogTable catalogTable ) {
public void dropTable( Context context, CatalogTable catalogTable, List<Long> partitionIds ) {
CassandraPhysicalNameProvider physicalNameProvider = new CassandraPhysicalNameProvider( this.getAdapterId() );
String physicalTableName = physicalNameProvider.getPhysicalTableName( catalogTable.id );
SimpleStatement dropTable = SchemaBuilder.dropTable( this.dbKeyspace, physicalTableName ).build();
Expand All @@ -315,7 +315,6 @@ public void addColumn( Context context, CatalogTable catalogTable, CatalogColumn
getAdapterId(),
catalogColumn.id,
this.dbKeyspace,
physicalTableName,
physicalColumnName,
false );
}
Expand All @@ -325,7 +324,10 @@ public void addColumn( Context context, CatalogTable catalogTable, CatalogColumn
public void dropColumn( Context context, CatalogColumnPlacement columnPlacement ) {
// public void dropColumn( Context context, CatalogCombinedTable catalogTable, CatalogColumn catalogColumn ) {
// CassandraPhysicalNameProvider physicalNameProvider = new CassandraPhysicalNameProvider( context.getStatement().getTransaction().getCatalog(), this.getStoreId() );
String physicalTableName = columnPlacement.physicalTableName;

CatalogPartitionPlacement partitionPlacement = catalog.getPartitionPlacement( getAdapterId(), catalog.getTable( columnPlacement.tableId ).partitionProperty.partitionIds.get( 0 ) );

String physicalTableName = partitionPlacement.physicalTableName;
String physicalColumnName = columnPlacement.physicalColumnName;

SimpleStatement dropColumn = SchemaBuilder.alterTable( this.dbKeyspace, physicalTableName )
Expand All @@ -337,14 +339,14 @@ public void dropColumn( Context context, CatalogColumnPlacement columnPlacement


@Override
public void addIndex( Context context, CatalogIndex catalogIndex ) {
public void addIndex( Context context, CatalogIndex catalogIndex, List<Long> partitionIds ) {
throw new RuntimeException( "Cassandra adapter does not support adding indexes" );
}


@Override
public void dropIndex( Context context, CatalogIndex catalogIndex ) {
throw new RuntimeException( "Cassandra adaper does not support dropping indexes" );
public void dropIndex( Context context, CatalogIndex catalogIndex, List<Long> partitionIds ) {
throw new RuntimeException( "Cassandra adapter does not support dropping indexes" );
}


Expand Down
Loading