Run JUnit CI Tests with Compiled Query Execution #1470

turingcompl33t · 2021-02-13T17:00:24Z

Closes #1223.

We have seen a number of bugs now that only surface under compiled or adaptive query execution modes, but our testing infrastructure fails to detect them. The reasons for this are manifold, but are primarily a result of the fact that:

Apart from one exception, unit tests are not run in compiled or adaptive modes
JUnit tests are not run run in compiled or adaptive modes

This PR takes a step towards increasing our coverage of compiled and adaptive query execution modes by adding support for running JUnit tests in CI with compiled query execution. This will hopefully decrease the probability of introducing new, uncaught bugs as we continue to modify and improve the execution engine and related aspects of the system.

I intended this PR to be solely an update to the CI infrastructure (i.e. the Jenkinsfile) but I also found it necessary to modify some of our testing infrastructure (Python implementation) to add the desired functionality.

…dify Jenkinsfile to run (some) of the JUnit tests in compiled mode, want to test on a subset first before changing all of them

turingcompl33t · 2021-02-13T17:01:38Z

There may be some kinks in this PR that remain to be worked out, but I'm marking it ready-for-ci now so I can verify that the modifications in their current form work in the CI pipeline.

noisepage-checks · 2021-02-13T22:20:44Z

Minor Decrease in Performance

Be warned: this PR may have decreased the throughput of the system slightly.

tps (%change)	benchmark_type	wal_device	details
-0.26%	tpcc	RAM disk	Details master tps=9025.3, commit tps=9001.61, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=RAM disk, max_connection_threads=32
0.86%	tpcc	None	Details master tps=10093.21, commit tps=10180.5, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=None, max_connection_threads=32
-0.93%	tpcc	HDD	Details master tps=424.85, commit tps=420.9, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=HDD, max_connection_threads=32
3.34%	tatp	RAM disk	Details master tps=3715.87, commit tps=3839.88, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=RAM disk, max_connection_threads=32
1.36%	tatp	None	Details master tps=4056.11, commit tps=4111.08, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=None, max_connection_threads=32
-2.08%	tatp	HDD	Details master tps=305.92, commit tps=299.55, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=HDD, max_connection_threads=32

codecov · 2021-02-13T22:20:46Z

Codecov Report

Merging #1470 (b2724f7) into master (c2635d3) will increase coverage by 0.00%.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #1470   +/-   ##
=======================================
  Coverage   81.78%   81.78%           
=======================================
  Files         707      707           
  Lines       50356    50356           
=======================================
+ Hits        41183    41184    +1     
+ Misses       9173     9172    -1

Impacted Files	Coverage Δ
src/transaction/transaction_manager.cpp	`92.51% <0.00%> (-2.73%)`	⬇️
src/traffic_cop/traffic_cop.cpp	`73.77% <0.00%> (+0.40%)`	⬆️
...network/postgres/postgres_protocol_interpreter.cpp	`86.66% <0.00%> (+5.33%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c2635d3...b2724f7. Read the comment docs.

noisepage-checks · 2021-02-13T23:04:31Z

Minor Decrease in Performance

Be warned: this PR may have decreased the throughput of the system slightly.

tps (%change)	benchmark_type	wal_device	details
0.61%	tpcc	RAM disk	Details master tps=9025.3, commit tps=9079.96, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=RAM disk, max_connection_threads=32
1.27%	tpcc	None	Details master tps=10093.21, commit tps=10221.46, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=None, max_connection_threads=32
-0.9%	tpcc	HDD	Details master tps=424.85, commit tps=421.03, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=HDD, max_connection_threads=32
2.78%	tatp	RAM disk	Details master tps=3715.87, commit tps=3819.21, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=RAM disk, max_connection_threads=32
1.51%	tatp	None	Details master tps=4056.11, commit tps=4117.21, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=None, max_connection_threads=32
-3.54%	tatp	HDD	Details master tps=305.92, commit tps=295.1, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=HDD, max_connection_threads=32

noisepage-checks · 2021-02-16T18:31:54Z

Minor Decrease in Performance

Be warned: this PR may have decreased the throughput of the system slightly.

tps (%change)	benchmark_type	wal_device	details
0.88%	tpcc	RAM disk	Details master tps=8861.33, commit tps=8939.66, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=RAM disk, max_connection_threads=32
-1.78%	tpcc	None	Details master tps=10337.48, commit tps=10153.42, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=None, max_connection_threads=32
6.67%	tpcc	HDD	Details master tps=553.79, commit tps=590.73, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=HDD, max_connection_threads=32
1.88%	tatp	RAM disk	Details master tps=3750.39, commit tps=3820.74, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=RAM disk, max_connection_threads=32
1.27%	tatp	None	Details master tps=3964.54, commit tps=4014.94, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=None, max_connection_threads=32
6.17%	tatp	HDD	Details master tps=399.03, commit tps=423.63, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=HDD, max_connection_threads=32

noisepage-checks · 2021-02-16T21:37:26Z

Minor Decrease in Performance

Be warned: this PR may have decreased the throughput of the system slightly.

tps (%change)	benchmark_type	wal_device	details
0.88%	tpcc	RAM disk	Details master tps=8861.33, commit tps=8939.66, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=RAM disk, max_connection_threads=32
-1.78%	tpcc	None	Details master tps=10337.48, commit tps=10153.42, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=None, max_connection_threads=32
6.67%	tpcc	HDD	Details master tps=553.79, commit tps=590.73, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=HDD, max_connection_threads=32
1.88%	tatp	RAM disk	Details master tps=3750.39, commit tps=3820.74, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=RAM disk, max_connection_threads=32
1.27%	tatp	None	Details master tps=3964.54, commit tps=4014.94, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=None, max_connection_threads=32
6.17%	tatp	HDD	Details master tps=399.03, commit tps=423.63, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=HDD, max_connection_threads=32

noisepage-checks · 2021-02-17T04:41:25Z

Minor Decrease in Performance

Be warned: this PR may have decreased the throughput of the system slightly.

tps (%change)	benchmark_type	wal_device	details
1.75%	tpcc	RAM disk	Details master tps=8861.33, commit tps=9016.42, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=RAM disk, max_connection_threads=32
-4.54%	tpcc	None	Details master tps=10337.48, commit tps=9868.15, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=None, max_connection_threads=32
-0.94%	tpcc	HDD	Details master tps=553.79, commit tps=548.56, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=HDD, max_connection_threads=32
2.37%	tatp	RAM disk	Details master tps=3750.39, commit tps=3839.41, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=RAM disk, max_connection_threads=32
2.54%	tatp	None	Details master tps=3964.54, commit tps=4065.36, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=None, max_connection_threads=32
-3.16%	tatp	HDD	Details master tps=399.03, commit tps=386.43, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=HDD, max_connection_threads=32

noisepage-checks · 2021-02-18T19:00:36Z

Major Decrease in Performance

STOP: this PR has a major negative performance impact

tps (%change)	benchmark_type	wal_device	details
0.92%	tpcc	RAM disk	Details master tps=8934.58, commit tps=9016.42, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=RAM disk, max_connection_threads=32
-3.01%	tpcc	None	Details master tps=10174.58, commit tps=9868.15, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=None, max_connection_threads=32
-2.54%	tpcc	HDD	Details master tps=562.85, commit tps=548.56, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=HDD, max_connection_threads=32
1.04%	tatp	RAM disk	Details master tps=3799.91, commit tps=3839.41, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=RAM disk, max_connection_threads=32
-0.11%	tatp	None	Details master tps=4069.66, commit tps=4065.36, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=None, max_connection_threads=32
-5.8%	tatp	HDD	Details master tps=410.23, commit tps=386.43, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=HDD, max_connection_threads=32

noisepage-checks · 2021-02-21T23:44:10Z

Major Decrease in Performance

STOP: this PR has a major negative performance impact

tps (%change)	benchmark_type	wal_device	details
0.16%	tpcc	RAM disk	Details master tps=8959.51, commit tps=8973.79, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=RAM disk, max_connection_threads=32
-0.8%	tpcc	None	Details master tps=10055.01, commit tps=9974.73, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=None, max_connection_threads=32
-0.51%	tpcc	HDD	Details master tps=542.82, commit tps=540.04, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=HDD, max_connection_threads=32
-0.22%	tatp	RAM disk	Details master tps=3762.93, commit tps=3754.49, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=RAM disk, max_connection_threads=32
0.56%	tatp	None	Details master tps=4058.9, commit tps=4081.81, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=None, max_connection_threads=32
-5.88%	tatp	HDD	Details master tps=399.67, commit tps=376.18, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=HDD, max_connection_threads=32

lmwnshn

Generally LGTM, nice use of Python, question about the None case, minor typo.

script/testing/util/db_server.py

…to them as meta-tests

noisepage-checks · 2021-03-06T23:47:14Z

Performance Boost!

Nice job! This PR has increased the throughput of the system.

Could not find any performance results to compare for this commit.

linmagit · 2021-03-07T00:01:30Z

The time to finish the junit tests under compiled mode seems really long. I don't believe it's an issue of this PR itself (nice work, @turingcompl33t). But it does seem alarming that it takes so long, which will significantly increase our Jenkins test time. For example, this build takes almost 4 hours to finish:
http://jenkins.db.cs.cmu.edu:8080/blue/organizations/jenkins/terrier/detail/PR-1470/43/pipeline/142

turingcompl33t · 2021-03-07T00:20:16Z

The time to finish the junit tests under compiled mode seems really long. I don't believe it's an issue of this PR itself (nice work, @turingcompl33t). But it does seem alarming that it takes so long, which will significantly increase our Jenkins test time. For example, this build takes almost 4 hours to finish:
http://jenkins.db.cs.cmu.edu:8080/blue/organizations/jenkins/terrier/detail/PR-1470/43/pipeline/142

Agreed @malin1993ml, was noticing the same thing, as you can see in the commits above where I had to bump the timeout for the JUnit tests twice in order to get them to run to completion before being terminated. I plan to look a bit closer at where exactly the huge jump in execution time is coming from, although its entirely possible that's just the current state of execution engine performance... do we have a robust comparison of the performance in interpreted vs compiled modes anywhere?

linmagit · 2021-03-07T00:37:56Z

do we have a robust comparison of the performance in interpreted vs compiled modes anywhere?

I'm not sure. Maybe others have an idea of what should be the expected performance.

noisepage-checks · 2021-03-09T00:36:32Z

Major Decrease in Performance

STOP: this PR has a major negative performance impact

tps (%change)	benchmark_type	wal_device	details
3.03%	tpcc	RAM disk	Details master tps=8832.55, commit tps=9100.34, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=RAM disk, max_connection_threads=32
-2.05%	tpcc	None	Details master tps=10133.65, commit tps=9925.53, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=None, max_connection_threads=32
-2.74%	tpcc	HDD	Details master tps=566.75, commit tps=551.24, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=HDD, max_connection_threads=32
2.1%	tatp	RAM disk	Details master tps=3781.11, commit tps=3860.52, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=RAM disk, max_connection_threads=32
0.41%	tatp	None	Details master tps=4157.76, commit tps=4174.95, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=None, max_connection_threads=32
-9.68%	tatp	HDD	Details master tps=424.69, commit tps=383.56, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=HDD, max_connection_threads=32

turingcompl33t · 2021-03-19T14:47:45Z

Adding ready-for-ci tag because in theory all of the compiled JUnit tests should pass in compiled mode now, with compiled aggregations fixed AND the update where we fail the test in the event the DBMS exits with nonzero exit code.

…removing asan even for debug builds

turingcompl33t · 2021-03-19T20:05:41Z

Don't have a passing CI build to link to because the pipeline failed (in end-to-end performance, unrelated to this PR), but I think the modifications to the bytecode compilation settings may have done the trick. After making the change, compiled JUnit tests consistently finished in ~20 minutes (often far fewer). I am hoping this will be deemed 'good enough' for the time being to push the PR through.

turingcompl33t · 2021-03-19T21:06:32Z

Per the discussion in standup, I am going to briefly investigate the breakdown of the runtime overhead for tests run in compiled execution mode. Related issue is here.

noisepage-checks · 2021-03-20T01:54:18Z

Minor Decrease in Performance

Be warned: this PR may have decreased the throughput of the system slightly.

tps (%change)	benchmark_type	wal_device	details
149.9%	tpcc	RAM disk	Details master tps=8915.61, commit tps=22279.9, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=RAM disk, max_connection_threads=32
202.22%	tpcc	None	Details master tps=9908.63, commit tps=29946.11, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=None, max_connection_threads=32
-1.27%	tpcc	HDD	Details master tps=21513.75, commit tps=21241.33, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=HDD, max_connection_threads=32
6.78%	tatp	RAM disk	Details master tps=6552.35, commit tps=6996.65, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=RAM disk, max_connection_threads=32
2.38%	tatp	None	Details master tps=7311.9, commit tps=7486.1, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=None, max_connection_threads=32
5.79%	tatp	HDD	Details master tps=6455.65, commit tps=6829.14, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=HDD, max_connection_threads=32

noisepage-checks · 2021-03-20T01:56:38Z

Minor Decrease in Performance

Be warned: this PR may have decreased the throughput of the system slightly.

tps (%change)	benchmark_type	wal_device	details
149.9%	tpcc	RAM disk	Details master tps=8915.61, commit tps=22279.9, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=RAM disk, max_connection_threads=32
202.22%	tpcc	None	Details master tps=9908.63, commit tps=29946.11, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=None, max_connection_threads=32
-1.27%	tpcc	HDD	Details master tps=21513.75, commit tps=21241.33, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=HDD, max_connection_threads=32
6.78%	tatp	RAM disk	Details master tps=6552.35, commit tps=6996.65, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=RAM disk, max_connection_threads=32
2.38%	tatp	None	Details master tps=7311.9, commit tps=7486.1, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=None, max_connection_threads=32
5.79%	tatp	HDD	Details master tps=6455.65, commit tps=6829.14, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=HDD, max_connection_threads=32

noisepage-checks · 2021-03-20T05:36:04Z

Minor Decrease in Performance

Be warned: this PR may have decreased the throughput of the system slightly.

tps (%change)	benchmark_type	wal_device	details
152.2%	tpcc	RAM disk	Details master tps=8915.61, commit tps=22484.77, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=RAM disk, max_connection_threads=32
190.25%	tpcc	None	Details master tps=9908.63, commit tps=28760.24, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=None, max_connection_threads=32
-0.72%	tpcc	HDD	Details master tps=21513.75, commit tps=21359.79, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=HDD, max_connection_threads=32
5.8%	tatp	RAM disk	Details master tps=6552.35, commit tps=6932.32, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=RAM disk, max_connection_threads=32
1.26%	tatp	None	Details master tps=7311.9, commit tps=7403.71, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=None, max_connection_threads=32
4.42%	tatp	HDD	Details master tps=6455.65, commit tps=6741.05, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=HDD, max_connection_threads=32

noisepage-checks · 2021-03-20T05:36:22Z

Performance Boost!

Nice job! This PR has increased the throughput of the system.

Could not find any performance results to compare for this commit.

noisepage-checks · 2021-03-22T02:20:16Z

Minor Decrease in Performance

Be warned: this PR may have decreased the throughput of the system slightly.

tps (%change)	benchmark_type	wal_device	details
1.63%	tpcc	RAM disk	Details master tps=22124.91, commit tps=22484.77, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=RAM disk, max_connection_threads=32
-0.9%	tpcc	None	Details master tps=29020.88, commit tps=28760.24, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=None, max_connection_threads=32
2.08%	tpcc	HDD	Details master tps=20924.55, commit tps=21359.79, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=HDD, max_connection_threads=32
7.91%	tatp	RAM disk	Details master tps=6424.01, commit tps=6932.32, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=RAM disk, max_connection_threads=32
-0.29%	tatp	None	Details master tps=7425.04, commit tps=7403.71, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=None, max_connection_threads=32
5.03%	tatp	HDD	Details master tps=6418.28, commit tps=6741.05, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=HDD, max_connection_threads=32

noisepage-checks · 2021-03-29T22:03:04Z

Minor Decrease in Performance

Be warned: this PR may have decreased the throughput of the system slightly.

tps (%change)	benchmark_type	wal_device	details
-1.36%	tpcc	RAM disk	Details master tps=22247.1, commit tps=21945.12, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=RAM disk, max_connection_threads=32
2.8%	tpcc	None	Details master tps=28340.31, commit tps=29132.55, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=None, max_connection_threads=32
0.54%	tpcc	HDD	Details master tps=21273.53, commit tps=21388.86, query_mode=extended, benchmark_type=tpcc, scale_factor=32.0000, terminals=32, client_time=60, weights={'Payment': 43, 'Delivery': 4, 'NewOrder': 45, 'StockLevel': 4, 'OrderStatus': 4}, wal_device=HDD, max_connection_threads=32
6.45%	tatp	RAM disk	Details master tps=6460.17, commit tps=6877.18, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=RAM disk, max_connection_threads=32
0.82%	tatp	None	Details master tps=7301.83, commit tps=7361.92, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=None, max_connection_threads=32
7.78%	tatp	HDD	Details master tps=6343.19, commit tps=6836.86, query_mode=extended, benchmark_type=tatp, scale_factor=1.0000, terminals=16, client_time=60, weights={'GetAccessData': 35, 'UpdateLocation': 14, 'GetNewDestination': 10, 'GetSubscriberData': 35, 'DeleteCallForwarding': 2, 'InsertCallForwarding': 2, 'UpdateSubscriberData': 2}, wal_device=HDD, max_connection_threads=32

lmwnshn

Rubber stamp.

modify NoisePageServer to make working with relative paths easier, mo…

17b355f

…dify Jenkinsfile to run (some) of the JUnit tests in compiled mode, want to test on a subset first before changing all of them

turingcompl33t added infrastructure Infrastructure related issues or changes (CMake, dependencies, CI) ready-for-ci Indicate that this build should be run through CI. tests This issue or PR concerns the testing infrastructure: Google Test, Google Benchmark, JUnit, etc. labels Feb 13, 2021

turingcompl33t added 2 commits February 13, 2021 15:21

handle type system issue that comes up on OLTPBench tests

d03db58

update remaining CI stages with compiled query execution tests

fbd1102

turingcompl33t added the ready-for-review This PR passes all checks and is ready to be reviewed. Mark PRs with this. label Feb 13, 2021

Merge branch 'master' into ci-compiled-junit-tests

224491b

Merge branch 'master' into ci-compiled-junit-tests

a101a88

turingcompl33t added in-progress This PR is being actively worked on and not ready to be reviewed or merged. Mark PRs with this. and removed ready-for-review This PR passes all checks and is ready to be reviewed. Mark PRs with this. labels Feb 19, 2021

turingcompl33t added 2 commits February 19, 2021 16:16

Merge branch 'master' into ci-compiled-junit-tests

306b7e8

Merge branch 'master' into ci-compiled-junit-tests

cd8026b

Merge branch 'master' into ci-compiled-junit-tests

37b18e1

lmwnshn suggested changes Feb 22, 2021

View reviewed changes

lmwnshn removed the ready-for-ci Indicate that this build should be run through CI. label Feb 22, 2021

turingcompl33t added 4 commits February 22, 2021 17:38

address (most) comments from review, still one open question

3c23d9e

add some basic tests for testing infrastructure utilities, referring …

28f3f3b

…to them as meta-tests

Merge branch 'master' into ci-compiled-junit-tests

9a30745

Merge branch 'master' into ci-compiled-junit-tests

a249f0e

turingcompl33t added the ready-for-ci Indicate that this build should be run through CI. label Mar 5, 2021

increase timeouts again

0bea976

Merge branch 'master' into ci-compiled-junit-tests

db23be1

lmwnshn removed the ready-for-ci Indicate that this build should be run through CI. label Mar 14, 2021

Merge branch 'master' into ci-compiled-junit-tests

8d9ef79

turingcompl33t added the ready-for-ci Indicate that this build should be run through CI. label Mar 19, 2021

turingcompl33t mentioned this pull request Mar 19, 2021

Fix Compiled Parallel Aggregations #1211

Open

turingcompl33t added 2 commits March 19, 2021 12:31

attempt to speed up compiled tests by forcing some optimizations and …

a436bdb

…removing asan even for debug builds

update relative path resolution, update some documentation

5ab6842

Merge branch 'master' into ci-compiled-junit-tests

575c6de

Merge branch 'master' into ci-compiled-junit-tests

b2724f7

apavlo added ready-to-merge This PR is ready to be merged. Mark PRs with this. and removed in-progress This PR is being actively worked on and not ready to be reviewed or merged. Mark PRs with this. labels Mar 23, 2021

lmwnshn approved these changes Mar 30, 2021

View reviewed changes

lmwnshn merged commit 7fb3e48 into cmu-db:master Mar 30, 2021

turingcompl33t mentioned this pull request Mar 30, 2021

fix broken import in binary size artifacts collection script #1527

Merged

turingcompl33t deleted the ci-compiled-junit-tests branch April 2, 2021 13:21

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Run JUnit CI Tests with Compiled Query Execution #1470

Run JUnit CI Tests with Compiled Query Execution #1470

turingcompl33t commented Feb 13, 2021

turingcompl33t commented Feb 13, 2021

noisepage-checks bot commented Feb 13, 2021

codecov bot commented Feb 13, 2021 •

edited

Loading

noisepage-checks bot commented Feb 13, 2021

noisepage-checks bot commented Feb 16, 2021

noisepage-checks bot commented Feb 16, 2021

noisepage-checks bot commented Feb 17, 2021

noisepage-checks bot commented Feb 18, 2021

noisepage-checks bot commented Feb 21, 2021

lmwnshn left a comment

noisepage-checks bot commented Mar 6, 2021

linmagit commented Mar 7, 2021

turingcompl33t commented Mar 7, 2021

linmagit commented Mar 7, 2021

noisepage-checks bot commented Mar 9, 2021

turingcompl33t commented Mar 19, 2021

turingcompl33t commented Mar 19, 2021

turingcompl33t commented Mar 19, 2021

noisepage-checks bot commented Mar 20, 2021

noisepage-checks bot commented Mar 20, 2021

noisepage-checks bot commented Mar 20, 2021

noisepage-checks bot commented Mar 20, 2021

noisepage-checks bot commented Mar 22, 2021

noisepage-checks bot commented Mar 29, 2021

lmwnshn left a comment

Run JUnit CI Tests with Compiled Query Execution #1470

Run JUnit CI Tests with Compiled Query Execution #1470

Conversation

turingcompl33t commented Feb 13, 2021

turingcompl33t commented Feb 13, 2021

noisepage-checks bot commented Feb 13, 2021

Minor Decrease in Performance

codecov bot commented Feb 13, 2021 • edited Loading

Codecov Report

noisepage-checks bot commented Feb 13, 2021

Minor Decrease in Performance

noisepage-checks bot commented Feb 16, 2021

Minor Decrease in Performance

noisepage-checks bot commented Feb 16, 2021

Minor Decrease in Performance

noisepage-checks bot commented Feb 17, 2021

Minor Decrease in Performance

noisepage-checks bot commented Feb 18, 2021

Major Decrease in Performance

noisepage-checks bot commented Feb 21, 2021

Major Decrease in Performance

lmwnshn left a comment

Choose a reason for hiding this comment

noisepage-checks bot commented Mar 6, 2021

Performance Boost!

linmagit commented Mar 7, 2021

turingcompl33t commented Mar 7, 2021

linmagit commented Mar 7, 2021

noisepage-checks bot commented Mar 9, 2021

Major Decrease in Performance

turingcompl33t commented Mar 19, 2021

turingcompl33t commented Mar 19, 2021

turingcompl33t commented Mar 19, 2021

noisepage-checks bot commented Mar 20, 2021

Minor Decrease in Performance

noisepage-checks bot commented Mar 20, 2021

Minor Decrease in Performance

noisepage-checks bot commented Mar 20, 2021

Minor Decrease in Performance

noisepage-checks bot commented Mar 20, 2021

Performance Boost!

noisepage-checks bot commented Mar 22, 2021

Minor Decrease in Performance

noisepage-checks bot commented Mar 29, 2021

Minor Decrease in Performance

lmwnshn left a comment

Choose a reason for hiding this comment

codecov bot commented Feb 13, 2021 •

edited

Loading