Skip to content
This repository has been archived by the owner on Aug 23, 2021. It is now read-only.

clean up config directory and use docker for travis #241

Open
wants to merge 9 commits into
base: master
Choose a base branch
from

Conversation

at15
Copy link

@at15 at15 commented Feb 8, 2018

Related issue #209

breaking changes

  • removed config/plugin.xml, added class Benchmarks under com.oltpbenchmark.benchmarks
    • use reflection might work, but this way is easier and requires no external library
  • removed most legacy config, all the config that is not sample_<bench>_config.xml are considered legacy and removed.
  • added benchmarks.yml and databases.yml as a catalog for supported benchmark and databases
  • added docker folder to run database using docker, it's (very) inaccurate, but can test correctness and is handy on CI systems
    • added create_db.py to create database using db shell inside containers
  • added TiDB and MemSQL, only TPC-C is tested, both have a lot of errors
  • updated doc in config and docker folder to address those changes
  • removed old travis test entirely, all the tests on travis is using docker, including mysql and postgres.
    • travis started mysql and postgres by default, they are stopped to avoid port conflict

known issues

PR that would conflict with this one, (actually I think all the pending PR will ...)

To play around with this PR

cd /tmp
git clone [email protected]:benchhub/oltpbench.git
cd oltpbench
git checkout config/cleanup-v2 
./config/config.py benchmarks
./config/config.py databases
# stop your local mysql/postgres
# make sure you have docker and docker-compose installed
export BENCH=tpcc
export DB=mysql
./config/config.py generate --bench ${BENCH} --db ${DB} --scalefactor=1
./docker/travis_start.sh
ant build
./oltpbenchmark --bench ${BENCH} --config config/generated_${BENCH}_${DB}_config.xml --create true --load true --execute true
./docker/travis_stop.sh

at15 added 9 commits February 7, 2018 12:23
- address oltpbenchmark#209
  - create two directory `benchmarks` and `databases`
  - traces, `plugin.xml` need separated PR
- many config are ad-hoc with remote host url hard coded,
some are there for several years without any change
- if any of them are still needed, just go back to this commit (or any
version before it) to grab them out ...
- document how to update the catalog yaml files
- copy config.py etc. from config/cleanup branch
- stop travis builtin mysql and postgres
- include docker compose file for mysql, postgres, tidb, memsql
- added `Benchmarks.java` in `com.oltpbenchmakr.benchmarks` package as
a registry for all existing benchmarks.
  - it is possible to use reflection to loop all the subpackages and
find those extends BenchmarkModule, but it may requires new libraries
- ignore generated config in config directory, config generated by
`config.py` all starts w/ `generated_` prefix
- some are disabled due to lack of config or requires external program
to generate loading data (i.e. tpc-h, tpc-ds, does tpc-ds ever
worked? oltpbenchmark#222 )
- auctionmark is failing, but it worked in upstream, need to check it
- tidb is disabled for now, because DatabaseType is not updated
- previously no default value is specified, now changed to `TIMESTAMP
DEFAULT CURRENT_TIMESTAMP`
- there are still errors, but it was there in old travis log as well
  - https://travis-ci.org/oltpbenchmark/oltpbench/jobs/337420401
````
The DBMS rejected the transaction without an error code
com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException:
Duplicate entry '40534595669786707-14750-2199023452243-196691' for key
'PRIMARY'
````
- update gitignore to ignore IDEA stuff and Mac .DS_Store
- auction mysql ddl should be fixed in previous commit
- jpab does not have ddl ...
- chbench got too many errors and has run for 40 mins on travis
  - https://travis-ci.org/benchhub/oltpbench/jobs/338773148
  - it seems the ddl didn't create the table as expected ...
- just copy mysql's ddl w/ some modification
- all have some errors, guess they are not that compatiable to MySQL
- MemSQL does not support `REFERENCE` and tooks a while to start
  - thus the `travis_start.sh` would sleep 30s for memsql
- travis limit is 4 MB, there are too many exceptions, and travis would
terminate those jobs (it can finish on local machine though ...)
  - https://travis-ci.org/benchhub/oltpbench/jobs/338789981
@apavlo
Copy link
Member

apavlo commented Feb 20, 2018

@at15 What is the status of this?

@at15
Copy link
Author

at15 commented Feb 20, 2018

@apavlo I think I need to wait for other PRs that introduces new databases to be merged first, so I can solve the conflicts in my fork's branch before it got merged into upstream

@apavlo
Copy link
Member

apavlo commented Feb 20, 2018

Those other PRs are not going to be merged (at least not for now). We can do that manually later.

@at15 at15 changed the title WIP: clean up config directory clean up config directory Feb 20, 2018
@at15
Copy link
Author

at15 commented Feb 20, 2018

@apavlo then I think it's ready to be merged, I don't have any update since the initial PR

@at15 at15 changed the title clean up config directory clean up config directory and use docker for travis Mar 3, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants