Skip to content

Commit

Permalink
renamed sdk back to test-harness (liquibase#40)
Browse files Browse the repository at this point in the history
  • Loading branch information
KushnirykOleh authored Jan 13, 2021
1 parent 90e38e4 commit 974f3ad
Show file tree
Hide file tree
Showing 317 changed files with 74 additions and 76 deletions.
8 changes: 4 additions & 4 deletions .github/workflows/main.yml
Original file line number Diff line number Diff line change
Expand Up @@ -25,13 +25,13 @@ jobs:
run: docker-compose up -d

- name: MySQL Test Run
run: mvn -Dtest=LiquibaseSdkSuiteTest -DdbName=mysql -Dmaven.test.failure.ignore=true test
run: mvn -Dtest=LiquibaseHarnessSuiteTest -DdbName=mysql -Dmaven.test.failure.ignore=true test

- name: MariaDB Test Run
run: mvn -Dtest=LiquibaseSdkSuiteTest -DdbName=mariadb -Dmaven.test.failure.ignore=true test
run: mvn -Dtest=LiquibaseHarnessSuiteTest -DdbName=mariadb -Dmaven.test.failure.ignore=true test

- name: Postgres Test Run
run: mvn -Dtest=LiquibaseSdkSuiteTest -DdbName=postgresql -Dmaven.test.failure.ignore=true test
run: mvn -Dtest=LiquibaseHarnessSuiteTest -DdbName=postgresql -Dmaven.test.failure.ignore=true test

- name: Tear down test infra
working-directory: src/test/resources/docker
Expand All @@ -42,7 +42,7 @@ jobs:
run: docker-compose -f docker-compose.cockroachdb.yml up -d

- name: cockroachDB Test Run
run: mvn -Dtest=LiquibaseSdkSuiteTest -DdbName=cockroachdb test
run: mvn -Dtest=LiquibaseHarnessSuiteTest -DdbName=cockroachdb test

- name: Tear down cockroachDB test infra
working-directory: src/test/resources/docker
Expand Down
12 changes: 6 additions & 6 deletions README.extensions.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,33 +12,33 @@ If you are using maven, you can add the dependency as follows:
<dependencies>
<dependency>
<groupId>org.liquibase</groupId>
<artifactId>liquibase-sdk</artifactId>
<artifactId>liquibase-test-harness</artifactId>
<version>1.0-SNAPSHOT</version>
<scope>test</scope>
</dependency>
</dependencies>
```
You will need to `git clone https://github.com/liquibase/liquibase-test-harness.git` and then run `mvn install -DskipTests` to install the liquibase-sdk in your `~/.m2/repository`.
You will need to `git clone https://github.com/liquibase/liquibase-test-harness.git` and then run `mvn install -DskipTests` to install the liquibase-test-harness in your `~/.m2/repository`.

#### Configuring your connections

Add a liquibase.sdk.test.yml file to your `src/test/resources` file.
Add a harness-config.yml file to your `src/test/resources` file.
This file should contain the connection information for all the databases you want your extension to be tested against.

See [src/test/resources/liquibase.sdk.test.yml] as an example.
See [src/test/resources/harness-config.yml] as an example.

#### Setting up your databases

If possible, provide a docker-compose.yml file that starts and configures your test databases.

The test harness requires certain objects to be pre-created in your database. See [src/test/resources/docker/postgres-init.sh] as an example of what setup is required.

#### Adding a LiquibaseSdkSuite file
#### Adding a LiquibaseHarnessSuite file

In your `src/test/groovy` directory, create a file like:

```
class LiquibaseSdkSuite extends liquibase.sdk.test.BaseLiquibaseSdkSuite {
class ExtensionHarnessTest extends BaseHarnessSuite {
}
```
Expand Down
33 changes: 16 additions & 17 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,18 +8,17 @@ versions and configurations
The test harness logically consists of three parts:
1. Test logic
1. Configuration files containing inputs for the test logic
1. Configuration filese containing outputs/expectations for the test logic.
1. Configuration files containing outputs/expectations for the test logic.

The built-in tests are designed to test an overall functional flow, iterating over all configured connections.
For each connection, it will run each applicable input configuration and compare it to the expected output/expectations.

Both the input and output configuration files can be defined in a way that makes them apply to all databases or to specific types and/or specific versions.

The general pattern is that for each directory containing configuration files:
- Files directly in that root apply to all databases. Example: `liquibase/sdk/test/changes/changelogs`
- Files in a subdirectory named for the database type apply to only that type of database. Example: `liquibase/sdk/test/changes/changelogs/mysql`
- Files in a subdirectory with a major version apply only to the major version of the database. Example: `liquibase/sdk/test/changes/changelogs/mysql/8`
- Files in a subdirectory with a minor version apply only to the minor version of the database. Example: `liquibase/sdk/test/changes/changelogs/mysql/8/0`
- Files directly in that root apply to all databases. Example: `liquibase/harness/change/changelogs`
- Files in a subdirectory named for the database type apply to only that type of database. Example: `liquibase/harness/change/changelogs/mysql`
- Files in a subdirectory with a version apply only to this version of the database. Example: `liquibase/harness/change/changelogs/mysql/8`

At each level in that hierarchy, new configurations can be added and/or can override configurations from a lower level.

Expand All @@ -36,11 +35,11 @@ Extensions that add support for additional databases and/or define additional fu

#### Configuration File

The test harness will look for a file called `liquibase.sdk.test.yml` in the root of your classpath.
The test harness will look for a file called `harness-config.yml` in the root of your classpath.

That file contains a list of the database connections to test against, as well as an ability to control which subsets of tests to run.

See `src/test/resources/liquibase.sdk.test.yml` to see what this repository is configured to use.
See `src/test/resources/harness-config.yml` to see what this repository is configured to use.

## For use in extensions

Expand All @@ -50,18 +49,18 @@ For more information on using the test harness in your extension, see [README.ex

## Change Objects Test

The `groovy/liquibase/sdk/test/ChangeObjectsTest.groovy` test executes changelogs against the database and validates the SQL generated by them as well as
The `groovy/liquibase/harness/ChangeObjectsTests.groovy` test executes changelogs against the database and validates the SQL generated by them as well as
whether they make the expected changes.

* The test behavior is as follows:
* It reads the changesets from the changelogs provided in `src/main/resources/liquibase/sdk/test/changelogs` folders (recursively)
* It reads the changesets from the changelogs provided in `src/main/resources/liquibase/harness/change/changelogs` folders (recursively)
* Runs the changeset thru the SqlGeneratorFactory to generate SQL
* Compares the generated SQL with the expected SQL (provided in `src/main/resources/liquibase/sdk/test/expectedSql`)
* Compares the generated SQL with the expected SQL (provided in `src/main/resources/liquibase/harness/change/expectedSql`)
* If the SQL generation is correct, the test then runs `liquibase update` to deploy the
changeset to the DB
* The test takes a snapshot of the database after deployment
* The deployed changes are then rolled back
* Finally, the actual DB snapshot is compared to the expected DB snapshot (provided in `src/main/resources/liquibase/sdk/test/expectedSnapshot`)
* Finally, the actual DB snapshot is compared to the expected DB snapshot (provided in `src/main/resources/liquibase/harness/change/expectedSnapshot`)

#### Types of input files
* The tests work with the 4 types of input files that are supported by liquibase itself - xml, yaml, json, sql.
Expand All @@ -71,20 +70,20 @@ To change it to another format, like 'sql' for instance, specify `-DinputFormat=


### Adding a change object test
1) Go to `src/main/resources/liquibase/sdk/test/changelogs` and add the xml (or other) changeset for the change type you
1) Go to `src/main/resources/liquibase/harness/change/changelogs` and add the xml (or other) changeset for the change type you
want to test.
- The framework tries to rollback changes after deploying them to DB. If liquibase knows how to do a rollback for that particular changeset, it will automatically do that.
If not, you will need to provide the rollback by yourself. To learn more about rollbacks read [Rolling back changesets](https://docs.liquibase.com/workflows/liquibase-community/using-rollback.html) article.
2) Go to `src/main/resources/liquibase/sdk/test/expectedSql` and add the expected generated SQL.
2) Go to `src/main/resources/liquibase/harness/change/expectedSql` and add the expected generated SQL.
- You will need to add this under the database specific folder.
- NOTE: If your changeset will generate multiple SQL statements, you should add each SQL statement as a separate line. (See `renameTable.sql` in the postgres folder for an example.)
- If you would like to test another DB type, please add the requisite folder.
3) Go to `src/main/resources/liquibase/sdk/test/expectedSnapshot` and add the expected DB Snapshot results.
3) Go to `src/main/resources/liquibase/harness/change/expectedSnapshot` and add the expected DB Snapshot results.
- To verify the absence of an object in a snapshot (such as with drop* commands) add `"_noMatch": true,` to that tree level where the missing object should be verified.
See [dropSequence.json](src/test/resources/expectedSnapshot/postgresql/dropSequence.json) as an example.
- You will need to add this under the database specific folder.
- If you would like to test another DB type, please add the requisite folder.
4) Go to your IDE and run the test class `ChangeObjectTests.groovy` (You can also choose to run `BaseLiquibaseSdkSuite`, or `LiquibaseSdkSuite` -- at present they all work the same).
4) Go to your IDE and run the test class `ChangeObjectTests.groovy` (You can also choose to run `BaseTestHarnessSuite`, or `LiquibaseHarnessSuiteTest` -- at present they all work the same).

# Running the Tests

Expand All @@ -94,12 +93,12 @@ Java 1.8
1) Make sure you have a docker container up and running first
2) Go to `src/test/resources/docker` and run `docker-compose up -d`.
Wait until the databases start up.
3) Open `src/test/groovy/liquibase/sdk/test/LiquibaseSdkSuite.groovy` in your IDE of choice and run it
3) Open `src/test/groovy/liquibase/harness/LiquibaseHarnessSuiteTest.groovy` in your IDE of choice and run it

## Running from the cmd line with Maven
Execute `mvn test` with the (optional) flags outlined below:
* `-DinputFormat=xml` or select from the other inputFormats listed in [Types of input files](#types-of-input-files)
* `-DchangeObjects=createTable,dropTable` flag allows you to override changeObjects configured in `liquibase.sdk.test.yml`. Use comma
* `-DchangeObjects=createTable,dropTable` flag allows you to run specific changeObjects rather than all. Use comma
separated lists.
* `-DdbName=mysql` overrides the database type. This is only a single value property for now.
* `-DdbVersion` overrides the database version. Works in conjunction with `-DdbName` flag.
Expand Down
2 changes: 1 addition & 1 deletion README.non-docker-test-environments.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ For instance, you may wish to run against a database running in the cloud or a n

## Getting Started Guide
#### The execution & writing of tests will remain the same -- but you will need to recreate the starting test environment yourself. See below for instructions:
1. In the `liquibase.sdk.test.yml` file, edit the jdbc URL to point to the hostname for your database.
1. In the `harness-config.yml` file, edit the jdbc URL to point to the hostname for your database.
1. If your username and/or password is different, you will need to modify those values as well.
1. Ensure that you set up the test environment before executing the test-harness tests
1. For example, if you need to run the tests against a postgres instance, then:
Expand Down
2 changes: 1 addition & 1 deletion pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<modelVersion>4.0.0</modelVersion>

<groupId>org.liquibase</groupId>
<artifactId>liquibase-sdk</artifactId>
<artifactId>liquibase-test-harness</artifactId>
<version>1.0-SNAPSHOT</version>
<properties>
<maven.compiler.source>1.8</maven.compiler.source>
Expand Down
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
package liquibase.sdk.test;
package liquibase.harness;

import org.junit.runner.RunWith;
import org.junit.runners.Suite;

@RunWith(Suite.class)
@Suite.SuiteClasses([ChangeObjectTests.class])
public abstract class BaseLiquibaseSdkSuite {
public abstract class BaseHarnessSuite {
}
Original file line number Diff line number Diff line change
@@ -1,18 +1,18 @@
package liquibase.sdk.test
package liquibase.harness

import liquibase.CatalogAndSchema
import liquibase.Liquibase
import liquibase.database.jvm.JdbcConnection
import liquibase.sdk.test.config.TestConfig
import liquibase.sdk.test.util.FileUtils
import liquibase.sdk.test.util.SnapshotHelpers
import liquibase.sdk.test.util.TestUtils
import liquibase.harness.config.TestConfig
import liquibase.harness.util.FileUtils
import liquibase.harness.util.SnapshotHelpers
import liquibase.harness.util.TestUtils
import org.junit.Assert
import org.junit.Assume
import spock.lang.Specification
import spock.lang.Unroll

import static liquibase.sdk.test.util.ChangeObjectTestHelper.*
import static liquibase.harness.util.ChangeObjectTestHelper.*

class ChangeObjectTests extends Specification {

Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
package liquibase.sdk.test
package liquibase.harness

import spock.lang.Specification

Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
package liquibase.sdk.test.config
package liquibase.harness.config

import liquibase.database.Database;

Expand Down
Original file line number Diff line number Diff line change
@@ -1,13 +1,12 @@
package liquibase.sdk.test.config
package liquibase.harness.config

import groovy.transform.ToString
import liquibase.Scope
import liquibase.database.DatabaseFactory
import liquibase.database.OfflineConnection
import liquibase.lockservice.LockServiceFactory
import liquibase.resource.ClassLoaderResourceAccessor
import liquibase.resource.ResourceAccessor
import liquibase.sdk.test.util.DatabaseConnectionUtil
import liquibase.harness.util.DatabaseConnectionUtil
import org.yaml.snakeyaml.Yaml

import java.util.logging.Logger
Expand All @@ -31,8 +30,8 @@ class TestConfig {
public static TestConfig getInstance() {
if (instance == null) {
Yaml configFileYml = new Yaml()
def testConfig = getClass().getResourceAsStream("/liquibase.sdk.test.yml")
assert testConfig != null : "Cannot find liquibase.sdk.test.yml in classpath"
def testConfig = getClass().getResourceAsStream("/harness-config.yml")
assert testConfig != null : "Cannot find harness-config.yml in classpath"

instance = configFileYml.loadAs(testConfig, TestConfig.class)

Expand Down
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
package liquibase.sdk.test.util
package liquibase.harness.util

import groovy.transform.builder.Builder
import liquibase.database.Database
import liquibase.sdk.test.config.DatabaseUnderTest
import liquibase.sdk.test.config.TestConfig
import liquibase.harness.config.DatabaseUnderTest
import liquibase.harness.config.TestConfig
import liquibase.util.StringUtil
import org.skyscreamer.jsonassert.JSONAssert

Expand Down Expand Up @@ -75,7 +75,7 @@ class ChangeObjectTestHelper {
}

static void saveAsExpectedSql(String generatedSql, TestInput testInput) {
File outputFile = "${TestConfig.instance.outputResourcesBase}/liquibase/sdk/test/expectedSql/" +
File outputFile = "${TestConfig.instance.outputResourcesBase}/liquibase/harness/change/expectedSql/" +
"${testInput.databaseName}/${testInput.changeObject}.sql" as File
outputFile.parentFile.mkdirs()
outputFile.write(generatedSql)
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
package liquibase.sdk.test.util
package liquibase.harness.util

import liquibase.Scope
import liquibase.changelog.ChangeLogHistoryServiceFactory
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
package liquibase.sdk.test.util
package liquibase.harness.util

import liquibase.Scope
import liquibase.database.Database
Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
package liquibase.sdk.test.util
package liquibase.harness.util


import liquibase.sdk.test.config.TestConfig
import liquibase.harness.config.TestConfig
import liquibase.util.StreamUtil

class FileUtils {
Expand Down Expand Up @@ -29,10 +29,10 @@ class FileUtils {
}

static String getExpectedSqlFileContent(String changeObject, String databaseName, String version) {
return getFileContent(changeObject, databaseName, version, "liquibase/sdk/test/expectedSql", ".sql")
return getFileContent(changeObject, databaseName, version, "liquibase/harness/change/expectedSql", ".sql")
}

static String getExpectedSnapshotFileContent(String changeObject, String databaseName, String version) {
return getFileContent(changeObject, databaseName, version, "liquibase/sdk/test/expectedSnapshot", ".json")
return getFileContent(changeObject, databaseName, version, "liquibase/harness/change/expectedSnapshot", ".json")
}
}
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
package liquibase.sdk.test.util
package liquibase.harness.util

import liquibase.CatalogAndSchema
import liquibase.command.core.SnapshotCommand
Expand Down
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
package liquibase.sdk.test.util
package liquibase.harness.util

import liquibase.CatalogAndSchema
import liquibase.Liquibase
import liquibase.change.Change
import liquibase.changelog.ChangeSet
import liquibase.database.Database
import liquibase.sdk.test.config.DatabaseUnderTest
import liquibase.sdk.test.config.TestConfig
import liquibase.harness.config.DatabaseUnderTest
import liquibase.harness.config.TestConfig
import liquibase.sql.Sql
import liquibase.sqlgenerator.SqlGeneratorFactory

Expand Down Expand Up @@ -80,16 +80,16 @@ class TestUtils {
static SortedMap<String, String> getChangeLogPaths(DatabaseUnderTest database, String inputFormat) {
inputFormat = inputFormat ?: ""
def returnPaths = new TreeMap<String, String>()
for (String changeLogPath : TestConfig.instance.resourceAccessor.list(null, "liquibase/sdk/test/changelogs", true, true, false)) {
for (String changeLogPath : TestConfig.instance.resourceAccessor.list(null, "liquibase/harness/change/changelogs", true, true, false)) {
def validChangeLog = false

//is it a common changelog?
if (changeLogPath =~ "liquibase/sdk/test/changelogs/[\\w.]+${inputFormat}+\$") {
if (changeLogPath =~ "liquibase/harness/change/changelogs/[\\w.]+${inputFormat}+\$") {
validChangeLog = true
} else if (changeLogPath =~ "liquibase/sdk/test/changelogs/${database.name}/[\\w.]+${inputFormat}+\$") {
} else if (changeLogPath =~ "liquibase/harness/change/changelogs/${database.name}/[\\w.]+${inputFormat}+\$") {
//is it a database-specific changelog?
validChangeLog = true
} else if (changeLogPath =~ "liquibase/sdk/test/changelogs/${database.name}/${database.version}/[\\w" +
} else if (changeLogPath =~ "liquibase/harness/change/changelogs/${database.name}/${database.version}/[\\w" +
".]+${inputFormat}+\$") {
//is it a database-major-version specific changelog?
validChangeLog = true
Expand All @@ -104,7 +104,7 @@ class TestUtils {
}

Logger.getLogger(this.class.name).info("Found " + returnPaths.size() + " changeLogs for " + database.name +
"/" + database.version + " in liquibase/sdk/test/changelogs")
"/" + database.version + " in liquibase/harness/change/changelogs")


return returnPaths
Expand Down
Loading

0 comments on commit 974f3ad

Please sign in to comment.