Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PostgreSQL quota manager and storage backend #3644

Merged
merged 65 commits into from
Nov 7, 2024
Merged
Changes from 1 commit
Commits
Show all changes
65 commits
Select commit Hold shift + click to select a range
21ef4b2
Duplicate MySQL files to new PostgreSQL directories, preserving git l…
robstradling Oct 1, 2024
78653c2
Restore MySQL files
robstradling Oct 1, 2024
9f48d08
Replace 'MySQL' references with 'PostgreSQL', preserving case of each…
robstradling Oct 1, 2024
b0f7dfe
Build the PostgreSQL protobuf definitions
robstradling Oct 1, 2024
18cf306
Rename PostgreSQL 'testdb' package to 'testdbpgx'
robstradling Oct 1, 2024
e01fb0b
Convert from database/sql API to jackc/pgx/v5 API
robstradling Oct 1, 2024
33451e6
Update imports
robstradling Oct 1, 2024
6d88880
Convert schema
robstradling Oct 1, 2024
74526b6
Use EXPLAIN output to estimate rows in PostgreSQL quota.Manager
robstradling Oct 1, 2024
e3583cd
PostgreSQL doesn't have an equivalent of MySQL's strict mode
robstradling Oct 1, 2024
a167577
Context required for pgx Commit() and Rollback() functions
robstradling Oct 1, 2024
c127965
Remove statement map and PrepareContext() calls, because pgx automati…
robstradling Oct 1, 2024
cb7bda2
Use ANY to bind arrays, instead of expanding placeholders
robstradling Oct 1, 2024
52f2867
Use PostgreSQL's COPY interface and a temporary table to upsert subtrees
robstradling Oct 1, 2024
51def32
Use PostgreSQL parameter placeholder syntax
robstradling Oct 1, 2024
22f955a
Format SQL queries consistently and without including unnecessary whi…
robstradling Oct 1, 2024
e8fdc37
Remove explicit delete from TreeControl, because no previous versions…
robstradling Oct 1, 2024
0f6f593
Adapt getVersion query to PostgreSQL
robstradling Oct 1, 2024
10d47f8
Context required for pgx Exec() and Ping() functions
robstradling Oct 1, 2024
e617388
No error returned by pgx RowsAffected() function
robstradling Oct 1, 2024
a1ccf26
Check Rows errors consistently and correctly
robstradling Oct 1, 2024
66d2c7f
Adapt errors.go to PostgreSQL
robstradling Oct 1, 2024
8218303
Adapt connection pool opening to PostgreSQL
robstradling Oct 1, 2024
92030b3
No error returned by pgxpool Close() function
robstradling Oct 1, 2024
f68b8b9
Pass arguments directly to Query in getLeavesByRangeInternal
robstradling Oct 1, 2024
1456b20
Adapt provider.go with plausible postgreSQLURI and drop calls to data…
robstradling Oct 1, 2024
aff11fc
Remove CockroachDB stuff from testdbpgx and specify plausible default…
robstradling Oct 1, 2024
bdd9faf
Tidy up / fix pgx imports
robstradling Oct 1, 2024
7b0ae0a
Plug PostgreSQL storage and quota providers into log_server and log_s…
robstradling Oct 1, 2024
2846d36
Enable the testdbpgx tokenizer to handle PL/pgSQL functions correctly…
robstradling Oct 1, 2024
0121720
To pass postgresql_quota_test requires fresher row estimates, so ANAL…
robstradling Oct 1, 2024
ba74553
PostgreSQL connection strings support TLS natively
robstradling Oct 1, 2024
2c5598f
Rebuild the PostgreSQL protobuf definitions with updated protoc-gen-go
robstradling Oct 1, 2024
b365aed
Use PostgreSQL's COPY interface for updating sequenced leaves
robstradling Oct 2, 2024
aa47b36
CopyFrom seems to require all lower case for table and column names
robstradling Oct 2, 2024
607c13a
Use PostgreSQL's COPY interface and a temporary table to queue leaves…
robstradling Oct 7, 2024
7a0df88
Fix conflict handling in multiple subtree insert query
robstradling Oct 8, 2024
7ce0258
Unmarshal storage settings before beginning transaction in beginTreeT…
robstradling Oct 8, 2024
55f0656
go get -u github.com/jackc/pgx/v5
robstradling Oct 8, 2024
b644630
go get github.com/google/trillian/storage/postgresql/testdbpgx
robstradling Oct 8, 2024
9ecaa58
Deleted all of the revisioned tree support
mhutchinson Oct 10, 2024
8e459da
Missed one
mhutchinson Oct 10, 2024
6698903
Disambiguate flag name
mhutchinson Oct 10, 2024
9c72070
Merge branch 'master' into postgresql_support
robstradling Oct 15, 2024
7c1f8c5
Merge branch 'google:master' into postgresql_support
robstradling Oct 25, 2024
2d5c941
Fix integration and unit tests
robstradling Oct 25, 2024
d1a21e1
Enable the BigBatch tests
robstradling Oct 25, 2024
0eac2ab
Hopefully fix trillian-pr-tests (trillian-opensource-ci) checks
robstradling Oct 28, 2024
965aa91
No need to sortLeavesForInsert
robstradling Oct 29, 2024
651b3b5
Update copyright details
robstradling Oct 30, 2024
72964d2
Remove vestigial Trees.PrivateKey and Trees.PublicKey fields
robstradling Oct 30, 2024
917f362
Remove vestigial Trees.HashStrategy, Trees.HashAlgorithm, and Trees.S…
robstradling Oct 31, 2024
dd85139
Add missing DROP FUNCTIONs.
robstradling Oct 31, 2024
32096c4
Call out the caveats in the quota manager more explicitly and up-front
robstradling Nov 5, 2024
8c40e33
Add a README.md that provides an overview of the PostgreSQL storage i…
robstradling Nov 5, 2024
28df5c0
Cover PostgreSQL in the mitigation for Issue #1297 in CODEOWNERS
robstradling Nov 5, 2024
286e278
Add PostgreSQL storage and quota implementations to the feature imple…
robstradling Nov 5, 2024
a4ee5cd
Full stops
robstradling Nov 5, 2024
1aa5094
Merge branch 'master' into postgresql_support
robstradling Nov 6, 2024
0462f62
Update CHANGELOG.md
robstradling Nov 6, 2024
6e3eb86
Remove Rob from CODEOWNERS
robstradling Nov 6, 2024
d41757e
Transfer PostgreSQL TODOs to Rob
robstradling Nov 6, 2024
378779f
Use the batched queue implementation, and remove the non-batched option
robstradling Nov 6, 2024
8102b6a
Remove vestigial Subtree.SubtreeRevision field
robstradling Nov 6, 2024
be96182
Remove vestigial TreeHead.TreeRevision field
robstradling Nov 6, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Remove CockroachDB stuff from testdbpgx and specify plausible default…
…TestPostgreSQLURI
  • Loading branch information
robstradling committed Oct 8, 2024
commit aff11fcf7601ca408c52f15d63063376edb5ed21
88 changes: 9 additions & 79 deletions storage/postgresql/testdbpgx/testdbpgx.go
Original file line number Diff line number Diff line change
Expand Up @@ -12,53 +12,40 @@
// See the License for the specific language governing permissions and
// limitations under the License.

// Package testdb creates new databases for tests.
// Package testdbpgx creates new PostgreSQL databases for tests.
package testdbpgx

import (
"bytes"
"context"
"fmt"
"log"
"net/url"
"os"
"strings"
"testing"
"time"

"github.com/google/trillian/testonly"
"github.com/jackc/pgx/v5/pgxpool"
"golang.org/x/sys/unix"
"k8s.io/klog/v2"

"github.com/jackc/pgx/v5/pgxpool"
_ "github.com/jackc/pgx/v5/pgxpool" // postgresql driver
_ "github.com/lib/pq" // postgres driver
)

const (
// PostgreSQLURIEnv is the name of the ENV variable checked for the test PostgreSQL
// instance URI to use. The value must have a trailing slash.
PostgreSQLURIEnv = "TEST_POSTGRESQL_URI"

// Note: pgxpool.New requires the URI to end with a slash.
defaultTestPostgreSQLURI = "root@tcp(127.0.0.1)/"

// CockroachDBURIEnv is the name of the ENV variable checked for the test CockroachDB
// instance URI to use. The value must have a trailing slash.
CockroachDBURIEnv = "TEST_COCKROACHDB_URI"

defaultTestCockroachDBURI = "postgres://root@localhost:26257/?sslmode=disable"
defaultTestPostgreSQLURI = "postgresql:///ctlog?host=localhost&user=ctlog"
)

type storageDriverInfo struct {
sqlDriverName string
schema string
uriFunc func(paths ...string) string
schema string
uriFunc func(paths ...string) string
}

var (
trillianPostgreSQLSchema = testonly.RelativeToPackage("../postgresql/schema/storage.sql")
trillianCRDBSchema = testonly.RelativeToPackage("../crdb/schema/storage.sql")
)

// DriverName is the name of a database driver.
Expand All @@ -67,20 +54,12 @@ type DriverName string
const (
// DriverPostgreSQL is the identifier for the PostgreSQL storage driver.
DriverPostgreSQL DriverName = "postgresql"
// DriverCockroachDB is the identifier for the CockroachDB storage driver.
DriverCockroachDB DriverName = "cockroachdb"
)

var driverMapping = map[DriverName]storageDriverInfo{
DriverPostgreSQL: {
sqlDriverName: "postgresql",
schema: trillianPostgreSQLSchema,
uriFunc: postgresqlURI,
},
DriverCockroachDB: {
sqlDriverName: "postgres",
schema: trillianCRDBSchema,
uriFunc: crdbURI,
schema: trillianPostgreSQLSchema,
uriFunc: postgresqlURI,
},
}

Expand All @@ -101,8 +80,8 @@ func postgresqlURI(dbRef ...string) string {
}

for _, ref := range dbRef {
separator := "/"
if strings.HasSuffix(stringurl, "/") {
separator := "&"
if strings.HasSuffix(stringurl, "&") {
separator = ""
}
stringurl = strings.Join([]string{stringurl, ref}, separator)
Expand All @@ -111,52 +90,12 @@ func postgresqlURI(dbRef ...string) string {
return stringurl
}

// crdbURI returns the CockroachDB connection URI to use for tests. It returns the
// value in the ENV variable defined by CockroachDBURIEnv. If the value is empty,
// returns defaultTestCockroachDBURI.
func crdbURI(dbRef ...string) string {
var uri *url.URL
if e := os.Getenv(CockroachDBURIEnv); len(e) > 0 {
uri = getURL(e)
} else {
uri = getURL(defaultTestCockroachDBURI)
}

return addPathToURI(uri, dbRef...)
}

func addPathToURI(uri *url.URL, paths ...string) string {
if len(paths) > 0 {
for _, ref := range paths {
currentPaths := uri.Path
// If the path is the root path, we don't want to append a slash.
if currentPaths == "/" {
currentPaths = ""
}
uri.Path = strings.Join([]string{currentPaths, ref}, "/")
}
}
return uri.String()
}

func getURL(unparsedurl string) *url.URL {
//nolint:errcheck // We're not expecting an error here.
u, _ := url.Parse(unparsedurl)
return u
}

// PostgreSQLAvailable indicates whether the configured PostgreSQL database is available.
func PostgreSQLAvailable() bool {
return dbAvailable(DriverPostgreSQL)
}

// CockroachDBAvailable indicates whether the configured CockroachDB database is available.
func CockroachDBAvailable() bool {
return dbAvailable(DriverCockroachDB)
}

func dbAvailable(driver DriverName) bool {
driverName := driverMapping[driver].sqlDriverName
uri := driverMapping[driver].uriFunc()
db, err := pgxpool.New(context.TODO(), uri)
if err != nil {
Expand Down Expand Up @@ -280,12 +219,3 @@ func SkipIfNoPostgreSQL(t *testing.T) {
}
t.Logf("Test PostgreSQL available at %q", postgresqlURI())
}

// SkipIfNoCockroachDB is a test helper that skips tests that require a local CockroachDB.
func SkipIfNoCockroachDB(t *testing.T) {
t.Helper()
if !CockroachDBAvailable() {
t.Skip("Skipping test as CockroachDB not available")
}
t.Logf("Test CockroachDB available at %q", crdbURI())
}