Skip to content

Commit

Permalink
Add pkg/pg with dialects.go & txdb.go (#910)
Browse files Browse the repository at this point in the history
* Add pkg/pg with dialects.go & txdb.go

Neither of these were in the actual pg package in chainlink repo.
dialects.go came from core/store/dialects and txdb.go from
core/internal/testutils/pgtest, but neither of these seem like they
deserve their own package in chainlink-common--we can lump all the
postgres specific common utilities under pkg/pg

* Add TestTxDBDriver, NewSqlxDB, SkipShort, SkipShortDB and SkipFlakey

* Add idempotency test of RegisterTxDb

* Create ctx from testing context, instead of using context.Background

* Only abort tx's when last connection is closed

Also: convert rest of panic isn't ordinary errors

* go mod tidy

* Split abort channel into one per connection object

All txdb connections share the same underlying connection to the
postgres db. Calling NewSqlxDB() or NewConnection() with dialect=txdb
doesn't create a new pg connection, it just creates a new tx with
BEGIN. Closing the connection with db.Close() issues ROLLBACK.

Both NewSqlxDB() and NewConneciton() choose random UUID's for their
dsn string, so we shouldn't have a case where the same dsn is opened
more than once. If that did happen, then these two different txdb
"connections" would be sharing the same transaction which would
mean closing the abort channel due to a query sent over one of them
would affect the other. Hopefully that's not a problem? If it is
I think our only option will be to go back to using context.Background
for all queries.

Before this commit, there was only one abort channel for the entire
txdb driver meaning that even two entirely different connections
opened with different dsn's could interfere with each other's queries.
This should fix that case, which is presumably the only case we
care about. Since each dsn corresponds to a different call to
NewSqlxDB() and the UUID's are generated randomly, there should no
longer be a conflict. Each txdb connection will have its own abort
channel.

* Errorf -> Fatalf on failure to register txdb driver

* Add in-memory DataSource using go-duckdb

* Fall back to testing txdb with in-memory backed db if CL_DATABASE_URL is not set

This allows us to test most of it in CI, and all locally

* Fix imports & fmt.Sprintf -> t.Log

* Add concurrency test for RegisterTxDb()

* Fix race condition

This showed up in some of the unit tests in the linked PR in chainlink repo

* Remove pg.SkipDB(), add DbUrlOrInMemory()

* pkg/pg -> pkg/sqlutil/pg

* NewSqlxDB -> NewTestDB, DbUrlOrInMemory -> TestURL
  • Loading branch information
reductionista authored and hendoxc committed Dec 17, 2024
1 parent 7f33239 commit 0b12d46
Show file tree
Hide file tree
Showing 8 changed files with 905 additions and 29 deletions.
40 changes: 29 additions & 11 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ module github.com/smartcontractkit/chainlink-common
go 1.23.3

require (
github.com/andybalholm/brotli v1.1.0
github.com/andybalholm/brotli v1.1.1
github.com/atombender/go-jsonschema v0.16.1-0.20240916205339-a74cd4e2851c
github.com/bytecodealliance/wasmtime-go/v23 v23.0.0
github.com/confluentinc/confluent-kafka-go/v2 v2.3.0
Expand All @@ -20,15 +20,18 @@ require (
github.com/hashicorp/go-plugin v1.6.2
github.com/iancoleman/strcase v0.3.0
github.com/invopop/jsonschema v0.12.0
github.com/jackc/pgx/v4 v4.18.3
github.com/jmoiron/sqlx v1.4.0
github.com/jonboulle/clockwork v0.4.0
github.com/jpillora/backoff v1.0.0
github.com/lib/pq v1.10.9
github.com/linkedin/goavro/v2 v2.12.0
github.com/marcboeker/go-duckdb v1.8.3
github.com/pelletier/go-toml/v2 v2.2.0
github.com/prometheus/client_golang v1.17.0
github.com/riferrei/srclient v0.5.4
github.com/santhosh-tekuri/jsonschema/v5 v5.2.0
github.com/scylladb/go-reflectx v1.0.1
github.com/shopspring/decimal v1.4.0
github.com/smartcontractkit/grpc-proxy v0.0.0-20240830132753-a7e17fec5ab7
github.com/smartcontractkit/libocr v0.0.0-20241007185508-adbe57025f12
Expand All @@ -51,17 +54,19 @@ require (
go.opentelemetry.io/otel/sdk/metric v1.30.0
go.opentelemetry.io/otel/trace v1.30.0
go.uber.org/goleak v1.3.0
go.uber.org/multierr v1.11.0
go.uber.org/zap v1.27.0
golang.org/x/crypto v0.27.0
golang.org/x/crypto v0.28.0
golang.org/x/exp v0.0.0-20240909161429-701f63a606c0
golang.org/x/tools v0.25.0
gonum.org/v1/gonum v0.15.0
google.golang.org/grpc v1.66.1
google.golang.org/protobuf v1.34.2
golang.org/x/tools v0.26.0
gonum.org/v1/gonum v0.15.1
google.golang.org/grpc v1.67.1
google.golang.org/protobuf v1.35.1
sigs.k8s.io/yaml v1.4.0
)

require (
github.com/apache/arrow-go/v18 v18.0.0 // indirect
github.com/bahlo/generic-list-go v0.2.0 // indirect
github.com/beorn7/perks v1.0.1 // indirect
github.com/buger/jsonparser v1.1.1 // indirect
Expand All @@ -73,20 +78,33 @@ require (
github.com/go-logr/stdr v1.2.2 // indirect
github.com/go-playground/locales v0.13.0 // indirect
github.com/go-playground/universal-translator v0.17.0 // indirect
github.com/goccy/go-json v0.10.3 // indirect
github.com/goccy/go-yaml v1.12.0 // indirect
github.com/golang/protobuf v1.5.4 // indirect
github.com/golang/snappy v0.0.5-0.20220116011046-fa5810519dcb // indirect; indirec
github.com/golang/snappy v0.0.5-0.20220116011046-fa5810519dcb // indirect
github.com/google/flatbuffers v24.3.25+incompatible // indirect
github.com/grpc-ecosystem/go-grpc-middleware/v2 v2.1.0 // indirect
github.com/grpc-ecosystem/grpc-gateway/v2 v2.22.0 // indirect
github.com/hashicorp/yamux v0.1.1 // indirect
github.com/jackc/chunkreader/v2 v2.0.1 // indirect
github.com/jackc/pgconn v1.14.3 // indirect
github.com/jackc/pgio v1.0.0 // indirect
github.com/jackc/pgpassfile v1.0.0 // indirect
github.com/jackc/pgproto3/v2 v2.3.3 // indirect
github.com/jackc/pgservicefile v0.0.0-20221227161230-091c0ba34f0a // indirect
github.com/jackc/pgtype v1.14.0 // indirect
github.com/klauspost/compress v1.17.11 // indirect
github.com/klauspost/cpuid/v2 v2.2.8 // indirect
github.com/leodido/go-urn v1.2.0 // indirect
github.com/mailru/easyjson v0.7.7 // indirect
github.com/mattn/go-colorable v0.1.13 // indirect
github.com/mattn/go-isatty v0.0.20 // indirect
github.com/matttproud/golang_protobuf_extensions v1.0.4 // indirect
github.com/mitchellh/go-wordwrap v1.0.1 // indirect
github.com/mitchellh/mapstructure v1.5.0 // indirect
github.com/mr-tron/base58 v1.2.0 // indirect
github.com/oklog/run v1.0.0 // indirect
github.com/pierrec/lz4/v4 v4.1.21 // indirect
github.com/pkg/errors v0.9.1 // indirect
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
github.com/prometheus/client_model v0.4.1-0.20230718164431-9a2bf3000d16 // indirect
Expand All @@ -96,14 +114,14 @@ require (
github.com/stretchr/objx v0.5.2 // indirect
github.com/wk8/go-ordered-map/v2 v2.1.8 // indirect
github.com/x448/float16 v0.8.4 // indirect
github.com/zeebo/xxh3 v1.0.2 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.30.0 // indirect
go.opentelemetry.io/proto/otlp v1.3.1 // indirect
go.uber.org/multierr v1.11.0 // indirect
golang.org/x/mod v0.21.0 // indirect
golang.org/x/net v0.29.0 // indirect
golang.org/x/net v0.30.0 // indirect
golang.org/x/sync v0.8.0 // indirect
golang.org/x/sys v0.25.0 // indirect
golang.org/x/text v0.18.0 // indirect
golang.org/x/sys v0.26.0 // indirect
golang.org/x/text v0.19.0 // indirect
golang.org/x/xerrors v0.0.0-20240903120638-7835f813f4da // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20240903143218-8af14fe29dc1 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20240903143218-8af14fe29dc1 // indirect
Expand Down
Loading

0 comments on commit 0b12d46

Please sign in to comment.