Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Submodules step 2/2: draw the rest of the owl #5609

Merged
merged 9 commits into from
Jan 23, 2024
Merged

Submodules step 2/2: draw the rest of the owl #5609

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
646c31c
not checked but builds
Groxx Jan 17, 2024
eeb52cf
hoepful dockerfile fix
Groxx Jan 20, 2024
1017112
add test for syncmap
Groxx Jan 22, 2024
8aa7c8a
fix grep check
Groxx Jan 22, 2024
f79e7c1
undo half-finished changes
Groxx Jan 22, 2024
aec4e58
minor updates to old readme
Groxx Jan 22, 2024
fedf776
a bit more readme cleanup
Groxx Jan 22, 2024
e373542
tidy should be checked in ci too
Groxx Jan 22, 2024
243374c
minor "what is this" note on relative imports in submodules
Groxx Jan 23, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,11 @@ WORKDIR /cadence
# Making sure that dependency is not touched
ENV GOFLAGS="-mod=readonly"

# Copy go mod dependencies and build cache
# Copy go mod dependencies and try to share the module download cache
COPY go.* ./
COPY cmd/server/go.* ./cmd/server/
COPY common/archiver/gcloud/go.* ./common/archiver/gcloud/
# go.work means this downloads everything, not just the top module
RUN go mod download

COPY . .
Expand Down
40 changes: 33 additions & 7 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,7 @@ endif
# note that vars that do not yet exist are empty, so any prerequisites defined below are ineffective here.
$(BUILD)/lint: $(BUILD)/fmt # lint will fail if fmt fails, so fmt first
$(BUILD)/proto-lint:
$(BUILD)/gomod-lint:
$(BUILD)/fmt: $(BUILD)/copyright # formatting must occur only after all other go-file-modifications are done
$(BUILD)/copyright: $(BUILD)/codegen # must add copyright to generated code, sometimes needs re-formatting
$(BUILD)/codegen: $(BUILD)/thrift $(BUILD)/protoc
Expand Down Expand Up @@ -139,9 +140,13 @@ LINT_SRC := $(filter-out %_test.go ./.gen/%, $(ALL_SRC))

# downloads and builds a go-gettable tool, versioned by go.mod, and installs
# it into the build folder, named the same as the last portion of the URL.
#
# unfortunately go.work and `go list -modfile=sub/module/go.mod` seem to interact badly,
# and some versions complain about duplicates, while others simply block it outright.
# the good news is that you can just drop that and `cd` to the folder and it works.
define go_build_tool
$Q echo "building $(or $(2), $(notdir $(1))) from internal/tools/go.mod..."
$Q go build -mod=readonly -modfile=internal/tools/go.mod -o $(BIN)/$(or $(2), $(notdir $(1))) $(1)
$Q cd internal/tools; go build -mod=readonly -o ../../$(BIN)/$(or $(2), $(notdir $(1))) $(1)
endef

# same as go_build_tool, but uses our main module file, not the tools one.
Expand Down Expand Up @@ -333,9 +338,18 @@ $(BUILD)/proto-lint: $(PROTO_FILES) $(STABLE_BIN)/$(BUF_VERSION_BIN) | $(BUILD)
$Q cd $(PROTO_ROOT) && ../$(STABLE_BIN)/$(BUF_VERSION_BIN) lint
$Q touch $@

# lints that go modules are as expected, e.g. parent does not import submodule.
# tool builds that need to be in sync with the parent are partially checked through go_mod_build_tool, but should probably be checked here too
$(BUILD)/gomod-lint: go.mod internal/tools/go.mod common/archiver/gcloud/go.mod | $(BUILD)
$Q # this is likely impossible as it'd be a cycle
$Q if grep github.com/uber/cadence/common/archiver/gcloud go.mod; then echo "gcloud submodule cannot be imported by main module" >&2; exit 1; fi
$Q # intentionally kept separate so the server does not include tool-only dependencies
$Q if grep github.com/uber/cadence/internal go.mod; then echo "internal module cannot be imported by main module" >&2; exit 1; fi
$Q touch $@

# note that LINT_SRC is fairly fake as a prerequisite.
# it's a coarse "you probably don't need to re-lint" filter, nothing more.
$(BUILD)/lint: $(LINT_SRC) $(BIN)/revive | $(BUILD)
$(BUILD)/code-lint: $(LINT_SRC) $(BIN)/revive | $(BUILD)
$Q echo "lint..."
$Q $(BIN)/revive -config revive.toml -exclude './vendor/...' -exclude './.gen/...' -formatter stylish ./...
$Q touch $@
Expand Down Expand Up @@ -388,7 +402,7 @@ endef
# useful to actually re-run to get output again.
# reuse the intermediates for simplicity and consistency.
lint: ## (re)run the linter
$(call remake,proto-lint lint)
$(call remake,proto-lint gomod-lint code-lint)

# intentionally not re-making, it's a bit slow and it's clear when it's unnecessary
fmt: $(BUILD)/fmt ## run gofmt / organize imports / etc
Expand Down Expand Up @@ -468,12 +482,22 @@ release: ## Re-generate generated code and run tests
$(MAKE) --no-print-directory test

build: ## Build all packages and all tests (ensures everything compiles)
$Q echo 'Building all packages...'
$Q echo 'Building all packages and submodules...'
$Q go build ./...
$Q cd common/archiver/gcloud; go build ./...
Groxx marked this conversation as resolved.
Show resolved Hide resolved
$Q cd cmd/server; go build ./...
Comment on lines 486 to +488
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if/when we get a ton of these, this is pretty easy to template. not worth it for 3 though IMO, make is pretty obtuse

$Q # "tests" by building and then running `true`, and hides test-success output
$Q echo 'Building all tests (~5x slower)...'
$Q # intentionally not -race due to !race build tags
$Q go test -exec /usr/bin/true ./... >/dev/null
$Q cd common/archiver/gcloud; go test -exec /usr/bin/true ./... >/dev/null
$Q cd cmd/server; go test -exec /usr/bin/true ./... >/dev/null

tidy: ## go mod tidy all packages
$Q # tidy in dependency order
$Q go mod tidy
$Q cd common/archiver/gcloud; go mod tidy || (echo "failed to tidy gcloud plugin, try manually copying go.mod contents into common/archiver/gcloud/go.mod and rerunning" >&2; exit 1)
$Q cd cmd/server; go mod tidy || (echo "failed to tidy main server module, try manually copying go.mod and common/archiver/gcloud/go.mod contents into cmd/server/go.mod and rerunning" >&2; exit 1)

clean: ## Clean build products
rm -f $(BINS)
Expand Down Expand Up @@ -509,8 +533,9 @@ endif

# all directories with *_test.go files in them (exclude host/xdc)
TEST_DIRS := $(filter-out $(INTEG_TEST_XDC_ROOT)%, $(sort $(dir $(filter %_test.go,$(ALL_SRC)))))
# all tests other than end-to-end integration test fall into the pkg_test category
PKG_TEST_DIRS := $(filter-out $(INTEG_TEST_ROOT)% $(OPT_OUT_TEST), $(TEST_DIRS))
# all tests other than end-to-end integration test fall into the pkg_test category.
# ?= allows passing specific (space-separated) dirs for faster testing
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

came in handy for my own testing, figured I might as well leave it in

PKG_TEST_DIRS ?= $(filter-out $(INTEG_TEST_ROOT)% $(OPT_OUT_TEST), $(TEST_DIRS))

# Code coverage output files
COVER_ROOT := $(BUILD)/coverage
Expand All @@ -536,11 +561,12 @@ COVER_PKGS = client common host service tools
# pkg -> pkg/... -> github.com/uber/cadence/pkg/... -> join with commas
GOCOVERPKG_ARG := -coverpkg="$(subst $(SPACE),$(COMMA),$(addprefix $(PROJECT_ROOT)/,$(addsuffix /...,$(COVER_PKGS))))"

test: bins ## Build and run all tests. This target is for local development. The pipeline is using cover_profile target
test: ## Build and run all tests. This target is for local development. The pipeline is using cover_profile target
$Q rm -f test
$Q rm -f test.log
$Q echo Running special test cases without race detector:
$Q go test -v ./cmd/server/cadence/
$Q # CAUTION: when changing to `go test ./...`, note that this DOES NOT test submodules. Those must be run separately.
$Q for dir in $(PKG_TEST_DIRS); do \
go test $(TEST_ARG) -coverprofile=$@ "$$dir" $(TEST_TAG) | tee -a test.log; \
done;
Expand Down
16 changes: 16 additions & 0 deletions cmd/server/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
This is the primary "kitchen sink" server build, which includes all first-party optional plugins, and is used as our Docker-provided binary.

For the most part, this means that day-to-day library upgrades should:
1. Update the main `go.mod`'s dependencies
2. `make tidy`

New submodules we want to include by default should:
1. Add a replace in this `go.mod` like others
2. Import and register it somehow
3. `make tidy`

And if you have problems tidying:
1. Copy/paste all included modules into this `go.mod` (`/go.mod` + `/common/archiver/gcloud/go.mod` currently)
2. `go mod tidy` this submodule and it will probably be correct, `make build` to make sure
3. Commit to save your results!
4. `make tidy` to make sure it's stable
158 changes: 158 additions & 0 deletions cmd/server/go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,158 @@
module github.com/uber/cadence/cmd/server

go 1.20

// build against the current code in the "main" (and gcloud) module, not a specific SHA.
//
// anyone outside this repo using this needs to ensure that both the "main" module and this module
// are at the same SHA for consistency, but internally we can cheat by telling Go that it's at a
// relative file path.
replace github.com/uber/cadence => ../..

replace github.com/uber/cadence/common/archiver/gcloud => ../../common/archiver/gcloud

// ringpop-go and tchannel-go depends on older version of thrift, yarpc brings up newer version
replace github.com/apache/thrift => github.com/apache/thrift v0.0.0-20161221203622-b2a4d4ae21c7

require (
github.com/Shopify/sarama v1.33.0 // indirect
github.com/VividCortex/mysqlerr v1.0.0 // indirect
github.com/aws/aws-sdk-go v1.44.180 // indirect
github.com/cactus/go-statsd-client/statsd v0.0.0-20191106001114-12b4e2b38748 // indirect
github.com/cch123/elasticsql v0.0.0-20190321073543-a1a440758eb9 // indirect
github.com/cristalhq/jwt/v3 v3.1.0 // indirect
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/dgryski/go-farm v0.0.0-20200201041132-a6ae2369ad13 // indirect
github.com/go-sql-driver/mysql v1.7.1 // indirect
github.com/gocql/gocql v0.0.0-20211015133455-b225f9b53fa1 // indirect
github.com/gogo/protobuf v1.3.2 // indirect
github.com/golang/mock v1.6.0 // indirect
github.com/google/uuid v1.5.0 // indirect
github.com/hashicorp/go-version v1.2.0 // indirect
github.com/iancoleman/strcase v0.2.0 // indirect
github.com/jmoiron/sqlx v1.2.1-0.20200615141059-0794cb1f47ee // indirect
github.com/lib/pq v1.2.0 // indirect
github.com/m3db/prometheus_client_golang v0.8.1 // indirect
github.com/olivere/elastic v6.2.37+incompatible // indirect
github.com/olivere/elastic/v7 v7.0.21 // indirect
github.com/opensearch-project/opensearch-go/v2 v2.2.0 // indirect
github.com/opentracing/opentracing-go v1.2.0 // indirect
github.com/pborman/uuid v0.0.0-20180906182336-adf5a7427709 // indirect
github.com/robfig/cron v1.2.0 // indirect
github.com/sirupsen/logrus v1.9.0 // indirect
github.com/startreedata/pinot-client-go v0.0.0-20230303070132-3b84c28a9e95 // latest release doesn't support pinot v0.12, so use master branch
github.com/stretchr/testify v1.8.3
github.com/uber-go/tally v3.3.15+incompatible // indirect
github.com/uber/cadence-idl v0.0.0-20240119174829-b5c878f44825
github.com/uber/ringpop-go v0.8.5 // indirect
github.com/uber/tchannel-go v1.22.2 // indirect
github.com/urfave/cli v1.22.4
github.com/valyala/fastjson v1.4.1 // indirect
github.com/xdg/scram v0.0.0-20180814205039-7eeb5667e42c // indirect
github.com/xwb1989/sqlparser v0.0.0-20180606152119-120387863bf2 // indirect
go.uber.org/atomic v1.10.0 // indirect
go.uber.org/cadence v0.19.0
go.uber.org/config v1.4.0 // indirect
go.uber.org/fx v1.13.1 // indirect
go.uber.org/multierr v1.6.0 // indirect
go.uber.org/thriftrw v1.29.2 // indirect
go.uber.org/yarpc v1.70.3 // indirect
go.uber.org/zap v1.13.0 // indirect
golang.org/x/net v0.19.0 // indirect
golang.org/x/sync v0.5.0 // indirect
golang.org/x/time v0.3.0 // indirect
golang.org/x/tools v0.16.0 // indirect
gonum.org/v1/gonum v0.7.0 // indirect
google.golang.org/grpc v1.59.0 // indirect
gopkg.in/validator.v2 v2.0.0-20180514200540-135c24b11c19 // indirect
gopkg.in/yaml.v2 v2.3.0 // indirect
)

require (
github.com/uber/cadence v0.0.0-00010101000000-000000000000
github.com/uber/cadence/common/archiver/gcloud v0.0.0-00010101000000-000000000000
)

require (
cloud.google.com/go v0.110.8 // indirect
cloud.google.com/go/compute v1.23.1 // indirect
cloud.google.com/go/compute/metadata v0.2.3 // indirect
cloud.google.com/go/iam v1.1.3 // indirect
cloud.google.com/go/storage v1.30.1 // indirect
github.com/BurntSushi/toml v0.4.1 // indirect
github.com/anmitsu/go-shlex v0.0.0-20161002113705-648efa622239 // indirect
github.com/apache/thrift v0.16.0 // indirect
github.com/benbjohnson/clock v0.0.0-20161215174838-7dc76406b6d3 // indirect
github.com/beorn7/perks v1.0.1 // indirect
github.com/cespare/xxhash/v2 v2.2.0 // indirect
github.com/cpuguy83/go-md2man/v2 v2.0.0 // indirect
github.com/eapache/go-resiliency v1.2.0 // indirect
github.com/eapache/go-xerial-snappy v0.0.0-20180814174437-776d5712da21 // indirect
github.com/eapache/queue v1.1.0 // indirect
github.com/facebookgo/clock v0.0.0-20150410010913-600d898af40a // indirect
github.com/fatih/structtag v1.2.0 // indirect
github.com/gogo/googleapis v1.3.2 // indirect
github.com/gogo/status v1.1.0 // indirect
github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
github.com/golang/protobuf v1.5.3 // indirect
github.com/golang/snappy v0.0.4 // indirect
github.com/google/s2a-go v0.1.4 // indirect
github.com/googleapis/enterprise-certificate-proxy v0.2.4 // indirect
github.com/googleapis/gax-go/v2 v2.12.0 // indirect
github.com/hailocab/go-hostpool v0.0.0-20160125115350-e80d13ce29ed // indirect
github.com/hashicorp/errwrap v1.0.0 // indirect
github.com/hashicorp/go-multierror v1.1.1 // indirect
github.com/hashicorp/go-uuid v1.0.2 // indirect
github.com/jcmturner/aescts/v2 v2.0.0 // indirect
github.com/jcmturner/dnsutils/v2 v2.0.0 // indirect
github.com/jcmturner/gofork v1.0.0 // indirect
github.com/jcmturner/gokrb5/v8 v8.4.2 // indirect
github.com/jcmturner/rpc/v2 v2.0.3 // indirect
github.com/jessevdk/go-flags v1.4.0 // indirect
github.com/jmespath/go-jmespath v0.4.0 // indirect
github.com/jonboulle/clockwork v0.4.0 // indirect
github.com/josharian/intern v1.0.0 // indirect
github.com/kisielk/errcheck v1.5.0 // indirect
github.com/klauspost/compress v1.15.9 // indirect
github.com/m3db/prometheus_client_model v0.1.0 // indirect
github.com/m3db/prometheus_common v0.1.0 // indirect
github.com/m3db/prometheus_procfs v0.8.1 // indirect
github.com/mailru/easyjson v0.7.6 // indirect
github.com/matttproud/golang_protobuf_extensions v1.0.2-0.20181231171920-c182affec369 // indirect
github.com/pierrec/lz4 v2.6.1+incompatible // indirect
github.com/pkg/errors v0.9.1 // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect
github.com/prometheus/client_golang v1.11.1 // indirect
github.com/prometheus/client_model v0.4.0 // indirect
github.com/prometheus/common v0.26.0 // indirect
github.com/prometheus/procfs v0.6.0 // indirect
github.com/rcrowley/go-metrics v0.0.0-20201227073835-cf1acfcdf475 // indirect
github.com/russross/blackfriday/v2 v2.0.1 // indirect
github.com/samuel/go-zookeeper v0.0.0-20201211165307-7117e9ea2414 // indirect
github.com/shurcooL/sanitized_anchor_name v1.0.0 // indirect
github.com/stretchr/objx v0.5.0 // indirect
github.com/uber-common/bark v1.2.1 // indirect
github.com/uber-go/mapdecode v1.0.0 // indirect
github.com/xdg/stringprep v1.0.0 // indirect
go.opencensus.io v0.24.0 // indirect
go.uber.org/dig v1.10.0 // indirect
go.uber.org/net/metrics v1.3.0 // indirect
golang.org/x/crypto v0.16.0 // indirect
golang.org/x/exp v0.0.0-20231226003508-02704c960a9b // indirect
golang.org/x/exp/typeparams v0.0.0-20220218215828-6cf2b201936e // indirect
golang.org/x/lint v0.0.0-20210508222113-6edffad5e616 // indirect
golang.org/x/mod v0.14.0 // indirect
golang.org/x/oauth2 v0.11.0 // indirect
golang.org/x/sys v0.15.0 // indirect
golang.org/x/text v0.14.0 // indirect
golang.org/x/xerrors v0.0.0-20220907171357-04be3eba64a2 // indirect
google.golang.org/api v0.128.0 // indirect
google.golang.org/appengine v1.6.7 // indirect
google.golang.org/genproto v0.0.0-20231016165738-49dd2c1f3d0b // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20231012201019-e917dd12ba7a // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20231030173426-d783a09b4405 // indirect
google.golang.org/protobuf v1.31.0 // indirect
gopkg.in/inf.v0 v0.9.1 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
honnef.co/go/tools v0.3.2 // indirect
)
Loading