Phil/workspace deps

[psFried]

Description:

Changes our Rust dependency configuration to use Workspace Inheritance. This is a very new feature, so this PR also increases the minimum supported rust version to 1.64 (the most recent stable release). All dependencies are now declared in in the root Cargo.toml file. Individual workspace crates now simply reference the workspace dependency using, for example, serde_json = { workspace = true }.

My initial position on how we should use this feature is to declare all dependencies in the root Cargo.toml. The rationale being that we should default to always ensuring that a single version of each dependency is used across all the crates in our workspace. If this becomes untenable, then we can revisit it later.

This also updates all dependencies to use explicit version requirements. There are no longer any version = "*" dependencies. Turns out that it's really a pain to use a crate having * dependencies as a dependency in an external crate. For example, depending on a workspace crate from another repo. Explicit version requirements make this much more robust. The * dependencies were also a barrier to us ever publishing crates to crates.io, so removing them seemed to move us in the right direction.

Many dependencies were updated as part of this, but not all of them. Notable updates are:

• tonic, prost, and the rest of the grpc gang: This brought some changes to the generated protobuf code. This included a cosmetic change to comment formatting that caused some code comments to get interpreted as code blocks, which then caused doc tests to fail. I disabled doc tests in proto-gazette to work around this.
• insta: Had a breaking change to how serde_json::value::RawValues are serialized. IMO the new behavior is preferable, but it did require updating a bunch of snapshots. issue here

Finally, I also made a few updates to the build in an attempt to reduce build times, which had gotten out of hand after the incorporate of animated-carnival:

• Allow catalog-test and end-to-end test to skip re-building stuff, and to only rely on the binaries that were output from the first two build stages.
• Rename some Makefile targets to clarify what they're building within the context of our now multi-platform build process.
• Group related outputs into a single cargo build command with multiple -p arguments instead of invoking cargo build separately for each desired output.

There's definitely lots more room for improvement with builds, but I don't want to spend a ton of time on build right now. Also, there seems to be a ton of variability of timing run-to-run in GH actions, so we may benefit from some additional time to characterize the build performance as it is.

---

This change is 

[williamhbaker]

Some nitpicks, but overall looks good.

I ran this locally to make sure it would build for me, and it seems like it does. Previously I could run make package and it would (re-)build everything I needed. Now, it seems that has changed, and to get an equivalent "rebuild everything if needed" I need to use make linux-binaries. This seems fine (although the nomenclature leaves something to be desired when building on mac), but will take a little bit of adjustment.

[psFried]

@williamhbaker thanks for the review. I took care of all the leftover comments, which had all been resolved.

to get an equivalent "rebuild everything if needed" I need to use make linux-binaries. This seems fine (although the nomenclature leaves something to be desired when building on mac), but will take a little bit of adjustment.

Yeah, you're correct that linux-binaries is the intended replacement for package. The naming was chosen because I'd assumed that the output would always be only linux binaries. But now that I take another look, it seems like we don't actually supply an explicit --target argument when building linux-gnu-binaries. Are you getting native mac binaries when you run make linux-binaries? I think I'll try to get my mac setup to test this locally (I've been meaning to, anyway), but it's seeming like we should have a simple and direct way of building this stuff on mac.

[williamhbaker]

Are you getting native mac binaries when you run make linux-binaries?

For the most part...the musl ones obviously are not:

agent:                 Mach-O 64-bit executable arm64
etcd:                  Mach-O 64-bit executable arm64
etcdctl:               Mach-O 64-bit executable arm64
fetch-open-graph:      Mach-O 64-bit executable arm64
flow-connector-proxy:  ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), static-pie linked, with debug_info, not stripped
flow-network-tunnel:   ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), static-pie linked, with debug_info, not stripped
flow-parser:           ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), static-pie linked, with debug_info, not stripped
flow-schema-inference: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), static-pie linked, with debug_info, not stripped
flow-schemalate:       ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), static-pie linked, with debug_info, not stripped
flowctl:               ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), static-pie linked, with debug_info, not stripped
flowctl-admin:         Mach-O 64-bit executable arm64
flowctl-go:            Mach-O 64-bit executable arm64
gazette:               Mach-O 64-bit executable arm64
sops:                  Mach-O 64-bit executable arm64

It's not immediately clear to me why flowctl would need to be a musl binary?

[psFried]

It's not immediately clear to me why flowctl would need to be a musl binary?

It doesn't need to be. It's basically just away to make it easy to install on a variety of linux systems. If you link against glibc, then the binary can only run on systems having a glibc version that's at least as new as the version it was built with. We kinda need to statically link some other dependencies, like librocksdb, since the versions in most apt/yum repositories aren't built with the RTTI feature. So it kinda just made sense to statically link the whole thing, and that way installation can be as simple as copying the binary to your local system and putting it on your PATH.

That said, if there was a good alternative to linux app packaging, then it might be worth moving to at some point. It's a little unfortunate to have to build some things with musl and others with glibc, and it'd be nice if we could just build everything with one or the other. I'm leaving that for a future exercise, though.