Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[prism] Good Non-Go Beam developer experience #32564

Open
Tracked by #29650
lostluck opened this issue Sep 25, 2024 · 0 comments
Open
Tracked by #29650

[prism] Good Non-Go Beam developer experience #32564

lostluck opened this issue Sep 25, 2024 · 0 comments
Labels

Comments

@lostluck
Copy link
Contributor

lostluck commented Sep 25, 2024

Right now, it's a bit of a chore to use Prism when on a Development Version of Beam off the Go SDK.

The current "PrismRunner" paradigm for both the Python and Java SDKs hinges on using a release version of Beam (at which point the SDK uses a binary from the Github releases), or using a version that you've built and spun up yourself somehow, that you somehow know how to use.

The proposal here is to detect whether the SDK is using a Development version, and if so, provide instructions on using building or otherwise using it.

There are a few cases to consider in the solution.

These all rely on the SDK's beam_version being marked as a "dev" version. (eg. __version__ = '2.60.0.dev' in Python and ReleaseInfo.getReleaseInfo().isDevSdkVersion() in Java).

The user does not have Go installed.
The user does have Go installed.

X

The user is executing the code from within a Beam github repo.

If we detect that PrismRunner is being used, and it's a Dev SDK version, the SDK should print/log in a user visible manner that that's the case, and that it's going to try using a "dev prism binary / in the Apache Beam cache".

Whether or not / exists, PrismRunner should do the following:

Check if the user has Go installed. If not, print provide a link to https://go.dev/doc/install for instructions on installing Go, with the recommendation to install the latest version of Go.

After that, Prism runner should provide a reminder instruction on a command line for building prism locally with Go. This should include a command line for the user to execute that will build a prism binary and put that binary into / directly. (eg. using the -o flag on go build.) The build instructions here should always be provided, so users know how to build a fresher instance of prism locally.

If / doesn't exist, execution should fail there, either referring to or possibly repeating the above instructions.


go build github.com/apache/beam/sdks/v2/go/cmd/prism -o /<Beam CACHE>/<NAME> is a command that will put the prism binary at the designated location.

In particular, if the user's working directory is under a Go Module (eg. in beam/sdks or deeper), it will be build the current repo's version of prism. This can be used to iterate on prism in non-Go SDKs.

If the user's working directory isn't in the Beam's Go module, then Go will return the following error:

no required module provides package github.com/apache/beam/sdks/go/v2/prism; to add it:
        go get github.com/apache/beam/sdks/go/v2/prism`

Or similarly the command go env GOMOD returns /dev/null, (on linux at least).

In that case, it's probably convenient to build and use the latest released version of Prism, which the following command does automatically.

go build github.com/apache/beam/sdks/v2/go/cmd/prism@latest -o /<Beam CACHE>/<NAME>


A more advanced version of this would be to build prism automatically if Go is available, without user intervention. Go will handle caching and similar for us. This enables always using go run instead of Go build, and avoids manipulating a binary WRT to the cache. I'll ask around and see how folks feel about that approach.

This would turn the flow to "Check if Go is installed, and if so, try and run with without @latest and then try and run with @latest".


A final alternative is using @master instead of @latest, but this then will do much heavier weight and less cacheable approach to fetching the prism code. I recommend instead being within the SDK context.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant