You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Right now, it's a bit of a chore to use Prism when on a Development Version of Beam off the Go SDK.
The current "PrismRunner" paradigm for both the Python and Java SDKs hinges on using a release version of Beam (at which point the SDK uses a binary from the Github releases), or using a version that you've built and spun up yourself somehow, that you somehow know how to use.
The proposal here is to detect whether the SDK is using a Development version, and if so, provide instructions on using building or otherwise using it.
There are a few cases to consider in the solution.
These all rely on the SDK's beam_version being marked as a "dev" version. (eg. __version__ = '2.60.0.dev' in Python and ReleaseInfo.getReleaseInfo().isDevSdkVersion() in Java).
The user does not have Go installed.
The user does have Go installed.
X
The user is executing the code from within a Beam github repo.
If we detect that PrismRunner is being used, and it's a Dev SDK version, the SDK should print/log in a user visible manner that that's the case, and that it's going to try using a "dev prism binary / in the Apache Beam cache".
Whether or not / exists, PrismRunner should do the following:
Check if the user has Go installed. If not, print provide a link to https://go.dev/doc/install for instructions on installing Go, with the recommendation to install the latest version of Go.
After that, Prism runner should provide a reminder instruction on a command line for building prism locally with Go. This should include a command line for the user to execute that will build a prism binary and put that binary into / directly. (eg. using the -o flag on go build.) The build instructions here should always be provided, so users know how to build a fresher instance of prism locally.
If / doesn't exist, execution should fail there, either referring to or possibly repeating the above instructions.
go build github.com/apache/beam/sdks/v2/go/cmd/prism -o /<Beam CACHE>/<NAME> is a command that will put the prism binary at the designated location.
In particular, if the user's working directory is under a Go Module (eg. in beam/sdks or deeper), it will be build the current repo's version of prism. This can be used to iterate on prism in non-Go SDKs.
If the user's working directory isn't in the Beam's Go module, then Go will return the following error:
no required module provides package github.com/apache/beam/sdks/go/v2/prism; to add it:
go get github.com/apache/beam/sdks/go/v2/prism`
Or similarly the command go env GOMOD returns /dev/null, (on linux at least).
In that case, it's probably convenient to build and use the latest released version of Prism, which the following command does automatically.
go build github.com/apache/beam/sdks/v2/go/cmd/prism@latest -o /<Beam CACHE>/<NAME>
A more advanced version of this would be to build prism automatically if Go is available, without user intervention. Go will handle caching and similar for us. This enables always using go run instead of Go build, and avoids manipulating a binary WRT to the cache. I'll ask around and see how folks feel about that approach.
This would turn the flow to "Check if Go is installed, and if so, try and run with without @latest and then try and run with @latest".
A final alternative is using @master instead of @latest, but this then will do much heavier weight and less cacheable approach to fetching the prism code. I recommend instead being within the SDK context.
The text was updated successfully, but these errors were encountered:
Right now, it's a bit of a chore to use Prism when on a Development Version of Beam off the Go SDK.
The current "PrismRunner" paradigm for both the Python and Java SDKs hinges on using a release version of Beam (at which point the SDK uses a binary from the Github releases), or using a version that you've built and spun up yourself somehow, that you somehow know how to use.
The proposal here is to detect whether the SDK is using a Development version, and if so, provide instructions on using building or otherwise using it.
There are a few cases to consider in the solution.
These all rely on the SDK's beam_version being marked as a "dev" version. (eg.
__version__ = '2.60.0.dev'
in Python andReleaseInfo.getReleaseInfo().isDevSdkVersion()
in Java).The user does not have Go installed.
The user does have Go installed.
X
The user is executing the code from within a Beam github repo.
If we detect that PrismRunner is being used, and it's a Dev SDK version, the SDK should print/log in a user visible manner that that's the case, and that it's going to try using a "dev prism binary / in the Apache Beam cache".
Whether or not / exists, PrismRunner should do the following:
Check if the user has Go installed. If not, print provide a link to https://go.dev/doc/install for instructions on installing Go, with the recommendation to install the latest version of Go.
After that, Prism runner should provide a reminder instruction on a command line for building prism locally with Go. This should include a command line for the user to execute that will build a prism binary and put that binary into / directly. (eg. using the
-o
flag ongo build
.) The build instructions here should always be provided, so users know how to build a fresher instance of prism locally.If / doesn't exist, execution should fail there, either referring to or possibly repeating the above instructions.
go build github.com/apache/beam/sdks/v2/go/cmd/prism -o /<Beam CACHE>/<NAME>
is a command that will put the prism binary at the designated location.In particular, if the user's working directory is under a Go Module (eg. in beam/sdks or deeper), it will be build the current repo's version of prism. This can be used to iterate on prism in non-Go SDKs.
If the user's working directory isn't in the Beam's Go module, then Go will return the following error:
Or similarly the command
go env GOMOD
returns/dev/null
, (on linux at least).In that case, it's probably convenient to build and use the latest released version of Prism, which the following command does automatically.
go build github.com/apache/beam/sdks/v2/go/cmd/prism@latest -o /<Beam CACHE>/<NAME>
A more advanced version of this would be to build prism automatically if Go is available, without user intervention. Go will handle caching and similar for us. This enables always using
go run
instead of Go build, and avoids manipulating a binary WRT to the cache. I'll ask around and see how folks feel about that approach.This would turn the flow to "Check if Go is installed, and if so, try and run with without
@latest
and then try and run with@latest
".A final alternative is using
@master
instead of@latest
, but this then will do much heavier weight and less cacheable approach to fetching the prism code. I recommend instead being within the SDK context.The text was updated successfully, but these errors were encountered: