From b5c87b38b52d862da4b3b6cbd0d37b0435942518 Mon Sep 17 00:00:00 2001 From: lostluck <13907733+lostluck@users.noreply.github.com> Date: Wed, 2 Aug 2023 12:08:39 -0700 Subject: [PATCH 1/3] mention prism in changes.md --- CHANGES.md | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/CHANGES.md b/CHANGES.md index c6e756ee9646..72abeabadf85 100644 --- a/CHANGES.md +++ b/CHANGES.md @@ -57,6 +57,7 @@ * New highly anticipated feature X added to Python SDK ([#X](https://github.com/apache/beam/issues/X)). * New highly anticipated feature Y added to Java SDK ([#Y](https://github.com/apache/beam/issues/Y)). * Spark 3.2.2 is used as default version for Spark runner ([#23804](https://github.com/apache/beam/issues/23804)). +* The Go SDK has a new default local runner, called Prism ([#24789](https://github.com/apache/beam/issues/24789)). ## I/Os @@ -70,10 +71,19 @@ * X feature added (Java/Python) ([#X](https://github.com/apache/beam/issues/X)). * The Go SDK now requires Go 1.20 to build. ([#27558](https://github.com/apache/beam/issues/27558)) +* The Go SDK has a new default local runner, Prism. ([#24789](https://github.com/apache/beam/issues/24789)). + * Prism is a portable runner that executes each transform independantly, ensuring coders. + * At this point it supercedes the Go direct runner in functionality. The Go direct runner is now deprecated. + * See https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/runners/prism/README.md for the goals and features of Prism. ## Breaking Changes -* Legacy runner support removed from Dataflow, all pipelines must use runner v2. +* Python SDK: Legacy runner support removed from Dataflow, all pipelines must use runner v2. +* The Go Direct Runner is now Deprecated. It remains available to reduce migration churn. + * Tests can be set back to the direct runner by overriding TestMain: `func TestMain(m *testing.M) { ptest.MainWithDefault(m, "direct") }` + * It's recommended to fix issues seen in tests using Prism, as they can also happen on any portable runner. + * Use the generic register package for your pipeline DoFns to ensure pipelines function on portable runners, like prism. + * Do not rely on closures or using package globals for DoFn configuration. They don't function on portable runners. ## Deprecations From 54c8d48d129333b015d5ba23e19729c34d90f03e Mon Sep 17 00:00:00 2001 From: lostluck <13907733+lostluck@users.noreply.github.com> Date: Wed, 2 Aug 2023 12:18:29 -0700 Subject: [PATCH 2/3] ws lint --- CHANGES.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGES.md b/CHANGES.md index 72abeabadf85..c3397f06a8f7 100644 --- a/CHANGES.md +++ b/CHANGES.md @@ -83,7 +83,7 @@ * Tests can be set back to the direct runner by overriding TestMain: `func TestMain(m *testing.M) { ptest.MainWithDefault(m, "direct") }` * It's recommended to fix issues seen in tests using Prism, as they can also happen on any portable runner. * Use the generic register package for your pipeline DoFns to ensure pipelines function on portable runners, like prism. - * Do not rely on closures or using package globals for DoFn configuration. They don't function on portable runners. + * Do not rely on closures or using package globals for DoFn configuration. They don't function on portable runners. ## Deprecations From 9a308c601ea0d5b066f8a2e505566968b415b565 Mon Sep 17 00:00:00 2001 From: lostluck <13907733+lostluck@users.noreply.github.com> Date: Wed, 2 Aug 2023 13:51:14 -0700 Subject: [PATCH 3/3] review comments --- CHANGES.md | 14 ++++---------- 1 file changed, 4 insertions(+), 10 deletions(-) diff --git a/CHANGES.md b/CHANGES.md index c3397f06a8f7..62e090a87324 100644 --- a/CHANGES.md +++ b/CHANGES.md @@ -54,14 +54,11 @@ ## Highlights -* New highly anticipated feature X added to Python SDK ([#X](https://github.com/apache/beam/issues/X)). -* New highly anticipated feature Y added to Java SDK ([#Y](https://github.com/apache/beam/issues/Y)). * Spark 3.2.2 is used as default version for Spark runner ([#23804](https://github.com/apache/beam/issues/23804)). * The Go SDK has a new default local runner, called Prism ([#24789](https://github.com/apache/beam/issues/24789)). ## I/Os -* Support for X source added (Java/Python) ([#X](https://github.com/apache/beam/issues/X)). * Python GCSIO is now implemented with GCP GCS Client instead of apitools ([#25676](https://github.com/apache/beam/issues/25676)) * Java KafkaIO now supports picking up topics via topicPattern ([#26948](https://github.com/apache/beam/pull/26948)) * Support for read from Cosmos DB Core SQL API ([#23604](https://github.com/apache/beam/issues/23604)) @@ -69,7 +66,6 @@ ## New Features / Improvements -* X feature added (Java/Python) ([#X](https://github.com/apache/beam/issues/X)). * The Go SDK now requires Go 1.20 to build. ([#27558](https://github.com/apache/beam/issues/27558)) * The Go SDK has a new default local runner, Prism. ([#24789](https://github.com/apache/beam/issues/24789)). * Prism is a portable runner that executes each transform independantly, ensuring coders. @@ -79,25 +75,23 @@ ## Breaking Changes * Python SDK: Legacy runner support removed from Dataflow, all pipelines must use runner v2. + +## Deprecations + * The Go Direct Runner is now Deprecated. It remains available to reduce migration churn. * Tests can be set back to the direct runner by overriding TestMain: `func TestMain(m *testing.M) { ptest.MainWithDefault(m, "direct") }` * It's recommended to fix issues seen in tests using Prism, as they can also happen on any portable runner. * Use the generic register package for your pipeline DoFns to ensure pipelines function on portable runners, like prism. * Do not rely on closures or using package globals for DoFn configuration. They don't function on portable runners. -## Deprecations - -* X behavior is deprecated and will be removed in X versions ([#X](https://github.com/apache/beam/issues/X)). - ## Bugfixes * Fixed DirectRunner bug in Python SDK where GroupByKey gets empty PCollection and fails when pipeline option `direct_num_workers!=1`.([#27373](https://github.com/apache/beam/pull/27373)) * Fixed BigQuery I/O bug when estimating size on queries that utilize row-level security ([#27474](https://github.com/apache/beam/pull/27474)) -* Fixed X (Java/Python) ([#X](https://github.com/apache/beam/issues/X)). ## Known Issues -* ([#X](https://github.com/apache/beam/issues/X)). +* TBD # [2.49.0] - 2023-07-17