Skip to content

Latest commit

 

History

History
45 lines (37 loc) · 2.07 KB

README.md

File metadata and controls

45 lines (37 loc) · 2.07 KB

Beam YAML API

The Beam YAML API provides a simple declarative syntax for describing pipelines that does not require coding experience or learning how to use an SDK—any text editor will do. Some installation may be required to actually execute a pipeline, but we envision various services (such as Dataflow) to accept yaml pipelines directly obviating the need for even that in the future. We also anticipate the ability to generate code directly from these higher-level yaml descriptions, should one want to graduate to a full Beam SDK (and possibly the other direction as well as far as possible).

Though we intend this syntax to be easily authored (and read) directly by humans, this may also prove a useful intermediate representation for tools to use as well, either as output (e.g. a pipeline authoring GUI) or consumption (e.g. a lineage analysis tool) and expect it to be more easily manipulated and semantically meaningful than the Beam protos themselves (which concern themselves more with execution).

More details

User-facing documentation for Beam YAML has moved to the main Beam site at https://beam.apache.org/documentation/sdks/yaml/

For information about contributing to Beam YAML see https://docs.google.com/document/d/19zswPXxxBxlAUmswYPUtSc-IVAu1qWvpjo1ZSDMRbu0