Document how to adopt and adapt CDEvents

Document a series of scenarios about adopting CDEvents producer and consumer side. Describe how to adapt CDEvents with customData and how to contribute new fields and events to the community. Document declarative vs. imperative events and the subscriber model. Fixes: cdevents#66 Fixes: cdevents#42 Signed-off-by: Andrea Frittoli <[email protected]>
afrittoli · Oct 13, 2022 · 36f3d67 · 36f3d67
1 parent 749b7c6
commit 36f3d67
Show file tree

Hide file tree

Showing 8 changed files with 235 additions and 20 deletions.
diff --git a/images/adapter.svg b/images/adapter.svg
diff --git a/images/cdevents.drawio b/images/cdevents.drawio
diff --git a/images/consumer-adapter.svg b/images/consumer-adapter.svg
diff --git a/images/multiple-produced.svg b/images/multiple-produced.svg
diff --git a/images/multiple-received.svg b/images/multiple-received.svg
diff --git a/images/original-adapter.svg b/images/original-adapter.svg
diff --git a/images/watcher-producer.svg b/images/watcher-producer.svg
diff --git a/primer.md b/primer.md
@@ -24,16 +24,34 @@ document is updated accordingly to reflect the design decisions behind the chang
 <!-- toc -->
 - [History](#history)
 - [Design reflections](#design-reflections)
-  - [How does CDEvents enable tools to communicate in an interoperable
-    way?](#how-does-cdevents-enable-tools-to-communicate-in-an-interoperable-way)
+  - [How does CDEvents enable tools to communicate in an interoperable way?](#how-does-cdevents-enable-tools-to-communicate-in-an-interoperable-way)
   - [Why use events?](#why-use-events)
-  - [Why not point-to-point
-    communication?](#why-not-point-to-point-communication)
+  - [Why not point-to-point communication?](#why-not-point-to-point-communication)
+  - [Declarative vs. imperative events](#declarative-vs-imperative-events)
 - [Relations to CloudEvents](#relations-to-cloudevents)
+- [Versioning](#versioning)
+  - [Versioning of CDEvents](#versioning-of-cdevents)
+  - [Versioning of the CDEvents specification](#versioning-of-the-cdevents-specification)
+  - [Development of a new version](#development-of-a-new-version)
+- [Extending CDEvents](#extending-cdevents)
+  - [Adding new data to CDEvents](#adding-new-data-to-cdevents)
+  - [Adding new event types](#adding-new-event-types)
+- [Adopting CDEvents](#adopting-cdevents)
+  - [Producer-side architectures](#producer-side-architectures)
+    - [External event producer](#external-event-producer)
+    - [External event adapter](#external-event-adapter)
+    - [Multiple event formats](#multiple-event-formats)
+  - [Consumer-side architectures](#consumer-side-architectures)
+    - [Multiple event formats through adapter](#multiple-event-formats-through-adapter)
+    - [Consumer side adapters](#consumer-side-adapters)
+    - [Multiple received](#multiple-received)
 - [Acknowledgments](#acknowledgments)
 - [Use Cases](#use-cases)
 - [Design Decisions](#design-decisions)
   - [Keys, Values and Types](#keys-values-and-types)
+  - [Simplified data model](#simplified-data-model)
+  - [Artifacts](#artifacts)
+- [Whitepaper](#whitepaper)
 <!-- /toc -->
 
 ## History
@@ -89,17 +107,23 @@ create a system that will:
 * Create a coupled architecture - using point-to-point communication creates a
   tightly intertwined architecture difficult to expand and monitor.
 
-### Simplified data model
+### Declarative vs. imperative events
 
-In the initial version of CDEvents we tackled a simple scenario in which
-each artifact is built from a single repository and each service is deployed
-from a single artifact.
+With imperative events we intend events that are sent with the intent to trigger
+a specific reaction, like "start a pipeline" or "deploy an application".
+Imperative events create coupling between producer and consumer, and typically
+require some form of acknowledgement to be send back by the consumer of the
+original event back to the producer.
 
-This data model is somewhat limited, but is has allowed us to put more focus
-on the overall structure of the protocol in the first release.
+CDEvents are declarative events. With "declarative" we intend that the event
+producer sends information about an occurrence, but it does not now how this
+event will be used on the receiving side.
 
-We plan to extend the data model to support more complex scenarios in upcoming
-releases.
+A behavior similar to that of imperative events can be achieved by moving part
+of the business logic to an adapter that listens for specific declarative events
+and decides based on a set of policy to trigger actions in a downstream system,
+similarly to what is described in the [receiving
+adapters](#consumer-side-adapters) scenario.
 
 ## Relations to CloudEvents
 
@@ -189,15 +213,162 @@ Once a specification is ready for release, its number if updated, the event vers
 are finalized (`-draft` is removed), schemas are updated and a git tag is created for
 this last commit.
 
-### Artifacts
+## Extending CDEvents
 
-The specification has chosen for v0.1.0 to adopt [package-urls][purl] (or purls)
-as the format for any artifact identifier included in the spec. Purls provide a
-consistent format for artifact identifiers across different package types.
+The CDEvents specification is designed to evolve over time, to accommodate the
+need of CDEvents users and cover a growing number of use cases.
+In all cases we prefer backward compatible changes, which could be new fields in
+existing events or new event types. 
+[Versioning](#versioning) of messages is used for producers to validate messages
+before they are sent, and for consumer to know how to parse them.
 
-CDEvents wishes for a format that can be used to reference to an artifact, or
-package, that is independent from the hosting storage, which is a property which
-purls satisfy for several artifact types.
+### Adding new data to CDEvents
+
+If the data model of a CDEvent is not sufficient, events producers may choose to
+pass extra data through the `customData` field. Using `customData` can be an
+effective interim step, as it's easy to implement and can be used to help the
+migration process from existing events to CDEvents.
+
+In most cases though `customData` should not be considered as a permanent
+solution, since consumers don't know how to process this extra data, unless they
+implement producer specific logic and sacrifice part of the interoperability
+benefit of using CDEvents.
+
+Adding a new field to existing an existing CDEvent type is a backward compatible
+change. Aspects to consider when proposing a new field are:
+
+- is the field generally useful to the CD community? Data that is unique to a
+  single platform is likely to be rejected
+- what are the use cases where this field will be used?
+- what is the format for the new field? Please be as specific as possible
+- what is the name of the new field? Check the [SIG interoperability
+  vocabulary][sig-interop-vocabulariy] if a standard name exists. If not
+  consider proposing the new field name for the vocabulary as well. 
+
+### Adding new event types
+
+If a new event type is needed, it's good practice to contribute the new type
+into the CDEvents specification. Custom events can be used, but they should not
+use the "dev.cdevents" namespace for their type.
+
+Custom events are not interoperable, so existing CDEvents consumers won't be
+able to handle them. Introducing a custom event is simple enough on the producer
+side but it doesn't scale well with the number of consumers.
+
+Adding a new event to an existing CDEvents bucket is a backward compatible
+change. Aspects to consider when proposing a new event type are:
+
+- is the event generally useful to the CD community? Events that are very
+  specific to a single platform are likely to be rejected. If the event
+  represents a functionality that is currently only implemented in one platform,
+  but nonetheless generally useful, it can still be introduced in CDEvents
+- what are the use cases where this event will be used?
+- what are the sources of these events?
+- if the event includes a new kind of subject, what is the data model of the
+  subject? What is the format of its ID? Please be as specific as possible
+- what is the name of the new type? Check the [SIG interoperability
+  vocabulary][sig-interop-vocabulariy] if a standard name exists. If not
+  consider proposing the new field name for the vocabulary as well. 
+
+## Adopting CDEvents
+
+When adopting CDEvents, producers and consumers alike may adopt different
+strategies to support existing event producers and consumers that want to
+consider existing messaging systems, formats and event producers and consumers
+that are in place.
+CDEvents is a new specifications, but neither CloudEvents not events in general
+are a new idea, and several tools may already be using events or webhooks with a
+tool specific data model.
+
+Below we consider a set of common scenarios and how CDEvents may be
+incrementally introduced in an existing system. 
+
+### Producer-side architectures
+
+In the first three scenarios, CDEvents are introduced in the producer side,
+either directly in the tool or through some external component.
+
+#### External event producer
+
+If a tool does not produce events, it may be possible to use an external
+component to "watch" a tool output (for instance logs) and produce CDEvents.
+
+If the output does not contain all information required for the events, this
+limitation can be worked-around by adding the missing data in the tool output.
+
+This solution may be brittle, because the tool output may not be a stable
+interface for the tool, and it may change over time without notice.
+
+![watcher-producer](images/watcher-producer.svg)
+
+This is approach is certainly valid to build a proof-of-concept or experiment
+with events in an existing environment.
+
+If the output of the tool is structured and part of the tool API, this may also
+be adopted as a permanent solution, to keep separation of concerns between the
+tool itself and the process of generating events. 
+
+#### External event adapter
+
+If a tool does produce events, it may be possible to use an external adapter
+component to convert the existing events into CDEvents.
+
+![adapter](images/adapter.svg)
+
+Similar to the [previous case](#external-event-producer), incoming events may be
+missing data required by CDEvents. If the events come from a tool that we do not
+control, we cannot alter the content of the events, so we may request the tool
+maintainers to either add the extra data or, like in the next scenario, start
+producing CDEvents natively.
+
+#### Multiple event formats
+
+A tool may start producing CDEvents natively. If the tool previously produced
+events, some consumers may expect the pre-existing event format. This can be
+solved on the producer side by sending both format of events in parallel.
+
+In some cases it may be possible to use a single broker for both event types,
+for instance if both formats are CloudEvents based.
+
+![multiple-produced](images/multiple-produced.svg)
+
+### Consumer-side architectures
+
+Typically it won't be possible for all existing event consumers to switch to
+CDEvents at the same time. The following scenarios show how an incremental
+approach can be used to migrate consumers through CDEvents gradually.
+
+#### Multiple event formats through adapter
+
+In a variation of the previous two producer scenarios, the tool produces only
+one format of events, which is sent to the broker. The adapter subscribes to
+the events, converts them and publishes them back to the broker. Consumer may
+then subscribe to the type of events that they prefer.
+
+![original-adapter](images/original-adapter.svg)
+
+With this architecture, the adapter may even be able to convert messages from
+different tools, instead of just one.
+
+#### Consumer side adapters
+
+In this scenario, the tool and some consumers use CDEvents. New consumers are
+added that do not understand CDEvents, or that do not support events in general.
+An adapter can be used to convert a CDEvent into the consumer specific format or
+to extract data from a CDEvent and use it to invoke an API for the receiving
+side.
+
+![consumer-adapter](images/consumer-adapter.svg)
+
+#### Multiple received
+
+In this scenario, a new tool is added that produces CDEvents. An existing
+consumer wants to benefit from existing events as well as the events from the
+new tool. 
+
+![multiple-received](images/multiple-received.svg)
+
+A single consumer may receive events from heterogenous sources. 
 
 ## Acknowledgments
 
@@ -245,6 +416,28 @@ Keys and ENUM values are always written in
 [lowerCamelCase](https://en.wikipedia.org/wiki/Camel_case) for readability
 purposes.
 
+### Simplified data model
+
+In the initial version of CDEvents we tackled a simple scenario in which
+each artifact is built from a single repository and each service is deployed
+from a single artifact.
+
+This data model is somewhat limited, but is has allowed us to put more focus
+on the overall structure of the protocol in the first release.
+
+We plan to extend the data model to support more complex scenarios in upcoming
+releases.
+
+### Artifacts
+
+The specification has chosen for v0.1.0 to adopt [package-urls][purl] (or purls)
+as the format for any artifact identifier included in the spec. Purls provide a
+consistent format for artifact identifiers across different package types.
+
+CDEvents wishes for a format that can be used to reference to an artifact, or
+package, that is independent from the hosting storage, which is a property which
+purls satisfy for several artifact types.
+
 ## Whitepaper
 
 The [CDEvents whitepaper](./CDEvents_Whitepaper.pdf) has been originally
@@ -259,4 +452,7 @@ created and its mission.
 [whitepaper]: https://cd.foundation/blog/2022/06/07/cdevents-publishes-first-whitepaper/
 [ce-design-goals]: https://github.com/cloudevents/spec/blob/v1.0.2/cloudevents/primer.md#design-goals
 [ce-partitioning]: https://github.com/cloudevents/spec/blob/v1.0.1/extensions/partitioning.md
-[purl]: https://github.com/package-url/purl-spec/blob/master/PURL-SPECIFICATION.rst
+[purl]:
+    https://github.com/package-url/purl-spec/blob/master/PURL-SPECIFICATION.rst
+[sig-interop-vocabulariy]:
+    https://github.com/cdfoundation/sig-interoperability/blob/main/docs/vocabulary.md