Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Document how to adopt and adapt CDEvents #77

Merged
merged 2 commits into from
Oct 19, 2022
Merged

Document how to adopt and adapt CDEvents #77

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
71d1feb
Document how to adopt and adapt CDEvents
afrittoli Oct 13, 2022
8d3ed95
Review comments
afrittoli Oct 18, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions images/adapter.svg
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
457 changes: 457 additions & 0 deletions images/cdevents.drawio
View file
Open in desktop

Large diffs are not rendered by default.

3 changes: 3 additions & 0 deletions images/consumer-adapter.svg
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
1 change: 1 addition & 0 deletions images/multiple-produced.svg
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
1 change: 1 addition & 0 deletions images/multiple-received.svg
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
1 change: 1 addition & 0 deletions images/original-adapter.svg
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
1 change: 1 addition & 0 deletions images/watcher-producer.svg
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
245 changes: 224 additions & 21 deletions primer.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,16 +24,34 @@ document is updated accordingly to reflect the design decisions behind the chang
<!-- toc -->
- [History](#history)
- [Design reflections](#design-reflections)
- [How does CDEvents enable tools to communicate in an interoperable
way?](#how-does-cdevents-enable-tools-to-communicate-in-an-interoperable-way)
- [How does CDEvents enable tools to communicate in an interoperable way?](#how-does-cdevents-enable-tools-to-communicate-in-an-interoperable-way)
- [Why use events?](#why-use-events)
- [Why not point-to-point
communication?](#why-not-point-to-point-communication)
- [Why not point-to-point communication?](#why-not-point-to-point-communication)
- [Declarative vs. imperative events](#declarative-vs-imperative-events)
- [Relations to CloudEvents](#relations-to-cloudevents)
- [Versioning](#versioning)
- [Versioning of CDEvents](#versioning-of-cdevents)
- [Versioning of the CDEvents specification](#versioning-of-the-cdevents-specification)
- [Development of a new version](#development-of-a-new-version)
- [Extending CDEvents](#extending-cdevents)
- [Adding new data to CDEvents](#adding-new-data-to-cdevents)
- [Adding new event types](#adding-new-event-types)
- [Adopting CDEvents](#adopting-cdevents)
- [Producer-side architectures](#producer-side-architectures)
- [External event producer](#external-event-producer)
- [External event adapter](#external-event-adapter)
- [Multiple event formats produced](#multiple-event-formats-produced)
- [Consumer-side architectures](#consumer-side-architectures)
- [Multiple event formats through adapter](#multiple-event-formats-through-adapter)
- [Consumer-side adapters](#consumer-side-adapters)
- [Multiple event formats consumed](#multiple-event-formats-consumed)
- [Acknowledgments](#acknowledgments)
- [Use Cases](#use-cases)
- [Design Decisions](#design-decisions)
- [Keys, Values and Types](#keys-values-and-types)
- [Simplified data model](#simplified-data-model)
- [Artifacts](#artifacts)
- [Whitepaper](#whitepaper)
<!-- /toc -->

## History
Expand Down Expand Up @@ -89,17 +107,29 @@ create a system that will:
* Create a coupled architecture - using point-to-point communication creates a
tightly intertwined architecture difficult to expand and monitor.

### Simplified data model
### Declarative vs. imperative events

In the initial version of CDEvents we tackled a simple scenario in which
each artifact is built from a single repository and each service is deployed
from a single artifact.
CDEvents are declarative events. With "declarative" we refer to event through
which the producer sends information about an occurrence, but it does not know
how the event will be used on the receiving side or even who will receive it.

This data model is somewhat limited, but is has allowed us to put more focus
on the overall structure of the protocol in the first release.
With imperative events we refer to events that are sent with the intent of
triggering a specific reaction, like "start a pipeline" or "deploy an
application". CDEvents do not support imperative events today; the specification
may include imperative events in future to foster interoperability in systems
that rely on imperative events today.

We plan to extend the data model to support more complex scenarios in upcoming
releases.
Imperative events create coupling between producer and consumer as they
typically require some form of acknowledgement to be send back by the
consumer of the original event back to the producer. Imperative events are
useful to implement workflows where the orchestration logic is centrally
managed by a single component.

A behavior similar to that of imperative events can be achieved by moving part
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Imperative events could also be achieved with point to point, right. Maybe this is a good use case for it? If it is, we may want to call that out, while explaining most events should be handled with a message bus, but in this particular use case, it does make sense for p2p. At least multiple ways to skin this cat, and the benefit with going to p2p here is they dont need to move any code anywhere, so may be easier approach?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I tend to associate imperative with p2p and API calls as well, however there are systems - like Keptn - that do imperative via events. In Keptn one defines the sequence of events like build test deploy and Keptn orchestrates that via events. It sends a "do a build" event, or "do a deployment" event, and one or more systems may take the job and do it an report back through a the original context ID provided by Keptn.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, right, Temporal also behaves this way except a little more vaguely. Okay, cool, I think we should also highlight imperative being a valid use case for p2p since we call out not doing it, but it may make sense for this particular instance.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, I will expand on this one.
What we discussed in the past is that we may include imperative events in future as a dedicated subgroup of the spec, but that may be something for the next release - if someone takes ownership of specifying them.

of the business logic to an adapter that listens for specific declarative events
and decides based on a set of policy to trigger actions in a downstream system,
similarly to what is described in the [receiving
adapters](#consumer-side-adapters) scenario.

## Relations to CloudEvents

Expand Down Expand Up @@ -189,15 +219,163 @@ Once a specification is ready for release, its number if updated, the event vers
are finalized (`-draft` is removed), schemas are updated and a git tag is created for
this last commit.

### Artifacts
## Extending CDEvents

The specification has chosen for v0.1.0 to adopt [package-urls][purl] (or purls)
as the format for any artifact identifier included in the spec. Purls provide a
consistent format for artifact identifiers across different package types.
The CDEvents specification is designed to evolve over time, to accommodate the
need of CDEvents users and cover a growing number of use cases.
In all cases we prefer backward compatible changes, which could be new fields in
existing events or new event types.
[Versioning](#versioning) of messages is used for producers to validate messages
before they are sent, and for consumer to know how to parse them.

CDEvents wishes for a format that can be used to reference to an artifact, or
package, that is independent from the hosting storage, which is a property which
purls satisfy for several artifact types.
### Adding new data to CDEvents

If the data model of a CDEvent is not sufficient, events producers may choose to
pass extra data through the `customData` field. Using `customData` can be an
effective interim step, as it's easy to implement and can be used to help the
migration process from existing events to CDEvents.

In most cases though `customData` should not be considered as a permanent
solution, since consumers don't know how to process this extra data, unless they
implement producer specific logic and sacrifice part of the interoperability
benefit of using CDEvents.

Adding a new field to an existing CDEvent type is considered a backward
compatible change - see the [versioning](#versioning) for more details.
Aspects to consider when proposing a new field are:

- is the field generally useful to the CD community? Data that is unique to a
single platform is likely to be rejected
- what are the use cases where this field will be used?
- what is the format for the new field? Please be as specific as possible
- what is the name of the new field? Check the [SIG interoperability
vocabulary][sig-interop-vocabulary] if a standard name exists. If not
propose the new field name for the vocabulary as well.

### Adding new event types

If a new event type is needed, it's good practice to contribute the new type
into the CDEvents specification. Custom events can be used, but they should not
use the "dev.cdevents" namespace for their type.

Custom events are not interoperable, so existing CDEvents consumers won't be
able to handle them. Introducing a custom event is simple enough on the producer
side but it doesn't scale well with the number of consumers.

Adding a new event to an existing CDEvents bucket is a backward compatible
change. Aspects to consider when proposing a new event type are:

- is the event generally useful to the CD community? Events that are very
specific to a single platform are likely to be rejected. If the event
represents a functionality that is currently only implemented in one platform,
but nonetheless generally useful, it can still be introduced in CDEvents
- what are the use cases where this event will be used?
- what are the sources of these events?
- if the event includes a new kind of subject, what is the data model of the
subject? What is the format of its ID? Please be as specific as possible
- what is the name of the new type? Check the [SIG interoperability
vocabulary][sig-interop-vocabulary] if a standard name exists. If not
consider proposing the new field name for the vocabulary as well.
afrittoli marked this conversation as resolved.
Show resolved Hide resolved

## Adopting CDEvents

When adopting CDEvents, producers and consumers alike may adopt different
strategies to support existing event producers and consumers that want to
consider existing messaging systems, formats and event producers and consumers
that are in place.
CDEvents is a new specification, but neither CloudEvents nor events in general
are a new idea, and several tools may already be using events or webhooks with a
tool specific data model.

Below we consider a set of common scenarios and how CDEvents may be
incrementally introduced in an existing system.

### Producer-side architectures

In the first three scenarios, CDEvents are introduced in the producer side,
either directly in the tool or through some external component.

#### External event producer

If a tool does not produce events, it may be possible to use an external
component to "watch" a tool output (for instance logs) and produce CDEvents.

If the output does not contain all information required for the events, this
limitation can be worked-around by adding the missing data in the tool output.

This solution may be brittle, because the tool output may not be a stable
interface for the tool, and it may change over time without notice.

![watcher-producer](images/watcher-producer.svg)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could benefit from "greying out" the consumer part of this picture, since there might be multiple consumer scenarios relevant for this producer scenario.
And it could be relevant to include the Cloudevents symbol in all pictures on this page, to not confuse the readers.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I split the scenarios into producer ones and consumer ones because they can be combined in different ways, I can make that more explicit in the first paragraph.

The colour of the consumer boxes has its relevance though since not all producer scenarios apply to consumer ones, but I can try to find a way to highlight this visually more. I'll try the greying option and see how it looks.


This approach is certainly valid to build a proof-of-concept or to experiment
with events in an existing environment.

If the output of the tool is structured and part of the tool API, this may also
be adopted as a permanent solution, to keep separation of concerns between the
tool itself and the process of generating events.

#### External event adapter

If a tool does produce events, it may be possible to use an external adapter
component to convert the existing events into CDEvents.

![adapter](images/adapter.svg)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same comment as for previous picture


Similar to the [previous case](#external-event-producer), incoming events may be
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Similar to the comment above, wouldn't the adapter need to have some context to what the event is? Why couldn't this just be a handler injected in the HTTP client's instead? It doesn't need to be external, from what I am gathering.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The lack of context is the main gotcha with this kind of external adapters approaches, I should do a better job of expressing that in the doc. Similar to the case above, if one does not control the source, adapting is a good interim solution.

I can adapt GitHub webhooks to CDEvents for instance.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Similar response above

missing data required by CDEvents. If the events come from a tool that we do not
control, we cannot alter the content of the events, so we may request the tool
maintainers to either add the extra data or, like in the next scenario, start
producing CDEvents natively.

#### Multiple event formats produced

A tool may start producing CDEvents natively. If the tool previously produced
events, some consumers may expect the pre-existing event format. This can be
solved on the producer side by sending both format of events in parallel.

In some cases it may be possible to use a single broker for both event types,
for instance if both formats are CloudEvents based.

![multiple-produced](images/multiple-produced.svg)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as for previous picture
And I don't think the "Event" keyword under the Cloudevents symbol adds any value. If it should be kept I believe the same should be used on the consuming side, but I prefer to omit the "Event" text completely here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I started with the word "event" only because it doesn't have to CloudEvents.
With CloudEvents it's easier because one can have the same broker passing CDEvents and CloudEvents around which is why I added a small CloudEvent logo there, but it's not obvious from the picture.

Perhaps I should simply remove CloudEvents from the picture, also to avoid fostering confusion between CloudEvents and CDEvents.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, since these pictures are just examples anyway, I see no issue with exemplifying using Cloudevents in them. But if some should contain the term "Event" and some not, and if some should be larger than others that needs to be explained in some way in a legend or similar I think.


### Consumer-side architectures

Typically it won't be possible for all existing event consumers to switch to
CDEvents at the same time. The following scenarios show how an incremental
approach can be used to migrate consumers towards CDEvents gradually.

#### Multiple event formats through adapter

In a variation of the previously mentioned producer-side architectures, the tool
produces only one format of events, which is sent to the broker. The adapter
subscribes to the events, converts them and publishes them back to the broker.
Consumer may then subscribe to the type of events that they prefer.

![original-adapter](images/original-adapter.svg)
e-backmark-ericsson marked this conversation as resolved.
Show resolved Hide resolved

With this architecture, the adapter may even be able to convert messages from
different tools, instead of just one.

#### Consumer-side adapters

In this scenario, the tool and some consumers use CDEvents. New consumers are
added that do not understand CDEvents, or that do not support events in general.
An adapter can be used to convert a CDEvent into the consumer specific format or
to extract data from a CDEvent and use it to invoke an API for the receiving
side.

![consumer-adapter](images/consumer-adapter.svg)

#### Multiple event formats consumed

In this scenario, a new tool is added that produces CDEvents. An existing
consumer wants to benefit from existing events as well as the events from the
new tool.

![multiple-received](images/multiple-received.svg)

A single consumer may receive events from heterogenous sources.

## Acknowledgments

Expand All @@ -211,7 +389,7 @@ There are two root use cases that we are considering:
- *Interoperability through CDEvents*: In this use case, platforms from the CD
landscape either produce or consume CDEvents. On the producing side, a system
broadcasts that certain value has been produced, like a code change, an
artifact or a test result. On the consumer side, a system takes an action that
artifact or a test result. On the consumer-side, a system takes an action that
takes advantage of that value that has been produced.

- *Observability & Metrics*: In this use case, platforms from the CD landscape
Expand Down Expand Up @@ -245,6 +423,28 @@ Keys and ENUM values are always written in
[lowerCamelCase](https://en.wikipedia.org/wiki/Camel_case) for readability
purposes.

### Simplified data model

In the initial version of CDEvents we tackled a simple scenario in which
each artifact is built from a single repository and each service is deployed
from a single artifact.

This data model is somewhat limited, but is has allowed us to put more focus
on the overall structure of the protocol in the first release.

We plan to extend the data model to support more complex scenarios in upcoming
releases.

### Artifacts

The specification has chosen for v0.1.0 to adopt [package-urls][purl] (or purls)
as the format for any artifact identifier included in the spec. Purls provide a
consistent format for artifact identifiers across different package types.

CDEvents wishes for a format that can be used to reference to an artifact, or
package, that is independent from the hosting storage, which is a property which
purls satisfy for several artifact types.

## Whitepaper

The [CDEvents whitepaper](./CDEvents_Whitepaper.pdf) has been originally
Expand All @@ -259,4 +459,7 @@ created and its mission.
[whitepaper]: https://cd.foundation/blog/2022/06/07/cdevents-publishes-first-whitepaper/
[ce-design-goals]: https://github.com/cloudevents/spec/blob/v1.0.2/cloudevents/primer.md#design-goals
[ce-partitioning]: https://github.com/cloudevents/spec/blob/v1.0.1/extensions/partitioning.md
[purl]: https://github.com/package-url/purl-spec/blob/master/PURL-SPECIFICATION.rst
[purl]:
https://github.com/package-url/purl-spec/blob/master/PURL-SPECIFICATION.rst
[sig-interop-vocabulary]:
https://github.com/cdfoundation/sig-interoperability/blob/main/docs/vocabulary.md