Skip to content

Commit

Permalink
update: typos and sequence diagram
Browse files Browse the repository at this point in the history 
Signed-off-by: benjamin-j-powell <[email protected]>
  • Loading branch information
benjamin-j-powell committed Aug 24, 2023
1 parent 5c4c241 commit 8c3a53e
Show file tree
Hide file tree
Showing 2 changed files with 105 additions and 57 deletions.
Binary file removed images/links_flow.jpeg
View file
Binary file not shown.
162 changes: 105 additions & 57 deletions links.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

## Abstract

This proposal will outline how to connect individual CDEvents to eachother.
This proposal will outline how to connect individual CDEvents to each other.
Right now there's no way of associating events to one another without needing
to backtrack across certain context attributes, eg
[id](https://github.com/CDEvents/spec/blob/main/spec.md#id-context). While
Expand All @@ -21,7 +21,7 @@ assumptions being made when we talk about linking events
* **CI** - [Continuous integration](https://bestpractices.cd.foundation/learn/ci/)
* **CD** - [Continuous delivery](https://bestpractices.cd.foundation/learn/cd/)
* **Span** - A span is an end to end representation of all activities performed
in the CI/CD lifecycle of some entity
in the CI/CD life cycle of some entity
* **Link** - A link is an object containing relations between two CDEvents.
* **Global ID** - A global ID is some overarching ID for a given span.

Expand Down Expand Up @@ -52,32 +52,22 @@ and lastly the CD system which consume these artifacts shown in the diagram
below.

```
+---------------+ +-------------+
+-->| Build for Mac |---->| Test on Mac | --------+
+-----------+ | +---------------+ +-------------+ |
| Static | | +----------------+ +--------------+ | +---------------------------+
| Code |---+-->| Build for SLES |-->| Test on SLES | --------+-->| Promote & Deliver Release |
| Analysis | | +----------------+ +--------------+ | +---------------------------+
+-------------------+ +-----------------+
+-->| Build for Mac |-->| Test on Mac |---+
+-----------+ | +-------------------+ +-----------------+ |
| Static | | +-------------------+ +-----------------+ | +---------------------------+
| Code |---+-->| Build for SLES |-->| Test on SLES |---+-->| Promote & Deliver Release |
| Analysis | | +-------------------+ +-----------------+ | +---------------------------+
+-----------+ | +-------------------+ +-----------------+ |
+-->| Build for Windows |-->| Test on Windows | --+
+-->| Build for Windows |-->| Test on Windows |---+
+-------------------+ +-----------------+
```

The diagram above shows the CI event creating 3 separate artifacts that will
make it's way to the artifact store. Some CD system would then consume those
artifact, but would need to group all the artifact when the CI system finishes
its pipeline. This is not meant to be a full diagram of the whole CDEvents
flow, but a simple representation of the artifact(s) lifecycle.
flow, but a simple representation of the artifact(s) life cycle.

### 2. Generic UI

Expand Down Expand Up @@ -109,7 +99,11 @@ global ID and parent IDs.
"source": "dev/jenkins",
"type": "dev.cdevents.testsuite.finished.0.1.1",
"global_id": "00000000-0000-0000-0000-000000000001", # new global id field
"parent_ids": ["271069a8-fc18-44f1-b38f-9d70a1695819"], # new parent id field
"parent_ids": [ # new parent id field
{
"link_id": "271069a8-fc18-44f1-b38f-9d70a1695819",
}
],
"timestamp": "2023-03-20T14:27:05.315384Z"
},
"subject": {
Expand All @@ -124,7 +118,7 @@ global ID and parent IDs.
The `global_id` is an ID that will be generated when a new CDEvent span is
wanted or if no CDEvent span is present. This ID will follow the
[UUID](https://datatracker.ietf.org/doc/html/rfc4122) format. Global IDs will
serve as a bucket for all CDEvents with some sort of relation to eachother.
serve as a bucket for all CDEvents with some sort of relation to each other.

The `parent_ids` is an array of `context.id`s rather than `subject.id` since
the `subject.id` do not have a specified format, and may not be unique across
Expand All @@ -141,8 +135,8 @@ Below represents an example of a few CDEvents and how they link to one another.
"context": {
"version": "0.4.0-draft",
"id": "11111111-0000-0000-0000-000000000000",
"global_id": "00000000-0000-0000-0000-000000000000",
"parent_ids": [],
"global_id": "00000000-0000-0000-0000-000000000000",
"parent_ids": [],
"source": "/event/source/123",
"type": "dev.cdevents.change.merged.0.2.0",
"timestamp": "2023-03-20T14:27:05.315384Z"
Expand All @@ -167,8 +161,8 @@ Below represents an example of a few CDEvents and how they link to one another.
"context": {
"version": "0.4.0-draft",
"id": "22222222-0000-0000-0000-000000000000",
"global_id": "00000000-0000-0000-0000-000000000000",
"parent_ids": ["11111111-0000-0000-0000-000000000000"],
"global_id": "00000000-0000-0000-0000-000000000000",
"parent_ids": ["11111111-0000-0000-0000-000000000000"],
"source": "/event/source/123",
"type": "dev.cdevents.pipelinerun.queued.0.1.1",
"timestamp": "2023-03-20T14:27:05.315384Z"
Expand All @@ -191,8 +185,8 @@ Below represents an example of a few CDEvents and how they link to one another.
"context": {
"version": "0.4.0-draft",
"id": "33333333-0000-0000-0000-000000000000",
"global_id": "00000000-0000-0000-0000-000000000000",
"parent_ids": ["22222222-0000-0000-0000-000000000000"],
"global_id": "00000000-0000-0000-0000-000000000000",
"parent_ids": ["22222222-0000-0000-0000-000000000000"],
"source": "/event/source/123",
"type": "dev.cdevents.pipelinerun.started.0.1.1",
"timestamp": "2023-03-20T14:27:05.315384Z"
Expand Down Expand Up @@ -226,11 +220,11 @@ By establishing who has what parent, we can infer how they connect.

## Propagation

Span propogation will be handled differently depending on the protocol that is
Span propagation will be handled differently depending on the protocol that is
used. This proposal will not cover the technical details but will cover various
use cases that can occur in large systems. A separate technical design document
can be found [here](TODO/link/to/tech/doc) for those who want some guidance on
how propagation could be handled for a given protocal
how propagation could be handled for a given protocol

### Partial CDEvent Support

Expand All @@ -252,15 +246,15 @@ span may look like:
```

From the example above we can see that persisting a CDEvent event across A and
B can be easily done as they both support CDEvents. However, that propogation
B can be easily done as they both support CDEvents. However, that propagation
is broken when B talks to C.

### Different Protocols

As mentioned before, propogation is up to the protocol, e.g. HTTP(S), MQTT,
As mentioned before, propagation is up to the protocol, e.g. HTTP(S), MQTT,
AMQP, etc. A full list can be found [here](https://github.com/cloudevents/spec#cloudevents-documents).
Protocols that handle propogation differently should rely on the SDKs to
consume and continue the propogation.
Protocols that handle propagation differently should rely on the SDKs to
consume and continue the propagation.

### Separate Event Buses

Expand Down Expand Up @@ -292,7 +286,7 @@ sharing occurs due to consumer(s).
```

This proposal will not tackle these problems, but identify them as issues that
the technical design docoument will need to solve.
the technical design document will need to solve.

For case 1, introducing a links service becomes much more difficult. If a user
is to query for a full span since the link data may be persisted in different
Expand Down Expand Up @@ -323,7 +317,7 @@ The link spec definition will look like this
{
"global_id": "05CEDE1F-8C13-4699-80C3-B4AC6468414E",
"link_type": "TO",
"event_kind": "TRIGGER",
"event_transport": "TRIGGER",
"from": "F27E36A4-5C78-43C0-840A-52524DFEED03",
"to": "F004290E-5E45-45F4-B97A-FA82499F534C",
"tags": [
Expand All @@ -332,14 +326,14 @@ The link spec definition will look like this
}
```

| Name | Description |
|------------|----------------------------------------------------------------------|
| global_id | CDEvent span ID |
| link_type | An enum that represents the type of link, e.g. 'TO', 'FROM', 'GROUP' |
| event_kind | An enum that represents what kind of an event was consumed/produced. |
| from | Where the link is coming from |
| to | Where the link is going to |
| tags | Custom metadata that an individual link can have |
| Name | Description |
|-----------------|----------------------------------------------------------------------|
| global_id | CDEvent span ID |
| link_type | An enum that represents the type of link, e.g. 'TO', 'FROM', 'GROUP' |
| event_transport | An enum that represents how the event message was transported |
| from | Where the link is coming from |
| to | Where the link is going to |
| tags | Custom metadata that an individual link can have |

Below defines a list of `link_type` enums a kind can have

Expand All @@ -349,13 +343,13 @@ Below defines a list of `link_type` enums a kind can have
| FROM | When a link is receiving an event, it will use FROM to signal what service called it |
| GROUP | When a link is to be grouped with other events, it will use GROUP to establish a grouping |

Below defines a list of `event_kind` enums a link can have
Below defines a list of `event_transport` enums a link can have

| Name | Description |
|---------|--------------------------------------------------------------------|
| TRIGGER | When an event was triggered rather than called directly, e.g. cron |
| DIRECT | When an event was called directly instead of using the event bus |
| BUS | When an event was propogated via event bus |
| TRIGGER | When an CDEvent was triggered, e.g. cron |
| P2P | When a CDEvent sent using p2p (peer to peer) instead of using the event bus |
| BUS | When a CDEvent was propagated via event bus |

These links can be, but are not limited to, sent when a CDEvent has completed
to some collector or to the link service itself. Further the link service will
Expand All @@ -367,11 +361,61 @@ simply turning on linking payload aggregation, will send all links in the
payload. Mind you, this can make the payload very large, but may be good for
debugging.

The global ID header will continue to propogate, unless the user explicitly
The global ID header will continue to propagate, unless the user explicitly
starts a new CDEvent span. If there is no global ID header, the client will
generate one and that will be used for the lifetime of the whole events span

![link flow](images/links_flow.jpeg)
```
+-----+ +-----+ +-----+ +--------------+ +-----------+
| Git | | CI | | CD | | Link Service | | Event Bus |
+--+--+ +--+--+ +--+--+ +-------+------+ +-----+-----+
| | | #1 (send change merged event) | |
+--------------------------------------------------------------------------------------------------------------------------->|
| | | #2 (source change links) | |
+---------------------------------------------------------------------------------------------------->| |
| | | | |
| | | #3 (receive change merged event) | |
| |<--------------------------------------------------------------------------------------------------------------+
| | | #4 (send pipeline queued event) | |
| +-------------------------------------------------------------------------------------------------------------->|
| | | #5 (pipeline queued links w/ parent: #1) | |
| +--------------------------------------------------------------------------------------->| |
| | | #6 (send pipeline started event) | |
| +-------------------------------------------------------------------------------------------------------------->|
| | | #7 (pipeline started links w/ parent: #5) | |
| +--------------------------------------------------------------------------------------->| |
| | | #8 (send build queued event) | |
| +-------------------------------------------------------------------------------------------------------------->|
| | | #9 (build queued event links w/ parent: #7) | |
| +--------------------------------------------------------------------------------------->| |
| | | #10 (send build started event) | |
| +-------------------------------------------------------------------------------------------------------------->|
| | | #11 (build started event links w/ parent: #9) | |
| +--------------------------------------------------------------------------------------->| |
| | | #12 (send build completed event) | |
| +-------------------------------------------------------------------------------------------------------------->|
| | | #13 (build completed event links w/ parent: #11) | |
| +--------------------------------------------------------------------------------------->| |
| | | #14 (send pipeline finished event) | |
| +-------------------------------------------------------------------------------------------------------------->|
| | | #15 (pipeline finished event links w/ parent: #13) | |
| +--------------------------------------------------------------------------------------->| |
| | | | |
| | | #16 (receive pipeline finished event) | |
| | |<-------------------------------------------------------------------------------------------------+
| | | #17 (send pipeline queued event) | |
| | +------------------------------------------------------------------------------------------------->|
| | | #18 (pipeline queued links w/ parent: #15) | |
| | +-------------------------------------------------------------------------->| |
| | | #19 (send pipeline started event) | |
| | +------------------------------------------------------------------------------------------------->|
| | | #20 (pipeline started links w/ parent: #18) | |
| | +-------------------------------------------------------------------------->| |
| | | #21 (send environment created event) | |
| | +------------------------------------------------------------------------------------------------->|
| | | #22 (environment created links w/ parent: #20) | |
| | +-------------------------------------------------------------------------->| |
```

This show's an example of how these different types would be used in a CI/CD setting.

Expand All @@ -388,13 +432,17 @@ fast lookups. This section is going to describe how the proposed links format
will be scalable and also provide tactics on how DB read/writes can be done.

The purpose of the global ID is to ensure very fast lookups no matter the
database. Without a global ID the database or its client would need to recursively follow event references, upstream or downstream depending on the use case. A graph DB would easily provide that, and it is also possible to implement on top of SQL like DBs and document DBs, but it will never be as fast as querying for a global ID.

Instead a link service that processes and agnostically stores to some DB is
much prefer as it gives companies and developers options to choose from. When
using an SQL database, the global ID could be the secondary key to easily
retrieve indexed entities. Links could be easily sorted by timestamp which
should roughly coordinate to their linked neighbors, parent and child.
database. Without a global ID the database or its client would need to
recursively follow event references, upstream or downstream depending on the
use case. A graph DB would easily provide that, and it is also possible to
implement on top of SQL like DBs and document DBs, but it will never be as fast
as querying for a global ID.

Instead a link service that processes and stores the links to some DB is much
prefer as it gives companies and developers options to choose from. When using
an SQL database, the global ID could be the secondary key to easily retrieve
indexed entities. Links could be easily sorted by timestamps which should
roughly coordinate to their linked neighbors, parent and child.

CDEvents that are to be ingested by some service would also have to worry about
the number of events returned. This problem is mitigated in that only the
Expand Down

0 comments on commit 8c3a53e

Please sign in to comment.