From 29494e8d712e9a4c26f516c3c2b1c74d611658d7 Mon Sep 17 00:00:00 2001
From: slinkydeveloper <francescoguard@gmail.com>
Date: Mon, 27 Nov 2023 11:46:43 +0100
Subject: [PATCH 1/4] Introduce the distinction fallible journal
entries/actionable journal entries
---
dev/restate/service/protocol.proto | 10 ++++----
service-invocation-protocol.md | 37 +++++++++++++++++++-----------
2 files changed, 29 insertions(+), 18 deletions(-)
diff --git a/dev/restate/service/protocol.proto b/dev/restate/service/protocol.proto
index 5be52b0..5216aae 100644
--- a/dev/restate/service/protocol.proto
+++ b/dev/restate/service/protocol.proto
@@ -102,7 +102,7 @@ message PollInputStreamEntryMessage {
bytes value = 14;
}
-// Kind: Non-Completable JournalEntry
+// Kind: Actionable JournalEntry
// Type: 0x0400 + 1
message OutputStreamEntryMessage {
oneof result {
@@ -124,14 +124,14 @@ message GetStateEntryMessage {
};
}
-// Kind: Non-Completable JournalEntry
+// Kind: Actionable JournalEntry
// Type: 0x0800 + 1
message SetStateEntryMessage {
bytes key = 1;
bytes value = 3;
}
-// Kind: Non-Completable JournalEntry
+// Kind: Actionable JournalEntry
// Type: 0x0800 + 2
message ClearStateEntryMessage {
bytes key = 1;
@@ -163,7 +163,7 @@ message InvokeEntryMessage {
};
}
-// Kind: Non-Completable JournalEntry
+// Kind: Fallible JournalEntry
// Type: 0x0C00 + 2
message BackgroundInvokeEntryMessage {
string service_name = 1;
@@ -188,7 +188,7 @@ message AwakeableEntryMessage {
};
}
-// Kind: Non-Completable JournalEntry
+// Kind: Fallible JournalEntry
// Type: 0x0C00 + 4
message CompleteAwakeableEntryMessage {
// Identifier of the awakeable. See the spec for more details.
diff --git a/service-invocation-protocol.md b/service-invocation-protocol.md
index e1053db..9fadb54 100644
--- a/service-invocation-protocol.md
+++ b/service-invocation-protocol.md
@@ -161,15 +161,21 @@ Flags:
### Entries and Completions
-We distinguish among two types of journal entries:
+For each journal entry the runtime commits the entry message and executes the action atomically. The runtime won't
+commit the entry, nor perform the action, if the entry is invalid. If an entry is not committed, all the subsequent
+entries are not committed as well.
+
+We distinguish among three types of journal entries:
- Completable journal entries. These represent actions the runtime will perform, and for which consequently provide a
completion value. All these entries have a `result` field defined in the message descriptor, defining the different
variants of the completion value, and have a `COMPLETED` flag in the header.
-- Non-completable journal entries. These represent actions the runtime will perform, but won't provide any completion
- value to it.
+- Fallible journal entries. These represent actions the runtime can perform, for which no completions are provided, but
+ it might reject them anyway in case they're invalid.
+- Actionable journal entries. Like fallible journal entries, but they can't be rejected and the runtime always considers
+ them valid.
-Whether a journal entry is completable or not is intrinsic in the definition of the journal action itself.
+The type of the journal entry is intrinsic in the definition of the journal action itself.
The header format for journal entries applies both when the runtime is sending entries to the SDK during a replay, and
when the SDK sends entries to the runtime during processing.
@@ -192,7 +198,7 @@ Flags:
- 14 bits: Reserved
- 1 bit `C`: `COMPLETED` flag. Mask: `0x0000_0001_0000_0000`
-Non-Completable journal entries:
+Fallible/Actionable journal entries:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
@@ -273,15 +279,20 @@ descriptions in [`protocol.proto`](dev/restate/service/protocol.proto).
| `InvokeEntryMessage` | `0x0C01` | Invoke another Restate service. |
| `AwakeableEntryMessage` | `0x0C03` | Arbitrary result container which can be completed from another service, given a specific id. See [Awakeable identifier](#awakeable-identifier) for more details. |
-**Non-Completable journal entries**
+**Fallible journal entries**
+
+| Message | Type | Description |
+| ------------------------------- | -------- | ---------------------------------------------------------------------------------------------------------- |
+| `BackgroundInvokeEntryMessage` | `0x0C02` | Invoke another Restate service at the given time, without waiting for the response. |
+| `CompleteAwakeableEntryMessage` | `0x0C04` | Complete an `Awakeable`, given its id. See [Awakeable identifier](#awakeable-identifier) for more details. |
+
+**Actionable journal entries**
-| Message | Type | Description |
-| ------------------------------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `OutputStreamEntryMessage` | `0x0401` | Carries the service method output message(s) or terminal failure of the invocation. Note: currently the runtime accepts only one entry of this type, but this may change in future. |
-| `SetStateEntryMessage` | `0x0800` | Set the value of a service instance state key. |
-| `ClearStateEntryMessage` | `0x0801` | Clear the value of a service instance state key. |
-| `BackgroundInvokeEntryMessage` | `0x0C02` | Invoke another Restate service at the given time, without waiting for the response. |
-| `CompleteAwakeableEntryMessage` | `0x0C04` | Complete an `Awakeable`, given its id. See [Awakeable identifier](#awakeable-identifier) for more details. |
+| Message | Type | Description |
+| -------------------------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `OutputStreamEntryMessage` | `0x0401` | Carries the service method output message(s) or terminal failure of the invocation. Note: currently the runtime accepts only one entry of this type, but this may change in future. |
+| `SetStateEntryMessage` | `0x0800` | Set the value of a service instance state key. |
+| `ClearStateEntryMessage` | `0x0801` | Clear the value of a service instance state key. |
#### Awakeable identifier
From 0faf79203655e08daa8b2ea41269324221d6cdae Mon Sep 17 00:00:00 2001
From: slinkydeveloper <francescoguard@gmail.com>
Date: Wed, 29 Nov 2023 11:54:51 +0100
Subject: [PATCH 2/4] Introduce StartMessage.cancelled
---
dev/restate/service/protocol.proto | 2 ++
1 file changed, 2 insertions(+)
diff --git a/dev/restate/service/protocol.proto b/dev/restate/service/protocol.proto
index 5216aae..9af732b 100644
--- a/dev/restate/service/protocol.proto
+++ b/dev/restate/service/protocol.proto
@@ -39,6 +39,8 @@ message StartMessage {
// protolint:disable:next REPEATED_FIELD_NAMES_PLURALIZED
repeated StateEntry state_map = 4;
bool partial_state = 5;
+
+ bool cancelled = 6;
}
// Type: 0x0000 + 1
From 276ea4902e404826ae2468a932d5f29fc8d012c7 Mon Sep 17 00:00:00 2001
From: slinkydeveloper <francescoguard@gmail.com>
Date: Mon, 4 Dec 2023 17:49:02 +0100
Subject: [PATCH 3/4] Introduce CancelledEntryMessage
---
dev/restate/service/protocol.proto | 6 ++++++
service-invocation-protocol.md | 1 +
2 files changed, 7 insertions(+)
diff --git a/dev/restate/service/protocol.proto b/dev/restate/service/protocol.proto
index 9af732b..e0313e9 100644
--- a/dev/restate/service/protocol.proto
+++ b/dev/restate/service/protocol.proto
@@ -202,6 +202,12 @@ message CompleteAwakeableEntryMessage {
};
}
+// Kind: Actionable JournalEntry
+// Type: 0x0C00 + 5
+message CancelledEntryMessage {
+
+}
+
// --- Nested messages
// This failure object carries user visible errors,
diff --git a/service-invocation-protocol.md b/service-invocation-protocol.md
index 9fadb54..86c563e 100644
--- a/service-invocation-protocol.md
+++ b/service-invocation-protocol.md
@@ -293,6 +293,7 @@ descriptions in [`protocol.proto`](dev/restate/service/protocol.proto).
| `OutputStreamEntryMessage` | `0x0401` | Carries the service method output message(s) or terminal failure of the invocation. Note: currently the runtime accepts only one entry of this type, but this may change in future. |
| `SetStateEntryMessage` | `0x0800` | Set the value of a service instance state key. |
| `ClearStateEntryMessage` | `0x0801` | Clear the value of a service instance state key. |
+| `CancelledEntryMessage` | `0x0C05` | Marks the beginning of the cancellation. See [Cancelling an invocation](#cancelling-an-invocation) |
#### Awakeable identifier
From a61802c273d5218bcfd80784c6aa52261fbf0b0a Mon Sep 17 00:00:00 2001
From: slinkydeveloper <francescoguard@gmail.com>
Date: Mon, 4 Dec 2023 17:49:35 +0100
Subject: [PATCH 4/4] Add some text for the description of the feature SDK
wise.
This text is meant to be open, so that SDKs can choose how to implement the handling of the cancellation signal.
---
service-invocation-protocol.md | 22 +++++++++++++++++++++-
1 file changed, 21 insertions(+), 1 deletion(-)
diff --git a/service-invocation-protocol.md b/service-invocation-protocol.md
index 86c563e..e0c5af1 100644
--- a/service-invocation-protocol.md
+++ b/service-invocation-protocol.md
@@ -41,7 +41,8 @@ Depending on the specific syscall, the Restate runtime generates as response eit
Each syscall defines a priori whether it replies with an ack or a completion, or doesn't reply at all.
-There are a couple of special message streams for initializing and closing the invocation.
+Depending on the programming language and/or specific SDK, the SDKs expose a way to await completions using some form of
+`Future` type.
### Replaying and Processing
@@ -357,6 +358,25 @@ closing the stream afterward.
The following section describes optional features SDK developers MAY implement to improve the experience and provide
additional features to the users.
+### Cancelling an invocation
+
+The Restate runtime has a feature to send a signal to "cancel" an invocation, in order to allow the user to eventually
+execute operations to revert/compensate its progress.
+
+The idea is that the SDK receives the signal at the beginning of the invocation, surfaces it in the user space and at
+the same time records in the journal where it surfaced.
+
+To start the cancellation process, the runtime invokes the SDK with the `StartMessage.cancelled` set to `true`. It's up
+to the SDKs to decide how and when to surface the cancellation signal to the user code, be it an exception, or a
+saga-like library, or a mix of both. During the handling of the cancellation, the SDK MUST be able to suspend.
+
+The SDK must guarantee that the cancellation signal is fired in the user code deterministically, meaning that on a
+subsequent replay of the invocation, e.g. due to a suspension or a network error, the signal should be fired again on
+the same point. For this purpose the SDKs can record, when needed, the point where the cancellation signal was fired by
+using the `CancelledEntryMessage`.
+
+When the SDK doesn't implement cancellation, the runtime cancellation signal has no effect on the target invocation.
+
### Custom entry messages
The protocol allows the SDK to register an arbitrary entry type within the journal. The type MUST be `>= 0xFC00`. The