Cancelling invocations

dev/restate/service/protocol.proto

@@ -39,6 +39,8 @@ message StartMessage {
   // protolint:disable:next REPEATED_FIELD_NAMES_PLURALIZED
   repeated StateEntry state_map = 4;
   bool partial_state = 5;
+
+  bool cancelled = 6;
 }
 
 // Type: 0x0000 + 1
@@ -102,7 +104,7 @@ message PollInputStreamEntryMessage {
   bytes value = 14;
 }
 
-// Kind: Non-Completable JournalEntry
+// Kind: Actionable JournalEntry
 // Type: 0x0400 + 1
 message OutputStreamEntryMessage {
   oneof result {
@@ -124,14 +126,14 @@ message GetStateEntryMessage {
   };
 }
 
-// Kind: Non-Completable JournalEntry
+// Kind: Actionable JournalEntry
 // Type: 0x0800 + 1
 message SetStateEntryMessage {
   bytes key = 1;
   bytes value = 3;
 }
 
-// Kind: Non-Completable JournalEntry
+// Kind: Actionable JournalEntry
 // Type: 0x0800 + 2
 message ClearStateEntryMessage {
   bytes key = 1;
@@ -163,7 +165,7 @@ message InvokeEntryMessage {
   };
 }
 
-// Kind: Non-Completable JournalEntry
+// Kind: Fallible JournalEntry
 // Type: 0x0C00 + 2
 message BackgroundInvokeEntryMessage {
   string service_name = 1;
@@ -188,7 +190,7 @@ message AwakeableEntryMessage {
   };
 }
 
-// Kind: Non-Completable JournalEntry
+// Kind: Fallible JournalEntry
 // Type: 0x0C00 + 4
 message CompleteAwakeableEntryMessage {
   // Identifier of the awakeable. See the spec for more details.
@@ -200,6 +202,12 @@ message CompleteAwakeableEntryMessage {
   };
 }
 
+// Kind: Actionable JournalEntry
+// Type: 0x0C00 + 5
+message CancelledEntryMessage {
+
+}
+
 // --- Nested messages
 
 // This failure object carries user visible errors,

service-invocation-protocol.md

@@ -41,7 +41,8 @@ Depending on the specific syscall, the Restate runtime generates as response eit
 
 Each syscall defines a priori whether it replies with an ack or a completion, or doesn't reply at all.
 
-There are a couple of special message streams for initializing and closing the invocation.
+Depending on the programming language and/or specific SDK, the SDKs expose a way to await completions using some form of
+`Future` type.
 
 ### Replaying and Processing
 
@@ -161,15 +162,21 @@ Flags:
 
 ### Entries and Completions
 
-We distinguish among two types of journal entries:
+For each journal entry the runtime commits the entry message and executes the action atomically. The runtime won't
+commit the entry, nor perform the action, if the entry is invalid. If an entry is not committed, all the subsequent
+entries are not committed as well.
+
+We distinguish among three types of journal entries:
 
 - Completable journal entries. These represent actions the runtime will perform, and for which consequently provide a
   completion value. All these entries have a `result` field defined in the message descriptor, defining the different
   variants of the completion value, and have a `COMPLETED` flag in the header.
-- Non-completable journal entries. These represent actions the runtime will perform, but won't provide any completion
-  value to it.
+- Fallible journal entries. These represent actions the runtime can perform, for which no completions are provided, but
+  it might reject them anyway in case they're invalid.
+- Actionable journal entries. Like fallible journal entries, but they can't be rejected and the runtime always considers
+  them valid.
 
-Whether a journal entry is completable or not is intrinsic in the definition of the journal action itself.
+The type of the journal entry is intrinsic in the definition of the journal action itself.
 
 The header format for journal entries applies both when the runtime is sending entries to the SDK during a replay, and
 when the SDK sends entries to the runtime during processing.
@@ -192,7 +199,7 @@ Flags:
 - 14 bits: Reserved
 - 1 bit `C`: `COMPLETED` flag. Mask: `0x0000_0001_0000_0000`
 
-Non-Completable journal entries:
+Fallible/Actionable journal entries:
 
     0                   1                   2                   3
     0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
@@ -273,15 +280,21 @@ descriptions in [`protocol.proto`](dev/restate/service/protocol.proto).
 | `InvokeEntryMessage`          | `0x0C01` | Invoke another Restate service.                                                                                                                                  |
 | `AwakeableEntryMessage`       | `0x0C03` | Arbitrary result container which can be completed from another service, given a specific id. See [Awakeable identifier](#awakeable-identifier) for more details. |
 
-**Non-Completable journal entries**
+**Fallible journal entries**
+
+| Message                         | Type     | Description                                                                                                |
+| ------------------------------- | -------- | ---------------------------------------------------------------------------------------------------------- |
+| `BackgroundInvokeEntryMessage`  | `0x0C02` | Invoke another Restate service at the given time, without waiting for the response.                        |
+| `CompleteAwakeableEntryMessage` | `0x0C04` | Complete an `Awakeable`, given its id. See [Awakeable identifier](#awakeable-identifier) for more details. |
+
+**Actionable journal entries**
 
-| Message                         | Type     | Description                                                                                                                                                                         |
-| ------------------------------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `OutputStreamEntryMessage`      | `0x0401` | Carries the service method output message(s) or terminal failure of the invocation. Note: currently the runtime accepts only one entry of this type, but this may change in future. |
-| `SetStateEntryMessage`          | `0x0800` | Set the value of a service instance state key.                                                                                                                                      |
-| `ClearStateEntryMessage`        | `0x0801` | Clear the value of a service instance state key.                                                                                                                                    |
-| `BackgroundInvokeEntryMessage`  | `0x0C02` | Invoke another Restate service at the given time, without waiting for the response.                                                                                                 |
-| `CompleteAwakeableEntryMessage` | `0x0C04` | Complete an `Awakeable`, given its id. See [Awakeable identifier](#awakeable-identifier) for more details.                                                                          |
+| Message                    | Type     | Description                                                                                                                                                                         |
+| -------------------------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `OutputStreamEntryMessage` | `0x0401` | Carries the service method output message(s) or terminal failure of the invocation. Note: currently the runtime accepts only one entry of this type, but this may change in future. |
+| `SetStateEntryMessage`     | `0x0800` | Set the value of a service instance state key.                                                                                                                                      |
+| `ClearStateEntryMessage`   | `0x0801` | Clear the value of a service instance state key.                                                                                                                                    |
+| `CancelledEntryMessage`    | `0x0C05` | Marks the beginning of the cancellation. See [Cancelling an invocation](#cancelling-an-invocation)                                                                                  |
 
 #### Awakeable identifier
 
@@ -345,6 +358,25 @@ closing the stream afterward.
 The following section describes optional features SDK developers MAY implement to improve the experience and provide
 additional features to the users.
 
+### Cancelling an invocation
+
+The Restate runtime has a feature to send a signal to "cancel" an invocation, in order to allow the user to eventually
+execute operations to revert/compensate its progress.
+
+The idea is that the SDK receives the signal at the beginning of the invocation, surfaces it in the user space and at
+the same time records in the journal where it surfaced.
+
+To start the cancellation process, the runtime invokes the SDK with the `StartMessage.cancelled` set to `true`. It's up
+to the SDKs to decide how and when to surface the cancellation signal to the user code, be it an exception, or a
+saga-like library, or a mix of both. During the handling of the cancellation, the SDK MUST be able to suspend.
+
+The SDK must guarantee that the cancellation signal is fired in the user code deterministically, meaning that on a
+subsequent replay of the invocation, e.g. due to a suspension or a network error, the signal should be fired again on
+the same point. For this purpose the SDKs can record, when needed, the point where the cancellation signal was fired by
+using the `CancelledEntryMessage`.
+
+When the SDK doesn't implement cancellation, the runtime cancellation signal has no effect on the target invocation.
+
 ### Custom entry messages
 
 The protocol allows the SDK to register an arbitrary entry type within the journal. The type MUST be `>= 0xFC00`. The