add support for correlation ids to websocket messages

[eiknat]

rationale:

there are two use cases here:

requests
when sending a request, client and servers be able to know exactly what to look for with their expected response. when they send off a request with a specific correlation id, they can wait for the response that also contains that correlation id to then be able to identify the response that correlates to the request they sent.

live updates
these can also really benefit from correlation ids as websockets can get pretty noisy, clients being able to filter out events easily can help keep them stable under periods of heavy load.

additionally, some clients may not want to support live updates, or clients may want to have the option to turn off all live updates, only some live updates, or live updates when they're just coming too quickly for the client to support. we should provide a way allow developers to implement ways to filter out updates they're not currently interested in.

this also enables clients to create a "live" version of certain sort options without additional effort on the API side or allow individual users to turn off live updates completely (assuming a new user setting).

solution:

websocket messages should implement a correlation id in the same manner that AMQP clients are able to do. the correlation id would be added to websocket messages in this manner:

{"op":"CreateComment","correlation_id":"<uuid>", "data":{"<data>"}}

they can easily be made nullable as well for clients who chose not to use them or for backwards compatibility, a client or server that utilizes the correlation id would be looking for a specific correlation id, so nulls wouldn't be an issue for those who implement them.

[eiknat]

@dessalines if this seems reasonable, i'd love to take it on for a chance to dig in to rust :)

[dessalines]

You could, although I'm not exactly sure what purpose it would serve. The back-end already has the concept of rooms / contexts, IE collections of websocket connections that receive messages based on room:

  /// A map from generated random ID to session addr
  pub sessions: HashMap<ConnectionId, SessionInfo>,

  /// A map from post_id to set of connectionIDs
  pub post_rooms: HashMap<PostId, HashSet<ConnectionId>>,

  /// A map from community to set of connectionIDs
  pub community_rooms: HashMap<CommunityId, HashSet<ConnectionId>>,

  /// A map from user id to its connection ID for joined users. Remember a user can have multiple
  /// sessions (IE clients)
  user_rooms: HashMap<UserId, HashSet<ConnectionId>>,

What matters most for websocket, is making sure that user has joined / left the correct contexts, and receiving the right messages it should for those contexts.

How would correlation ids help me make sure, for example, if I'm in a post room, that I'm receiving all the correct CreateComment, EditComment, etc for that room?

[Nutomic]

@eiknat Do you still think this is needed, or can we close the issue?

[dessalines]

I did add the optional form_id field to comment create and receives, which serves the same purpose. lemmy-ui uses it help clear out or collapse comment boxes after you've sent a comment.

[Nutomic]

Okay then I'm closing this.

[Elara6331]

I think this would be very useful. I think it should be in the websocket message itself, not the form data, and should apply to all requests, not just some of them. The reason it would be useful is that currently, all requests must be done synchronously, as in you must wait for a response before sending another request of the same type. The reason for this is that if you want to read the response and you send multiple requests, you will not be able to tell which response corresponds to which request.

Another issue is the fact that the websocket API can send events when things happen. For example, when a comment is created. The lack of IDs necessitates using multiple connections for events and requests. The reason for this is that if I were to create a comment using the same connection as the one used to listen for comments, I wouldn't be able to tell which CommentResponse corresponds to my CreateComment request, because there may be other CommentResponses being sent as events.

I think this issue should be re-opened, because currently, the lack of such IDs makes using the WebSocket API for anything complex extremely annoying and requires very error-prone request-specific code (like checking the content of a comment, for example). This is an extremely simple feature that I believe wouldn't be difficult to implement (though I may be wrong), and would reduce the complexity of a lot of client handling code by a huge amount.

[dessalines]

That makes sense. I have some very hacky code in the front end to work around this very problem anyway.

[Nutomic]

Websocket is gone