Optionally emit location updates for indexing purposes

[mi-yu]

Proposal

Implement an optional feature where the Index::update method can emit inscription location updates to downstream consumers. This would make custom indexers depending on ord easier to build and maintain across version upgrades.

Concretely, this would be implemented using channels. We can modify InscriptionUpdater to have a tokio::sync::mpsc::channel::Sender, then inside update_inscription_location send a message containing the inscription id, old and new location, and other metadata.

I would like to get some feedback on this idea before starting on any implementation!

Example

Below is some pseudocode demonstrating how the channel might set up and used:

struct LocationUpdateEvent {
  inscription_id: InscriptionId,
  offset: u64,
  origin: Origin,
  sat: Option<Sat>,
  charm: Option<Charm>,
  sequence_number: u32,
  block_number: u64,
  tx_id: TxId,
  tx_block_index: u32,
}

// Create channel
let (location_update_sender, location_update_receiver) = tokio::sync::mpsc::channel::<LocationUpdateEvent>(512);

// Initialize Index with sender
let index = Index::open(&options)
  .with_location_update_channel(location_update_sender);

// Inside update_inscription_location, send updates via the sender
fn update_inscription_location(
  &mut self,
  input_sat_ranges: Option<&VecDeque<(u64, u64)>>,
  flotsam: Flotsam,
  new_satpoint: SatPoint,
) {
  // ...

  if let Some(sender) = self.location_update_sender {
    self.location_update_sender.send(LocationUpdateEvent::new(...));
  }
}

// Start indexing, handle events in separate thread

let handler = tokio::spawn(async move {
  while let Some(i) = location_update_receiver.recv().await {
    // process message
  }
});

index.update()?;

[casey]

This seems easy enough. I want to make sure I understand the use case though.

When you say a custom indexer, is it a fork of ord, or is it a codebase that only uses the Index? Also, what is the consumer of the updates? Is an in-process channel the best interface, or would it ideal for you if ord wrote updates to a socket so that another process could consume the updates? The latter would be more complex, but it would allow the custom indexer to be a separate binary, and not require tight integration with ord.

[mi-yu]

Our use case specifically is for Magic Eden's indexer, we currently maintain a fork https://github.com/magicoss/ord-kafka/tree/dev, the main change is to send location updates to Kafka for our internal indexing flows. But we would prefer to rely on ord as a library, treating it as a black box and receiving location updates from it. This could be either as an in-process channel or over a socket, but I imagine an in-process channel would be easier to implement on the ord side

[casey]

Gotcha. Sending over a socket is maybe the long term solution, but that requires a bunch of choices, like what protocol, how to pass in the socket or address, etc, and how to serialize the data into the socket, so a channel probably makes sense. If you want to open a PR, it should be pretty easy.