Optimizing the write path for mixed storage v1/v2 state

[yurishkuro]

I was thinking more about this. In both the read and write paths, we want to avoid introducing obvious inefficiencies by requiring multiple data transformations. This applies not just to Jaeger v2 but also Jaeger v1, since many user are still running it at scale and model transformations are the major source of performance overhead (esp. memory allocations).

Take the write paths:

Legacy (1):

graph LR
  Client -->|model| Collector
  Collector --> |model| Storage[Storage v1]
  Storage --> |dbmodel| Database[(Database)]

Loading

OTLP (2):

graph LR
  Client -->|OTLP| Receiver
  Receiver --> |model| Collector
  Collector --> |model| Storage[Storage v1]
  Storage --> |dbmodel| Database[(Database)]

Loading

In these two examples most model transformations are necessary, although out could argue that in the OTLP case it should be possible to bypass the model part and go directly OTLP --> dbmodel. This is what Storage v2 API gives us:

OTLP with v2 storage (3):

graph LR
  Client -->|OTLP| Receiver
  Receiver --> |OTLP| Collector
  Collector --> |OTLP| Storage[Storage v2]
  Storage --> |dbmodel| Database[(Database)]

Loading

This change requires the v1 collector pipeline to support OTLP as the payload, which it does not. If we upgrade it (just the collector part), but still use the underlying v1 storage implementations, then the OTLP path still looks ok:

OTLP with v1 storage pretending to be v2 storage (4):

graph LR
  Client -->|OTLP| Receiver
  Receiver --> |OTLP| Collector
  Collector --> |OTLP| Storage2[Storage Adapter v2]
  Storage2 --> |model| Storage1[Storage v1]
  Storage1 --> |dbmodel| Database[(Database)]

Loading

(4) has the same amount of transformations as (2), so no regression. But (1) not looks bad:

Legacy with v1 storage pretending to be v2 storage (5):

graph LR
  Client -->|model| Receiver
  Receiver --> |OTLP| Collector
  Collector --> |OTLP| Storage2[Storage Adapter v2]
  Storage2 --> |model| Storage1[Storage v1]
  Storage1 --> |dbmodel| Database[(Database)]

Loading

Here we introduced an unnecessary transformation into OTLP that makes the path less efficient. This will improve once the storage is upgraded to v2 proper, but that will take some time.

My proposal is to consider upgrading the internal pipeline to support both model and OTLP simultaneously, and also to utilize the fact that the storage v2 might be an adapter over v1.

graph LR
  Client -->|model or OTLP| Collector
  Collector --> |model or OTLP| Processor{Processor}
  Processor --> |model| Storage1[Storage v2 Adapter over v1]
  Processor --> |OTLP| Storage2[Storage v2]
  Storage1 --> |dbmodel| Database[(Database)]
  Storage2 --> |dbmodel| Database[(Database)]

Loading