Add Generic Forwarding documentation (#1872)

* Add generic forwarder documentation

* Update generic_forwarding.md based on comments

docs/tempo/website/configuration/_index.md

@@ -128,6 +128,41 @@ distributor:
         opencensus:
         kafka:
 
+    # Optional.
+    # Configures forwarders that asynchronously replicate ingested traces
+    # to specified endpoints. Forwarders work on per-tenant basis, so to
+    # fully enable this feature, overrides configuration must also be updated.
+    #
+    # Note: Forwarders work asynchronously and can fail or decide not to forward
+    # some traces. This feature works in a "best-effort" manner.
+    forwarders:
+
+        # Forwarder name. Must be unique within the list of forwarders. 
+        # This name can be referenced in the overrides configuration to
+        # enable forwarder for a tenant.
+      - name: <string>
+
+        # The forwarder backend to use
+        # Should be "otlpgrpc".
+        backend: <string>
+
+        # otlpgrpc configuration. Will be used only if value of backend is "otlpgrpc".
+        otlpgrpc:
+
+          # List of otlpgrpc compatible endpoints.
+          endpoints: <list of string>
+          tls:
+
+            # Optional.
+            # Disables TSL if set to true.
+            [insecure: <boolean> | default = false]
+
+            # Optional.
+            # Path to the TLS certificate. This field must be set if insecure = false.
+            [cert_file: <string | default = "">]
+      - (repetition of above...)
+
+
     # Optional.
     # Enable to log every received trace id to help debug ingestion
     # WARNING: Deprecated. Use log_received_spans instead.
@@ -1082,6 +1117,12 @@ overrides:
     # A value of 0 disables the limit.
     [max_bytes_per_tag_values_query: <int> | default = 5000000 (5MB) ]
 
+    # Generic forwarding configuration
+
+    # Per-user configuration of generic forwarder feature. Each forwarder in the list
+    # must refer by name to a forwarder defined in the distributor.forwarders configuration.
+    [forwarders: <list of strings>]
+
     # Metrics-generator configurations
 
     # Per-user configuration of the metrics-generator ring size. If set, the tenant will use a
@@ -1104,7 +1145,7 @@ overrides:
     # overrides settings in the global configuration.
     [metrics_generator_processor_service_graphs_histogram_buckets: <list of float>]
     [metrics_generator_processor_service_graphs_dimensions: <list of string>]
-    [metrics_generator_processor_span_metrics_histogram_buckets: <<list of float>]
+    [metrics_generator_processor_span_metrics_histogram_buckets: <list of float>]
     [metrics_generator_processor_span_metrics_dimensions: <list of string>]
 
     # Maximum number of active series in the registry, per instance of the metrics-generator. A

docs/tempo/website/operations/generic_forwarding.md

@@ -0,0 +1,16 @@
+---
+title: Generic forwarding
+weight: 13
+---
+
+# Generic forwarding
+
+Generic forwarding allows asynchronous replication of ingested traces. The distributor writes received spans to both the ingester and defined endpoints, if enabled. This feature works in a "best-effort" manner, meaning that no retries happen if an error occurs during replication. 
+
+>**Warning:** Generic forwarding does not work retroactively. Once enabled, the distributor only replicates freshly ingested spans.
+
+## Configuration
+
+Enabling generic forwarding requires the configuration of two sections. First, define a list of forwarders in the `distributor` section. Each forwarder must specify a unique `name`, supported `backend`, and backend-specific configuration. Second, reference these forwarders in the `overrides` section. This allows for fine-grained control over forwarding and makes it possible to enable this feature globally or on a per-tenant basis.
+
+For a detailed view of all the config options for the generic forwarding feature, please refer to [distributor]({{< relref "../configuration/#distributor" >}}) and [overrides]({{< relref "../configuration/#overrides" >}}) config pages.