OpenTelemetry Protocol with Apache Arrow Exporter initial skeleton (o…

…pen-telemetry#30619)

First PR to introduce the OpenTelemetry Protocol with
Apache Arrow exporter. From the upstream repository:
https://github.com/open-telemetry/otel-arrow/blob/main/collector/exporter/otelarrowexporter

**Link to tracking Issue:**  open-telemetry#26491 

This is a skeleton PR, therefore only the skeleton contains tests.
Compared with the upstream repository, the `factory_test.go` and
`config_test.go` files have been kept, the implementation tests in
`otelarrow_test.go` were excluded in this PR.

New README, [copied from the upstream
repository](https://github.com/open-telemetry/otel-arrow/blob/main/collector/exporter/otelarrowexporter/README.md).

.chloggen/new-otelarrow-exporter.yaml

@@ -0,0 +1,6 @@
+change_type: new_component
+component: otelarrow
+note: Skeleton of new OpenTelemetry Protocol with Apache Arrow Exporter.
+issues: [26491]
+subtext:
+change_logs: [user]

.github/CODEOWNERS

@@ -68,6 +68,7 @@ exporter/lokiexporter/                                   @open-telemetry/collect
 exporter/mezmoexporter/                                  @open-telemetry/collector-contrib-approvers @dashpole @billmeyer @gjanco
 exporter/opencensusexporter/                             @open-telemetry/collector-contrib-approvers @open-telemetry/collector-approvers
 exporter/opensearchexporter/                             @open-telemetry/collector-contrib-approvers @Aneurysm9 @MitchellGale @MaxKsyunz @YANG-DB
+exporter/otelarrowexporter/                              @open-telemetry/collector-contrib-approvers @jmacd @moh-osman3
 exporter/prometheusexporter/                             @open-telemetry/collector-contrib-approvers @Aneurysm9
 exporter/prometheusremotewriteexporter/                  @open-telemetry/collector-contrib-approvers @Aneurysm9 @rapphil
 exporter/pulsarexporter/                                 @open-telemetry/collector-contrib-approvers @dmitryax @dao-jun

.github/ISSUE_TEMPLATE/bug_report.yaml

@@ -68,6 +68,7 @@ body:
       - exporter/mezmo
       - exporter/opencensus
       - exporter/opensearch
+      - exporter/otelarrow
       - exporter/prometheus
       - exporter/prometheusremotewrite
       - exporter/pulsar

.github/ISSUE_TEMPLATE/feature_request.yaml

@@ -62,6 +62,7 @@ body:
       - exporter/mezmo
       - exporter/opencensus
       - exporter/opensearch
+      - exporter/otelarrow
       - exporter/prometheus
       - exporter/prometheusremotewrite
       - exporter/pulsar

.github/ISSUE_TEMPLATE/other.yaml

@@ -62,6 +62,7 @@ body:
       - exporter/mezmo
       - exporter/opencensus
       - exporter/opensearch
+      - exporter/otelarrow
       - exporter/prometheus
       - exporter/prometheusremotewrite
       - exporter/pulsar

exporter/otelarrowexporter/Makefile

@@ -0,0 +1 @@
+include ../../Makefile.Common

exporter/otelarrowexporter/README.md

@@ -0,0 +1,243 @@
+# OTel Arrow Exporter
+
+<!-- status autogenerated section -->
+| Status        |           |
+| ------------- |-----------|
+| Stability     | [development]: traces, metrics, logs   |
+| Distributions | [contrib] |
+| Issues        | [![Open issues](https://img.shields.io/github/issues-search/open-telemetry/opentelemetry-collector-contrib?query=is%3Aissue%20is%3Aopen%20label%3Aexporter%2Fotelarrow%20&label=open&color=orange&logo=opentelemetry)](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aopen+is%3Aissue+label%3Aexporter%2Fotelarrow) [![Closed issues](https://img.shields.io/github/issues-search/open-telemetry/opentelemetry-collector-contrib?query=is%3Aissue%20is%3Aclosed%20label%3Aexporter%2Fotelarrow%20&label=closed&color=blue&logo=opentelemetry)](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aclosed+is%3Aissue+label%3Aexporter%2Fotelarrow) |
+| [Code Owners](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CONTRIBUTING.md#becoming-a-code-owner)    | [@jmacd](https://www.github.com/jmacd), [@moh-osman3](https://www.github.com/moh-osman3) |
+
+[development]: https://github.com/open-telemetry/opentelemetry-collector#development
+[contrib]: https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol-contrib
+<!-- end autogenerated section -->
+
+Exports telemetry data using [OpenTelemetry Protocol with Apache
+Arrow](https://github.com/open-telemetry/otel-arrow) components with
+support for both OpenTelemetry Protocol with Apache
+Arrow and standard [OpenTelemetry Protocol](
+https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/protocol/otlp.md)
+(OTLP) protocol via gRPC.
+
+## Getting Started
+
+The [OpenTelemetry Protocol with Apache
+Arrow](https://github.com/open-telemetry/otel-arrow) exporter combines
+the features and configuration syntax of the core OpenTelemetry
+Collector [OTLP
+exporter](https://github.com/open-telemetry/opentelemetry-collector/tree/main/exporter/otlpexporter)
+component with additional support for the OpenTelemetry Protocol with
+Apache Arrow.
+
+OpenTelemetry Protocol with Apache Arrow supports column-oriented data
+transport using the Apache Arrow data format.  This component converts
+OTLP data into an optimized representation and then sends batches of
+data using Apache Arrow to encode the stream.  The OpenTelemetry
+Protocol with Apache Arrow receiver <!-- TODO add link when the
+corresponding skeleton is introduced.
+[](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/otelarrowreceiver)
+--> component contains logic to reverse the process used in this
+component.
+
+The use of an OpenTelemetry Protocol with Apache Arrow
+exporter-receiver pair is recommended when the network is expensive.
+Typically, expect to see a 50% reduction in bandwidth compared with
+the same data being sent using standard OTLP/gRPC with Zstd
+compression, batch sizes being equal.
+
+This component includes all the features and configuration of the core
+OTLP exporter, making it possible to upgrade from the core OTLP
+exporter component.  This is as simple as replacing "otlp" with
+"otelarrow" as the component name in the collector configuration.
+
+To enable the OpenTelemetry Protocol with Apache Arrow exporter,
+include it in the list of exporters for a pipeline.  The `endpoint`
+setting is required.  The `tls` setting is requirede for insecure
+transport.
+
+- `endpoint` (no default): host:port to which the exporter is going to send OTLP trace data,
+using the gRPC protocol. The valid syntax is described
+[here](https://github.com/grpc/grpc/blob/master/doc/naming.md).
+If a scheme of `https` is used then client transport security is enabled and overrides the `insecure` setting.
+- `tls`: see [TLS Configuration Settings](https://github.com/open-telemetry/opentelemetry-collector/blob/main/config/configtls/README.md) for the full set of available options.
+
+Example:
+
+```yaml
+exporters:
+  otelarrow/secure:
+    endpoint: external-collector:4317
+    tls:
+      cert_file: file.cert
+      key_file: file.key
+  otelarrow/insecure:
+    endpoint: internal-collector:4317
+    tls:
+      insecure: true
+```
+
+By default, `zstd` compression is enabled at the gRPC level.  See
+[compression configuration](#compression-configuration) below.  To
+disable gRPC-level compression, configure "none":
+
+```yaml
+exporters:
+  otelarrow:
+    compression: none
+    endpoint: ...
+    tls: ...
+```
+
+## Configuration
+
+Several helper files are leveraged to provide additional capabilities automatically:
+
+- [gRPC settings](https://github.com/open-telemetry/opentelemetry-collector/blob/main/config/configgrpc/README.md)
+- [TLS and mTLS settings](https://github.com/open-telemetry/opentelemetry-collector/blob/main/config/configtls/README.md)
+- [Queuing, retry and timeout settings](https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/exporterhelper/README.md)
+
+### Arrow-specific Configuration
+
+In the `arrow` configuration block, the following settings enable and
+disable the use of OTel Arrow as opposed to standard OTLP.
+
+- `disabled` (default: false): disables use of Arrow, causing the exporter to use standard OTLP
+- `disable_downgrade` (default: false): prevents this exporter from using standard OTLP.
+
+The following settings determine the resources that the exporter will use:
+
+- `num_streams` (default: number of CPUs): the number of concurrent Arrow streams
+- `max_stream_lifetime` (default: unlimited): duration after which streams are recycled.
+
+### Network Configuration
+
+This component uses `round_robin` by default as the gRPC load
+balancer.  This can be modified using the `balancer_name` setting, for
+example, to configure the `pick_first` balancer:
+
+```yaml
+exporters:
+  otelarrow:
+    balancer_name: pick_first
+    endpoint: ...
+    tls: ...
+```
+
+When the server or an intermediate proxy uses a keepalive setting, the
+Arrow-specific `max_stream_lifetime` setting is critical to avoiding
+abrupt termination of Arrow streams, which causes retries of the
+in-flight requests.  The maximum stream lifetime should be set to a
+value less than the minimum of the server's keepalive parameter (and
+any of the intermediate proxies), plus the export timeout.
+
+```yaml
+exporters:
+  otelarrow:
+    timeout: 30s
+    arrow:
+      max_stream_lifetime: 9m30s
+    endpoint: ...
+    tls: ...
+```
+
+When this is configured, the stream will terminate cleanly without
+causing retries, with `OK` gRPC status.
+
+The corresponding `otelarrowreceiver` keepalive setting, that is
+compatible with the one above, reads:
+
+<!-- TODO add a link to the (../../receiver/otelarrowreceiver/README.md) section
+discussing this topic from the receiver perspective after both READMEs are present
+in collector-contrib -->
+
+```
+receivers:
+  otelarrow:
+    protocols:
+      grpc:
+        keepalive:
+          server_parameters:
+            max_connection_age: 1m
+            max_connection_age_grace: 10m
+```
+
+### Exporter metrics
+
+In addition to the the standard
+[exporterhelper](https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/exporterhelper/README.md)
+and
+[obsreport](https://pkg.go.dev/go.opentelemetry.io/collector/obsreport)
+metrics, this component provides network-level measurement instruments
+which we anticipate will become part of `exporterhelper` and/or
+`obsreport` in the future.  At the `normal` level of metrics detail:
+
+- `exporter_sent`: uncompressed bytes sent, prior to compression
+- `exporter_sent_wire`: compressed bytes sent, on the wire.
+
+Arrow's compression performance can be derived by dividing the average
+`exporter_sent` value by the average `exporter_sent_wire` value.
+
+At the `detailed` metrics detail level, information about the stream
+of data being returned to the exporter will be instrumented:
+
+- `exporter_recv`: uncompressed bytes received, prior to compression
+- `exporter_recv_wire`: compressed bytes received, on the wire.
+
+### Compression Configuration
+
+The exporter supports configuring Zstd compression at both the gRPC
+and the Arrow level.  The exporter metrics described above will be
+correct in either case.  The default settings are subject to change as
+we gain experience.
+
+See the Collector [compression
+comparison](https://github.com/open-telemetry/opentelemetry-collector/blob/main/config/configgrpc/README.md#compression-comparison)
+for general information about the choice of Zstd by default, for other
+general compression configuration and benchmark information.
+
+For the OpenTelemetry Protocol with Apache Arrow streams specifically,
+gRPC-level the Zstd compression level can be configured.  However, there
+is an important caveat: the gRPC-Go library requires that compressor
+implementations be registered statically.  These libraries use
+compressors named `zstdarrow1`, `zstdarrow2`, ..., `zstdarrow10`,
+supporting 10 configurable compression levels.  Note, however that
+these configurations are static and only one unique configuration is
+possible per level.  It is possible to configure multiple OTel-Arrow
+exporters with different Zstd configuration simply by using distinct
+levels.
+
+Under `arrow`, the `zstd` sub-configuration has the following fields:
+
+- `level`: in the range 1-10 determines a number of defaults (default 5)
+- `window_size_mib`: size of the Zstd window in MiB, 0 indicates to determine based on level (default 0)
+- `concurrency`: controls background CPU used for compression, 0 indicates to let `zstd` library decide (default 1)
+
+The exporter supports configuring compression at the [Arrow
+columnar-protocol
+level](https://arrow.apache.org/docs/format/Columnar.html#format-ipc).
+
+- `payload_compression`: compression applied at the Arrow IPC level, "none" by default, "zstd" supported.
+
+      payload_compression: zstd  # describes Arrow-IPC compression (default "none")
+
+Compression settings at the Arrow IPC level cannot be further
+configured.  We do not recommend configuring both payload and
+gRPC-level compression at once, hwoever these settings are
+independent.
+
+For example, two exporters may be configured with multiple zstd
+configurations, provided they use different levels:
+
+```yaml
+exporters:
+  otelarrow/best:
+    compression: zstd  # describes gRPC-level compression (default "zstd")
+    arrow:
+      zstd:
+        level: 10      # describes gRPC-level compression level (default 5)
+  otelarrow/fastest:
+    compression: zstd
+    arrow:
+      zstd:
+        level: 1       # 1 is the "fastest" compression level
+```