Build integration for Logstash metrics via OTLP

[estolfo]

Build an integration for the metrics exported from Logstash via OTLP to Elasticsearch.

Related to #19021

[estolfo]

@perk Do you think the integration should be a content type package (just kibana dashboards, no Elastic Agent involvement) or an input type package with otelcol input (the EDOT collector embedded in Elastic Agent as an OTLP receiver)

Here are some details about each approach:

1. content type package

• Just Kibana dashboards and saved objects, no Elastic Agent config
• Logstash pushes OTLP directly to mOTel
• The package uses discovery.datasets to associate dashboards with the arriving data
• This is the model used by otel_collector_internal_telemetry — an OTel-instrumented service that pushes its own metrics directly to ES

2. input type package with otelcol input

• Configures EDOT embedded in Elastic Agent as an OTLP receiver
• Logstash pushes to the EDOT collector, which forwards to ES
• Only justified if you need Fleet-managed configuration or multi-destination routing

[perk]

Let's start with the content type package for now. Thank you!

[estolfo]

Note the limitations of building a dashboard with OTel metrics:

Pipelines Overview

Pipelines table

Batch byte size is a flow metric, can’t display that column

Worker Utilization

uses logstash.pipeline.total.flow.worker_utilization.last_1_minute, a flow rate metric

Batch event count

uses logstash.pipeline.total.batch.event_count.current, a flow rate metric

Batch size

uses logstash.pipeline.total.batch.byte_size.current, a flow rate metric