[connector/spanmetrics] OpenTelemetry Collector gateway deployment mode cannot parse agent metrics data.

[ixiaoyi93]

Component(s)

connector/spanmetrics

Describe the issue you're reporting

I used opentelmetry collector operator for opentelmetry collector deployment in a kubernetes cluster, and kubernetes daemonset for opentelmetry collector agent mode deployment, and then the The traces/metrics are forwarded to the collector cluster in gateway deployment mode, which is used to unify the processing of data from virtual hosts other than the kubernetes cluster. The problem encountered is that the collector in the gateway mode cluster is unable to process the metrics data.

OpenTelmetry Collector agent mode configuration in a kubernetes cluster:

apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: test-ack-public-collector
spec:
  mode: daemonset
  hostNetwork: true
  config:
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
          http:
            endpoint: 0.0.0.0:4318
    exporters:
      otlp:
        endpoint: "http://192.168.255.155:4317"
        tls:
          insecure: true
      otlphttp:
        endpoint: "http://192.168.255.155:4318"
        tls:
          insecure: true
    service:
      pipelines:
        traces:
          receivers:
            - otlp
          exporters:
            - otlp
            - otlphttp
        metrics:
          receivers:
            - otlp
          exporters:
            - otlp
            - otlphttp
  resources:
    limits:
      cpu: 100m
      memory: 200M

192.168.255.155 is a collector gateway access address. OpenTelmetry Collector gateway mode configuration:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
  prometheus:
    config:
      scrape_configs:
      - job_name: 'otelcol'
        scrape_interval: 10s
        static_configs:
        - targets: ['0.0.0.0:8888']

processors:
  filter:
    error_mode: ignore
    traces:
      span:
        - IsMatch(attributes["http.route"], ".*/health.*")
  transform:
    metric_statements:
      - context: datapoint
        statements:
        - set(attributes["ip"], resource.attributes["host.name"])
  memory_limiter:
    check_interval: 1s
    limit_percentage: 75
    spike_limit_percentage: 15
  batch:
    send_batch_size: 10000
    timeout: 10s

connectors:
  spanmetrics:
    histogram:
      explicit:
        buckets: [100us, 1ms, 2ms, 6ms, 10ms, 100ms, 250ms]

exporters:
  debug:
    verbosity: detailed
  otlp:
    endpoint: tempo:4317
    tls:
      insecure: true
  prometheus:
    endpoint: "0.0.0.0:8889"

extensions:
  health_check:

service:
  extensions: [health_check]
  telemetry:
    logs:
      level: "debug"
    metrics:
      address: "0.0.0.0:8888"
  pipelines:
    traces:
      receivers: [otlp]
      processors: [filter, transform, memory_limiter, batch]
      exporters: [otlp, spanmetrics]
    metrics:
      receivers: [otlp, prometheus, spanmetrics]
      processors: [transform, memory_limiter, batch]
      exporters: [prometheus]

The status quo is that currently there is only trace data and the ServiceGrap. I'm not sure how the metrics metrics are relayed.

[github-actions]

Pinging code owners:

• connector/spanmetrics: @portertech @Frapschen

See Adding Labels via Comments if you do not have permissions to add labels yourself.