OpenTelemetry plugin not sending logs to collector

[DMaxter]

Is there an existing issue for this?

• I have searched the existing issues

Kong version ($ kong version)

3.9.0

Current Behavior

When setting the OpenTelemetry plugin and setting the logs endpoint, it sends a request to the collector (sometimes 2), but after that nothing else is sent (even after waiting some minutes).

I noticed that if the log level is debug the requests are then sent.

I compiled kong from source and played around a bit, and the behavior is super strange:

If I add

ngx.log(ngx.ERR, "TESTING OTEL")

in either of these functions

kong/kong/plugins/opentelemetry/logs.lua

Line 22 in 0118b3c

local function http_export_logs(conf, logs_batch)

kong/kong/plugins/opentelemetry/utils.lua

Line 16 in 0118b3c

local function http_export_request(conf, pb_data, headers)

in the path of success, I get both my log message and the request on OpenTelemetry collector.

To check that this was indeed a problem on Kong side, I even checked with tcpdump and there seems to be no traffic at all. Connection is established and around 1min later we have the connection being closed and nothing more.

This seems to be either some buffer that needs to be flushed or the worker thread that is supposed to make the request not running. If I used ngx.DEBUG instead of ngx.ERR, I would have the same behavior as if the statement wasn't there, because default log level is higher (info)

Expected Behavior

Logs sent to OpenTelemetry collector after a timeout (which should be some seconds) or right after making them if timeout is too low.

Steps To Reproduce

1. Deploy an OpenTelemetry Collector with the following configuration

receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:

exporters:
  debug/basic:
    verbosity: basic

service:
  pipelines:
    logs:
      receivers:
        - otlp
      processors:
        - batch
      exporters:
        - debug/basic

  telemetry:
    logs:
      level: info

2. Deploy kong with a minimal configuration

_format_version: "3.0"
plugins:
- config:
    connect_timeout: 1000
    header_type: ot
    logs_endpoint: http://localhost:4318/v1/logs
    propagation:
      clear: null
      default_format: ot
      extract: null
      inject:
      - preserve
    queue:
      concurrency_limit: -1
      initial_retry_delay: 0.01
      max_batch_size: 1
      max_bytes: null
      max_coalescing_delay: 1
      max_entries: 10000
      max_retry_delay: 30
      max_retry_time: 30
    read_timeout: 60000
    sampling_rate: 0
    send_timeout: 60000
  enabled: true
  name: opentelemetry
  protocols:
  - grpc
  - grpcs
  - http
  - https
routes:
- https_redirect_status_code: 426
  name: otel
  path_handling: v0
  paths:
  - /
  plugins:
  - config:
      anonymous: null
      hide_credentials: false
      key_in_body: false
      key_in_header: true
      key_in_query: true
      key_names:
      - apikey
      realm: null
      run_on_preflight: true
    enabled: true
    name: key-auth
    protocols:
    - grpc
    - grpcs
    - http
    - https
  preserve_host: false
  protocols:
  - http
  - https
  regex_priority: 0
  request_buffering: true
  response_buffering: true
  strip_path: true

3. Perform some requests to the gateway
4. Should only see 2 lines similar to

2024-12-31T15:15:50.346Z        info    Logs    {"kind": "exporter", "data_type": "logs", "name": "debug/basic", "resource logs": 1, "log records": 1}
2024-12-31T15:15:59.377Z        info    Logs    {"kind": "exporter", "data_type": "logs", "name": "debug/basic", "resource logs": 1, "log records": 1}

Anything else?

No response

[samugi]

Hi @DMaxter, thank you for opening this issue.

Is any log produced in your Kong error_log output, which you are expecting to be reported? I tested the configuration that you provided, and with that, no log is produced for incoming requests, in which case the behavior of no logs being reported would be expected, as described in our docs:

Logs are reported based on the log level that is configured for Kong Gateway. If a log is emitted with a level that is lower than the configured log level, it is not exported.

Note: Not all logs are guaranteed to be exported. Logs that are not exported include those produced by the Nginx master process and low-level errors produced by Nginx. Operators are expected to capture the Nginx error.log file in addition to using this feature for observability purposes.

If understood correctly, the moment you added a log with a level that is compatible to your logging configuration, i.e. higher than INFO: ngx.log(ngx.ERR, "TESTING OTEL"), you started receiving logs on your collector as expected.

The two log entries you mention in your description are likely the following:

worker #0 is ready to accept events from unix:/usr/local/kong/sockets/we
worker #1 is ready to accept events from unix:/usr/local/kong/sockets/we

Which are expected to only be outputed once for each worker.

To reiterate, you should only expect logs that exist in the error_log to be reported in OTel format via the configured logs_endpoint.
Let me know in case you think I missed anything or if you have any doubts.