feature: opentelemetry logs exporter for all logs, not only monitor

[MattiasDC]

Feature request

I would like it if it was possible to redirect all logs easily to an opentelemetry collector. Currently I only see monitoring related logs in my collector. I'm using these opentelemetry environment variables:

• OTEL_LOGS_EXPORTER
• OTEL_PYTHON_LOG_LEVEL
• OTEL_EXPORTER_OTLP_ENDPOINT
• OTEL_EXPORTER_OTLP_PROTOCOL
• OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED
• OTEL_PYTHON_LOG_CORRELATION

I run my bentoml command prepended with opentelemetry-instrument, but it seems that bentoml starts subprocesses which are not automatically instrumented.

Motivation

This feature would make it easy to set up opentelemetry log collector for all logs outputted by bentoml directly with otlp

Other

The reason for this seems to come from this code: https://github.com/open-telemetry/opentelemetry-python-contrib/blob/e2ba6d43c0d9906e664da09abe8f5404e9026ec0/opentelemetry-instrumentation/src/opentelemetry/instrumentation/auto_instrumentation/__init__.py#L123

[aarnphm]

See https://docs.bentoml.com/en/latest/build-with-bentoml/observability/tracing.html#id3

[MattiasDC]

But isn't that only for the traces themselves? E.g. in this example there is both a traces exporter and a logs exporter

export OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED=true
opentelemetry-instrument \
  --traces_exporter console,otlp \
  --metrics_exporter console,otlp \
  --logs_exporter console,otlp \
  --service_name python-logs-example \
  python $(pwd)/example.py

[MattiasDC]

For example, I ran into an issue with a deployed model and found a trace with a 500 (internal server error) response. I then wanted to see the callstack of the internal server error. For this I had to check the container logs, as the logs themselves are not available in ELK, as they are not being send to the collector. Only the logs of the bentoml_monitor_data logger are being sent to opentelemetry.

BentoML/src/bentoml/_internal/monitoring/otlp.py

Lines 202 to 220 in 1b09445

	self.logger_provider.add_log_record_processor(BatchLogRecordProcessor(exporter))
	handler = LoggingHandler(
	level=logging.NOTSET, logger_provider=self.logger_provider
	)
	logging.config.dictConfig(
	{
	"version": 1,
	"disable_existing_loggers": False,
	"loggers": {
	"bentoml_monitor_data": {
	"level": "INFO",
	"handlers": [],
	"propagate": False,
	}
	},
	}
	)
	self.data_logger = logging.getLogger("bentoml_monitor_data")
	self.data_logger.addHandler(handler)

There is no way to configure this for the root logger and since BentoML uses Watcher, autoinstrumentation is to my understanding not possible