Open Telemetry Data Model?

[deniskrizanovic]

https://opentelemetry.io/docs/reference/specification/logs/data-model/

Why not?

I understand we can't always participate in a scoped context, but wouldn't it be reasonable to store logs in this format so that other log aggregators can suck up the logs in Salesforce.

This thinking was inspired by this article -> https://blog.salesforceairesearch.com/logai/
From Salesforce Research, where they add value to a log, by doing AI things to it.. But assume a Data Model.

I saw no other discussions with the phrase "open telemetry" in them..

regards,
denis-

[jongpie]

@deniskrizanovic I love the idea of making Nebula Logger's data available in a format compatible with OpenTelemetry! I haven't worked with OpenTelemetry much before, so I think I'll need to learn more about it - offhand, I think the main challenge is there are specific field names that should be used in the log data, such as severityNumber, timeUnixNano etc., and these can't be used as the names of custom fields (e.g., a custom SeverityNumber field in Salesforce would end up with the API name SeverityNumber__c). I could add some Apex code to convert Log__c and LogEntry__c records into OpenTelemetry-compatible JSON that's stored in Salesforce Files. That would keep the data within Salesforce, and the files could be exported/extracted into other tools that support OpenTelemetry's data model.

Let me know what you think of this approach, and thanks for the suggestion!

[deniskrizanovic]

I'd like to throw out the idea that your main data model should be the Open Telemetry one. Understanding that Salesforce is gonna do the __c thing. The cognitive load is lesser if the model is as closely aligned as possible. Of course, this bold assertion should be validated with an actual attempt. With OTEL you can even get into tools like Honeycomb..

My one issue with Files is that you are now using loads of storage from db and files.. And I'm already unhappy about how much storage log records take.

[jongpie]

@deniskrizanovic I've looked through OpenTelemetry's logs data model, and I don't think it makes sense to try to make Nebula Logger's data model conform to it directly

• Some of OpenTelemetry's concepts don't make sense in the context of a Salesforce org (especially one that's not using OpenTelemetry). For example, its SeverityNumber for a log supports ranges that correspond to different enum values (e.g., TRACE has possible values 1-4). These enum values don't match the LoggingLevel values used in Apex or in Salesforce's debug logs, and the use of ranges would add unnecessary complexity to Nebula Logger

  

• It would also increase the data storage used in Salesforce orgs - OpenTelemetry's data model stores some fields as key-value pairs, which could involve creating several additional records per LogEntry__c (if trying to mimic OpenTelemetry's data model as closely as possible). I've tried to design Nebula Logger's data model to be as simple as possible (while still providing a robust & reportable data model) to try to take Salesforce data storage into consideration - OpenTelemetry's data model doesn't share that concern.

• It would involve some breaking changes - lots of fields & expected values would need to be renamed to conform to OpenTelemetry's data model.

I think a better option it to add support for sending Nebula Logger data to an OTLP endpoint using an async job (either batchable or queueable) and some DTOs.

• This would keep Nebula Logger's existing data model as-is,
• The custom objects' data (and/or possibly LogEntryEvent__e data) would be transformed in the async job to match OpenTelemetry's logs data model to a specified (configurable) endpoint.
• This approach would also avoid using up any additional data or file storage.
• It'd also make it fairly easy to display a specific Log__c record in OpenTelemetry's format (as an extra "nice to have" feature for admins/developers). The Apex code used by the async job to transform the data to DTOs could also be re-used by the 'Log Viewer' quick action on Log__c.

[deniskrizanovic]

Hi Jonathan, Awesome analysis. And it makes perfect sense to optimise for Salesforce simplicity whilst attending to those that expect OTEL structures. When the time is appropriate it might be good to document this somewhere so that future "otel Salesforce" searches have somewhere to go. Thanks for clarifying how this should work with your logger. regards, denis-

…

On Tue, 25 Apr 2023 at 00:34, Jonathan Gillespie ***@***.***> wrote: @deniskrizanovic <https://github.com/deniskrizanovic> I've looked through OpenTelemetry's logs data model <https://opentelemetry.io/docs/reference/specification/logs/data-model/>, and I don't think it makes sense to try to make Nebula Logger's data model conform to it directly - Some of OpenTelemetry's concepts don't make sense in the context of a Salesforce org (especially one that's not using OpenTelemetry). For example, its SeverityNumber for a log supports ranges that correspond to different enum values (e.g., TRACE has possible values 1-4). These enum values don't match the LoggingLevel values used in Apex or in Salesforce's debug logs, and the use of ranges would add unnecessary complexity to Nebula Logger [image: image] <https://user-images.githubusercontent.com/1267157/234025864-3db228ab-8f53-4b22-bc95-90a662441f9c.png> - It would also increase the data storage used in Salesforce orgs - OpenTelemetry's data model stores some fields as key-value pairs, which could involve creating several additional records per LogEntry__c (if trying to mimic OpenTelemetry's data model as closely as possible). I've tried to design Nebula Logger's data model to be as simple as possible (while still providing a robust & reportable data model) to try to take Salesforce data storage into consideration - OpenTelemetry's data model doesn't share that concern. - It would involve some breaking changes - lots of fields & expected values would need to be renamed to conform to OpenTelemetry's data model. I think a better option it to add support for sending Nebula Logger data to an OTLP <https://opentelemetry.io/docs/reference/specification/protocol/otlp/> endpoint using an async job (either batchable or queueable) and some DTOs. - This would keep Nebula Logger's existing data model as-is, - The custom objects' data (and/or possibly LogEntryEvent__e data) would be transformed in the async job to match OpenTelemetry's logs data model to a specified (configurable) endpoint. - This approach would also avoid using up any additional data or file storage. - It'd also make it fairly easy to display a specific Log__c record in OpenTelemetry's format (as an extra "nice to have" feature for admins/developers). The Apex code used by the async job to transform the data to DTOs could also be re-used by the 'Log Viewer' quick action on Log__c. — Reply to this email directly, view it on GitHub <#479 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAG3SAZFOVCMAMT7Q62KGILXC2FPHANCNFSM6AAAAAAWXXQEPE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[jongpie]

Awesome, thanks for the feedback! Later this week or next week, I'll document some of this in a new issue so I can track this - I'm hoping that I'll have some time in the next few months to at least build out a proof of concept to ensure this approach will work. I'll keep you updated!