Add support for structured metadata in logs #310
Conversation
- Introduced `propertiesAsStructuredMetadata` to extract high-cardinality data as structured metadata. - Enhanced `LokiBatchFormatter` and Loki serialization model to handle structured metadata. - Updated documentation and examples for structured metadata integration. - Adjusted tests to verify proper metadata serialization behavior. Signed-off-by: Andy Blyler <[email protected]>
There was a problem hiding this comment.
Pull request overview
This PR introduces support for Loki's structured metadata feature, allowing high-cardinality data like trace IDs and user IDs to be attached to log entries without being indexed as labels. The implementation extracts specified properties as structured metadata and handles their serialization according to the Loki API format.
Key changes:
- Added
propertiesAsStructuredMetadataconfiguration parameter to specify which properties should be extracted as structured metadata - Modified the Loki batch format to support a third element in log entry arrays (the structured metadata dictionary)
- Added
leaveStructuredMetadataPropertiesIntactoption to control whether extracted properties remain in the log JSON
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| test/Serilog.Sinks.Grafana.Loki.Tests/IntegrationTests/LokiJsonTextFormatterRequestPayloadTests.cs | Added integration tests for structured metadata serialization with and without the leavePropertiesIntact option |
| src/Serilog.Sinks.Grafana.Loki/Models/LokiStream.cs | Changed Entries type from IList<IList<string>> to IList<IList<object>> to accommodate structured metadata dictionary as third array element; added overload for AddEntry with structured metadata parameter |
| src/Serilog.Sinks.Grafana.Loki/Models/LokiSerializationContext.cs | Added JSON serialization support for Dictionary<string, string>, object[], and string types needed for structured metadata |
| src/Serilog.Sinks.Grafana.Loki/LokiBatchFormatter.cs | Implemented structured metadata extraction logic that removes quotes from property values and optionally removes properties from log events; added constructor parameters for structured metadata configuration |
| src/Serilog.Sinks.Grafana.Loki/LoggerConfigurationLokiExtensions.cs | Added propertiesAsStructuredMetadata and leaveStructuredMetadataPropertiesIntact parameters to public API with XML documentation |
| sample/Serilog.Sinks.Grafana.Loki.SampleWebApp/appsettings.json | Added example configuration showing RequestId and RequestPath as structured metadata properties |
| README.md | Added comprehensive documentation section explaining structured metadata, version requirements, usage examples, and configuration options |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| You can configure which log event properties should be extracted as structured metadata using the `propertiesAsStructuredMetadata` parameter: | ||
|
|
||
| ```csharp | ||
| Log.Logger = new LoggerConfiguration() | ||
| .WriteTo.GrafanaLoki( | ||
| "http://localhost:3100", | ||
| propertiesAsStructuredMetadata: new[] { "trace_id", "user_id", "request_id" }) | ||
| .CreateLogger(); | ||
|
|
||
| Log.Information("User {user_id} performed action with trace {trace_id}", "user123", "abc-xyz-123"); | ||
| ``` | ||
|
|
||
| This produces a log entry with structured metadata attached: | ||
| ```json | ||
| { | ||
| "streams": [{ | ||
| "stream": {}, | ||
| "values": [ | ||
| ["1234567890000000000", "{\"Message\":\"...\",\"level\":\"info\"}", {"trace_id":"abc-xyz-123","user_id":"user123"}] | ||
| ] | ||
| }] | ||
| } | ||
| ``` | ||
|
|
||
| By default, properties extracted as structured metadata are removed from the log line JSON to avoid duplication. To keep them in both places, set `leaveStructuredMetadataPropertiesIntact` to `true`: | ||
|
|
||
| ```csharp | ||
| Log.Logger = new LoggerConfiguration() | ||
| .WriteTo.GrafanaLoki( | ||
| "http://localhost:3100", | ||
| propertiesAsStructuredMetadata: new[] { "trace_id", "user_id" }, | ||
| leaveStructuredMetadataPropertiesIntact: true) | ||
| .CreateLogger(); | ||
| ``` |
There was a problem hiding this comment.
The documentation doesn't clarify what happens when a property is configured in both propertiesAsLabels and propertiesAsStructuredMetadata. Based on the implementation, such a property would appear in both places, which might lead to data duplication and unexpected behavior. The documentation should clarify this edge case and either recommend against it or explicitly document the behavior.
| foreach (var property in propertiesToExtract) | ||
| { | ||
| // Remove quotes from the value string representation | ||
| var value = property.Value.ToString().Replace("\"", string.Empty); |
There was a problem hiding this comment.
The value extraction using ToString().Replace(""", string.Empty) is overly simplistic and can produce incorrect results for complex property values. This approach will strip quotes from anywhere in the string, not just the surrounding quotes. For example, a value like "He said "hello"" would become "He said hello", losing the internal quotes that are part of the actual data. Consider using a more robust approach that only removes the outer quotes added by Serilog's rendering, or use a proper serialization method that handles various value types correctly.
| // Extract structured metadata | ||
| Dictionary<string, string>? structuredMetadata = null; | ||
| if (_propertiesAsStructuredMetadata.Any()) | ||
| { | ||
| structuredMetadata = new Dictionary<string, string>(); | ||
| var propertiesToExtract = logEvent.Properties | ||
| .Where(kvp => _propertiesAsStructuredMetadata.Contains(kvp.Key)) | ||
| .ToList(); | ||
|
|
||
| foreach (var property in propertiesToExtract) | ||
| { | ||
| // Remove quotes from the value string representation | ||
| var value = property.Value.ToString().Replace("\"", string.Empty); | ||
| structuredMetadata[property.Key] = value; | ||
|
|
||
| // Remove the property from the log event if configured to do so | ||
| if (!_leaveStructuredMetadataPropertiesIntact) | ||
| { | ||
| logEvent.RemovePropertyIfPresent(property.Key); | ||
| } | ||
| } | ||
| } |
There was a problem hiding this comment.
The structured metadata extraction happens after label extraction in the GenerateLabels method, but operates on the original LokiLogEvent's LogEvent. If a property is specified in both propertiesAsLabels and propertiesAsStructuredMetadata, the property will be in both places (labels and structured metadata), which may not be the intended behavior. Consider adding validation to prevent properties from being in both lists, or document the expected behavior when a property appears in both configurations.
| structuredMetadata = new Dictionary<string, string>(); | ||
| var propertiesToExtract = logEvent.Properties | ||
| .Where(kvp => _propertiesAsStructuredMetadata.Contains(kvp.Key)) | ||
| .ToList(); | ||
|
|
||
| foreach (var property in propertiesToExtract) | ||
| { | ||
| // Remove quotes from the value string representation | ||
| var value = property.Value.ToString().Replace("\"", string.Empty); | ||
| structuredMetadata[property.Key] = value; | ||
|
|
||
| // Remove the property from the log event if configured to do so | ||
| if (!_leaveStructuredMetadataPropertiesIntact) | ||
| { | ||
| logEvent.RemovePropertyIfPresent(property.Key); |
There was a problem hiding this comment.
An empty Dictionary is created even when no properties match the configured propertiesAsStructuredMetadata. This results in unnecessary object allocation that will be discarded. Consider checking if any properties match before creating the Dictionary, or use a null check pattern to avoid the allocation when propertiesToExtract is empty.
| structuredMetadata = new Dictionary<string, string>(); | |
| var propertiesToExtract = logEvent.Properties | |
| .Where(kvp => _propertiesAsStructuredMetadata.Contains(kvp.Key)) | |
| .ToList(); | |
| foreach (var property in propertiesToExtract) | |
| { | |
| // Remove quotes from the value string representation | |
| var value = property.Value.ToString().Replace("\"", string.Empty); | |
| structuredMetadata[property.Key] = value; | |
| // Remove the property from the log event if configured to do so | |
| if (!_leaveStructuredMetadataPropertiesIntact) | |
| { | |
| logEvent.RemovePropertyIfPresent(property.Key); | |
| var propertiesToExtract = logEvent.Properties | |
| .Where(kvp => _propertiesAsStructuredMetadata.Contains(kvp.Key)) | |
| .ToList(); | |
| if (propertiesToExtract.Count > 0) | |
| { | |
| structuredMetadata = new Dictionary<string, string>(); | |
| foreach (var property in propertiesToExtract) | |
| { | |
| // Remove quotes from the value string representation | |
| var value = property.Value.ToString().Replace("\"", string.Empty); | |
| structuredMetadata[property.Key] = value; | |
| // Remove the property from the log event if configured to do so | |
| if (!_leaveStructuredMetadataPropertiesIntact) | |
| { | |
| logEvent.RemovePropertyIfPresent(property.Key); | |
| } |
| Loki supports [structured metadata](https://grafana.com/docs/loki/latest/reference/loki-http-api/#ingest-logs) - additional key-value pairs that can be attached to each log line without being indexed as labels. This is useful for high-cardinality data like trace IDs, user IDs, or request IDs that you want to query but don't want to use as labels. | ||
|
|
||
| > **Requirements:** | ||
| > - Structured metadata requires **Loki 1.2.0 or newer** |
There was a problem hiding this comment.
The documentation states "Loki 1.2.0 or newer" but this appears to be incorrect. Structured metadata support was actually introduced in Loki 2.9.0 (released in 2023). The documentation should be updated to reflect the correct minimum version requirement to avoid confusion for users who may be running older versions of Loki.
| > - Structured metadata requires **Loki 1.2.0 or newer** | |
| > - Structured metadata requires **Loki 2.9.0 or newer** |
| if (structuredMetadata != null && structuredMetadata.Count > 0) | ||
| { | ||
| stream.AddEntry(timestamp, entry, structuredMetadata); | ||
| } | ||
| else | ||
| { | ||
| stream.AddEntry(timestamp, entry); | ||
| } |
There was a problem hiding this comment.
The condition checks if structuredMetadata is null or has Count == 0, but structuredMetadata is initialized as null and only set to a non-null value inside the if block at line 196. If _propertiesAsStructuredMetadata.Any() is false, structuredMetadata remains null. The null check at line 221 is correct, but you could simplify the logic by just checking for null since an empty dictionary should not be created in the first place (see the performance comment on lines 198-201).
1 participant
propertiesAsStructuredMetadatato extract high-cardinality data as structured metadata.LokiBatchFormatterand Loki serialization model to handle structured metadata.