Add PostHog.AI Package for LLM Observability

[matjanos]

Summary

Adds a new PostHog.AI package to provide AI observability for .NET applications. Intercepts OpenAI API requests and responses to capture events with properties per the PostHog AI event spec.

Features

• Automatic event capture: captures $ai_generation and $ai_embedding events from OpenAI API calls
• Model parameters: extracts temperature, max_tokens, stream, tools from requests
• Token usage: captures input/output/total tokens from responses
• Streaming support: handles Server-Sent Events (SSE) with proper message parsing
• Context management: PostHogAIContext for session/span tracking and custom properties
• Cache tracking: captures cache token usage when available
• Error handling: captures errors and HTTP status codes

Implementation Details

Core Components

• PostHogOpenAIHandler: Delegating handler that intercepts HTTP requests/responses
• PostHogAIContext: Async-local context for setting session IDs, span IDs, and custom properties
• PostHogAIExtensions: Dependency injection extensions for setup

Captured Properties

The handler extracts and captures properties including:

• Core: trace_id, provider, model, latency, URLs, HTTP status
• Model parameters: temperature, max_tokens, stream, tools
• Token usage: input_tokens, output_tokens, total_tokens
• Context: session_id, span_id, span_name, parent_id (via PostHogAIContext)
• Cache: cache_read_input_tokens, cache_creation_input_tokens
• Errors: is_error, error messages

Streaming Support

• Proper SSE message parsing that handles partial messages across chunks
• Buffer management for incomplete lines
• Support for both \n and \r\n line endings (with \r\n checked first)

Usage Example

// Configure PostHog
builder.Services.AddPostHog(options => {
    options.ProjectApiKey = "YOUR_KEY";
    options.HostUrl = new Uri("https://us.i.posthog.com");
});

// Register OpenAI client with PostHog integration
builder.Services.AddPostHogOpenAIClient("YOUR_OPENAI_API_KEY");

// Use with context
using (PostHogAIContext.BeginScope(
    sessionId: "session-123",
    spanId: "span-1",
    spanName: "summarize_text"))
{
    var response = await openAIClient.GetChatCompletionsAsync(...);
}

Testing

• Unit tests for non-streaming requests
• Tests for embedding events
• Tests for streaming responses with SSE parsing
• All tests passing

Files Changed

• New package: src/PostHog.AI/PostHog.AI.csproj
• New tests: tests/PostHog.AI.Tests/PostHog.AI.Tests.csproj
• Updated solution file to include new projects
• Updated root README to document the new package
• Fixed typo in samples/HogTied.Web/Program.cs (PostHoc → PostHog)

Notes

• Cost properties ($ai_input_cost_usd, etc.) are only set when explicitly provided via PostHogAIContext.Properties
• The handler skips null values when merging context properties
• Backward compatible with existing PostHog setup

[greptile-apps]

Additional Comments (3)

1. src/PostHog.AI/PostHogOpenAIHandler.cs, line 280 (link)

   style: temperature is a numeric field but stored as string (same inconsistency as max_tokens which uses type checking). Inconsistent handling across model parameters.

2. src/PostHog.AI/PostHogOpenAIHandler.cs, line 500-501 (link)

   style: unnecessary await Task.CompletedTask at the end of an async method when there's no actual async work

3. src/PostHog.AI/PostHogOpenAIHandler.cs, line 36-62 (link)

   style: duplicated code pattern for reading content and parsing JSON (appears at lines 36-62 for request and 161-188 for response)

_{10 files reviewed, 3 comments}

_{Edit Code Review Agent Settings | Greptile}