Skip to content

Commit 6bf5507

Browse files
prathamesh-sonpatkiclaudekarthikeyangs9
authored
feat: add route exclusion for health check endpoints (#27)
* feat: add route exclusion to skip tracing health check endpoints Health check endpoints (/health, /metrics, /ready, etc.) generate noise and inflate trace volume. This adds configurable path exclusion that leverages each OTel middleware's native filter mechanism (WithFilter/ WithSkipper), so excluded paths never create spans. Three-layer matcher (exact, prefix, glob) configured via env vars: - LAST9_EXCLUDED_PATHS (default: /health,/healthz,/metrics,/ready,/live,/ping) - LAST9_EXCLUDED_PATH_PREFIXES (default: none) - LAST9_EXCLUDED_PATH_PATTERNS (default: /*/health,/*/healthz,/*/metrics,/*/ready,/*/live,/*/ping) Set any env var to "" to opt out of its defaults. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: resolve lint errors in routematcher tests Fix fieldalignment (govet) and gofmt issues: - Reorder struct fields for optimal GC pointer scanning - Use named fields in TestIsEmpty struct literals - Run gofmt for consistent formatting Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs: add route exclusion section to README Document the route exclusion feature including default excluded paths, configuration env vars, usage examples, and matching behavior. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(config): reorder Config struct fields to satisfy fieldalignment lint Move SampleRate and SamplerRatio (float64, non-pointer) after all slice fields so the GC scan range shrinks from 184 to 168 bytes. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * refactor(instrumentation): prefer var declaration over empty slice literal Replace `opts := []T{}` with `var opts []T` in echo, gin, and gorilla middleware — idiomatic Go for nil-initialized slices that may stay empty. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: karthikeyangs9 <karthikeyan@last9.io>
1 parent 32b1658 commit 6bf5507

13 files changed

Lines changed: 641 additions & 29 deletions

File tree

README.md

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@ A drop-in OpenTelemetry agent for Go applications that minimizes code changes wh
1717
- [Log-Trace Correlation (zap)](#-log-trace-correlation-zap) - zap
1818
- [Log-Trace Correlation (slog)](#-log-trace-correlation) - slog
1919
- [Metrics Support](#-metrics-support) - Automatic • Custom • Runtime
20+
- [Route Exclusion](#-route-exclusion) - Skip tracing for health checks & internal paths
2021
- [Configuration](#️-configuration)
2122
- [Requirements & Compatibility](#-requirements--compatibility)
2223
- [Testing](#-testing)
@@ -29,6 +30,7 @@ A drop-in OpenTelemetry agent for Go applications that minimizes code changes wh
2930
- 🎯 **Auto-instrumentation** - HTTP, gRPC, SQL, MongoDB, Redis, Kafka automatically traced with proper span nesting
3031
- 📊 **Automatic metrics** - Runtime (memory, GC, goroutines), HTTP, gRPC, database, MongoDB, Kafka, Redis metrics out-of-the-box
3132
- 📈 **Custom metrics** - Simple helpers for counters, histograms, gauges for business metrics
33+
- 🚫 **Route exclusion** - Automatically skips health checks and infrastructure endpoints from tracing
3234
- ⚙️ **Environment-based config** - Uses standard OpenTelemetry environment variables (no hardcoded config)
3335
- 🔗 **Log-trace correlation** - Automatic `trace_id`/`span_id` injection into `log/slog` log entries
3436
- 🔍 **Complete observability** - Full distributed tracing + metrics across all layers (HTTP → gRPC → DB → External APIs)
@@ -907,6 +909,45 @@ func processOrder(ctx context.Context, value float64) {
907909
}
908910
```
909911

912+
## 🚫 Route Exclusion
913+
914+
The agent automatically skips tracing for common health check and infrastructure endpoints, reducing noise in your traces. This works across all supported frameworks (net/http, Gin, Chi, Echo, Gorilla Mux, gRPC-Gateway).
915+
916+
### Default Excluded Routes
917+
918+
Out of the box, the following paths are excluded from tracing:
919+
920+
- **Exact paths**: `/health`, `/healthz`, `/metrics`, `/ready`, `/live`, `/ping`
921+
- **Glob patterns**: `/*/health`, `/*/healthz`, `/*/metrics`, `/*/ready`, `/*/live`, `/*/ping`
922+
923+
### Configuration
924+
925+
Three environment variables control route exclusion:
926+
927+
| Variable | Default | Description |
928+
|----------|---------|-------------|
929+
| `LAST9_EXCLUDED_PATHS` | `/health,/healthz,/metrics,/ready,/live,/ping` | Exact path matches |
930+
| `LAST9_EXCLUDED_PATH_PREFIXES` | *(none)* | Prefix matches (e.g., `/internal/`) |
931+
| `LAST9_EXCLUDED_PATH_PATTERNS` | `/*/health,/*/healthz,/*/metrics,/*/ready,/*/live,/*/ping` | Glob patterns using `path.Match` semantics |
932+
933+
### Examples
934+
935+
```bash
936+
# Add a prefix exclusion for internal debug endpoints
937+
export LAST9_EXCLUDED_PATH_PREFIXES="/internal/,/debug/"
938+
939+
# Custom exact paths to exclude
940+
export LAST9_EXCLUDED_PATHS="/health,/healthz,/status,/version"
941+
942+
# Disable all default exclusions (trace everything)
943+
export LAST9_EXCLUDED_PATHS=""
944+
export LAST9_EXCLUDED_PATH_PATTERNS=""
945+
```
946+
947+
### How It Works
948+
949+
Route matching is evaluated in order: exact match (O(1) map lookup) → prefix match → glob pattern match. The first match wins. Setting an environment variable to an empty string (`""`) disables its defaults, allowing you to opt out of default exclusions.
950+
910951
## ⚙️ Configuration
911952

912953
The agent reads configuration from environment variables following OpenTelemetry standards:
@@ -919,6 +960,9 @@ The agent reads configuration from environment variables following OpenTelemetry
919960
| `OTEL_SERVICE_VERSION` | No | - | Service version (e.g., git commit SHA) |
920961
| `OTEL_RESOURCE_ATTRIBUTES` | No | - | Additional attributes (key=value pairs) |
921962
| `OTEL_TRACES_SAMPLER` | No | `always_on` | Sampling strategy |
963+
| `LAST9_EXCLUDED_PATHS` | No | `/health,/healthz,...` | Exact paths to exclude from tracing |
964+
| `LAST9_EXCLUDED_PATH_PREFIXES` | No | - | Path prefixes to exclude from tracing |
965+
| `LAST9_EXCLUDED_PATH_PATTERNS` | No | `/*/health,/*/healthz,...` | Glob patterns to exclude from tracing |
922966

923967
### Resource Attributes
924968

agent.go

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,7 @@ import (
2323
"time"
2424

2525
"github.com/last9/go-agent/config"
26+
"github.com/last9/go-agent/internal/routematcher"
2627
"go.opentelemetry.io/otel"
2728
"go.opentelemetry.io/otel/attribute"
2829
"go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc"
@@ -109,6 +110,7 @@ func WithSamplingRate(rate float64) Option {
109110
// Agent represents the Last9 telemetry agent
110111
type Agent struct {
111112
config *config.Config
113+
routeMatcher *routematcher.RouteMatcher
112114
tracerProvider *sdktrace.TracerProvider
113115
meterProvider *metric.MeterProvider
114116
shutdown func(context.Context) error
@@ -185,8 +187,11 @@ func Start(opts ...Option) error {
185187
log.Printf("[Last9 Agent] Warning: Failed to start runtime metrics: %v", runtimeErr)
186188
}
187189

190+
rm := routematcher.New(cfg.ExcludedPaths, cfg.ExcludedPathPrefixes, cfg.ExcludedPathPatterns)
191+
188192
globalAgent.Store(&Agent{
189193
config: cfg,
194+
routeMatcher: rm,
190195
tracerProvider: tp,
191196
meterProvider: mp,
192197
shutdown: func(ctx context.Context) error {
@@ -245,6 +250,15 @@ func GetConfig() *config.Config {
245250
return globalAgent.Load().config
246251
}
247252

253+
// GetRouteMatcher returns the route matcher for path exclusion (or nil if not initialized).
254+
// Nil is safe to use — RouteMatcher.ShouldExclude on nil returns false (no exclusion).
255+
func GetRouteMatcher() *routematcher.RouteMatcher {
256+
if a := globalAgent.Load(); a != nil {
257+
return a.routeMatcher
258+
}
259+
return nil
260+
}
261+
248262
// createResource creates an OpenTelemetry resource with service information
249263
func createResource(cfg *config.Config) (*resource.Resource, error) {
250264
// Build base attributes

agent_test.go

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -363,6 +363,52 @@ func TestSampleRateUnsetFallsBackToSampler(t *testing.T) {
363363
}
364364
}
365365

366+
func TestGetRouteMatcher(t *testing.T) {
367+
defer Reset()
368+
369+
// Before Start, returns nil
370+
rm := GetRouteMatcher()
371+
if rm != nil {
372+
t.Error("GetRouteMatcher() should return nil before Start()")
373+
}
374+
// Nil receiver is safe — should not exclude anything
375+
if rm.ShouldExclude("/health") {
376+
t.Error("nil RouteMatcher should not exclude /health")
377+
}
378+
379+
os.Setenv("OTEL_SERVICE_NAME", "test-service")
380+
defer os.Unsetenv("OTEL_SERVICE_NAME")
381+
// Use defaults (health endpoints excluded)
382+
os.Unsetenv("LAST9_EXCLUDED_PATHS")
383+
os.Unsetenv("LAST9_EXCLUDED_PATH_PREFIXES")
384+
os.Unsetenv("LAST9_EXCLUDED_PATH_PATTERNS")
385+
386+
if err := Start(); err != nil {
387+
t.Fatalf("Start() failed: %v", err)
388+
}
389+
390+
rm = GetRouteMatcher()
391+
if rm == nil {
392+
t.Fatal("GetRouteMatcher() should return non-nil after Start()")
393+
}
394+
if rm.IsEmpty() {
395+
t.Error("RouteMatcher should not be empty with default exclusion rules")
396+
}
397+
398+
// Default exact excludes /health
399+
if !rm.ShouldExclude("/health") {
400+
t.Error("default RouteMatcher should exclude /health")
401+
}
402+
// Default pattern excludes /v1/health
403+
if !rm.ShouldExclude("/v1/health") {
404+
t.Error("default RouteMatcher should exclude /v1/health via glob pattern")
405+
}
406+
// Should not exclude normal paths
407+
if rm.ShouldExclude("/api/users") {
408+
t.Error("default RouteMatcher should not exclude /api/users")
409+
}
410+
}
411+
366412
func TestParseSamplerRatio(t *testing.T) {
367413
tests := []struct {
368414
input string

config/config.go

Lines changed: 53 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,23 @@ type Config struct {
1919
Endpoint string
2020
Sampler string
2121
ResourceAttributes []attribute.KeyValue
22-
SampleRate float64
22+
23+
// ExcludedPaths is a list of exact URL paths to exclude from tracing (from LAST9_EXCLUDED_PATHS).
24+
// Default: /health,/healthz,/metrics,/ready,/live,/ping
25+
// Set LAST9_EXCLUDED_PATHS="" to disable defaults.
26+
ExcludedPaths []string
27+
28+
// ExcludedPathPrefixes is a list of URL path prefixes to exclude from tracing (from LAST9_EXCLUDED_PATH_PREFIXES).
29+
// Default: none
30+
ExcludedPathPrefixes []string
31+
32+
// ExcludedPathPatterns is a list of glob patterns to exclude from tracing (from LAST9_EXCLUDED_PATH_PATTERNS).
33+
// Uses path.Match semantics (not filepath.Match).
34+
// Default: /*/health,/*/healthz,/*/metrics,/*/ready,/*/live,/*/ping
35+
// Set LAST9_EXCLUDED_PATH_PATTERNS="" to disable defaults.
36+
ExcludedPathPatterns []string
37+
38+
SampleRate float64
2339
// SamplerRatio is the sampling ratio for traceidratio samplers (0.0-1.0).
2440
// Only used when Sampler is "traceidratio" or "parentbased_traceidratio".
2541
// Set via WithSamplingRate() option. Zero value means use OTEL_TRACES_SAMPLER_ARG env var.
@@ -47,6 +63,20 @@ func Load() *Config {
4763
cfg.ServiceVersion,
4864
)
4965

66+
// Parse route exclusion configuration
67+
cfg.ExcludedPaths = parseCommaSeparatedWithDefault(
68+
"LAST9_EXCLUDED_PATHS",
69+
"/health,/healthz,/metrics,/ready,/live,/ping",
70+
)
71+
cfg.ExcludedPathPrefixes = parseCommaSeparatedWithDefault(
72+
"LAST9_EXCLUDED_PATH_PREFIXES",
73+
"",
74+
)
75+
cfg.ExcludedPathPatterns = parseCommaSeparatedWithDefault(
76+
"LAST9_EXCLUDED_PATH_PATTERNS",
77+
"/*/health,/*/healthz,/*/metrics,/*/ready,/*/live,/*/ping",
78+
)
79+
5080
// Validate configuration
5181
if cfg.Endpoint == "" {
5282
log.Println("[Last9 Agent] Warning: OTEL_EXPORTER_OTLP_ENDPOINT not set - telemetry will not be exported")
@@ -110,6 +140,28 @@ func parseResourceAttributes(attrsStr string, serviceVersion string) ([]attribut
110140
return attrs, environment, serviceVersion
111141
}
112142

143+
// parseCommaSeparatedWithDefault reads an env var and splits it by comma.
144+
// If the env var is not set, defaultVal is used. If the env var is explicitly
145+
// set to "", an empty slice is returned (opt-out of defaults).
146+
func parseCommaSeparatedWithDefault(envKey, defaultVal string) []string {
147+
raw, ok := os.LookupEnv(envKey)
148+
if !ok {
149+
raw = defaultVal
150+
}
151+
if raw == "" {
152+
return nil
153+
}
154+
155+
parts := strings.Split(raw, ",")
156+
result := make([]string, 0, len(parts))
157+
for _, p := range parts {
158+
if v := strings.TrimSpace(p); v != "" {
159+
result = append(result, v)
160+
}
161+
}
162+
return result
163+
}
164+
113165
// getEnvOrDefault returns the environment variable value or a default
114166
func getEnvOrDefault(key, defaultValue string) string {
115167
if value := os.Getenv(key); value != "" {

config/config_test.go

Lines changed: 132 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@ package config
22

33
import (
44
"os"
5+
"reflect"
56
"testing"
67
)
78

@@ -70,3 +71,134 @@ func TestLoad_SampleRate(t *testing.T) {
7071
}
7172
})
7273
}
74+
75+
func TestParseCommaSeparatedWithDefault(t *testing.T) {
76+
tests := []struct {
77+
name string
78+
envKey string
79+
envVal *string // nil = not set
80+
defaultVal string
81+
want []string
82+
}{
83+
{
84+
name: "env not set uses default",
85+
envKey: "TEST_PARSE_CSV_1",
86+
envVal: nil,
87+
defaultVal: "/health,/metrics",
88+
want: []string{"/health", "/metrics"},
89+
},
90+
{
91+
name: "env set to empty opts out",
92+
envKey: "TEST_PARSE_CSV_2",
93+
envVal: strPtr(""),
94+
defaultVal: "/health,/metrics",
95+
want: nil,
96+
},
97+
{
98+
name: "env set to custom value",
99+
envKey: "TEST_PARSE_CSV_3",
100+
envVal: strPtr("/custom,/paths"),
101+
defaultVal: "/health,/metrics",
102+
want: []string{"/custom", "/paths"},
103+
},
104+
{
105+
name: "whitespace trimmed",
106+
envKey: "TEST_PARSE_CSV_4",
107+
envVal: strPtr(" /health , /metrics "),
108+
defaultVal: "",
109+
want: []string{"/health", "/metrics"},
110+
},
111+
{
112+
name: "empty default with env not set",
113+
envKey: "TEST_PARSE_CSV_5",
114+
envVal: nil,
115+
defaultVal: "",
116+
want: nil,
117+
},
118+
{
119+
name: "trailing comma ignored",
120+
envKey: "TEST_PARSE_CSV_6",
121+
envVal: strPtr("/health,,/metrics,"),
122+
defaultVal: "",
123+
want: []string{"/health", "/metrics"},
124+
},
125+
}
126+
127+
for _, tt := range tests {
128+
t.Run(tt.name, func(t *testing.T) {
129+
os.Unsetenv(tt.envKey)
130+
if tt.envVal != nil {
131+
os.Setenv(tt.envKey, *tt.envVal)
132+
defer os.Unsetenv(tt.envKey)
133+
}
134+
135+
got := parseCommaSeparatedWithDefault(tt.envKey, tt.defaultVal)
136+
if !reflect.DeepEqual(got, tt.want) {
137+
t.Errorf("parseCommaSeparatedWithDefault(%q) = %v, want %v", tt.envKey, got, tt.want)
138+
}
139+
})
140+
}
141+
}
142+
143+
func TestLoadExcludedPathsDefaults(t *testing.T) {
144+
// Ensure env vars are not set so defaults apply
145+
os.Unsetenv("LAST9_EXCLUDED_PATHS")
146+
os.Unsetenv("LAST9_EXCLUDED_PATH_PREFIXES")
147+
os.Unsetenv("LAST9_EXCLUDED_PATH_PATTERNS")
148+
149+
cfg := Load()
150+
151+
expectedPaths := []string{"/health", "/healthz", "/metrics", "/ready", "/live", "/ping"}
152+
if !reflect.DeepEqual(cfg.ExcludedPaths, expectedPaths) {
153+
t.Errorf("ExcludedPaths = %v, want %v", cfg.ExcludedPaths, expectedPaths)
154+
}
155+
156+
if cfg.ExcludedPathPrefixes != nil {
157+
t.Errorf("ExcludedPathPrefixes = %v, want nil", cfg.ExcludedPathPrefixes)
158+
}
159+
160+
expectedPatterns := []string{"/*/health", "/*/healthz", "/*/metrics", "/*/ready", "/*/live", "/*/ping"}
161+
if !reflect.DeepEqual(cfg.ExcludedPathPatterns, expectedPatterns) {
162+
t.Errorf("ExcludedPathPatterns = %v, want %v", cfg.ExcludedPathPatterns, expectedPatterns)
163+
}
164+
}
165+
166+
func TestLoadExcludedPathsOptOut(t *testing.T) {
167+
// Set env vars to empty to opt out
168+
os.Setenv("LAST9_EXCLUDED_PATHS", "")
169+
os.Setenv("LAST9_EXCLUDED_PATH_PATTERNS", "")
170+
defer os.Unsetenv("LAST9_EXCLUDED_PATHS")
171+
defer os.Unsetenv("LAST9_EXCLUDED_PATH_PATTERNS")
172+
173+
cfg := Load()
174+
175+
if cfg.ExcludedPaths != nil {
176+
t.Errorf("ExcludedPaths = %v, want nil (opted out)", cfg.ExcludedPaths)
177+
}
178+
if cfg.ExcludedPathPatterns != nil {
179+
t.Errorf("ExcludedPathPatterns = %v, want nil (opted out)", cfg.ExcludedPathPatterns)
180+
}
181+
}
182+
183+
func TestLoadExcludedPathsCustom(t *testing.T) {
184+
os.Setenv("LAST9_EXCLUDED_PATHS", "/custom-health")
185+
os.Setenv("LAST9_EXCLUDED_PATH_PREFIXES", "/internal/")
186+
os.Setenv("LAST9_EXCLUDED_PATH_PATTERNS", "/v*/status")
187+
defer os.Unsetenv("LAST9_EXCLUDED_PATHS")
188+
defer os.Unsetenv("LAST9_EXCLUDED_PATH_PREFIXES")
189+
defer os.Unsetenv("LAST9_EXCLUDED_PATH_PATTERNS")
190+
191+
cfg := Load()
192+
193+
if !reflect.DeepEqual(cfg.ExcludedPaths, []string{"/custom-health"}) {
194+
t.Errorf("ExcludedPaths = %v, want [/custom-health]", cfg.ExcludedPaths)
195+
}
196+
if !reflect.DeepEqual(cfg.ExcludedPathPrefixes, []string{"/internal/"}) {
197+
t.Errorf("ExcludedPathPrefixes = %v, want [/internal/]", cfg.ExcludedPathPrefixes)
198+
}
199+
if !reflect.DeepEqual(cfg.ExcludedPathPatterns, []string{"/v*/status"}) {
200+
t.Errorf("ExcludedPathPatterns = %v, want [/v*/status]", cfg.ExcludedPathPatterns)
201+
}
202+
}
203+
204+
func strPtr(s string) *string { return &s }

0 commit comments

Comments
 (0)