Vets-API in-app PII scrubbing

[rmtolmach]

User Story

As a platform engineer, I want to implement additional in-app PII scrubbing mechanisms in vets-api so that sensitive data is filtered before logs are emitted or sent to Datadog, complementing logstop and Datadog Agent–level scrubbing.

Issue Description

Vets-API already filters request parameters through filter_parameter_logging.rb, but this only covers Rails params. Other log sources—such as custom log messages, ActiveRecord/SQL logs—may still include sensitive data depending on how they are logged.

Since we've decided against the logstop gem, the next step is to apply additional scrubbing at the vets-api application level, specifically in areas not covered by parameter filtering. Two likely areas for improvement are:

1. Adding optional PII scrubbing within the vets-api logging stack (for example, a logger wrapper that redacts sensitive patterns before messages are emitted).
2. Enabling or expanding Datadog APM/trace obfuscation features (such as SQL literal obfuscation) through config/initializers/datadog.rb.

This work should remain small in scope: introduce minimal, high-value scrubbing for a small set of patterns (for example, email addresses and SSNs) in places where parameter filtering does not apply.

Tasks

• Review how vets-api logging flows through Rails.logger, Sidekiq logging, and Datadog tracer logging to determine the safest interception point for additional scrubbing.
• Implement a minimal PII-scrubbing layer for non-parameter log content (for example, redact email + SSN patterns before logs are emitted).
• Evaluate Datadog APM configuration options (for example: ActiveRecord query obfuscation) and enable necessary settings in config/initializers/datadog.rb.
• Verify that existing log filtering (filter_parameter_logging.rb) remains unaffected.
• Test in lower environments to ensure:
  • the application still logs correctly,
  • SQL queries are obfuscated (if configured),
  • non-param log messages are scrubbed as expected,
  • no regressions or unexpected scrubbing occur.
• Prepare follow-up tickets for broader scrubbing enhancements if needed.

Acceptance Criteria

• Additional vets-api log scrubbing is implemented for at least one high-value pattern not covered by filter_parameter_logging.rb (for example, emails).
• Datadog APM obfuscation settings are enabled or updated as appropriate.
• Logging behavior is verified in a lower environment with no unexpected side effects.
• Application-level scrubbing complements (does not conflict with) Datadog Agent–level scrubbing.

Reference

Existing parameter filtering:
config/initializers/filter_parameter_logging.rb

Datadog APM tracing + SQL obfuscation:
https://docs.datadoghq.com/tracing/trace_collection/compatibility/ruby/?tab=containers#active-record

Datadog log processing (Agent-level, for context):
https://docs.datadoghq.com/agent/logs/advanced_log_collection/?tab=kubernetes&site=gov

---

Validation

Assignee to add steps to this section. List the actions that need to be taken to confirm this issue is complete. Include any necessary links or context. State the expected outcome(s).