AWS SO - Mitigated the PCAP carving problem

[vibrantsculpture-ai]

Version

2.4.190

Installation Method

Cloud image (Amazon, Azure, Google)

Description

other (please provide detail below)

Installation Type

Distributed

Location

cloud

Hardware Specs

Meets minimum requirements

CPU

16

RAM

64G

Storage for /

400G

Storage for /nsm

400G

Network Traffic Collection

other (please provide detail below)

Network Traffic Speeds

Less than 1Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

No, there are no failures

Logs

No, there are no additional clues

Detail

PCAP Correlation Mitigation for AWS Distributed Sensors

This document describes a mitigation solution for a PCAP correlation issue in the AWS version of a distributed Security Onion deployment using separate EC2 sensor instances.

The issue affects environments that rely on AWS Traffic Mirroring and optionally AWS Load Balancers, where packet encapsulation prevents proper PCAP carving and alert correlation in Steno.

---

Problem Summary

When Suricata generated alerts, correlating those alerts to the related PCAP in Steno frequently failed.

Although alerts appeared correctly in the SOC, the corresponding packet data could not be located during PCAP carving.

---

Root Cause Analysis

AWS Traffic Mirroring encapsulates network traffic before it reaches the sensor:

• Mirrored traffic is encapsulated using VXLAN over UDP port 4789
• When traffic passes through AWS load balancers, an additional encapsulation layer is added using Geneve over TCP port 6081
• This results in nested tunnel encapsulation around the original packet

Steno only sees the outer VXLAN wrapper in the PCAP files.

Example Failure Scenario

• Suricata triggers an alert on TCP port 443
• During PCAP correlation, Steno only sees UDP port 4789 (VXLAN)
• The inner packet is hidden by encapsulation
• SOC to PCAP correlation fails

---

Solution Overview

To restore correct PCAP visibility and correlation, an OS-level decapsulation approach was implemented directly on the sensor.

The solution removes VXLAN and Geneve encapsulation before packets are processed by Steno.

---

Implementation Details

Virtual Interface

A virtual interface named decap0 was created on the sensor host.

This interface receives fully decapsulated packets.

Decapsulation Logic

A custom Perl script performs the following actions:

1. Reads mirrored traffic from the sensor’s monitor ENI (for example, eni1)
2. Removes the outer VXLAN encapsulation
3. Removes the inner Geneve encapsulation when present
4. Extracts the original inner packet
5. Writes the decapsulated packet to the decap0 interface

Steno Configuration

Steno was reconfigured to:

• Monitor the decap0 interface
• Ignore the raw ENI interface carrying encapsulated traffic

This ensures Steno sees the actual inner packet rather than the tunnel wrapper.

---

Operationalization

To ensure persistence across reboots:

• A systemd service was created
• The service starts the Perl decapsulation script automatically at boot
• No manual intervention is required after system startup

---

Result

• PCAP carving functions correctly
• Alert to PCAP correlation is restored
• Steno and SOC views are consistent
• VXLAN and Geneve encapsulation no longer interfere with analysis

---

Availability

If there is interest, the following can be provided:

• The Perl decapsulation script
• The systemd service unit file
• Example interface configuration

These can be added directly to this post.

---

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.