Skip to content

Commit 25a7f4e

Browse files
al3xneal3xne
authored
Merge branch 'main' into readme_updates
2 parents 1cfadc7 + 31e742a commit 25a7f4e

36 files changed

+5050
-0
lines changed

‎security/security-design/shared-assets/oci-security-health-check-forensics/LICENSE

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
Copyright (c) 2025 Oracle and/or its affiliates.
2+
3+
The Universal Permissive License (UPL), Version 1.0
4+
5+
Subject to the condition set forth below, permission is hereby granted to any
6+
person obtaining a copy of this software, associated documentation and/or data
7+
(collectively the "Software"), free of charge and under any and all copyright
8+
rights in the Software, and any and all patent rights owned or freely
9+
licensable by each licensor hereunder covering either (i) the unmodified
10+
Software as contributed to or provided by such licensor, or (ii) the Larger
11+
Works (as defined below), to deal in both
12+
13+
(a) the Software, and
14+
(b) any piece of software and/or hardware listed in the lrgrwrks.txt file if
15+
one is included with the Software (each a "Larger Work" to which the Software
16+
is contributed by such licensors),
17+
18+
without restriction, including without limitation the rights to copy, create
19+
derivative works of, display, perform, and distribute the Software and make,
20+
use, sell, offer for sale, import, export, have made, and have sold the
21+
Software and the Larger Work(s), and to sublicense the foregoing rights on
22+
either these or other terms.
23+
24+
This license is subject to the following condition:
25+
The above copyright notice and either this complete permission notice or at
26+
a minimum a reference to the UPL must be included in all copies or
27+
substantial portions of the Software.
28+
29+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
30+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
31+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
32+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
33+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
34+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
35+
SOFTWARE.

‎security/security-design/shared-assets/oci-security-health-check-forensics/README.md

Lines changed: 298 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,298 @@
1+
# OCI Security Health Check Forensics Tool
2+
3+
Last updated: 11 June 2025
4+
5+
The OCI Security Health Check Forensics Tool (the tool) is designed to load and analyze data from Oracle Cloud Infrastructure (OCI) environments. This tool enables users to import CSV files containing OCI resource information (e.g., compute instances, users, compartments) and perform SQL queries on the data. This data is used to investigate configuration issues etc.
6+
7+
The tool can also digest audit events and cloud guard problems. These resources can be loaded with different snapshots from a certain date with a number of days prior to that date.
8+
9+
This data can be used to investiage anomalies.
10+
11+
## Features
12+
- Automatic OCI data fetching using showoci integration
13+
- **Audit events** and **Cloud Guard problems** fetching with parallel processing
14+
- Advanced filtering capabilities for age-based and compartment analysis
15+
- Interactive tenancy selection from combined OCI configuration files
16+
- Load CSV files with OCI data from multiple tenancies
17+
- Execute SQL queries on the loaded data using DuckDB backend. Stay tuned for autonomous DB support.
18+
- Support for `SHOW TABLES` and `DESCRIBE table_name` commands
19+
- Command history and help system
20+
- Batch query execution from YAML files
21+
22+
The tool will be used for forensic purposes. Data can be collected by the customer and shipped to Oracle for forensic research.
23+
24+
The tool is in development and the following is on the backlog:
25+
- Switch back-end DB for large data sets. ADB support.
26+
- Customer documentation to extract data and ship to Oracle in a secure way
27+
28+
## Know Errors
29+
- Error shown when a query results in an empty data frame when a filter is applied.
30+
31+
## Installation
32+
33+
Clone the repository:
34+
```bash
35+
git clone <repository-url>
36+
cd healthcheck-forensic
37+
```
38+
39+
Set up a Python virtual environment and install dependencies:
40+
```bash
41+
python3.10 -m venv .venv
42+
source .venv/bin/activate
43+
pip install -r requirements.txt
44+
```
45+
46+
The `requirements.txt` file contains dependencies for DuckDB, pandas, OCI SDK, and other required libraries.
47+
48+
### OCI Configuration Files
49+
50+
The tool now supports split OCI configuration:
51+
52+
- **`~/.oci/config`**: Contains only the DEFAULT domain configuration
53+
- **`qt_config`**: Contains additional tenancy configurations
54+
55+
The tool automatically combines these files when selecting tenancies. This separation allows you to keep your main OCI config clean while managing multiple tenancies in a separate file.
56+
57+
## Usage
58+
59+
### Running the Tool
60+
61+
To start the tool, use:
62+
```bash
63+
python healthcheck_forensic_tool.py
64+
```
65+
### Interactive Mode
66+
67+
The tool supports an interactive mode for running SQL queries dynamically. Available commands include:
68+
69+
#### Basic Commands
70+
- `show tables`: Lists all loaded tables
71+
- `describe <table_name>`: Displays columns and data types for a given table
72+
- `history`: Shows command history
73+
- `help [command]`: Shows help for commands
74+
- `exit` or `quit`: Exits the application
75+
76+
#### Data Management
77+
- `set tenancy`: Switch between different OCI tenancies
78+
- `set queries [directory]`: Load queries from YAML files for batch execution
79+
- `run queries`: Execute all loaded queries in sequence
80+
81+
#### Data Fetching
82+
- `audit_events fetch <DD-MM-YYYY> <days>`: Fetch <days> of audit events prior to specified date.
83+
- `audit_events fetch`: Interactive loader for existing audit data
84+
- `audit_events delete`: Delete audit events files and tables
85+
- `cloudguard fetch <DD-MM-YYYY> <days>`: Fetch <days> of cloud guard problems prior to specified date.
86+
- `cloudguard fetch`: Interactive loader for existing Cloud Guard data
87+
- `cloudguard delete`: Delete Cloud Guard files and tables
88+
89+
#### Filtering and Analysis
90+
- `filter age <column> <older|younger> <days>`: Filter results by date age
91+
- `filter compartment <subcommand>`: Analyze compartment structures
92+
- `root`: Show root compartment
93+
- `depth`: Show maximum depth
94+
- `tree_view`: Display compartment tree
95+
- `path_to <compartment>`: Show path to specific compartment
96+
97+
### Command-line Switches
98+
99+
| Switch | Description |
100+
|------------------|---------------------------------------------------|
101+
| `--config-file` | Path to the configuration file (`config.yaml`). |
102+
| `--interactive` | Enable interactive SQL mode. <Default> |
103+
104+
Example usage:
105+
```bash
106+
python healthcheck_forensic_tool.py
107+
```
108+
109+
## Configuration Options (`config.yaml`)
110+
111+
| Setting | Description |
112+
|----------------------------|-------------|
113+
| `oci_config_file` | Path to the main OCI config file (default: `~/.oci/config`) |
114+
| `tqt_config_file` | Path to the additional tenancies config file (default: `config/qt_config`) |
115+
| `csv_dir` | Directory for CSV files |
116+
| `prefix` | Filename prefix for filtering CSVs |
117+
| `resource_argument` | Resource argument for showoci (a: all, i: identity, n: network, c: compute, etc.) |
118+
| `delimiter` | Delimiter used in CSV files |
119+
| `case_insensitive_headers` | Convert column headers to lowercase |
120+
| `log_level` | Logging level (`INFO`, `DEBUG`, `ERROR`) |
121+
| `interactive` | Enable interactive mode |
122+
| `audit_worker_count` | Number of parallel workers for audit/Cloud Guard fetching (default: 10) |
123+
| `audit_worker_window` | Hours per batch for parallel fetching (default: 1) |
124+
125+
### Example `config.yaml`
126+
```yaml
127+
# OCI Configuration
128+
oci_config_file: "~/.oci/config" # Main OCI config (DEFAULT domain)
129+
tqt_config_file: "qt_config" # Additional tenancies
130+
131+
# Data Management
132+
csv_dir: "data"
133+
prefix: "oci"
134+
resource_argument: "a"
135+
136+
# Output Settings
137+
output_format: "DataFrame"
138+
log_level: "INFO"
139+
delimiter: ","
140+
case_insensitive_headers: true
141+
142+
# Interactive Mode
143+
interactive: true
144+
145+
# Parallel Fetching Configuration
146+
audit_worker_count: 10
147+
audit_worker_window: 1
148+
```
149+
150+
## Predefined Queries
151+
152+
Queries can be defined in YAML files for batch execution. Example `queries.yaml`:
153+
```yaml
154+
queries:
155+
- description: "List all users with API access"
156+
sql: "SELECT display_name, can_use_api_keys FROM identity_domains_users WHERE can_use_api_keys = 1"
157+
- description: "Show compute instances by compartment"
158+
sql: "SELECT server_name, compartment_name, status FROM compute WHERE status = 'STOPPED'"
159+
filter: "age last_modified older 30"
160+
sql: "sql: "SELECT server_name, compartment_name, status FROM compute WHERE compartment_name = '<YOUR COMPARTMENT_NAME>'"
161+
```
162+
163+
## Example Usage Scenarios
164+
165+
### Getting Started
166+
```bash
167+
# Start the tool
168+
python healthcheck_forensic_tool.py
169+
170+
# Select tenancy and load data
171+
# Tool will prompt for tenancy selection from combined configs
172+
173+
# Basic exploration
174+
CMD> show tables
175+
CMD> describe identity_domains_users
176+
CMD> SELECT COUNT(*) FROM compute;
177+
```
178+
179+
### Data Fetching
180+
```bash
181+
# Fetch 2 days of audit events ending June 15, 2025
182+
CMD> audit_events fetch 15-06-2025 2
183+
184+
# Fetch 30 days of Cloud Guard problems ending January 1, 2025
185+
CMD> cloudguard fetch 01-01-2025 30
186+
187+
# Load existing audit data interactively
188+
CMD> audit_events fetch
189+
```
190+
191+
### Advanced Analysis
192+
```bash
193+
# Filter API keys older than 90 days
194+
CMD> SELECT display_name, api_keys FROM identity_domains_users;
195+
CMD> filter age api_keys older 90
196+
197+
# Analyze compartment structure
198+
CMD> SELECT path FROM identity_compartments;
199+
CMD> filter compartment tree_view
200+
CMD> filter compartment path_to my-compartment
201+
```
202+
203+
### Batch Operations
204+
```bash
205+
# Load and run predefined queries
206+
CMD> set queries < Select a query file using the query file browser >
207+
CMD> run queries
208+
209+
# Switch between tenancies
210+
CMD> set tenancy
211+
```
212+
213+
## Data Organization
214+
215+
The tool organizes data in the following structure:
216+
```
217+
data/
218+
├── tenancy1/
219+
│ ├── tenancy1_20241215_143022/
220+
│ │ ├── oci_compute.csv
221+
│ │ ├── oci_identity_domains_users.csv
222+
│ │ ├── audit_events_15-06-2025_7.json
223+
│ │ └── cloudguard_problems_15062025_7.json
224+
│ └── tenancy1_20241214_091545/
225+
└── tenancy2/
226+
└── tenancy2_20241215_100530/
227+
```
228+
229+
## Logging
230+
231+
Logging is configured via the `log_level` setting in `config.yaml`. The tool provides detailed logging for:
232+
- Configuration loading and validation
233+
- CSV file loading and table creation
234+
- Query execution and results
235+
- Data fetching operations with progress tracking
236+
- Error handling and troubleshooting information
237+
238+
## Troubleshooting
239+
240+
### Common Issues
241+
242+
**OCI Configuration Problems**
243+
- Ensure both `~/.oci/config` and `config/qt_config` exist and are readable
244+
- Verify that tenancy profiles are properly configured with required keys
245+
- Check that API keys and permissions are correctly set up
246+
247+
**CSV Loading Issues**
248+
- Ensure CSV files are properly formatted with consistent delimiters
249+
- Column names in queries should match those in the loaded data (case-sensitive by default)
250+
- Check that the specified prefix matches your CSV file naming convention
251+
252+
**Data Fetching Problems**
253+
- Verify OCI permissions for audit events and Cloud Guard APIs
254+
- Check network connectivity and OCI service availability
255+
- Ensure the date range doesn't exceed OCI's retention periods (365 days for audit events)
256+
257+
**Query Execution**
258+
- Use DuckDB-compatible SQL syntax
259+
- Table names are derived from CSV filenames (minus prefix and extension)
260+
- Check available tables with `show tables` and column structure with `describe <table>`
261+
262+
### Getting Help
263+
264+
For detailed command help:
265+
```bash
266+
CMD> help # Show all commands
267+
CMD> help audit_events fetch # Show audit_events fetch options
268+
CMD> help filter age # Show filter age options
269+
```
270+
271+
## Advanced Features
272+
273+
### Parallel Data Fetching
274+
The tool supports parallel fetching for large datasets:
275+
- Configurable worker count and time windows
276+
- Progress tracking with detailed summaries
277+
- Automatic retry handling for failed intervals
278+
- Clean temporary file management
279+
280+
### Smart Configuration Management
281+
- Automatic detection and combination of split OCI configs
282+
- Interactive tenancy selection with metadata display
283+
- Temporary file creation for showoci integration
284+
- Graceful handling of missing or invalid configurations
285+
286+
### Comprehensive Filtering
287+
- Date-based filtering with flexible column support
288+
- Compartment hierarchy analysis and visualization
289+
- Support for complex nested data structures
290+
- Chainable filter operations on query results
291+
292+
# License
293+
294+
Copyright (c) 2025 Oracle and/or its affiliates.
295+
296+
Licensed under the Universal Permissive License (UPL), Version 1.0.
297+
298+
See [LICENSE](https://github.com/oracle-devrel/technology-engineering/blob/main/LICENSE) for more details.

‎security/security-design/shared-assets/oci-security-health-check-forensics/classes/__init__.py

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
from .output_formatter import OutputFormatter

0 commit comments

Comments
 (0)