WatchGuard OSINT_URL

Comprehensive URL OSINT and Security Assessment Microservice

A production-ready, dockerized OSINT reconnaissance service for automated security assessment of web applications and domains. Part of the WatchGuard security platform by CyberLink Security.

---

🎯 Features

Core Capabilities

• DNS Reconnaissance: A, AAAA, MX, NS, TXT, CAA, CNAME records, DNSSEC validation
• Subdomain Enumeration: Automated discovery of common subdomains
• TLS/SSL Analysis: Certificate validation, chain inspection, protocol support testing
• HTTP Security Headers: Comprehensive security header auditing (HSTS, CSP, CORS, etc.)
• Technology Detection: Server, CDN, and framework fingerprinting
• Sensitive File Detection: Automated checks for exposed .git, .env, security.txt
• Risk Scoring: Automated security risk assessment with severity classification
• Report Generation: Visual HTML and PDF reports with executive summaries

Technical Stack

• Framework: FastAPI (async Python web framework)
• OSINT Tools: dig, nmap, wget, openssl, whois
• Report Engine: Jinja2 templates with WeasyPrint PDF export
• Containerization: Docker with Debian 12 (Bookworm) base
• Deployment: Render.com compatible with health checks

---

🚀 Quick Start

Prerequisites

• Docker 20.10+
• Docker Compose 2.0+
• 2GB RAM minimum

Local Development

1. Clone/Navigate to project:

cd /path/to/WatchGuard/tools/OSINT_URL

2. Build and run with Docker Compose:

docker-compose up --build

3. Access the service:

• Web Interface: http://localhost:8000
• API Documentation: http://localhost:8000/api/docs
• Health Check: http://localhost:8000/health

---

📡 API Usage

1. Initiate OSINT Scan

curl -X POST http://localhost:8000/api/scan \
  -H "Content-Type: application/json" \
  -d '{"target": "example.com"}'

Response:

{
  "scan_id": "123e4567-e89b-12d3-a456-426614174000",
  "target": "example.com",
  "status": "processing",
  "message": "Scan initiated for example.com. Use scan_id to retrieve results."
}

2. Retrieve Scan Results

curl http://localhost:8000/api/scan/{scan_id}

3. Download HTML Report

curl -O http://localhost:8000/api/report/{scan_id}/html

4. Download PDF Report

curl -O http://localhost:8000/api/report/{scan_id}/pdf

---

🔧 Configuration

Environment Variables

Variable	Default	Description
PORT	8000	HTTP server port
PYTHONUNBUFFERED	1	Disable Python output buffering

Docker Build Arguments

Build with custom Python version:

docker build --build-arg PYTHON_VERSION=3.11 -t osint_url .

---

📊 Sample Scan Output

Security Assessment Structure

{
  "target": "example.com",
  "scan_timestamp": "2025-11-17T22:30:00",
  "scan_duration_seconds": 45.2,
  "security_assessment": {
    "risk_level": "MEDIUM",
    "risk_score": 12,
    "summary": {
      "total_issues": 15,
      "critical": 0,
      "high": 3,
      "medium": 7,
      "low": 3,
      "info": 2
    }
  },
  "dns": { ... },
  "tls": { ... },
  "http": { ... }
}

---

🏗️ Architecture

Project Structure

OSINT_URL/
├── app/
│   ├── main.py                  # FastAPI application
│   ├── osint/
│   │   ├── __init__.py          # OSINT orchestrator
│   │   ├── dns_recon.py         # DNS reconnaissance
│   │   ├── tls_analysis.py      # TLS/SSL analysis
│   │   └── http_analysis.py     # HTTP security headers
│   └── reports/
│       ├── __init__.py
│       └── generator.py         # HTML/PDF report generator
├── templates/                   # Jinja2 templates
├── static/                      # Static assets
├── reports/                     # Generated reports
├── Dockerfile                   # Container definition
├── docker-compose.yml           # Local orchestration
├── render.yaml                  # Render.com deployment
├── requirements.txt             # Python dependencies
└── README.md                    # This file

Scan Workflow

1. User submits target URL via REST API
2. Background task initiates OSINT scan
3. Parallel execution of DNS, TLS, HTTP modules
4. Security assessment engine analyzes results
5. Risk score calculation and issue classification
6. HTML/PDF report generation on demand

---

🚢 Production Deployment

Render.com Deployment

1. Push to Git repository:

git init
git add .
git commit -m "Initial commit - WatchGuard OSINT_URL"
git remote add origin <your-repo-url>
git push -u origin main

2. Configure Render.com:

• Connect repository
• Select "Deploy from Git"
• Render will auto-detect render.yaml
• Service will deploy automatically

3. Access deployed service:

https://watchguard-osint-url.onrender.com

Custom Infrastructure

For AWS/GCP/Azure deployment:

# Build image
docker build -t watchguard-osint-url:1.0.0 .

# Push to registry
docker tag watchguard-osint-url:1.0.0 registry.example.com/watchguard-osint-url:1.0.0
docker push registry.example.com/watchguard-osint-url:1.0.0

# Deploy to Kubernetes
kubectl apply -f k8s/deployment.yaml

---

🔒 Security Considerations

Operational Security

• Service runs as non-root user (osint)
• No sensitive credentials stored in container
• Network egress restricted to standard ports (53, 80, 443)
• Health checks prevent failed deployments

Input Validation

• URL sanitization and normalization
• Rate limiting recommended for production (implement via API gateway)
• CORS configured (update allow_origins for production)

Data Retention

• In-memory storage by default (use Redis/PostgreSQL for persistence)
• Generated reports stored in /app/reports (mount persistent volume)

---

🧪 Testing

Run Unit Tests

docker-compose exec osint_url pytest

Manual Test Scan

# Test against test domain
curl -X POST http://localhost:8000/api/scan \
  -H "Content-Type: application/json" \
  -d '{"target": "example.com"}'

---

📈 Monitoring

Health Check Endpoint

curl http://localhost:8000/health

Response:

{
  "status": "healthy",
  "service": "OSINT_URL",
  "version": "1.0.0",
  "timestamp": "2025-11-17T22:30:00"
}

Logs

# Docker Compose
docker-compose logs -f osint_url

# Docker
docker logs -f watchguard_osint_url

---

🛠️ Development

Install Dependencies Locally

python -m venv venv
source venv/bin/activate  # Linux/Mac
pip install -r requirements.txt

Run Without Docker

uvicorn app.main:app --host 0.0.0.0 --port 8000 --reload

Add New OSINT Module

1. Create module in app/osint/
2. Import in app/osint/__init__.py
3. Integrate in OSINTScanner.scan() method
4. Update security assessment logic
5. Add to report template

---

📝 API Reference

Endpoints

Method	Endpoint	Description
GET	/	Service landing page
GET	/health	Health check
POST	/api/scan	Initiate OSINT scan
GET	/api/scan/{scan_id}	Retrieve scan results
GET	/api/report/{scan_id}/html	Download HTML report
GET	/api/report/{scan_id}/pdf	Download PDF report
GET	/api/docs	Interactive API documentation (Swagger UI)

---

🐛 Troubleshooting

Common Issues

Problem: Container fails to start

# Check logs
docker-compose logs osint_url

# Rebuild without cache
docker-compose build --no-cache

Problem: Scan times out

• Increase Docker memory limit (Settings > Resources)
• Check network connectivity from container
• Verify target domain is accessible

Problem: PDF generation fails

• Ensure WeasyPrint dependencies are installed (included in Dockerfile)
• Check /app/reports directory permissions

---

🤝 Contributing

This tool is part of the CyberLink Security WatchGuard platform. For internal development:

1. Create feature branch: git checkout -b feature/your-feature
2. Make changes with comprehensive tests
3. Update documentation
4. Submit pull request for review

---

📜 License

Proprietary - CyberLink Security
Internal use only. All rights reserved.

---

👥 Team

CyberLink Security - WatchGuard Platform

• Engineering Lead: Development Team
• Product Owner: Project Lead

---

📞 Support

For issues or questions:

• Internal Slack: #watchguard-platform
• Email: [email protected]
• Documentation: Internal Wiki

---

Version: 1.0.0
Last Updated: November 2025
Status: Production Ready