Update Python version requirements to support up to 3.14 and adjust CI workflow accordingly

Author: fedelemantuano

.github/copilot-instructions.md

@@ -1,219 +1,226 @@
 # Copilot Instructions for mail-parser
 
-## Project Overview
+mail-parser is a **production-grade email parsing library** for Python that transforms raw email messages into
+structured Python objects. Originally built as the foundation for [SpamScope](https://github.com/SpamScope/spamscope),
+it excels at security analysis, forensics, and RFC-compliant email processing.
 
-mail-parser is a Python library that parses raw email messages into structured Python objects,
-serving as the foundation for [SpamScope](https://github.com/SpamScope/spamscope). It handles both
-standard email formats and Outlook .msg files, with a focus on security analysis and forensics.
+## Core Architecture
 
-## Architecture & Key Components
+### Factory-Based API Pattern
 
-### Core Parser (`src/mailparser/core.py`)
+**Always use factory functions** instead of direct `MailParser()` instantiation:
 
-- **MailParser class**: Main parser with factory methods (`from_file`, `from_string`, `from_bytes`,
-  etc.)
-- **Property-based API**: Email components accessible as properties (`.subject`, `.from_`,
-  `.attachments`)
-- **Multi-format access**: Each property has `_raw`, `_json` variants (e.g., `mail.to`,
-  `mail.to_raw`, `mail.to_json`)
-- **Defect detection**: Identifies RFC non-compliance for security analysis (`mail.defects`,
-  `mail.defects_categories`)
+```python
+import mailparser
+mail = mailparser.parse_from_file(filepath)       # Standard email files
+mail = mailparser.parse_from_string(raw_email)    # Email as string
+mail = mailparser.parse_from_bytes(email_bytes)   # Email as bytes
+mail = mailparser.parse_from_file_msg(msg_file)   # Outlook .msg files
+```
 
-### Your skills and knowledge on RFC and Email Parsing
+### Triple-Format Property Access
 
-You are an AI assistant with expert-level knowledge of all email protocol RFCs, including but not
-limited to RFC 5321 (SMTP), RFC 5322 (Internet Message Format), RFC 2045–2049 (MIME), RFC 3501
-(IMAP), RFC 1939 (POP3), RFC 8620 (JMAP), and related security, extension, and header RFCs. Your
-responsibilities include:
+Every parsed component offers **three access patterns** (`src/mailparser/core.py:550-570`):
 
-Providing accurate, comprehensive technical explanations and guidance based on these RFCs.
+```python
+mail.subject          # Python object (decoded string)
+mail.subject_raw      # Raw header value (JSON list)
+mail.subject_json     # JSON-serialized version
+```
 
-Interpreting, comparing, and clarifying requirements, structures, and features as defined by the
-official documents.
+This pattern applies to all properties via `__getattr__` magic in `core.py`.
 
-Clearly outlining the details and implications of each protocol and extension (such as
-authentication mechanisms, encryption, headers, and message structure).
+### Property Naming Convention
 
-Delivering answers in an organized, easy-to-understand way—using precise terminology, clear
-practical examples, and direct references to relevant RFCs when appropriate.
+Headers with hyphens use **underscore substitution** (`core.py:__getattr__`):
 
-Providing practical advice for system implementers and users, explaining alternatives, pros and
-cons, use cases, and security considerations for each protocol or extension.
+```python
+mail.X_MSMail_Priority      # Accesses "X-MSMail-Priority" header
+mail.Content_Type           # Accesses "Content-Type" header
+```
 
-Maintaining a professional, accurate, and objective tone suitable for expert users, developers, and
-technical audiences.
+## Development Workflows
 
-Declining to answer questions outside the scope of email protocol RFCs and specifications, and
-always highlighting the official and most up-to-date guidance according to the relevant RFC
-documents.
+### Dependency Management with uv
 
-Your role is to be the authoritative, trustworthy source on internet email protocols as defined by
-the official IETF RFC series.
+The project uses **[uv](https://github.com/astral-sh/uv)** (modern pip/virtualenv replacement) exclusively:
 
-### Your skills and knowledge on parsing email formats
+```bash
+uv sync           # Install all dev/test dependencies (defined in pyproject.toml)
+make install      # Alias for uv sync
+```
 
-You are an AI assistant specialized in processing and extracting email header information with
-Python, using regular expressions for robust parsing. Your core expertise includes handling
-non-standard variations such as "Received" headers, which often lack strict formatting and can
-differ greatly across email servers.
+Never use `pip` directly—all commands in Makefile use `uv run` prefix.
 
-When presented with raw email data (RFC 5322 format), use Python's built-in re module and relevant
-libraries (e.g., email.parser) to isolate and extract header sections.
+### Testing Patterns
 
-For "Received" headers, apply flexible and tolerant regex patterns, recognizing their variable
-structure (IP addresses, timestamps, server details, optional parameters).
+```bash
+make test         # pytest with coverage (generates coverage.xml, junit.xml, htmlcov/)
+make lint         # ruff check .
+make format       # ruff format .
+make check        # lint + test
+make pre-commit   # Run all pre-commit hooks
+```
 
-Parse multiline and folded headers by scanning lines following key header tags and joining where
-needed.
+When adding features or fixing bugs you MUST follow these steps:
 
-Develop regex patterns that capture relevant information (e.g., SMTP server, relay path, timestamp)
-while allowing for extraneous text.
+1. Add relevant test email to `tests/mails/` if demonstrating new case
+2. Write tests in the corresponding test file following existing patterns, under `tests/`
+3. Run `make test` to verify all tests pass before committing
+4. Run `uv run mail-parser -f tests/mails/mail_test_11 -j` to manually verify JSON output and that new changes
+   work as expected
+5. Run `make pre-commit` to ensure code style compliance before pushing
 
-Document the extraction process: explain which regexes are designed for typical cases and how to
-adapt them for mismatches, edge cases, or partial matches.
+**Test data location**: `tests/mails/` contains malformed emails, Outlook files, and various encodings
+(`mail_test_1` through `mail_test_17`, `mail_malformed_1-3`, `mail_outlook_1`).
 
-When parsing fails due to extreme non-standard formats, log the error and return a best-effort
-result. Always explain any limitations or ambiguities in the extraction.
+**Critical testing rule**: When modifying parsing logic, test against malformed emails to ensure security defect
+detection still works.
 
-Example generic regex for a "Received" header: Received:\s*(.*?);(.*) (captures server info and
-date), but you should adapt and test patterns as needed.
+### Build & Release Process
 
-Provide code comments, extraction summaries, and references for each regex used to ensure
-maintainability and clarity.
+```bash
+make build        # uv build → creates dist/*.tar.gz and dist/*.whl
+make release      # build + twine upload to PyPI
+```
 
-Avoid making assumptions about the order or presence of specific header fields, and handle edge
-cases gracefully.
+Version is **dynamically loaded** from `src/mailparser/version.py` (see
+`pyproject.toml:tool.hatch.version`).
 
-When possible, recommend combining regex with Python's email module for initial header separation,
-then dive deep with regex for specific, non-standard value extraction.
+## Security-First Parsing
 
-Your responses must prioritize accuracy, transparency in limitations, and practical utility for
-anyone parsing complex email headers.
+### Defect Detection System
 
-### Entry Points (`src/mailparser/__init__.py`)
+The parser identifies RFC violations that could indicate malicious intent (`core.py:240-268`):
 
 ```python
-# Factory functions are the primary API
-import mailparser
-mail = mailparser.parse_from_file(filepath)
-mail = mailparser.parse_from_string(raw_email)
-mail = mailparser.parse_from_bytes(email_bytes)
-mail = mailparser.parse_from_file_msg(outlook_file)  # .msg files
+mail.has_defects          # Boolean flag
+mail.defects              # List of defect dicts by content type
+mail.defects_categories   # Set of defect class names (e.g., "StartBoundaryNotFoundDefect")
 ```
 
-### CLI Tool (`src/mailparser/__main__.py`)
-
-- Entry point: `mail-parser` command
-- JSON output mode (`-j`) for integration with other tools
-- Multiple input methods: file (`-f`), string (`-s`), stdin (`-k`)
-- Outlook support (`-o`) with system dependency on `libemail-outlook-message-perl`
+**Epilogue defect handling** (`core.py:320-335`): When `EPILOGUE_DEFECTS` are detected, parser extracts hidden
+content between MIME boundaries that could contain malicious payloads.
 
-## Development Workflows
+### IP Address Extraction
 
-### Setup & Dependencies
+`get_server_ipaddress(trust)` method (`core.py:487-528`) extracts sender IPs with **trust-level validation**:
 
-```bash
-# Use uv for dependency management (modern pip replacement)
-uv sync  # Installs all dev/test dependencies
-make install  # Alias for uv sync
+```python
+# Finds first non-private IP in trusted headers
+mail.get_server_ipaddress(trust="Received")
 ```
 
-### Testing & Quality
+Filters out private IP ranges using Python's `ipaddress` module.
 
-```bash
-make test     # pytest with coverage (outputs coverage.xml, junit.xml)
-make lint     # ruff linting
-make format   # ruff formatting
-make check    # lint + test
-make pre-commit  # runs pre-commit hooks
-```
-
-For all unittest use `pytest` framework and mock external dependencies as needed.
-When you modify code, ensure all tests pass and coverage remains high.
+### Received Header Parsing
 
-### Build & Release
+Complex regex-based parsing (`utils.py:302-360`, patterns in `const.py:24-73`) extracts hop-by-hop routing:
 
-```bash
-make build    # uv build (creates wheel/sdist in dist/)
-make release  # build + twine upload to PyPI
+```python
+# Returns list of dicts with: by, from, date, date_utc, delay, envelope_from, hop, with
+mail.received
 ```
 
-### Docker Development
+**Key pattern**: `RECEIVED_COMPILED_LIST` contains pre-compiled regexes for "from", "by", "with", "id", "for",
+"via", "envelope-from", "envelope-sender", and date patterns. Recent fixes addressed IBM gateway duplicate matches
+(see comments in `const.py:26-38`).
 
-- Dockerfile uses Python 3.10-slim with `libemail-outlook-message-perl`
-- docker-compose.yml mounts `~/mails` for testing
-- Image available as `fmantuano/spamscope-mail-parser`
+If parsing fails, falls back to `receiveds_not_parsed()` returning `{"raw": <header>, "hop": <n>}`
+structure.
 
-## Key Patterns & Conventions
+## Project Structure Specifics
 
-### Header Access Pattern
+### src/ Layout
 
-Headers with hyphens use underscore substitution:
+Package uses modern **src-layout** (`src/mailparser/`) for cleaner imports and testing isolation:
 
-```python
-mail.X_MSMail_Priority  # for X-MSMail-Priority header
+```text
+src/mailparser/
+├── __init__.py      # Exports factory functions
+├── __main__.py      # CLI entry point (mail-parser command)
+├── core.py          # MailParser class (760 lines)
+├── utils.py         # Parsing utilities (582 lines)
+├── const.py         # Regex patterns and constants
+├── exceptions.py    # Exception hierarchy
+└── version.py       # Version string
 ```
 
-### Attachment Structure
+### External Dependency: Outlook Support
 
-```python
-# Each attachment is a dict with standardized keys
-for attachment in mail.attachments:
-    attachment['filename']
-    attachment['payload']  # base64 encoded
-    attachment['content_transfer_encoding']
-    attachment['binary']  # boolean flag
+Outlook `.msg` file parsing requires **system-level Perl module**:
+
+```bash
+apt-get install libemail-outlook-message-perl  # Debian/Ubuntu
 ```
 
-### Received Header Parsing
+Triggered via `msgconvert()` function in `utils.py` that shells out to Perl script. Raises `MailParserOutlookError`
+if unavailable.
 
-Complex parsing in `receiveds_parsing()` extracts hop-by-hop email routing:
+### CLI Tool Pattern
 
-```python
-mail.received  # List of parsed received headers with structured data
-# Each hop contains: by, from, date, delay, envelope_from, etc.
+`__main__.py` provides production CLI with mutually exclusive input modes (`-f`, `-s`, `-k`), JSON output (`-j`),
+and selective printing (`-b`, `-a`, `-r`, `-t`).
+
+**Entry point defined** in `pyproject.toml:project.scripts`:
+
+```toml
+[project.scripts]
+mail-parser = "mailparser.__main__:main"
 ```
 
-### Error Handling Hierarchy
+## Code Style & Tooling
 
-```python
-MailParserError  # Base exception
-├── MailParserOutlookError  # Outlook .msg issues
-├── MailParserEnvironmentError  # Missing dependencies
-├── MailParserOSError  # File system issues
-└── MailParserReceivedParsingError  # Header parsing failures
+### Ruff Configuration
+
+Single linter/formatter (replaces black, isort, flake8):
+
+```toml
+[tool.ruff.lint]
+select = ["E", "F", "I"]  # pycodestyle, pyflakes, isort
+# "UP", "B", "SIM", "S", "PT" commented out in pyproject.toml
 ```
 
-## Testing Approach
+### Pytest Configuration
 
-- Test emails in `tests/mails/` (malformed, Outlook, various encodings)
-- Comprehensive property testing for all email components
-- CLI integration tests in CI pipeline
-- Coverage reporting with pytest-cov
+Key markers in `pyproject.toml:tool.pytest.ini_options`:
 
-## Security Focus
+- `integration`: marks integration tests
+- Coverage outputs: XML (for CI), HTML (for local), terminal
+- JUnit XML for CI integration
 
-- **Defect detection**: Identifies malformed boundaries that could hide malicious content
-- **IP extraction**: `get_server_ipaddress()` with trust levels for forensic analysis
-- **Epilogue analysis**: Detects hidden content in malformed MIME boundaries
-- **Fingerprinting**: Mail and attachment hashing for threat intelligence
+## Common Pitfalls
 
-## Build System Specifics
+1. **Don't instantiate `MailParser()` directly**—use factory functions from `__init__.py`
+2. **Don't use `pip`**—always use `uv` or Makefile targets
+3. **Don't ignore defects**—they're critical for security analysis
+4. **Don't assume headers exist**—use `.get()` pattern or handle `None`
+5. **Test against malformed emails**—`tests/mails/mail_malformed_*` files exist for this reason
+
+## Docker Development
+
+Dockerfile uses **Python 3.10-slim-bookworm** with Outlook dependencies pre-installed. Container runs as non-root
+`mailparser` user.
+
+```bash
+docker build -t mail-parser .
+docker run mail-parser -f /path/to/email
+```
 
-- **pyproject.toml**: Modern Python packaging with hatch backend
-- **uv**: Used instead of pip for faster, reliable dependency resolution
-- **src/ layout**: Package in `src/mailparser/` for cleaner imports
-- **Dynamic versioning**: Version from `src/mailparser/version.py`
+## Key Reference Points
 
-## External Dependencies
+- **Property implementation**: `core.py:540-730` (all `@property` decorators)
+- **Attachment extraction**: `core.py:355-475` (walks multipart, handles encoding)
+- **Received parsing logic**: `utils.py:302-455` + `const.py:24-73` (regex patterns)
+- **CLI implementation**: `__main__.py:30-347` (argparse + output formatting)
+- **Exception hierarchy**: `exceptions.py:20-60` (5 exception types)
 
-- **Outlook support**: Requires system package `libemail-outlook-message-perl` + Perl module `Email::Outlook::Message`
-- **six**: Python 2/3 compatibility (legacy requirement)
-- **Minimal runtime deps**: Only `six>=1.17.0` required
+## Testing Strategy
 
-When working with this codebase:
+When adding features:
 
-- Use factory functions, not direct MailParser() instantiation
-- Test with various malformed emails from `tests/mails/`
-- Remember header property naming (underscores for hyphens)
-- Consider security implications of email parsing edge cases
+1. Add test email to `tests/mails/` if demonstrating new case
+2. Write tests in `tests/test_mail_parser.py` following existing patterns
+3. Test both normal and `_raw`/`_json` property variants
+4. Verify defect detection for security-relevant changes
+5. Run `make check` before committing

.github/workflows/main.yml

@@ -12,7 +12,7 @@ jobs:
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        python-version: ['3.8', '3.9', '3.10', '3.11', '3.12', '3.13']
+        python-version: ['3.8', '3.9', '3.10', '3.11', '3.12', '3.13', '3.14']
 
     steps:
       - uses: actions/checkout@v4