[FEATURE] Conditional HTTP GET for Blacklist Downloads (ETag / Last-Modified) #3282

[mkf-sysangels]

[FEATURE] Conditional HTTP GET for Blacklist Downloads (ETag / Last-Modified) #3282

[Copilot]

Pull request overview

This PR implements conditional HTTP GET support (via ETag / Last-Modified headers) for the blacklist download job in BunkerWeb. Instead of performing a full download on every job run after the 1-hour TTL expires, the job now sends If-None-Match or If-Modified-Since headers to allow servers to respond with a lightweight 304 Not Modified. A HEAD-request-based Content-Length comparison is included as a last-resort fallback when neither ETag nor Last-Modified are available. The feature reduces bandwidth, CPU load, and load on upstream list providers.

Changes:

• New parse_http_metadata() helper to extract ETag, Last-Modified, and Content-Length from cached file headers.
• New build_cache_header() helper to persist HTTP metadata (ETag, Last-Modified, Content-Length) in the cache file's comment header block.
• Updated download loop to send conditional GET headers, handle 304 responses by refreshing the cache timestamp, and fall back to a HEAD request + Content-Length comparison when no standard validators are present. Also upgrades several log calls from warning to error where appropriate.