🐛: Spam scores not adding up

[creativebastard]

What happened?

I've noticed at times the spam filter seems to be confused.

X-Spam-Result: RBL_SENDERSCORE_REPUT_9 (-1.00),
DMARC_POLICY_ALLOW (-0.50),
DKIM_ALLOW (-0.20),
ARC_ALLOW (0.00),
ARC_SIGNED (0.00),
DKIM_SIGNED (0.00),
DMARC_POLICY_ALLOW_WITH_FAILURES (0.00),
FREEMAIL_TO (0.00),
FROM_HAS_DN (0.00),
FROM_NEQ_ENV_FROM (0.00),
HAS_EXTERNAL_IMG (0.00),
HAS_LINK_TO_LARGE_IMG (0.00),
HAS_PHPMAILER_SIG (0.00),
HAS_REPLYTO (0.00),
HTML_SHORT_1 (0.00),
PREVIOUSLY_DELIVERED (0.00),
RCPT_COUNT_ONE (0.00),
RCVD_COUNT_SEVEN (0.00),
RCVD_IN_DNSWL_NONE (0.00),
RCVD_TLS_LAST (0.00),
REPLYTO_DN_EQ_FROM_DN (0.00),
REPLYTO_DOM_NEQ_FROM_DOM (0.00),
TO_DN_NONE (0.00),
FORGED_SENDER (0.30),
MID_RHS_NOT_FQDN (0.50),
FORGED_RCVD_TRAIL (1.00),
HELO_IPREV_MISMATCH (1.00),
PARTS_DIFFER (1.00),
SPF_FAIL (1.00),
FORGED_RECIPIENTS (2.00),
HTML_SHORT_LINK_IMG_1 (2.00),
SPAM_FLAG (5.00),
SPOOF_REPLYTO (6.00)
X-Spam-Status: No, score=0.41

This should have been Yes for the spam score, but instead it's no and the score is 0.41.

How can we reproduce the problem?

I can reproduce the problem by doing the following steps:

Version

v0.13.x

What database are you using?

None

What blob storage are you using?

S3-compatible

Where is your directory located?

Internal

What operating system are you using?

Linux

Relevant log output

Code of Conduct

• I agree to follow this project's Code of Conduct

[max-baz]

Interesting ticket, I have also noticed the same!

X-Spam-Result: DMARC_POLICY_ALLOW (-0.50),
        DKIM_ALLOW (-0.20),
        ARC_NA (0.00),
        DBL_BLOCKED_OPENRESOLVER (0.00),
        DKIM_SIGNED (0.00),
        FROMTLD_EQ_ENV_FROMTLD (0.00),
        FROM_HAS_DN (0.00),
        HAS_EXTERNAL_IMG (0.00),
        HAS_REPLYTO (0.00),
        HTML_SHORT_3 (0.00),
        RBL_SPAMHAUS_BLOCKED_OPENRESOLVER (0.00),
        RCPT_COUNT_ONE (0.00),
        REPLYTO_DOM_NEQ_FROM_DOM (0.00),
        SPF_NEUTRAL (0.00),
        TO_DN_NONE (0.00),
        TO_MATCH_ENVRCPT_ALL (0.00),
        URIBL_BLOCKED (0.00),
        WP_COMPROMISED (0.00),
        RCVD_COUNT_ZERO (0.10),
        RCVD_NO_TLS_LAST (0.10),
        MIME_HTML_ONLY (0.20),
        SPOOF_REPLYTO (6.00)
X-Spam-Status: No, score=3.70

[creativebastard]

I got another one

X-Spam-Result: DMARC_POLICY_ALLOW (-0.50),
DKIM_ALLOW (-0.20),
SPF_ALLOW (-0.20),
ARC_NA (0.00),
DKIM_SIGNED (0.00),
FROMTLD_EQ_ENV_FROMTLD (0.00),
FROM_HAS_DN (0.00),
HAS_EXTERNAL_IMG (0.00),
HAS_LINK_TO_LARGE_IMG (0.00),
HAS_REPLYTO (0.00),
HTML_SHORT_2 (0.00),
MID_RHS_MATCH_ENV_FROMTLD (0.00),
RCPT_COUNT_ONE (0.00),
REPLYTO_DN_EQ_FROM_DN (0.00),
REPLYTO_DOM_EQ_FROM_DOM (0.00),
TO_DN_NONE (0.00),
TO_MATCH_ENVRCPT_ALL (0.00),
RCVD_COUNT_ZERO (0.10),
RCVD_NO_TLS_LAST (0.10),
MISSING_CHARSET (0.50),
MV_CASE (0.50),
URI_COUNT_ODD (0.50),
CT_EXTRA_SEMI (1.00),
HELO_IPREV_MISMATCH (1.00),
HTML_SHORT_LINK_IMG_2 (1.00),
MID_RHS_MATCH_FROM (1.00),
PARTS_DIFFER (1.00),
RBL_MAILSPIKE_WORST (2.00),
RBL_SENDERSCORE_REPUT_4 (2.00),
RDNS_NONE (2.00),
SEM_URIBL_FRESH15 (3.00),
RBL_SPAMCOP (4.00),
RBL_SPAMHAUS_SBL (4.00),
ABUSE_SURBL (5.00),
BAYES_SPAM (5.10),
DBL_SPAM (6.50),
URIBL_BLACK (7.50)
X-Spam-Status: No, score=4.93

X-Spam-Result: DMARC_POLICY_ALLOW (-0.50),
SPF_ALLOW (-0.20),
ARC_NA (0.00),
DKIM_NA (0.00),
FROMTLD_EQ_ENV_FROMTLD (0.00),
FROM_HAS_DN (0.00),
HAS_EXTERNAL_IMG (0.00),
HAS_LINK_TO_LARGE_IMG (0.00),
HAS_REPLYTO (0.00),
HTML_SHORT_1 (0.00),
MID_RHS_MATCH_ENV_FROMTLD (0.00),
RCPT_COUNT_ONE (0.00),
REPLYTO_EQ_FROM (0.00),
TO_DN_NONE (0.00),
TO_MATCH_ENVRCPT_ALL (0.00),
RCVD_COUNT_ZERO (0.10),
RCVD_NO_TLS_LAST (0.10),
MISSING_CHARSET (0.50),
MV_CASE (0.50),
URI_COUNT_ODD (0.50),
CT_EXTRA_SEMI (1.00),
MID_RHS_MATCH_FROM (1.00),
PARTS_DIFFER (1.00),
HTML_SHORT_LINK_IMG_1 (2.00),
RBL_MAILSPIKE_WORST (2.00),
RBL_SPAMHAUS_CSS (2.00),
RDNS_NONE (2.00),
ABUSE_SURBL (5.00),
BAYES_SPAM (5.10),
DBL_SPAM (6.50),
URIBL_BLACK (7.50)
X-Spam-Status: No, score=2.06

[creativebastard]

X-Spam-Result: DMARC_POLICY_ALLOW (-0.50),
DKIM_ALLOW (-0.20),
SPF_ALLOW (-0.20),
ARC_NA (0.00),
DKIM_SIGNED (0.00),
FROMTLD_EQ_ENV_FROMTLD (0.00),
FROM_HAS_DN (0.00),
HAS_EXTERNAL_IMG (0.00),
HAS_LINK_TO_IMG (0.00),
HAS_LINK_TO_LARGE_IMG (0.00),
HAS_REPLYTO (0.00),
HTML_SHORT_2 (0.00),
MID_RHS_MATCH_ENV_FROMTLD (0.00),
RCPT_COUNT_ONE (0.00),
REPLYTO_EQ_FROM (0.00),
SUBJECT_ENDS_EXCLAIM (0.00),
TO_DN_NONE (0.00),
TO_MATCH_ENVRCPT_ALL (0.00),
RCVD_COUNT_ZERO (0.10),
RCVD_NO_TLS_LAST (0.10),
MISSING_CHARSET (0.50),
MV_CASE (0.50),
URI_COUNT_ODD (0.50),
CT_EXTRA_SEMI (1.00),
HELO_IPREV_MISMATCH (1.00),
HTML_SHORT_LINK_IMG_2 (1.00),
MID_RHS_MATCH_FROM (1.00),
RBL_MAILSPIKE_WORST (2.00),
RBL_VIRUSFREE_BOTNET (2.00),
RDNS_NONE (2.00),
SEM_URIBL_FRESH15 (3.00),
RBL_SENDERSCORE_REPUT_1 (3.50),
RBL_SPAMCOP (4.00),
RBL_SPAMHAUS_SBL (4.00),
ABUSE_SURBL (5.00),
BAYES_SPAM (5.10),
DBL_SPAM (6.50),
URIBL_BLACK (7.50)
X-Spam-Status: Yes, score=8.39

[creativebastard]

X-Spam-Result: DMARC_POLICY_ALLOW (-0.50),
SPF_ALLOW (-0.20),
ARC_NA (0.00),
DKIM_PERMFAIL (0.00),
DKIM_SIGNED (0.00),
FROMTLD_EQ_ENV_FROMTLD (0.00),
FROM_HAS_DN (0.00),
HAS_EXTERNAL_IMG (0.00),
HAS_LINK_TO_IMG (0.00),
HAS_LINK_TO_LARGE_IMG (0.00),
HAS_REPLYTO (0.00),
HTML_SHORT_2 (0.00),
MID_RHS_MATCH_ENV_FROMTLD (0.00),
RCPT_COUNT_ONE (0.00),
REPLYTO_EQ_FROM (0.00),
RWL_MAILSPIKE_POSSIBLE (0.00),
SUBJECT_HAS_EXCLAIM (0.00),
TO_DN_NONE (0.00),
TO_MATCH_ENVRCPT_ALL (0.00),
RCVD_COUNT_ZERO (0.10),
RCVD_NO_TLS_LAST (0.10),
MISSING_CHARSET (0.50),
MV_CASE (0.50),
URI_COUNT_ODD (0.50),
CT_EXTRA_SEMI (1.00),
HELO_IPREV_MISMATCH (1.00),
HTML_SHORT_LINK_IMG_2 (1.00),
MID_RHS_MATCH_FROM (1.00),
RBL_SENDERSCORE_REPUT_6 (1.00),
RBL_VIRUSFREE_BOTNET (2.00),
RDNS_NONE (2.00),
RBL_SPAMCOP (4.00),
RBL_SPAMHAUS_SBL (4.00),
ABUSE_SURBL (5.00),
BAYES_SPAM (5.10),
DBL_SPAM (6.50),
URIBL_BLACK (7.50)
X-Spam-Status: No, score=3.62

[creativebastard]

Even using the LLM classifier it's not adding up.

X-Spam-LLM: HARMFUL (The email uses a deceptive health-related quiz subject line to lure clicks)
X-Spam-Result: DMARC_POLICY_ALLOW (-0.50),
DKIM_ALLOW (-0.20),
SPF_ALLOW (-0.20),
ARC_NA (0.00),
DKIM_SIGNED (0.00),
FROMTLD_EQ_ENV_FROMTLD (0.00),
FROM_HAS_DN (0.00),
HAS_EXTERNAL_IMG (0.00),
HAS_LINK_TO_LARGE_IMG (0.00),
HAS_REPLYTO (0.00),
HTML_SHORT_1 (0.00),
MID_RHS_MATCH_ENV_FROMTLD (0.00),
RCPT_COUNT_ONE (0.00),
REPLYTO_EQ_FROM (0.00),
TO_DN_NONE (0.00),
TO_MATCH_ENVRCPT_ALL (0.00),
RCVD_COUNT_ZERO (0.10),
RCVD_NO_TLS_LAST (0.10),
MISSING_CHARSET (0.50),
MV_CASE (0.50),
URI_COUNT_ODD (0.50),
CT_EXTRA_SEMI (1.00),
MID_RHS_MATCH_FROM (1.00),
PARTS_DIFFER (1.00),
SUBJECT_ENDS_QUESTION (1.00),
HTML_SHORT_LINK_IMG_1 (2.00),
RBL_MAILSPIKE_WORST (2.00),
RBL_SPAMHAUS_CSS (2.00),
RDNS_NONE (2.00),
LLM_HARMFUL_HIGH (3.00),
ABUSE_SURBL (5.00),
BAYES_SPAM (5.10),
DBL_SPAM (6.50),
URIBL_BLACK (7.50)
X-Spam-Status: No, score=2.08

[creativebastard]

X-Spam-LLM: HARMFUL (The email contains suspicious shortened URLs from an unknown domain (justlost.store) promising fuel cost savings without any legitimate company branding or contact information)
X-Spam-Result: BAYES_HAM (-3.00),
DMARC_POLICY_ALLOW (-0.50),
DKIM_ALLOW (-0.20),
SPF_ALLOW (-0.20),
ARC_NA (0.00),
DKIM_SIGNED (0.00),
FROMTLD_EQ_ENV_FROMTLD (0.00),
FROM_HAS_DN (0.00),
HAS_EXTERNAL_IMG (0.00),
HAS_LINK_TO_LARGE_IMG (0.00),
HAS_REPLYTO (0.00),
HTML_SHORT_2 (0.00),
MID_RHS_MATCH_ENV_FROMTLD (0.00),
RCPT_COUNT_ONE (0.00),
REPLYTO_EQ_FROM (0.00),
TO_DN_NONE (0.00),
TO_MATCH_ENVRCPT_ALL (0.00),
RCVD_COUNT_ZERO (0.10),
RCVD_NO_TLS_LAST (0.10),
MISSING_CHARSET (0.50),
MV_CASE (0.50),
RBL_SENDERSCORE_REPUT_7 (0.50),
URI_COUNT_ODD (0.50),
CT_EXTRA_SEMI (1.00),
HELO_IPREV_MISMATCH (1.00),
HTML_SHORT_LINK_IMG_2 (1.00),
MID_RHS_MATCH_FROM (1.00),
RBL_MAILSPIKE_WORST (2.00),
RBL_SPAMHAUS_CSS (2.00),
RBL_VIRUSFREE_BOTNET (2.00),
RDNS_NONE (2.00),
LLM_HARMFUL_HIGH (3.00),
SEM_URIBL_FRESH15 (3.00),
RBL_SPAMCOP (4.00),
ABUSE_SURBL (5.00),
DBL_SPAM (6.50),
URIBL_BLACK (7.50)
X-Spam-Status: No, score=3.79

[mdecimus]

Have you enabled reputation tracking? The spam filter currently does not include a header when there adjustments for reputation.

[creativebastard]

I have. Should I disable it?

[creativebastard]

Alright well that seems to have done the trick....

[mdecimus]

The reputation tracking is going to be reviewed/fixed in a few months so it does not perform incorrect adjustments and also logs any changes to the scores.

[creativebastard]

Awesome. How does the LLM classifier contribute to the score?

[mdecimus]

LLMs scores are added as a separate header.