Skip to content

backfill: recover never-bridged messages in forward backfill#286

Draft
Fizzadar wants to merge 1 commit into
mainfrom
nick/plat-36990-backfill-id-dedup
Draft

backfill: recover never-bridged messages in forward backfill#286
Fizzadar wants to merge 1 commit into
mainfrom
nick/plat-36990-backfill-id-dedup

Conversation

@Fizzadar

@Fizzadar Fizzadar commented Jun 16, 2026

Copy link
Copy Markdown
Member

Problem

Messenger messages that arrive while the bridge is offline can be permanently lost. A reconnect snapshot advances the Meta sync cursor past skipped threads, so their mid-timeline messages are never bridged. A later forward backfill covering the gap then dropped those messages in because ts <= anchor.Timestamp, so the gap was lost forever.

Change

  • Remove the forward timestamp DeleteFunc(ts <= anchor) in wrapBackfillEvents.
  • Set AggressiveDeduplication: forward on the returned FetchMessagesResponse so bridgev2 deduplicates by message ID — dropping already-bridged messages while retaining never-bridged ones older than the anchor.
  • Keep the backward timestamp trim unchanged.

Regression risk (please scrutinize)

The removed forward timestamp trim was an intentional dedup guard. Dedup now relies on bridgev2's ID-based pass. Meta message IDs are deterministic from the network MessageId and the live and backfill paths store under the same ID, so already-bridged messages are still found and dropped; intra-batch dupes are handled by the existing CompactFunc; in-flight messages are covered by the TxnID pending check. Assessed acceptable (high confidence), but maintainer review of the dedup assumption is requested. New minor cost: one indexed GetFirstPartByID lookup per forward-backfill message.

Cross-repo dependency

Requires mautrix/go#517 (skips the forward timestamp prefix-trim in cutoffMessages when AggressiveDeduplication is set). Without that change this PR has no effect, since bridgev2 would re-trim the retained messages by timestamp. Merge order: mautrix/go first, then bump the dependency here.

🤖 Generated with Claude Code

Messenger messages that arrived while the bridge was offline could be permanently lost: a reconnect snapshot advanced the sync cursor past skipped threads, and forward backfill then dropped those never-bridged mid-timeline messages purely because their timestamp was older than the portal anchor.

Remove the forward timestamp trim and set AggressiveDeduplication on forward responses so bridgev2 deduplicates by message ID, dropping already-bridged messages while retaining never-bridged ones. Keep the backward timestamp trim unchanged.
@Fizzadar Fizzadar changed the title PLAT-36990: recover never-bridged messages in forward backfill backfill: recover never-bridged messages in forward backfill Jun 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant