backfill: recover never-bridged messages in forward backfill#286
Draft
Fizzadar wants to merge 1 commit into
Draft
backfill: recover never-bridged messages in forward backfill#286Fizzadar wants to merge 1 commit into
Fizzadar wants to merge 1 commit into
Conversation
Messenger messages that arrived while the bridge was offline could be permanently lost: a reconnect snapshot advanced the sync cursor past skipped threads, and forward backfill then dropped those never-bridged mid-timeline messages purely because their timestamp was older than the portal anchor. Remove the forward timestamp trim and set AggressiveDeduplication on forward responses so bridgev2 deduplicates by message ID, dropping already-bridged messages while retaining never-bridged ones. Keep the backward timestamp trim unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Messenger messages that arrive while the bridge is offline can be permanently lost. A reconnect snapshot advances the Meta sync cursor past skipped threads, so their mid-timeline messages are never bridged. A later forward backfill covering the gap then dropped those messages in because
ts <= anchor.Timestamp, so the gap was lost forever.Change
DeleteFunc(ts <= anchor)inwrapBackfillEvents.AggressiveDeduplication: forwardon the returnedFetchMessagesResponseso bridgev2 deduplicates by message ID — dropping already-bridged messages while retaining never-bridged ones older than the anchor.Regression risk (please scrutinize)
The removed forward timestamp trim was an intentional dedup guard. Dedup now relies on bridgev2's ID-based pass. Meta message IDs are deterministic from the network
MessageIdand the live and backfill paths store under the same ID, so already-bridged messages are still found and dropped; intra-batch dupes are handled by the existingCompactFunc; in-flight messages are covered by the TxnID pending check. Assessed acceptable (high confidence), but maintainer review of the dedup assumption is requested. New minor cost: one indexedGetFirstPartByIDlookup per forward-backfill message.Cross-repo dependency
Requires mautrix/go#517 (skips the forward timestamp prefix-trim in
cutoffMessageswhenAggressiveDeduplicationis set). Without that change this PR has no effect, since bridgev2 would re-trim the retained messages by timestamp. Merge order: mautrix/go first, then bump the dependency here.🤖 Generated with Claude Code