Implement in-place seed for assembly pipeline#323
Open
folbricht wants to merge 16 commits into
Open
Conversation
Introduce AssemblePlan that separates planning from execution in file assembly. The plan pre-computes all chunk placements (self-seed, file seeds, store fetches, skip-in-place) into a DAG of steps with explicit dependencies, replacing the interleaved sequencer approach. This lays the groundwork for #312 (destination-as-seed) by making assembly sources composable and the planning phase extensible. Key changes: - New AssemblePlan with functional options and step-based execution - Split assembly sources into separate files (fileseed, selfseed, store, skip) - Self-seed matching now uses longestMatchFrom for longer sequences - Plan validation detects stale file seeds before execution - Comprehensive tests for plan generation and in-place detection - Remove sequencer.go, selfseed.go in favor of new plan types Closes #312
- Replace deprecated os.SEEK_SET with io.SeekStart in fileSeedSegment.copy - Fix stale NewIndexSeed doc comment on NewFileSeed - Update test comment referring to the removed writeChunk function - Use the min builtin instead of shadowing max in selfSeed matching - Drop the never-failing error return from AssemblePlan.generate - Return only the match count from maxMatchFrom; the position is the caller's own argument - Extract a newSeed helper in the CLI for the in-place-vs-file seed decision duplicated between readSeeds and readSeedDirs - Remove Size from the SeedSegment interface; its last interface-level caller went away with the sequencer
LongestMatchFrom returned a 4-tuple of byte and chunk coordinates of which every caller used a different half: the computed byte length had no callers at all, the null seed had to fabricate meaningless offsets, and FileSeed.GetSegment re-derived the chunk range from the byte offset with a binary search even though LongestMatchFrom had just computed it. LongestMatchFrom now returns only (pos, n) in chunk coordinates and GetSegment takes the same coordinates, slicing the seed index directly. The nullChunkSection no longer tracks a size; WriteInto already receives the length to write.
- Extract a shared seedIndex type (index + position lookup + matching kernel) used by both FileSeed and selfSeed, replacing two near- identical LongestMatchFrom implementations and three copies of the position-map construction. The kernel resolves the tie-break drift between the two copies: later candidates now win ties for both seeds, which for file seeds picks a byte-identical range - Extract claimRun for the run-claiming loop that was duplicated between the self-seed and file-seed passes in generate(), and chunkRangeLength for the byte-length expression repeated across the plan and segment code - Move fileSeedSegment's copy/clone to package-level copyRange and cloneOrCopyRange, and use them from selfSeedSegment.Execute too. The self-seed previously called CloneRange directly for its whole range, without the unaligned head/tail handling or the copy fallback that cloneOrCopyRange has, so cloning failed on unaligned segments or filesystems like ZFS - The self-seed now opens its read handle per segment execution, the same pattern fileSeedSegment uses for its seed file. That removes the reader-handle pool and with it the entire Close lifecycle on selfSeed and AssemblePlan as well as the error return of NewPlan - Share a chunkInPlace helper between the two seed validation paths and the generateSkips scan - Drop the unused needValidation field from fileSeedSegment
- Exclude the null-chunk ID from the self-seed lookup. The self-seed pass runs before the seeds are matched, so runs of zero chunks were claimed as chained self-copies: real read/write I/O serialized by dependencies, allocating disk blocks on sparse targets. The null seed handles them as no-ops on blank targets and without dependencies - Merge generateInPlace's two classification passes into one and find in-place chunks with a binary search over the (ascending) source offsets instead of scanning every occurrence, which was quadratic for chunks that repeat often in the seed - Compute the target blocksize once per plan instead of calling stat through blocksizeOfFile on every seed step execution - Run generateSkips on a fixed pool of workers with a reused read buffer per worker, instead of spawning one goroutine and one buffer allocation per chunk - Bound self-seed matching: the overlap clamp is now applied as the match limit, which also bounds the work done per candidate, and at most 100 candidate positions are examined per match, mirroring the 100-chunk run limit that already applies without reflinks
- Pass the in-place seed to the plan explicitly via PlanWithInPlaceSeed instead of fishing it out of the generic seed list with type assertions in AssembleFile and twice in plan generation. AssembleFile partitions the seeds once - Move chunk accounting onto the assembleSource interface as recordStats, replacing the worker's type switch over all six concrete source types, which new sources would silently fall out of - Replace the File() == "" sentinel in plan validation with an explicit needsValidation method on assembleSeedSource - Replace the stepSet wrapper with a plain map and a link helper that sets both directions of a dependency edge, halving each linking site - Sort SCCs with SortStableFunc and a key closure instead of three auxiliary index slices - Replace the backward goto in AssembleFile's plan validation with a retry loop
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Refactors the assembly pipeline into a plan-based architecture and adds in-place seed support, enabling efficient file reconstruction when the target file already contains chunk data at different offsets.
AssemblePlanthat pre-computes all chunk placements into a DAG of steps with explicit dependencies, replacing the interleaved sequencer approachInPlaceSeedwhich rearranges chunks already present in the target file using Tarjan's SCC algorithm for cycle detection and buffer-break resolutionskipInPlace,inPlaceCopy,fileSeedSource,selfSeedSegment,copyFromStorelongestMatchFromfor longer contiguous sequencesCloses #312