Parallelize and optimize dwarf body-generation setup (bit-identical results)#238
Open
IanSteveC wants to merge 1 commit into
Open
Parallelize and optimize dwarf body-generation setup (bit-identical results)#238IanSteveC wants to merge 1 commit into
IanSteveC wants to merge 1 commit into
Conversation
…ults) The v1.96 cored/NFW-with-rcut dwarf has no analytic distribution function, so body generation rejection-samples against numerical gauss_quad integrals. Profiling shows dist_fun (one gauss_quad per call) is essentially 100% of the single-threaded setup phase: ~10-16 minutes per 40k-body workunit before any simulation starts. Three changes, all provably bit-neutral (identical operations in the identical order; only scheduling/reuse changes): 1. gauss_quad: enumerate the integration nodes first (verbatim original arithmetic), evaluate the pure integrand at all ~330 nodes in parallel with OpenMP (honors --nthreads / -n), then accumulate the terms in the original sequential order. Results are identical for any thread count. 2. Hoist the constant prefix 4*pi*pcut*rcut^-delta*e^(rcut/rdecay)* rdecay^(delta+3) into the Dwarf struct (computed once in set_model_params as the same left-associated product) instead of re-evaluating three transcendentals per integrand call. 3. Memoize gammln(z) with a thread-local single-entry cache: it is called millions of times with only ~2 distinct arguments (delta+2, delta+3). Measured on a 40k-body v1.96 workunit (Ryzen, --nthreads 4): setup 985s -> 244s (4.0x); 16 threads: 108s (9.1x). Full-workunit results bit-identical at 1, 4, and 16 threads (all 8 likelihood metrics).
Contributor
|
The ST phase currently (1.96) accounts for a significant portion of the total task runtime, making the use of MT impractical. Parallelizing this phase helps a lot. @kgimmy thanks for considering it! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Parallelize and optimize dwarf body-generation setup (bit-identical results)
Motivation
The v1.96 cored/NFW-with-rcut dwarf has no analytic distribution function, so body generation rejection-samples against numerical
gauss_quadintegrals. Profiling showsdist_fun(onegauss_quadper call) is essentially 100% of the setup phase, which runs single-threaded for ~10-16 minutes on a 40k-body workunit before the simulation starts.Changes (all bit-neutral: identical operations in identical order, only scheduling/reuse changes)
gauss_quad: enumerate the integration nodes first (verbatim original arithmetic), evaluate the pure integrand at all ~330 nodes in parallel with OpenMP (honors-n/--nthreads), then accumulate the terms in the original sequential order. Results are identical for any thread count.4π·pcut·rcut^-δ·e^(rcut/rdecay)·rdecay^(δ+3)is computed once per component inset_model_params(same left-associated product) instead of re-evaluating three crlibm transcendentals per integrand call in ther > rcutbranches.gammlnmemo: thread-local single-entry cache; it is called millions of times with only ~2 distinct arguments (delta+2,delta+3).Verification
Full 40k-body v1.96 workunit: all 8
<search_likelihood*>metrics bit-identical to the unmodified build at--nthreads1, 4, and 16.Measured (40k v1.96 workunit)
Diff is 4 files, +68/−13.