Optimize compilation phase for codec registration checker

[PawelLipski]

Time measurement of compilation phases on Hydra (commercial) project, with -Ystatistics flag:

[info]   parser                      : 1 spans, ()473.728ms (0.2%)
[info]   namer                       : 1 spans, ()118.276ms (0.0%)
[info]   packageobjects              : 1 spans, ()1.863ms (0.0%)
[info]   typer                       : 1 spans, ()124027.494ms (52.1%)
[info]   semanticdb-typer            : 1 spans, ()10044.861ms (4.2%)
[info]   dump-persistence-schema     : 1 spans, ()844.721ms (0.4%)
[info]   superaccessors              : 1 spans, ()353.506ms (0.1%)
[info]   extmethods                  : 1 spans, ()74.814ms (0.0%)
[info]   pickler                     : 1 spans, ()150.427ms (0.1%)
[info]   xsbt-api                    : 1 spans, ()1645.657ms (0.7%)
[info]   xsbt-dependency             : 1 spans, ()1767.425ms (0.7%)
[info]   refchecks                   : 1 spans, ()7264.879ms (3.1%)
[info]   codec-registration-class-sweep: 1 spans, ()129.464ms (0.1%)
[info]   patmat                      : 1 spans, ()11113.825ms (4.7%)
[info]   serializability-checker     : 1 spans, ()662.502ms (0.3%)
[info]   codec-registration-serializer-check: 1 spans, ()38826.128ms (16.3%)
[info]   uncurry                     : 1 spans, ()7006.531ms (2.9%)
[info]   fields                      : 1 spans, ()5867.108ms (2.5%)
[info]   tailcalls                   : 1 spans, ()352.656ms (0.1%)
[info]   specialize                  : 1 spans, ()954.033ms (0.4%)
[info]   explicitouter               : 1 spans, ()1027.995ms (0.4%)
[info]   erasure                     : 1 spans, ()10823.126ms (4.6%)
[info]   posterasure                 : 1 spans, ()684.086ms (0.3%)
[info]   lambdalift                  : 1 spans, ()1444.621ms (0.6%)
[info]   constructors                : 1 spans, ()496.269ms (0.2%)
[info]   flatten                     : 1 spans, ()535.716ms (0.2%)
[info]   mixin                       : 1 spans, ()675.139ms (0.3%)
[info]   cleanup                     : 1 spans, ()589.357ms (0.2%)
[info]   delambdafy                  : 1 spans, ()1370.026ms (0.6%)
[info]   jvm                         : 1 spans, ()8177.186ms (3.4%)
[info]   semanticdb-jvm              : 1 spans, ()13.902ms (0.0%)
[info]   xsbt-analyzer               : 1 spans, ()168.936ms (0.1%)

or just including the phases added by ash:

[info]   dump-persistence-schema     : 1 spans, ()844.721ms (0.4%)
[info]   codec-registration-class-sweep: 1 spans, ()129.464ms (0.1%)
[info]   serializability-checker     : 1 spans, ()662.502ms (0.3%)
[info]   codec-registration-serializer-check: 1 spans, ()38826.128ms (16.3%)

To be researched what specifically in org.virtuslab.ash.SerializerCheckCompilerPluginComponent#newPhase makes it so slow...
probably b/c it checks for every combination of types discovered in the @Serializer-annotated class and the types detected in the preceding codec-registration-class-sweep phase

[MarconZet]

This problem will likely be solved by replacing some Lists with Sets.

[LukaszKontowski]

Results from my compilation statistics for our phases (slightly different from PawelLipski's snippet):

[info] dump-persistence-schema : 1 spans, ()2651.299ms (2.4%)
[info] codec-registration-class-sweep: 1 spans, ()46.157ms (0.0%)
[info] serializability-checker : 1 spans, ()255.628ms (0.2%)
[info] codec-registration-serializer-check: 1 spans, ()12359.217ms (11.4%)

And flamegraph generated by https://github.com/jvm-profiling-tools/async-profiler profiler on Hydra - in attachment. Potentially correlated fact is that:

1. for dump-persistence-schema phase's (2,4% compilation time)newPhase - there were 160 samples ddetected by profiler
2. for codec-registration-serializer-check (11,4% compilation time) phase's newPhase - there were 955 such samples

So, it might be possible, that this long time is not caused by some lack of optimization inside SerializerCheckCompilerPluginComponent#newPhase (tried a lot of things to optimize it more, but failed to shorten it's duration). The problem might be caused by the fact, that this SerializerCheckCompilerPluginComponent#newPhase method gets invoked round 6 * more times than similar method from dump-persistence-schema phase. (this last finding is not true)

[PawelLipski]

The problem might be caused by the fact, that this SerializerCheckCompilerPluginComponent#newPhase method gets invoked round 6 * more times than similar method from dump-persistence-schema phase.

Wow, that's a non-trivial discovery indeed 🤔

[LukaszKontowski]

Sorry, in my last comment I was wrong - SerializerCheckCompilerPluginComponent#newPhase gets invoked only one time and creates one instance of StdPhase type, but - underlying StdPhase#apply method gets invoked hundreds of times - e.g. 541 times (from my last test).
For now, I do not know yet, why it gets invoked so many times.

[PawelLipski]

Reopening as @LukaszKontowski's research is still ongoing