More actuator optimizations

[sdeleuze]

Despite our effort to modularize and optimize actuator hints in our 0.8.0 milestone, adding actuator on a Spring Boot + Spring MVC @Controller + Tomcat + Jackson increase RSS memory by 34M of RSS (97M with, 63M without). This is a lot especially when we think that most users just want their default web endpoints.

I think there are multiple things to explore here in order to find the best way how to optimize this:

• @dsyer How much using spring-init functional configuration decrease the footprint? How much remaining hints required after?
• @bclozel What is the footprint gain with your functional actuator branch? How much remaining hints when used with spring-init?
• @snicoll @wilkinsona Could it be possible to ship less infra on Spring Boot side by default?

[dsyer]

The spring-init sample with functional actuators with has only /health and /info (but adding more might not impact the footprint much). The footprint is currently around 40MB (cf 36MB for plain webflux). Some of the difference is probably Jackson - those would be the hints required (see reflect-config.json for details).

[sdeleuze]

See related spring-projects/spring-boot#20290 issue.

[sdeleuze]

The first step is to review this diff of used classes between a basic Spring MVC app and the same with the actuator starter.
diff.txt

There are 3 levels where we could do something:

• Fine tune the hints declarations, especially the access attribute
• Fine tune the hint engine with @aclement
• Contact Boot team to ask specific changes if we identify specific points they could change to make things easier here

[aclement]

First round of optimizations have gone in purely related to hint adjustment and smarter feature analysis, I wanted to share the results. I have a variant of the actuator-webflux sample. I've added a single / endpoint so I can exercise the app and the actuator separately.

Initial build with the starter for actuator removed from the pom to give us the baseline. (Macbook Pro)

Exercising the `/` endpoint:                  RSS=49.5m/ImageSize=43.3m

Now with the actuator added back to the pom.xml

Exercising `/` but not the health endpoint:   RSS=71.1m/ImageSize=52.1m
Exercising `/` *and* the health endpoint:     RSS=73.3m/ImageSize=52.1m

Recompiling with evaluate-cop=true set - this evaluates conditional on property checks at image build time, throwing more away as the properties are not typically set.

Exercising `/` but not the health endpoint:   RSS=69.1m/ImageSize=51.1m
Exercising `/` *and* the health endpoint:     RSS=71.4m/ImageSize=51.1m

Very roughly speaking if the observation in the first comment is that actuator had a 34M impact on RSS, we have reduced that to ~22M with a first pass. (I recognize the actuator-webflux sample is a little different to the original app). And down to ~20M with evaluate-cop=true.

We should probably review what is in the latter image to see if anything is obvious that doesn't need to be there.

[sdeleuze]

@aclement Could we compare the additional classes loaded by the JVM with actuator + exercising the /health endpoint with the one added to the native image? That would allow use to check with Boot team if there are or not optimization possible on the JVM and if we ship too much on native.

[aclement]

Now we can see the wood for the trees after the first round of optimizations, I was able to dig a little deeper. As I commented before reviewing the difference between a pure app with no actuator with actuator added and only wanting to exercise /health. Using the diff tool we have between the -H:+PrintAOTCompilation output in each case. There are 1623 extra things added to the actuator app. Looking through I could see Cloud related info and that is because we don't eagerly evaluate the @ConditionalOnCloudPlatform condition - so the ReactiveCloudFoundryActuatorAutoConfiguration kicks in. Yes it is additionally guarded with:

@ConditionalOnProperty(prefix = "management.cloudfoundry", name = "enabled", matchIfMissing = true)

But that matchIfMissing=true means even if you do eagerly evaluate conditional on property the fact that this matches if no property is specified means we aren't deactivating these things.

So a crude hack to do eager evaluation of ConditionalOnCloudPlatform. The changes are from:

Entries in the PrintAOTCompilation diff: 1623
Build memory: 6.42GB
Image build time: 114.3s
RSS memory: 71.4M
Image size: 51.2M
Startup time: 0.113 (JVM running for 0.115)

to:

Entries in the PrintAOTCompilation diff: 1215
Build memory: 6.97GB
Image build time: 102.2s
RSS memory: 70.4M
Image size: 48.6M
Startup time: 0.121 (JVM running for 0.123)

A meg saved in RSS. I also noticed a lot of micrometer stuff in there. I excluded micrometer in the pom (from the actuator dependency) to see the impact it would have:

Entries in the PrintAOTCompilation diff: 860
Build memory: 7.04GB
Image build time: 106.2s
RSS memory: 63.9M
Image size: 47.1M
Startup time: 0.09 (JVM running for 0.092)

Notice the giant decrease in RSS in this case. I suspect something similar could be achieved via not excluding micrometer as a dependency but evaluating conditions like ConditionalOnEnabledMetricsExport more eagerly.

[aclement]

Eager evaluation of ConditionalOnEnabledMetricsExport doesn't seem to be sufficient - it helps but there is still a lot of metrics stuff included - there seem to be other routes to creating meter registry related beans that are not guarded with anything beyond class path checks. For example:

@Configuration(proxyBeanMethods = false)
@ConditionalOnBean(Clock.class)
@ConditionalOnMissingBean(MeterRegistry.class)
class NoOpMeterRegistryConfiguration {

[dsyer]

Just checking: eager evaluation of @ConditionalOnCloudPlatform is a bad idea right? It's just to collect a data point?

[sdeleuze]

After more discussions with @wilkinsona and @dsyer I think the solution is the following:

• Introduce a spring.native.remove-metrics-support flag (Boot team is not ok to add such flag on their side because there is a clear classpath signal via the io.micrometer:micrometer-core dependency) that would discard metrics related beans (and do not add the related reflection entries) when set to true (default would be false)
• Introduce a build time transformation that set spring.native.remove-metrics-support to true when spring.native.remove-jmx-support=true and (management.endpoint.metrics.enabled=false or management.endpoints.web.exposure.include does not include metrics). It would remove metrics by default since the endpoint is not exposed by default.

@ConditionalOnCloudPlatform seems to be mostly Cloud Foundry specific right now and I am not sure this is a pattern we would like to increase. I am even wondering if we could think about removing it in Boot 3 timeframe. For now I would suggest we set management.cloudfoundry.enabled=false by default in spring-gralvm-native and document it, that will allow people to enable it if needed at build time.

[snicoll]

In Spring Boot 2.4 there is management.metrics.export.defaults.enabled=false to disable all known metric exporters. That does not prevent the metrics infrastructure to be auto-configured though as you can still opt-in for a given exporter. For instance adding management.metrics.export.simple.enabled=true alongside the previous property would disable all available metrics exporters but the simple in-memory one.

That's what we do to disable metrics export in Integration tests.

[wilkinsona]

Introduce a build time transformation that set spring.native.remove-metrics-support to true when spring.native.remove-jmx-support=true and (management.endpoint.metrics.enabled=false or management.endpoints.web.exposure.include does not include metrics). It would remove metrics by default since the endpoint is not exposed by default.

I'm not sure that this makes sense. If you're exporting metrics out-of-process – to Prometheus or whatever – you will care about metrics but in all likelihood won't have the metrics endpoint enabled.