Quarkus 3.32.1 has config classloader issues when deployed to AWS ECS

[julianhowarth]

Describe the bug

Our application no longer starts up on AWS ECS after upgrading from 3.31.4 to 3.32.1

We have a Java 21 application built into a docker image via JIB and deployed via CI to AWS ECS Fargate. Amongst other things, it has a component that attempts to periodically read data from AWS S3. This component is configured via config properties and has not been changed for many months:

  @ApplicationScoped
  S3ReplicaPoller<GeoTemplate> geoTemplateReplicaPoller(
      @ConfigProperty(name = "replication.consumer.geotemplate_legacy.bucket") String bucket,
      @ConfigProperty(name = "replication.consumer.geotemplate_legacy.replicaTypeName")
          String replicaTypeName,
      S3AsyncClient s3AsyncClient) {
  ...
}

After updating Quarkus to 3.32.1:

• our integration tests using localstack run locally, and all pass (exercising this component)
• running the application locally (in IntelliJ) but configured to point to AWS S3 buckets works correctly
• when the application is deployed to AWS ECS the application fails to start due to java.lang.IllegalArgumentException: SRCFG00015: No configuration is available for this class loader

Expected behavior

The application should run normally with no exceptions due to configuration exceptions. (And this is the behaviour when using 3.31.4)

Actual behavior

We hit config exceptions which prevent startup of the application. we see the following stacktrace when this component is called:

java.lang.IllegalArgumentException: SRCFG00015: No configuration is available for this class loader
	at io.smallrye.config.SmallRyeConfigProviderResolver.get(SmallRyeConfigProviderResolver.java:110)
	at io.smallrye.config.Config.get(Config.java:1211)
	at io.smallrye.config.inject.ConfigProducer.getConfig(ConfigProducer.java:44)
	at io.smallrye.config.inject.ConfigProducer.produceStringConfigProperty(ConfigProducer.java:51)
	at io.smallrye.config.inject.ConfigProducer_Subclass.produceStringConfigProperty$$superforward(Unknown Source)
	at io.smallrye.config.inject.ConfigProducer_Subclass$9.apply(Unknown Source)
	at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:73)
	at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:62)
	at io.quarkus.arc.runtime.ConfigStaticInitCheckInterceptor.aroundInvoke(ConfigStaticInitCheckInterceptor.java:47)
	at io.quarkus.arc.runtime.ConfigStaticInitCheckInterceptor_Bean.intercept(Unknown Source)
	at io.quarkus.arc.impl.InterceptorInvocation.invoke(InterceptorInvocation.java:42)
	at io.quarkus.arc.impl.AroundInvokeInvocationContext.perform(AroundInvokeInvocationContext.java:30)
	at io.quarkus.arc.impl.InvocationContexts.performAroundInvoke(InvocationContexts.java:27)
	at io.smallrye.config.inject.ConfigProducer_Subclass.produceStringConfigProperty(Unknown Source)
	at io.smallrye.config.inject.ConfigProducer_ProducerMethod_produceStringConfigProperty_Slb1-DO4BbQxgDjjOw2BuNv12-I_Bean.create(Unknown Source)
	at io.smallrye.config.inject.ConfigProducer_ProducerMethod_produceStringConfigProperty_Slb1-DO4BbQxgDjjOw2BuNv12-I_Bean.get(Unknown Source)
	at io.smallrye.config.inject.ConfigProducer_ProducerMethod_produceStringConfigProperty_Slb1-DO4BbQxgDjjOw2BuNv12-I_Bean.get(Unknown Source)
	at io.quarkus.arc.impl.CurrentInjectionPointProvider.get(CurrentInjectionPointProvider.java:44)
	at com.endeavorstreaming.vesper.content_engine.loader.replication.ReplicationProducers_ProducerMethod_geoRestrictionTemplatesReplicaConsumer_2wGmyNxQallwkY5iO0IDWj-7Cdo_Bean.create(Unknown Source)
	at com.endeavorstreaming.vesper.content_engine.loader.replication.ReplicationProducers_ProducerMethod_geoRestrictionTemplatesReplicaConsumer_2wGmyNxQallwkY5iO0IDWj-7Cdo_Bean.create(Unknown Source)
	at io.quarkus.arc.impl.AbstractSharedContext.createInstanceHandle(AbstractSharedContext.java:119)
	at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:38)
	at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:35)
	at io.quarkus.arc.generator.Default_jakarta_enterprise_context_ApplicationScoped_ContextInstances.ch10(Unknown Source)
	at io.quarkus.arc.generator.Default_jakarta_enterprise_context_ApplicationScoped_ContextInstances.computeIfAbsent(Unknown Source)
	at io.quarkus.arc.impl.AbstractSharedContext.get(AbstractSharedContext.java:35)
	at io.quarkus.arc.impl.ClientProxies.getApplicationScopedDelegate(ClientProxies.java:23)
	at technology.dice.commons.datareplication.ReplicationProducers_ProducerMethod_geoRestrictionTemplatesReplicaConsumer_2wGmyNxQallwkY5iO0IDWj-7Cdo_ClientProxy.arc$delegate(Unknown Source)
	at technology.dice.commons.datareplication.ReplicationProducers_ProducerMethod_geoRestrictionTemplatesReplicaConsumer_2wGmyNxQallwkY5iO0IDWj-7Cdo_ClientProxy.consume(Unknown Source)
	at technology.dice.commons.datareplication.S3ReplicaPoller.lambda$poll$7(S3ReplicaPoller.java:71)
	at java.base/java.util.Optional.map(Optional.java:260)
	at technology.dice.commons.datareplication.S3ReplicaPoller.lambda$poll$8(S3ReplicaPoller.java:68)
	at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1150)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncApiCallMetricCollectionStage.lambda$execute$0(AsyncApiCallMetricCollectionStage.java:58)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncApiCallTimeoutTrackingStage.lambda$execute$2(AsyncApiCallTimeoutTrackingStage.java:69)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.lambda$attemptExecute$1(AsyncRetryableStage.java:132)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$execute$0(MakeAsyncHttpRequestStage.java:110)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.completeResponseFuture(MakeAsyncHttpRequestStage.java:253)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$executeHttpRequest$3(MakeAsyncHttpRequestStage.java:167)
	at java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:934)
	at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:911)
	at java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:482)
	at io.quarkus.vertx.core.runtime.VertxCoreRecorder$15.runWith(VertxCoreRecorder.java:682)
	at org.jboss.threads.EnhancedQueueExecutor$Task.doRunWith(EnhancedQueueExecutor.java:2651)
	at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2630)
	at org.jboss.threads.EnhancedQueueExecutor.runThreadBody(EnhancedQueueExecutor.java:1622)
	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1589)
	at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:11)
	at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:11)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:1583)

How to Reproduce?

No reproducer yet as it's not clear why it is failing

Output of uname -a or ver

amazonlinux:2023.10.20260216.1-minimal

Output of java -version

OpenJDK 64-Bit Server VM Corretto-21.0.9.11.1 (build 21.0.9+11-LTS, mixed mode, sharing)

Quarkus version or git rev

3.32.1

Build tool (ie. output of mvnw --version or gradlew --version)

maven 3.8.7

Additional information

No response

[quarkus-bot]

/cc @edeandrea (container-image), @geoand (container-image), @radcortez (config)

[gsmet]

The application runs locally in Prod mode?

Do you use the same Corretto JVM?

@radcortez we should probably add the class loader toString to the error message

[radcortez]

I'll improve the error message.

I'll see if I can figure out the issue.

[gsmet]

What I was wondering is if the async stack could play a role. As for TCCL not being properly set up for instance.

[julianhowarth]

The application runs locally in Prod mode?

I have set quarkus.profile = prod, which I believe achieves this?

Do you use the same Corretto JVM?

locally I'm on Corretto 21.08 rather than 21.09 but still 64-bit on linux

[geoand]

Have you been able to reproduce this locally at all?

[julianhowarth]

Have you been able to reproduce this locally at all?

Not yet, but let me try it running it via docker

[julianhowarth]

Have you been able to reproduce this locally at all?

Not yet, but let me try it running it via docker

So building a docker image, then running it, I get the same IllegalArgumentException as per the original description

[radcortez]

Any chance you can share with us the reproducer? Thank you.

[julianhowarth]

Any chance you can share with us the reproducer? Thank you.

I don't have a reproducer, just our whole 100k+ line Java application which also includes multiple internal libs. So there's no easy way to cut it down to get a simple reproducer.

Equally without knowing roughly what is likely going wrong, it's difficult to know what to build as a simple app starting from scratch. For instance, it's not as straightforward as building an app with AWS S3 access configured via config properties as we have other S3 accesses within the app which are working without issue (along with DDB too).

I have now built the app into another docker image using Quarkus 3.31.4 and can confirm that that version runs correctly locally (as expected).

[radcortez]

Equally without knowing roughly what is likely going wrong, it's difficult to know what to build as a simple app starting from scratch. For instance, it's not as straightforward as building an app with AWS S3 access configured via config properties as we have other S3 accesses within the app which are working without issue (along with DDB too).

Do you mean that you have other applications that use the same pattern with Quarkus 3.32.x and are working without issues?

[radcortez]

Can you share the code executed at:

at technology.dice.commons.datareplication.S3ReplicaPoller.lambda$poll$8(S3ReplicaPoller.java:68)