BUG - ~60 threads in WAITING stage are remains when working with latest version of azure ( issue when azure-core-http-okhttp is 1.12.6 and up)

[ihudedi]

We are using azure version 1.2.31 (bom)
When we run multiple transfer from/to azure the number of azure thread remains: (see in jstack(
azure-sdk-global-thread-1
..
..
..
..
..
azure-sdk-global-thread-59

We are using new httpclient for every job we are running. (different clients and configuration)

When we use the previous version (1.2.23) - bom those threads were deleted after use.

Does the threads should be remains after actions finished or the thread pool is go away.

Is there an option to delete the connection pool after finish the action?
Today we don't set any ConnectionPool in our code.we use the default (should be set to 5 but we run in parallel multiple instances of httpclient)

Thanks.
Itay

[ihudedi]

Hi,
The issue occur in azure-core-http-okhttp version 1.12.6 and more
In version 1.12.5 all thread are released
I think it's because of this change mage in 1.12.6 - #42887
Thanks,
Itay

[ihudedi]

Hi @alzimmermsft
Could you please check if your fix cause this issue? #42887
Thanks,
Itay

[alzimmermsft]

Thanks for asking this @ihudedi!

First want to ask, is this a situation where the application is failing to run appropriately? Or, is this more a question around "why are there so many threads waiting with the names azure-sdk-global-thread-<number>"?

If it's the former, we'll need a reproduction of the issue.

If it's the latter, these threads should eventually be removed if they aren't being used any longer as all threads spawned by the default configuration of SharedExecutorService have a time-to-live. Why there are so many waiting, well they are just tracking if any part of the network request if blocked longer than we want to allow.

[ihudedi]

Hi @alzimmermsft
I am asking about the threads that are not being removed.in the former version of this jar azure-core-http-okhttp 1.12.5 all those threads were removed after they finished their action but from version 1.12.6 those threads are in waiting stage for a long time and I would like to know what was the change and why they don't removed.
Thanks,
Itay

[alzimmermsft]

@ihudedi, the change made between 1.12.5 and 1.12.6 transitioned azure-core-http-okhttp from using a single-thread Timer and TimerTasks to track whether a request timed out due to inactivity to using SharedExecutorService. This was done to resolve a bug where if too many requests were being tracked at once it could cause an OOME due to the way Timer removed cancelled tasks once their timeout period elapsed and not when the task was cancelled.

Why there all the azure-sdk-global-thread-* threads that are existing in the WAITING state. This is due to the change above where a shared ExecutorService (SharedExecutorService) is using a thread to track timing out. Once that thread completes tracking it is added back to its thread pool for further use and while waiting for re-use it's put into the WAITING state. Eventually, if unused the thread will be terminated, but if they keep being used they'll continually go in and out of the WAITING state.

[ihudedi]

Hi @alzimmermsft
In my env after all env in idle no job is running still see all thread for ever.
Thanks,
Itay

[alzimmermsft]

Thanks for the additional information @ihudedi. I made a very small sample application using SharedExecutorService directly:

public static void main(String[] args) throws InterruptedException {
    Collection<Callable<Void>> asyncCalls = IntStream.range(0, 60)
        .mapToObj(ignored -> (Callable<Void>) () -> null)
        .collect(Collectors.toList());

    List<Future<Void>> asyncResults = SharedExecutorService.getInstance().invokeAll(asyncCalls);

    for (Future<Void> asyncResult : asyncResults) {
        try {
            asyncResult.get();
        } catch (Exception e) {
            e.printStackTrace();
        }
    }

    ThreadMXBean threadMXBean = ManagementFactory.getThreadMXBean();
    ThreadInfo[] threadInfos = threadMXBean.getThreadInfo(threadMXBean.getAllThreadIds(), 100);

    for (ThreadInfo threadInfo : threadInfos) {
        System.out.println(threadInfo);
    }

    // Sleep for longer than the time for the thread pool to reclaim threads.
    Thread.sleep(120_000);

    threadInfos = threadMXBean.getThreadInfo(threadMXBean.getAllThreadIds(), 100);

    for (ThreadInfo threadInfo : threadInfos) {
        System.out.println(threadInfo);
    }
}

In the second round of ThreadInfo printing I saw that all the threads in the pool were terminated. During the first print I had many

"azure-sdk-global-thread-59" daemon prio=5 Id=102 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@11028347
	at java.base@21.0.6/jdk.internal.misc.Unsafe.park(Native Method)
	-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@11028347
	at java.base@21.0.6/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371)
	at java.base@21.0.6/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(AbstractQueuedSynchronizer.java:519)
	at java.base@21.0.6/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3780)
	at java.base@21.0.6/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3725)
	at java.base@21.0.6/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1712)
	at java.base@21.0.6/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1170)
	at java.base@21.0.6/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
	...

But in the second I had none. I'll expand this investigation out to using the OkHttpAsyncHttpClient we provide making calls to the Storage service and will report what I find shortly.

[ihudedi]

Hi @alzimmermsft
I found why the pool doesn't release the threads.in the sharedexecuterservice you create threadpool with coresize.the core size is depend number of processors ×10.if I set the system params to set the shared pool size to 0 all threads terminated.if not set all threads until core size remains in the pool
Thanks,
Itay

[alzimmermsft]

@ihudedi I tweaked my small sample application slightly and was able to replicate the behavior you're seeing.

public static void main(String[] args) throws InterruptedException {
    List<ScheduledFuture<Void>> asyncCalls = new ArrayList<>(60);
    for (int i = 0; i < 60; i++) {
        asyncCalls.add(SharedExecutorService.getInstance().schedule(() -> null, 1000, TimeUnit.MILLISECONDS));
    }

    for (Future<Void> asyncResult : asyncCalls) {
        try {
            asyncResult.get();
        } catch (Exception e) {
            e.printStackTrace();
        }
    }

    ThreadMXBean threadMXBean = ManagementFactory.getThreadMXBean();
    ThreadInfo[] threadInfos = threadMXBean.getThreadInfo(threadMXBean.getAllThreadIds(), 100);

    for (ThreadInfo threadInfo : threadInfos) {
        System.out.println(threadInfo);
    }

    // Sleep for longer than the time for the thread pool to reclaim threads.
    Thread.sleep(120_000);

    threadInfos = threadMXBean.getThreadInfo(threadMXBean.getAllThreadIds(), 100);

    for (ThreadInfo threadInfo : threadInfos) {
        System.out.println(threadInfo);
    }
}

Interestingly, using schedule instead of invoke results in the thread being maintained by the pool. I'll continue looking into this.

Fortunately, these threads shouldn't cause issue with shutting down of an application as the threads we create by default are daemon threads which won't block JVM shutdown.