[Coverity] Review kernel program cache locks

[AlexeySachkov]

You can see Coverity issues at https://scan.coverity.com/projects/intel-llvm?tab=overview

This report is motivated by 512040 - from what I understand, Coverity's concern is that here we acquire multiple locks and it may lead to a deadlock if there is another place where the same set (or a subset bigger than 1 lock) is acquired as well, but in a different order.

I don't know if we have a precedent for this, but the comment around the function says that it should only be used in unit tests.
That makes me think that it may not be ready for a production usage in a multi-threaded environment.

llvm/sycl/source/detail/kernel_program_cache.hpp

Lines 716 to 722 in 98cd464

	/// Clears cache state.
	///
	/// This member function should only be used in unit tests.
	void reset() {
	std::lock_guard<std::mutex> EvictionListLock(MProgramEvictionListMutex);
	std::lock_guard<std::mutex> L1(MProgramCacheMutex);
	std::lock_guard<std::mutex> L2(MKernelsPerProgramCacheMutex);

However, it is used several times in the SYCL RT:

llvm/sycl/source/detail/kernel_program_cache.hpp

Line 811 in 98cd464

reset();

llvm/sycl/source/detail/program_manager/program_manager.cpp

Line 1813 in 98cd464

Context.getKernelProgramCache().reset();

llvm/sycl/source/detail/program_manager/program_manager.cpp

Line 3008 in 98cd464

ContextImpl.getKernelProgramCache().reset();

I think that we should clarify the comment and double-check if a deadlock Coverity is talking about is indeed possible or not

[KseniyaTikhomirova]

we have fixed something similar before https://github.com/intel/llvm/pull/16359/files#diff-99114f8fd849025a6f86e930779900a3458758d8f5dd38e4a9bfbe21af7af015.
I checked the code and I think the problem is real: e.g. when out of memory error is reported during program build https://github.com/intel/llvm/blob/sycl/sycl/source/detail/kernel_program_cache.hpp#L811 and removeImages is called at kernel bundle destruction that ends in https://github.com/intel/llvm/blob/sycl/sycl/source/detail/kernel_program_cache.hpp#L828 simultaneously.
Rare case but still possible. Will assign to runtime team.