[SYCL] Reduce urKernelRetain, Release calls when not using kernel bundle or RTC

[uditagarwal97]

Problem
Every time we fetch a kernel/program from in-memory cache, we make a call to urKernelRetain/urKernelRelease/urProgramRelease/urProgramRetain. These calls are expensive.

Solution (Proposed in this PR)

Case 1 (When in-memory cache is enabled):

Instead of a shared ownership, transfer the ownership of kernel/program handles to the cache. Cache manages the lifetime of program/kernel handle and so there are no more calls to urKernelRetain/Release when fetching items from cache. During program shutdown, cache will be destroyed, and the kernel, program objects it holds will be also released.

• (a) Caveat: (When eviction is used or cache has to be cleared due to OOM)
  Consider the case when one thread is using the kernel/program handle fetched from cache and another thread just evicted that item from cache, perhaps due to OOM. In this case, we might run into a use-after-free bug. To prevent that, this PR, guards cache access with a reader, writer lock. Threads fetching from cache will hold a reader lock which will be release when kernel/program handles are no longer needed. Thread trying to evict from cache will hold a writer lock.

Case 2 (When in-memory cache is explicitly disabled):

When cache is disabled, there no transfer of ownership to cache (obviously) and the code will work just like it works right now.

[uditagarwal97]

@Alexandr-Konovalov FYI. PR is ready for review.

[uditagarwal97]

Performance: https://intel.github.io/llvm/benchmarks/?runs=Baseline_PVC_L0%2CPR18324_PVC_L0 (max 3-4% improvement)

[vinser52]

Did I understand correctly that the RW lock you introduced is per the whole cache, not per cache entry? How is this approach compared to #18565?

[vinser52]

Why do you need to invent your own RW lock?

[uditagarwal97]

IIUC, the closest to reader/writer lock in C++ is shared_lock (https://en.cppreference.com/w/cpp/thread/shared_lock.html) and unique_lock. Both of these operate over a mutex. In my implementation, I've used an atomic variable instead, which I suppose will be faster than mutex here as contention between threads is low (w'll evict from cache rarely). In my understanding, for simple atomic counter-like applications, std::atomic performs better than mutex as the former can leverage HW support for atomic ops while mutex would also require OS support (like futex syscall on Linux?).

[vinser52]

Should it be TransferOwnershipToCache instead of TransferownerShipToCache?

[vinser52]

Also I think it breaks encapsulation: the cache config should be read inside the cache implementation, not by the caller of the cache

[vinser52]

Performance: https://intel.github.io/llvm/benchmarks/?runs=Baseline_PVC_L0%2CPR18324_PVC_L0 (max 3-4% improvement)

we need to test with the v2 adapter.

[vinser52]

As we agreed at the meeting, we will proceed with the approach in #18565.