Clarification on concurrency of jobs on KAI-Scheduler

[kev-iotairx]

Are jobs scheduled using KAI Scheduler running concurrently? Given that the mechanism is time slicing, how exactly does it run concurrent jobs?

Say I have 2 workloads in the queue requesting fractions gpu and memory. If my workload #1 requests 0.5 fraction of gpu and memory (in mb), is the other 0.5 idle when the job is running? Does 0.5 fraction represent the slice of time or % of GPU resource / compute cores?

Is the Lease Time configurable? And does it switch context at the end of the lease time?

[romanbaron]

Hey @kev-iotairx 👋

KAI Scheduler enables pod co-location on the same GPU device. The GPU fraction API permits a pod to request a partial GPU allocation, leaving the remaining capacity available for other pods.

It is important to note that KAI does not manage low-level GPU access or enforce GPU device sharing policies. Consequently, lease time configurations are not supported.

When multiple pods share a GPU device, each pod receives equal time slices for GPU access, independent of the GPU fraction requested. Within its allocated time slice, a pod has unbounded GPU access. Accordingly, if a process utilizes only 50% of the GPU cores during its time slice, the remaining capacity will remain idle for the duration of that slice.

[kev-iotairx]

Thanks for the clarification @romanbaron.

Between time slices; how does KAI maintain state? How are memory read/writes handled between time slices? Is memory for the process cleared between time slices to avoid frequent read/writes?

How do I monitor metrics on fractional gpu usage at the pod level and time slices? For example, if I have 4 pods each consuming 25% of GPU resource, how can I obtain pod-level resource consumption ? What kind of metrics and observability tools are available to monitor resource utilization?

We have a k8s cluster with kai-scheduler installed and a multiple processes running. We would like to view process / pod level resource utilization with timestamps. Is this available? We have prometheus / grafana setup for monitoring.

[romanbaron]

KAI does not manage resource allocation at this granularity. Once four pods are scheduled on the same GPU, it is the responsibility of the workloads to self-regulate and utilize only one-fourth of the GPU device memory each. Consequently, explicit memory clearing is not required.

We have plans to introduce native GPU sharing pod metrics in the future. At present, you may utilize nvidia-smi pmon to monitor memory allocation at the pod level. For monitoring aggregate GPU utilization, you can leverage DCGM exporter metrics (e.g. DCGM_FI_DEV_GPU_UTIL) by filtering on the KAI resource reservation pod operating on the same GPU device.

Example metric:

DCGM_FI_DEV_GPU_UTIL{gpu="0",UUID="GPU-3666b35e-61f4-d92f-83ca-269ddb7c2712",pci_bus_id="00000000:00:1B.0",device="nvidia0",modelName="Tesla T4",Hostname="ip-10-0-189-100.ec2.internal",DCGM_FI_DRIVER_VERSION="550.163.01",container="resource-reservation",namespace="kai-resource-reservation",pod="gpu-reservation-ip-10-0-189-100.ec2.internal-hgr22"} 83

To identify the reservation pod name efficiently:

1. Retrieve the runai-gpu-group from the desired GPU-sharing pod:

kubectl get pod -n <NAMESPACE> <PODNAME> -o yaml | grep runai-gpu-group

Locate the reservation pod using the runai-gpu-group label obtained in the previous step:

kubectl get pod -n kai-resource-reservation -l runai-gpu-group=<GPU-GROUP>

[kev-iotairx]

I understand it is the responsibility of the pods to self-regulate once they're scheduled on the same device. To be clear, the memory state for each pod is maintained on the gpu device between time slices? i.e KAI doesn't clear any gpu memory of the pods between time slices, so the requested memory remains occupied?

[kev-iotairx]

@romanbaron Is there another command I can run to get gpu compute utilization at the pod level?

[romanbaron]

I am not aware of anything else available at the moment.