CSI 3.14.0 - RBD QoS parameters

[mustafaakgunn]

Hi everyone!

After the contribution of @YiteGu , we are able to set QoS limits with parameters on storageclasses.
REFERENCE: 7595e20

I added some parameters to my RBD storageclass:

BaseReadIops: "500"
BaseWriteIops: "500"
BaseReadBytesPerSecond: "0"
BaseWriteBytesPerSecond: "0"
ReadIopsPerGB: "0"
WriteIopsPerGB: "0"
ReadBpsPerGB: "0"
WriteBpsPerGB: "0"
BaseVolSizeBytes: "107374182400"

but unfortunately I don't get different results in my FIO benchmarks. Am I doing something wrong?

Thanks.

[YiteGu]

Please provide a complete definition of SC @mustafaakgunn

[YiteGu]

this my SC yaml, use your QOS params:

allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ceph-ssd-rbd-flex2
parameters:
  clusterID: ceph02
  csi.storage.k8s.io/controller-expand-secret-name: rook-csi-rbd-provisioner
  csi.storage.k8s.io/controller-expand-secret-namespace: ceph02
  csi.storage.k8s.io/fstype: xfs
  csi.storage.k8s.io/node-stage-secret-name: rook-csi-rbd-node
  csi.storage.k8s.io/node-stage-secret-namespace: ceph02
  csi.storage.k8s.io/provisioner-secret-name: rook-csi-rbd-provisioner
  csi.storage.k8s.io/provisioner-secret-namespace: ceph02
  imageFeatures: layering,exclusive-lock,object-map,fast-diff,deep-flatten
  imageFormat: "2"
  pool: replicapool-ssd
  mounter: rbd-nbd
  BaseReadIops: "500"
  BaseWriteIops: "500"
  BaseReadBytesPerSecond: "0"
  BaseWriteBytesPerSecond: "0"
  ReadIopsPerGiB: "0"
  WriteIopsPerGiB: "0"
  ReadBpsPerGiB: "0"
  WriteBpsPerGiB: "0"
  BaseVolSizeBytes: "107374182400"
provisioner: ceph02.rbd.csi.ceph.com
reclaimPolicy: Delete
volumeBindingMode: Immediate

create a pvc:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: gyt-pvc-flex
  namespace: ceph02
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 100Gi
  storageClassName: ceph-ssd-rbd-flex2
  volumeMode: Filesystem

create a test pod:

apiVersion: v1
kind: Pod
metadata:
  name: gyt-demo-pod
  namespace: ceph02
spec:
  containers:
   - name: debian-fio
     image: registry.zjcdn.com/public/alto/debian-fio:v1
     command: ["sleep"]
     args: ["infinity"]
     volumeMounts:
       - name: gyt-volume1
         mountPath: /tmp/fio-test1/
  restartPolicy: Never
  volumes:
   - name: gyt-volume1
     persistentVolumeClaim:
       claimName: gyt-pvc-flex
       readOnly: false

enter test pod and run fio to test pvc mount directory:

root@gyt-demo-pod:~# fio --name=test --numjobs=1 --iodepth=1 --direct=1 --ioengine=libaio --rw=randwrite --bs=4K --size=1G --time_based --runtime=60 --filename=/tmp/fio-test1/tes-file --eta-newline 1
test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
Starting 1 process
Jobs: 1 (f=1): [w(1)][5.0%][w=2000KiB/s][w=500 IOPS][eta 00m:57s]
Jobs: 1 (f=1): [w(1)][8.3%][w=2000KiB/s][w=500 IOPS][eta 00m:55s]
Jobs: 1 (f=1): [w(1)][11.7%][w=2002KiB/s][w=500 IOPS][eta 00m:53s]
Jobs: 1 (f=1): [w(1)][15.0%][w=2002KiB/s][w=500 IOPS][eta 00m:51s]
Jobs: 1 (f=1): [w(1)][18.3%][w=2000KiB/s][w=500 IOPS][eta 00m:49s]
Jobs: 1 (f=1): [w(1)][21.7%][w=2000KiB/s][w=500 IOPS][eta 00m:47s]
Jobs: 1 (f=1): [w(1)][25.0%][w=2002KiB/s][w=500 IOPS][eta 00m:45s]
Jobs: 1 (f=1): [w(1)][28.3%][w=2002KiB/s][w=500 IOPS][eta 00m:43s]
Jobs: 1 (f=1): [w(1)][31.7%][w=2000KiB/s][w=500 IOPS][eta 00m:41s]
Jobs: 1 (f=1): [w(1)][35.0%][w=2000KiB/s][w=500 IOPS][eta 00m:39s]
Jobs: 1 (f=1): [w(1)][38.3%][w=1953KiB/s][w=488 IOPS][eta 00m:37s]
Jobs: 1 (f=1): [w(1)][41.7%][w=2000KiB/s][w=500 IOPS][eta 00m:35s]
^Cbs: 1 (f=1): [w(1)][43.3%][w=2002KiB/s][w=500 IOPS][eta 00m:34s]

[mustafaakgunn]

Hi @YiteGu,

Firstly thanks for your detailed answers 🙋🏼

Here is my full sc:

allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: csi-qos
parameters:
  BaseReadIops: "500"
  BaseWriteIops: "500"
  BaseReadBytesPerSecond: "0"
  BaseWriteBytesPerSecond: "0"
  ReadIopsPerGB: "0"
  WriteIopsPerGB: "0"
  ReadBpsPerGB: "0"
  WriteBpsPerGB: "0"
  BaseVolSizeBytes: "107374182400"
  clusterID: myceph
  csi.storage.k8s.io/controller-expand-secret-name: csi-rbd-secret
  csi.storage.k8s.io/controller-expand-secret-namespace: csi-system
  csi.storage.k8s.io/fstype: ext4
  csi.storage.k8s.io/node-stage-secret-name: csi-rbd-secret
  csi.storage.k8s.io/node-stage-secret-namespace: csi-system
  csi.storage.k8s.io/provisioner-secret-name: csi-rbd-secret
  csi.storage.k8s.io/provisioner-secret-namespace: csi-system
  imageFeatures: layering
  pool: multi-zone
provisioner: rbd.csi.ceph.com
reclaimPolicy: Delete
volumeBindingMode: Immediate

Also I demonstrated your test scenario, here is my result:

kubectl exec gyt-demo-pod -it -- /bin/sh

fio --name=test --numjobs=1 --iodepth=1 --direct=1 --ioengine=libaio --rw=randwrite --bs=4K --size=1G --time_based --runtime=60 --filename=/tmp/fio-test1/tes-file --eta-newline 1

test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.12
Starting 1 process
test: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1): [w(1)][5.0%][w=5765KiB/s][w=1441 IOPS][eta 00m:57s]
Jobs: 1 (f=1): [w(1)][8.3%][w=6012KiB/s][w=1503 IOPS][eta 00m:55s]
Jobs: 1 (f=1): [w(1)][11.7%][w=6224KiB/s][w=1556 IOPS][eta 00m:53s]
Jobs: 1 (f=1): [w(1)][15.0%][w=6256KiB/s][w=1564 IOPS][eta 00m:51s]
Jobs: 1 (f=1): [w(1)][18.3%][w=6234KiB/s][w=1558 IOPS][eta 00m:49s]
Jobs: 1 (f=1): [w(1)][21.7%][w=6178KiB/s][w=1544 IOPS][eta 00m:47s]
Jobs: 1 (f=1): [w(1)][25.0%][w=6246KiB/s][w=1561 IOPS][eta 00m:45s]
Jobs: 1 (f=1): [w(1)][28.3%][w=6380KiB/s][w=1595 IOPS][eta 00m:43s]

I noticed that I didn't used parameters such as "imageFeatures: layering,exclusive-lock,object-map,fast-diff,deep-flatten", "mounter: rbd-nbd" in my storageclass when compared to yours. Is it important for this process?

Thanks in advance 👍🏼

[YiteGu]

QOS must use: "mounter: rbd-nbd", not related with "imageFeatures: layering,exclusive-lock,object-map,fast-diff,deep-flatten"

[mustafaakgunn]

I'm trying with:

imageFeatures: layering,exclusive-lock,object-map,fast-diff,deep-flatten
mounter: rbd-nbd" in my storageclass

in my storageclass and looks like it works!

but one more question:

When applying the limit, I get results slightly below the limit I want, What could be the reason?

  BaseReadIops: "50"
  BaseWriteIops: "50"
  BaseReadBytesPerSecond: "0"
  BaseWriteBytesPerSecond: "0"
  ReadIopsPerGB: "0"
  WriteIopsPerGB: "0"
  ReadBpsPerGB: "0"
  WriteBpsPerGB: "0"
  BaseVolSizeBytes: "107374182400"

RESULT:
Jobs: 1 (f=1): [w(1)][5.0%][w=32KiB/s][w=8 IOPS][eta 00m:57s]
Jobs: 1 (f=1): [w(1)][8.3%][w=32KiB/s][w=8 IOPS][eta 00m:55s]
Jobs: 1 (f=1): [w(1)][11.7%][w=28KiB/s][w=7 IOPS][eta 00m:53s]

  BaseReadIops: "1500"
  BaseWriteIops: "1500"
  BaseReadBytesPerSecond: "0"
  BaseWriteBytesPerSecond: "0"
  ReadIopsPerGB: "0"
  WriteIopsPerGB: "0"
  ReadBpsPerGB: "0"
  WriteBpsPerGB: "0"
  BaseVolSizeBytes: "107374182400"

RESULT:
Jobs: 1 (f=1): [w(1)][5.0%][w=5349KiB/s][w=1337 IOPS][eta 00m:57s]
Jobs: 1 (f=1): [w(1)][8.3%][w=5360KiB/s][w=1340 IOPS][eta 00m:55s]
Jobs: 1 (f=1): [w(1)][11.7%][w=5360KiB/s][w=1340 IOPS][eta 00m:53s]

Thanks!

[YiteGu]

please tell me your fio command params

[YiteGu]

your ceph rbd verison?

[YiteGu]

RESULT:
Jobs: 1 (f=1): [w(1)][5.0%][w=32KiB/s][w=8 IOPS][eta 00m:57s]
Jobs: 1 (f=1): [w(1)][8.3%][w=32KiB/s][w=8 IOPS][eta 00m:55s]
Jobs: 1 (f=1): [w(1)][11.7%][w=28KiB/s][w=7 IOPS][eta 00m:53s]

I had not reproduct this.

RESULT:
Jobs: 1 (f=1): [w(1)][5.0%][w=5349KiB/s][w=1337 IOPS][eta 00m:57s]
Jobs: 1 (f=1): [w(1)][8.3%][w=5360KiB/s][w=1340 IOPS][eta 00m:55s]
Jobs: 1 (f=1): [w(1)][11.7%][w=5360KiB/s][w=1340 IOPS][eta 00m:53s]

try to use --iodepth=2 in fio command

[mustafaakgunn]

After a while the tests went back to normal after the pod was running, it might be related to my environment.

3.14 for CSI.
for CEPH it's squid.

Everything is working smoothly now, thank you very much for your support and contributions!

[YiteGu]

ReadIopsPerGB: "0"
WriteIopsPerGB: "0"
ReadBpsPerGB: "0"
WriteBpsPerGB: "0"

please change to:

ReadIopsPerGiB: "0"
WriteIopsPerGiB: "0"
ReadBpsPerGiB: "0"
WriteBpsPerGiB: "0"

[mustafaakgunn]

Also this is good point to be careful, noted.
Thanks for your help @YiteGu !

[microyahoo]

QOS must use: "mounter: rbd-nbd", not related with "imageFeatures: layering,exclusive-lock,object-map,fast-diff,deep-flatten"

hi @YiteGu, does QoS currently only supported for rbd-nbd mounter? Thanks.

[YiteGu]

Yes, QoS is implemented by librbd, krbd itself does not support QoS features.

[microyahoo]

Got it, thanks!