memory cgroups cannot be enabled with recent kernel version

[joelddiaz]

Describe the bug

After upgrading to a recent kernel on a Raspberry Pi 5, the memory cgroup functionality has stopped working.

Looking at the non-working configuration (kernel 6.12.34+rpt-rpi-2712), you can see the active command line parameters passed to the linux kernel look like:

root@k8s-master-3:~# uname -a
Linux k8s-master-3 6.12.34+rpt-rpi-2712 #1 SMP PREEMPT Debian 1:6.12.34-1+rpt1~bookworm (2025-06-26) aarch64 GNU/Linux

root@k8s-master-3:~# cat /proc/cmdline 
reboot=w coherent_pool=1M 8250.nr_uarts=1 pci=pcie_bus_safe cgroup_disable=memory numa_policy=interleave nvme.max_host_mem_size_mb=0  numa=fake=8 system_heap.max_order=0 smsc95xx.macaddr=2C:CF:67:FB:89:3C vc_mem.mem_base=0x3fc00000 vc_mem.mem_size=0x40000000  cgroup_memory=1 cgroup_enable=memory console=ttyAMA10,115200 console=tty1 root=PARTUUID=4d8097e2-02 rootfstype=ext4 fsck.repair=yes rootwait

And /proc/cgroups shows no evidence of memory cgroups being active;

root@k8s-master-3:~# cat /proc/cgroups 
#subsys_name	hierarchy	num_cgroups	enabled
cpuset	0	70	1
cpu	0	70	1
cpuacct	0	70	1
blkio	0	70	1
devices	0	70	1
freezer	0	70	1
net_cls	0	70	1
perf_event	0	70	1
net_prio	0	70	1
pids	0	70	1

But looking at an otherwise identical Raspberry Pi 5 which has an older linux kernel (6.6.74+rpt-rpi-2712), we see the command line and /proc/cgroups output showing that memory cgroups are availble:

root@k8s-master-1:~# uname -a
Linux k8s-master-1 6.6.74+rpt-rpi-2712 #1 SMP PREEMPT Debian 1:6.6.74-1+rpt1 (2025-01-27) aarch64 GNU/Linux
root@k8s-master-1:~# cat /proc/cmdline 
reboot=w coherent_pool=1M 8250.nr_uarts=1 pci=pcie_bus_safe cgroup_disable=memory numa_policy=interleave  numa=fake=8 system_heap.max_order=0 smsc95xx.macaddr=2C:CF:67:CF:25:FD vc_mem.mem_base=0x3fc00000 vc_mem.mem_size=0x40000000  cgroup_memory=1 cgroup_enable=memory console=ttyAMA10,115200 console=tty1 root=PARTUUID=612e095f-02 rootfstype=ext4 fsck.repair=yes rootwait

/proc/cgroups:

root@k8s-master-1:~# cat /proc/cgroups 
#subsys_name	hierarchy	num_cgroups	enabled
cpuset	0	123	1
cpu	0	123	1
cpuacct	0	123	1
blkio	0	123	1
memory	0	123	1
devices	0	123	1
freezer	0	123	1
net_cls	0	123	1
perf_event	0	123	1
net_prio	0	123	1
pids	0	123	1

Steps to reproduce the behaviour

Enable memory cgroups by adding cgroup_memory=1 cgroup_enable=memory to /boot/firmware/cmdline.txt. Install/upgrade to linux 6.12.34+rpt-rpi-2712, reboot and see how /proc/cgroups doesn't list memory cgroups as enabled.

Device (s)

Raspberry Pi CM5

System

https://gist.github.com/joelddiaz/d4b75825c9e3f0493ac67cc49d0481b4

Logs

No response

Additional context

Maybe related to this commit...: 86099de

[nbuchwitz]

Does adding cgroup_enable=memory in cmdline.txt fix this?

[joelddiaz]

Does adding cgroup_enable=memory in cmdline.txt fix this?

The /proc/cmdline output is admittedly a mess, but it is in there:

root@k8s-master-3:~# cat /proc/cmdline
reboot=w coherent_pool=1M 8250.nr_uarts=1 pci=pcie_bus_safe cgroup_disable=memory numa_policy=interleave nvme.max_host_mem_size_mb=0 numa=fake=8 system_heap.max_order=0 smsc95xx.macaddr=2C:CF:67:FB:89:3C vc_mem.mem_base=0x3fc00000 vc_mem.mem_size=0x40000000 cgroup_memory=1 ##HERE!!!## cgroup_enable=memory console=ttyAMA10,115200 console=tty1 root=PARTUUID=4d8097e2-02 rootfstype=ext4 fsck.repair=yes rootwait

[pelwell]

I'm totally AFK and feeling too lazy to dig, but does removing cgroup_disable=memory (also present) help?

[mjmammoth]

I'm having the same problem.
It seems that cgroup_disable=memory is prepended as an argument by some other process, probably the bootloader.
Having cgroup_memory=1 cgroup_enable=memory appended to /boot/firmware/cmdline.txt and rebooting does not solve the issue.
cgroup_disable=memory is not in the cmdline.txt but still shows up when doing cat /proc/cmdline

OP did actually find this related commit that adds cgroup_disable=memory by default.
The commit states

To avoid such an issue, revert the artificial disablement and disable it
through the command line. If a user wants to enable the feature, it can
use the cgroup_enable= command line.

I would assume that override isn't working?

[nbuchwitz]

Yes, this comes from the bootargs in dt. @pelwell probably something which can be fixed with a dtparam easily. If you'd like I can give it a try

[popcornmix]

I think it's something else. I removed cgroup_disable=memory from DT but it doesn't fix it.

$ dmesg | grep cgroup
[    0.000000] Kernel command line: reboot=w coherent_pool=1M 8250.nr_uarts=1 pci=pcie_bus_safe numa_policy=interleave nvme.max_host_mem_size_mb=0  numa=fake=8 system_heap.max_order=0 smsc95xx.macaddr=2C:CF:67:DA:22:70 vc_mem.mem_base=0x3fc00000 vc_mem.mem_size=0x40000000  console=ttyAMA10,115200 console=tty1 root=PARTUUID=e7dd8211-02 rootfstype=ext4 fsck.repair=yes rootwait quiet splash plymouth.ignore-serial-consoles cfg80211.ieee80211_regdom=GB cgroup_enable=memory
[    0.000000] cgroup: Enabling memory control group subsystem

We do try to enable it, but no output from grep memory /proc/cgroups.

[nbuchwitz]

Without dt change

$ cat /proc/cmdline                  
reboot=w coherent_pool=1M 8250.nr_uarts=1 pci=pcie_bus_safe numa_policy=interleave nvme.max_host_mem_size_mb=0  numa=fake=8 system_heap.max_order=0 smsc95xx.macaddr=D8:3A:DD:83:E8:F1 vc_mem.mem_base=0x3fc00000 vc_mem.mem_size=0x40000000  console=ttyAMA10,115200 console=tty1 root=PARTUUID=2ab63b31-02 rootfstype=ext4 fsck.repair=yes rootwait quiet splash plymouth.ignore-serial-consoles cfg80211.ieee80211_regdom=DE cgroup_enable=memory cgroup_memory=1 cgroup_disable=memory

$ cat /proc/cgroups                  
#subsys_name	hierarchy	num_cgroups	enabled
cpuset	0	76	1
cpu	0	76	1
cpuacct	0	76	1
blkio	0	76	1
devices	0	76	1
freezer	0	76	1
net_cls	0	76	1
perf_event	0	76	1
net_prio	0	76	1
pids	0	76	1

$ cat /sys/fs/cgroup/cgroup.subtree_control
cpuset cpu io pids

With dt change

$ cat /proc/cmdline 
reboot=w coherent_pool=1M 8250.nr_uarts=1 pci=pcie_bus_safe numa_policy=interleave nvme.max_host_mem_size_mb=0  numa=fake=8 system_heap.max_order=0 smsc95xx.macaddr=D8:3A:DD:83:E8:F1 vc_mem.mem_base=0x3fc00000 vc_mem.mem_size=0x40000000  console=ttyAMA10,115200 console=tty1 root=PARTUUID=2ab63b31-02 rootfstype=ext4 fsck.repair=yes rootwait quiet splash plymouth.ignore-serial-consoles cfg80211.ieee80211_regdom=DE cgroup_enable=memory cgroup_memory=1

$ cat /proc/cgroups                        
#subsys_name	hierarchy	num_cgroups	enabled
cpuset	0	116	1
cpu	0	116	1
cpuacct	0	116	1
blkio	0	116	1
devices	0	116	1
freezer	0	116	1
net_cls	0	116	1
perf_event	0	116	1
net_prio	0	116	1
pids	0	116	1
nbw@rpi5 ~ $ 

$ cat /sys/fs/cgroup/cgroup.subtree_control
cpuset cpu io memory pids

/proc/cgroups is afaik only for legacy support, but the system is using cgroup v2. So everything as I would expect it with cgroups v2 and modfied cmdline. Also docker info doesn't show any warnings like WARNING: No memory limit support anymore.

Enabling memory cgroups with enable_cgroup=memory with disable_cgroup=memory already present from dt does seem to work (order seems to matter):

$ cat /proc/cmdline 
reboot=w coherent_pool=1M 8250.nr_uarts=1 pci=pcie_bus_safe cgroup_disable=memory numa_policy=interleave nvme.max_host_mem_size_mb=0  numa=fake=8 system_heap.max_order=0 smsc95xx.macaddr=D8:3A:DD:83:E8:F1 vc_mem.mem_base=0x3fc00000 vc_mem.mem_size=0x40000000  console=ttyAMA10,115200 console=tty1 root=PARTUUID=2ab63b31-02 rootfstype=ext4 fsck.repair=yes rootwait quiet splash plymouth.ignore-serial-consoles cfg80211.ieee80211_regdom=DE cgroup_enable=memory

$ dmesg|grep cgroup
[    0.000000] Kernel command line: reboot=w coherent_pool=1M 8250.nr_uarts=1 pci=pcie_bus_safe cgroup_disable=memory numa_policy=interleave nvme.max_host_mem_size_mb=0  numa=fake=8 system_heap.max_order=0 smsc95xx.macaddr=D8:3A:DD:83:E8:F1 vc_mem.mem_base=0x3fc00000 vc_mem.mem_size=0x40000000  console=ttyAMA10,115200 console=tty1 root=PARTUUID=2ab63b31-02 rootfstype=ext4 fsck.repair=yes rootwait quiet splash plymouth.ignore-serial-consoles cfg80211.ieee80211_regdom=DE cgroup_enable=memory
[    0.000000] cgroup: Disabling memory control group subsystem
[    0.000000] cgroup: Enabling memory control group subsystem

$ cat /sys/fs/cgroup/cgroup.subtree_control
cpuset cpu io memory pids

[boshkash1151]

for anyone facing a similar issue after the recent upgrade this helped me solve it.
I just popped the SD card, mounted it to another devices, edited the cmdline.txt file to include "cgroup_enable=memory cgroup_memory=1", inserting the SD card back

[pelwell]

Now that I'm back, I think everything is behaving as expected (at least by me):

1. Our preferred configuration is to have the memory cgroup available but disabled by default. Over the years, the way the kernel enables cgroups has evolved, and as a result we've had to change the way we re-enable the memory cgroup. The current version, which is relatively clean from a kernel code perspective, is to use the standard cgroup_disable=memory setting in our default command line settings (as inherited from the DTB files), but to allow users to override it using our (downstream only) cgroup_enable=memory setting. That should be all that is required - no cgroup_memory=1 - as long as you bear in mind that the settings are applied in the order they appear in /proc/cmdline, and that the contents of cmdline.txt appears after the values in the DTB.
2. Between 6.6 and 6.12 the kernel gained the CONFIG_MEMCG_V1 build setting. Without this, the old V1 memory cgroup support, as seen in /proc/cgroups, is not available. We have decided not to continue to support the old API, as it only increases the kernel size without adding any capabilities. This explains the absence of memory from /proc/cgroups.

Does anybody disagree with this?

[nbuchwitz]

My own investigation came to the same conclusion (eg. it's working with the new API, if the right cmdline parameter is added). So I agree with your summary.

[pelwell]

Thanks. In anticipation of no dissenting voices I'm marking this closed, but that won't prevent further comments if that turns out not to be the case.

[joelddiaz]

Not sure why it was failing for me...i guess i might have had a typo at first and then became focused on the output of /proc/cgroups.

Anyway, I can confirm that updating to 6.12.X with the right cgroup_enable=memory croup_memory=1 in /boot/firmware/cmdlline.txt does work as far as cgroups (v2) being enabled.

Sorry for the noise, and thanks

[pelwell]

No problem - it's good to have to remember this stuff every now and again (but cgroup_memory=1 is not needed - the kernel will complain that it is unrecognised in its log output).