podman build inside EKS pods is abnormally slow

[danielap-ma]

Issue Description

We're using podman to build images on Jenkins pods running in EKS. The builds can take up to 10-12 minutes, whereas on EC2 Jenkins agents, building the same image in the same context never takes longer than 3 minutes.
We're running podman 4.6.2, but I tried upgrading to 5.2.0 and the issue persisted.

Steps to reproduce the issue

Steps to reproduce the issue

1. podman build inside EKS pods.

Describe the results you received

Builds are unusually slow.

Describe the results you expected

Reasonable build times.

podman info output

host:
  arch: arm64
  buildahVersion: 1.31.2
  cgroupControllers:
  - cpuset
  - cpu
  - cpuacct
  - blkio
  - memory
  - devices
  - freezer
  - net_cls
  - perf_event
  - net_prio
  - hugetlb
  - pids
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: Unknown
    path: /usr/local/libexec/podman/conmon
    version: 'conmon version 2.1.12, commit: 3bc422cd8aaec542d85d1a80f2d38e6e69046b5b'
  cpuUtilization:
    idlePercent: 93.37
    systemPercent: 3.08
    userPercent: 3.55
  cpus: 16
  databaseBackend: boltdb
  distribution:
    codename: bookworm
    distribution: debian
    version: "12"
  eventLogger: file
  freeLocks: 2048
  hostname: ci-arm-q1xbw
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.10.213-201.855.amzn2.aarch64
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 10537287680
  memTotal: 33023348736
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns_1.4.0-3_arm64
      path: /usr/lib/podman/aardvark-dns
      version: aardvark-dns 1.4.0
    package: netavark_1.4.0-3_arm64
    path: /usr/lib/podman/netavark
    version: netavark 1.4.0
  ociRuntime:
    name: crun
    package: crun_1.8.1-1+deb12u1_arm64
    path: /usr/bin/crun
    version: |-
      crun version 1.8.6
      commit: 73f759f4a39769f60990e7d225f561b4f4f06bcf
      rundir: /run/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  pasta:
    executable: ""
    package: ""
    version: ""
  remoteSocket:
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns_1.2.0-1_arm64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.4
  swapFree: 0
  swapTotal: 0
  uptime: 23h 45m 13.00s (Approximately 0.96 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /tmp/podman_storage
  graphRootAllocated: 161039233024
  graphRootUsed: 57681346560
  graphStatus:
    Backing Filesystem: overlayfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 12
  runRoot: /tmp/podman_storage
  transientStore: true
  volumePath: /tmp/podman_storage/volumes
version:
  APIVersion: 4.6.2
  Built: 1722336775
  BuiltTime: Tue Jul 30 10:52:55 2024
  GitCommit: 5db42e86862ef42c59304c38aa583732fd80f178
  GoVersion: go1.21.11
  Os: linux
  OsArch: linux/arm64
  Version: 4.6.2

Podman in a container

Yes

Privileged Or Rootless

Privileged

Upstream Latest Release

Yes

Additional environment details

EKS

Additional information

Happens in both amd and arm architectures.

[zarko-a]

Did you ever figure out a solution? We switched to podman builds inside of EKS and are facing the same issue.

Larger apps (that copy a bunch of files to a image) are taking 4x more time than docker buildkit.

We seem to be using overlay already.

podman info --debug
host:
  arch: amd64
  buildahVersion: 1.42.2
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - hugetlb
  - pids
  - misc
  cgroupManager: cgroupfs
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.13-2.fc43.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.13, commit: '
  cpuUtilization:
    idlePercent: 88.78
    systemPercent: 2.41
    userPercent: 8.81
  cpus: 16
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: container
    version: "43"
  eventLogger: file
  freeLocks: 2048
  hostname: linux-self-hosted-podman-10gb-5cpu-htjm9-runner-pq7ws-workflow
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 6.12.40-64.114.amzn2023.x86_64
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 2036965376
  memTotal: 32907362304
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.17.0-1.fc43.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.17.0
    package: netavark-1.17.1-1.fc43.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.17.1
  ociRuntime:
    name: crun
    package: crun-1.25.1-1.fc43.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.25.1
      commit: 156ae065d4a322d149c7307034f98d9637aa92a2
      rundir: /run/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20260117.g81c97f6-1.fc43.x86_64
    version: |
      pasta 0^20260117.g81c97f6-1.fc43.x86_64
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 0
  swapTotal: 0
  uptime: 0h 50m 1.00s
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
  - quay.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.additionalImageStores:
    - /var/lib/shared
    - /usr/lib/containers/storage
    overlay.imagestore: /usr/lib/containers/storage
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.13-4.fc43.x86_64
      Version: |-
        fusermount3 version: 3.16.2
        fuse-overlayfs: version 1.13-dev
        FUSE library version 3.16.2
        using FUSE kernel interface version 7.38
    overlay.mountopt: nodev,fsync=0
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 107294470144
  graphRootUsed: 34832113664
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Supports shifting: "true"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 7
  runRoot: /run/containers/storage
  transientStore: false
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 5.7.1
  BuildOrigin: Fedora Project
  Built: 1765324800
  BuiltTime: Wed Dec 10 00:00:00 2025
  GitCommit: f845d14e941889ba4c071f35233d09b29d363c75
  GoVersion: go1.25.4 X:nodwarf5
  Os: linux
  OsArch: linux/amd64
  Version: 5.7.1

[zarko-a]

I realized what my issue was. It was fuse-overlayfs. I skipped over this part initially because I saw overlay. Everyone warns against VFS 😀. fuse-overlay runs in namespaces and is slower because it is on top of the existing overlay created for the container in EKS.

After changing runRoot in /etc/containers/storage.conf to a path that was backed by emptyDir on the host, podman started using (kernel) overlay instead of userspace fuse-overlay and things were back to normal speed. Without noticeable difference between docker's buildkit and podman.