podman build inside EKS pods is abnormally slow

[danielap-ma]

Issue Description

We're using podman to build images on Jenkins pods running in EKS. The builds can take up to 10-12 minutes, whereas on EC2 Jenkins agents, building the same image in the same context never takes longer than 3 minutes.
We're running podman 4.6.2, but I tried upgrading to 5.2.0 and the issue persisted.

Steps to reproduce the issue

Steps to reproduce the issue

1. podman build inside EKS pods.

Describe the results you received

Builds are unusually slow.

Describe the results you expected

Reasonable build times.

podman info output

host:
  arch: arm64
  buildahVersion: 1.31.2
  cgroupControllers:
  - cpuset
  - cpu
  - cpuacct
  - blkio
  - memory
  - devices
  - freezer
  - net_cls
  - perf_event
  - net_prio
  - hugetlb
  - pids
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: Unknown
    path: /usr/local/libexec/podman/conmon
    version: 'conmon version 2.1.12, commit: 3bc422cd8aaec542d85d1a80f2d38e6e69046b5b'
  cpuUtilization:
    idlePercent: 93.37
    systemPercent: 3.08
    userPercent: 3.55
  cpus: 16
  databaseBackend: boltdb
  distribution:
    codename: bookworm
    distribution: debian
    version: "12"
  eventLogger: file
  freeLocks: 2048
  hostname: ci-arm-q1xbw
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.10.213-201.855.amzn2.aarch64
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 10537287680
  memTotal: 33023348736
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns_1.4.0-3_arm64
      path: /usr/lib/podman/aardvark-dns
      version: aardvark-dns 1.4.0
    package: netavark_1.4.0-3_arm64
    path: /usr/lib/podman/netavark
    version: netavark 1.4.0
  ociRuntime:
    name: crun
    package: crun_1.8.1-1+deb12u1_arm64
    path: /usr/bin/crun
    version: |-
      crun version 1.8.6
      commit: 73f759f4a39769f60990e7d225f561b4f4f06bcf
      rundir: /run/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  pasta:
    executable: ""
    package: ""
    version: ""
  remoteSocket:
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns_1.2.0-1_arm64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.4
  swapFree: 0
  swapTotal: 0
  uptime: 23h 45m 13.00s (Approximately 0.96 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /tmp/podman_storage
  graphRootAllocated: 161039233024
  graphRootUsed: 57681346560
  graphStatus:
    Backing Filesystem: overlayfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 12
  runRoot: /tmp/podman_storage
  transientStore: true
  volumePath: /tmp/podman_storage/volumes
version:
  APIVersion: 4.6.2
  Built: 1722336775
  BuiltTime: Tue Jul 30 10:52:55 2024
  GitCommit: 5db42e86862ef42c59304c38aa583732fd80f178
  GoVersion: go1.21.11
  Os: linux
  OsArch: linux/arm64
  Version: 4.6.2

Podman in a container

Yes

Privileged Or Rootless

Privileged

Upstream Latest Release

Yes

Additional environment details

EKS

Additional information

Happens in both amd and arm architectures.