Extremely Slow Startup

[rnemeth90]

Version

5.3.1

What Kubernetes platforms are you running on?

AKS Azure

Steps to reproduce

1. Deploy nginx to AKS
2. Deploy several thousand ingress objects (8000+)
3. Scale out nginx or perform a rollout
4. Watch the pods take 20+ minutes to pass the readiness probe

We have around 150 namespaces that have ingresses configured to use this particular ingress controller. Though, that number may vary from day-to-day.

These namespace each have 5 master ingresses (each for a different domain name) and 9 or 10 different deployments that create a minion for each master (so 5 minion ingresses per service). In addition, we have a 'headless' service that has an endpoint that is mapped to the IP address of a VM in Azure (these are the '-proxy-' minion ingress objects you see below.

ingress-master-apiautortn67
ingress-master-excelsiorautortn67
ingress-master-rtn67-financial1
ingress-master-rtn67-financial2
ingress-master-us1-rtn67

λ kubernetes-workspace (wip/586041) $ k get ing | grep -i minion| awk '{print $1}'
aprimo-csp-rtn67-minion--apiautortn67-labs-aprimo-com
aprimo-csp-rtn67-minion--excelsiorautortn67-labs-aprimo-com
aprimo-csp-rtn67-minion--rtn67-financial1-labs-aprimo-com
aprimo-csp-rtn67-minion--rtn67-financial2-labs-aprimo-com
aprimo-csp-rtn67-minion--us1-rtn67-labs-aprimo-com
aprimo-upload-rtn67-minion--apiautortn67-labs-aprimo-com
aprimo-upload-rtn67-minion--excelsiorautortn67-labs-aprimo-com
aprimo-upload-rtn67-minion--rtn67-financial1-labs-aprimo-com
aprimo-upload-rtn67-minion--rtn67-financial2-labs-aprimo-com
aprimo-upload-rtn67-minion--us1-rtn67-labs-aprimo-com
budget-ui-rtn67-minion--apiautortn67-labs-aprimo-com
budget-ui-rtn67-minion--excelsiorautortn67-labs-aprimo-com
budget-ui-rtn67-minion--rtn67-financial1-labs-aprimo-com
budget-ui-rtn67-minion--rtn67-financial2-labs-aprimo-com
budget-ui-rtn67-minion--us1-rtn67-labs-aprimo-com
cube-stack-pm-rtn67-minion-us1-rtn67-labs-aprimo-com
idealab-api-rtn67-minion--apiautortn67-labs-aprimo-com
idealab-api-rtn67-minion--excelsiorautortn67-labs-aprimo-com
idealab-api-rtn67-minion--rtn67-financial1-labs-aprimo-com
idealab-api-rtn67-minion--rtn67-financial2-labs-aprimo-com
idealab-api-rtn67-minion--us1-rtn67-labs-aprimo-com
lab-proxy-rtn67-minion-apiauto
lab-proxy-rtn67-minion-excelsiorauto
lab-proxy-rtn67-minion-financial1
lab-proxy-rtn67-minion-financial2
lab-proxy-rtn67-minion-us1
marketingopsapi-rtn67-minion--apiautortn67-labs-aprimo-com
marketingopsapi-rtn67-minion--excelsiorautortn67-labs-aprimo-com
marketingopsapi-rtn67-minion--rtn67-financial1-labs-aprimo-com
marketingopsapi-rtn67-minion--rtn67-financial2-labs-aprimo-com
marketingopsapi-rtn67-minion--us1-rtn67-labs-aprimo-com
marketingopsui-rtn67-minion--apiautortn67-labs-aprimo-com
marketingopsui-rtn67-minion--excelsiorautortn67-labs-aprimo-com
marketingopsui-rtn67-minion--rtn67-financial1-labs-aprimo-com
marketingopsui-rtn67-minion--rtn67-financial2-labs-aprimo-com
marketingopsui-rtn67-minion--us1-rtn67-labs-aprimo-com
pushnotification-rtn67-minion--apiautortn67-labs-aprimo-com
pushnotification-rtn67-minion--excelsiorautortn67-labs-aprimo-com
pushnotification-rtn67-minion--rtn67-financial1-labs-aprimo-com
pushnotification-rtn67-minion--rtn67-financial2-labs-aprimo-com
pushnotification-rtn67-minion--us1-rtn67-labs-aprimo-com
reviewfileservice-rtn67-minion--apiautortn67-labs-aprimo-com
reviewfileservice-rtn67-minion--excelsiorautortn67-labs-aprimo-com
reviewfileservice-rtn67-minion--rtn67-financial1-labs-aprimo-com
reviewfileservice-rtn67-minion--rtn67-financial2-labs-aprimo-com
reviewfileservice-rtn67-minion--us1-rtn67-labs-aprimo-com
uls-rtn67-minion--apiautortn67-labs-aprimo-com
uls-rtn67-minion--excelsiorautortn67-labs-aprimo-com
uls-rtn67-minion--rtn67-financial1-labs-aprimo-com
uls-rtn67-minion--rtn67-financial2-labs-aprimo-com
uls-rtn67-minion--us1-rtn67-labs-aprimo-com

In addition, we 'shutdown' some deployments in AKS on a schedule (i.e. we scale deployments to 0 replicas) to save on costs. When we do this, the endpoint slices associated with these deployments have no IP addresses. It seems as though Nginx adds a default IP address to the upstream (127.0.0.1:8181) if it 'discovers' these endpoint slices with no IP address (we have observed this in the logs).

kubernetes-ingress/internal/configs/version1/config.go

Line 351 in c2389d2

Address: "127.0.0.1:8181",

[github-actions]

Hi @rnemeth90 thanks for reporting!

Be sure to check out the docs and the Contributing Guidelines while you wait for a human to take a look at this 🙂

If your issue concerns any of the enterprise features please open a ticket with F5 directly (For NGINX Plus customers NGINX Ingress Controller (when used with NGINX Plus) is covered by the support contract.)

Cheers!

[vepatel]

Hi @rnemeth90 ,

• can you please provide us with the describe output of your NIC deployment please?
• please specify if you're using OSS or Plus
• any other logs which might help

Deploy several thousand ingress objects (8000+)

an example of one if them would be great as well

From my test on 5.3.1 with 1000 ingress in single ns and single pod, startup took around 12 sec

[rnemeth90]

λ kubernetes-workspace (wip/586041) $ k describe deploy lab-pm-public-nginx-pm-r09-nginx-ingress-controller
Name:                   lab-pm-public-nginx-pm-r09-nginx-ingress-controller
Namespace:              pm-r09
CreationTimestamp:      Thu, 05 Mar 2026 15:42:23 -0500
Labels:                 app.kubernetes.io/instance=lab-pm-public-nginx-pm-r09
                        app.kubernetes.io/managed-by=Helm
                        app.kubernetes.io/name=nginx-ingress
                        app.kubernetes.io/version=5.3.1
                        helm.sh/chart=nginx-ingress-2.4.1
Annotations:            deployment.kubernetes.io/revision: 15
                        meta.helm.sh/release-name: lab-pm-public-nginx-pm-r09
                        meta.helm.sh/release-namespace: pm-r09
Selector:               app.kubernetes.io/instance=lab-pm-public-nginx-pm-r09,app.kubernetes.io/name=nginx-ingress
Replicas:               10 desired | 10 updated | 10 total | 10 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:           app.kubernetes.io/instance=lab-pm-public-nginx-pm-r09
                    app.kubernetes.io/name=nginx-ingress
  Annotations:      kubectl.kubernetes.io/restartedAt: 2026-04-15T19:29:01-04:00
                    prometheus.io/port: 9113
                    prometheus.io/scheme: http
                    prometheus.io/scrape: true
  Service Account:  lab-pm-public-nginx-pm-r09-nginx-ingress
  Containers:
   nginx-ingress:
    Image:       nginx/nginx-ingress:5.3.1
    Ports:       80/TCP, 443/TCP, 9113/TCP, 8081/TCP
    Host Ports:  0/TCP, 0/TCP, 0/TCP, 0/TCP
    Args:
      -nginx-plus=false
      -nginx-reload-timeout=60000
      -enable-app-protect=false
      -enable-app-protect-dos=false
      -nginx-configmaps=$(POD_NAMESPACE)/lab-pm-public-nginx-pm-r09-nginx-ingress
      -default-server-tls-secret=pm-r09/azure-pm-lab-tls-csi
      -ingress-class=lab-pm-public-nginx-pm-r09
      -health-status=true
      -health-status-uri=/healthz
      -nginx-debug=false
      -log-level=info
      -log-format=glog
      -nginx-status=true
      -nginx-status-port=8080
      -nginx-status-allow-cidrs=127.0.0.1
      -report-ingress-status
      -external-service=lab-pm-public-nginx-pm-r09-nginx-ingress-controller
      -enable-leader-election=true
      -leader-election-lock-name=lab-pm-public-nginx-pm-r09-nginx-ingress-leader-election
      -wildcard-tls-secret=pm-r09/azure-pm-lab-tls-csi
      -enable-prometheus-metrics=true
      -prometheus-metrics-listen-port=9113
      -prometheus-tls-secret=
      -enable-service-insight=false
      -service-insight-listen-port=9114
      -service-insight-tls-secret=
      -enable-custom-resources=true
      -enable-snippets=true
      -disable-ipv6=false
      -enable-tls-passthrough=false
      -enable-cert-manager=false
      -enable-oidc=false
      -enable-external-dns=false
      -default-http-listener-port=80
      -default-https-listener-port=443
      -ready-status=true
      -ready-status-port=8081
      -enable-latency-metrics=false
      -ssl-dynamic-reload=true
      -enable-telemetry-reporting=true
      -weight-changes-dynamic-reload=false
    Requests:
      cpu:      2
      memory:   4Gi
    Readiness:  http-get http://:readiness-port/nginx-ready delay=120s timeout=1s period=1s #success=1 #failure=3
    Environment:
      POD_NAMESPACE:   (v1:metadata.namespace)
      POD_NAME:        (v1:metadata.name)
    Mounts:           <none>
  Volumes:            <none>
  Node-Selectors:     k8s.aprimo.com/workloadType=web
                      kubernetes.io/os=linux
  Tolerations:        <none>
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    True    NewReplicaSetAvailable
OldReplicaSets:  lab-pm-public-nginx-pm-r09-nginx-ingress-controller-7c64b4b69c (0/0 replicas created), lab-pm-public-nginx-pm-r09-nginx-ingress-controller-7c4d9c67dc (0/0 replicas created), lab-pm-public-nginx-pm-r09-nginx-ingress-controller-587dbb5bd8 (0/0 replicas created), lab-pm-public-nginx-pm-r09-nginx-ingress-controller-657bd796d4 (0/0 replicas created), lab-pm-public-nginx-pm-r09-nginx-ingress-controller-5875867d45 (0/0 replicas created), lab-pm-public-nginx-pm-r09-nginx-ingress-controller-5cc44cd6bf (0/0 replicas created), lab-pm-public-nginx-pm-r09-nginx-ingress-controller-5ff6c46b96 (0/0 replicas created), lab-pm-public-nginx-pm-r09-nginx-ingress-controller-67c656c84 (0/0 replicas created), lab-pm-public-nginx-pm-r09-nginx-ingress-controller-74c6fbbcd5 (0/0 replicas created), lab-pm-public-nginx-pm-r09-nginx-ingress-controller-65579d78fd (0/0 replicas created)
NewReplicaSet:   lab-pm-public-nginx-pm-r09-nginx-ingress-controller-69cb56d5b6 (10/10 replicas created)
Events:          <none>

We are using OSS

Here is an excerpt of the logs from one of the NIC pods:

I20260417 17:06:50.713967   1 main.go:112] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"pm-g023", Name:"budget-ui-g023-minion--excelsiorautog023-labs-aprimo-com", UID:"8aed0cd4-ecb3-49f6-a3b2-fe927dedf005", APIVersion:"networking.k8s.io/v1", ResourceVersion:"2575826761", FieldPath:""}): type: 'Normal' reason: 'AddedOrUpdated' Configuration for pm-g023/budget-ui-g023-minion--excelsiorautog023-labs-aprimo-com was added or updated
I20260417 17:06:50.713976   1 main.go:112] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"pm-g023", Name:"idealab-api-g023-minion--excelsiorautog023-labs-aprimo-com", UID:"d1fd7495-10b0-41a9-a00b-39ecf17f0d17", APIVersion:"networking.k8s.io/v1", ResourceVersion:"2575826873", FieldPath:""}): type: 'Normal' reason: 'AddedOrUpdated' Configuration for pm-g023/idealab-api-g023-minion--excelsiorautog023-labs-aprimo-com was added or updated
I20260417 17:06:50.713987   1 main.go:112] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"pm-g023", Name:"lab-proxy-g023-minion-excelsiorauto", UID:"1347705f-991a-47d0-a7fb-bc64747d4b99", APIVersion:"networking.k8s.io/v1", ResourceVersion:"2575819513", FieldPath:""}): type: 'Normal' reason: 'AddedOrUpdated' Configuration for pm-g023/lab-proxy-g023-minion-excelsiorauto was added or updated
W20260417 17:06:50.718663   1 controller.go:2167] Error trying to get the secret  for Ingress ingress-master-g023-financial1: secret doesn't exist or of an unsupported type
W20260417 17:06:50.719561   1 controller.go:2322] Error retrieving endpoints for the service aprimo-csp-g023: no endpointslices for target port 8080 in service aprimo-csp-g023
W20260417 17:06:50.720304   1 controller.go:2322] Error retrieving endpoints for the service aprimo-upload-g023: no endpointslices for target port 8080 in service aprimo-upload-g023
W20260417 17:06:50.721013   1 controller.go:2322] Error retrieving endpoints for the service budget-ui-g023: no endpointslices for target port 8080 in service budget-ui-g023
W20260417 17:06:50.721745   1 controller.go:2322] Error retrieving endpoints for the service idealab-api-g023: no endpointslices for target port 80 in service idealab-api-g023
I20260417 17:06:50.725712   1 main.go:112] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"pm-g023", Name:"ingress-master-g023-financial1", UID:"b29a1789-8cb9-4d23-a4ae-bd37a18170fd", APIVersion:"networking.k8s.io/v1", ResourceVersion:"2575819444", FieldPath:""}): type: 'Normal' reason: 'AddedOrUpdated' Configuration for pm-g023/ingress-master-g023-financial1 was added or updated
I20260417 17:06:50.725746   1 main.go:112] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"pm-g023", Name:"aprimo-csp-g023-minion--g023-financial1-labs-aprimo-com", UID:"6f9aa8ed-e181-4213-9292-7e6bdab9866d", APIVersion:"networking.k8s.io/v1", ResourceVersion:"2575827909", FieldPath:""}): type: 'Normal' reason: 'AddedOrUpdated' Configuration for pm-g023/aprimo-csp-g023-minion--g023-financial1-labs-aprimo-com was added or updated
I20260417 17:06:50.725758   1 main.go:112] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"pm-g023", Name:"aprimo-upload-g023-minion--g023-financial1-labs-aprimo-com", UID:"69cb88d0-6787-422d-85ce-3977107c8edf", APIVersion:"networking.k8s.io/v1", ResourceVersion:"2575829548", FieldPath:""}): type: 'Normal' reason: 'AddedOrUpdated' Configuration for pm-g023/aprimo-upload-g023-minion--g023-financial1-labs-aprimo-com was added or updated
I20260417 17:06:50.725767   1 main.go:112] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"pm-g023", Name:"budget-ui-g023-minion--g023-financial1-labs-aprimo-com", UID:"53a7b98f-edb2-4b3e-8c36-03c8b202d2bb", APIVersion:"networking.k8s.io/v1", ResourceVersion:"2575826682", FieldPath:""}): type: 'Normal' reason: 'AddedOrUpdated' Configuration for pm-g023/budget-ui-g023-minion--g023-financial1-labs-aprimo-com was added or updated
I20260417 17:06:50.725776   1 main.go:112] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"pm-g023", Name:"idealab-api-g023-minion--g023-financial1-labs-aprimo-com", UID:"edcd0bab-1e10-42a4-bf80-293cb2a0cbb4", APIVersion:"networking.k8s.io/v1", ResourceVersion:"2575826820", FieldPath:""}): type: 'Normal' reason: 'AddedOrUpdated' Configuration for pm-g023/idealab-api-g023-minion--g023-financial1-labs-aprimo-com was added or updated
I20260417 17:06:50.725785   1 main.go:112] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"pm-g023", Name:"lab-proxy-g023-minion-financial1", UID:"f097b013-b82e-46fb-af3a-7ea2b0f55ee1", APIVersion:"networking.k8s.io/v1", ResourceVersion:"2575819554", FieldPath:""}): type: 'Normal' reason: 'AddedOrUpdated' Configuration for pm-g023/lab-proxy-g023-minion-financial1 was added or updated
W20260417 17:06:50.730501   1 controller.go:2167] Error trying to get the secret  for Ingress ingress-master-g023-financial2: secret doesn't exist or of an unsupported type
W20260417 17:06:50.731293   1 controller.go:2322] Error retrieving endpoints for the service aprimo-csp-g023: no endpointslices for target port 8080 in service aprimo-csp-g023
W20260417 17:06:50.732038   1 controller.go:2322] Error retrieving endpoints for the service aprimo-upload-g023: no endpointslices for target port 8080 in service aprimo-upload-g023
W20260417 17:06:50.732719   1 controller.go:2322] Error retrieving endpoints for the service budget-ui-g023: no endpointslices for target port 8080 in service budget-ui-g023
W20260417 17:06:50.733442   1 controller.go:2322] Error retrieving endpoints for the service idealab-api-g023: no endpointslices for target port 80 in service idealab-api-g023
I20260417 17:06:50.737393   1 main.go:112] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"pm-g023", Name:"ingress-master-g023-financial2", UID:"8ab69225-e6d3-411f-9b24-7ad16105a57a", APIVersion:"networking.k8s.io/v1", ResourceVersion:"2575819469", FieldPath:""}): type: 'Normal' reason: 'AddedOrUpdated' Configuration for pm-g023/ingress-master-g023-financial2 was added or updated
I20260417 17:06:50.737428   1 main.go:112] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"pm-g023", Name:"aprimo-csp-g023-minion--g023-financial2-labs-aprimo-com", UID:"2b45a86b-8428-4d92-8065-73cf49af684d", APIVersion:"networking.k8s.io/v1", ResourceVersion:"2575827826", FieldPath:""}): type: 'Normal' reason: 'AddedOrUpdated' Configuration for pm-g023/aprimo-csp-g023-minion--g023-financial2-labs-aprimo-com was added or updated
I20260417 17:06:50.737444   1 main.go:112] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"pm-g023", Name:"aprimo-upload-g023-minion--g023-financial2-labs-aprimo-com", UID:"b02f5a8d-d346-47e1-8eda-417911fd0686", APIVersion:"networking.k8s.io/v1", ResourceVersion:"2575829591", FieldPath:""}): type: 'Normal' reason: 'AddedOrUpdated' Configuration for pm-g023/aprimo-upload-g023-minion--g023-financial2-labs-aprimo-com was added or updated
I20260417 17:06:50.737452   1 main.go:112] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"pm-g023", Name:"budget-ui-g023-minion--g023-financial2-labs-aprimo-com", UID:"d0b668f1-026e-4bfd-bb09-dd793fea06ed", APIVersion:"networking.k8s.io/v1", ResourceVersion:"2575826738", FieldPath:""}): type: 'Normal' reason: 'AddedOrUpdated' Configuration for pm-g023/budget-ui-g023-minion--g023-financial2-labs-aprimo-com was added or updated
I20260417 17:06:50.737482   1 main.go:112] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"pm-g023", Name:"idealab-api-g023-minion--g023-financial2-labs-aprimo-com", UID:"f4273fd0-d94b-4f22-aa36-8b0e44a80e7b", APIVersion:"networking.k8s.io/v1", ResourceVersion:"2575826920", FieldPath:""}): type: 'Normal' reason: 'AddedOrUpdated' Configuration for pm-g023/idealab-api-g023-minion--g023-financial2-labs-aprimo-com was added or updated
I20260417 17:06:50.737494   1 main.go:112] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"pm-g023", Name:"lab-proxy-g023-minion-financial2", UID:"910dbe86-dceb-45ea-8510-b130d05dea44", APIVersion:"networking.k8s.io/v1", ResourceVersion:"2575819581", FieldPath:""}): type: 'Normal' reason: 'AddedOrUpdated' Configuration for pm-g023/lab-proxy-g023-minion-financial2 was added or updated
W20260417 17:06:50.742527   1 controller.go:2167] Error trying to get the secret  for Ingress ingress-master-us1-g023: secret doesn't exist or of an unsupported type
W20260417 17:06:50.743476   1 controller.go:2322] Error retrieving endpoints for the service aprimo-csp-g023: no endpointslices for target port 8080 in service aprimo-csp-g023
W20260417 17:06:50.744177   1 controller.go:2322] Error retrieving endpoints for the service aprimo-upload-g023: no endpointslices for target port 8080 in service aprimo-upload-g023
W20260417 17:06:50.744856   1 controller.go:2322] Error retrieving endpoints for the service budget-ui-g023: no endpointslices for target port 8080 in service budget-ui-g023
W20260417 17:06:50.745532   1 controller.go:2322] Error retrieving endpoints for the service cube-stack-pm-g023: error determining target port for port {http 0} in Ingress: no pods of service cube-stack-pm-g023
W20260417 17:06:50.746215   1 controller.go:2322] Error retrieving endpoints for the service idealab-api-g023: no endpointslices for target port 80 in service idealab-api-g023
I20260417 17:06:50.750099   1 main.go:112] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"pm-g023", Name:"ingress-master-us1-g023", UID:"c38bed02-c85e-4bd4-a8fb-d3439de5b678", APIVersion:"networking.k8s.io/v1", ResourceVersion:"2575819357", FieldPath:""}): type: 'Normal' reason: 'AddedOrUpdated' Configuration for pm-g023/ingress-master-us1-g023 was added or updated
I20260417 17:06:50.750134   1 main.go:112] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"pm-g023", Name:"aprimo-csp-g023-minion--us1-g023-labs-aprimo-com", UID:"e11f77a1-f481-4244-a5c7-ca41cab0dec2", APIVersion:"networking.k8s.io/v1", ResourceVersion:"2575827884", FieldPath:""}): type: 'Normal' reason: 'AddedOrUpdated' Configuration for pm-g023/aprimo-csp-g023-minion--us1-g023-labs-aprimo-com was added or updated
I20260417 17:06:50.750150   1 main.go:112] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"pm-g023", Name:"aprimo-upload-g023-minion--us1-g023-labs-aprimo-com", UID:"778a2a29-21a5-4bf6-ab3f-f8e588d38088", APIVersion:"networking.k8s.io/v1", ResourceVersion:"2575829660", FieldPath:""}): type: 'Normal' reason: 'AddedOrUpdated' Configuration for pm-g023/aprimo-upload-g023-minion--us1-g023-labs-aprimo-com was added or updated
I20260417 17:06:50.750159   1 main.go:112] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"pm-g023", Name:"budget-ui-g023-minion--us1-g023-labs-aprimo-com", UID:"9f0dff14-2bc3-48d7-951b-8e10c046588e", APIVersion:"networking.k8s.io/v1", ResourceVersion:"2575826708", FieldPath:""}): type: 'Normal' reason: 'AddedOrUpdated' Configuration for pm-g023/budget-ui-g023-minion--us1-g023-labs-aprimo-com was added or updated
I20260417 17:06:50.750173   1 main.go:112] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"pm-g023", Name:"cube-stack-pm-g023-minion-us1-g023-labs-aprimo-com", UID:"d8d0b61f-9072-4bf4-a454-223cbe29b2b4", APIVersion:"networking.k8s.io/v1", ResourceVersion:"2575827082", FieldPath:""}): type: 'Normal' reason: 'AddedOrUpdated' Configuration for pm-g023/cube-stack-pm-g023-minion-us1-g023-labs-aprimo-com was added or updated
I20260417 17:06:50.750182   1 main.go:112] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"pm-g023", Name:"idealab-api-g023-minion--us1-g023-labs-aprimo-com", UID:"598e08fe-da3c-4bf2-9b87-fcc59b123460", APIVersion:"networking.k8s.io/v1", ResourceVersion:"2575826851", FieldPath:""}): type: 'Normal' reason: 'AddedOrUpdated' Configuration for pm-g023/idealab-api-g023-minion--us1-g023-labs-aprimo-com was added or updated
I20260417 17:06:50.750193   1 main.go:112] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"pm-g023", Name:"lab-proxy-g023-minion-us1", UID:"668434b9-9444-4eef-b9bf-0375c7baad4d", APIVersion:"networking.k8s.io/v1", ResourceVersion:"2575819493", FieldPath:""}): type: 'Normal' reason: 'AddedOrUpdated' Configuration for pm-g023/lab-proxy-g023-minion-us1 was added or updated
W20260417 17:06:50.759722   1 controller.go:2167] Error trying to get the secret  for Ingress ingress-master-apiautog023: secret doesn't exist or of an unsupported type
W20260417 17:06:50.761238   1 controller.go:2322] Error retrieving endpoints for the service aprimo-csp-g023: no endpointslices for target port 8080 in service aprimo-csp-g023
W20260417 17:06:50.762410   1 controller.go:2322] Error retrieving endpoints for the service aprimo-upload-g023: no endpointslices for target port 8080 in service aprimo-upload-g023
W20260417 17:06:50.763201   1 controller.go:2322] Error retrieving endpoints for the service budget-ui-g023: no endpointslices for target port 8080 in service budget-ui-g023
W20260417 17:06:50.763960   1 controller.go:2322] Error retrieving endpoints for the service idealab-api-g023: no endpointslices for target port 80 in service idealab-api-g023
W20260417 17:06:50.767985   1 controller.go:2322] Error retrieving endpoints for the service marketingopsapi-g023: no endpointslices for target port 8080 in service marketingopsapi-g023
I20260417 17:06:50.768855   1 main.go:112] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"pm-g023", Name:"ingress-master-apiautog023", UID:"17a391e8-af84-47c9-be06-92d9be060936", APIVersion:"networking.k8s.io/v1", ResourceVersion:"2575819426", FieldPath:""}): type: 'Normal' reason: 'AddedOrUpdated' Configuration for pm-g023/ingress-master-apiautog023 was added or updated

kubectl describe on a master ingress:

Name:             ingress-master-us1-rtn67
Labels:           <none>
Namespace:        pm-rtn67
Address:          20.232.80.204
Ingress Class:    lab-pm-public-nginx-pm-r09
Default backend:  <default>
TLS:
  SNI routes us1-rtn67.labs.aprimo.com
Rules:
  Host        Path  Backends
  ----        ----  --------
  *           *     <default>
Annotations:  nginx.org/mergeable-ingress-type: master
              nginx.org/path-regex: case_insensitive
Events:       <none>

kubectl describe on a minion:

λ kubernetes-workspace (wip/586041) $ k describe ing uls-rtn67-minion--us1-rtn67-labs-aprimo-com
Name:             uls-rtn67-minion--us1-rtn67-labs-aprimo-com
Labels:           app.kubernetes.io/instance=uls-rtn67
                  app.kubernetes.io/managed-by=Helm
                  app.kubernetes.io/name=uls
                  app.kubernetes.io/version=1.0.0
                  helm.sh/chart=uls-1.1.0
                  k8s.aprimo.com/billing.serviceCode=l19
                  k8s.aprimo.com/billing.subCategory=uls
                  k8s.aprimo.com/build.date=2026-04-15T23__30__39.000Z
                  k8s.aprimo.com/build.id=1079853
                  k8s.aprimo.com/build.number=uls-ci-master-20260415.1
                  k8s.aprimo.com/release.date=2026-04-16T14__19__04.000Z
                  k8s.aprimo.com/release.id=433167
                  k8s.aprimo.com/release.number=MO-Labs-V3-rtn67.3
                  k8s.aprimo.com/repo.name=UnifiedLoginService
                  k8s.aprimo.com/source.branch=refs__heads__master
                  k8s.aprimo.com/source.version=d740b57eb09689664ac69077e0be92453bd75456
Namespace:        pm-rtn67
Address:          20.232.80.204
Ingress Class:    lab-pm-public-nginx-pm-r09
Default backend:  <default>
Rules:
  Host                       Path  Backends
  ----                       ----  --------
  us1-rtn67.labs.aprimo.com
                             /login   uls-rtn67:80 ()
Annotations:                 meta.helm.sh/release-name: uls-rtn67
                             meta.helm.sh/release-namespace: pm-rtn67
                             nginx.org/mergeable-ingress-type: minion
                             nginx.org/path-regex: case_insensitive
                             nginx.org/proxy-buffer-size: 128k
                             nginx.org/proxy-buffers: 8 128k
Events:                      <none>
λ kubernetes-workspace (wip/586041) $

[rnemeth90]

I added additional details to the description.

[rnemeth90]

So here is an interesting finding... I created a new ingress class (new controllers, etc.) also using F5 OSS charts. I then created 600ish 'test' apps running alpine, each with a VirtualServer object (rather than an ingress). I then did a rollout of the ingress controller deployment. Each pod took just over 2 minutes to startup. That's with 600ish VirtualServer objects.. So scale that up to 8000 (or however many ingresses we have), and the time to startup scales linearly with it.

I had a thoery that maybe this was caused by using ingresses, rather than VirtualServer objects, but I believe that is now disproven.

Here is an example VS:

λ rnemeth-test $ k describe vs test-app-0998
Warning: short name "vs" could also match lower priority resource volumesnapshots.snapshot.storage.k8s.io
Name:         test-app-0998
Namespace:    rnemeth-test
Labels:       test-harness=nginx-ingress-startup
Annotations:  <none>
API Version:  k8s.nginx.org/v1
Kind:         VirtualServer
Metadata:
  Creation Timestamp:  2026-04-20T14:20:04Z
  Generation:          1
  Resource Version:    2584857718
  UID:                 91b36061-81d7-4809-bdf3-087ecc58a3f5
Spec:
  Host:                test-app-0998.labs.aprimo.com
  Ingress Class Name:  rnemeth-test-ingress
  Routes:
    Action:
      Pass:  app
    Path:    /
  Tls:
    Secret:  azure-pm-lab-tls-csi
  Upstreams:
    Name:     app
    Port:     80
    Service:  test-app-0998
Status:
  External Endpoints:
    Ip:     4.157.43.220
    Ports:  [80,443]
  Message:  Configuration for rnemeth-test/test-app-0998 was added or updated
  Reason:   AddedOrUpdated
  State:    Valid
Events:
  Type    Reason          Age    From                      Message
  ----    ------          ----   ----                      -------
  Normal  AddedOrUpdated  14m    nginx-ingress-controller  Configuration for rnemeth-test/test-app-0998 was added or updated
  Normal  AddedOrUpdated  14m    nginx-ingress-controller  Configuration for rnemeth-test/test-app-0998 was added or updated
  Normal  AddedOrUpdated  11m    nginx-ingress-controller  Configuration for rnemeth-test/test-app-0998 was added or updated
  Normal  AddedOrUpdated  8m57s  nginx-ingress-controller  Configuration for rnemeth-test/test-app-0998 was added or updated
λ rnemeth-test $

[hugolevino]

I am facing same issue in v5.4.1 with a cluster that has around 3000+ ingresses.

I tried different approaches:

• using Ingress Master/Minion strategy
• using VirtualServer/VirtualServerRoutes with RouteSelector

Both scenarios +20min startup

In my opnion issue is that rebuildHosts() is very expensive and is being called on every AddOrUpdateIngress event.

kubernetes-ingress/internal/k8s/configuration.go

Lines 468 to 487 in 10b86a5

	func (c *Configuration) AddOrUpdateIngress(ing *networking.Ingress) ([]ResourceChange, []ConfigurationProblem) {
	c.lock.Lock()
	defer c.lock.Unlock()
	key := getResourceKey(&ing.ObjectMeta)
	var validationError error
	if !c.hasCorrectIngressClass(ing) {
	delete(c.ingresses, key)
	} else {
	validationError = validateIngress(ing, c.isPlus, c.appProtectEnabled, c.appProtectDosEnabled, c.internalRoutesEnabled, c.snippetsEnabled, c.isDirectiveAutoadjustEnabled).ToAggregate()
	if validationError != nil {
	delete(c.ingresses, key)
	} else {
	c.ingresses[key] = ing
	}
	}
	changes, problems := c.rebuildHosts()

[vepatel]

hey @rnemeth90 thanks, I have had some time now to run tests and here are the results for 250 master and 1000 minion ingresses divided equally across 50 namespaces and it seems to be aligning with your observation

  All NIC pods ready in: 762.06s
  NIC pods ready:        3/3 (expected 3)
  Reload count:          1
  Last reload (ms):      1998
  Regular Ingresses:     0
  Master Ingresses:      250
  Minion Ingresses:      1000
  VirtualServers:        0

For regular 1000 regular Ingresses in 50 ns:

  All NIC pods ready in: 211.33s
  NIC pods ready:        3/3 (expected 3)
  Reload count:          76
  Last reload (ms):      2993
  Regular Ingresses:     1000
  Master Ingresses:      0
  Minion Ingresses:      0
  VirtualServers:        0

and thanks for the PR @hugolevino, we'll take a look at this!