WIP: Add support for dual stack load balancers by nrb · Pull Request #1313 · kubernetes/cloud-provider-aws

nrb · 2025-12-02T21:18:23Z

What type of PR is this?
/kind feature

What this PR does / why we need it:

Adds basic support for configuring the IP address family on Services.

This functionality overlaps with the ALBC somewhat, but the goal here is to provide much more basic functionality, not the full scope of ALBC.

Which issue(s) this PR fixes:

Fixes #1219

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

Introduce `service.beta.kubernetes.io/aws-load-balancer-ip-address-type` to enable dual stack Services.

k8s-ci-robot · 2025-12-02T21:18:26Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

k8s-ci-robot · 2025-12-02T21:18:31Z

This issue is currently awaiting triage.

If cloud-provider-aws contributors determine this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

nrb · 2025-12-02T21:32:24Z

/test pull-cloud-provider-aws-test

damdo · 2025-12-09T19:49:55Z

Hey @kmala do we know what's happening with the Netlify checks? It looks like they are broken across many open PRs. TY

damdo · 2025-12-10T07:50:27Z

/retest

nrb · 2025-12-11T16:00:12Z

/retest

nrb · 2025-12-12T14:00:46Z

/retest

nrb · 2025-12-12T16:06:24Z

/test pull-cloud-provider-aws-test

One more run to see if this is a rate limiting issue or some sort of genuine issue with my code.

tthvo

I tested with a simple service below:

apiVersion: v1
kind: Service
metadata:
  name: wordpress-mysql
  labels:
    app: wordpress
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: nlb
    service.beta.kubernetes.io/aws-load-balancer-ip-address-type: dualstack
spec:
  ipFamilyPolicy: PreferDualStack
  ports:
    - port: 3306
  selector:
    app: wordpress
  type: LoadBalancer

The CCM does create the dualstack NLB as expected 🚀

$ kubectl -n myns get svc/wordpress-mysql -o yaml | yq .status.loadBalancer.ingress
- hostname: <id>.elb.us-east-1.amazonaws.com

$ nslookup <id>.elb.us-east-1.amazonaws.com
# nslookup shows both public IPv4 EIP and IPv6 GUA addresses

$ aws elbv2 describe-load-balancers --names=<nlb-id> --query='LoadBalancers[0].[Type,IpAddressType,Scheme]'
[
    "network",
    "dualstack",
    "internet-facing"
]

pkg/providers/v1/aws_validations_test.go

pkg/providers/v1/aws.go

nrb · 2026-01-16T15:08:12Z

/test pull-cloud-provider-aws-test

nrb · 2026-01-22T22:41:32Z

/test pull-cloud-provider-aws-test

nrb · 2026-01-22T22:42:51Z

/test pull-cloud-provider-aws-check

nrb · 2026-01-23T17:38:16Z

/test pull-cloud-provider-aws-test
/test pull-cloud-provider-aws-check

nrb · 2026-01-26T15:29:02Z

@kmala I'm going to do some manual testing with this PR starting today before I mark it as ready for review, but would you be able to give it a look by any chance?

nrb · 2026-02-24T19:56:46Z

I've updated the change to only use the Kubernetes API fields in service.Spec rather than annotations. I think this keeps the implementation straightforward, and there should be nothing in this PR that blocks us from adding support for more granular settings via annotations in the future.

- Create load balancers according to the Kubernetes Service API - Do not reconcile Services with spec.loadBalancerClass defined. These are handled by external controllers, and the CCM should ignore them. Signed-off-by: Nolan Brubaker <nolan@nbrubaker.com> Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> Signed-off-by: Nolan Brubaker <nolan@nbrubaker.com>

nrb · 2026-02-24T20:09:45Z

govulncheck notice will be resolved by pulling kubernetes/kubernetes#136820 in during a kube bumpl.

kmala · 2026-02-25T05:59:58Z

pkg/providers/v1/aws.go

 			internalELB,
 			annotations,
 			securityGroups,
+			apiService,


nit: if we are passing the entire service then we can skip passing other params like service name, annotation right?

pkg/providers/v1/aws.go

tthvo

I just have a few questions 🙏 But overall, I think we are having great progress!

pkg/providers/v1/aws.go

pkg/providers/v1/aws_loadbalancer.go

tthvo · 2026-02-25T06:35:12Z

pkg/providers/v1/aws.go

 		err = c.updateInstanceSecurityGroupsForNLB(ctx, loadBalancerName, instances, subnetCidrs, sourceRangeCidrs, v2Mappings)
 		if err != nil {
 			klog.Warningf("Error opening ingress rules for the load balancer to the instances: %q", err)
 			return nil, err
 		}


We should consider/allow IPv6 CIDRs when updating the instance's security groups (to allow traffic from NLB), right? IIUC, there are 2 types of CIDRs here:

sourceRangeCidrs: Already defined on line 2396-2405

subnetCidrs: IPv6 CIDRs can be extracted from subnet info returned by AWS.

I guess that means we need to update the following functions to set the Ipv6Range field correctly?

cloud-provider-aws/pkg/providers/v1/aws_loadbalancer.go

Line 957 in b6c9d76

func (c *Cloud) updateInstanceSecurityGroupsForNLB(ctx context.Context, lbName string, instances map[InstanceID]*ec2types.Instance, subnetCIDRs []string, clientCIDRs []string, portMappings []nlbPortMapping) error {

cloud-provider-aws/pkg/providers/v1/aws_loadbalancer.go

Line 1049 in b6c9d76

func (c *Cloud) updateInstanceSecurityGroupForNLBTraffic(ctx context.Context, sgID string, sgPerms IPPermissionSet, ruleDesc string, protocol string, ports sets.Set[int32], cidrs []string) error {

tthvo · 2026-02-25T06:42:40Z

pkg/providers/v1/aws_loadbalancer.go

+	if service.Spec.IPFamilyPolicy != nil && *service.Spec.IPFamilyPolicy == v1.IPFamilyPolicySingleStack {
+		if serviceRequestsIPv6(service) {
+			return nil, fmt.Errorf("single stack IPv6 is not supported for network load balancers")
+		}
+	}


I wonder if we can:

Place this validation in pkg/providers/v1/aws_validations.go

Additionally, we should add a validation that only allows dualstack settings (i.e. ipFamilies: [IPv4, IPv6], ipFamilyPolicy: RequireDualStack) when LB type is NLB.

WDYT?

Yep, makes sense.

pkg/providers/v1/aws_loadbalancer.go

tthvo · 2026-02-25T06:58:31Z

docs/service_controller.md

+
+Some limitations to be aware of when using dual stack load balancers:
+
+- The `spec.ipFamilies` field can have a second family added or removed, but the first entry is immutable after Service creation.


Maybe, we should mention what would happen if the secondary family is removed? Like what will be the state of:

The Load Balancer

The Security Groups (those for LB and those for instances)

Target Group

(Optional) Listeners

tthvo · 2026-02-25T06:59:14Z

docs/service_controller.md

+        - IPv6
+        - IPv4


nit: The list indentation is a lot more than usual. Let's format it a bit 😁

remove loadbalancerclass check Signed-off-by: Nolan Brubaker <nolan@nbrubaker.com>

pass IPv6 source ranges to security group rule Signed-off-by: Nolan Brubaker <nolan@nbrubaker.com>

Set the IP address type on load balancer creation Signed-off-by: Nolan Brubaker <nolan@nbrubaker.com>

damdo · 2026-02-26T12:15:29Z

/retest

damdo · 2026-02-26T13:09:22Z

/test pull-cloud-provider-aws-e2e-kubetest2-quick

damdo · 2026-02-27T10:09:17Z

/test pull-cloud-provider-aws-e2e-kubetest2-quick

Signed-off-by: Nolan Brubaker <nolan@nbrubaker.com>

Check for empty fields in validation logic Signed-off-by: Nolan Brubaker <nolan@nbrubaker.com>

nrb · 2026-03-03T23:46:14Z

/test pull-cloud-provider-aws-test

nrb · 2026-03-03T23:47:17Z

Current e2e failure is legit - the provisioned cluster is not set up for IPv6.

   STEP: creating LoadBalancer service in Kubernetes @ 03/03/26 17:45:37.338
  I0303 17:45:37.386843 32011 loadbalancer.go:508] Unexpected error: 
      <*errors.errorString | 0xc000321140>: 
      failed to create LoadBalancer Service "lbconfig-test-clb-ds": Service "lbconfig-test-clb-ds" is invalid: spec.ipFamilies[1]: Invalid value: "IPv6": not configured on this cluster
      {
          s: "failed to create LoadBalancer Service \"lbconfig-test-clb-ds\": Service \"lbconfig-test-clb-ds\" is invalid: spec.ipFamilies[1]: Invalid value: \"IPv6\": not configured on this cluster",
      }
  [FAILED] in [It] - /home/prow/go/src/k8s.io/cloud-provider-aws/tests/e2e/loadbalancer.go:508 @ 03/03/26 17:45:37.387
  STEP: Destroying namespace "cloud-provider-aws-593" for this suite. @ 03/03/26 17:45:37.387
  << Timeline

Will work on getting support for that plumbed in to kubetest-ec2.

Remove LoadBalancerClass check Signed-off-by: Nolan Brubaker <nolan@nbrubaker.com>

k8s-ci-robot · 2026-03-04T15:01:51Z

@nrb: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-cloud-provider-aws-e2e	`2dd0dac`	link	true	`/test pull-cloud-provider-aws-e2e`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. release-note Denotes a PR that will be considered when it comes time to generate release notes. labels Dec 2, 2025

k8s-ci-robot added kind/feature Categorizes issue or PR as related to a new feature. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Dec 2, 2025

k8s-ci-robot requested review from bridgetkromhout and cheftako December 2, 2025 21:18

k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Dec 2, 2025

nrb force-pushed the dual-stack-lb branch from ba4b93b to 83110d8 Compare December 11, 2025 19:04

tthvo reviewed Jan 7, 2026

View reviewed changes

pkg/providers/v1/aws_validations_test.go Outdated Show resolved Hide resolved

pkg/providers/v1/aws.go Outdated Show resolved Hide resolved

nrb force-pushed the dual-stack-lb branch from 83110d8 to ffbb4b8 Compare January 16, 2026 13:58

k8s-ci-robot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Feb 24, 2026

nrb marked this pull request as ready for review February 24, 2026 19:55

k8s-ci-robot requested review from olemarkus and yue9944882 February 24, 2026 19:55

nrb force-pushed the dual-stack-lb branch from db6cfcf to 19b6337 Compare February 24, 2026 19:58

kmala reviewed Feb 25, 2026

View reviewed changes

tthvo reviewed Feb 25, 2026

View reviewed changes

nrb added 3 commits February 25, 2026 16:10

fixup! Add support for dual networking stack services

9261ed5

remove loadbalancerclass check Signed-off-by: Nolan Brubaker <nolan@nbrubaker.com>

fixup! Add support for dual networking stack services

d423cd3

pass IPv6 source ranges to security group rule Signed-off-by: Nolan Brubaker <nolan@nbrubaker.com>

fixup! Add support for dual networking stack services

4f313e4

Set the IP address type on load balancer creation Signed-off-by: Nolan Brubaker <nolan@nbrubaker.com>

nrb added 5 commits February 27, 2026 16:58

fixup! Add support for dual networking stack services

30d2e78

Signed-off-by: Nolan Brubaker <nolan@nbrubaker.com>

fixup! Add support for dual networking stack services

41225bb

Signed-off-by: Nolan Brubaker <nolan@nbrubaker.com>

fixup! Add support for dual networking stack services

4655e60

Signed-off-by: Nolan Brubaker <nolan@nbrubaker.com>

Add some e2e test cases

4d88132

Signed-off-by: Nolan Brubaker <nolan@nbrubaker.com>

Fix ICMPv6 handling

a8dca77

Signed-off-by: Nolan Brubaker <nolan@nbrubaker.com>

k8s-ci-robot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Mar 2, 2026

fixup! Add support for dual networking stack services

21f8cdd

Check for empty fields in validation logic Signed-off-by: Nolan Brubaker <nolan@nbrubaker.com>

fixup! Add support for dual networking stack services

2dd0dac

Remove LoadBalancerClass check Signed-off-by: Nolan Brubaker <nolan@nbrubaker.com>

nrb force-pushed the dual-stack-lb branch from 1b3815c to 2dd0dac Compare March 4, 2026 14:20

alebedev87 mentioned this pull request Mar 5, 2026

[WIP] NE-2421: Support dual-stack IngressController on AWS openshift/cluster-ingress-operator#1376

Open


		Some limitations to be aware of when using dual stack load balancers:

		- The `spec.ipFamilies` field can have a second family added or removed, but the first entry is immutable after Service creation.

Conversation

nrb commented Dec 2, 2025

Uh oh!

k8s-ci-robot commented Dec 2, 2025

Uh oh!

k8s-ci-robot commented Dec 2, 2025

Uh oh!

nrb commented Dec 2, 2025

Uh oh!

damdo commented Dec 9, 2025

Uh oh!

damdo commented Dec 10, 2025

Uh oh!

nrb commented Dec 11, 2025

Uh oh!

nrb commented Dec 12, 2025

Uh oh!

nrb commented Dec 12, 2025

Uh oh!

tthvo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

nrb commented Jan 16, 2026

Uh oh!

nrb commented Jan 22, 2026

Uh oh!

nrb commented Jan 22, 2026

Uh oh!

nrb commented Jan 23, 2026

Uh oh!

nrb commented Jan 26, 2026

Uh oh!

nrb commented Feb 24, 2026

Uh oh!

nrb commented Feb 24, 2026

Uh oh!

kmala Feb 25, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

tthvo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

tthvo Feb 25, 2026

Choose a reason for hiding this comment

Uh oh!

tthvo Feb 25, 2026

Choose a reason for hiding this comment

Uh oh!

tthvo Feb 25, 2026

Choose a reason for hiding this comment

Uh oh!

nrb Feb 25, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

tthvo Feb 25, 2026

Choose a reason for hiding this comment

Uh oh!

tthvo Feb 25, 2026

Choose a reason for hiding this comment

Uh oh!

damdo commented Feb 26, 2026

Uh oh!

damdo commented Feb 26, 2026

Uh oh!

damdo commented Feb 27, 2026

Uh oh!

nrb commented Mar 3, 2026

Uh oh!

nrb commented Mar 3, 2026

Uh oh!

k8s-ci-robot commented Mar 4, 2026

Uh oh!

Reviewers

Assignees