API for Off-Cluster Gateways #3894

kflynn · 2025-07-01T17:44:21Z

As with GEP-3793, I'm sure this will be noncontroversial. 😇

/kind gep

NONE

k8s-ci-robot · 2025-07-01T17:44:27Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: kflynn
Once this PR has been reviewed and has the lgtm label, please assign robscott for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

geps/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

howardjohn · 2025-07-01T18:04:43Z

geps/gep-3792/index.md

+   meshed workloads in the cluster, including providing the OCG with the
+   information it needs to authenticate itself as a workload.
+
+2. The API must allow the mesh to be configured to accept requests from the


In mTLS there are 4 auth pieces. Each party provides credentials and verifies the peer certificates. We seem to only mention "OCG to provide credentials" and "Mesh to verify OCG".

We should require OCG to verify the workload (and the mesh to provide a credential, though this is probably obvious)

Yeah, this a good point: I'm at the point where I think of verifying the peer identity as a required piece of mTLS, but of course it really isn't, and I'll add that as a requirement.

howardjohn · 2025-07-01T18:06:20Z

geps/gep-3792/index.md

+  gatewayClassName: ocg-gateway-class
+  ...
+  mesh:   # All mesh-related configuration goes here
+    trustBundle:


I want to put in consideration adoption of ClusterTrustBundle which is now Beta in Kubernetes. It was designed exactly for this goal and is a perfect fit.

This isn't a "no", but... what would that look like? do we basically vendor the Go type for ClusterTrustBundle for codegen and hope that it doesn't change out from under us? 🤔

No, just an object ref to a ClusterTrustBundle. Or maybe not even a 'ref' needed, just a name (its cluster scoped so no namespace needed)

ClusterTrustBundle, if I understand correctly, is an alpha feature from K8s 1.27 to K8s 1.32, which is to say that in nearly every version supported by Gateway API, if you don't have the feature gate enabled, the ClusterTrustBundle resource type won't exist and there will be no way to refer to a ClusterTrustBundle resource in the Gateway resource... which is why I'm asking what using ClusterTrustBundle would look like. 🙂

(I really, really do not want Gateway API to ship definitions for resources defined by the Kubernetes core, ever.)

ClusterTrustBundle is also a cluster-scoped resource, which feels a bit less than ideal, but that's a much much smaller problem in my mind.

This is also an alpha (experimental) feature. It seems problematic that an experimental feature cannot depend on a Beta feature, and instead we are just re-inventing the exact thing ClusterTrustBundle was designed for

Yeah, this isn't a question about semantics, I kind of agree with you in principle. 🙂 I'm asking about mechanics here: how, technically, can we use ClusterTrustBundle on a cluster where the APIserver does not have any resource type defined under that name?

Saying "you have use K8s 1.33 or enable the feature gate" feels problematic.

The options here are:

we only add trustBundle inline for now, with a note about how we will add ClusterTrustBundle in the future.

we add both for now, with a note that once ClusterTrustBundle is more available in Gateway API supported versions, we will remove the inline trustBundle. We can get away with that as long as this resource is Experimental.

We only add trustBundle, and say "run 1.33 or later, or enable an alpha feature to use this". This would require us to specify the behavior for implementations; since this has a FeatureName anyway, there is a mechanism to say "requires k8s 1.33 or alpha feature enabled to work", although we've never done that before.

Because Gateway API's claimed Kubernetes support is five versions, the last would mean that we would need to wait nearly two years before bringing this API to GA in options 2 or 3. That feels like a longer timeline than people would expect for this. Unless we are okay with doing 3 and adding features that are not supported across all supported Kubernetes versions, which does not seem optimal to me.

@robscott for his thoughts on this one too.

Actually, I just checked, and ClusterTrustBundle is only beta, and not enabled by default in v1.33, so this would be even longer. (See https://kubernetes.io/docs/reference/access-authn-authz/certificate-signing-requests/#cluster-trust-bundles for the reference)

At a minimum, if ClusterTrustBundle moves to Stable in v1.34, then the only time we could allow it with no caveats is as of Kubernetes v1.38, which is in just over two years.

In short; my thoughts here are that we can add a ClusterTrustBundle reference with a feature name and a version caveat at best. But we need something for before then as well.

We have similar (though less restrictive) versioning issues for using ClusterTrustBundle within GKE. The approach we took was to make a CustomResource ("GKEClusterTrustBundle") that had exactly the same shape as ClusterTrustBundle, and have our controllers understand both types.

Elsewhere in GW API when we've dealt with configure CA certs we've started with config maps with the idea that we'd add support for ClusterTrustBundle as it became more broadly available. That seems like a reasonable approach here, with the distinction that ClusterTrustBundle is likely going to go to GA in v1.35, so it would be good to consider that our default starting point with ConfigMap as the legacy backfill option from the outset.

howardjohn · 2025-07-01T18:14:53Z

geps/gep-3792/index.md

+- The mesh MUST use an mTLS certificate ultimately signed by a certificate in
+  the trust bundle provided to the OCG.
+
+- The OCG MUST send the `ocg.gateway.networking.k8s.io/v1` ALPN protocol


This applies to the whole proposal in general but specifically this line. There are 3 OCG implementations today. I would think to make this viable we need at least 2 to agree to this.

I know that, unless things have changed, using a custom ALPN is the hill at least one cloud will die on and will never support

I'm not comfortable making this a MUST. Custom ALPNs are certainly a way to signal mTLS, but different implementations may use tunneling protocols like HBONE instead.

@keithmattix and I discussed this on Slack a bit, since I'm very uncomfortable with making this a MAY and then having, say, the OCG expecting ALPN, the mesh expecting HBONE, but they both believe that they "support GEP-3792 OCG".

As Keith noted on Slack, though, it doesn't really make sense to require a mesh to only support one mechanism for solving the protocol problem, and it follows that it probably does make sense to add an explicit field to the Gateway and Mesh resources to say what the signalling mechanism is. That also permits warning Chihiro if the Gateway and Mesh disagree on what's offered. I'll add that.

Also, to @howardjohn's point: we explicitly say in "Graduation Criteria" section that we require at least two OCG implementations and at least two mesh implementations. Leaving aside the choice of hills upon which to die... 🙂

it follows that it probably does make sense to add an explicit field to the Gateway and Mesh resources to say what the signalling mechanism is. That also permits warning Chihiro if the Gateway and Mesh disagree on what's offered. I'll add that.

This makes me a bit nervous in that we could end up with n different protocols that OCGs would need to support here. Maybe this is unavoidable, but I'd really like to minimize the variations here if we can. The larger the number of variations here, the less likely it is that an OCG is going to be able to support them.

keithmattix · 2025-07-01T18:56:53Z

geps/gep-3792/index.md

+  that the OCG should trust when communicating with meshed workloads in the
+  cluster.
+
+- The trust bundle in the Mesh resource will define the CA certificate(s)


I may have missed this in another GEP, but who's creating the Mesh resource: the user or the mesh controller? How does this get plumbed through? IIRC, this ambiguity was one of the reasons we didn't originally create a mesh resource

I was kinda hoping that maybe someone else would write the Mesh-resource GEP. 😉 It probably makes sense to include it in this one, though.

My thought here is that Chihiro creates the Mesh resource, and that the Mesh resource includes a controllerName field that identifies its owner. (I don't think that a MeshClass resource makes sense now.) If Chihiro doesn't create the Mesh resource, then the mesh won't enable OCG (because it won't have the trust bundle, etc.).

Would it make sense for most Mesh installations to be bundled with a Mesh resource that users can configure?

keithmattix · 2025-07-01T19:00:07Z

geps/gep-3792/index.md

+- The mesh MUST use an mTLS certificate ultimately signed by a certificate in
+  the trust bundle provided to the OCG.
+
+- The OCG MUST send the `ocg.gateway.networking.k8s.io/v1` ALPN protocol


I'm not comfortable making this a MUST. Custom ALPNs are certainly a way to signal mTLS, but different implementations may use tunneling protocols like HBONE instead.

keithmattix · 2025-07-01T19:05:09Z

geps/gep-3792/index.md

+   for [Chihiro] and [Ian], _and_ it limits us to only having entire
+   namespaces meshed.
+
+6. Add a label selector to the Gateway resource and declare that Routes with


One other option that randomly popped in my head (not saying it's a great one): add some metadata to backendRef. This shouldn't at all take any of backendTLSPolicy's scope, but it's a way for the route to say "this backend is meshed"

I would really like to arrange things so that Ana needn't take on that burden. But let's keep it in mind.

keithmattix · 2025-07-01T19:06:21Z

geps/gep-3792/index.md

+Gateway resource, and to create a Mesh resource which will also have a trust
+bundle:
+
+- The trust bundle in the Gateway resource will define the CA certificate(s)


~~A single trust bundle implies that multitenancy (i.e. multiple meshes with different trust domains served by the same gateway) is a non-goal for now?~~

I see below that it's a list of secrets; should this be pluralized?

"Trust bundle" to mean "one or more CA certificates that you're going to trust" is pretty well-established in the PKI world. Pluralizing it would imply having multiple lists, rather than multiple certificates; I don't really think that that makes sense.

Oh wait I was interpreting this in light of John's comment about using CTB. In which case it got sort of confusing

To respond to your overstruck comment: you are correct, support for multiple meshes in the same cluster is a non-goal in this pass. I'll make that explicit as well.

geps/gep-3792/index.md

keithmattix · 2025-07-01T19:08:48Z

geps/gep-3792/index.md

+
+```yaml
+apiVersion: networking.x-k8s.io/v1
+kind: Mesh


s/Mesh/XMesh?

Uhhhhhhhhh wow, that's a great question. Yeah, probably so.

+1, good catch.

geps/gep-3792/index.md

youngnick · 2025-07-02T03:14:27Z

I won't get much of a chance to watch this one further, but the main point I wanted to add is that, because this is TLS config for a Gateway, it's important that we define the interaction (or lack thereof) with consumer BackendTLSPolicy as per #3875.

FTR, I think that saying "they are for different things, the only interaction is to end up with TLS-in-TLS, which is dumb, so you should avoid that". is fine by me.

karthikbox · 2025-07-07T17:02:49Z

geps/gep-3792/index.md

+   for [Chihiro] and [Ian], _and_ it limits us to only having entire
+   namespaces meshed.
+
+6. Add a label selector to the Gateway resource and declare that Routes with


just a thought - how about an explicit mesh MTLS policy that targetRefs a service? the meshMTLS policy would indicate to both the OCG and Mesh Clients that they need to use Mesh MTLS to talk to the service.
the explicit meshMTLS policy can be either created by the user or the mesh controller creates a default one for every service that is meshed. The user would override the default one if they wish to tweak the default settings - for e.g override the SAN checks, or SNI or disable/enable MTLS perhaps (any kind of outbound setting).

FWIW in Istio (and probably linkerd and every other mesh), the mesh TLS is a property of a workload (Pod) not service. So workloads within the same Service may not actually have the same properties

@howardjohn I don't think that we should expect Gateways to be able to differentiate TLS configuration per workload, that would be a huge increase in scope/complexity for many Gateway implementations. Instead, it seems simpler to focus on the common attributes that Gateways can understand (Service, Namespace, etc).

(You may not have been suggesting that Gateways should be able to understand per-Pod TLS config, so sorry if I'm misunderstanding here).

I am not necessarily suggesting we do, but to properly handle all cases you would need to

karthikbox · 2025-07-07T17:11:30Z

geps/gep-3792/index.md

+  as application-level mTLS that they should treat as an opaque data stream,
+  or they may simply refuse it. This alternative, therefore, both shifts the
+  entire burden of implementation to the meshes, and probably makes it
+  impossible to correctly handle application-level mTLS.


just wanted to mention that GKE Mesh is a notable exception (atleast partially). We use simply use bare mtls connection. And yes, we're subject to the downsides mentioned above. But the upside of interoperability between various GCP (off-GCP) data planes with GKE Mesh workloads using standard MTLS was/is deemed to be more important (i.e don't run into the protocol problem).

my intention with this comment is NOT to say custom protocol shouldn't be supported, but to point out that there are implementations that just use bare/standard mtls.

karthikbox · 2025-07-07T17:19:53Z

geps/gep-3792/index.md

+  gatewayClassName: ocg-gateway-class
+  ...
+  mesh:   # All mesh-related configuration goes here
+    trustBundle:


can we allow for some kind of extensibility here to indicate Platform MTLS? The Platform MTLS is a feature of the cloud platform (e.g GKE Managed Workload Certificates) or the mesh vendor (e.g Istio) where the certs and trust bundle is provisioned by the platform for each workload and OCG automatically. In such cases, User need not configure the k8s Secret explicitly reducing toil.

same comment for the counterpart in Mesh resource.

or would clusterTrustBundle (or the gkeClusterTrustBundle) be able carry the info for the platform MTLS? @ahmedtd to confirm. If that's the case, then supporting clusterTrustBundle be the path forward for indicating platform MTLS.

GKE Managed Workload Certificates uses GKEClusterTrustBundle, which is an identical copy of ClusterTrustBundle as a custom resource. We intend to migrate to ClusterTrustBundle for in-cluster distribution of trust anchors.

@ahmedtd @karthikbox just to make sure I'm following this thread correctly, it sounds like ClusterTrustBundle (or an equivalent) would work well here, even for Platform mTLS?

LiorLieberman · 2025-07-08T16:44:07Z

geps/gep-3792/index.md

+3. Add a field to the Route resource that indicates whether the Route is
+   meshed. This feels like quite a bit of an imposition on [Ana] (and, again,
+   if we do our jobs correctly, [Ana] shouldn't need to think about this).


Maybe I am missing something, cant we use parentRef: <ANY_SERVICE>?

robscott

Thanks @kflynn, this is really well written, appreciate the time you've put into this!

robscott · 2025-07-10T05:44:35Z

geps/gep-3792/index.md

+  that the OCG should trust when communicating with meshed workloads in the
+  cluster.
+
+- The trust bundle in the Mesh resource will define the CA certificate(s)


Would it make sense for most Mesh installations to be bundled with a Mesh resource that users can configure?

robscott · 2025-07-10T05:47:51Z

geps/gep-3792/index.md

+- The OCG MUST use an mTLS certificate ultimately signed by a certificate in
+  the trust bundle provided to the mesh.


I thought this trust bundle was configured on the mesh resource? This may be saying that, but it's a bit confusing to me.

robscott · 2025-07-10T05:49:06Z

geps/gep-3792/index.md

+- The OCG MUST use an mTLS certificate ultimately signed by a certificate in
+  the trust bundle provided to the mesh.
+
+- The mesh MUST use an mTLS certificate ultimately signed by a certificate in


Can we call this a client cert? Is the Gateway supposed to generate this on their own, or can/should a mesh provide client cert(s) for Gateway(s) to use?

robscott · 2025-07-10T05:51:56Z

geps/gep-3792/index.md

+- The mesh MUST use an mTLS certificate ultimately signed by a certificate in
+  the trust bundle provided to the OCG.
+
+- The OCG MUST send the `ocg.gateway.networking.k8s.io/v1` ALPN protocol


it follows that it probably does make sense to add an explicit field to the Gateway and Mesh resources to say what the signalling mechanism is. That also permits warning Chihiro if the Gateway and Mesh disagree on what's offered. I'll add that.

This makes me a bit nervous in that we could end up with n different protocols that OCGs would need to support here. Maybe this is unavoidable, but I'd really like to minimize the variations here if we can. The larger the number of variations here, the less likely it is that an OCG is going to be able to support them.

robscott · 2025-07-10T05:55:27Z

geps/gep-3792/index.md

+The discovery problem is that not every workload in the cluster is required to
+be meshed, which means that the OCG needs a way to know whether a given
+connection must participate in the mesh or not. In practice, this isn't
+actually a question of _workloads_ but of _Routes_: the point of interface
+between a Gateway and a workload in the cluster is not a Pod or a Service, but
+rather a Route.


It seems like we care less about which workloads have a mesh Route, and more about which workloads have automatic mTLS provided by a Mesh. My understanding is that most mesh implementations provide automatic mTLS/mesh transport for a broader set of workloads, and that's what we want Gateways to care about when connecting to backends with mTLS following this mesh config.

robscott · 2025-07-10T05:58:19Z

geps/gep-3792/index.md

+   for [Chihiro] and [Ian], _and_ it limits us to only having entire
+   namespaces meshed.
+
+6. Add a label selector to the Gateway resource and declare that Routes with


I'm trying to understand the mechanics of this. Does this selector only apply to Routes that are attached to each relevant Gateway? Are Gateway implementations supposed to look at mesh Routes?

robscott · 2025-07-10T06:01:41Z

geps/gep-3792/index.md

+   for [Chihiro] and [Ian], _and_ it limits us to only having entire
+   namespaces meshed.
+
+6. Add a label selector to the Gateway resource and declare that Routes with


@howardjohn I don't think that we should expect Gateways to be able to differentiate TLS configuration per workload, that would be a huge increase in scope/complexity for many Gateway implementations. Instead, it seems simpler to focus on the common attributes that Gateways can understand (Service, Namespace, etc).

(You may not have been suggesting that Gateways should be able to understand per-Pod TLS config, so sorry if I'm misunderstanding here).

robscott · 2025-07-10T06:04:38Z

geps/gep-3792/index.md

+  gatewayClassName: ocg-gateway-class
+  ...
+  mesh:   # All mesh-related configuration goes here
+    trustBundle:


Elsewhere in GW API when we've dealt with configure CA certs we've started with config maps with the idea that we'd add support for ClusterTrustBundle as it became more broadly available. That seems like a reasonable approach here, with the distinction that ClusterTrustBundle is likely going to go to GA in v1.35, so it would be good to consider that our default starting point with ConfigMap as the legacy backfill option from the outset.

robscott · 2025-07-10T06:05:32Z

geps/gep-3792/index.md

+spec:
+  gatewayClassName: ocg-gateway-class
+  ...
+  mesh:   # All mesh-related configuration goes here


Why is this on the Gateway? This feels like configuration that could live on the Mesh resource?

robscott · 2025-07-10T06:11:51Z

geps/gep-3792/index.md

+    selector:
+      matchLabels:
+        linkerd.io/inject: "true"       # or whatever label is appropriate


As I understand it, this selector is intended to select namespaces and routes. I'm very nervous about a multi-resource selector here though. My understanding is that the primary reason for considering Routes here is when only a portion of a namespace is part of a mesh. Would it be reasonable to keep a label selector for namespaces, and then opt-out per Service or Service Port?

The rationale for that would be:

The selector is clearly for a single kind of resource

We still provide a way to opt-out subsets of a namespace

We're using granularities that are already well understood by a Gateway (my guess is that it will be difficult for some implementations to configure mTLS per route rule, but quite simple per backend as we're already doing with BackendTLSPolicy)

Signed-off-by: Flynn <[email protected]>

…on-goal Signed-off-by: Flynn <[email protected]>

Signed-off-by: Flynn <[email protected]>

k8s-ci-robot added release-note-none Denotes a PR that doesn't merit a release note. kind/gep PRs related to Gateway Enhancement Proposal(GEP) cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Jul 1, 2025

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 1, 2025

k8s-ci-robot requested review from gcs278 and robscott July 1, 2025 17:44

k8s-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Jul 1, 2025

howardjohn reviewed Jul 1, 2025

View reviewed changes

keithmattix reviewed Jul 1, 2025

View reviewed changes

kflynn changed the title ~~Add API for Off-Cluster Gateways~~ API for Off-Cluster Gateways Jul 1, 2025

karthikbox reviewed Jul 7, 2025

View reviewed changes

LiorLieberman reviewed Jul 8, 2025

View reviewed changes

robscott reviewed Jul 10, 2025

View reviewed changes

robscott mentioned this pull request Jul 11, 2025

[GEP-2162] Updated a new field on supported features inference from boolean to enum and remove from report. #3885

Merged

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 17, 2025

kflynn added 6 commits July 21, 2025 23:19

API for GEP-3792

4547991

Signed-off-by: Flynn <[email protected]>

Typo

c70eccb

Signed-off-by: Flynn <[email protected]>

DEFER for Gateway API 1.4.

a5324f8

Signed-off-by: Flynn <[email protected]>

Note that an OCG simultaneously operating with multiple meshes is a n…

028be91

…on-goal Signed-off-by: Flynn <[email protected]>

Wordsmithing, and tighten up language around mTLS verification

42c6cc6

Signed-off-by: Flynn <[email protected]>

WIP: talk about protocol selection

7a2111a

Signed-off-by: Flynn <[email protected]>

kflynn force-pushed the flynn/gep-3792 branch from a1bc550 to 7a2111a Compare July 22, 2025 03:21

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 22, 2025

kflynn marked this pull request as draft July 22, 2025 03:24

k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 22, 2025

kflynn mentioned this pull request Jul 22, 2025

API for Default Gateways #3887

Open

karthikbox mentioned this pull request Jul 22, 2025

API for Off-Cluster Gateways (superseeds #3894) #3941

Open

		- The OCG MUST use an mTLS certificate ultimately signed by a certificate in
		the trust bundle provided to the mesh.

API for Off-Cluster Gateways #3894

Are you sure you want to change the base?

API for Off-Cluster Gateways #3894

Conversation

kflynn commented Jul 1, 2025

Uh oh!

k8s-ci-robot commented Jul 1, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

youngnick Jul 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kflynn Jul 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

youngnick commented Jul 2, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

youngnick Jul 2, 2025 •

edited

Loading

kflynn Jul 1, 2025 •

edited

Loading

karthikbox Jul 7, 2025 •

edited

Loading

LiorLieberman Jul 8, 2025 •

edited

Loading