Run Conformance Tests on implementations in CI

[arkodg]

What would you like to be added:
Run the entire conformance test suite when a PR is raised on Gateway API implementations, but run it as a non blocking step.

Why this is needed:
As a developer writing conformance tests, I do not how to test my changes.

[arkodg]

Ive been creating my k8s cluster and running this script
hack/run-conformance-envoy-gateway.sh

#!/usr/bin/env bash

set -euo pipefail

# Deploy latest version of Envoy Gateway and Gateway API CRDs
kubectl apply -f https://github.com/envoyproxy/gateway/releases/download/latest/install.yaml

# Wait for Envoy Gateway to be ready
kubectl wait --timeout=5m -n gateway-system deployment/gateway-api-admission-server --for=condition=Available
kubectl wait --timeout=5m -n envoy-gateway-system deployment/envoy-gateway --for=condition=Available
kubectl wait --timeout=5m -n gateway-system job/gateway-api-admission --for=condition=Complete

# Deploy GatewayClass
kubectl apply -f - <<EOF
kind: GatewayClass
apiVersion: gateway.networking.k8s.io/v1beta1
metadata:
  name: envoy-gateway
spec:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller
EOF

# Run conformance test
go test -v ./conformance/... --run TestConformance/HTTPRouteHostnameIntersection  --gateway-class=envoy-gateway --debug=true

would help the community if something like this is incorporated into the CI

Sucessful conformance tests take around ~5min to finish today so running multiple in parallel on multiple implementations shouldnt slow down the CI too much imho.

A suggestion for acceptable criteria could be - implementation must be passing all conformance tests for it to be added into the CI

[arkodg]

another suggestion is to specify implementation specific steps in https://gateway-api.sigs.k8s.io/concepts/conformance/?h=conf#3-conformance-tests and linking this page to the PR template #1733 making it a one click discovery (CI would be the ideal auto magic way)

[shaneutt]

So interestingly enough we were already planning on this and actively working on this (the https://github.com/kong/blixt project (which is being donated to Kubernetes SIGs) is related as we intend to plug that implementation into CI).

You can find the relevant announcement #1706, but after searching I don't think we had a tracking issue for it so this works as the tracking issue, thank you. 👍

[arkodg]

@shaneutt I completely missed that discussion, thanks for surfacing it. This is great that Blixt will be incorporated into the CI. My follow up questions are

• Will Blixt support L7 APIs (HTTPRoute) ?
• If yes, any rough estimate timelines on when this will happen

The velocity of contributions for conformance related PRs should increase once ^ happens.
Speaking from personal experience, after introducing a regression in the conformance test suite, im hesitant to add more tests or review PRs without knowing that the PR has been tested with a known implementation

[shaneutt]

* Will Blixt support L7 APIs (HTTPRoute) ?
* If yes, any rough estimate timelines on when this will happen

Currently we don't have plans to do this, though it's not the first time it's been asked.

For right this moment, we're going to focus on Layer4 as that will get a fair amount of the ground work covered, while also having the side benefit of driving L4 conformance so that we can work on getting UDPRoute and TCPRoute matured to BETA sooner. Once we have that groundwork done, we could definitely have a discussion about the possibilities for L7 support. Just note that we're using the Linux kernel for the dataplane in Blixt because we consider this to be vendor neutral and we absolutely want to stay away from playing favorites with well established base tech like envoy or nginx, and that makes adding L7 support potentially costly.

If you're interested in pursuing this I would definitely recommend contributing to the project and checking in on our friday [SIG Network] Gateway API Code Jam meetings, which are currently mostly focused on the topic of Blixt.

[danehans]

It's great to see that Blixt will help drive L4 adoption. I know Blixt is meant to be used for testing only, so it seems like a good fit for L4 conformance testing. However, I'm concerned that most implementations are L7 with a focus on supporting the Beta APIs. What existing L4 implementations is Blixt supporting? I can't tell from this list.

[shaneutt]

It's great to see that Blixt will help drive L4 adoption. I know Blixt is meant to be used for testing only, so it seems like a good fit for L4 conformance testing. However, I'm concerned that most implementations are L7 with a focus on supporting the Beta APIs. What existing L4 implementations is Blixt supporting? I can't tell from this list.

Not sure I understand the question perfectly, but my take of the question was that you're asking what other L4 implementations there are today. If that's the case, we don't know exactly and that's actually something we're trying to solve by driving forward L4 conformance tests (with Blixt as a catalyst) and also the conformance profiles work which will include reporting back to upstream.