Skip to content

Experimental conformance suite: Extended skipped tests not being reported #2206

Open
@sunjayBhatia

Description

@sunjayBhatia

With a conformance suite config similar to the below:

	cSuite, err := suite.NewExperimentalConformanceTestSuite(suite.ExperimentalConformanceOptions{
		Options: suite.Options{
                        ...
                        // Note this vs. using the EnableAllSupportedFeatures and ExemptFeatures field
			SupportedFeatures: suite.AllFeatures.Delete(suite.MeshCoreFeatures.UnsortedList()...),
			SkipTests: []string{
				tests.HTTPRouteRedirectPortAndScheme.ShortName,
			},
		},
		Implementation: conformance_v1alpha1.Implementation{
                         ...
		},
		ConformanceProfiles: sets.New(
			suite.HTTPConformanceProfileName,
			suite.TLSConformanceProfileName,
		),
	})

The resulting conformance report does not show any tests are skipped:

        apiVersion: gateway.networking.k8s.io/v1alpha1
        date: "2023-07-19T20:34:17Z"
        gatewayAPIVersion: TODO
        implementation:
          contact:
          - '@projectcontour/maintainers'
          organization: projectcontour
          project: contour
          url: https://github.com/projectcontour/contour
          version: ""
        kind: ConformanceReport
        profiles:
        - core:
            result: success
            statistics:
              Failed: 0
              Passed: 10
              Skipped: 0
            summary: ""
          name: TLS
        - core:
            result: success
            statistics:
              Failed: 0
              Passed: 28
              Skipped: 0
            summary: ""
          extended:
            result: success
            statistics:
              Failed: 0
              Passed: 9
              Skipped: 0
            summary: ""
            supportedFeatures:
            - HTTPRouteQueryParamMatching
            - HTTPRouteSchemeRedirect
            - HTTPRouteHostRewrite
            - HTTPRoutePathRewrite
            - HTTPRouteRequestMirror
            - HTTPResponseHeaderModification
            - HTTPRoutePortRedirect
            - HTTPRoutePathRedirect
            - HTTPRouteMethodMatching
          name: HTTP

Metadata

Metadata

Assignees

Labels

area/conformance-machineryIssues or PRs related to the machinery and the suite used to run conformance tests.lifecycle/staleDenotes an issue or PR has remained open with no activity and has become stale.needs-triageIndicates an issue or PR lacks a `triage/foo` label and requires one.priority/important-longtermImportant over the long term, but may not be staffed and/or may need multiple releases to complete.

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions