Add Microsoft.Databricks support

[Copilot]

What this PR does

Adds support for Microsoft.Databricks resources using the stable 2024-05-01 API version.

Closes #4345

Resources Added

• AccessConnector - Secure data access for Databricks workspaces
• Workspace - Main Databricks workspace resource
• WorkspacesPrivateEndpointConnection - Private endpoint management
• WorkspacesVirtualNetworkPeering - VNet peering configuration

Implementation

Added databricks service configuration to azure-arm.yaml:

databricks:
  2024-05-01:
    AccessConnector:
      $exportAs: AccessConnector
      $supportedFrom: v2.17.0
    Workspace:
      $exportAs: Workspace
      $supportedFrom: v2.17.0
    Workspaces_VirtualNetworkPeering:
      $exportAs: WorkspacesVirtualNetworkPeering
      $supportedFrom: v2.17.0
    VirtualNetworkPeeringPropertiesFormat_RemoteVirtualNetwork:
      Id:
        $referenceType: arm

Configured ARM reference types for VirtualNetworkPeering properties to enable cross-resource references. Removed readonly properties (SystemData from AccessConnector_Spec and CreatedDateTime from WorkspaceProperties) that should only appear in status. Code generator produced 53 files including CRDs, Go types, ARM conversions, storage types, webhooks, and tests.

Usage

apiVersion: databricks.azure.com/v1api20240501
kind: Workspace
metadata:
  name: my-databricks-workspace
spec:
  location: westus2
  owner:
    name: my-resource-group
  managedResourceGroupId: /subscriptions/.../resourceGroups/managed-rg
  sku:
    name: premium

How does this PR make you feel?

Checklist

• this PR contains documentation
• this PR contains tests
• this PR contains YAML Samples

Original prompt

---

This section details on the original issue you should resolve

<issue_title>Feature: Add Microsoft.Databricks capabilities</issue_title>
<issue_description>It would be great to be able to manage our Azure Databricks instances via ASO. Here is a link to the ARM template for databricks workspaces.

</issue_description>

Comments on the Issue (you are @copilot in this section)

@theunrepentantgeek New [resource types](https://learn.microsoft.com/en-us/azure/templates/microsoft.databricks/workspaces?pivots=deployment-language-arm-template):

• Microsoft.Databricks/accessConnectors
• Microsoft.Databricks/workspaces
• Microsoft.Databricks/workspaces/privateEndpointConnections
• Microsoft.Databricks/workspaces/virtualNetworkPeerings</comment_new>
  <comment_new>@theunrepentantgeek
  We're still interested in adding these resources.</comment_new>
  <comment_new>@matthchr
  We're still interested in adding these.</comment_new>

• Fixes Feature: Add Microsoft.Databricks capabilities #3299

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

[matthchr]

This field should be readonly. @copilot please prune this SystemData field to the v2/azure-arm.yaml file (see other examples in the file for reference).

[Copilot]

Done. Removed SystemData from AccessConnector_Spec in 1889f01.

[matthchr]

This should probably be readonly as well. @copilot please prune this

[Copilot]

Done. Removed CreatedDateTime from WorkspaceProperties in 1889f01.

[matthchr]

We need to do something here but I am not sure what.

[matthchr]

I think ideally we would be able to swap each of these objects (or at least the ones with ARM ID refs) to refer to a special variant of WorkspaceCustomStringParameter (which is the parameter they use for both IDs and names here).

We want (for the ID parameters) to use an ASOWorkspaceCustomStringParameter which has the arm-id flag on the value, but leave the things like subnet names as just strings