REST code generation for 2024-07 prerelease (#361)

## Problem

We need to generate code off of the openapi specs stored in our apis
repo. We'll need to massage the output a bit in order to avoid having a
lot of confusing duplication of generated exception classes and other
utils.

## Solution

- I create a new `codegen` directory to act as the home for everything
codegen related. Inside this directory there are several things:
- Git submodule of our template files (slightly tweaked from the open
source generator, but in a separate repo so they can be shared with
plugins)
  - Git submodule of our apis repo
- A `build-oas.sh` script which does everything to generate the code off
the spec, adjust import paths, and place things into the `pinecone`
module package structure.

The script does the following:
- Pulls in the latest from the apis repo
- Builds the api repo to produce versioned spec files
- For each of data plane and control plane, generates a python package.
- Performs surgery to extract the duplicated files across these two
packages into a `shared` package.
- Use `sed` to adjust import paths to reflect the modified code
structure.

The surgery to extract shared code is more than just a cosmetic change,
as without it you can end up with a lot of confusion in the areas of
configuration and error handling due to there being multiple objects
that despite being identical in name and functionality remain distinct
from the perspective of language utils such as `isinstance`,
`.__class__` and `except`.

When the script is done running, the generated outputs have this
structure:

```txt
pinecone/core
└── openapi
    ├── control
    │   ├── api
    │   │   ├── inference_api.py
    │   │   └── manage_indexes_api.py
    │   └── model
    │       ├── __init__.py
    │       ├── create_index_request.py
    │       ├── index_model.py
    │       └── ...etc
    ├── data
    │   ├── api
    │   │   └── data_plane_api.py
    │   └── model
    │       ├── __init__.py
    │       ├── upsert_request.py
    │       ├── vector.py
    │       └── ...etc
    └── shared
        ├── api_client.py
        ├── configuration.py
        ├── exceptions.py
        ├── model_utils.py
        └── rest.py
```

## What actually changed in here?

It seems the main substantive change is in how the `spec` info is
expected to be passed when creating an index. So I had to write some
additional tests and make some modifications to the `create_index`
method.

Also, inference-related stuff is currently generated. But for now, all
the wrapper implementations for that functionality are only available by
installing a separate plugin,
[pinecone-plugin-inference](https://github.com/pinecone-io/python-plugin-inference).

## Type of Change

- [x] Breaking change (fix or feature that would cause existing
functionality to not work as expected)

## Test Plan

To run the script, run `make generate-oas`. Before pushing, I ran tests
locally with `make test-unit` and `PINECONE_API_KEY='key' make
test-integration`. This helped me catch a lot of small issues related to
the sed rewrite of import paths.

Want to see tests passing even though I changed quite a bit about how
the generated code is structured.

Author: jhamon

.gitmodules

@@ -0,0 +1,6 @@
+[submodule "codegen/apis"]
+	path = codegen/apis
+	url = [email protected]:pinecone-io/apis.git
+[submodule "codegen/python-oas-templates"]
+	path = codegen/python-oas-templates
+	url = [email protected]:pinecone-io/python-oas-templates.git

Makefile

@@ -13,13 +13,20 @@ test-unit:
 	@echo "Running tests..."
 	poetry run pytest --cov=pinecone --timeout=120 tests/unit
 
+test-integration:
+	@echo "Running integration tests..."
+	PINECONE_ENVIRONMENT="us-east4-gcp" SPEC='{"serverless": {"cloud": "aws", "region": "us-east-1" }}' DIMENSION=2 METRIC='cosine' GITHUB_BUILD_NUMBER='local' poetry run pytest tests/integration
+
 test-grpc-unit:
 	@echo "Running tests..."
 	poetry run pytest --cov=pinecone --timeout=120 tests/unit_grpc
 
 make type-check:
 	poetry run mypy pinecone --exclude pinecone/core
 
+make generate-oas:
+	./codegen/build-oas.sh "2024-07"
+
 version:
 	poetry version
 
@@ -28,21 +35,3 @@ package:
 
 upload:
 	poetry publish --verbose --username ${PYPI_USERNAME} --password ${PYPI_PASSWORD}
-
-upload-spruce:
-	# Configure Poetry for publishing to testpypi
-	poetry config repositories.test-pypi https://test.pypi.org/legacy/
-	poetry publish --verbose -r test-pypi --username ${PYPI_USERNAME} --password ${PYPI_PASSWORD}
-	
-license:
-	# Add license header using https://github.com/google/addlicense.
-	# If the license header already exists in a file, re-running this command has no effect.
-	pushd ${mkfile_path}/pinecone && \
-		docker run --rm -it -v ${mkfile_path}/pinecone:/src ghcr.io/google/addlicense:latest -f ./license_header.txt *.py */*.py */*/*.py */*/*/*.py */*/*/*/*.py */*/*/*/*/*.py */*/*/*/*/*/*.py; \
-		popd
-
-set-production:
-	echo "production" > pinecone/__environment__
-
-set-development:
-	echo "" > pinecone/__environment__

codegen/apis

@@ -0,0 +1,148 @@
+#!/bin/bash
+
+set -eux -o pipefail
+
+version=$1 # e.g. 2024-07
+modules=("control" "data")
+
+destination="pinecone/core/openapi"
+build_dir="build"
+
+update_apis_repo() {
+	echo "Updating apis repo"
+	pushd codegen/apis
+		git fetch
+		git pull
+		just build
+	popd
+}
+
+update_templates_repo() {
+	echo "Updating templates repo"
+	pushd codegen/python-oas-templates
+		git fetch
+		git pull
+	popd
+}
+
+verify_spec_version() {
+	local version=$1
+	echo "Verifying spec version $version exists in apis repo"
+	if [ -z "$version" ]; then
+		echo "Version is required"
+		exit 1
+	fi
+
+	verify_directory_exists "codegen/apis/_build/${version}"
+}
+
+verify_file_exists() {
+	local filename=$1
+	if [ ! -f "$filename" ]; then
+		echo "File does not exist at $filename"
+		exit 1
+	fi
+}
+
+verify_directory_exists() {
+	local directory=$1
+	if [ ! -d "$directory" ]; then
+		echo "Directory does not exist at $directory"
+		exit 1
+	fi
+}
+
+generate_client() {
+	local module_name=$1
+
+	oas_file="codegen/apis/_build/${version}/${module_name}_${version}.oas.yaml"
+	openapi_generator_config="codegen/openapi-config.${module_name}.json"
+	template_dir="codegen/python-oas-templates/templates5.2.0"
+	
+	verify_file_exists $oas_file
+	verify_file_exists $openapi_generator_config
+	verify_directory_exists $template_dir
+
+	# Cleanup previous build files
+	echo "Cleaning up previous build files"
+	rm -rf "${build_dir}"
+
+	# Generate client module
+	docker run --rm -v $(pwd):/workspace openapitools/openapi-generator-cli:v5.2.0 generate \
+		--input-spec "/workspace/$oas_file" \
+		--generator-name python \
+		--config "/workspace/$openapi_generator_config" \
+		--output "/workspace/${build_dir}" \
+		--template-dir "/workspace/$template_dir"
+
+	# Copy the generated module to the correct location
+	rm -rf "${destination}/${module_name}"
+	cp -r "build/pinecone/core/openapi/${module_name}" "${destination}/${module_name}"
+}
+
+extract_shared_classes() {
+	target_directory="${destination}/shared"
+	mkdir -p "$target_directory"
+
+	# Define the list of shared source files
+	sharedFiles=(
+		"api_client"
+		"configuration" 
+		"exceptions" 
+		"model_utils" 
+		"rest"
+	)
+
+	source_directory="${destination}/${modules[0]}"
+
+	# Loop through each file we want to share and copy it to the target directory
+	for file in "${sharedFiles[@]}"; do
+		cp "${source_directory}/${file}.py" "$target_directory"
+	done
+
+	# Cleanup shared files in each module
+	for module in "${modules[@]}"; do
+		source_directory="${destination}/${module}"
+		for file in "${sharedFiles[@]}"; do
+			rm "${source_directory}/${file}.py"
+		done
+	done
+
+	# Remove the docstring headers that aren't really correct in the 
+	# context of this new shared package structure
+	find "$target_directory" -name "*.py" -print0 | xargs -0 -I {} sh -c 'sed -i "" "/^\"\"\"/,/^\"\"\"/d" "{}"'
+
+	echo "All shared files have been copied to $target_directory."
+
+	# Adjust import paths in every file
+	find "${destination}" -name "*.py" | while IFS= read -r file; do
+		sed -i '' 's/from \.\.model_utils/from pinecone\.core\.openapi\.shared\.model_utils/g' "$file"
+
+		for module in "${modules[@]}"; do
+			sed -i '' "s/from pinecone\.core\.openapi\.$module import rest/from pinecone\.core\.openapi\.shared import rest/g" "$file"
+
+			for sharedFile in "${sharedFiles[@]}"; do
+				sed -i '' "s/from pinecone\.core\.openapi\.$module\.$sharedFile/from pinecone\.core\.openapi\.shared\.$sharedFile/g" "$file"
+			done
+		done
+	done
+}
+
+update_apis_repo
+update_templates_repo
+verify_spec_version $version
+
+rm -rf "${destination}"
+mkdir -p "${destination}"
+
+for module in "${modules[@]}"; do
+	generate_client $module
+done
+
+# Even though we want to generate multiple packages, we
+# don't want to duplicate every exception and utility class.
+# So we do a bit of surgery to combine the shared files.
+extract_shared_classes
+
+# Format generated files
+poetry run black "${destination}"

codegen/build-oas.sh

@@ -0,0 +1,4 @@
+{
+    "packageName": "pinecone.core.openapi.control",
+    "pythonAttrNoneIfUnset": true
+}

codegen/openapi-config.control.json

@@ -0,0 +1,4 @@
+{
+    "packageName": "pinecone.core.openapi.data",
+    "pythonAttrNoneIfUnset": true
+}

codegen/openapi-config.data.json

@@ -14,7 +14,7 @@
 from .data import *
 from .models import *
 
-from .core.client.models import (
+from .core.openapi.control.models import (
     IndexModel,
 )
 

codegen/python-oas-templates

@@ -1,9 +1,11 @@
 from typing import NamedTuple, Optional, Dict
 import os
 
-from pinecone.exceptions import PineconeConfigurationError
+from pinecone.exceptions.exceptions import PineconeConfigurationError
 from pinecone.config.openapi import OpenApiConfigFactory
-from pinecone.core.client.configuration import Configuration as OpenApiConfiguration
+from pinecone.core.openapi.shared.configuration import (
+    Configuration as OpenApiConfiguration,
+)
 from pinecone.utils import normalize_host
 from pinecone.utils.constants import SOURCE_TAG
 
@@ -57,15 +59,28 @@ def build(
         if not host:
             raise PineconeConfigurationError("You haven't specified a host.")
 
-        return Config(api_key, host, proxy_url, proxy_headers, ssl_ca_certs, ssl_verify, additional_headers, source_tag)
+        return Config(
+            api_key,
+            host,
+            proxy_url,
+            proxy_headers,
+            ssl_ca_certs,
+            ssl_verify,
+            additional_headers,
+            source_tag,
+        )
 
     @staticmethod
     def build_openapi_config(
-        config: Config, openapi_config: Optional[OpenApiConfiguration] = None, **kwargs
+        config: Config,
+        openapi_config: Optional[OpenApiConfiguration] = None,
+        **kwargs,
     ) -> OpenApiConfiguration:
         if openapi_config:
             openapi_config = OpenApiConfigFactory.copy(
-                openapi_config=openapi_config, api_key=config.api_key, host=config.host
+                openapi_config=openapi_config,
+                api_key=config.api_key,
+                host=config.host,
             )
         elif openapi_config is None:
             openapi_config = OpenApiConfigFactory.build(api_key=config.api_key, host=config.host)

pinecone/__init__.py

@@ -7,7 +7,9 @@
 
 from urllib3.connection import HTTPConnection
 
-from pinecone.core.client.configuration import Configuration as OpenApiConfiguration
+from pinecone.core.openapi.shared.configuration import (
+    Configuration as OpenApiConfiguration,
+)
 
 TCP_KEEPINTVL = 60  # Sec
 TCP_KEEPIDLE = 300  # Sec
@@ -84,7 +86,13 @@ def _get_socket_options(
             and hasattr(socket, "TCP_KEEPCNT")
         ):
             socket_params += [(socket.IPPROTO_TCP, socket.TCP_KEEPIDLE, keep_alive_idle_sec)]
-            socket_params += [(socket.IPPROTO_TCP, socket.TCP_KEEPINTVL, keep_alive_interval_sec)]
+            socket_params += [
+                (
+                    socket.IPPROTO_TCP,
+                    socket.TCP_KEEPINTVL,
+                    keep_alive_interval_sec,
+                )
+            ]
             socket_params += [(socket.IPPROTO_TCP, socket.TCP_KEEPCNT, keep_alive_tries)]
 
         # TCP Keep Alive Probes for Windows OS