Implement docs with sphinx (#508)

## Problem

Recent changes to implement lazy loading in the SDK have broken
compatibility with our previous docs tool, pdoc. After some
investigation, it seemed easier to adopt a new docs solution than to try
to manipulate pdoc internals to work for our usecase.

## Solution

- Implement configurations for docs in a new `docs` folder. Main points
of interest: `docs/conf.py` and `docs/index.rst`
- Adjust docstrings to use rst format throughout
- Add a few custom styles in `docs/_static/custom.css` to tweak the look
of the default sphinx theme which is using Garamond font and doesn't
really match our branding at all. Changing up to use sans-serif fonts
and tweaking a few other things gets us a long way there, although I'm
sure there are still things to refine on that over time.
- Build docs with `poetry run sphinx-build -b html docs docsbuild`
- Implement some minor shenanigans to work around poetry and sphinx's
rigidness about python version support. To do this in a nicer way I
would either have to drop python 3.9 and 3.10 from our SDK's supported
python versions, or setup a separate package outside this repo that
depended on `sphinx` and the published `pinecone`. Neither one of those
sound like good options to me.

## Type of Change

- [x] New feature (non-breaking change which adds functionality)

## Testing

New docs visible at https://sdk.pinecone.io/python

I published these by manually triggering the build-and-publish-docs job
from this branch

Author: jhamon

.github/actions/build-docs/action.yml

@@ -1,5 +1,5 @@
 name: 'Build client documentation'
-description: 'Generates client documentation using pdoc'
+description: 'Generates client documentation using sphinx'
 inputs:
   python-version:
     description: 'Python version to use'
@@ -13,15 +13,31 @@ runs:
       with:
         include_grpc: 'true'
         include_dev: 'true'
+        include_asyncio: 'true'
         python_version: ${{ inputs.python-version }}
+
+    - name: Pretend this project requires Python 3.11
+      shell: bash
+      run: |
+        # Poetry won't let me install sphinx as a dev dependency in this project
+        # because of the wide range of versions our library supports. So during this
+        # action, we'll pretend this project requires Python 3.11 or greater.
+        sed -i 's/python = "^3.9"/python = "^3.11"/' pyproject.toml
+        poetry lock
+        poetry install -E grpc -E asyncio
+
+    - name: Install sphinx
+      shell: bash
+      run: |
+        poetry add sphinx myst-parser --group dev
+
     - name: Build html documentation
       shell: bash
       run: |
-        poetry run pdoc pinecone '!pinecone.core' '!pinecone.utils' --favicon ./favicon-32x32.png --docformat google -o ./pdoc
+        poetry run sphinx-build -b html docs docsbuild
 
-    - name: Fix relative links
+    - name: Discard changes to pyproject.toml and poetry.lock
       shell: bash
       run: |
-        poetry run python3 ./.github/actions/build-docs/fix-relative-links.py ./pdoc ./pdoc
-      env:
-        BASE_URL: "https://github.com/pinecone-io/pinecone-python-client/blob/main/"
+        git checkout pyproject.toml
+        git checkout poetry.lock

.github/actions/build-docs/fix-relative-links.py

@@ -14,7 +14,7 @@ jobs:
       - name: Checkout
         uses: actions/checkout@v4
 
-      - name: Generate pdoc documentation
+      - name: Generate sphinx documentation
         uses: ./.github/actions/build-docs
         with:
           python-version: 3.11
@@ -24,7 +24,7 @@ jobs:
         env:
           SSH_DEPLOY_KEY: ${{ secrets.SSH_DEPLOY_KEY }}
         with:
-          source-directory: pdoc
+          source-directory: docsbuild
           destination-github-username: pinecone-io
           destination-repository-name: sdk-docs
           user-email: [email protected]

.github/workflows/build-and-publish-docs.yaml

@@ -30,6 +30,7 @@ jobs:
     if: ${{ always() }}
     needs:
       - dependency-tests
+      - create-project
     uses: './.github/workflows/project-cleanup.yaml'
     secrets: inherit
     with:

.github/workflows/on-pr-dep-change.yaml

@@ -144,6 +144,7 @@ venv.bak/
 pdoc/*
 !pdoc/pinecone-python-client-fork.png
 !pdoc/favicon-32x32.png
+docsbuild
 
 # mypy
 .mypy_cache/

.gitignore

@@ -0,0 +1,33 @@
+body,
+div.body h1,
+div.body h2,
+div.body h3,
+div.body h4,
+div.body h5,
+div.body h6,
+div.admonition p.admonition-title {
+    font-family: "Inter", "Helvetica Neue", "Helvetica", "Arial", sans-serif;
+}
+
+.blurb {
+    font-size: 16px;
+}
+
+p.admonition-title:after {
+    content: "";
+}
+
+div.code-block-caption {
+    background-color: #EEE;
+    border-bottom: 1px solid #CCC;
+    font-size: 17px;
+    padding: 10px;
+}
+
+dt:target {
+    background-color: #E8E8E8;
+}
+
+.highlight {
+    background-color: #F8F8F8;
+}

docs/_static/custom.css

@@ -0,0 +1,107 @@
+===============
+PineconeAsyncio
+===============
+
+.. autoclass:: pinecone::PineconeAsyncio
+
+.. automethod:: pinecone::PineconeAsyncio.__init__
+
+DB Control Plane
+================
+
+Indexes
+-------
+
+.. automethod:: pinecone::PineconeAsyncio.create_index
+
+.. automethod:: pinecone::PineconeAsyncio.create_index_for_model
+
+.. automethod:: pinecone::PineconeAsyncio.create_index_from_backup
+
+.. automethod:: pinecone::PineconeAsyncio.list_indexes
+
+.. automethod:: pinecone::PineconeAsyncio.describe_index
+
+.. automethod:: pinecone::PineconeAsyncio.configure_index
+
+.. automethod:: pinecone::PineconeAsyncio.delete_index
+
+.. automethod:: pinecone::PineconeAsyncio.has_index
+
+Backups
+-------
+
+.. automethod:: pinecone::PineconeAsyncio.create_backup
+
+.. automethod:: pinecone::PineconeAsyncio.list_backups
+
+.. automethod:: pinecone::PineconeAsyncio.describe_backup
+
+.. automethod:: pinecone::PineconeAsyncio.delete_backup
+
+Collections
+-----------
+
+.. automethod:: pinecone::PineconeAsyncio.create_collection
+
+.. automethod:: pinecone::PineconeAsyncio.list_collections
+
+.. automethod:: pinecone::PineconeAsyncio.describe_collection
+
+.. automethod:: pinecone::PineconeAsyncio.delete_collection
+
+Restore Jobs
+------------
+
+.. automethod:: pinecone::PineconeAsyncio.list_restore_jobs
+
+.. automethod:: pinecone::PineconeAsyncio.describe_restore_job
+
+DB Data Plane
+=============
+
+.. autoclass:: pinecone.db_data::IndexAsyncio
+
+.. automethod:: pinecone.db_data::IndexAsyncio.__init__
+
+.. automethod:: pinecone.db_data::IndexAsyncio.describe_index_stats
+
+Vectors
+-------
+
+.. automethod:: pinecone.db_data::IndexAsyncio.upsert
+
+.. automethod:: pinecone.db_data::IndexAsyncio.query
+
+.. automethod:: pinecone.db_data::IndexAsyncio.query_namespaces
+
+.. automethod:: pinecone.db_data::IndexAsyncio.delete
+
+.. automethod:: pinecone.db_data::IndexAsyncio.fetch
+
+.. automethod:: pinecone.db_data::IndexAsyncio.list
+
+.. automethod:: pinecone.db_data::IndexAsyncio.list_paginated
+
+Records
+-------
+
+If you have created an index using integrated inference, you can use the following methods to
+search and retrieve records.
+
+.. automethod:: pinecone.db_data::IndexAsyncio.search
+
+.. automethod:: pinecone.db_data::IndexAsyncio.search_records
+
+
+
+Inference
+=========
+
+.. automethod:: pinecone.inference::Inference.embed
+
+.. automethod:: pinecone.inference::Inference.rerank
+
+.. automethod:: pinecone.inference::Inference.list_models
+
+.. automethod:: pinecone.inference::Inference.get_model

docs/_static/pinecone-logo.svg

@@ -0,0 +1,42 @@
+import pinecone
+
+project = "Pinecone Python SDK"
+author = "Pinecone Systems, Inc."
+version = pinecone.__version__
+copyright = "%Y, Pinecone Systems, Inc."
+
+html_baseurl = "https://sdk.pinecone.io/python"
+html_static_path = ["_static"]
+html_favicon = "favicon-32x32.png"
+
+extensions = [
+    "sphinx.ext.autodoc",
+    "sphinx.ext.viewcode",
+    "sphinx.ext.todo",
+    "sphinx.ext.napoleon",
+    "sphinx.ext.coverage",
+    "sphinx.ext.autodoc.typehints",
+    "myst_parser",
+]
+
+# -- HTML Configuration -------------------------------------------------
+
+html_theme = "alabaster"
+html_theme_options = {
+    "logo": "pinecone-logo.svg",
+    "description": "Pinecone Python SDK",
+    "github_user": "pinecone-io",
+    "github_repo": "pinecone-python-client",
+    "github_button": True,
+    "fixed_sidebar": True,
+    "page_width": "1140px",
+    "sidebar_width": "300px",
+    "show_related": False,
+    "show_powered_by": False,
+    "extra_nav_links": {
+        "Github Source": "https://github.com/pinecone-io/pinecone-python-client",
+        "Pinecone Home": "https://pinecone.io",
+        "Pinecone Docs": "https://docs.pinecone.io",
+        "Pinecone Console": "https://app.pinecone.io",
+    },
+}

docs/asyncio.rst

@@ -151,6 +151,6 @@ pc.configure_index(
 )
 ```
 
-# Configuring, listing, describing, and deleting
+## Configuring, listing, describing, and deleting
 
 See [shared index actions](shared-index-actions.md) to learn about how to manage the lifecycle of your index after it is created.

docs/conf.py

@@ -126,6 +126,6 @@ pc.create_index(
 )
 ```
 
-# Configuring, listing, describing, and deleting
+## Configuring, listing, describing, and deleting
 
 See [shared index actions](shared-index-actions.md) to learn about how to manage the lifecycle of your index after it is created.

docs/db_control/pod-indexes.md

@@ -1,4 +1,4 @@
-# Optional Index configurations
+# Index configuration
 
 This page covers some optional configurations that can be used with all index types.
 

docs/db_control/serverless-indexes.md

@@ -1,4 +1,4 @@
-# Using your Pinecone index
+# Vectors
 
 ## Describe index statistics
 

docs/db_control/shared-index-configs.md

@@ -1,3 +1,5 @@
+# FAQ
+
 ## How does connection pooling work in the Pinecone SDK?
 
 Before any data can be sent or received from Pinecone, your application must first establish a TCP connection with our API gateway. Establishing a TCP connection is a costly operation from a performance standpoint and so we use connection pooling to cache and reuse these connections across many different requests.