Skip to content

Commit 96d6045

Browse files
authored
Update docs for maintainers (#499)
## Problem We need better docs on how to work in this repository. ## Solution Add information about testing and debugging. ## Type of Change - [x] Non-code change (docs, etc)
1 parent 7860ca1 commit 96d6045

File tree

9 files changed

+586
-137
lines changed

9 files changed

+586
-137
lines changed

.env.example

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
## Fill in real values and rename this file to .env before
2+
## running integration tests on your machine.
3+
4+
## This should be your personal API key. These will get picked up
5+
## and used any time you run integration tests under
6+
## "poetry run pytest tests/integration"
7+
##
8+
## This key is also read and used to setup the pc client instance
9+
## when running "poetry run repl". This makes it easy to do
10+
## one-off manual testing.
11+
PINECONE_API_KEY=''
12+
13+
## If you set this variable, you can also use the pcci client instance
14+
## when running "poetry run repl" in order to do cleanup/management
15+
## on the project used from CI.
16+
PINECONE_API_KEY_CI_TESTING=''
17+
18+
## These headers get picked up and attached to every request by the code in
19+
## pinecone/config/pinecone_config.py
20+
##
21+
## The x-environment header is used to route requests to preprod. The value needs to be
22+
## a JSON string so it can be properly stored and read from an env var.
23+
PINECONE_ADDITIONAL_HEADERS='{"sdk-test-suite": "pinecone-python-client", "x-environment": "preprod-aws-0"}'
24+
25+
## There's a bunch of tests in tests/integration/data/test_weird_ids.py
26+
## that we don't need to run most of the time. Only when refactoring the rat's nest
27+
## of generated code to ensure we haven't broken something subtle with string handling.
28+
SKIP_WEIRD=true
29+
30+
## Some tests can run with either the Pinecone or PineconeGrpc client depending on
31+
## whether this value is set.
32+
USE_GRPC=false
33+
34+
## When debugging, you may want to enable PINECONE_DEBUG_CURL this to see some requests translated into
35+
## curl syntax. These are useful when reporting API issues to the backend team so they
36+
## can be reproduced without having to setup a python repro. WARNING: This output will
37+
## include the Api-Key header.
38+
# PINECONE_DEBUG_CURL='true'

.github/workflows/pr.yaml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,7 @@ on:
1515
- '*.jpeg'
1616
- '*.gif'
1717
- '*.svg'
18+
- '*.example'
1819
push:
1920
branches:
2021
- main
@@ -31,6 +32,7 @@ on:
3132
- '*.jpeg'
3233
- '*.gif'
3334
- '*.svg'
35+
- '*.example'
3436
workflow_dispatch: {}
3537

3638
concurrency:

CONTRIBUTING.md

Lines changed: 22 additions & 132 deletions
Original file line numberDiff line numberDiff line change
@@ -2,9 +2,7 @@
22

33
## Installing development versions
44

5-
If you want to explore a potential code change, investigate
6-
a bug, or just want to try unreleased features, you can also install
7-
specific git shas.
5+
If you want to explore a potential code change, investigate a bug, or just want to try unreleased features, you can also install specific git shas.
86

97
Some example commands:
108

@@ -16,20 +14,9 @@ pip3 install git+https://[email protected]/pinecone-io/pinecone-python-client.git@4
1614
poetry add git+https://github.com/pinecone-io/pinecone-python-client.git@44fc7ed
1715
```
1816

19-
2017
## Developing locally with Poetry
2118

22-
[Poetry](https://python-poetry.org/) is a tool that combines [virtualenv](https://virtualenv.pypa.io/en/latest/) usage with dependency management, to provide a consistent experience for project maintainers and contributors who need to develop the pinecone-python-client
23-
as a library.
24-
25-
A common need when making changes to the Pinecone client is to test your changes against existing Python code or Jupyter Notebooks that `pip install` the Pinecone Python client as a library.
26-
27-
Developers want to be able to see their changes to the library immediately reflected in their main application code, as well as to track all changes they make in git, so that they can be contributed back in the form of a pull request.
28-
29-
The Pinecone Python client therefore supports Poetry as its primary means of enabling a consistent local development experience. This guide will walk you through the setup process so that you can:
30-
1. Make local changes to the Pinecone Python client that are separated from your system's Python installation
31-
2. Make local changes to the Pinecone Python client that are immediately reflected in other local code that imports the pinecone client
32-
3. Track all your local changes to the Pinecone Python client so that you can contribute your fixes and feature additions back via GitHub pull requests
19+
[Poetry](https://python-poetry.org/) is a tool that combines [virtualenv](https://virtualenv.pypa.io/en/latest/) usage with dependency management, to provide a consistent experience for project maintainers and contributors who need to develop the pinecone-python-client as a library.
3320

3421
### Step 1. Fork the Pinecone python client repository
3522

@@ -41,149 +28,52 @@ It will take a few seconds for your fork to be ready. When it's ready, **clone y
4128

4229
Change directory into the repository, as we'll be setting up a virtualenv from within the root of the repository.
4330

44-
### Step 1. Install Poetry
31+
### Step 2. Install Poetry
4532

4633
Visit [the Poetry site](https://python-poetry.org/) for installation instructions.
47-
To use the [Poetry `shell` command](https://python-poetry.org/docs/cli#shell), install the [`shell` plugin](https://github.com/python-poetry/poetry-plugin-shell).
48-
49-
### Step 2. Install dependencies
50-
51-
Run `poetry install` from the root of the project.
52-
53-
### Step 3. Activate the Poetry virtual environment and verify success
5434

55-
Run `poetry shell` from the root of the project. At this point, you now have a virtualenv set up in this directory, which you can verify by running:
35+
### Step 3. Install dependencies
5636

57-
`poetry env info`
58-
59-
You should see something similar to the following output:
60-
61-
```bash
62-
Virtualenv
63-
Python: 3.9.16
64-
Implementation: CPython
65-
Path: /home/youruser/.cache/pypoetry/virtualenvs/pinecone-fWu70vbC-py3.9
66-
Executable: /home/youruser/.cache/pypoetry/virtualenvs/pinecone-fWu70vbC-py3.9/bin/python
67-
Valid: True
68-
69-
System
70-
Platform: linux
71-
OS: posix
72-
Python: 3.9.16
73-
Path: /home/linuxbrew/.linuxbrew/opt/[email protected]
74-
```
75-
If you want to extract only the path to your new virtualenv, you can run `poetry env info --path`
37+
Run `poetry install -E grpc -E asyncio` from the root of the project.
7638

7739
### Step 4. Enable pre-commit hooks.
7840

7941
Run `poetry run pre-commit install` to enable checks to run when you commit so you don't have to find out during your CI run that minor lint issues need to be addressed.
8042

8143
## Common tasks
8244

83-
### Running tests
84-
85-
- Unit tests: `make test-unit`
86-
- Integration tests: `PINECONE_API_KEY="YOUR API KEY" make test-integration`
87-
- Run the tests in a single file: `poetry run pytest tests/unit/data/test_bulk_import.py -s -vv`
88-
89-
### Running the ruff linter / formatter
90-
91-
These should automatically trigger if you have enabled pre-commit hooks with `poetry run pre-commit install`. But in case you want to trigger these yourself, you can run them like this:
92-
93-
```
94-
poetry run ruff check --fix # lint rules
95-
poetry run ruff format # formatting
96-
```
45+
### Debugging
9746

98-
If you want to adjust the behavior of ruff, configurations are in `pyproject.toml`.
47+
See the [debugging guide](./docs/maintainers/debugging.md). If you find an issue and would like to report it as a github issue, make sure you do not leak your API key that may be included in debug outputs.
9948

49+
### Running tests
10050

101-
### Consuming API version upgrades
51+
- Unit tests: `make test-unit`
52+
- Run the tests in a single file: `poetry run pytest tests/unit/data/test_bulk_import.py`
10253

103-
These instructions can only be followed by Pinecone employees with access to our private APIs repository.
54+
For more information on testing, see the [Testing guide](./docs/maintainers/testing-guide.md). External contributors should not worry about running integration tests as they make live calls to Pinecone and will incur significant costs.
10455

105-
Prerequisites:
106-
- You must be an employee with access to private Pinecone repositories
107-
- You must have [Docker Desktop](https://www.docker.com/products/docker-desktop/) installed and running. Our code generation script uses a dockerized version of the OpenAPI CLI.
108-
- You must have initialized the git submodules under codegen
56+
### Running the type checker
10957

110-
```sh
111-
git submodule
112-
```
58+
If you are adding new code, you should make an effort to annotate it with [type hints](https://mypy.readthedocs.io/en/stable/cheat_sheet_py3.html).
11359

114-
To regenerate the generated portions of the client with the latest version of the API specifications, you need to have Docker Desktop running on your local machine.
60+
You can run the type-checker to check for issues with:
11561

11662
```sh
117-
./codegen/
118-
```
119-
120-
121-
## Loading your virtualenv in another shell
122-
123-
It's a common need when developing against this client to load it as part of some other application or Jupyter Notebook code, modify
124-
it directly, see your changes reflected immediately and also have your changes tracked in git so you can contribute them back.
125-
126-
It's important to understand that, by default, if you open a new shell or terminal window, or, for example, a new pane in a tmux session,
127-
your new shell will not yet reference the new virtualenv you created in the previous step.
128-
129-
### Step 1. Get the path to your virtualenv
130-
131-
We're going to first get the path to the virtualenv we just created, by running:
132-
133-
```bash
134-
poetry env info --path
135-
```
136-
137-
You'll get a path similar to this one: `/home/youruser/.cache/pypoetry/virtualenvs/pinecone-fWu70vbC-py3.9/`
138-
139-
### Step 2. Load your existing virtualenv in your new shell
140-
141-
Within this path is a shell script that lives at `<your-virtualenv-path>/bin/activate`. Importantly, you cannot simply run this script, but you
142-
must instead source it like so:
143-
144-
```bash
145-
source /home/youruser/.cache/pypoetry/virtualenvs/pinecone-fWu70vbC-py3.9/bin/activate
146-
```
147-
In the above example, ensure you're using your own virtualenv path as returned by `poetry env info --path`.
148-
149-
### Step 3. Test out your virtualenv
150-
151-
Now, we can test that our virtualenv is working properly by adding a new test module and function to the `pinecone` client within our virtualenv
152-
and running it from the second shell.
153-
154-
#### Create a new test file in pinecone-python-client
155-
In the root of your working directory of the `pinecone-python-client` where you first ran `poetry shell`, add a new file named `hello_virtualenv.py` under the `pinecone` folder.
156-
157-
In that file write the following:
158-
159-
```python
160-
def hello():
161-
print("Hello, from your virtualenv!")
63+
poetry run mypy pinecone
16264
```
163-
Save the file.
16465

165-
#### Create a new test file in your second shell
166-
This step demonstrates how you can immediately test your latest Pinecone client code from any local Python application or Jupyter Notebook:
167-
168-
In your second shell, where you ran `source` to load your virtualenv, create a python file named `test.py` and write the following:
66+
### Running the ruff linter / formatter
16967

170-
```python
171-
from pinecone import hello_virtualenv
68+
These should automatically trigger if you have enabled pre-commit hooks with `poetry run pre-commit install`. But in case you want to trigger these yourself, you can run them like this:
17269

173-
hello_virtualenv.hello()
17470
```
175-
176-
Save the file. Run it with your Python binary. Depending on your system, this may either be `python` or `python3`:
177-
178-
```bash
179-
python3 test.py
71+
poetry run ruff check --fix # lint rules
72+
poetry run ruff format # formatting
18073
```
18174

182-
You should see the following output:
75+
If you experience any issues please [file a new issue](https://github.com/pinecone-io/pinecone-python-client/issues/new).
18376

184-
```bash
185-
❯ python3 test.py
186-
Hello, from your virtualenv!
187-
```
77+
### Submitting a Pull Request
18878

189-
If you experience any issues please [file a new issue](https://github.com/pinecone-io/pinecone-python-client/issues/new).
79+
Once you have a change in your fork you feel good about, confirm you are able to run unit tests, pass the ruff and mypy type-checking steps, please submit a [Pull Request](https://github.com/pinecone-io/pinecone-python-client/compare). All code contributed to the pinecone-python-client repository is licensed under the [Apache 2.0 license](./LICENSE.txt).

MAINTAINERS.md

Lines changed: 96 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,96 @@
1+
# Maintainers
2+
3+
This guide is aimed primarily at Pinecone employees working on maintaining and developing the python SDK.
4+
5+
## Setup
6+
7+
### 1. Clone the repo
8+
9+
```sh
10+
git clone [email protected]:pinecone-io/pinecone-python-client.git
11+
```
12+
13+
### 2. Install Poetry
14+
15+
Visit [the Poetry site](https://python-poetry.org/docs/#installation) for installation instructions.
16+
17+
### 3. Install dependencies
18+
19+
Run this from the root of the project.
20+
21+
```sh
22+
poetry install -E grpc -E asyncio
23+
```
24+
25+
These extra groups for `grpc` and `asyncio` are optional but required to do development on those optional parts of the SDK.
26+
27+
### 4. Enable pre-commit hooks
28+
29+
Run `poetry run pre-commit install` to enable checks to run when you commit so you don't have to find out during your CI run that minor lint issues need to be addressed.
30+
31+
### 5. Setup environment variables
32+
33+
Some tests require environment variables to be set in order to run.
34+
35+
```sh
36+
cp .env.example .env
37+
```
38+
39+
After copying the template, you will need to fill in your secrets. `.env` is in `.gitignore`, so there's no concern about accidentally committing your secrets.
40+
41+
### Testing
42+
43+
There is a lot to say about testing the Python SDK. See the [testing guide](./docs/maintainers/testing-guide.md).
44+
45+
### Debugging
46+
47+
See the [debugging guide](./docs/maintainers/debugging.md)
48+
49+
50+
### Preparing a new release candidate (for major release / API version bump)
51+
52+
These instructions can only be followed by Pinecone employees with access to our private APIs repository.
53+
54+
Prerequisites:
55+
- You must be an employee with access to private Pinecone repositories
56+
- You must have [Docker Desktop](https://www.docker.com/products/docker-desktop/) installed and running. Our code generation script uses a dockerized version of the OpenAPI CLI.
57+
- You must have initialized the git submodules under `codegen/`
58+
59+
First create a prerelease branch where all the work for the upcoming release will be integrated. For example, for 2025-04 API release I worked off of this branch:
60+
61+
```
62+
git checkout main
63+
git pull
64+
git checkout release-candidate/2025-04
65+
git push origin release-candidate/2025-04
66+
```
67+
68+
The release-candidate branch is where we will integrate all changes for an upcoming release which may include work from many different PRs and commits. So we can push it directly with no changes.
69+
70+
Next, to regenerate the generated parts of the SDK using a new API spec, I make a second branch to hold my changes
71+
72+
```sh
73+
git checkout jhamon/regen-2025-04
74+
```
75+
76+
Then you run the build script by passing the API version as an argument, like this:
77+
78+
```sh
79+
./codegen/build-oas.sh 2025-07
80+
```
81+
82+
For grpc updates, it's a similar story:
83+
84+
```sh
85+
./codegen/build-grpc.sh 2025-07
86+
```
87+
88+
Commit the generated files which should be mainly placed under `pinecone/core`. Commit the sha changes in the git submodule at `codegen/apis`.
89+
90+
Run the type check with `poetry run mypy pinecone`. This will usually surface breaking changes as a result of things being renamed or modified.
91+
92+
Push your branch (`git push origin jhamon/regen-2025-04` in this example) and open a PR **against the RC branch** (in this example `release-candidate/2025-04`). This will allow the full PR test suite to kick off and help you discover what other changes you need to make.
93+
94+
### Releasing
95+
96+
See the [release instructions](./docs/maintainers/releasing.md)

0 commit comments

Comments
 (0)