You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
## Problem
We need better docs on how to work in this repository.
## Solution
Add information about testing and debugging.
## Type of Change
- [x] Non-code change (docs, etc)
[Poetry](https://python-poetry.org/) is a tool that combines [virtualenv](https://virtualenv.pypa.io/en/latest/) usage with dependency management, to provide a consistent experience for project maintainers and contributors who need to develop the pinecone-python-client
23
-
as a library.
24
-
25
-
A common need when making changes to the Pinecone client is to test your changes against existing Python code or Jupyter Notebooks that `pip install` the Pinecone Python client as a library.
26
-
27
-
Developers want to be able to see their changes to the library immediately reflected in their main application code, as well as to track all changes they make in git, so that they can be contributed back in the form of a pull request.
28
-
29
-
The Pinecone Python client therefore supports Poetry as its primary means of enabling a consistent local development experience. This guide will walk you through the setup process so that you can:
30
-
1. Make local changes to the Pinecone Python client that are separated from your system's Python installation
31
-
2. Make local changes to the Pinecone Python client that are immediately reflected in other local code that imports the pinecone client
32
-
3. Track all your local changes to the Pinecone Python client so that you can contribute your fixes and feature additions back via GitHub pull requests
19
+
[Poetry](https://python-poetry.org/) is a tool that combines [virtualenv](https://virtualenv.pypa.io/en/latest/) usage with dependency management, to provide a consistent experience for project maintainers and contributors who need to develop the pinecone-python-client as a library.
33
20
34
21
### Step 1. Fork the Pinecone python client repository
35
22
@@ -41,149 +28,52 @@ It will take a few seconds for your fork to be ready. When it's ready, **clone y
41
28
42
29
Change directory into the repository, as we'll be setting up a virtualenv from within the root of the repository.
43
30
44
-
### Step 1. Install Poetry
31
+
### Step 2. Install Poetry
45
32
46
33
Visit [the Poetry site](https://python-poetry.org/) for installation instructions.
47
-
To use the [Poetry `shell` command](https://python-poetry.org/docs/cli#shell), install the [`shell` plugin](https://github.com/python-poetry/poetry-plugin-shell).
48
-
49
-
### Step 2. Install dependencies
50
-
51
-
Run `poetry install` from the root of the project.
52
-
53
-
### Step 3. Activate the Poetry virtual environment and verify success
54
34
55
-
Run `poetry shell` from the root of the project. At this point, you now have a virtualenv set up in this directory, which you can verify by running:
35
+
### Step 3. Install dependencies
56
36
57
-
`poetry env info`
58
-
59
-
You should see something similar to the following output:
If you want to extract only the path to your new virtualenv, you can run `poetry env info --path`
37
+
Run `poetry install -E grpc -E asyncio` from the root of the project.
76
38
77
39
### Step 4. Enable pre-commit hooks.
78
40
79
41
Run `poetry run pre-commit install` to enable checks to run when you commit so you don't have to find out during your CI run that minor lint issues need to be addressed.
80
42
81
43
## Common tasks
82
44
83
-
### Running tests
84
-
85
-
- Unit tests: `make test-unit`
86
-
- Integration tests: `PINECONE_API_KEY="YOUR API KEY" make test-integration`
87
-
- Run the tests in a single file: `poetry run pytest tests/unit/data/test_bulk_import.py -s -vv`
88
-
89
-
### Running the ruff linter / formatter
90
-
91
-
These should automatically trigger if you have enabled pre-commit hooks with `poetry run pre-commit install`. But in case you want to trigger these yourself, you can run them like this:
92
-
93
-
```
94
-
poetry run ruff check --fix # lint rules
95
-
poetry run ruff format # formatting
96
-
```
45
+
### Debugging
97
46
98
-
If you want to adjust the behavior of ruff, configurations are in `pyproject.toml`.
47
+
See the [debugging guide](./docs/maintainers/debugging.md). If you find an issue and would like to report it as a github issue, make sure you do not leak your API key that may be included in debug outputs.
99
48
49
+
### Running tests
100
50
101
-
### Consuming API version upgrades
51
+
- Unit tests: `make test-unit`
52
+
- Run the tests in a single file: `poetry run pytest tests/unit/data/test_bulk_import.py`
102
53
103
-
These instructions can only be followed by Pinecone employees with access to our private APIs repository.
54
+
For more information on testing, see the [Testing guide](./docs/maintainers/testing-guide.md). External contributors should not worry about running integration tests as they make live calls to Pinecone and will incur significant costs.
104
55
105
-
Prerequisites:
106
-
- You must be an employee with access to private Pinecone repositories
107
-
- You must have [Docker Desktop](https://www.docker.com/products/docker-desktop/) installed and running. Our code generation script uses a dockerized version of the OpenAPI CLI.
108
-
- You must have initialized the git submodules under codegen
56
+
### Running the type checker
109
57
110
-
```sh
111
-
git submodule
112
-
```
58
+
If you are adding new code, you should make an effort to annotate it with [type hints](https://mypy.readthedocs.io/en/stable/cheat_sheet_py3.html).
113
59
114
-
To regenerate the generated portions of the client with the latest version of the API specifications, you need to have Docker Desktop running on your local machine.
60
+
You can run the type-checker to check for issues with:
115
61
116
62
```sh
117
-
./codegen/
118
-
```
119
-
120
-
121
-
## Loading your virtualenv in another shell
122
-
123
-
It's a common need when developing against this client to load it as part of some other application or Jupyter Notebook code, modify
124
-
it directly, see your changes reflected immediately and also have your changes tracked in git so you can contribute them back.
125
-
126
-
It's important to understand that, by default, if you open a new shell or terminal window, or, for example, a new pane in a tmux session,
127
-
your new shell will not yet reference the new virtualenv you created in the previous step.
128
-
129
-
### Step 1. Get the path to your virtualenv
130
-
131
-
We're going to first get the path to the virtualenv we just created, by running:
132
-
133
-
```bash
134
-
poetry env info --path
135
-
```
136
-
137
-
You'll get a path similar to this one: `/home/youruser/.cache/pypoetry/virtualenvs/pinecone-fWu70vbC-py3.9/`
138
-
139
-
### Step 2. Load your existing virtualenv in your new shell
140
-
141
-
Within this path is a shell script that lives at `<your-virtualenv-path>/bin/activate`. Importantly, you cannot simply run this script, but you
In the above example, ensure you're using your own virtualenv path as returned by `poetry env info --path`.
148
-
149
-
### Step 3. Test out your virtualenv
150
-
151
-
Now, we can test that our virtualenv is working properly by adding a new test module and function to the `pinecone` client within our virtualenv
152
-
and running it from the second shell.
153
-
154
-
#### Create a new test file in pinecone-python-client
155
-
In the root of your working directory of the `pinecone-python-client` where you first ran `poetry shell`, add a new file named `hello_virtualenv.py` under the `pinecone` folder.
156
-
157
-
In that file write the following:
158
-
159
-
```python
160
-
defhello():
161
-
print("Hello, from your virtualenv!")
63
+
poetry run mypy pinecone
162
64
```
163
-
Save the file.
164
65
165
-
#### Create a new test file in your second shell
166
-
This step demonstrates how you can immediately test your latest Pinecone client code from any local Python application or Jupyter Notebook:
167
-
168
-
In your second shell, where you ran `source` to load your virtualenv, create a python file named `test.py` and write the following:
66
+
### Running the ruff linter / formatter
169
67
170
-
```python
171
-
from pinecone import hello_virtualenv
68
+
These should automatically trigger if you have enabled pre-commit hooks with `poetry run pre-commit install`. But in case you want to trigger these yourself, you can run them like this:
172
69
173
-
hello_virtualenv.hello()
174
70
```
175
-
176
-
Save the file. Run it with your Python binary. Depending on your system, this may either be `python` or `python3`:
177
-
178
-
```bash
179
-
python3 test.py
71
+
poetry run ruff check --fix # lint rules
72
+
poetry run ruff format # formatting
180
73
```
181
74
182
-
You should see the following output:
75
+
If you experience any issues please [file a new issue](https://github.com/pinecone-io/pinecone-python-client/issues/new).
183
76
184
-
```bash
185
-
❯ python3 test.py
186
-
Hello, from your virtualenv!
187
-
```
77
+
### Submitting a Pull Request
188
78
189
-
If you experience any issues please [file a new issue](https://github.com/pinecone-io/pinecone-python-client/issues/new).
79
+
Once you have a change in your fork you feel good about, confirm you are able to run unit tests, pass the ruff and mypy type-checking steps, please submit a [Pull Request](https://github.com/pinecone-io/pinecone-python-client/compare). All code contributed to the pinecone-python-client repository is licensed under the [Apache 2.0 license](./LICENSE.txt).
Visit [the Poetry site](https://python-poetry.org/docs/#installation) for installation instructions.
16
+
17
+
### 3. Install dependencies
18
+
19
+
Run this from the root of the project.
20
+
21
+
```sh
22
+
poetry install -E grpc -E asyncio
23
+
```
24
+
25
+
These extra groups for `grpc` and `asyncio` are optional but required to do development on those optional parts of the SDK.
26
+
27
+
### 4. Enable pre-commit hooks
28
+
29
+
Run `poetry run pre-commit install` to enable checks to run when you commit so you don't have to find out during your CI run that minor lint issues need to be addressed.
30
+
31
+
### 5. Setup environment variables
32
+
33
+
Some tests require environment variables to be set in order to run.
34
+
35
+
```sh
36
+
cp .env.example .env
37
+
```
38
+
39
+
After copying the template, you will need to fill in your secrets. `.env` is in `.gitignore`, so there's no concern about accidentally committing your secrets.
40
+
41
+
### Testing
42
+
43
+
There is a lot to say about testing the Python SDK. See the [testing guide](./docs/maintainers/testing-guide.md).
44
+
45
+
### Debugging
46
+
47
+
See the [debugging guide](./docs/maintainers/debugging.md)
48
+
49
+
50
+
### Preparing a new release candidate (for major release / API version bump)
51
+
52
+
These instructions can only be followed by Pinecone employees with access to our private APIs repository.
53
+
54
+
Prerequisites:
55
+
- You must be an employee with access to private Pinecone repositories
56
+
- You must have [Docker Desktop](https://www.docker.com/products/docker-desktop/) installed and running. Our code generation script uses a dockerized version of the OpenAPI CLI.
57
+
- You must have initialized the git submodules under `codegen/`
58
+
59
+
First create a prerelease branch where all the work for the upcoming release will be integrated. For example, for 2025-04 API release I worked off of this branch:
60
+
61
+
```
62
+
git checkout main
63
+
git pull
64
+
git checkout release-candidate/2025-04
65
+
git push origin release-candidate/2025-04
66
+
```
67
+
68
+
The release-candidate branch is where we will integrate all changes for an upcoming release which may include work from many different PRs and commits. So we can push it directly with no changes.
69
+
70
+
Next, to regenerate the generated parts of the SDK using a new API spec, I make a second branch to hold my changes
71
+
72
+
```sh
73
+
git checkout jhamon/regen-2025-04
74
+
```
75
+
76
+
Then you run the build script by passing the API version as an argument, like this:
77
+
78
+
```sh
79
+
./codegen/build-oas.sh 2025-07
80
+
```
81
+
82
+
For grpc updates, it's a similar story:
83
+
84
+
```sh
85
+
./codegen/build-grpc.sh 2025-07
86
+
```
87
+
88
+
Commit the generated files which should be mainly placed under `pinecone/core`. Commit the sha changes in the git submodule at `codegen/apis`.
89
+
90
+
Run the type check with `poetry run mypy pinecone`. This will usually surface breaking changes as a result of things being renamed or modified.
91
+
92
+
Push your branch (`git push origin jhamon/regen-2025-04` in this example) and open a PR **against the RC branch** (in this example `release-candidate/2025-04`). This will allow the full PR test suite to kick off and help you discover what other changes you need to make.
93
+
94
+
### Releasing
95
+
96
+
See the [release instructions](./docs/maintainers/releasing.md)
0 commit comments