Support poetry virtual environments

Motivation

Support for poetry environments is currently our highest upvoted feature request on github. It attempts to solve problems with all predating tools to manage Python packages: virtualenv, pip, pipenv, etc by being an all-in-one tool, and hence is becoming increasingly popular.

Scope

• Discover environments created using Poetry
• Identify if a provided custom interpreter path belongs to a Poetry environment
• Install dev tools using Poetry when related poetry environment is selected

Things we decided we need not address,

• Activate environment using poetry CLI: poetry shell command is similar to pipenv shell, which has its issues #4404. Besides, we don't need to activate these environments via poetry, as it manages venv/virtualenv environments which can directly be activated using the activate scripts.
• Run scripts using poetry CLI: poetry run command can be useful to run scripts if for some reason activating poetry environment doesn't work properly. But that is not the case observed with these environments for now, so we need not use it.

Design

Identifier

Global poetry environments

Poetry environments doesn't have a .project file equivalent like Pipenv, using which we can locate the project for which the current environment is created; hence we can't say for sure whether it's poetry environment by just looking at the environment path.

• Our best guess is to use the environment name pattern which Global poetry environments follow, as evident from the code here:

<sanitized_env_name>-<cwd_hash>-py<major>.<micro>

Implementation details behind <sanitized_env_name> and <hash of cwd> are too much rely upon, so for our purposes the best we can do is verify if the pattern matches:

<anything>-<anything>-py<number>.<number>

• Should be a virtualenv/venv type environment.

If this doesn't work well, we can follow other alternatives mentioned in the spike.

Local poetry environments

• Environment folder is named .venv
• One of the parent directories contain pyproject.toml which contains a poetry section.
  • We can directly run poetry check to verify this assumption.
  • If running command for every .venv-named folder turns out to be expensive (probably won't be), look for pyproject.toml file in the current workspace folder. (the correct thing would be to look up all the parent directories as well, but that is expensive). If found, read the file and see if contains the string [tool.poetry].

Discovery

Location for poetry environments is decided by the virtualenvs.path setting. It defaults to {cache-dir}/virtualenvs where {cache-dir} is another setting with its own defaults. Poetry looks at the following entries in order to get a setting,

• Environment variable: Every setting key has an associated environment variable which begins with POETRY_ (more). For eg. for the virtualenvs.path setting the environment variable to look into is POETRY_VIRTUALENVS_PATH.
• Local config: Stored in poetry.toml which is located besides pyproject.toml.
• Global config: Stored in these locations, in a file named config.toml. Parse file using npm package toml to get the setting value.
• Default value

Unfortunately poetry CLI doesn't provide APIs individually query for these 3 types of configs.

We have two locators to search for all these locations for,

• Global poetry locator:

  • Env variable POETRY_VIRTUALENVS_PATH
  • Global virtualenvs.path setting

  Now we look at {cache-dir}/virtualenvs, possible {cache-dir} values are,

  • Env variable POETRY_CACHE_DIR
  • Global cache-dir setting
  • Default setting value based on OS

• Workspace poetry locator

  • Local virtualenvs.path setting
  • Local cache-dir setting
  • Looks into .venv folder if it exists

Use identifier, pyvenv.cfg files and other utils to get type, version and other details.

API: Each locator extends FSWatchingLocator for file watching, and uses the existing ILocator interface.

Installer

API: It uses the existing IModuleInstaller interface which exposes two methods:

• isSupported: Returns whether we can install dev tools using poetry for a particular resource. If the selected environment for the resource is the active poetry environment which was created for the resource, return true, otherwise return false. Steps,

  • Get the associated poetry environment for the resource using poetry env info -p command.
  • Verify if it's the same as the selected environment for the resource.

  Running command here is not expensive as we already have the extension activated, and this only happens after user-consent. But we can use other alternatives if for some reason this doesn't work out. Users can specify the poetry executable to use using the python.poetryPath setting.

• installModule: Use poetry add --dev <moduleName> to install dev tools. Add --allow-prereleases for packages for which released version is not available yet. (Example, black)

Special Considerations

We're depending on certain internal implementation details of poetry where appropriate poetry CLI is not available:

• The way global environment directories are named: <anything>-<anything>-py<number>.<number>.
• Location and name of global config file: config.toml.
• Location and name of local config file: poetry.toml.
• Default value of cache-dir setting.

All the locations are different per OS, with different environment variables to rely upon.

Telemetry and Experimentation

Except for poetry check, we do not run additional commands for discovery. So it should be just like locators and doesn't require specific telemetry. Success of the feature can be measured along with other locators using the pythonDiscoveryModule experiment.

Testing Strategy

For discovery, unit tests uses real files to mock the file system, set the environment variables, and stubs shell execution. The corresponding ILocator APIs are then called to verify if we're getting all the environments. For installation, we have unit tests to stub shell execution, active interpreter and verify the implementation.