Stubs not found when installed in separate venvs

[brandonchinn178]

Bug Report

We're using mypy with Bazel, which installs dependencies into separate directories and manually places the directories on PYTHONPATH. But it seems like this doesn't play well with protobuf, probably because of the initial google/ directory. This only seems to happen with protobuf, can't reproduce with other libraries.

To Reproduce

class FooProto(google.protobuf.message.Message):
    x: str

@dataclasses.dataclass
class Foo:
    x: str = ""

google.protobuf.message.non_existent()

proto_to_dataclass: Foo = FooProto()
dataclass_to_proto: FooProto = Foo()

See README in https://github.com/brandonchinn178/mypy-repro-protobuf for more details

Expected Behavior

• Module has no attribute "non_existent"
• Incompatible types in assignment on proto_to_dataclass
• Incompatible types in assignment on dataclass_to_proto

Actual Behavior

When types-protobuf and protobuf are installed in the same venv, this works (whether mypy is in that environment or a different one). When they're in different venvs, this does not.

Your Environment

• Mypy version used: 1.18.1
• Mypy command-line flags:
• Mypy configuration options from mypy.ini (and other config files):
• Python version used:

[ofarukdogancay]

Won't this issue occur with any stub-only package that covers a namespace package? If so I think this issue should be closed since this is the intended consequence of PEP 561 + PEP 420 . Fixing it would require changes at the packaging/PEP level or Bazel integration, not a mypy update.

[brandonchinn178]

Can you elaborate on how those PEPs result in this behavior? The most relevant quote I can see is

If typecheckers identify a stub-only namespace package without the desired module in step 3, they should continue to step 4/5. Typecheckers should identify namespace packages by the absence of init.pyi. This allows different subpackages to independently opt for inline vs stub-only.

But that leaves the question of what "without the desired module" means; does it require the desired module be in the same environment, or can it be found in another directory in sys.path? If the latter is allowed, then it seems like this does fall under the scope of mypy's implementation handling it.

[ofarukdogancay]

Related, wording feels like it is recognized that different path entries for the packages' stub resolution will not be supported but it's not explicitly specified

But I agree, it is not clear to me now whether this should be a type checker responsibility or not.

[brandonchinn178]

Are you referring to language like

If you have installed typed packages in another Python installation or environment, mypy won’t automatically find them. One option is to install another copy of those packages in the environment in which you installed mypy.
[...]
Note that stub-only packages cannot be used with MYPYPATH. If you want mypy to find the package, it must be installed.

? Yeah, that language might imply that mypy only officially supports third-party deps and stubs installed in the same environment as mypy, however, mypy does respect sys.path in most cases (as seen in my repro); just not in the case of namespace packages, it seems.

[ofarukdogancay]

I guess you are right,

In the second example you append the second venv to sys path so its virtually the same as installing to the same environment.

I couldn't find an argument as to why the 3rd example shouldn't behave the same.

Edit: It's not virtually the same package metadata is not propagated.

[emmatyping]

I believe the issue is that protobuf is first in the PYTHONPATH. Mypy searches similarly (in some ways) to the interpreter at runtime, so the first result found (in this case probably a .py file from the protobuf package) for a given module is used. I am not able to test right now but I'd wager putting the stubs path before the protobuf path fixes the issue.