feat: lazy imports gapic v1#17673
Conversation
There was a problem hiding this comment.
Code Review
This pull request adds an import_profile test target to the CI script and attempts to implement lazy loading for gapic_v1 submodules using Python 3.15's __lazy_modules__ feature. However, a critical issue was identified: placing the imports inside a conditional block (sys.version_info < (3, 15)) prevents them from being executed at all on Python 3.15+, which will lead to AttributeErrors at runtime. To fix this, the imports should remain at the top level so that Python 3.15+ can lazily load them while older versions load them eagerly.
2439765 to
603236b
Compare
|
I'm going to switch this to draft until tests are added. Please mark this as ready for review once #17673 (review) is addressed |
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request implements PEP 0810 explicit lazy imports for Python 3.15+ in the gapic_v1 package and adds unit tests to verify both lazy and fallback eager import behaviors. Feedback points out a critical inconsistency when grpc is not installed: on Python 3.15+, the deferred imports prevent ImportError from being caught eagerly, resulting in missing dependencies being added to __all__ and causing runtime failures during wildcard imports. The reviewer suggests using importlib.util.find_spec to conditionally define lazy modules and imports based on the presence of grpc.
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request attempts to implement lazy imports in gapic_v1 using a hypothetical Python 3.15 feature (__lazy_modules__ / PEP 810), falling back to eager imports on older versions. However, because this feature and PEP do not exist, the implementation currently results in eager imports on all Python versions. The reviewer recommends implementing standard lazy loading using PEP 562 (__getattr__ and __dir__), which is compatible with all supported Python versions (3.7+). This approach would also allow the accompanying unit tests to run and verify lazy loading behavior across all environments instead of being skipped on versions below 3.15.
| from google.api_core.gapic_v1 import client_info # noqa: E402 | ||
| from google.api_core.gapic_v1 import routing_header # noqa: E402 | ||
|
|
||
| if _has_grpc: | ||
| from google.api_core.gapic_v1 import config # noqa: F401 | ||
| from google.api_core.gapic_v1 import config_async # noqa: F401 | ||
| from google.api_core.gapic_v1 import method # noqa: F401 | ||
| from google.api_core.gapic_v1 import method_async # noqa: F401 |
There was a problem hiding this comment.
why do we need these # noqa: E402 and # noqa: F401?
|
|
||
| import pytest | ||
|
|
||
| SCRIPT_PYTHON_315 = """ |
There was a problem hiding this comment.
Is there a better way to test this instead of using this script within a test file? Do we even want to test this?
| __lazy_modules__: Set[str] = { | ||
| "google.api_core.gapic_v1.client_info", | ||
| "google.api_core.gapic_v1.routing_header", | ||
| } | ||
|
|
||
| if _has_grpc: | ||
| __lazy_modules__.update( | ||
| { | ||
| "google.api_core.gapic_v1.config", | ||
| "google.api_core.gapic_v1.config_async", | ||
| "google.api_core.gapic_v1.method", | ||
| "google.api_core.gapic_v1.method_async", | ||
| } | ||
| ) | ||
|
|
||
| from google.api_core.gapic_v1 import client_info # noqa: E402 | ||
| from google.api_core.gapic_v1 import routing_header # noqa: E402 | ||
|
|
||
| if _has_grpc: | ||
| from google.api_core.gapic_v1 import config # noqa: F401 | ||
| from google.api_core.gapic_v1 import config_async # noqa: F401 | ||
| from google.api_core.gapic_v1 import method # noqa: F401 | ||
| from google.api_core.gapic_v1 import method_async # noqa: F401 | ||
|
|
||
| __all__ = [ | ||
| "client_info", | ||
| "routing_header", | ||
| "config", | ||
| "config_async", | ||
| "method", | ||
| "method_async", | ||
| ] | ||
| else: | ||
| __all__ = [ | ||
| "client_info", | ||
| "routing_header", | ||
| ] |
There was a problem hiding this comment.
| __lazy_modules__: Set[str] = { | |
| "google.api_core.gapic_v1.client_info", | |
| "google.api_core.gapic_v1.routing_header", | |
| } | |
| if _has_grpc: | |
| __lazy_modules__.update( | |
| { | |
| "google.api_core.gapic_v1.config", | |
| "google.api_core.gapic_v1.config_async", | |
| "google.api_core.gapic_v1.method", | |
| "google.api_core.gapic_v1.method_async", | |
| } | |
| ) | |
| from google.api_core.gapic_v1 import client_info # noqa: E402 | |
| from google.api_core.gapic_v1 import routing_header # noqa: E402 | |
| if _has_grpc: | |
| from google.api_core.gapic_v1 import config # noqa: F401 | |
| from google.api_core.gapic_v1 import config_async # noqa: F401 | |
| from google.api_core.gapic_v1 import method # noqa: F401 | |
| from google.api_core.gapic_v1 import method_async # noqa: F401 | |
| __all__ = [ | |
| "client_info", | |
| "routing_header", | |
| "config", | |
| "config_async", | |
| "method", | |
| "method_async", | |
| ] | |
| else: | |
| __all__ = [ | |
| "client_info", | |
| "routing_header", | |
| ] | |
| __lazy_modules__: Set[str] = { | |
| "google.api_core.gapic_v1.client_info", | |
| "google.api_core.gapic_v1.routing_header", | |
| } | |
| __all__ = ["client_info", "routing_header"] | |
| if _has_grpc: | |
| __lazy_modules__.update({ | |
| "google.api_core.gapic_v1.config", | |
| "google.api_core.gapic_v1.config_async", | |
| "google.api_core.gapic_v1.method", | |
| "google.api_core.gapic_v1.method_async", | |
| }) | |
| from google.api_core.gapic_v1 import client_info # noqa: E402 | |
| from google.api_core.gapic_v1 import routing_header # noqa: E402 | |
| if _has_grpc: | |
| from google.api_core.gapic_v1 import config # noqa: F401 | |
| from google.api_core.gapic_v1 import config_async # noqa: F401 | |
| from google.api_core.gapic_v1 import method # noqa: F401 | |
| from google.api_core.gapic_v1 import method_async # noqa: F401 | |
| __all__.extend(["config", "config_async", "method", "method_async"]) | |
Description
This PR initiates the rollout of our PEP 0810 lazy-loading architecture to google-api-core, starting with the
gapic_v1module.By applying Python 3.15's native explicit lazy imports, we defer the eager execution of inner modules (like
method.pyandconfig.py) which recursively pull in massive third-party C-extensions likegrpcio. This acts as the first step in addressing the high initialization latency and peak memory footprints we're seeing on Serverless cold-starts.Older Python runtimes (3.14 and below) safely ignore the
__lazy_modules__set and fall back to standard eager execution, meaning this introduces zero backwards compatibility risk.(Note: We use fully-qualified, hardcoded module strings rather than dynamic
__name__injection, and provide explicit type annotations (Set[str]), to ensure static analysis tools and type checkers can safely infer the proxies without complaints).Related Documents
Note: This PR was kept intentionally small for review speed. We will be executing an iterative rollout, extending this pattern to other heavy modules like
exceptionsandoperations_v1in fast follow-up PRs once this structural pattern is approved.