IMPR: 'import datajoint' slow on Mac amd64 #1220
Description
Improvement Request
Problem
Not really a problem, but Mac import datajoint is way less performant comparing to import datajoint on Linux.
Requirements
# make a directory
mkdir test_dj_import
cd ~/Downloads/test_dj_import
## make import.py
echo "import datajoint" > import.py
## CLI
conda create -n dj313 -c defaults python=3.13
pip install datajoint snakeviz
# generate cProfiler result
python -m cProfile -o result.log import.py
# filter and sort result to top 20 longest cumulative cpu time - check cProfile doc
python -c 'import pstats; p=pstats.Stats("result.log");p.sort_stats("cumulative").print_stats(20)' > sorted_result.log
# if you want more
# visualize non filtered result.log, since can't visualize filtered result, the func recursive format was broken
pip install snakeviz
snakeviz result.log
# you will get an error saying too many depth, just change the filter to "depth: 3", "cutoff: 1/100"
Justification
On either Mac/amd64 or Linux/amd64, the results are in the same order, but it takes ~6 sec to import on Mac, but ~1 sec on Linux.
On Mac/amd64
# sort_result.log
cat sorted_result.log | grep .py
1 0.000 0.000 6.138 6.138 import.py:1(<module>)
1 0.000 0.000 6.136 6.136 /Users/yam/Documents/DataJoint/datajoint-python/datajoint/__init__.py:1(<module>)
1 0.000 0.000 3.010 3.010 /Users/yam/Documents/DataJoint/datajoint-python/datajoint/diagram.py:1(<module>)
1 0.000 0.000 2.531 2.531 /Users/yam/Documents/DataJoint/datajoint-python/datajoint/admin.py:1(<module>)
1 0.000 0.000 1.986 1.986 /Users/yam/Documents/DataJoint/datajoint-python/datajoint/connection.py:1(<module>)
1 0.000 0.000 1.543 1.543 /Users/yam/Documents/DataJoint/datajoint-python/datajoint/table.py:1(<module>)
1 0.000 0.000 1.416 1.416 /Users/yam/opt/miniconda3/envs/dj313/lib/python3.13/site-packages/pandas/__init__.py:1(<module>)
1 0.000 0.000 1.068 1.068 /Users/yam/opt/miniconda3/envs/dj313/lib/python3.13/site-packages/pandas/core/api.py:1(<module>)
On Linux/amd64
cat sorted_result.log | grep .py
1 0.000 0.000 1.058 1.058 import.py:1(<module>)
1 0.000 0.000 1.058 1.058 /home/drewyang/Documents/DataJoint/datajoint-python/datajoint/__init__.py:1(<module>)
1 0.000 0.000 0.574 0.574 /home/drewyang/Documents/DataJoint/datajoint-python/datajoint/diagram.py:1(<module>)
1 0.000 0.000 0.353 0.353 /home/drewyang/Documents/DataJoint/datajoint-python/datajoint/admin.py:1(<module>)
1 0.000 0.000 0.310 0.310 /home/drewyang/Documents/DataJoint/datajoint-python/datajoint/connection.py:1(<module>)
1 0.000 0.000 0.292 0.292 /home/drewyang/miniforge3/envs/dj313/lib/python3.13/site-packages/matplotlib/pyplot.py:1(<module>)
1 0.000 0.000 0.203 0.203 /home/drewyang/Documents/DataJoint/datajoint-python/datajoint/table.py:1(<module>)
106 0.002 0.000 0.185 0.002 /home/drewyang/miniforge3/envs/dj313/lib/python3.13/site-packages/matplotlib/artist.py:158(_update_set_signature_and_docstring)
105 0.000 0.000 0.182 0.002 /home/drewyang/miniforge3/envs/dj313/lib/python3.13/site-packages/matplotlib/artist.py:125(__init_subclass__)
1 0.000 0.000 0.169 0.169 /home/drewyang/miniforge3/envs/dj313/lib/python3.13/site-packages/pandas/__init__.py:1(<module>)
Please include steps to reproduce provided errors as follows:
- OS (MACOS | Linux)
- Python Version 3.13
- DataJoint Version 0.14.3
Screenshots
Snakeviz on Mac
