Python package incompatibilities, can't run pipeline

[Traubert]

With an unmodified requirements.txt I get import problems related to pytorch-lightning trying to import things from torchmetrics. Then if I downgrade pytorch-lightning to 0.6.0 as suggested in other discussions, I get

$ echo "Minulla on koira." | python3 tnpp_parse.py --conf models_fi_tdt_dia/pipelines.yaml parse_plaintext
/scratch/clarin/hardwick/Turku-neural-parser-pipeline/venv-tnpp/lib64/python3.9/site-packages/pytorch_lightning/core/decorators.py:65: LightningDeprecationWarning: The `@auto_move_data` decorator is deprecated in v1.3 and will be removed in v1.5. Please use `trainer.predict` instead for inference. The decorator was applied to `predict`
  rank_zero_deprecation(
INFO:root:Loading model from /scratch/clarin/hardwick/Turku-neural-parser-pipeline/models_fi_tdt_dia/Tagger/best.ckpt
/scratch/clarin/hardwick/Turku-neural-parser-pipeline/venv-tnpp/lib64/python3.9/site-packages/sklearn/base.py:288: UserWarning: Trying to unpickle estimator LabelEncoder from version 0.24.2 when using version 1.2.0. This might lead to breaking code or invalid results. Use at your own risk. For more info please refer to:
https://scikit-learn.org/stable/model_persistence.html#security-maintainability-limitations
  warnings.warn(
Lemmatizer device: cpu / -1
Waiting for input
Feeding final batch
Some weights of the model checkpoint at TurkuNLP/bert-base-finnish-cased-v1 were not used when initializing BertModel: ['cls.predictions.transform.dense.weight', 'cls.predictions.transform.LayerNorm.weight', 'cls.seq_relationship.weight', 'cls.seq_relationship.bias', 'cls.predictions.bias', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight']
- This IS expected if you are initializing BertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing BertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of the model checkpoint at TurkuNLP/bert-base-finnish-cased-v1 were not used when initializing BertModel: ['cls.predictions.transform.dense.weight', 'cls.predictions.transform.LayerNorm.weight', 'cls.seq_relationship.weight', 'cls.seq_relationship.bias', 'cls.predictions.bias', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight']
- This IS expected if you are initializing BertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing BertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Process ForkProcess-6:
Traceback (most recent call last):
  File "/usr/lib64/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/lib64/python3.9/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/scratch/clarin/hardwick/Turku-neural-parser-pipeline/tnparser/lemmatizer_mod.py", line 234, in launch
    lemmatizer=LemmatizerWrapper(args)
  File "/scratch/clarin/hardwick/Turku-neural-parser-pipeline/tnparser/lemmatizer_mod.py", line 224, in __init__
    self.lemmatizer_model.init_model(args)
  File "/scratch/clarin/hardwick/Turku-neural-parser-pipeline/tnparser/lemmatizer_mod.py", line 60, in init_model
    self.translator = self.build_my_translator(args.model, self.f_output, use_gpu=use_gpu, gpu_device=device, beam_size=args.beam_size, max_length=args.max_length)
  File "/scratch/clarin/hardwick/Turku-neural-parser-pipeline/tnparser/lemmatizer_mod.py", line 85, in build_my_translator
    fields, model, model_opt = self.load_model(model_name, use_gpu=use_gpu, gpu_device=gpu_device)
  File "/scratch/clarin/hardwick/Turku-neural-parser-pipeline/tnparser/lemmatizer_mod.py", line 68, in load_model
    checkpoint = torch.load(model, map_location=lambda storage, loc: storage)
  File "/scratch/clarin/hardwick/Turku-neural-parser-pipeline/venv-tnpp/lib64/python3.9/site-packages/torch/serialization.py", line 789, in load
    return _load(opened_zipfile, map_location, pickle_module, **pickle_load_args)
  File "/scratch/clarin/hardwick/Turku-neural-parser-pipeline/venv-tnpp/lib64/python3.9/site-packages/torch/serialization.py", line 1131, in _load
    result = unpickler.load()
  File "/scratch/clarin/hardwick/Turku-neural-parser-pipeline/venv-tnpp/lib64/python3.9/site-packages/torch/serialization.py", line 1124, in find_class
    return super().find_class(mod_name, name)
ModuleNotFoundError: No module named 'onmt.inputters.text_dataset'
Error: pipeline stage died with exit code 256: lemmatizer_mod --model /scratch/clarin/hardwick/Turku-neural-parser-pipeline/models_fi_tdt_dia/Lemmatizer/lemmatizer.pt
Error: pipeline stage died with exit code 15: bert512_mod --vocabfile TurkuNLP/bert-base-finnish-cased-v1 --max 400
Exception ignored in: <Finalize object, dead>
Traceback (most recent call last):
  File "/usr/lib64/python3.9/multiprocessing/util.py", line 224, in __call__
  File "/usr/lib64/python3.9/multiprocessing/queues.py", line 198, in _finalize_join
TypeError: 'NoneType' object is not callable

[Traubert]

Hm, does this in fact mean that it's the fi_tdt_dia model that's out of date?

[fginter]

There's been (once again) a backward-incompatible update of torch lightning. Definitely not the first time this happens. We'll check and fix this, but it might take a while while we are busy with deadlines. Sorry about that. If you are in an extreme hurry, do try the trankit parser, it is very good also for Finnish.

[kjalonen]

Great to know the team is looking into this - my work is stalled, too. I completely understand that this is not on the top of your list in a hurry and don't mean to be impolite; would you have an estimate on when you're able to get back to this?

[haamis]

There's a lot of moving parts here in terms of package versions.
I was able to get it running with python3 -m pip install opennmt-py==2.2.0 torchmetrics==0.6.2 after a fresh clone (as of time of writing) and install (i.e. pip install -r requirements.txt).

For reference, here's my pip freeze:

absl-py==1.4.0
aiohttp==3.8.3
aiosignal==1.3.1
async-timeout==4.0.2
attrs==22.2.0
cachetools==5.3.0
certifi==2022.12.7
charset-normalizer==2.1.1
click==8.1.3
colorama==0.4.6
ConfigArgParse==1.5.3
ctranslate2==3.4.0
diaparser==1.1.2
emoji==2.2.0
filelock==3.9.0
Flask==2.2.2
frozenlist==1.3.3
fsspec==2023.1.0
future==0.18.3
google-auth==2.16.0
google-auth-oauthlib==0.4.6
grpcio==1.51.1
huggingface-hub==0.12.0
idna==3.4
importlib-metadata==6.0.0
itsdangerous==2.1.2
Jinja2==3.1.2
joblib==1.2.0
lxml==4.9.2
Markdown==3.4.1
MarkupSafe==2.1.2
multidict==6.0.4
nltk==3.8.1
numpy==1.19.4
nvidia-cublas-cu11==11.10.3.66
nvidia-cuda-nvrtc-cu11==11.7.99
nvidia-cuda-runtime-cu11==11.7.99
nvidia-cudnn-cu11==8.5.0.96
oauthlib==3.2.2
OpenNMT-py==2.2.0
packaging==23.0
pkg_resources==0.0.0
portalocker==2.7.0
protobuf==3.20.3
pyasn1==0.4.8
pyasn1-modules==0.2.8
pyDeprecate==0.3.1
pyonmttok==1.36.0
pytorch-lightning==1.4.9
PyYAML==6.0
rapidfuzz==2.13.7
regex==2022.10.31
requests==2.28.2
requests-oauthlib==1.3.1
rsa==4.9
sacrebleu==2.3.1
scikit-learn==1.2.1
scipy==1.9.3
sentencepiece==0.1.97
six==1.16.0
stanza==1.4.2
tabulate==0.9.0
tensorboard==2.11.2
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.1
threadpoolctl==3.1.0
tokenizers==0.13.2
torch==1.13.1
torchmetrics==0.6.2
torchtext==0.5.0
tqdm==4.64.1
transformers==4.26.0
typing_extensions==4.4.0
ufal.udpipe==1.2.0.3
urllib3==1.26.14
waitress==2.1.2
Werkzeug==2.2.2
yarl==1.8.2
zipp==3.12.1

[tony-shepherd]

Also been hacking away to get package versions to agree with each other.

Synch'd all versions with the freeze posted by haamis above and got quite far but stuck on this one:
ModuleNotFoundError: No module named 'onmt.inputters.text_dataset'

Digging down through Libs/site-packages to the onmt module confirms that the text_dataset.py is not there (so it's not a path variable problem) and googling around returns no evidence of such a .py file anywhere.

I'll take a look at trankit as prompted by fginter (all I want is lemmatization) but any comments here appreciated

(--- full traceback below......)

Traceback (most recent call last):
File "C:\Users<username>\AppData\Local\Programs\Python\Python39\lib\multiprocessing\process.py", line 315, in _bootstrap
self.run()
File "C:\Users<username>\AppData\Local\Programs\Python\Python39\lib\multiprocessing\process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "C:\Users<username><repos_folder>\Turku-neural-parser-pipeline\tnparser\lemmatizer_mod.py", line 234, in launch
lemmatizer=LemmatizerWrapper(args)
File "C:\Users<username><repos_folder>\Turku-neural-parser-pipeline\tnparser\lemmatizer_mod.py", line 224, in init
self.lemmatizer_model.init_model(args)
File "C:\Users<username><repos_folder>\Turku-neural-parser-pipeline\tnparser\lemmatizer_mod.py", line 60, in init_model
self.translator = self.build_my_translator(args.model, self.f_output, use_gpu=use_gpu, gpu_device=device, beam_size=args.beam_size, max_length=args.max_length)
File "C:\Users<username><repos_folder>\Turku-neural-parser-pipeline\tnparser\lemmatizer_mod.py", line 85, in build_my_translator
fields, model, model_opt = self.load_model(model_name, use_gpu=use_gpu, gpu_device=gpu_device)
File "C:\Users<username><repos_folder>\Turku-neural-parser-pipeline\tnparser\lemmatizer_mod.py", line 68, in load_model
checkpoint = torch.load(model, map_location=lambda storage, loc: storage)
File "C:\Users<username>\AppData\Local\Programs\Python\Python39\lib\site-packages\torch\serialization.py", line 789, in load
return _load(opened_zipfile, map_location, pickle_module, **pickle_load_args)
File "C:\Users<username>\AppData\Local\Programs\Python\Python39\lib\site-packages\torch\serialization.py", line 1131, in _load
result = unpickler.load()
File "C:\Users<username>\AppData\Local\Programs\Python\Python39\lib\site-packages\torch\serialization.py", line 1124, in find_class
return super().find_class(mod_name, name)
ModuleNotFoundError: No module named 'onmt.inputters.text_dataset'

[perttu]

I managed to get this work when I run it in server mode using dockerfile. Tested with Python 3.9.17. Compiles also with Python 3.7 and 3.8. Won't work with Python 3.10 and 3.11 because of Numpy.

torch>=1.9
requests
flask
numpy==1.19.4
ufal.udpipe==1.2.0.3
pyyaml
configargparse
transformers
OpenNMT-py==2.2.0
git+https://github.com/TurkuNLP/diaparser.git@master
pytorch_lightning<1.5 
torchmetrics<=0.7.3
scikit-learn>=0.24

'onmt.inputters' errors refer to wrong version of OpenNMT-py