Skip to content

[Bug]: error encountered when trying to quantize model with modelslim #2078

@fataswellassad

Description

@fataswellassad

Your current environment

I know this isn't strictly vllm-ascend but it's heavily related.
I was trying to follow the instructions from https://vllm-ascend.readthedocs.io/en/latest/user_guide/feature_guide/quantization.html.

🐛 Describe the bug

However, when I run the script, (cmd: python3 msit/msmodelslim/example/Qwen/quant_qwen.py --model_path Qwen2.5-32B-Instruct --model_type qwen2 --save_directory Qwen2.5-32B-Instruct-W8A16 --a_bit 16 --w_bit 8 --tp_size 2), it shows:

Image

That file msmodelslim.pytorch.llm_ptq.anti_outlier.anti_utils doesn't even exist (for this tag at least). How do I work around this?

Thx in advance.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions